3.1.22.18. unit_scaling.functional.silu_glu

unit_scaling.functional.silu_glu(input: Tensor, gate: Tensor, mult: float = 1.0) → Tensor

Applies a unit-scaled gated linear unit for input * silu(gate).

$$\text{silu\_glu}(x,g)=x\ast g\ast \sigma (g),\text{where }\sigma (g)\text{ is the logistic sigmoid.}$$

Parameters:

• input (Tensor) – linear input

• gate (Tensor) – gate (SiLU) input

• mult (float, optional) – a multiplier to be applied to change the shape of a nonlinear function. Typically, high multipliers (> 1) correspond to a ‘sharper’ (low temperature) function, while low multipliers (< 1) correspond to a ‘flatter’ (high temperature) function.

Returns:

a scaled output, the same shape as input

Return type:

Tensor