Skip to content

activation

Custom activation functions.

Classes

fastvideo.layers.activation.GeluAndMul 

GeluAndMul(approximate: str = 'none')

Bases: CustomOp

An activation function for GeGLU.

The function computes x -> GELU(x[:d]) * x[d:] where d = x.shape[-1] // 2.

Shapes

x: (batch_size, seq_len, 2 * d) or (num_tokens, 2 * d) return: (batch_size, seq_len, d) or (num_tokens, d)

Source code in fastvideo/layers/activation.py 
45
46
47
48
49
def __init__(self, approximate: str = "none"):
    super().__init__()
    self.approximate = approximate
    if approximate not in ("none", "tanh"):
        raise ValueError(f"Unknown approximate mode: {approximate}")

Methods:

fastvideo.layers.activation.GeluAndMul.forward_native 
forward_native(x: Tensor) -> Tensor

PyTorch-native implementation equivalent to forward().

Source code in fastvideo/layers/activation.py 
51
52
53
54
def forward_native(self, x: torch.Tensor) -> torch.Tensor:
    """PyTorch-native implementation equivalent to forward()."""
    d = x.shape[-1] // 2
    return F.gelu(x[..., :d], approximate=self.approximate) * x[..., d:]

fastvideo.layers.activation.NewGELU 

NewGELU()

Bases: CustomOp

Source code in fastvideo/layers/activation.py 
63
64
def __init__(self):
    super().__init__()

Methods:

fastvideo.layers.activation.NewGELU.forward_native 
forward_native(x: Tensor) -> Tensor

PyTorch-native implementation equivalent to forward().

Source code in fastvideo/layers/activation.py 
66
67
68
69
def forward_native(self, x: torch.Tensor) -> torch.Tensor:
    """PyTorch-native implementation equivalent to forward()."""
    c = math.sqrt(2.0 / math.pi)
    return 0.5 * x * (1.0 + torch.tanh(c * (x + 0.044715 * torch.pow(x, 3.0))))

fastvideo.layers.activation.QuickGELU 

QuickGELU()

Bases: CustomOp

Source code in fastvideo/layers/activation.py 
75
76
def __init__(self):
    super().__init__()

Methods:

fastvideo.layers.activation.QuickGELU.forward_native 
forward_native(x: Tensor) -> Tensor

PyTorch-native implementation equivalent to forward().

Source code in fastvideo/layers/activation.py 
78
79
80
def forward_native(self, x: torch.Tensor) -> torch.Tensor:
    """PyTorch-native implementation equivalent to forward()."""
    return x * torch.sigmoid(1.702 * x)

fastvideo.layers.activation.SiluAndMul 

SiluAndMul()

Bases: CustomOp

An activation function for SwiGLU.

The function computes x -> silu(x[:d]) * x[d:] where d = x.shape[-1] // 2.

Shapes

x: (num_tokens, 2 * d) or (batch_size, seq_len, 2 * d) return: (num_tokens, d) or (batch_size, seq_len, d)

Source code in fastvideo/layers/activation.py 
25
26
def __init__(self) -> None:
    super().__init__()

Methods:

fastvideo.layers.activation.SiluAndMul.forward_native 
forward_native(x: Tensor) -> Tensor

PyTorch-native implementation equivalent to forward().

Source code in fastvideo/layers/activation.py 
28
29
30
31
def forward_native(self, x: torch.Tensor) -> torch.Tensor:
    """PyTorch-native implementation equivalent to forward()."""
    d = x.shape[-1] // 2
    return F.silu(x[..., :d]) * x[..., d:]

Functions:

fastvideo.layers.activation.get_act_and_mul_fn 

get_act_and_mul_fn(act_fn_name: str) -> Module

Get an activation-and-mul (i.e. SiluAndMul) function by name.

Source code in fastvideo/layers/activation.py 
109
110
111
112
113
114
115
def get_act_and_mul_fn(act_fn_name: str) -> nn.Module:
    """Get an activation-and-mul (i.e. SiluAndMul) function by name."""
    act_fn_name = act_fn_name.lower()
    if act_fn_name not in _ACTIVATION_AND_MUL_REGISTRY:
        raise ValueError(f"Activation function {act_fn_name!r} is not supported.")

    return _ACTIVATION_AND_MUL_REGISTRY[act_fn_name]()

fastvideo.layers.activation.get_act_fn 

get_act_fn(act_fn_name: str) -> Module

Get an activation function by name.

Source code in fastvideo/layers/activation.py 
 94
 95
 96
 97
 98
 99
100
def get_act_fn(act_fn_name: str) -> nn.Module:
    """Get an activation function by name."""
    act_fn_name = act_fn_name.lower()
    if act_fn_name not in _ACTIVATION_REGISTRY:
        raise ValueError(f"Activation function {act_fn_name!r} is not supported.")

    return _ACTIVATION_REGISTRY[act_fn_name]()