--- title: siglip ---

SigLIP Vision Encoder for FastVideo.

SigLIP (Sigmoid Loss for Language-Image Pre-training) is similar to CLIP but uses sigmoid loss instead of contrastive loss, and doesn't use a CLS token.

fastvideo.models.encoders.siglip.LocalAttention

LocalAttention(num_heads: int, head_size: int, num_kv_heads: int | None = None, softmax_scale: float | None = None, causal: bool = False, supported_attention_backends: tuple[AttentionBackendEnum, ...] | None = None, default_backend: AttentionBackendEnum | None = None, **extra_impl_args)

Attention layer.

fastvideo.models.encoders.siglip.LocalAttention.forward

forward(q: torch.Tensor, k: torch.Tensor, v: torch.Tensor, freqs_cis: tuple[torch.Tensor, torch.Tensor] | None = None) -> torch.Tensor

Apply local attention between query, key and value tensors.

Parameters

q: torch.Tensor

Query tensor of shape [batch_size, seq_len, num_heads, head_dim]

k: torch.Tensor

Key tensor of shape [batch_size, seq_len, num_heads, head_dim]

v: torch.Tensor

Value tensor of shape [batch_size, seq_len, num_heads, head_dim]

Returns

torch.Tensor

torch.Tensor: Output tensor after local attention

fastvideo.models.encoders.siglip.BaseEncoderOutput

BaseEncoderOutput(last_hidden_state: torch.FloatTensor | None = None, pooler_output: torch.FloatTensor | None = None, hidden_states: tuple[torch.FloatTensor, ...] | None = None, attentions: tuple[torch.FloatTensor, ...] | None = None, attention_mask: torch.Tensor | None = None)

fastvideo.models.encoders.siglip.SiglipVisionArchConfig

SiglipVisionArchConfig(stacked_params_mapping: list = (lambda: [('qkv_proj', 'q_proj', 'q'), ('qkv_proj', 'k_proj', 'k'), ('qkv_proj', 'v_proj', 'v')])(), architectures: list[str] = (lambda: ['SiglipVisionModel'])(), _supported_attention_backends: tuple[AttentionBackendEnum, ...] = (AttentionBackendEnum.FLASH_ATTN, AttentionBackendEnum.TORCH_SDPA), output_hidden_states: bool = False, use_return_dict: bool = True, attention_dropout: float = 0.0, dtype: str | None = None, hidden_act: str = 'gelu_pytorch_tanh', hidden_size: int = 1152, image_size: int = 384, intermediate_size: int = 4304, layer_norm_eps: float = 1e-06, model_type: str = 'siglip_vision_model', num_attention_heads: int = 16, num_channels: int = 3, num_hidden_layers: int = 27, patch_size: int = 14)

Architecture configuration for SigLIP vision encoder.

Fields match the config.json from HuggingFace SigLIP checkpoints.

fastvideo.models.encoders.siglip.SiglipVisionConfig

SiglipVisionConfig(arch_config: ImageEncoderArchConfig = SiglipVisionArchConfig(), prefix: str = 'siglip', quant_config: QuantizationConfig | None = None, lora_config: Any | None = None, num_hidden_layers_override: int | None = None, require_post_norm: bool | None = None, enable_scale: bool = True, is_causal: bool = False, *, _resolved_attention_backend: AttentionBackendEnum | None = None)

Configuration for SigLIP vision encoder.

fastvideo.models.encoders.siglip.divide

divide(numerator: int, denominator: int) -> int

Ensure that numerator is divisible by the denominator and return the division value.

fastvideo.models.encoders.siglip.get_tp_world_size

get_tp_world_size() -> int

Return world size for the tensor model parallel group.

fastvideo.models.encoders.siglip.get_act_fn

get_act_fn(act_fn_name: str) -> nn.Module

Get an activation function by name.

fastvideo.models.encoders.siglip.ColumnParallelLinear

ColumnParallelLinear(input_size: int, output_size: int, bias: bool = True, gather_output: bool = False, skip_bias_add: bool = False, params_dtype: torch.dtype | None = None, quant_config: QuantizationConfig | None = None, output_sizes: list[int] | None = None, prefix: str = '')

Linear layer with column parallelism.

The linear layer is defined as Y = XA + b. A is parallelized along its second dimension as A = [A_1, ..., A_p].

Parameters

input_size: int

first dimension of matrix A.

output_size: int

second dimension of matrix A.

bias: bool

If true, add bias.

gather_output: bool

If true, call all-gather on output and make Y available to all GPUs, otherwise, every GPU will have its output which is Y_i = XA_i

skip_bias_add: bool

This was added to enable performance optimizations where bias can be fused with other element-wise operations. we skip adding bias but instead return it.

params_dtype: torch.dtype | None

Data type for the parameters.

quant_config: QuantizationConfig | None

Quantization configure.

output_sizes: list[int] | None

list of output sizes packed into one output, like for QKV the list would be size 3.

prefix: str

The name of the layer in the state dict, including all parents (e.g. model.layers.0.qkv_proj)

fastvideo.models.encoders.siglip.ColumnParallelLinear.weight_loader

weight_loader(param: Parameter, loaded_weight: torch.Tensor) -> None

fastvideo.models.encoders.siglip.ColumnParallelLinear.weight_loader_v2

weight_loader_v2(param: Parameter, loaded_weight: torch.Tensor) -> None

fastvideo.models.encoders.siglip.ColumnParallelLinear.forward

forward(input_: torch.Tensor) -> tuple[torch.Tensor, Parameter | None]

fastvideo.models.encoders.siglip.ColumnParallelLinear.extra_repr

extra_repr() -> str

fastvideo.models.encoders.siglip.QKVParallelLinear

QKVParallelLinear(hidden_size: int, head_size: int, total_num_heads: int, total_num_kv_heads: int | None = None, bias: bool = True, skip_bias_add: bool = False, params_dtype: torch.dtype | None = None, quant_config: QuantizationConfig | None = None, prefix: str = '')

Linear layers for the attention's QKV transformation.

Linear layers for the linear transformation of the query, key, and value vectors in the attention layer. The weight matrix is concatenated along the output dimension. The layer is parallelized along the head dimension. When the number of key/value heads is smaller than the number of query heads (e.g., multi-query/grouped-query attention), the key/value head may be replicated while the query heads are partitioned.

Parameters

hidden_size: int

input hidden state size of the transformer.

head_size: int

size of each attention head.

total_num_heads: int

total number of attention query heads.

total_num_kv_heads: int | None

total number of attention key/value heads. If None, assume total_num_kv_heads = total_num_heads.

bias: bool

If true, add bias.

skip_bias_add: bool

This was added to enable performance optimizations where bias can be fused with other element-wise operations. we skip adding bias but instead return it.

params_dtype: torch.dtype | None

Data type for the parameters.

quant_config: QuantizationConfig | None

Quantization configure.

prefix: str

The name of the layer in the state dict, including all parents (e.g. model.layers.0.qkv_proj)

fastvideo.models.encoders.siglip.QKVParallelLinear.weight_loader_v2

weight_loader_v2(param: BasevLLMParameter, loaded_weight: torch.Tensor, loaded_shard_id: str | None = None)

fastvideo.models.encoders.siglip.QKVParallelLinear.weight_loader

weight_loader(param: Parameter, loaded_weight: torch.Tensor, loaded_shard_id: str | None = None)

fastvideo.models.encoders.siglip.RowParallelLinear

RowParallelLinear(input_size: int, output_size: int, bias: bool = True, input_is_parallel: bool = True, skip_bias_add: bool = False, params_dtype: torch.dtype | None = None, reduce_results: bool = True, quant_config: QuantizationConfig | None = None, prefix: str = '')

Linear layer with row parallelism.

The linear layer is defined as Y = XA + b. A is parallelized along its first dimension and X along its second dimension as: - - | A_1 | | . | A = | . | X = [X_1, ..., X_p] | . | | A_p | - - Arguments: input_size: first dimension of matrix A. output_size: second dimension of matrix A. bias: If true, add bias. Note that bias is not parallelized. input_is_parallel: If true, we assume that the input is already split across the GPUs and we do not split again. skip_bias_add: This was added to enable performance optimization where bias can be fused with other element-wise operations. We skip adding bias but instead return it. params_dtype: Data type for the parameters. quant_config: Quantization configure.

fastvideo.models.encoders.siglip.RowParallelLinear.weight_loader

weight_loader(param: Parameter, loaded_weight: torch.Tensor)

fastvideo.models.encoders.siglip.RowParallelLinear.weight_loader_v2

weight_loader_v2(param: BasevLLMParameter, loaded_weight: torch.Tensor)

fastvideo.models.encoders.siglip.RowParallelLinear.forward

forward(input_) -> tuple[torch.Tensor, Parameter | None]

fastvideo.models.encoders.siglip.RowParallelLinear.extra_repr

extra_repr() -> str

fastvideo.models.encoders.siglip.QuantizationConfig

QuantizationConfig()

Base class for quantization configs.

fastvideo.models.encoders.siglip.QuantizationConfig.get_name

get_name() -> QuantizationMethods

Name of the quantization method.

fastvideo.models.encoders.siglip.QuantizationConfig.get_supported_act_dtypes

get_supported_act_dtypes() -> list[torch.dtype]

List of supported activation dtypes.

fastvideo.models.encoders.siglip.QuantizationConfig.get_min_capability

get_min_capability() -> int

Minimum GPU capability to support the quantization method.

E.g., 70 for Volta, 75 for Turing, 80 for Ampere. This requirement is due to the custom CUDA kernels used by the quantization method.

fastvideo.models.encoders.siglip.QuantizationConfig.get_config_filenames

get_config_filenames() -> list[str]

List of filenames to search for in the model directory.

fastvideo.models.encoders.siglip.QuantizationConfig.from_config

from_config(config: dict[str, Any]) -> QuantizationConfig

Create a config class from the model's quantization config.

fastvideo.models.encoders.siglip.QuantizationConfig.override_quantization_method

override_quantization_method(hf_quant_cfg, user_quant) -> QuantizationMethods | None

Detects if this quantization method can support a given checkpoint format by overriding the user specified quantization method -- this method should only be overwritten by subclasses in exceptional circumstances

fastvideo.models.encoders.siglip.QuantizationConfig.get_from_keys

get_from_keys(config: dict[str, Any], keys: list[str]) -> Any

Get a value from the model's quantization config.

fastvideo.models.encoders.siglip.QuantizationConfig.get_from_keys_or

get_from_keys_or(config: dict[str, Any], keys: list[str], default: Any) -> Any

Get a optional value from the model's quantization config.

fastvideo.models.encoders.siglip.QuantizationConfig.get_quant_method

get_quant_method(layer: torch.nn.Module, prefix: str) -> QuantizeMethodBase | None

Get the quantize method to use for the quantized layer.

Parameters

layer: torch.nn.Module

The layer for the quant method.

prefix: str

The full name of the layer in the state dict

Returns: The quantize method. None if the given layer doesn't support quant method.

fastvideo.models.encoders.siglip.QuantizationConfig.validate_runtime

validate_runtime(device: torch.device) -> None

Fail fast on a device the scheme cannot execute on.

Called once by loaders that select a serialized checkpoint scheme, before any weight is read. The default accepts every device; schemes with kernel or capability requirements override it.

fastvideo.models.encoders.siglip.QuantizationConfig.get_cache_scale

get_cache_scale(name: str) -> str | None

fastvideo.models.encoders.siglip.init_logger

init_logger(name: str) -> _FastvideoLogger

The main purpose of this function is to ensure that loggers are retrieved in such a way that we can be sure the root fastvideo logger has already been configured.

fastvideo.models.encoders.siglip.ImageEncoder

ImageEncoder(config: ImageEncoderConfig)

fastvideo.models.encoders.siglip.ImageEncoder.forward

forward(pixel_values: torch.Tensor, **kwargs) -> BaseEncoderOutput

fastvideo.models.encoders.siglip.default_weight_loader

default_weight_loader(param: torch.Tensor, loaded_weight: torch.Tensor) -> None

Default weight loader.

fastvideo.models.encoders.siglip.SiglipVisionEmbeddings

SiglipVisionEmbeddings(arch_config: SiglipVisionArchConfig)

SigLIP vision embeddings - similar to CLIP but without class embedding.

Source
class SiglipVisionEmbeddings(nn.Module):
    """
    SigLIP vision embeddings - similar to CLIP but without class embedding.
    """

    def __init__(self, arch_config: SiglipVisionArchConfig):
        super().__init__()
        self.arch_config = arch_config
        self.embed_dim = arch_config.hidden_size
        self.image_size = arch_config.image_size
        self.patch_size = arch_config.patch_size
        # SigLIP uses valid padding, so non-divisible sizes work (edge pixels are ignored)

        self.patch_embedding = nn.Conv2d(
            in_channels=arch_config.num_channels,
            out_channels=self.embed_dim,
            kernel_size=self.patch_size,
            stride=self.patch_size,
            padding="valid",  # SigLIP uses valid padding
        )

        # Integer division - with valid padding, edge pixels are ignored
        self.num_patches = (self.image_size // self.patch_size)**2
        self.num_positions = self.num_patches
        self.position_embedding = nn.Embedding(self.num_positions, self.embed_dim)
        self.register_buffer(
            "position_ids",
            torch.arange(self.num_positions).expand((1, -1)),
            persistent=False,
        )

    def forward(self, pixel_values: torch.Tensor) -> torch.Tensor:
        target_dtype = self.patch_embedding.weight.dtype
        patch_embeds = self.patch_embedding(pixel_values.to(dtype=target_dtype))  # shape = [*, embed_dim, grid, grid]
        embeddings = patch_embeds.flatten(2).transpose(1, 2)
        embeddings = embeddings + self.position_embedding(self.position_ids)
        return embeddings

fastvideo.models.encoders.siglip.SiglipVisionEmbeddings.forward

forward(pixel_values: torch.Tensor) -> torch.Tensor
Source
def forward(self, pixel_values: torch.Tensor) -> torch.Tensor:
    target_dtype = self.patch_embedding.weight.dtype
    patch_embeds = self.patch_embedding(pixel_values.to(dtype=target_dtype))  # shape = [*, embed_dim, grid, grid]
    embeddings = patch_embeds.flatten(2).transpose(1, 2)
    embeddings = embeddings + self.position_embedding(self.position_ids)
    return embeddings

fastvideo.models.encoders.siglip.SiglipAttention

SiglipAttention(arch_config: SiglipVisionArchConfig, enable_scale: bool = True, is_causal: bool = False, quant_config: QuantizationConfig | None = None, prefix: str = '')

Multi-headed attention for SigLIP.

Source
class SiglipAttention(nn.Module):
    """Multi-headed attention for SigLIP."""

    def __init__(
        self,
        arch_config: SiglipVisionArchConfig,
        enable_scale: bool = True,
        is_causal: bool = False,
        quant_config: QuantizationConfig | None = None,
        prefix: str = "",
    ):
        super().__init__()
        self.arch_config = arch_config
        self.embed_dim = arch_config.hidden_size
        self.num_heads = arch_config.num_attention_heads
        self.head_dim = self.embed_dim // self.num_heads

        if self.head_dim * self.num_heads != self.embed_dim:
            raise ValueError(f"embed_dim must be divisible by num_heads "
                             f"(got `embed_dim`: {self.embed_dim} and `num_heads`: {self.num_heads}).")

        self.scale = self.head_dim**-0.5 if enable_scale else None
        self.dropout = arch_config.attention_dropout

        self.qkv_proj = QKVParallelLinear(
            hidden_size=self.embed_dim,
            head_size=self.head_dim,
            total_num_heads=self.num_heads,
            quant_config=quant_config,
            prefix=f"{prefix}.qkv_proj",
        )

        self.out_proj = RowParallelLinear(
            input_size=self.embed_dim,
            output_size=self.embed_dim,
            quant_config=quant_config,
            prefix=f"{prefix}.out_proj",
        )

        self.tp_size = get_tp_world_size()
        self.num_heads_per_partition = divide(self.num_heads, self.tp_size)

        self.attn = LocalAttention(
            self.num_heads_per_partition,
            self.head_dim,
            self.num_heads_per_partition,
            softmax_scale=self.scale,
            causal=is_causal,
            supported_attention_backends=arch_config._supported_attention_backends,
        )

    def forward(self, hidden_states: torch.Tensor):
        """Input shape: Batch x Time x Channel"""
        qkv_states, _ = self.qkv_proj(hidden_states)
        query_states, key_states, value_states = qkv_states.chunk(3, dim=-1)

        query_states = query_states.reshape(query_states.shape[0], query_states.shape[1], self.num_heads_per_partition,
                                            self.head_dim)
        key_states = key_states.reshape(key_states.shape[0], key_states.shape[1], self.num_heads_per_partition,
                                        self.head_dim)
        value_states = value_states.reshape(value_states.shape[0], value_states.shape[1], self.num_heads_per_partition,
                                            self.head_dim)

        attn_output = self.attn(query_states, key_states, value_states)
        attn_output = attn_output.reshape(attn_output.shape[0], attn_output.shape[1],
                                          self.num_heads_per_partition * self.head_dim)
        attn_output, _ = self.out_proj(attn_output)
        return attn_output, None

fastvideo.models.encoders.siglip.SiglipAttention.forward

forward(hidden_states: torch.Tensor)

Input shape: Batch x Time x Channel

Source
def forward(self, hidden_states: torch.Tensor):
    """Input shape: Batch x Time x Channel"""
    qkv_states, _ = self.qkv_proj(hidden_states)
    query_states, key_states, value_states = qkv_states.chunk(3, dim=-1)

    query_states = query_states.reshape(query_states.shape[0], query_states.shape[1], self.num_heads_per_partition,
                                        self.head_dim)
    key_states = key_states.reshape(key_states.shape[0], key_states.shape[1], self.num_heads_per_partition,
                                    self.head_dim)
    value_states = value_states.reshape(value_states.shape[0], value_states.shape[1], self.num_heads_per_partition,
                                        self.head_dim)

    attn_output = self.attn(query_states, key_states, value_states)
    attn_output = attn_output.reshape(attn_output.shape[0], attn_output.shape[1],
                                      self.num_heads_per_partition * self.head_dim)
    attn_output, _ = self.out_proj(attn_output)
    return attn_output, None

fastvideo.models.encoders.siglip.SiglipMLP

SiglipMLP(arch_config: SiglipVisionArchConfig, quant_config: QuantizationConfig | None = None, prefix: str = '')
Source
class SiglipMLP(nn.Module):

    def __init__(
        self,
        arch_config: SiglipVisionArchConfig,
        quant_config: QuantizationConfig | None = None,
        prefix: str = "",
    ) -> None:
        super().__init__()
        self.arch_config = arch_config
        self.activation_fn = get_act_fn(arch_config.hidden_act)
        self.fc1 = ColumnParallelLinear(
            arch_config.hidden_size,
            arch_config.intermediate_size,
            bias=True,
            quant_config=quant_config,
            prefix=f"{prefix}.fc1",
        )
        self.fc2 = RowParallelLinear(
            arch_config.intermediate_size,
            arch_config.hidden_size,
            bias=True,
            quant_config=quant_config,
            prefix=f"{prefix}.fc2",
        )

    def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
        hidden_states, _ = self.fc1(hidden_states)
        hidden_states = self.activation_fn(hidden_states)
        hidden_states, _ = self.fc2(hidden_states)
        return hidden_states

fastvideo.models.encoders.siglip.SiglipMLP.forward

forward(hidden_states: torch.Tensor) -> torch.Tensor
Source
def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
    hidden_states, _ = self.fc1(hidden_states)
    hidden_states = self.activation_fn(hidden_states)
    hidden_states, _ = self.fc2(hidden_states)
    return hidden_states

fastvideo.models.encoders.siglip.SiglipEncoderLayer

SiglipEncoderLayer(arch_config: SiglipVisionArchConfig, enable_scale: bool = True, is_causal: bool = False, quant_config: QuantizationConfig | None = None, prefix: str = '')
Source
class SiglipEncoderLayer(nn.Module):

    def __init__(
        self,
        arch_config: SiglipVisionArchConfig,
        enable_scale: bool = True,
        is_causal: bool = False,
        quant_config: QuantizationConfig | None = None,
        prefix: str = "",
    ) -> None:
        super().__init__()
        self.self_attn = SiglipAttention(
            arch_config,
            enable_scale=enable_scale,
            is_causal=is_causal,
            quant_config=quant_config,
            prefix=f"{prefix}.self_attn",
        )
        self.layer_norm1 = nn.LayerNorm(arch_config.hidden_size, eps=arch_config.layer_norm_eps)
        self.mlp = SiglipMLP(
            arch_config,
            quant_config=quant_config,
            prefix=f"{prefix}.mlp",
        )
        self.layer_norm2 = nn.LayerNorm(arch_config.hidden_size, eps=arch_config.layer_norm_eps)

    def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
        # SigLIP uses post-norm (like original ViT)
        residual = hidden_states
        hidden_states = self.layer_norm1(hidden_states)
        hidden_states, _ = self.self_attn(hidden_states=hidden_states)
        hidden_states = residual + hidden_states

        residual = hidden_states
        hidden_states = self.layer_norm2(hidden_states)
        hidden_states = self.mlp(hidden_states)
        hidden_states = residual + hidden_states

        return hidden_states

fastvideo.models.encoders.siglip.SiglipEncoderLayer.forward

forward(hidden_states: torch.Tensor) -> torch.Tensor
Source
def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
    # SigLIP uses post-norm (like original ViT)
    residual = hidden_states
    hidden_states = self.layer_norm1(hidden_states)
    hidden_states, _ = self.self_attn(hidden_states=hidden_states)
    hidden_states = residual + hidden_states

    residual = hidden_states
    hidden_states = self.layer_norm2(hidden_states)
    hidden_states = self.mlp(hidden_states)
    hidden_states = residual + hidden_states

    return hidden_states

fastvideo.models.encoders.siglip.SiglipEncoder

SiglipEncoder(arch_config: SiglipVisionArchConfig, enable_scale: bool = True, is_causal: bool = False, quant_config: QuantizationConfig | None = None, num_hidden_layers_override: int | None = None, prefix: str = '')

SigLIP encoder consisting of transformer layers.

Source
class SiglipEncoder(nn.Module):
    """SigLIP encoder consisting of transformer layers."""

    def __init__(
        self,
        arch_config: SiglipVisionArchConfig,
        enable_scale: bool = True,
        is_causal: bool = False,
        quant_config: QuantizationConfig | None = None,
        num_hidden_layers_override: int | None = None,
        prefix: str = "",
    ) -> None:
        super().__init__()
        self.arch_config = arch_config

        if num_hidden_layers_override is None:
            num_hidden_layers = arch_config.num_hidden_layers
        else:
            num_hidden_layers = num_hidden_layers_override

        self.layers = nn.ModuleList([
            SiglipEncoderLayer(
                arch_config=arch_config,
                enable_scale=enable_scale,
                is_causal=is_causal,
                quant_config=quant_config,
                prefix=f"{prefix}.layers.{layer_idx}",
            ) for layer_idx in range(num_hidden_layers)
        ])

    def forward(
        self,
        inputs_embeds: torch.Tensor,
        return_all_hidden_states: bool,
    ) -> torch.Tensor | list[torch.Tensor]:
        hidden_states_pool = [inputs_embeds]
        hidden_states = inputs_embeds

        for encoder_layer in self.layers:
            hidden_states = encoder_layer(hidden_states)
            if return_all_hidden_states:
                hidden_states_pool.append(hidden_states)

        if return_all_hidden_states:
            return hidden_states_pool
        return [hidden_states]

fastvideo.models.encoders.siglip.SiglipEncoder.forward

forward(inputs_embeds: torch.Tensor, return_all_hidden_states: bool) -> torch.Tensor | list[torch.Tensor]
Source
def forward(
    self,
    inputs_embeds: torch.Tensor,
    return_all_hidden_states: bool,
) -> torch.Tensor | list[torch.Tensor]:
    hidden_states_pool = [inputs_embeds]
    hidden_states = inputs_embeds

    for encoder_layer in self.layers:
        hidden_states = encoder_layer(hidden_states)
        if return_all_hidden_states:
            hidden_states_pool.append(hidden_states)

    if return_all_hidden_states:
        return hidden_states_pool
    return [hidden_states]

fastvideo.models.encoders.siglip.SiglipVisionTransformer

SiglipVisionTransformer(config: SiglipVisionConfig, quant_config: QuantizationConfig | None = None, num_hidden_layers_override: int | None = None, require_post_norm: bool | None = None, prefix: str = '')
Source
class SiglipVisionTransformer(nn.Module):

    def __init__(
        self,
        config: SiglipVisionConfig,
        quant_config: QuantizationConfig | None = None,
        num_hidden_layers_override: int | None = None,
        require_post_norm: bool | None = None,
        prefix: str = "",
    ) -> None:
        super().__init__()
        self.config = config
        arch_config = config.arch_config
        embed_dim = arch_config.hidden_size

        self.embeddings = SiglipVisionEmbeddings(arch_config)

        self.encoder = SiglipEncoder(
            arch_config=arch_config,
            enable_scale=config.enable_scale,
            is_causal=config.is_causal,
            quant_config=quant_config,
            num_hidden_layers_override=num_hidden_layers_override,
            prefix=f"{prefix}.encoder",
        )

        num_hidden_layers = arch_config.num_hidden_layers
        if len(self.encoder.layers) > arch_config.num_hidden_layers:
            raise ValueError(f"The original encoder only has {num_hidden_layers} "
                             f"layers, but you requested {len(self.encoder.layers)} layers.")

        # Post layer norm (applied to output)
        if require_post_norm is None:
            require_post_norm = len(self.encoder.layers) == num_hidden_layers

        if require_post_norm:
            self.post_layernorm = nn.LayerNorm(embed_dim, eps=arch_config.layer_norm_eps)
        else:
            self.post_layernorm = None

    def forward(
        self,
        pixel_values: torch.Tensor,
        feature_sample_layers: list[int] | None = None,
    ) -> torch.Tensor:
        hidden_states = self.embeddings(pixel_values)

        return_all_hidden_states = feature_sample_layers is not None
        encoder_outputs = self.encoder(
            inputs_embeds=hidden_states,
            return_all_hidden_states=return_all_hidden_states,
        )

        if not return_all_hidden_states:
            encoder_outputs = encoder_outputs[0]

        # Apply post-layernorm
        if self.post_layernorm is not None:
            if isinstance(encoder_outputs, list):
                encoder_outputs[-1] = self.post_layernorm(encoder_outputs[-1])
            else:
                encoder_outputs = self.post_layernorm(encoder_outputs)

        return encoder_outputs

fastvideo.models.encoders.siglip.SiglipVisionTransformer.forward

forward(pixel_values: torch.Tensor, feature_sample_layers: list[int] | None = None) -> torch.Tensor
Source
def forward(
    self,
    pixel_values: torch.Tensor,
    feature_sample_layers: list[int] | None = None,
) -> torch.Tensor:
    hidden_states = self.embeddings(pixel_values)

    return_all_hidden_states = feature_sample_layers is not None
    encoder_outputs = self.encoder(
        inputs_embeds=hidden_states,
        return_all_hidden_states=return_all_hidden_states,
    )

    if not return_all_hidden_states:
        encoder_outputs = encoder_outputs[0]

    # Apply post-layernorm
    if self.post_layernorm is not None:
        if isinstance(encoder_outputs, list):
            encoder_outputs[-1] = self.post_layernorm(encoder_outputs[-1])
        else:
            encoder_outputs = self.post_layernorm(encoder_outputs)

    return encoder_outputs

fastvideo.models.encoders.siglip.SiglipVisionModel

SiglipVisionModel(config: SiglipVisionConfig)

SigLIP Vision Model for FastVideo.

Source
class SiglipVisionModel(ImageEncoder):
    """SigLIP Vision Model for FastVideo."""

    config_class = SiglipVisionConfig
    main_input_name = "pixel_values"
    packed_modules_mapping = {"qkv_proj": ["q_proj", "k_proj", "v_proj"]}

    def __init__(self, config: SiglipVisionConfig) -> None:
        super().__init__(config)
        self.config = config
        self.vision_model = SiglipVisionTransformer(
            config=config,
            quant_config=config.quant_config,
            num_hidden_layers_override=config.num_hidden_layers_override,
            require_post_norm=config.require_post_norm,
            prefix=f"{config.prefix}.vision_model",
        )

    def forward(
        self,
        pixel_values: torch.Tensor,
        feature_sample_layers: list[int] | None = None,
        **kwargs,
    ) -> BaseEncoderOutput:
        last_hidden_state = self.vision_model(pixel_values, feature_sample_layers)
        return BaseEncoderOutput(last_hidden_state=last_hidden_state)

    @property
    def device(self):
        return next(self.parameters()).device

    def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]:
        params_dict = dict(self.named_parameters())
        loaded_params: set[str] = set()
        layer_count = len(self.vision_model.encoder.layers)

        for name, loaded_weight in weights:
            # Skip projection layers if any
            if name.startswith("visual_projection"):
                continue

            # Skip head if any
            if "head" in name:
                continue

            # Post layernorm handling
            if (name.startswith("vision_model.post_layernorm") and self.vision_model.post_layernorm is None):
                continue

            # Omit layers when num_hidden_layers_override is set
            if name.startswith("vision_model.encoder.layers"):
                layer_idx = int(name.split(".")[3])
                if layer_idx >= layer_count:
                    continue

            # Handle QKV projection weight mapping
            for (param_name, weight_name, shard_id) in self.config.arch_config.stacked_params_mapping:
                if weight_name not in name:
                    continue
                name = name.replace(weight_name, param_name)

                if name in params_dict:
                    param = params_dict[name]
                    weight_loader = param.weight_loader
                    weight_loader(param, loaded_weight, shard_id)
                break
            else:
                if name in params_dict:
                    param = params_dict[name]
                    weight_loader = getattr(param, "weight_loader", default_weight_loader)
                    weight_loader(param, loaded_weight)
            loaded_params.add(name)

        return loaded_params

fastvideo.models.encoders.siglip.SiglipVisionModel.forward

forward(pixel_values: torch.Tensor, feature_sample_layers: list[int] | None = None, **kwargs) -> BaseEncoderOutput
Source
def forward(
    self,
    pixel_values: torch.Tensor,
    feature_sample_layers: list[int] | None = None,
    **kwargs,
) -> BaseEncoderOutput:
    last_hidden_state = self.vision_model(pixel_values, feature_sample_layers)
    return BaseEncoderOutput(last_hidden_state=last_hidden_state)

fastvideo.models.encoders.siglip.SiglipVisionModel.load_weights

load_weights(weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]
Source
def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]:
    params_dict = dict(self.named_parameters())
    loaded_params: set[str] = set()
    layer_count = len(self.vision_model.encoder.layers)

    for name, loaded_weight in weights:
        # Skip projection layers if any
        if name.startswith("visual_projection"):
            continue

        # Skip head if any
        if "head" in name:
            continue

        # Post layernorm handling
        if (name.startswith("vision_model.post_layernorm") and self.vision_model.post_layernorm is None):
            continue

        # Omit layers when num_hidden_layers_override is set
        if name.startswith("vision_model.encoder.layers"):
            layer_idx = int(name.split(".")[3])
            if layer_idx >= layer_count:
                continue

        # Handle QKV projection weight mapping
        for (param_name, weight_name, shard_id) in self.config.arch_config.stacked_params_mapping:
            if weight_name not in name:
                continue
            name = name.replace(weight_name, param_name)

            if name in params_dict:
                param = params_dict[name]
                weight_loader = param.weight_loader
                weight_loader(param, loaded_weight, shard_id)
            break
        else:
            if name in params_dict:
                param = params_dict[name]
                weight_loader = getattr(param, "weight_loader", default_weight_loader)
                weight_loader(param, loaded_weight)
        loaded_params.add(name)

    return loaded_params