---
slug: attention
title: FastVideo Attention Kernels
---
<a id="fastvideo-attention-kernels"></a>

FastVideo provides highly optimized custom attention kernels to accelerate video generation.

## Supported Kernels {#supported-kernels}

* **[Video Sparse Attention (VSA)](/FastVideo/attention/vsa)**: Sparse attention mechanism selecting top-k blocks.
* **[Sliding Tile Attention (STA)](/FastVideo/attention/sta)**: STA kernel support is kept in
  `fastvideo-kernel`; full FastVideo STA pipeline workflow is archived in
  `sta_do_not_delete`.
* **[Attn-QAT Training](/FastVideo/training/attn_qat)**: Runtime-JIT Triton forward
  and backward kernels for role-local quantization-aware training.
* **Backend development guide**: See the developer guide at
  [Attention Backend Development](/FastVideo/contributing/attention_backend).

## General Build Instructions {#general-build-instructions}

These instructions apply to building the `fastvideo-kernel` package from
source, which includes STA, VSA, and Attn-QAT kernels.

### Prerequisites {#prerequisites}

* **PyTorch**: 2.5.0+
* **CUDA**: 12.6+ (CUDA 13 recommended for Blackwell GPUs)
* **C++ Compiler**: GCC 11+ (C++20 support required for ThunderKittens)

Install system dependencies:

```bash
sudo apt update
sudo apt install -y gcc-11 g++-11 clang-11 ninja-build

# Set gcc-11 as default
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-11 100 --slave /usr/bin/g++ g++ /usr/bin/g++-11
```

Set up your CUDA environment variables (adjust version as needed):

```bash
export CUDA_HOME=/usr/local/cuda
export PATH=${CUDA_HOME}/bin:${PATH}
export LD_LIBRARY_PATH=${CUDA_HOME}/lib64:$LD_LIBRARY_PATH
```

### Compile and Install {#compile-and-install}

Clone the repository and build the kernel:

```bash
# Clone recursively to get ThunderKittens submodule
git clone https://github.com/hao-ai-lab/FastVideo.git
cd FastVideo/fastvideo-kernel

# Build and install
./build.sh
```

The build script automatically detects your GPU architecture:
* **H100 (sm_90a)**: Compiles optimized C++ ThunderKittens kernels.
* **Other (A100, etc.)**: Skips C++ compilation; installs Python package with Triton kernels.