- Binding: require compute capability 12.1. Other archs in the same build get
an empty kernel that returned uninitialized output without an error.
- Kernel: a NaN softmax sum now gives a NaN output row (l != 0 instead of
l > 0), as on the ptx_blackwell and Triton routes; rows without keys stay 0.
- Kernel header: the kernel, encode_map and launch get internal linkage, so a
second translation unit can include the header (it failed to link before).
The launch limits move into the header (MaxBatchHeads, MaxRows, MaxTiles),
used by both launch() and the binding and mirrored in Python.
- Header comments: describe the QK/PV overlap that PhaseLead=8 gives; drop
the stale cp.async comment and the unused cp_async_mbar_arrive/prefetch_l2.
- H3 backend: an odd tile count without tile()'s partner tile falls back to
split instead of reaching ptx_blackwell; tile() no longer resolves the
tile-128 route (or warns) under an eager fine-attention override.
- Python module docstring states the replacement contract for the kernel.
- Kernel: zero V padding rows on partial 64-key chunks (and skip 16-key
steps wholly past the valid keys), so NaN/Inf padding in K or V cannot
reach the output; cache the SM count and the shared-memory attribute per
device in atomics instead of an unsynchronized function-static slot.
- is_supported now checks the launch limits (batch * heads <= 65535,
rows < 2^31, tiles < 2^22) via within_launch_limits; the binding checks
the row limit too. The H3 route checks the limits per call and falls
back to split-Triton instead of failing in the binding.
- H3 route: one tile-128 route cache (ptx_blackwell / sm121 / split),
resolved on a two-tile probe before tile(), so the sm121 and split routes
no longer pad odd tile counts with the ptx_blackwell partner tile.
- Warn once when a GB10 falls back to split-Triton; the fail-closed error
names the sm_121a build.
- CMake: accept 12.1f and +PTX spellings; warn when CUDA < 13 skips it.
- Docs: bf16 fallback leaves FASTVIDEO_H3_VSA_FINE_ATTENTION unset on Spark.
- Tests: the real-kernel oracle tests assert the route and spy on the
kernel entry; new tests for NaN/Inf padding, launch limits, the GB10
warning and the over-limit fallback.
FastH3's tile-128 route runs the ptx_blackwell CUDA forward on sm_100/sm_103 and, on every other GPU,
splits each 128-token tile into two 64-token halves on the Triton kernels. On the DGX Spark that split
route is what OmniRef's BF16 attention runs today. This adds a 128-token forward built for sm_121a
(mma.sync + TMA; the GB10 has no tcgen05 or tensor memory) and routes the tile-128 call to it on a
GB10, ahead of the split.
- csrc/attention/block_sparse_kernel_blk128_sm121.cuh and block_sparse_blk128_sm121.cu: the kernel and
its binding block_sparse_sm121_blk128_fwd(q, k, v, q2k_idx, q2k_num, variable_block_sizes) -> {out},
inference only, no LSE.
- CMakeLists.txt: built when TORCH_CUDA_ARCH_LIST has 12.1a or 12.1 and CUDA >= 13, so both the
aarch64 wheel and the GB10 Docker image carry it.
- fastvideo_kernel.block_sparse_attn_sm121: is_supported and an opaque custom op with a fake. It is a
separate module because block_sparse_attn_ptx_blackwell.is_supported also gates an LSE-requiring
autograd path.
- video_sparse_attn_h3: the tile-128 route tries it after ptx_blackwell and before split_tile128.
- Tests: kernel correctness against an FP32 reference (ragged blocks, empty selections, odd block
counts, many waves, a NaN-filled allocator, determinism), and the H3 route (fakes, plus a real-GB10
oracle test).
For Wan VSA inference with 64-token tiles, each attention layer scattered Q, K,
V and the gate into a tile buffer and then made four transposed copies. On
SM100, one Triton kernel (tile_to_bhsd) now does the scatter and the transposes
in one pass. Training, 256-token tiles, other GPUs and builds without the kernel
use the previous path. FASTVIDEO_DISABLE_VSA64_FUSED_LAYOUT=1 turns the fused
path off.
On GB200 the layout and attention step goes from 15.7 ms to 8.6 ms. FastWan 1.3B
frames are bit-exact with the previous path at sequence parallel 1 and 2.
On SM100, the Attn-QAT training kernel now uses a faster route for non-causal
BF16 attention with equal query and key lengths of 16384 tokens or more. The
forward pass processes full KV tiles without bounds masks, and a separate masked
step handles the partial tail tile. The backward pass uses 64x128 dQ and dK/dV
tiles. Outputs, softmax statistics and all gradients are bitwise identical to the
previous route. FASTVIDEO_ATTN_QAT_SM100_OPTIMIZED=0 turns off the SM100 route,
and FASTVIDEO_ATTN_QAT_SM100_WIDE_BWD=0 keeps the 64x64 backward tiles.
On GB200 with 6 heads and 31200 tokens, the forward kernel time goes from 9.14 ms
to 6.96 ms and the backward kernel time from 17.41 ms to 15.85 ms.
SolitaryThinker