Will Lin and Claude Fable 5
95f4f54712
[bugfix] Triton block-sparse backward: scale QK^T after the dot, keep K raw
...
The backward pre-scaled K by sm_scale*log2(e) in bf16 before recomputing
logits, so the recompute drifted from the forward's saved LSE proportionally
to |logit|; exp2 amplifies that into exponentially wrong probabilities and
garbage dK/dV (plus a mis-scaled dQ) at real activation magnitudes. Forward
was exact and unit-scale tests passed, which is how it survived — the Triton
path is only load-bearing off-Hopper (sm_90 uses the ThunderKittens CUDA
backward).
Fix: pass raw K, apply sm_scale*RCP_LN2 after tl.dot in fp32 in both
_attn_bwd_dkdv and _attn_bwd_dq, and scale dq by sm_scale instead of LN2.
Verified against SDPA to 4 significant digits at sm_scale 1/8/32 and at the
real H3 geometry (37,48,84 tiles, 128-dim heads).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com >
2026-08-11 04:40:58 +00:00
William Lin
9028953625
[misc] yapf pass under CI's interpreter (3.12) + pin hook language_version ( #1702 )
2026-08-08 22:24:17 -07:00
9709b7513b
[feat] Port NVFP4 QAT/QAD to modular train framework ( #1619 )
...
Co-authored-by: Peiyuan Zhang <email>
Co-authored-by: Peiyuan <a>
2026-07-18 15:01:09 -07:00
William Lin
0d626ef2d1
[kernel] Bump fastvideo-kernel pin to 0.3.1 and version the FA4 tile_mn port as 0.3.2 ( #1539 )
2026-07-03 14:11:08 -07:00
William Lin
b36d0ef085
[infra] Add DGX Spark and multi-architecture CUDA support ( #1447 )
2026-07-03 13:41:43 -07:00
William Lin
31e26abec4
[chore] release fastvideo-kernel 0.3.1 ( #1520 )
2026-06-30 12:25:27 -07:00
sumyyyyy and SolitaryThinker
9e83ba630c
[kernel] Extract VSA utility functions into fastvideo_kernel ( #1408 )
...
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com >
2026-06-30 12:19:25 -07:00
xsank and mergify[bot]
31b719ae49
[perf] optimize compress & topk kernel ( #1421 )
...
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-06-23 18:51:22 +00:00
William Lin
b1d89eba1f
[chore] release fastvideo-kernel 0.3.0 ( #1478 )
2026-06-23 10:02:27 -07:00
sumyyyyy
87f98c9b8b
[kernel] Add varlen support for block-sparse attention ( #1319 )
2026-06-18 01:06:00 +00:00
eed9c4bfbf
[kernel] QAD 5090: Add Attn-QAT training Triton kernels (11/12) ( #1460 )
...
Co-authored-by: William Lin <SolitaryThinker@users.noreply.github.com >
Co-authored-by: Loay Rashid <42599591+loaydatrain@users.noreply.github.com >
Co-authored-by: Kaiqin Kong <k1kong@ucsd.edu >
2026-06-17 13:35:12 -07:00
alexzms
be548a78ea
[feat] VSA-256 fastpath on Blackwell via FA4 CuTe block-sparse attention ( #1354 )
2026-05-26 12:58:29 -07:00
William Lin
140bd1a6cf
[misc]: standardize install instructions on uv pip install ( #1279 )
2026-05-02 12:45:50 -07:00
Mook
7b872cc41e
[Perf] Skip bool-mask round-trip in block-sparse VSA attention ( #1243 )
2026-04-26 15:14:37 -07:00
1a383f3f66
[refactor] train v1 clean up
...
Co-authored-by: alexzms <3036648523@qq.com >
Co-authored-by: Peiyuan Zhang <a1286225768@slurm-h200-204-215.slurm-compute.tenant-slurm.svc.cluster.local >
2026-03-09 18:58:56 -07:00
alexzms
d925ad05f3
[bugfix] fastvideo-kernel: fix VSA Triton padding NaNs and support q/kv length mismatch ( #1094 )
2026-02-13 12:39:49 -08:00
alexzms
dbf3917bf4
[fastvideo-kernel] replace map to index with Triton implementation + add vsa benchmark ( #1029 )
2026-01-22 11:35:02 -08:00
alexzms and William Lin
31f44110b5
[kernel] [bugfix] [ci] bump v0.2.4. Fix STA output handling, TurboDiffusion CUDA norm dtypes for fastvideo-kernel unit tests. ( #1020 )
...
Co-authored-by: William Lin <SolitaryThinker@users.noreply.github.com >
2026-01-19 17:42:42 -08:00
William Lin
d58c551c11
[chore] release fastvideo-kernel 0.2.3 ( #1018 )
2026-01-17 02:24:23 -08:00
alexzms
560628709c
[Bug Fix] Add autograd wrapper for block-sparse attention in fastvideo-kernel + fix CMake extension linking ( #1015 )
2026-01-16 21:16:43 -08:00
William Lin
61e63ea0d7
[chore] release fastvideo-kernel 0.2.2 ( #986 )
2026-01-04 21:21:06 -06:00
Loay Rashid
dea69cf651
[New Model] Turbodiffusion ( #971 )
2026-01-02 17:55:56 -06:00
William Lin
40ce2d72f5
[kernel] add turbodiffusion kernels ( #972 )
2025-12-30 04:23:10 -06:00
RoyWang and roywang
de264fcc99
[fix]: fix STA trition kernel for AMD RDNA archs ( #969 )
...
Co-authored-by: roywang <roywang@amd.com >
2025-12-29 14:25:32 -06:00
William Lin
cbbb09d7b8
[kernel] Release fastvideo-kernel v0.2.1 ( #963 )
2025-12-26 13:59:03 -06:00
William Lin and ShreejithSG
2f6230abcf
[kernel] Reorg and fix fastvideo-kernel ( #962 )
...
Co-authored-by: ShreejithSG <shreejithsg@gmail.com >
2025-12-26 01:50:24 -06:00
