Raghav
e4eef6ed20
Per-layer bf16 diagnostic test on Z-Image-Turbo (Qwen3 35 layers, not the 24 originally assumed) confirmed the divergence between FastVideo's Qwen3 port and HF Qwen3 is a textbook bf16-tail signature, not a code bug: - fp32 forward is bit-exact (last_hidden_state max=0.0000) - bf16 per-layer mean drift grows smoothly from 5e-4 to 7.5e-2 over 35 layers; no single-layer spike (consecutive layer ratio ≤ 1.5x) - embedding (idx 0) is bit-exact in bf16 too — weight load is right Root cause is well-understood: FastVideo's TP-aware fused linears (QKVParallel, MergedColumnParallel, SiluAndMul) execute matmul ops in a different order than HF's unfused equivalents. In fp32 the per-element differences cancel statistically (~zero mean drift); in bf16 the rounding-order accumulates into a long max tail. Per add-model-02-parity calibration block: replace element-wise assert_close (meaningless when the max tail is 4.0) with distribution checks (mean + median). Thresholds calibrated to empirical observation: last_hidden_state (post-norm): mean < 0.025, median < 0.020 hidden_states[-2] (pre-norm): mean < 0.120, median < 0.100 Headroom is ~1.6x over observed (0.0152, 0.0117, 0.0754, 0.0625). fp32 path keeps strict assert_close at atol=1e-4. Also revert `torch_dtype=dtype` -> `dtype=dtype`: transformers 4.57.3 emits a deprecation warning recommending `dtype=`. The earlier switch (Copilot review) was wrong-direction for the pinned transformers version. The two config-key strips that mention `torch_dtype` stay — they're stripping the legacy serialized config key, not the constructor kwarg. PORT_STATUS.md updated: pinned Z-Image SHA, pinned HF id (Z-Image-Turbo), empirical bf16 signature recorded in the decisions table, 35-layer correction reflected in the parity table.
Local Tests
Local-only parity, smoke, and component tests for FastVideo model ports. They compare FastVideo against the official reference implementations and are skipped in CI; run them locally on a single GPU (or CPU where noted).
For the CI-backed test suite, see fastvideo/tests/.
Layout
Tests are organized by model family, one directory per port. Each family
directory follows the layout produced by the
add-model-prep skill:
tests/local_tests/<family>/
├── README.md # reference assets, setup, test inventory
├── __init__.py
└── test_<family>_<component>_*.py
Families
| Family | Workload | Dir |
|---|---|---|
| Flux2 | T2I | flux2/ |
| Hunyuan GameCraft | T2V / I2V | gamecraft/ |
| GEN3C | T2V | gen3c/ |
| Kandinsky-5 | T2V | kandinsky5/ |
| LTX-2 | T2V (+ audio) | ltx2/ |
| Stable Diffusion 3.5 | T2I | sd35/ |
| Stable Audio Open 1.0 | T2A | stable_audio/ |
| Wan2.2 | I2V | wan22/ |
Running a family
pytest tests/local_tests/<family>/ -v -s
See each family's README.md for setup (clones, weights, env vars, gated
tokens) and the per-component parity coverage.
Adding a new family
Run the add-model-prep
skill from the FastVideo repo root. It scaffolds the family directory and
README.md and stages reference clones + weights. (In-progress ports may also
add a PORT_STATUS.md from the same skill template; remove it once the port
lands.)