Per-layer bf16 diagnostic test on Z-Image-Turbo (Qwen3 35 layers, not the
24 originally assumed) confirmed the divergence between FastVideo's
Qwen3 port and HF Qwen3 is a textbook bf16-tail signature, not a code
bug:
- fp32 forward is bit-exact (last_hidden_state max=0.0000)
- bf16 per-layer mean drift grows smoothly from 5e-4 to 7.5e-2 over 35
layers; no single-layer spike (consecutive layer ratio ≤ 1.5x)
- embedding (idx 0) is bit-exact in bf16 too — weight load is right
Root cause is well-understood: FastVideo's TP-aware fused linears
(QKVParallel, MergedColumnParallel, SiluAndMul) execute matmul ops in a
different order than HF's unfused equivalents. In fp32 the per-element
differences cancel statistically (~zero mean drift); in bf16 the
rounding-order accumulates into a long max tail.
Per add-model-02-parity calibration block: replace element-wise
assert_close (meaningless when the max tail is 4.0) with distribution
checks (mean + median). Thresholds calibrated to empirical observation:
last_hidden_state (post-norm): mean < 0.025, median < 0.020
hidden_states[-2] (pre-norm): mean < 0.120, median < 0.100
Headroom is ~1.6x over observed (0.0152, 0.0117, 0.0754, 0.0625).
fp32 path keeps strict assert_close at atol=1e-4.
Also revert `torch_dtype=dtype` -> `dtype=dtype`: transformers 4.57.3
emits a deprecation warning recommending `dtype=`. The earlier switch
(Copilot review) was wrong-direction for the pinned transformers version.
The two config-key strips that mention `torch_dtype` stay — they're
stripping the legacy serialized config key, not the constructor kwarg.
PORT_STATUS.md updated: pinned Z-Image SHA, pinned HF id (Z-Image-Turbo),
empirical bf16 signature recorded in the decisions table, 35-layer
correction reflected in the parity table.
tests/local_tests/ holds local-only parity, smoke, and component tests
organized by model family (e.g. tests/local_tests/ltx2/,
tests/local_tests/stable_audio/). They typically require a checked-out
reference repo and/or model weights and are skipped in CI; see each family's
README.md for setup. The top-level
tests/local_tests/README.md is the family index.
The CI-backed test suite still lives in fastvideo/tests/.
Raghav
e4eef6ed20