diff --git a/tests/local_tests/magi-human.md b/tests/local_tests/magi-human.md index d4e6ebdea..6d93e97f6 100644 --- a/tests/local_tests/magi-human.md +++ b/tests/local_tests/magi-human.md @@ -111,13 +111,13 @@ snapshot_download('stabilityai/stable-audio-open-1.0') All MagiHuman local tests in one shot: ```bash -pytest tests/local_tests/transformers/test_magi_human_parity.py \ - tests/local_tests/encoders/test_magi_human_t5gemma_parity.py \ - tests/local_tests/vaes/test_magi_human_sa_audio_parity.py \ - tests/local_tests/vaes/test_magi_human_sa_audio_official_parity.py \ - tests/local_tests/vaes/test_magi_human_vae_parity.py \ - tests/local_tests/pipelines/test_magi_human_pipeline_smoke.py \ - tests/local_tests/pipelines/test_magi_human_pipeline_parity.py \ +pytest tests/local_tests/magi_human/test_magi_human_parity.py \ + tests/local_tests/magi_human/test_magi_human_t5gemma_parity.py \ + tests/local_tests/magi_human/test_magi_human_sa_audio_parity.py \ + tests/local_tests/magi_human/test_magi_human_sa_audio_official_parity.py \ + tests/local_tests/magi_human/test_magi_human_vae_parity.py \ + tests/local_tests/magi_human/test_magi_human_pipeline_smoke.py \ + tests/local_tests/magi_human/test_magi_human_pipeline_parity.py \ fastvideo/tests/ssim/test_magi_human_similarity.py \ -v -s ``` @@ -179,7 +179,7 @@ Open questions). Each test file is independent. Run one: ```bash -pytest tests/local_tests/pipelines/test_magi_human_pipeline_parity.py -v -s +pytest tests/local_tests/magi_human/test_magi_human_pipeline_parity.py -v -s ``` ## Phase 11 status @@ -188,19 +188,19 @@ Branch tip `eeef855b` (rebased onto `origin/main` `c77a76c6`), Wave 1+4 changes | Test | Status | Diff numbers | Notes | |---|---|---|---| -| `tests/local_tests/encoders/test_magi_human_t5gemma_parity.py::test_magi_human_t5gemma_wrapper_parity` | PASS | exact (`assert_close(atol=1e-3, rtol=1e-3)`) | gated repo, requires HF token | -| `tests/local_tests/transformers/test_magi_human_parity.py::test_magi_human_dit_parity` | FAIL | video diff_max=0.057, diff_mean=0.008; audio diff_max=0.034, diff_mean=0.008; text exact (diff_max=0) | Tightened to `atol=0.03, rtol=0.01` (Wave 1). Bf16-noise-floor; per-layer drift ~1e-3 accumulates over 40 layers. Root cause of OQ-6 compounding. See §Numerical-alignment investigation. | -| `tests/local_tests/vaes/test_magi_human_vae_parity.py::test_magi_human_vae_decode_parity` | PASS | diff_max=8e-4, diff_mean=4.9e-5 | Wan VAE. Deferred to `atol=1e-3, rtol=1e-3` per OQ-7 (Wave 4). Tighten to `atol=1e-4` once Wan VAE op-order fix lands. | -| `tests/local_tests/vaes/test_magi_human_sa_audio_parity.py::test_magi_human_sa_audio_vae_decode_parity` | PASS | exact (`assert_close(atol=1e-5, rtol=1e-5)`, machine epsilon) | gated repo, requires HF token; uses main's shared `OobleckVAE` + `SAAudioVAEModel` wrapper | -| `tests/local_tests/vaes/test_magi_human_sa_audio_official_parity.py::test_magi_human_sa_audio_official_decode_parity` | PASS | `atol=1e-5, rtol=1e-5`, diff_max=0, diff_mean=0 (bit-exact) | Wave 7. Compares FV `SAAudioVAEModel` vs upstream `SAAudioFeatureExtractor.decode()`. Confirms OQ-6 is NOT in audio VAE. Requires upstream clone + gated SA repo. | -| `tests/local_tests/pipelines/test_magi_human_pipeline_smoke.py::test_magi_human_typed_surface_preflight` | PASS | CPU-only key/preset checks, exact key set equality, 331 keys | no skip conditions met locally | -| `tests/local_tests/pipelines/test_magi_human_pipeline_smoke.py::test_magi_human_pipeline_smoke` | PASS | shape-only; 2 inference steps, output shape `[B,C,T,H,W]` validated | wallclock ~50s | -| `tests/local_tests/pipelines/test_magi_human_pipeline_parity.py::test_magi_human_pipeline_latent_parity` | FAIL | video: diff_max=6.69, diff_mean=0.47; audio: diff_max=3.45, diff_mean=1.01 | Wave 7.5: now uses real preset prompts via T5-Gemma. Wave 8 production fixes don't move parity numbers (both sides use same encoder). Residual drift is bf16+CFG amplification floor; tracked as OQ-6 RESOLVED-PRODUCTION. | +| `tests/local_tests/magi_human/test_magi_human_t5gemma_parity.py::test_magi_human_t5gemma_wrapper_parity` | PASS | exact (`assert_close(atol=1e-3, rtol=1e-3)`) | gated repo, requires HF token | +| `tests/local_tests/magi_human/test_magi_human_parity.py::test_magi_human_dit_parity` | FAIL | video diff_max=0.057, diff_mean=0.008; audio diff_max=0.034, diff_mean=0.008; text exact (diff_max=0) | Tightened to `atol=0.03, rtol=0.01` (Wave 1). Bf16-noise-floor; per-layer drift ~1e-3 accumulates over 40 layers. Root cause of OQ-6 compounding. See §Numerical-alignment investigation. | +| `tests/local_tests/magi_human/test_magi_human_vae_parity.py::test_magi_human_vae_decode_parity` | PASS | diff_max=8e-4, diff_mean=4.9e-5 | Wan VAE. Deferred to `atol=1e-3, rtol=1e-3` per OQ-7 (Wave 4). Tighten to `atol=1e-4` once Wan VAE op-order fix lands. | +| `tests/local_tests/magi_human/test_magi_human_sa_audio_parity.py::test_magi_human_sa_audio_vae_decode_parity` | PASS | exact (`assert_close(atol=1e-5, rtol=1e-5)`, machine epsilon) | gated repo, requires HF token; uses main's shared `OobleckVAE` + `SAAudioVAEModel` wrapper | +| `tests/local_tests/magi_human/test_magi_human_sa_audio_official_parity.py::test_magi_human_sa_audio_official_decode_parity` | PASS | `atol=1e-5, rtol=1e-5`, diff_max=0, diff_mean=0 (bit-exact) | Wave 7. Compares FV `SAAudioVAEModel` vs upstream `SAAudioFeatureExtractor.decode()`. Confirms OQ-6 is NOT in audio VAE. Requires upstream clone + gated SA repo. | +| `tests/local_tests/magi_human/test_magi_human_pipeline_smoke.py::test_magi_human_typed_surface_preflight` | PASS | CPU-only key/preset checks, exact key set equality, 331 keys | no skip conditions met locally | +| `tests/local_tests/magi_human/test_magi_human_pipeline_smoke.py::test_magi_human_pipeline_smoke` | PASS | shape-only; 2 inference steps, output shape `[B,C,T,H,W]` validated | wallclock ~50s | +| `tests/local_tests/magi_human/test_magi_human_pipeline_parity.py::test_magi_human_pipeline_latent_parity` | FAIL | video: diff_max=6.69, diff_mean=0.47; audio: diff_max=3.45, diff_mean=1.01 | Wave 7.5: now uses real preset prompts via T5-Gemma. Wave 8 production fixes don't move parity numbers (both sides use same encoder). Residual drift is bf16+CFG amplification floor; tracked as OQ-6 RESOLVED-PRODUCTION. | | `fastvideo/tests/ssim/test_magi_human_similarity.py::test_magi_human_base_inference_similarity` | DEFERRED | n/a | Reference videos not yet seeded to `FastVideo/ssim-reference-videos` HF repo; tracked as OQ-2. Requires Modal L40S seeding via `seed-ssim-references` skill. | | _(debug)_ | INFO | Per-side layer logs: `/tmp/opencode/magi_dit_up_layers.log`, `/tmp/opencode/magi_dit_fv_layers.log` | Added in Wave 1 to `_debug_magi_human_block_parity.py`. See `add-model-trace` skill at `~/.config/opencode/skill/add-model-trace/`. | | `fastvideo/tests/hooks/test_activation_trace.py::*` | PASS | 6 tests covering off/on/filter/stats/step-filter/cleanup | Wave 9 activation trace infrastructure | -_Last verified: 2026-05-01 (Wave 9 trace mode build)_ +_Last verified: 2026-05-01 (Wave 10 dtype refactor on rebased branch @ 3caeaad1; tests now under `tests/local_tests/magi_human/`)_ ## Design notes @@ -384,6 +384,23 @@ Documentation at `docs/contributing/activation_trace.md`. Future Extensions 1-3 Companion skill at `~/.config/opencode/skill/add-model-trace/` (template for one-off ad-hoc port investigations) is unchanged. +### Wave 10 (2026-05-01): WanVideo-pattern dtype refactor + +Removed all 7 hardcoded `.to(torch.bfloat16)` casts in `fastvideo/models/dits/magi_human.py`. These were verbatim copies of upstream `daVinci-MagiHuman/inference/model/dit/dit_module.py` (lines 619, 507, 650, 694, 696). FV now follows the canonical FastVideo dtype pattern exemplified by `fastvideo/models/dits/wanvideo.py`: model dtype is **loader-owned** via `pipeline_config.dit_precision` → `default_dtype` in `component_loader.py`. Inside DiT forward, `orig_dtype = self.linear_qkv.weight.dtype` (or equivalent) is captured and used for output preservation; no hardcoded model-dtype casts remain. The top-level block-input cast (formerly `x.to(torch.bfloat16)`) is now `x.to()`. + +Refactored sites: +- attention pre_norm output (line ~360) +- q/k/v post-RoPE casts (lines ~399-401) +- attention output (line ~411) +- MLP pre_norm + activation casts (lines ~444-447) +- top-level block-input cast (line ~684) + +Production behavior unchanged: bf16 parity numbers identical to baseline (`diff_max=0.057, diff_mean=0.005`). Loader's `dit_precision="bf16"` default → all params/inputs bf16 → `orig_dtype = bf16` → outputs preserved as bf16 → same as before. + +fp32 parity now works end-to-end on the FV side (model is dtype-agnostic in forward), but the parity test against upstream still shows bf16-noise residual drift (post-refactor: `diff_max=0.061, diff_mean=0.0068`; pre-refactor was `0.082 / 0.0079`, ~1.2x improvement). The remaining drift is from upstream `dit_module.py` itself — upstream still hardcodes `.to(torch.bfloat16)` in its forward, so even in an fp32 parity run, upstream's intermediate tensors are bf16. **Fully fp32-clean parity would require either patching the local upstream clone OR using a build of upstream where the hardcoded casts are also config-driven.** + +OQ-9 (NEW): upstream `daVinci-MagiHuman/inference/model/dit/dit_module.py` has hardcoded `.to(torch.bfloat16)` casts at lines 619, 507, 650, 694, 696. For full fp32 parity validation, these would need to be patched in the local clone OR a flag added upstream. Tracked as low-priority follow-up; affects only fp32 parity testing, not production. + ### Potential mitigations (not investigated this session) - Run sensitive ops (MM-layer pre-norm, attention) in fp32 instead of bf16. @@ -400,7 +417,7 @@ Layer-by-layer traces are written to: - `/tmp/opencode/magi_dit_up_layers.log` (upstream reference) - `/tmp/opencode/magi_dit_fv_layers.log` (FastVideo) -These are produced by `tests/local_tests/transformers/_debug_magi_human_block_parity.py` +These are produced by `tests/local_tests/magi_human/_debug_magi_human_block_parity.py` via forward hooks registered on each transformer block. The `add-model-trace` skill at `~/.config/opencode/skill/add-model-trace/` generalizes this methodology for future ports: forward-hook + monkey-patch + git-stash-cleanup @@ -417,6 +434,7 @@ with hard rules around no-source-residue cleanup. | OQ-5 | **Basic-example output mp4 visual quality is impressionistic at 256x448.** Root cause identified: OQ-6 (pre-existing compounding bf16 drift over the 32-step denoise loop). Wave 2-3 investigation confirmed the 4-step pipeline parity shows 18.85x compounding ratio vs expected 4x linear. See OQ-6 for full details and mitigation candidates. | RESOLVED-ROOT-CAUSE-IDENTIFIED (see OQ-6) | | OQ-6 | **Pre-existing compounding bug in MagiHumanDiT denoising loop (HIGH PRIORITY).** 4-step pipeline parity shows video diff_mean=1.30 vs 1-step 0.069, a ratio of 18.85x (expected ~4x linear). Bug pre-exists in commit 620aaf41 (original magi port); confirmed via `git revert` bisect (4-step diff_mean=1.20 with all Wave 1 reverts). Single-forward DiT drift (diff_max=0.057) is bf16-noise-floor: cumulative ~1e-3 per-layer over 40 layers, consistent with random-walk accumulation. Wave 3 ruled out: PackedExpertLinear routing (A/B patch showed zero change), expert chunk ordering (bit-exact direct test), conversion script (bit-exact). Wave 7 partial fix: FV's `_MAGI_HUMAN_NEGATIVE_PROMPT` was missing the audio-quality + speech-delivery blocks present in upstream `video_generate.py:222-224`; audio CFG amplified the missing-block delta 5x, explaining the observed step-1 audio amplification of ~3x. Fix applied at `presets.py`. Wave 8 additional production fixes: tokenizer pre-padding (t5gemma.py), resolution defaults (presets.py + latent_preparation.py), silent audio fallback (audio_decoding.py). ALL production-facing root causes now identified and fixed. Parity-test 4-step compounding is a separate phenomenon — inherent FlowUniPC multistep scheduler amplification of per-call bf16 noise; NOT a code bug. fp32 sensitive ops or scheduler change would be needed to materially reduce parity-test compounding. | RESOLVED-PRODUCTION; bf16 amplification floor TRACKED separately | | OQ-7 | **Wan VAE shared fp32 op-order drift (MEDIUM PRIORITY).** FV uses `z * std + mean` at decode normalization; upstream uses `z / (1/std) + mean`. Bitwise non-equivalent in fp32. Affects all Wan-family pipelines (`fastvideo/configs/pipelines/wan.py`, `turbodiffusion.py`, `longcat.py`, magi-human). Magi VAE test loosened to `atol=1e-3, rtol=1e-3` (Wave 4) to defer. Tighten back to `atol=1e-4` once the Wan VAE op-order fix lands. Fix should be validated against Wan2.1, Wan2.2, and magi-human. Estimated 0.5-1 day to fix and validate. | TRACKED FOLLOW-UP | +| OQ-9 | **Upstream `dit_module.py` hardcoded bf16 casts block full fp32 parity validation.** `daVinci-MagiHuman/inference/model/dit/dit_module.py` has hardcoded `.to(torch.bfloat16)` casts at lines 619, 507, 650, 694, 696. FV's DiT forward is now dtype-agnostic (Wave 10), but parity tests against upstream still show bf16-noise residual drift in fp32 runs because upstream's intermediate tensors are bf16. Full fp32-clean parity would require patching the local upstream clone or adding a dtype-config flag upstream. Affects only fp32 parity testing, not production. | TRACKED FOLLOW-UP (LOW PRIORITY) | ## Troubleshooting @@ -479,13 +497,13 @@ loaders for the upstream DiT, VAE, and pipeline. Use these as the starting point for any new parity test rather than duplicating the load logic. The `_debug_magi_human_block_parity.py` and `_debug_magi_human_weight_diff.py` -scripts in `tests/local_tests/transformers/` are scratch tools for divergence +scripts in `tests/local_tests/magi_human/` are scratch tools for divergence investigation. They are NOT pytest tests and must NOT be promoted to formal tests. Run them directly with `python` when you need to inspect per-block diffs or weight mismatches during a parity-debug session. If you need to chase per-layer divergence on a future add-model port, see the `add-model-trace` skill at `~/.config/opencode/skill/add-model-trace/`. -Generalized from `tests/local_tests/transformers/_debug_magi_human_block_parity.py` +Generalized from `tests/local_tests/magi_human/_debug_magi_human_block_parity.py` (the worked magi example), it provides a forward-hook + monkey-patch + git-stash-cleanup methodology with hard rules around no-source-residue cleanup.