nodes/hao-ai-lab-FastVideo
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

[docs] magi-human: Wave 10 dtype refactor + post-rebase test paths + OQ-9

Browse Source 
- Update all test path references from old bucket dirs
  (transformers/, encoders/, vaes/, pipelines/) to the post-rebase
  consolidated tests/local_tests/magi_human/ directory.
- Append Wave 10 subsection to Numerical-alignment investigation:
  WanVideo-pattern dtype refactor, 7 hardcoded bf16 casts removed,
  fp32 parity improvement, upstream residual drift explanation.
- Add OQ-9 to open questions: upstream dit_module.py hardcoded bf16
  casts block full fp32 parity validation (low-priority follow-up).
- Update Phase 11 status: Last verified 2026-05-01 Wave 10 @ 3caeaad1,
  note tests now under tests/local_tests/magi_human/.
This commit is contained in:
SolitaryThinker
2026-05-05 10:55:49 -07:00
parent 8f0493637e
commit ccfcc3042b
1 changed files with 38 additions and 20 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/local_tests/magi-human.md
+38 -20
View File
@@ -111,13 +111,13 @@ snapshot_download('stabilityai/stable-audio-open-1.0')
All MagiHuman local tests in one shot:
```bash
pytest tests/local_tests/transformers/test_magi_human_parity.py \
tests/local_tests/encoders/test_magi_human_t5gemma_parity.py \
tests/local_tests/vaes/test_magi_human_sa_audio_parity.py \
tests/local_tests/vaes/test_magi_human_sa_audio_official_parity.py \
tests/local_tests/vaes/test_magi_human_vae_parity.py \
tests/local_tests/pipelines/test_magi_human_pipeline_smoke.py \
tests/local_tests/pipelines/test_magi_human_pipeline_parity.py \
pytest tests/local_tests/magi_human/test_magi_human_parity.py \
tests/local_tests/magi_human/test_magi_human_t5gemma_parity.py \
tests/local_tests/magi_human/test_magi_human_sa_audio_parity.py \
tests/local_tests/magi_human/test_magi_human_sa_audio_official_parity.py \
tests/local_tests/magi_human/test_magi_human_vae_parity.py \
tests/local_tests/magi_human/test_magi_human_pipeline_smoke.py \
tests/local_tests/magi_human/test_magi_human_pipeline_parity.py \
fastvideo/tests/ssim/test_magi_human_similarity.py \
-v -s
```
@@ -179,7 +179,7 @@ Open questions).
Each test file is independent. Run one:
```bash
pytest tests/local_tests/pipelines/test_magi_human_pipeline_parity.py -v -s
pytest tests/local_tests/magi_human/test_magi_human_pipeline_parity.py -v -s
```
## Phase 11 status
@@ -188,19 +188,19 @@ Branch tip `eeef855b` (rebased onto `origin/main` `c77a76c6`), Wave 1+4 changes
| Test | Status | Diff numbers | Notes |
|---|---|---|---|
| `tests/local_tests/encoders/test_magi_human_t5gemma_parity.py::test_magi_human_t5gemma_wrapper_parity` | PASS | exact (`assert_close(atol=1e-3, rtol=1e-3)`) | gated repo, requires HF token |
| `tests/local_tests/transformers/test_magi_human_parity.py::test_magi_human_dit_parity` | FAIL | video diff_max=0.057, diff_mean=0.008; audio diff_max=0.034, diff_mean=0.008; text exact (diff_max=0) | Tightened to `atol=0.03, rtol=0.01` (Wave 1). Bf16-noise-floor; per-layer drift ~1e-3 accumulates over 40 layers. Root cause of OQ-6 compounding. See §Numerical-alignment investigation. |
| `tests/local_tests/vaes/test_magi_human_vae_parity.py::test_magi_human_vae_decode_parity` | PASS | diff_max=8e-4, diff_mean=4.9e-5 | Wan VAE. Deferred to `atol=1e-3, rtol=1e-3` per OQ-7 (Wave 4). Tighten to `atol=1e-4` once Wan VAE op-order fix lands. |
| `tests/local_tests/vaes/test_magi_human_sa_audio_parity.py::test_magi_human_sa_audio_vae_decode_parity` | PASS | exact (`assert_close(atol=1e-5, rtol=1e-5)`, machine epsilon) | gated repo, requires HF token; uses main's shared `OobleckVAE` + `SAAudioVAEModel` wrapper |
| `tests/local_tests/vaes/test_magi_human_sa_audio_official_parity.py::test_magi_human_sa_audio_official_decode_parity` | PASS | `atol=1e-5, rtol=1e-5`, diff_max=0, diff_mean=0 (bit-exact) | Wave 7. Compares FV `SAAudioVAEModel` vs upstream `SAAudioFeatureExtractor.decode()`. Confirms OQ-6 is NOT in audio VAE. Requires upstream clone + gated SA repo. |
| `tests/local_tests/pipelines/test_magi_human_pipeline_smoke.py::test_magi_human_typed_surface_preflight` | PASS | CPU-only key/preset checks, exact key set equality, 331 keys | no skip conditions met locally |
| `tests/local_tests/pipelines/test_magi_human_pipeline_smoke.py::test_magi_human_pipeline_smoke` | PASS | shape-only; 2 inference steps, output shape `[B,C,T,H,W]` validated | wallclock ~50s |
| `tests/local_tests/pipelines/test_magi_human_pipeline_parity.py::test_magi_human_pipeline_latent_parity` | FAIL | video: diff_max=6.69, diff_mean=0.47; audio: diff_max=3.45, diff_mean=1.01 | Wave 7.5: now uses real preset prompts via T5-Gemma. Wave 8 production fixes don't move parity numbers (both sides use same encoder). Residual drift is bf16+CFG amplification floor; tracked as OQ-6 RESOLVED-PRODUCTION. |
| `tests/local_tests/magi_human/test_magi_human_t5gemma_parity.py::test_magi_human_t5gemma_wrapper_parity` | PASS | exact (`assert_close(atol=1e-3, rtol=1e-3)`) | gated repo, requires HF token |
| `tests/local_tests/magi_human/test_magi_human_parity.py::test_magi_human_dit_parity` | FAIL | video diff_max=0.057, diff_mean=0.008; audio diff_max=0.034, diff_mean=0.008; text exact (diff_max=0) | Tightened to `atol=0.03, rtol=0.01` (Wave 1). Bf16-noise-floor; per-layer drift ~1e-3 accumulates over 40 layers. Root cause of OQ-6 compounding. See §Numerical-alignment investigation. |
| `tests/local_tests/magi_human/test_magi_human_vae_parity.py::test_magi_human_vae_decode_parity` | PASS | diff_max=8e-4, diff_mean=4.9e-5 | Wan VAE. Deferred to `atol=1e-3, rtol=1e-3` per OQ-7 (Wave 4). Tighten to `atol=1e-4` once Wan VAE op-order fix lands. |
| `tests/local_tests/magi_human/test_magi_human_sa_audio_parity.py::test_magi_human_sa_audio_vae_decode_parity` | PASS | exact (`assert_close(atol=1e-5, rtol=1e-5)`, machine epsilon) | gated repo, requires HF token; uses main's shared `OobleckVAE` + `SAAudioVAEModel` wrapper |
| `tests/local_tests/magi_human/test_magi_human_sa_audio_official_parity.py::test_magi_human_sa_audio_official_decode_parity` | PASS | `atol=1e-5, rtol=1e-5`, diff_max=0, diff_mean=0 (bit-exact) | Wave 7. Compares FV `SAAudioVAEModel` vs upstream `SAAudioFeatureExtractor.decode()`. Confirms OQ-6 is NOT in audio VAE. Requires upstream clone + gated SA repo. |
| `tests/local_tests/magi_human/test_magi_human_pipeline_smoke.py::test_magi_human_typed_surface_preflight` | PASS | CPU-only key/preset checks, exact key set equality, 331 keys | no skip conditions met locally |
| `tests/local_tests/magi_human/test_magi_human_pipeline_smoke.py::test_magi_human_pipeline_smoke` | PASS | shape-only; 2 inference steps, output shape `[B,C,T,H,W]` validated | wallclock ~50s |
| `tests/local_tests/magi_human/test_magi_human_pipeline_parity.py::test_magi_human_pipeline_latent_parity` | FAIL | video: diff_max=6.69, diff_mean=0.47; audio: diff_max=3.45, diff_mean=1.01 | Wave 7.5: now uses real preset prompts via T5-Gemma. Wave 8 production fixes don't move parity numbers (both sides use same encoder). Residual drift is bf16+CFG amplification floor; tracked as OQ-6 RESOLVED-PRODUCTION. |
| `fastvideo/tests/ssim/test_magi_human_similarity.py::test_magi_human_base_inference_similarity` | DEFERRED | n/a | Reference videos not yet seeded to `FastVideo/ssim-reference-videos` HF repo; tracked as OQ-2. Requires Modal L40S seeding via `seed-ssim-references` skill. |
| _(debug)_ | INFO | Per-side layer logs: `/tmp/opencode/magi_dit_up_layers.log`, `/tmp/opencode/magi_dit_fv_layers.log` | Added in Wave 1 to `_debug_magi_human_block_parity.py`. See `add-model-trace` skill at `~/.config/opencode/skill/add-model-trace/`. |
| `fastvideo/tests/hooks/test_activation_trace.py::*` | PASS | 6 tests covering off/on/filter/stats/step-filter/cleanup | Wave 9 activation trace infrastructure |
_Last verified: 2026-05-01 (Wave 9 trace mode build)_
_Last verified: 2026-05-01 (Wave 10 dtype refactor on rebased branch @ 3caeaad1; tests now under `tests/local_tests/magi_human/`)_
## Design notes
@@ -384,6 +384,23 @@ Documentation at `docs/contributing/activation_trace.md`. Future Extensions 1-3
Companion skill at `~/.config/opencode/skill/add-model-trace/` (template for one-off ad-hoc port investigations) is unchanged.
### Wave 10 (2026-05-01): WanVideo-pattern dtype refactor
Removed all 7 hardcoded `.to(torch.bfloat16)` casts in `fastvideo/models/dits/magi_human.py`. These were verbatim copies of upstream `daVinci-MagiHuman/inference/model/dit/dit_module.py` (lines 619, 507, 650, 694, 696). FV now follows the canonical FastVideo dtype pattern exemplified by `fastvideo/models/dits/wanvideo.py`: model dtype is **loader-owned** via `pipeline_config.dit_precision` → `default_dtype` in `component_loader.py`. Inside DiT forward, `orig_dtype = self.linear_qkv.weight.dtype` (or equivalent) is captured and used for output preservation; no hardcoded model-dtype casts remain. The top-level block-input cast (formerly `x.to(torch.bfloat16)`) is now `x.to(<loader-owned dtype>)`.
Refactored sites:
- attention pre_norm output (line ~360)
- q/k/v post-RoPE casts (lines ~399-401)
- attention output (line ~411)
- MLP pre_norm + activation casts (lines ~444-447)
- top-level block-input cast (line ~684)
Production behavior unchanged: bf16 parity numbers identical to baseline (`diff_max=0.057, diff_mean=0.005`). Loader's `dit_precision="bf16"` default → all params/inputs bf16 → `orig_dtype = bf16` → outputs preserved as bf16 → same as before.
fp32 parity now works end-to-end on the FV side (model is dtype-agnostic in forward), but the parity test against upstream still shows bf16-noise residual drift (post-refactor: `diff_max=0.061, diff_mean=0.0068`; pre-refactor was `0.082 / 0.0079`, ~1.2x improvement). The remaining drift is from upstream `dit_module.py` itself — upstream still hardcodes `.to(torch.bfloat16)` in its forward, so even in an fp32 parity run, upstream's intermediate tensors are bf16. **Fully fp32-clean parity would require either patching the local upstream clone OR using a build of upstream where the hardcoded casts are also config-driven.**
OQ-9 (NEW): upstream `daVinci-MagiHuman/inference/model/dit/dit_module.py` has hardcoded `.to(torch.bfloat16)` casts at lines 619, 507, 650, 694, 696. For full fp32 parity validation, these would need to be patched in the local clone OR a flag added upstream. Tracked as low-priority follow-up; affects only fp32 parity testing, not production.
### Potential mitigations (not investigated this session)
- Run sensitive ops (MM-layer pre-norm, attention) in fp32 instead of bf16.
@@ -400,7 +417,7 @@ Layer-by-layer traces are written to:
- `/tmp/opencode/magi_dit_up_layers.log` (upstream reference)
- `/tmp/opencode/magi_dit_fv_layers.log` (FastVideo)
These are produced by `tests/local_tests/transformers/_debug_magi_human_block_parity.py`
These are produced by `tests/local_tests/magi_human/_debug_magi_human_block_parity.py`
via forward hooks registered on each transformer block. The `add-model-trace`
skill at `~/.config/opencode/skill/add-model-trace/` generalizes this
methodology for future ports: forward-hook + monkey-patch + git-stash-cleanup
@@ -417,6 +434,7 @@ with hard rules around no-source-residue cleanup.
| OQ-5 | **Basic-example output mp4 visual quality is impressionistic at 256x448.** Root cause identified: OQ-6 (pre-existing compounding bf16 drift over the 32-step denoise loop). Wave 2-3 investigation confirmed the 4-step pipeline parity shows 18.85x compounding ratio vs expected 4x linear. See OQ-6 for full details and mitigation candidates. | RESOLVED-ROOT-CAUSE-IDENTIFIED (see OQ-6) |
| OQ-6 | **Pre-existing compounding bug in MagiHumanDiT denoising loop (HIGH PRIORITY).** 4-step pipeline parity shows video diff_mean=1.30 vs 1-step 0.069, a ratio of 18.85x (expected ~4x linear). Bug pre-exists in commit 620aaf41 (original magi port); confirmed via `git revert` bisect (4-step diff_mean=1.20 with all Wave 1 reverts). Single-forward DiT drift (diff_max=0.057) is bf16-noise-floor: cumulative ~1e-3 per-layer over 40 layers, consistent with random-walk accumulation. Wave 3 ruled out: PackedExpertLinear routing (A/B patch showed zero change), expert chunk ordering (bit-exact direct test), conversion script (bit-exact). Wave 7 partial fix: FV's `_MAGI_HUMAN_NEGATIVE_PROMPT` was missing the audio-quality + speech-delivery blocks present in upstream `video_generate.py:222-224`; audio CFG amplified the missing-block delta 5x, explaining the observed step-1 audio amplification of ~3x. Fix applied at `presets.py`. Wave 8 additional production fixes: tokenizer pre-padding (t5gemma.py), resolution defaults (presets.py + latent_preparation.py), silent audio fallback (audio_decoding.py). ALL production-facing root causes now identified and fixed. Parity-test 4-step compounding is a separate phenomenon — inherent FlowUniPC multistep scheduler amplification of per-call bf16 noise; NOT a code bug. fp32 sensitive ops or scheduler change would be needed to materially reduce parity-test compounding. | RESOLVED-PRODUCTION; bf16 amplification floor TRACKED separately |
| OQ-7 | **Wan VAE shared fp32 op-order drift (MEDIUM PRIORITY).** FV uses `z * std + mean` at decode normalization; upstream uses `z / (1/std) + mean`. Bitwise non-equivalent in fp32. Affects all Wan-family pipelines (`fastvideo/configs/pipelines/wan.py`, `turbodiffusion.py`, `longcat.py`, magi-human). Magi VAE test loosened to `atol=1e-3, rtol=1e-3` (Wave 4) to defer. Tighten back to `atol=1e-4` once the Wan VAE op-order fix lands. Fix should be validated against Wan2.1, Wan2.2, and magi-human. Estimated 0.5-1 day to fix and validate. | TRACKED FOLLOW-UP |
| OQ-9 | **Upstream `dit_module.py` hardcoded bf16 casts block full fp32 parity validation.** `daVinci-MagiHuman/inference/model/dit/dit_module.py` has hardcoded `.to(torch.bfloat16)` casts at lines 619, 507, 650, 694, 696. FV's DiT forward is now dtype-agnostic (Wave 10), but parity tests against upstream still show bf16-noise residual drift in fp32 runs because upstream's intermediate tensors are bf16. Full fp32-clean parity would require patching the local upstream clone or adding a dtype-config flag upstream. Affects only fp32 parity testing, not production. | TRACKED FOLLOW-UP (LOW PRIORITY) |
## Troubleshooting
@@ -479,13 +497,13 @@ loaders for the upstream DiT, VAE, and pipeline. Use these as the starting
point for any new parity test rather than duplicating the load logic.
The `_debug_magi_human_block_parity.py` and `_debug_magi_human_weight_diff.py`
scripts in `tests/local_tests/transformers/` are scratch tools for divergence
scripts in `tests/local_tests/magi_human/` are scratch tools for divergence
investigation. They are NOT pytest tests and must NOT be promoted to formal
tests. Run them directly with `python` when you need to inspect per-block diffs
or weight mismatches during a parity-debug session.
If you need to chase per-layer divergence on a future add-model port, see the
`add-model-trace` skill at `~/.config/opencode/skill/add-model-trace/`.
Generalized from `tests/local_tests/transformers/_debug_magi_human_block_parity.py`
Generalized from `tests/local_tests/magi_human/_debug_magi_human_block_parity.py`
(the worked magi example), it provides a forward-hook + monkey-patch +
git-stash-cleanup methodology with hard rules around no-source-residue cleanup.