Ports the daVinci-MagiHuman SR-1080p inference flow to FastVideo. Builds
on the SR-540p two-stage pipeline plus block-sparse local-window
video->video attention on 32 of 40 SR DiT layers. Mirrors upstream
SR2_1080 config override at inference/common/config.py:225-244 and
calc_local_qk_range at inference/pipeline/data_proxy.py:31-79.
Files added:
* tests/local_tests/magi_human/test_magi_human_sr1080p_pipeline_parity.py:
parametric parity test for both T2V and TI2V modes. Both pass
diff_max=0.0000 / diff_mean=0.0000 -- BIT-EXACT.
Files modified (key changes):
* fastvideo/models/dits/magi_human.py:
- AttentionSubConfig.use_local_attn / frame_receptive_field
- MagiAttention.configure_local_attention(): per-layer toggle
- MagiAttention._sdpa(): thin SDPA wrapper for [L,H,D] tensors
- MagiAttention._local_window_attention(): 3-block accumulator
mirroring upstream FFA semantics with vanilla SDPA segments
(per-frame video window + all video->audio+text + audio+text->all).
- MagiAttention.forward() dispatches to _local_window_attention
when use_local_attn flag is set.
- MagiTransformerLayer wires use_local_attn from arch.local_attn_layers.
- MagiHumanDiT.configure_local_attention() top-level toggle.
* pipeline_configs.py: MagiHumanSR1080pConfig + I2V variant with
sr_local_attn_layers populated to upstream's 32 indices.
* presets.py: MAGI_HUMAN_SR_1080P + MAGI_HUMAN_SR_1080P_TI2V presets.
* registry.py: SR-1080p config entries with detectors.
* magi_human_pipeline.py: SR-1080p pipeline classes activating
local_attn_layers on the SR DiT at construction.
* scripts/checkpoint_conversion/convert_magi_human_to_diffusers.py:
--sr-subfolder 1080p_sr support.
* tests/local_tests/helpers/magi_human_upstream.py: arch override
support for SR-1080p parity test.
* test_magi_human_pipeline_smoke.py: preset set expanded.
* basic_magi_human_sr1080p{,_ti2v}.py: stubs -> runnable.
Verification:
* pytest tests/local_tests/magi_human/ -v -s: 14 passed, 0 failed.
All bit-exact including SR-1080p T2V and TI2V parity. The
block-sparse SDPA-segmented implementation matches upstream FFA's
q_ranges/k_ranges accumulator semantics exactly for the 3-block
layout (overlap accumulation handled by explicit '+' at
magi_human.py:421-425).
* basic_magi_human.py (T2V regression check): mp4 byte-identical
md5 dcf5f2bf6534c7c0d91e7353e42b23db.
* Base/Distill/TI2V/SR-540p flows untouched.
Notes:
* Converted SR-1080p artifact at /raid/.../magi_human_sr_1080p
(~58 GB), symlinked at converted_weights/magi_human_sr_1080p
(gitignored).
* Upstream's flex_flash_attn_func via SandAI-org/MagiAttention is
NOT a dependency; FV's pure-SDPA segmented implementation is
mathematically equivalent for this 3-block layout.
Ports the daVinci-MagiHuman SR-540p inference flow to FastVideo for
both T2V and TI2V modes. SR-540p is a TWO-STAGE pipeline: the base
model produces a 256x480 latent, then a separate SR DiT (same arch as
base, different weights) refines it to 896x512. Mirrors upstream
MagiEvaluator.evaluate at video_generate.py:300-360.
Files added:
* fastvideo/pipelines/basic/magi_human/stages/sr_latent_preparation.py
* fastvideo/pipelines/basic/magi_human/stages/sr_denoising.py
* tests/local_tests/magi_human/test_magi_human_sr540p_pipeline_parity.py
Files modified (key changes):
* pipeline_configs.py: SR config classes with sr_* knobs sourced
from upstream EvaluationConfig
* presets.py: MAGI_HUMAN_SR_540P + MAGI_HUMAN_SR_540P_TI2V
* registry.py: SR-540p config entries
* magi_human_pipeline.py: MagiHumanSRPipeline + MagiHumanSRI2VPipeline
classes wiring the 9-stage chain (base denoise -> sr latent prep
-> sr denoise -> decode)
* fastvideo/models/loader/component_loader.py: registers
'sr_transformer' alongside transformer / transformer_2
* scripts/checkpoint_conversion/convert_magi_human_to_diffusers.py:
--sr-source / --sr-subfolder flags for SR DiT into sr_transformer/
* examples/inference/basic/basic_magi_human_sr540p{,_ti2v}.py:
runnable
Verification:
* pytest tests/local_tests/magi_human/ -v -s: 12 passed, 0 failed.
All bit-exact (diff_max=0.0, diff_mean=0.0) including new SR-540p
T2V + TI2V parity tests.
* basic_magi_human.py (T2V regression check): mp4 byte-identical
md5 dcf5f2bf6534c7c0d91e7353e42b23db.
* basic_magi_human_sr540p.py: 896x512 mp4, coherent reading-on-
park-bench scene, much higher quality than base 480x256.
* basic_magi_human_sr540p_ti2v.py: 896x512 mp4 with reference-image-
conditioned saxophonist; reference conditioning preserved through
SR upscale.
Notes:
* Converted SR-540p artifact at converted_weights/magi_human_sr_540p
(~86 GB; both transformer/ and sr_transformer/) is gitignored.
* Base T2V/TI2V/distill flows untouched and bit-exact.
The conversion script's _FP32_KEEP_SUFFIXES list was missing 8 keys
that the BASE checkpoint stores as fp32:
adapter.video_embedder.{weight,bias}
adapter.text_embedder.{weight,bias}
adapter.audio_embedder.{weight,bias}
final_linear_video.weight
final_linear_audio.weight
For the BASE conversion this never surfaced because the BASE checkpoint
already ships these as fp32 (no --cast-bf16 needed; conversion was
identity for these weights). For the DISTILL conversion (which ships
ALL 331 weights as fp32 and relies on --cast-bf16 to produce a 30 GB
bf16 artifact), the omission caused these 8 fp32 layers to be cast
to bf16, which mismatched FV's MagiAdapter/final_linear modules
(declared dtype=torch.float32 at magi_human.py:519-527 and 645-648,
mirroring upstream Adapter at dit_module.py:721-723 and DiTModel at
dit_module.py:896-900).
Symptom: distill DiT parity vs upstream had video diff_mean=0.114
(19% relative error) instead of the expected 0.0. Adding the 8 keys
to _FP32_KEEP_SUFFIXES restores bit-exact parity.
Also adds tests/local_tests/magi_human/test_magi_human_distill_parity.py
mirroring the base DiT parity test but pointing at the distill shards
and converted weights. Bit-exact diff_max=0.0, diff_mean=0.0 vs
upstream daVinci-MagiHuman/inference/model/dit/dit_module.py:DiTModel
loaded from the distill subfolder.
Verified with --cast-bf16 reconversion of GAIR/daVinci-MagiHuman/distill
into converted_weights/magi_human_distill (29 GB).
Extends the existing lazy-load pattern for text_encoder, tokenizer,
and audio_vae to the video VAE: each MagiHuman variant's converted
repo no longer needs to bundle a copy of the Wan 2.2 TI2V-5B VAE.
The four cross-variant shared components are now all fetched from
their canonical upstream HF repos at first build:
* text_encoder, tokenizer -> google/t5gemma-9b-9b-ul2 (gated)
* audio_vae -> stabilityai/stable-audio-open-1.0 (gated)
* vae -> Wan-AI/Wan2.2-TI2V-5B-Diffusers
Per-variant converted repo shrinks to transformer/ + scheduler/ +
model_index.json (~5 GB for base bf16, ~30 GB for distill bf16). All
variants share the same ~25 GB cache of upstream weights, so a user
running 4 variants ends up with ~75 GB total instead of ~400 GB.
Implementation:
* fastvideo/utils.py:verify_model_config_and_directory now treats
the contents of model_index.json as authoritative for which
component subfolders must exist locally. Pipelines that emit a
minimal model_index.json (omitting vae / text_encoder / etc.)
pass verification; pipelines that DO declare a component must
still ship its subfolder. transformer/ remains mandatory.
* fastvideo/pipelines/basic/magi_human/magi_human_pipeline.py adds
vae to the deferred list in load_modules and a new
_load_video_vae helper that prefers a bundled vae/ subfolder
(legacy converted repos) and falls back to snapshot_download +
the standard FV VAELoader. Both paths produce the same FV
AutoencoderKLWan, so production behavior is unchanged.
* convert_magi_human_to_diffusers.py docstring updated; --bundle-vae
flag is unchanged (still optional) but the README example now
omits it so new converted repos default to the minimal layout.
Verification:
* basic_magi_human.py with bundled vae/: byte-identical mp4 (md5
dcf5f2bf...) to pre-refactor output.
* basic_magi_human.py with vae/ moved out and removed from
model_index.json: same byte-identical mp4 via the lazy-load path.
* 4/4 magi-human parity tests pass with diff_max=0.0 / diff_mean=0.0
(DiT, pipeline, smoke + typed surface preflight).
- Drop misleading "T2V" framing — base MagiHuman is a joint audio-visual
generator. Rename MagiHumanT2VConfig→MagiHumanBaseConfig,
MagiHumanDistillT2VConfig→MagiHumanDistillConfig, presets
magi_human_base_t2v→magi_human_base, magi_human_distill_t2v→
magi_human_distill. Update comments / docstrings / example output
filename. Keep workload_type="t2v" string (framework enum has no
T2AV variant yet — same placeholder Stable Audio uses for T2A).
- Pipeline parity test: tighten atol/rtol from 0.5/0.5 to 0.35/0.05.
atol absorbs the observed worst-element drift (~0.31 on signal abs_mean
~2.4); the tight rtol still flags gross structural bugs.
- Wan VAE parity: import FastVideo's AutoencoderKLWan from
fastvideo.models.vaes.wanvae instead of diffusers. The test now
actually validates the class MagiHumanBaseConfig.vae_config resolves
to in production.
- Pipeline parity scheduler init: split per-side so upstream mirrors
MagiEvaluator's single-shift pattern (fresh FlowUniPCMultistepScheduler
+ shift only in set_timesteps) and FastVideo mirrors
MagiHumanDenoisingStage's double-shift pattern (constructor + set_timesteps
both with shift). Surfaces a real production divergence between
FastVideo's magi denoise loop and the official one for N>1 steps.
Ports GAIR-NLP/daVinci-MagiHuman's 15B-param joint-AV DiT into FastVideo:
text + video + audio joint denoise in one flat token stream, 32-step
FlowUniPC w/ CFG=2, Wan 2.2 TI2V-5B video VAE, Stable Audio Open 1.0
audio VAE (first-class port at fastvideo/models/vaes/oobleck.py, no
runtime diffusers import), T5-Gemma 9B UL2 text encoder, auto-muxed
mp4 with h264 video + aac stereo audio.
Parity tests all pass on converted weights: DiT (diff_median=2.6e-3),
text encoder (exact), video VAE (8e-4), audio VAE (exact), pipeline
latent (5e-2).
See .agents/skills/add-model/REVIEW.md for porting procedure + open
review items.
SolitaryThinker