Ports the daVinci-MagiHuman TI2V branch to FastVideo for both base
and distill variants. The TI2V case takes a reference image, encodes
it through the Wan VAE, and overwrites the first frame's video latent
with the encoded image latent at every denoise step (mirrors upstream
inference/pipeline/video_generate.py:300-360 evaluate + 424-425
per-step overwrite).
Files added:
* fastvideo/pipelines/basic/magi_human/stages/reference_image.py:
new MagiHumanReferenceImageStage. Loads PIL image (or path),
resizecrops to (height, width) matching upstream resizecrop,
runs VideoProcessor.preprocess at vae_scale_factor=16, encodes
via the Wan VAE (uses .mean for deterministic latent), applies
shift_factor / scaling_factor normalization, stashes on
batch.image_latent.
* tests/local_tests/magi_human/test_magi_human_ti2v_pipeline_parity.py:
bit-exact parity test against upstream MagiEvaluator's TI2V denoise
(with reference image conditioning). Passes
ti2v video diff_max=0.0000 diff_mean=0.0000.
Files modified:
* pipeline_configs.py: MagiHumanBaseI2VConfig keeps the VAE encoder
loaded (load_encoder=True) so the reference image path can encode.
* presets.py: MAGI_HUMAN_BASE_TI2V and MAGI_HUMAN_DISTILL_TI2V presets
with workload_type=i2v.
* registry.py: TI2V config entries for both base and distill variants.
* magi_human_pipeline.py: MagiHumanI2VPipeline subclass that inserts
the MagiHumanReferenceImageStage between prompt encoding and latent
preparation. Reuses the lazy-load path for shared components.
* stages/latent_preparation.py: pre-loop overwrite of
latent_video[:, :, :1] with batch.image_latent[:, :, :1] when
image conditioning is present (matches upstream
evaluate_with_latent line 425 first-iteration overwrite).
* stages/denoising.py: per-step _overwrite_first_frame helper that
applies the same overwrite at the start of every denoise step
(matches upstream evaluate_with_latent line 424 in-loop overwrite).
static_packed rebuild moved inside the loop after the overwrite
so packed video tokens reflect the conditioned latent.
* examples/inference/basic/basic_magi_human_ti2v.py: rewritten from
NotImplementedError stub to runnable. Uses local
converted_weights/magi_human_base + the existing example
saxophonist reference image; produces a coherent mp4 with the
image-conditioned subject.
* examples/inference/basic/basic_magi_human_distill_ti2v.py: same
pattern against converted_weights/magi_human_distill (runnable
once distill weights are converted).
Verification:
* pytest tests/local_tests/magi_human/ -v -s: 10 passed, 0 failed.
All bit-exact (diff_max=0.0, diff_mean=0.0) including new
test_magi_human_distill_dit_parity (Phase 1) and
test_magi_human_ti2v_pipeline_parity (Phase 2) tests.
* basic_magi_human.py (T2V regression check): mp4 byte-identical
md5 dcf5f2bf6534c7c0d91e7353e42b23db.
* basic_magi_human_ti2v.py: produces coherent saxophonist scene
matching the reference image conditioning. Frames at
/tmp/opencode/ti2v_frame_*.png.
T2V flow unchanged: both T2V example and base parity test produce
identical output to pre-change. TI2V is purely additive on top.
Extends the existing lazy-load pattern for text_encoder, tokenizer,
and audio_vae to the video VAE: each MagiHuman variant's converted
repo no longer needs to bundle a copy of the Wan 2.2 TI2V-5B VAE.
The four cross-variant shared components are now all fetched from
their canonical upstream HF repos at first build:
* text_encoder, tokenizer -> google/t5gemma-9b-9b-ul2 (gated)
* audio_vae -> stabilityai/stable-audio-open-1.0 (gated)
* vae -> Wan-AI/Wan2.2-TI2V-5B-Diffusers
Per-variant converted repo shrinks to transformer/ + scheduler/ +
model_index.json (~5 GB for base bf16, ~30 GB for distill bf16). All
variants share the same ~25 GB cache of upstream weights, so a user
running 4 variants ends up with ~75 GB total instead of ~400 GB.
Implementation:
* fastvideo/utils.py:verify_model_config_and_directory now treats
the contents of model_index.json as authoritative for which
component subfolders must exist locally. Pipelines that emit a
minimal model_index.json (omitting vae / text_encoder / etc.)
pass verification; pipelines that DO declare a component must
still ship its subfolder. transformer/ remains mandatory.
* fastvideo/pipelines/basic/magi_human/magi_human_pipeline.py adds
vae to the deferred list in load_modules and a new
_load_video_vae helper that prefers a bundled vae/ subfolder
(legacy converted repos) and falls back to snapshot_download +
the standard FV VAELoader. Both paths produce the same FV
AutoencoderKLWan, so production behavior is unchanged.
* convert_magi_human_to_diffusers.py docstring updated; --bundle-vae
flag is unchanged (still optional) but the README example now
omits it so new converted repos default to the minimal layout.
Verification:
* basic_magi_human.py with bundled vae/: byte-identical mp4 (md5
dcf5f2bf...) to pre-refactor output.
* basic_magi_human.py with vae/ moved out and removed from
model_index.json: same byte-identical mp4 via the lazy-load path.
* 4/4 magi-human parity tests pass with diff_max=0.0 / diff_mean=0.0
(DiT, pipeline, smoke + typed surface preflight).
Upstream daVinci-MagiHuman ships 4 model variants x 2 input modes = 8
inference entrypoints (base / distill / sr_540p / sr_1080p, each in
T2V and TI2V mode). FastVideo currently has working code for the base
T2V path (basic_magi_human.py) and a registered preset for distill
T2V (magi_human_distill, but no example until now).
Add example files for the 7 remaining variants:
* basic_magi_human_distill.py -- runnable T2V example for the
DMD-2 distilled model. Just point conversion at the distill
subfolder and the existing magi_human_distill preset takes over.
* basic_magi_human_ti2v.py
basic_magi_human_distill_ti2v.py -- not-yet-ported TI2V (image
conditioning) variants. Each docstring lists the pipeline-side
work that is missing in FastVideo (VAE encoder load, reference
image stage, latent_video[..., :1] overwrite at every denoise
step, new I2V config + preset). main() raises NotImplementedError
with a pointer to magi-human.md.
* basic_magi_human_sr540p.py
basic_magi_human_sr540p_ti2v.py -- not-yet-ported super-resolution
to 540p. Docstring describes the upstream two-stage flow (base ->
SR latent prep with trilinear up + ZeroSNR noise -> SR DiT -> Wan
VAE) and lists the FV components needed (MagiHumanSR540pConfig,
SR latent prep stage, SR denoise stage with cfg-trick guidance
tensor, conversion script invocation for the 540p_sr subfolder,
new preset + registry entry).
* basic_magi_human_sr1080p.py
basic_magi_human_sr1080p_ti2v.py -- not-yet-ported SR-1080p.
Same SR-540p scaffolding plus a block-sparse local-window
attention path (32 of 40 SR DiT layers). The docstring points at
upstream's FFAHandler q_ranges/k_ranges blocks in dit_module.py
and notes that MagiAttention currently always runs full SDPA.
All stubs follow the existing example file convention (SPDX header,
focused docstring, single main()). The not-yet-ported stubs exit with
NotImplementedError so they fail loudly rather than silently misbehaving;
each error message points at the docstring for the missing-component
checklist.
- Drop misleading "T2V" framing — base MagiHuman is a joint audio-visual
generator. Rename MagiHumanT2VConfig→MagiHumanBaseConfig,
MagiHumanDistillT2VConfig→MagiHumanDistillConfig, presets
magi_human_base_t2v→magi_human_base, magi_human_distill_t2v→
magi_human_distill. Update comments / docstrings / example output
filename. Keep workload_type="t2v" string (framework enum has no
T2AV variant yet — same placeholder Stable Audio uses for T2A).
- Pipeline parity test: tighten atol/rtol from 0.5/0.5 to 0.35/0.05.
atol absorbs the observed worst-element drift (~0.31 on signal abs_mean
~2.4); the tight rtol still flags gross structural bugs.
- Wan VAE parity: import FastVideo's AutoencoderKLWan from
fastvideo.models.vaes.wanvae instead of diffusers. The test now
actually validates the class MagiHumanBaseConfig.vae_config resolves
to in production.
- Pipeline parity scheduler init: split per-side so upstream mirrors
MagiEvaluator's single-shift pattern (fresh FlowUniPCMultistepScheduler
+ shift only in set_timesteps) and FastVideo mirrors
MagiHumanDenoisingStage's double-shift pattern (constructor + set_timesteps
both with shift). Surfaces a real production divergence between
FastVideo's magi denoise loop and the official one for N>1 steps.
Ports GAIR-NLP/daVinci-MagiHuman's 15B-param joint-AV DiT into FastVideo:
text + video + audio joint denoise in one flat token stream, 32-step
FlowUniPC w/ CFG=2, Wan 2.2 TI2V-5B video VAE, Stable Audio Open 1.0
audio VAE (first-class port at fastvideo/models/vaes/oobleck.py, no
runtime diffusers import), T5-Gemma 9B UL2 text encoder, auto-muxed
mp4 with h264 video + aac stereo audio.
Parity tests all pass on converted weights: DiT (diff_median=2.6e-3),
text encoder (exact), video VAE (8e-4), audio VAE (exact), pipeline
latent (5e-2).
See .agents/skills/add-model/REVIEW.md for porting procedure + open
review items.
SolitaryThinker