[feat] Port daVinci-MagiHuman base model with AV output

Ports GAIR-NLP/daVinci-MagiHuman's 15B-param joint-AV DiT into FastVideo:
text + video + audio joint denoise in one flat token stream, 32-step
FlowUniPC w/ CFG=2, Wan 2.2 TI2V-5B video VAE, Stable Audio Open 1.0
audio VAE (first-class port at fastvideo/models/vaes/oobleck.py, no
runtime diffusers import), T5-Gemma 9B UL2 text encoder, auto-muxed
mp4 with h264 video + aac stereo audio.

Parity tests all pass on converted weights: DiT (diff_median=2.6e-3),
text encoder (exact), video VAE (8e-4), audio VAE (exact), pipeline
latent (5e-2).

See .agents/skills/add-model/REVIEW.md for porting procedure + open
review items.
This commit is contained in:
SolitaryThinker
2026-05-05 10:55:49 -07:00
parent 12a7cb53ff
commit e7d6c40860
31 changed files with 4515 additions and 2 deletions
@@ -0,0 +1,53 @@
# SPDX-License-Identifier: Apache-2.0
"""Minimal user-runnable example for the daVinci-MagiHuman base AV pipeline.
Produces an mp4 with both video (Wan 2.2 TI2V-5B VAE) and audio (Stable
Audio Open 1.0 VAE, first-class FastVideo port in
`fastvideo/models/vaes/oobleck.py`) muxed together via PyAV.
Prerequisites (one-off):
# 1) Accept terms of use on the gated HF repos with your HF_TOKEN:
# - https://huggingface.co/google/t5gemma-9b-9b-ul2
# - https://huggingface.co/stabilityai/stable-audio-open-1.0
# Lazy-loaded on first forward; the pipeline fails loudly if either
# access is denied.
# 2) Clone upstream repo (only needed for parity tests, not this example):
# git clone --depth 1 https://github.com/GAIR-NLP/daVinci-MagiHuman.git
# 3) Convert the raw MagiHuman base checkpoint to a Diffusers layout
# and bundle the Wan 2.2 TI2V-5B VAE:
python scripts/checkpoint_conversion/convert_magi_human_to_diffusers.py \\
--source GAIR/daVinci-MagiHuman \\
--output converted_weights/magi_human_base \\
--bundle-vae
# Optional: also bundle the audio VAE (`--bundle-audio-vae`) and/or the
# T5-Gemma encoder (`--bundle-text-encoder`). Either works lazy too.
"""
from fastvideo import VideoGenerator
PROMPT = (
"A warm afternoon scene: a person sits on a park bench reading a book, "
"surrounded by softly swaying trees."
)
def main() -> None:
generator = VideoGenerator.from_pretrained(
"converted_weights/magi_human_base",
num_gpus=1,
)
output_path = "outputs_video/magi_human_basic/output_magi_human_t2v.mp4"
generator.generate_video(
prompt=PROMPT,
output_path=output_path,
save_video=True,
# Defaults pulled from the registered preset (magi_human_base_t2v):
# height=256, width=448, fps=25, num_inference_steps=32, seed=42.
# Override here only if you have a specific QA scenario.
)
generator.shutdown()
if __name__ == "__main__":
main()