SolitaryThinker
3464cb8b03
Extends the existing lazy-load pattern for text_encoder, tokenizer,
and audio_vae to the video VAE: each MagiHuman variant's converted
repo no longer needs to bundle a copy of the Wan 2.2 TI2V-5B VAE.
The four cross-variant shared components are now all fetched from
their canonical upstream HF repos at first build:
* text_encoder, tokenizer -> google/t5gemma-9b-9b-ul2 (gated)
* audio_vae -> stabilityai/stable-audio-open-1.0 (gated)
* vae -> Wan-AI/Wan2.2-TI2V-5B-Diffusers
Per-variant converted repo shrinks to transformer/ + scheduler/ +
model_index.json (~5 GB for base bf16, ~30 GB for distill bf16). All
variants share the same ~25 GB cache of upstream weights, so a user
running 4 variants ends up with ~75 GB total instead of ~400 GB.
Implementation:
* fastvideo/utils.py:verify_model_config_and_directory now treats
the contents of model_index.json as authoritative for which
component subfolders must exist locally. Pipelines that emit a
minimal model_index.json (omitting vae / text_encoder / etc.)
pass verification; pipelines that DO declare a component must
still ship its subfolder. transformer/ remains mandatory.
* fastvideo/pipelines/basic/magi_human/magi_human_pipeline.py adds
vae to the deferred list in load_modules and a new
_load_video_vae helper that prefers a bundled vae/ subfolder
(legacy converted repos) and falls back to snapshot_download +
the standard FV VAELoader. Both paths produce the same FV
AutoencoderKLWan, so production behavior is unchanged.
* convert_magi_human_to_diffusers.py docstring updated; --bundle-vae
flag is unchanged (still optional) but the README example now
omits it so new converted repos default to the minimal layout.
Verification:
* basic_magi_human.py with bundled vae/: byte-identical mp4 (md5
dcf5f2bf...) to pre-refactor output.
* basic_magi_human.py with vae/ moved out and removed from
model_index.json: same byte-identical mp4 via the lazy-load path.
* 4/4 magi-human parity tests pass with diff_max=0.0 / diff_mean=0.0
(DiT, pipeline, smoke + typed surface preflight).
54 lines
2.0 KiB
Python
54 lines
2.0 KiB
Python
# SPDX-License-Identifier: Apache-2.0
|
|
"""Minimal user-runnable example for the daVinci-MagiHuman base AV pipeline.
|
|
|
|
Produces an mp4 with both video (Wan 2.2 TI2V-5B VAE) and audio (Stable
|
|
Audio Open 1.0 VAE, first-class FastVideo port in
|
|
`fastvideo/models/vaes/oobleck.py`) muxed together via PyAV.
|
|
|
|
Prerequisites (one-off):
|
|
|
|
# 1) Accept terms of use on the gated HF repos with your HF_TOKEN:
|
|
# - https://huggingface.co/google/t5gemma-9b-9b-ul2
|
|
# - https://huggingface.co/stabilityai/stable-audio-open-1.0
|
|
# All four cross-variant shared components — Wan 2.2 VAE, T5-Gemma
|
|
# encoder + tokenizer, and Stable Audio VAE — are lazy-loaded from
|
|
# their canonical upstream HF repos on first build, so a single
|
|
# cache (~25 GB) is shared across every MagiHuman variant.
|
|
# 2) Convert the raw MagiHuman base DiT to a minimal Diffusers layout
|
|
# (~5 GB; only transformer/ + scheduler/ + model_index.json):
|
|
python scripts/checkpoint_conversion/convert_magi_human_to_diffusers.py \\
|
|
--source GAIR/daVinci-MagiHuman \\
|
|
--subfolder base \\
|
|
--output converted_weights/magi_human_base
|
|
# Pass --bundle-vae / --bundle-audio-vae / --bundle-text-encoder only
|
|
# if you want a self-contained snapshot.
|
|
"""
|
|
from fastvideo import VideoGenerator
|
|
|
|
|
|
PROMPT = (
|
|
"A warm afternoon scene: a person sits on a park bench reading a book, "
|
|
"surrounded by softly swaying trees."
|
|
)
|
|
|
|
|
|
def main() -> None:
|
|
generator = VideoGenerator.from_pretrained(
|
|
"converted_weights/magi_human_base",
|
|
num_gpus=1,
|
|
)
|
|
output_path = "outputs_video/magi_human_basic/output_magi_human.mp4"
|
|
generator.generate_video(
|
|
prompt=PROMPT,
|
|
output_path=output_path,
|
|
save_video=True,
|
|
# Defaults pulled from the registered preset (magi_human_base):
|
|
# height=256, width=448, fps=25, num_inference_steps=32, seed=42.
|
|
# Override here only if you have a specific QA scenario.
|
|
)
|
|
generator.shutdown()
|
|
|
|
|
|
if __name__ == "__main__":
|
|
main()
|