SolitaryThinker
e114fba53f
Upstream daVinci-MagiHuman ships 4 model variants x 2 input modes = 8
inference entrypoints (base / distill / sr_540p / sr_1080p, each in
T2V and TI2V mode). FastVideo currently has working code for the base
T2V path (basic_magi_human.py) and a registered preset for distill
T2V (magi_human_distill, but no example until now).
Add example files for the 7 remaining variants:
* basic_magi_human_distill.py -- runnable T2V example for the
DMD-2 distilled model. Just point conversion at the distill
subfolder and the existing magi_human_distill preset takes over.
* basic_magi_human_ti2v.py
basic_magi_human_distill_ti2v.py -- not-yet-ported TI2V (image
conditioning) variants. Each docstring lists the pipeline-side
work that is missing in FastVideo (VAE encoder load, reference
image stage, latent_video[..., :1] overwrite at every denoise
step, new I2V config + preset). main() raises NotImplementedError
with a pointer to magi-human.md.
* basic_magi_human_sr540p.py
basic_magi_human_sr540p_ti2v.py -- not-yet-ported super-resolution
to 540p. Docstring describes the upstream two-stage flow (base ->
SR latent prep with trilinear up + ZeroSNR noise -> SR DiT -> Wan
VAE) and lists the FV components needed (MagiHumanSR540pConfig,
SR latent prep stage, SR denoise stage with cfg-trick guidance
tensor, conversion script invocation for the 540p_sr subfolder,
new preset + registry entry).
* basic_magi_human_sr1080p.py
basic_magi_human_sr1080p_ti2v.py -- not-yet-ported SR-1080p.
Same SR-540p scaffolding plus a block-sparse local-window
attention path (32 of 40 SR DiT layers). The docstring points at
upstream's FFAHandler q_ranges/k_ranges blocks in dit_module.py
and notes that MagiAttention currently always runs full SDPA.
All stubs follow the existing example file convention (SPDX header,
focused docstring, single main()). The not-yet-ported stubs exit with
NotImplementedError so they fail loudly rather than silently misbehaving;
each error message points at the docstring for the missing-component
checklist.
57 lines
2.4 KiB
Python
57 lines
2.4 KiB
Python
# SPDX-License-Identifier: Apache-2.0
|
|
"""Stub example for daVinci-MagiHuman SR-540p text-to-AV — NOT YET PORTED.
|
|
|
|
Mirrors upstream `daVinci-MagiHuman/example/sr_540p/run_T2V.sh`.
|
|
|
|
Upstream pipeline overview (see
|
|
`daVinci-MagiHuman/inference/pipeline/video_generate.py:300-360`,
|
|
`MagiEvaluator.evaluate`):
|
|
|
|
1. Run base model at 256x480 (`magi_human_base`) to get
|
|
`(br_latent_video, br_latent_audio)`.
|
|
2. Trilinear-interpolate the BR latent up to the SR latent shape
|
|
`(latent_length, sr_latent_height, sr_latent_width)` and add
|
|
ZeroSNR noise via `noise_value=220`.
|
|
3. Replace audio with mostly-fresh noise:
|
|
`randn_like(br_latent_audio) * 0.7 + br_latent_audio * 0.3`.
|
|
4. Run a separate SR DiT (`sr_arch_config`, same arch as base, 5 steps)
|
|
with `sr_video_txt_guidance_scale=3.5` and the cfg-trick that drops
|
|
guidance for the first `cfg_trick_start_frame=13` frames to avoid
|
|
overexposure on I2V hand-offs.
|
|
5. Decode with the Wan VAE.
|
|
|
|
What is missing in FastVideo:
|
|
|
|
* `MagiHumanSR540pConfig` (pipeline_config) wiring two DiTs (base + SR)
|
|
plus the `sr_video_txt_guidance_scale`, `cfg_trick_*`,
|
|
`noise_value`, `sr_audio_noise_scale`, `sr_num_inference_steps`
|
|
knobs surfaced in `inference/common/config.py:EvaluationConfig`.
|
|
* A two-stage pipeline: base denoise → SR latent prep
|
|
(`MagiHumanSRLatentPreparationStage`: trilinear up + noise mix) →
|
|
SR denoise → decode. Either a new `ComposedPipelineBase` subclass
|
|
that runs base + SR back-to-back or a meta-pipeline that calls
|
|
both as inner pipelines.
|
|
* `MagiHumanSRDenoisingStage` with the per-frame cfg-trick guidance
|
|
tensor (`evaluate_with_latent` lines 412-418).
|
|
* Conversion script support for the `540p_sr` subfolder (the existing
|
|
`--subfolder` choices already accept it; just need a separate
|
|
`convert_magi_human_to_diffusers ... --subfolder 540p_sr ...`
|
|
invocation that produces a `converted_weights/magi_human_sr_540p`
|
|
layout with `transformer/` AND `sr_transformer/`).
|
|
* A new preset (e.g. `magi_human_sr_540p`) and registry entry.
|
|
"""
|
|
import sys
|
|
|
|
|
|
def main() -> None:
|
|
raise NotImplementedError(
|
|
"MagiHuman SR-540p T2V is not yet ported. See the docstring at the "
|
|
"top of this file for the missing pipeline-side work, and "
|
|
"`tests/local_tests/magi-human.md` for the port-status journal.",
|
|
)
|
|
|
|
|
|
if __name__ == "__main__":
|
|
main()
|
|
sys.exit(0)
|