SolitaryThinker
0d99e47e16
Ports the daVinci-MagiHuman TI2V branch to FastVideo for both base
and distill variants. The TI2V case takes a reference image, encodes
it through the Wan VAE, and overwrites the first frame's video latent
with the encoded image latent at every denoise step (mirrors upstream
inference/pipeline/video_generate.py:300-360 evaluate + 424-425
per-step overwrite).
Files added:
* fastvideo/pipelines/basic/magi_human/stages/reference_image.py:
new MagiHumanReferenceImageStage. Loads PIL image (or path),
resizecrops to (height, width) matching upstream resizecrop,
runs VideoProcessor.preprocess at vae_scale_factor=16, encodes
via the Wan VAE (uses .mean for deterministic latent), applies
shift_factor / scaling_factor normalization, stashes on
batch.image_latent.
* tests/local_tests/magi_human/test_magi_human_ti2v_pipeline_parity.py:
bit-exact parity test against upstream MagiEvaluator's TI2V denoise
(with reference image conditioning). Passes
ti2v video diff_max=0.0000 diff_mean=0.0000.
Files modified:
* pipeline_configs.py: MagiHumanBaseI2VConfig keeps the VAE encoder
loaded (load_encoder=True) so the reference image path can encode.
* presets.py: MAGI_HUMAN_BASE_TI2V and MAGI_HUMAN_DISTILL_TI2V presets
with workload_type=i2v.
* registry.py: TI2V config entries for both base and distill variants.
* magi_human_pipeline.py: MagiHumanI2VPipeline subclass that inserts
the MagiHumanReferenceImageStage between prompt encoding and latent
preparation. Reuses the lazy-load path for shared components.
* stages/latent_preparation.py: pre-loop overwrite of
latent_video[:, :, :1] with batch.image_latent[:, :, :1] when
image conditioning is present (matches upstream
evaluate_with_latent line 425 first-iteration overwrite).
* stages/denoising.py: per-step _overwrite_first_frame helper that
applies the same overwrite at the start of every denoise step
(matches upstream evaluate_with_latent line 424 in-loop overwrite).
static_packed rebuild moved inside the loop after the overwrite
so packed video tokens reflect the conditioned latent.
* examples/inference/basic/basic_magi_human_ti2v.py: rewritten from
NotImplementedError stub to runnable. Uses local
converted_weights/magi_human_base + the existing example
saxophonist reference image; produces a coherent mp4 with the
image-conditioned subject.
* examples/inference/basic/basic_magi_human_distill_ti2v.py: same
pattern against converted_weights/magi_human_distill (runnable
once distill weights are converted).
Verification:
* pytest tests/local_tests/magi_human/ -v -s: 10 passed, 0 failed.
All bit-exact (diff_max=0.0, diff_mean=0.0) including new
test_magi_human_distill_dit_parity (Phase 1) and
test_magi_human_ti2v_pipeline_parity (Phase 2) tests.
* basic_magi_human.py (T2V regression check): mp4 byte-identical
md5 dcf5f2bf6534c7c0d91e7353e42b23db.
* basic_magi_human_ti2v.py: produces coherent saxophonist scene
matching the reference image conditioning. Frames at
/tmp/opencode/ti2v_frame_*.png.
T2V flow unchanged: both T2V example and base parity test produce
identical output to pre-change. TI2V is purely additive on top.
35 lines
992 B
Python
35 lines
992 B
Python
# SPDX-License-Identifier: Apache-2.0
|
|
"""Minimal daVinci-MagiHuman base text+image-to-AV example."""
|
|
from fastvideo import VideoGenerator
|
|
from fastvideo.pipelines.basic.magi_human.pipeline_configs import (
|
|
MagiHumanBaseI2VConfig,
|
|
)
|
|
|
|
|
|
PROMPT = (
|
|
"A cheerful saxophonist performs a short line with expressive facial "
|
|
"motion, natural head movement, and synchronized audio in a small jazz club."
|
|
)
|
|
IMAGE_PATH = "assets/images/saxophonist.jpg"
|
|
|
|
|
|
def main() -> None:
|
|
generator = VideoGenerator.from_pretrained(
|
|
"converted_weights/magi_human_base",
|
|
num_gpus=1,
|
|
workload_type="i2v",
|
|
override_pipeline_cls_name="MagiHumanI2VPipeline",
|
|
pipeline_config=MagiHumanBaseI2VConfig(),
|
|
)
|
|
generator.generate_video(
|
|
prompt=PROMPT,
|
|
image_path=IMAGE_PATH,
|
|
output_path="outputs_video/magi_human_ti2v/output_magi_human_ti2v.mp4",
|
|
save_video=True,
|
|
)
|
|
generator.shutdown()
|
|
|
|
|
|
if __name__ == "__main__":
|
|
main()
|