Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
e54106110e |
@@ -0,0 +1,77 @@
|
||||
# MagiHuman — Codebase Map Entry
|
||||
|
||||
**Family:** daVinci-MagiHuman (joint audio-visual generative model)
|
||||
**Reference:** [GAIR-NLP/daVinci-MagiHuman](https://github.com/GAIR-NLP/daVinci-MagiHuman)
|
||||
**Architecture:** 15B-param single-stream DiT, 40 layers, hidden=5120,
|
||||
head_dim=128, GQA num_query_groups=8. Joint AV denoising in a unified token
|
||||
sequence; **no cross-attention**.
|
||||
|
||||
## Variant Matrix
|
||||
|
||||
| Variant | T2V | TI2V | DiT | Steps | CFG | Resolution |
|
||||
|---|---|---|---|---|---|---|
|
||||
| `base` | yes | yes | base | 32 | 2 | 480x256 |
|
||||
| `distill` | yes | yes | distill (DMD-2) | 8 | 1 (no CFG) | 480x256 |
|
||||
| `sr_540p` | yes | yes | base + sr_540p | 32 + 5 | 2 + cfg-trick | 896x512 |
|
||||
| `sr_1080p` | yes | yes | base + sr_1080p | 32 + 5 | 2 + cfg-trick | 1920x1056 |
|
||||
|
||||
SR-1080p uses block-sparse video→video local-window attention on 32 of 40
|
||||
SR DiT layers (`frame_receptive_field=11`), implemented as a 3-block SDPA
|
||||
accumulator that mirrors upstream `flex_flash_attn_func`.
|
||||
|
||||
## File Locations
|
||||
|
||||
| Role | Path |
|
||||
|---|---|
|
||||
| Pipeline class | `fastvideo/pipelines/basic/magi_human/magi_human_pipeline.py` |
|
||||
| Pipeline package AGENTS.md | `fastvideo/pipelines/basic/magi_human/AGENTS.md` |
|
||||
| Stages | `fastvideo/pipelines/basic/magi_human/stages/*.py` |
|
||||
| DiT | `fastvideo/models/dits/magi_human.py` |
|
||||
| Text encoder (T5-Gemma) | `fastvideo/models/encoders/t5gemma.py` |
|
||||
| Audio VAE wrapper | `fastvideo/models/vaes/sa_audio.py` (shared with `stable_audio` pipeline) |
|
||||
| Conversion script | `scripts/checkpoint_conversion/convert_magi_human_to_diffusers.py` |
|
||||
| Examples | `examples/inference/basic/basic_magi_human*.py` (8 files, one per variant × mode) |
|
||||
| SSIM regression | `fastvideo/tests/ssim/test_magi_human_similarity.py` |
|
||||
| Local parity battery | `tests/local_tests/magi_human/` (14 tests, GPU-gated) |
|
||||
| Port journal | `fastvideo/pipelines/basic/magi_human/JOURNAL.md` |
|
||||
|
||||
## Canonical HF Repo
|
||||
|
||||
[FastVideo/MagiHuman-Diffusers](https://huggingface.co/FastVideo/MagiHuman-Diffusers)
|
||||
— umbrella repo with sibling subfolders per variant.
|
||||
|
||||
```python
|
||||
from fastvideo import VideoGenerator
|
||||
gen = VideoGenerator.from_pretrained("FastVideo/MagiHuman-Diffusers/base")
|
||||
gen.generate_video(prompt="...", output_path="out.mp4", save_video=True)
|
||||
```
|
||||
|
||||
Four shared upstream components are lazy-loaded by `MagiHumanPipeline.load_modules`:
|
||||
|
||||
| Component | Upstream repo |
|
||||
|---|---|
|
||||
| Wan 2.2 VAE | `Wan-AI/Wan2.2-TI2V-5B` |
|
||||
| T5-Gemma 9B UL2 | `google/t5gemma-9b-9b-ul2` |
|
||||
| Stable Audio VAE | `stabilityai/stable-audio-open-1.0` |
|
||||
| MagiHuman DiT weights | `GAIR/daVinci-MagiHuman` (gated) |
|
||||
|
||||
## Parity Invariants
|
||||
|
||||
Three load-bearing invariants. See `fastvideo/pipelines/basic/magi_human/AGENTS.md`
|
||||
for the full discussion.
|
||||
|
||||
1. **Channel-major video token packing** (`stages/latent_preparation.py`)
|
||||
2. **DiT dtype boundary**: residual stream stays fp32 across blocks
|
||||
3. **Conversion `_FP32_KEEP_SUFFIXES`** (`scripts/checkpoint_conversion/convert_magi_human_to_diffusers.py`)
|
||||
|
||||
## Lessons
|
||||
|
||||
- `.agents/lessons/2026-05-07_silent-channel-major-packing-bugs.md`
|
||||
- `.agents/lessons/2026-05-07_dit-dtype-boundary-with-flash-attn.md`
|
||||
- `.agents/lessons/2026-05-07_conversion-cast-bf16-suffix-allowlist.md`
|
||||
|
||||
## Provenance
|
||||
|
||||
Decomposed from PR [#1280](https://github.com/hao-ai-lab/FastVideo/pull/1280)
|
||||
(`will/magi` @ `4e1603634d27c8e1b5c4cc5d9387f046547f5c49`). See the package
|
||||
AGENTS.md for the full PR-stack table.
|
||||
@@ -232,25 +232,13 @@ by the harness and is not config-declarable.)
|
||||
|
||||
Recipe fingerprinting, hardware/software profile IDs, exact-identity
|
||||
comparison, and dashboard cohort grouping land with this change: v2 records
|
||||
compare only within their identity cohort and are never compared against v1
|
||||
records (or vice versa). Legacy records missing any comparison identity field
|
||||
still run and are normalized for reporting, but they skip rolling-baseline
|
||||
comparison entirely (`baseline_status: "skipped_missing_identity"`, comparator
|
||||
verdict `PASS`, never baseline eligible); only static thresholds gate them.
|
||||
Each comparison reports an explicit `comparator_status` on the normalized
|
||||
record and in the Markdown summary:
|
||||
|
||||
| Status | Meaning | CI |
|
||||
|---|---|---|
|
||||
| `PASS` | Comparable baseline found with no gated regression, or comparison was skipped for a record without the v2 identity block. | passes |
|
||||
| `REGRESSION` | A gated metric regressed past both its percent and absolute floors. | fails |
|
||||
| `CALIBRATION_NEEDED` | No comparable baseline exists. Gating is inactive and the record does **not** seed a baseline — seed new cohorts explicitly via the reseed workflow. The record also carries `baseline_status: "initialized_new_cohort"`. | passes |
|
||||
| `RECIPE_MISMATCH` | Baseline history exists for the same variant/hardware/software cohort but under a different `recipe_fingerprint`. Represent recipe changes as a new `variant_id`, or reseed. | fails |
|
||||
| `INFRA_ERROR` | The comparison itself failed (for example, baseline history could not be loaded). | fails |
|
||||
| `QUALITY_BLOCKED` | Reserved for the promoted-baseline workflow; never emitted by the comparator. | n/a |
|
||||
|
||||
Metric-specific threshold policies and promoted baselines remain separate
|
||||
follow-ups.
|
||||
compare only within their identity cohort, and a record that opens a NEW
|
||||
cohort is marked `baseline_status: "initialized_new_cohort"` (regression
|
||||
gating starts once that cohort accumulates history). Legacy v1 configs still
|
||||
run and are normalized for reporting, but their records skip rolling-baseline
|
||||
comparison entirely (`baseline_status: "skipped_missing_identity"`, never
|
||||
baseline eligible); only static thresholds gate them. Metric-specific
|
||||
threshold policies and promoted baselines remain separate follow-ups.
|
||||
|
||||
### Raw record (`results/perf_*.json`)
|
||||
|
||||
@@ -379,8 +367,6 @@ result, used as the rolling-baseline source of truth.
|
||||
"build_id": "<buildkite-build-id>",
|
||||
"job_id": "<buildkite-job-id>",
|
||||
"quality_metadata": { "quality_status": "canonical" },
|
||||
"baseline_status": "compared",
|
||||
"comparator_status": "PASS",
|
||||
"success": true
|
||||
}
|
||||
```
|
||||
@@ -501,10 +487,10 @@ When the rolling-baseline phase runs, it emits:
|
||||
2. The pytest test auto-discovers all configs — no test code needed. CI
|
||||
picks it up on the next `/test performance` run.
|
||||
|
||||
3. The first run with no HF history reports `CALIBRATION_NEEDED`: it passes,
|
||||
but it does not seed the shared baseline. Seed the cohort explicitly with
|
||||
the `reseed-performance-baseline` skill; subsequent scheduled-main runs
|
||||
then compare against it and keep the rolling baseline advancing.
|
||||
3. The first persisted main-branch run with no HF history initializes the
|
||||
baseline (passes automatically). Subsequent runs compare against it. Local
|
||||
and pull-request runs with no HF history also pass, but they do not seed the
|
||||
shared baseline.
|
||||
|
||||
4. If the benchmark targets a GPU not currently in `thresholds`, either add
|
||||
that GPU as a key or rely on the `default` block. Note that `default` is
|
||||
@@ -524,13 +510,8 @@ When the rolling-baseline phase runs, it emits:
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**`CALIBRATION_NEEDED: no comparable baseline`** — first run for this cohort.
|
||||
The run passes but does not seed a baseline; seed it explicitly with the
|
||||
`reseed-performance-baseline` skill.
|
||||
|
||||
**`RECIPE_MISMATCH`** — the benchmark recipe changed without a new
|
||||
`variant_id`. Either bump `variant_id` to open a new cohort, or reseed the
|
||||
baseline if the existing variant should adopt the new recipe.
|
||||
**"No baseline for ... Initializing"** — first run for this comparison cohort.
|
||||
Run will pass and (if persisting) seed the first record.
|
||||
|
||||
**Persistent failure right after a torch / kernel / image upgrade** —
|
||||
genuine regression *or* baseline drift. Compare the failing normalized record
|
||||
|
||||
@@ -0,0 +1,51 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""Minimal user-runnable example for the daVinci-MagiHuman base AV pipeline.
|
||||
|
||||
Produces an mp4 with both video (Wan 2.2 TI2V-5B VAE) and audio (Stable
|
||||
Audio Open 1.0 VAE, first-class FastVideo port in
|
||||
`fastvideo/models/vaes/oobleck.py`) muxed together via PyAV.
|
||||
|
||||
Prerequisites (one-off):
|
||||
|
||||
# Accept terms of use on the gated HF repos with your HF_TOKEN:
|
||||
# - https://huggingface.co/google/t5gemma-9b-9b-ul2
|
||||
# - https://huggingface.co/stabilityai/stable-audio-open-1.0
|
||||
# All four cross-variant shared components (Wan 2.2 VAE, T5-Gemma
|
||||
# encoder + tokenizer, Stable Audio VAE) are lazy-loaded from their
|
||||
# canonical upstream HF repos on first build, so a single ~25 GB
|
||||
# cache is shared across every MagiHuman variant.
|
||||
|
||||
The umbrella HF repo `FastVideo/MagiHuman-Diffusers` holds all four
|
||||
variants (base / distill / sr_540p / sr_1080p) under sibling subfolders
|
||||
and FastVideo will download just the requested subfolder. Local
|
||||
conversion via `scripts/checkpoint_conversion/convert_magi_human_to_diffusers.py`
|
||||
is also supported.
|
||||
"""
|
||||
from fastvideo import VideoGenerator
|
||||
|
||||
|
||||
PROMPT = (
|
||||
"A warm afternoon scene: a person sits on a park bench reading a book, "
|
||||
"surrounded by softly swaying trees."
|
||||
)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
generator = VideoGenerator.from_pretrained(
|
||||
"FastVideo/MagiHuman-Diffusers/base",
|
||||
num_gpus=1,
|
||||
)
|
||||
output_path = "outputs_video/magi_human_basic/output_magi_human.mp4"
|
||||
generator.generate_video(
|
||||
prompt=PROMPT,
|
||||
output_path=output_path,
|
||||
save_video=True,
|
||||
# Defaults pulled from the registered preset (magi_human_base):
|
||||
# height=256, width=448, fps=25, num_inference_steps=32, seed=42.
|
||||
# Override here only if you have a specific QA scenario.
|
||||
)
|
||||
generator.shutdown()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,53 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""Minimal user-runnable example for the daVinci-MagiHuman DMD-2 distilled
|
||||
text-to-AV pipeline.
|
||||
|
||||
Same arch as the base model (`basic_magi_human.py`) but with DMD-2 distilled
|
||||
weights: 8 denoising steps, no classifier-free guidance. ~4x faster than
|
||||
base at the same 256x480 resolution. Mirrors upstream
|
||||
`daVinci-MagiHuman/example/distill/run_T2V.sh`.
|
||||
|
||||
Prerequisites (one-off):
|
||||
|
||||
# 1) Accept terms on the gated HF repos with your HF_TOKEN:
|
||||
# - https://huggingface.co/google/t5gemma-9b-9b-ul2
|
||||
# - https://huggingface.co/stabilityai/stable-audio-open-1.0
|
||||
# Cross-variant shared components (Wan 2.2 VAE + T5-Gemma + Stable
|
||||
# Audio VAE) are lazy-loaded from their canonical upstream HF repos
|
||||
# and shared with the base variant cache.
|
||||
# 2) Convert the distill subfolder of GAIR/daVinci-MagiHuman:
|
||||
python scripts/checkpoint_conversion/convert_magi_human_to_diffusers.py \\
|
||||
--source GAIR/daVinci-MagiHuman \\
|
||||
--subfolder distill \\
|
||||
--output converted_weights/magi_human_distill \\
|
||||
--cast-bf16
|
||||
# `--cast-bf16` is recommended (61 GB fp32 -> 30 GB bf16); the FV pipeline
|
||||
# loads bf16 anyway, and the conversion keeps norms / RoPE bands fp32.
|
||||
"""
|
||||
from fastvideo import VideoGenerator
|
||||
|
||||
|
||||
PROMPT = (
|
||||
"A warm afternoon scene: a person sits on a park bench reading a book, "
|
||||
"surrounded by softly swaying trees."
|
||||
)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
generator = VideoGenerator.from_pretrained(
|
||||
"FastVideo/MagiHuman-Diffusers/distill",
|
||||
num_gpus=1,
|
||||
)
|
||||
output_path = "outputs_video/magi_human_basic/output_magi_human_distill.mp4"
|
||||
generator.generate_video(
|
||||
prompt=PROMPT,
|
||||
output_path=output_path,
|
||||
save_video=True,
|
||||
# Defaults pulled from the registered preset (magi_human_distill):
|
||||
# height=256, width=480, fps=25, num_inference_steps=8, cfg=1, seed=42.
|
||||
)
|
||||
generator.shutdown()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,34 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""Minimal daVinci-MagiHuman DMD-2 distilled text+image-to-AV example."""
|
||||
from fastvideo import VideoGenerator
|
||||
from fastvideo.pipelines.basic.magi_human.pipeline_configs import (
|
||||
MagiHumanDistillI2VConfig,
|
||||
)
|
||||
|
||||
|
||||
PROMPT = (
|
||||
"A cheerful saxophonist performs a short line with expressive facial "
|
||||
"motion, natural head movement, and synchronized audio in a small jazz club."
|
||||
)
|
||||
IMAGE_PATH = "assets/images/saxophonist.jpg"
|
||||
|
||||
|
||||
def main() -> None:
|
||||
generator = VideoGenerator.from_pretrained(
|
||||
"FastVideo/MagiHuman-Diffusers/distill",
|
||||
num_gpus=1,
|
||||
workload_type="i2v",
|
||||
override_pipeline_cls_name="MagiHumanI2VPipeline",
|
||||
pipeline_config=MagiHumanDistillI2VConfig(),
|
||||
)
|
||||
generator.generate_video(
|
||||
prompt=PROMPT,
|
||||
image_path=IMAGE_PATH,
|
||||
output_path="outputs_video/magi_human_distill_ti2v/output_magi_human_distill_ti2v.mp4",
|
||||
save_video=True,
|
||||
)
|
||||
generator.shutdown()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,45 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""Run daVinci-MagiHuman SR-1080p text-to-AV in FastVideo.
|
||||
|
||||
Build the converted repo on large local storage, then symlink it into the
|
||||
workspace:
|
||||
|
||||
python scripts/checkpoint_conversion/convert_magi_human_to_diffusers.py \
|
||||
--source GAIR/daVinci-MagiHuman \
|
||||
--subfolder base \
|
||||
--sr-source GAIR/daVinci-MagiHuman \
|
||||
--sr-subfolder 1080p_sr \
|
||||
--output /raid/william5lin_converted_weights/magi_human_sr_1080p \
|
||||
--cast-bf16
|
||||
ln -s /raid/william5lin_converted_weights/magi_human_sr_1080p \
|
||||
converted_weights/magi_human_sr_1080p
|
||||
"""
|
||||
from fastvideo import VideoGenerator
|
||||
from fastvideo.pipelines.basic.magi_human.pipeline_configs import (
|
||||
MagiHumanSR1080pConfig,
|
||||
)
|
||||
|
||||
|
||||
PROMPT = (
|
||||
"A warm afternoon scene: a person sits on a park bench reading a book, "
|
||||
"surrounded by softly swaying trees."
|
||||
)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
generator = VideoGenerator.from_pretrained(
|
||||
"FastVideo/MagiHuman-Diffusers/sr_1080p",
|
||||
num_gpus=1,
|
||||
override_pipeline_cls_name="MagiHumanSR1080pPipeline",
|
||||
pipeline_config=MagiHumanSR1080pConfig(),
|
||||
)
|
||||
generator.generate_video(
|
||||
prompt=PROMPT,
|
||||
output_path="outputs_video/magi_human_sr1080p/output_magi_human_sr1080p.mp4",
|
||||
save_video=True,
|
||||
)
|
||||
generator.shutdown()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,34 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""Run daVinci-MagiHuman SR-1080p text+image-to-AV in FastVideo."""
|
||||
from fastvideo import VideoGenerator
|
||||
from fastvideo.pipelines.basic.magi_human.pipeline_configs import (
|
||||
MagiHumanSR1080pI2VConfig,
|
||||
)
|
||||
|
||||
|
||||
PROMPT = (
|
||||
"A cheerful saxophonist performs a short line with expressive facial "
|
||||
"motion, natural head movement, and synchronized audio in a small jazz club."
|
||||
)
|
||||
IMAGE_PATH = "assets/images/saxophonist.jpg"
|
||||
|
||||
|
||||
def main() -> None:
|
||||
generator = VideoGenerator.from_pretrained(
|
||||
"FastVideo/MagiHuman-Diffusers/sr_1080p",
|
||||
num_gpus=1,
|
||||
workload_type="i2v",
|
||||
override_pipeline_cls_name="MagiHumanSR1080pI2VPipeline",
|
||||
pipeline_config=MagiHumanSR1080pI2VConfig(),
|
||||
)
|
||||
generator.generate_video(
|
||||
prompt=PROMPT,
|
||||
image_path=IMAGE_PATH,
|
||||
output_path="outputs_video/magi_human_sr1080p_ti2v/output_magi_human_sr1080p_ti2v.mp4",
|
||||
save_video=True,
|
||||
)
|
||||
generator.shutdown()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,37 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""Run daVinci-MagiHuman SR-540p text-to-AV in FastVideo.
|
||||
|
||||
The converted repo must contain both ``transformer/`` (base DiT) and
|
||||
``sr_transformer/`` (540p SR DiT). Build it with:
|
||||
|
||||
python scripts/checkpoint_conversion/convert_magi_human_to_diffusers.py \
|
||||
--source GAIR/daVinci-MagiHuman \
|
||||
--subfolder base \
|
||||
--sr-source GAIR/daVinci-MagiHuman \
|
||||
--sr-subfolder 540p_sr \
|
||||
--output converted_weights/magi_human_sr_540p
|
||||
"""
|
||||
from fastvideo import VideoGenerator
|
||||
|
||||
|
||||
PROMPT = (
|
||||
"A warm afternoon scene: a person sits on a park bench reading a book, "
|
||||
"surrounded by softly swaying trees."
|
||||
)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
generator = VideoGenerator.from_pretrained(
|
||||
"FastVideo/MagiHuman-Diffusers/sr_540p",
|
||||
num_gpus=1,
|
||||
)
|
||||
generator.generate_video(
|
||||
prompt=PROMPT,
|
||||
output_path="outputs_video/magi_human_sr540p/output_magi_human_sr540p.mp4",
|
||||
save_video=True,
|
||||
)
|
||||
generator.shutdown()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,34 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""Run daVinci-MagiHuman SR-540p text+image-to-AV in FastVideo."""
|
||||
from fastvideo import VideoGenerator
|
||||
from fastvideo.pipelines.basic.magi_human.pipeline_configs import (
|
||||
MagiHumanSR540pI2VConfig,
|
||||
)
|
||||
|
||||
|
||||
PROMPT = (
|
||||
"A cheerful saxophonist performs a short line with expressive facial "
|
||||
"motion, natural head movement, and synchronized audio in a small jazz club."
|
||||
)
|
||||
IMAGE_PATH = "assets/images/saxophonist.jpg"
|
||||
|
||||
|
||||
def main() -> None:
|
||||
generator = VideoGenerator.from_pretrained(
|
||||
"FastVideo/MagiHuman-Diffusers/sr_540p",
|
||||
num_gpus=1,
|
||||
workload_type="i2v",
|
||||
override_pipeline_cls_name="MagiHumanSRI2VPipeline",
|
||||
pipeline_config=MagiHumanSR540pI2VConfig(),
|
||||
)
|
||||
generator.generate_video(
|
||||
prompt=PROMPT,
|
||||
image_path=IMAGE_PATH,
|
||||
output_path="outputs_video/magi_human_sr540p_ti2v/output_magi_human_sr540p_ti2v.mp4",
|
||||
save_video=True,
|
||||
)
|
||||
generator.shutdown()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,34 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""Minimal daVinci-MagiHuman base text+image-to-AV example."""
|
||||
from fastvideo import VideoGenerator
|
||||
from fastvideo.pipelines.basic.magi_human.pipeline_configs import (
|
||||
MagiHumanBaseI2VConfig,
|
||||
)
|
||||
|
||||
|
||||
PROMPT = (
|
||||
"A cheerful saxophonist performs a short line with expressive facial "
|
||||
"motion, natural head movement, and synchronized audio in a small jazz club."
|
||||
)
|
||||
IMAGE_PATH = "assets/images/saxophonist.jpg"
|
||||
|
||||
|
||||
def main() -> None:
|
||||
generator = VideoGenerator.from_pretrained(
|
||||
"FastVideo/MagiHuman-Diffusers/base",
|
||||
num_gpus=1,
|
||||
workload_type="i2v",
|
||||
override_pipeline_cls_name="MagiHumanI2VPipeline",
|
||||
pipeline_config=MagiHumanBaseI2VConfig(),
|
||||
)
|
||||
generator.generate_video(
|
||||
prompt=PROMPT,
|
||||
image_path=IMAGE_PATH,
|
||||
output_path="outputs_video/magi_human_ti2v/output_magi_human_ti2v.mp4",
|
||||
save_video=True,
|
||||
)
|
||||
generator.shutdown()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -15,14 +15,8 @@ class SDPABackend(AttentionBackend):
|
||||
accept_output_buffer: bool = True
|
||||
|
||||
@staticmethod
|
||||
def get_supported_head_sizes() -> list[int] | None:
|
||||
# torch.nn.functional.scaled_dot_product_attention is head-size
|
||||
# agnostic: the math backend handles any size and the fused kernels
|
||||
# fall back internally when they cannot. The previous list
|
||||
# ([32, 64, ..., 256]) was copied from FlashAttentionBackend in the
|
||||
# v1 refactor (#270) and wrongly excluded sizes such as 80 (e.g.
|
||||
# CLIP vision encoders). None means "no restriction".
|
||||
return None
|
||||
def get_supported_head_sizes() -> list[int]:
|
||||
return [32, 64, 96, 128, 160, 192, 224, 256]
|
||||
|
||||
@staticmethod
|
||||
def get_name() -> str:
|
||||
|
||||
@@ -72,13 +72,8 @@ def _rms_norm_dispatch(
|
||||
"""Use QuACK RMSNorm only for LTX-2 refine stage, else Torch RMSNorm."""
|
||||
if _is_ltx2_refine_stage():
|
||||
return _quack_rmsnorm(x, weight=weight, eps=eps)
|
||||
# Official LTX-2 runs the DiT natively in bf16 with no autocast, so its
|
||||
# rms_norm outputs stay bf16 on every torch version. torch 2.12 added
|
||||
# rms_norm to autocast's float32 cast policy, which under our
|
||||
# autocast(bfloat16) forward returns float32 instead. Restore the input
|
||||
# dtype to match the official numerics (and torch <= 2.11 behavior).
|
||||
return torch.nn.functional.rms_norm(
|
||||
x, (x.shape[-1], ), weight=weight, eps=eps).to(x.dtype)
|
||||
x, (x.shape[-1], ), weight=weight, eps=eps)
|
||||
|
||||
|
||||
class StageAwareRMSNorm(nn.RMSNorm):
|
||||
|
||||
@@ -183,8 +183,6 @@ focused PRs:
|
||||
| 3/8 | [#1297](https://github.com/hao-ai-lab/FastVideo/pull/1297) | `will/magi-03-dit` | DiT + parity tests |
|
||||
| 4/8 | [#1298](https://github.com/hao-ai-lab/FastVideo/pull/1298) | `will/magi-04a-stages` | Pipeline stages + sr_transformer alias |
|
||||
| 5/8 | [#1299](https://github.com/hao-ai-lab/FastVideo/pull/1299) | `will/magi-04b-orchestrator` | Pipeline orchestrator + parity battery |
|
||||
| 6/8 | this PR | `will/magi-04c-provenance` | This AGENTS.md, JOURNAL.md, lessons, parent AGENTS.md hook |
|
||||
| 7/8 | (next) | `will/magi-05-conversion` | Checkpoint conversion + 3rd lesson |
|
||||
| 8/8 | (last) | `will/magi-06-activate` | Registry + examples + SSIM + codebase-map |
|
||||
|
||||
Provenance section will be finalized in 8/8 with the actual PR numbers for 7/8 and 8/8.
|
||||
| 6/8 | [#1300](https://github.com/hao-ai-lab/FastVideo/pull/1300) | `will/magi-04c-provenance` | This AGENTS.md, JOURNAL.md, lessons, parent AGENTS.md hook |
|
||||
| 7/8 | [#1301](https://github.com/hao-ai-lab/FastVideo/pull/1301) | `will/magi-05-conversion` | Checkpoint conversion + 3rd lesson |
|
||||
| 8/8 | this PR | `will/magi-06-activate` | Registry + examples + SSIM + codebase-map |
|
||||
|
||||
@@ -38,6 +38,16 @@ from fastvideo.configs.pipelines.flux_2 import (
|
||||
)
|
||||
from fastvideo.configs.pipelines.matrixgame2 import MatrixGame2I2V480PConfig
|
||||
from fastvideo.configs.pipelines.matrixgame3 import MatrixGame3I2V720PConfig
|
||||
from fastvideo.pipelines.basic.magi_human.pipeline_configs import (
|
||||
MagiHumanBaseConfig,
|
||||
MagiHumanBaseI2VConfig,
|
||||
MagiHumanDistillConfig,
|
||||
MagiHumanDistillI2VConfig,
|
||||
MagiHumanSR1080pConfig,
|
||||
MagiHumanSR1080pI2VConfig,
|
||||
MagiHumanSR540pConfig,
|
||||
MagiHumanSR540pI2VConfig,
|
||||
)
|
||||
from fastvideo.configs.pipelines.turbodiffusion import (
|
||||
TurboDiffusionI2V_A14B_Config,
|
||||
TurboDiffusionT2V_14B_Config,
|
||||
@@ -379,6 +389,146 @@ def _register_configs() -> None:
|
||||
default_preset="flux2_dev",
|
||||
)
|
||||
|
||||
# daVinci-MagiHuman SR-1080p (two-stage base + local-window SR text-to-AV).
|
||||
register_configs(
|
||||
sampling_param_cls=None,
|
||||
pipeline_config_cls=MagiHumanSR1080pConfig,
|
||||
workload_types=(WorkloadType.T2V, ),
|
||||
hf_model_paths=[
|
||||
"FastVideo/MagiHuman-SR-1080p-Diffusers",
|
||||
"FastVideo/MagiHuman-Diffusers/sr_1080p",
|
||||
],
|
||||
model_detectors=[
|
||||
lambda path:
|
||||
(("magihuman" in path.lower() or "magi_human" in path.lower() or "magi-human" in path.lower()) and
|
||||
("sr_1080p" in path.lower() or "sr-1080p" in path.lower() or "1080p_sr" in path.lower() or "sr1080p" in
|
||||
path.lower()) and "ti2v" not in path.lower()),
|
||||
],
|
||||
model_family="magi_human",
|
||||
default_preset="magi_human_sr_1080p",
|
||||
)
|
||||
register_configs(
|
||||
sampling_param_cls=None,
|
||||
pipeline_config_cls=MagiHumanSR1080pI2VConfig,
|
||||
workload_types=(WorkloadType.I2V, ),
|
||||
hf_model_paths=[
|
||||
"FastVideo/MagiHuman-SR-1080p-TI2V-Diffusers",
|
||||
],
|
||||
model_detectors=[
|
||||
lambda path:
|
||||
(("magihuman" in path.lower() or "magi_human" in path.lower() or "magi-human" in path.lower()) and
|
||||
("sr_1080p" in path.lower() or "sr-1080p" in path.lower() or "1080p_sr" in path.lower() or "sr1080p" in
|
||||
path.lower()) and "ti2v" in path.lower()),
|
||||
],
|
||||
model_family="magi_human",
|
||||
default_preset="magi_human_sr_1080p_ti2v",
|
||||
)
|
||||
|
||||
# daVinci-MagiHuman SR-540p (two-stage base + SR text-to-AV).
|
||||
register_configs(
|
||||
sampling_param_cls=None,
|
||||
pipeline_config_cls=MagiHumanSR540pConfig,
|
||||
workload_types=(WorkloadType.T2V, ),
|
||||
hf_model_paths=[
|
||||
"FastVideo/MagiHuman-SR-540p-Diffusers",
|
||||
"FastVideo/MagiHuman-Diffusers/sr_540p",
|
||||
],
|
||||
model_detectors=[
|
||||
lambda path:
|
||||
(("magihuman" in path.lower() or "magi_human" in path.lower() or "magi-human" in path.lower()) and
|
||||
("sr_540p" in path.lower() or "sr-540p" in path.lower() or "540p_sr" in path.lower() or "srpipeline" in
|
||||
path.lower()) and "1080" not in path.lower() and "ti2v" not in path.lower()),
|
||||
],
|
||||
model_family="magi_human",
|
||||
default_preset="magi_human_sr_540p",
|
||||
)
|
||||
register_configs(
|
||||
sampling_param_cls=None,
|
||||
pipeline_config_cls=MagiHumanSR540pI2VConfig,
|
||||
workload_types=(WorkloadType.I2V, ),
|
||||
hf_model_paths=[
|
||||
"FastVideo/MagiHuman-SR-540p-TI2V-Diffusers",
|
||||
],
|
||||
model_detectors=[
|
||||
lambda path:
|
||||
(("magihuman" in path.lower() or "magi_human" in path.lower() or "magi-human" in path.lower()) and
|
||||
("sr_540p" in path.lower() or "sr-540p" in path.lower() or "540p_sr" in path.lower() or "srpipeline" in
|
||||
path.lower()) and "1080" not in path.lower() and "ti2v" in path.lower()),
|
||||
],
|
||||
model_family="magi_human",
|
||||
default_preset="magi_human_sr_540p_ti2v",
|
||||
)
|
||||
|
||||
# daVinci-MagiHuman (base text-to-AV).
|
||||
# NOTE: WorkloadType has no T2AV variant yet; using T2V as the
|
||||
# placeholder until the enum is extended (same as Stable Audio).
|
||||
register_configs(
|
||||
sampling_param_cls=None,
|
||||
pipeline_config_cls=MagiHumanBaseConfig,
|
||||
workload_types=(WorkloadType.T2V, ),
|
||||
hf_model_paths=[
|
||||
"GAIR/daVinci-MagiHuman",
|
||||
"FastVideo/MagiHuman-Base-Diffusers",
|
||||
"FastVideo/MagiHuman-Diffusers/base",
|
||||
],
|
||||
model_detectors=[
|
||||
lambda path:
|
||||
(("magihuman" in path.lower() or "magi_human" in path.lower() or "magi-human" in path.lower()
|
||||
) and "distill" not in path.lower() and "ti2v" not in path.lower() and "sr_540p" not in path.lower() and
|
||||
"sr-540p" not in path.lower() and "540p_sr" not in path.lower() and "sr_1080p" not in path.lower() and
|
||||
"sr-1080p" not in path.lower() and "1080p_sr" not in path.lower() and "srpipeline" not in path.lower()),
|
||||
],
|
||||
model_family="magi_human",
|
||||
default_preset="magi_human_base",
|
||||
)
|
||||
register_configs(
|
||||
sampling_param_cls=None,
|
||||
pipeline_config_cls=MagiHumanBaseI2VConfig,
|
||||
workload_types=(WorkloadType.I2V, ),
|
||||
hf_model_paths=[
|
||||
"FastVideo/MagiHuman-Base-TI2V-Diffusers",
|
||||
],
|
||||
model_detectors=[
|
||||
lambda path:
|
||||
(("magihuman" in path.lower() or "magi_human" in path.lower() or "magi-human" in path.lower()
|
||||
) and "ti2v" in path.lower() and "distill" not in path.lower() and "sr_540p" not in path.lower() and
|
||||
"sr-540p" not in path.lower() and "540p_sr" not in path.lower() and "sr_1080p" not in path.lower() and
|
||||
"sr-1080p" not in path.lower() and "1080p_sr" not in path.lower() and "srpipeline" not in path.lower()),
|
||||
],
|
||||
model_family="magi_human",
|
||||
default_preset="magi_human_base_ti2v",
|
||||
)
|
||||
# daVinci-MagiHuman (DMD-2 distilled text-to-AV)
|
||||
register_configs(
|
||||
sampling_param_cls=None,
|
||||
pipeline_config_cls=MagiHumanDistillConfig,
|
||||
workload_types=(WorkloadType.T2V, ),
|
||||
hf_model_paths=[
|
||||
"FastVideo/MagiHuman-Distilled-Diffusers",
|
||||
"FastVideo/MagiHuman-Diffusers/distill",
|
||||
],
|
||||
model_detectors=[
|
||||
lambda path: (("magihuman" in path.lower() or "magi_human" in path.lower() or "magi-human" in path.lower())
|
||||
and "distill" in path.lower() and "ti2v" not in path.lower()),
|
||||
],
|
||||
model_family="magi_human",
|
||||
default_preset="magi_human_distill",
|
||||
)
|
||||
register_configs(
|
||||
sampling_param_cls=None,
|
||||
pipeline_config_cls=MagiHumanDistillI2VConfig,
|
||||
workload_types=(WorkloadType.I2V, ),
|
||||
hf_model_paths=[
|
||||
"FastVideo/MagiHuman-Distilled-TI2V-Diffusers",
|
||||
],
|
||||
model_detectors=[
|
||||
lambda path: (("magihuman" in path.lower() or "magi_human" in path.lower() or "magi-human" in path.lower())
|
||||
and "ti2v" in path.lower() and "distill" in path.lower()),
|
||||
],
|
||||
model_family="magi_human",
|
||||
default_preset="magi_human_distill_ti2v",
|
||||
)
|
||||
|
||||
# Hunyuan 1.5 (specific)
|
||||
register_configs(
|
||||
sampling_param_cls=None,
|
||||
@@ -1214,6 +1364,8 @@ def _register_presets() -> None:
|
||||
ALL_PRESETS as LONGCAT_PRESETS, )
|
||||
from fastvideo.pipelines.basic.ltx2.presets import (
|
||||
ALL_PRESETS as LTX2_PRESETS, )
|
||||
from fastvideo.pipelines.basic.magi_human.presets import (
|
||||
ALL_PRESETS as MAGI_HUMAN_PRESETS, )
|
||||
from fastvideo.pipelines.basic.matrixgame2.presets import (
|
||||
ALL_PRESETS as MATRIXGAME2_PRESETS, )
|
||||
from fastvideo.pipelines.basic.matrixgame3.presets import (
|
||||
@@ -1242,6 +1394,7 @@ def _register_presets() -> None:
|
||||
LINGBOTWORLD_PRESETS,
|
||||
LONGCAT_PRESETS,
|
||||
LTX2_PRESETS,
|
||||
MAGI_HUMAN_PRESETS,
|
||||
MATRIXGAME2_PRESETS,
|
||||
MATRIXGAME3_PRESETS,
|
||||
SD35_PRESETS,
|
||||
|
||||
@@ -1,49 +0,0 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""CPU tests: SDPA must not restrict attention head sizes.
|
||||
|
||||
``torch.nn.functional.scaled_dot_product_attention`` is head-size
|
||||
agnostic, but ``SDPABackend.get_supported_head_sizes()`` used to declare
|
||||
a list copied from FlashAttentionBackend that omitted e.g. 80 (the head
|
||||
size of CLIP-L vision encoders such as matrixgame2's). That made
|
||||
capability validation flag SDPA as incompatible with head-80 layers even
|
||||
though it handles them fine.
|
||||
|
||||
CPU-only; no GPU or flash-attn dependency.
|
||||
"""
|
||||
|
||||
import torch
|
||||
|
||||
from fastvideo.attention.backends.sdpa import SDPABackend, SDPAImpl, SDPAMetadata
|
||||
from fastvideo.platforms.cuda import CudaPlatformBase
|
||||
from fastvideo.platforms.interface import AttentionBackendEnum
|
||||
|
||||
|
||||
def test_sdpa_head_sizes_unrestricted() -> None:
|
||||
"""SDPA declares no head-size restriction (None), so 80 is allowed."""
|
||||
assert SDPABackend.get_supported_head_sizes() is None
|
||||
# Once the capability-validation API (#1494) lands, head-80 must pass.
|
||||
validate = getattr(SDPABackend, "validate_compatibility", None)
|
||||
if validate is not None:
|
||||
assert validate(head_size=80, dtype=torch.bfloat16) is None
|
||||
|
||||
|
||||
def test_explicit_torch_sdpa_pin_head_80_resolves() -> None:
|
||||
"""An explicit TORCH_SDPA pin on a head-80 layer resolves to SDPABackend."""
|
||||
cls_str = CudaPlatformBase.get_attn_backend_cls(AttentionBackendEnum.TORCH_SDPA, head_size=80, dtype=torch.bfloat16)
|
||||
assert cls_str == "fastvideo.attention.backends.sdpa.SDPABackend"
|
||||
|
||||
|
||||
def test_sdpa_impl_computes_head_80_attention() -> None:
|
||||
"""SDPAImpl produces correct output for head_size=80 (not in the old list)."""
|
||||
bs, seq, heads, head_dim = 2, 8, 2, 80
|
||||
softmax_scale = head_dim**-0.5
|
||||
impl = SDPAImpl(num_heads=heads, head_size=head_dim, causal=False, softmax_scale=softmax_scale)
|
||||
torch.manual_seed(0)
|
||||
q, k, v = (torch.randn(bs, seq, heads, head_dim) for _ in range(3))
|
||||
|
||||
out = impl.forward(q, k, v, SDPAMetadata(current_timestep=0))
|
||||
|
||||
assert out.shape == (bs, seq, heads, head_dim)
|
||||
qt, kt, vt = (t.transpose(1, 2) for t in (q, k, v))
|
||||
ref = torch.softmax(qt @ kt.transpose(-2, -1) * softmax_scale, dim=-1) @ vt
|
||||
torch.testing.assert_close(out, ref.transpose(1, 2), rtol=1e-4, atol=1e-4)
|
||||
@@ -89,14 +89,6 @@ COMPARISON_IDENTITY_KEYS = (
|
||||
"hardware_profile_id",
|
||||
"software_profile_id",
|
||||
)
|
||||
# Comparator verdicts (issue #1532), stored as record["comparator_status"].
|
||||
STATUS_PASS = "PASS"
|
||||
STATUS_REGRESSION = "REGRESSION"
|
||||
STATUS_CALIBRATION_NEEDED = "CALIBRATION_NEEDED"
|
||||
STATUS_RECIPE_MISMATCH = "RECIPE_MISMATCH"
|
||||
STATUS_INFRA_ERROR = "INFRA_ERROR"
|
||||
# Reserved for the promoted-baseline workflow; never emitted by this comparator.
|
||||
STATUS_QUALITY_BLOCKED = "QUALITY_BLOCKED"
|
||||
|
||||
|
||||
def _should_persist_tracking() -> bool:
|
||||
@@ -351,90 +343,6 @@ def _check_regressions(
|
||||
return failures
|
||||
|
||||
|
||||
def _recipe_mismatch_fingerprints(
|
||||
record: dict[str, Any],
|
||||
identity_filters: dict[str, str],
|
||||
) -> list[str]:
|
||||
"""Return baseline recipe fingerprints for the same variant cohort.
|
||||
|
||||
Non-empty means the same (workload, variant, version, hardware, software)
|
||||
cohort has baseline history under a DIFFERENT recipe fingerprint: the
|
||||
recipe changed without being represented as a new variant.
|
||||
"""
|
||||
if not identity_filters:
|
||||
return []
|
||||
variant_filters = {key: value for key, value in identity_filters.items() if key != "recipe_fingerprint"}
|
||||
same_variant = load_records_for_model(
|
||||
TRACKING_ROOT,
|
||||
record["model_id"],
|
||||
record["gpu_type"],
|
||||
**variant_filters,
|
||||
successful_only=True,
|
||||
baseline_eligible_only=True,
|
||||
)
|
||||
fingerprints = {str(r.get("recipe_fingerprint")) for r in same_variant}
|
||||
return sorted(fingerprints - {identity_filters["recipe_fingerprint"]})
|
||||
|
||||
|
||||
def _compare_record(
|
||||
record: dict[str, Any],
|
||||
metric_policies: tuple[MetricPolicy, ...],
|
||||
) -> tuple[list[str], list[dict[str, Any]]]:
|
||||
"""Compare one normalized record against its exact-identity cohort.
|
||||
|
||||
Sets ``record["comparator_status"]`` (and ``record["baseline_status"]``
|
||||
for cohort bookkeeping) and returns ``(failures, baseline_records)``.
|
||||
"""
|
||||
model = record.get("model_id", "unknown")
|
||||
try:
|
||||
identity_filters = _comparison_identity_filters(record)
|
||||
baseline_records = load_records_for_model(
|
||||
TRACKING_ROOT,
|
||||
record["model_id"],
|
||||
record["gpu_type"],
|
||||
**identity_filters,
|
||||
last_n=5,
|
||||
successful_only=True,
|
||||
baseline_eligible_only=True,
|
||||
)
|
||||
if baseline_records:
|
||||
record["baseline_status"] = "compared"
|
||||
failures = _check_regressions(record, baseline_records, metric_policies)
|
||||
record["comparator_status"] = STATUS_REGRESSION if failures else STATUS_PASS
|
||||
return failures, baseline_records
|
||||
|
||||
mismatched = _recipe_mismatch_fingerprints(record, identity_filters)
|
||||
except Exception as exc:
|
||||
record["comparator_status"] = STATUS_INFRA_ERROR
|
||||
return [f"{model} baseline comparison hit an infra error: {exc}"], []
|
||||
|
||||
if mismatched:
|
||||
record["comparator_status"] = STATUS_RECIPE_MISMATCH
|
||||
return [
|
||||
f"{model} recipe fingerprint {record.get('recipe_fingerprint')} does not "
|
||||
f"match baseline fingerprint(s) {', '.join(mismatched)} for variant "
|
||||
f"{record.get('variant_id')}. Represent recipe changes as a new "
|
||||
"variant_id, or reseed the baseline."
|
||||
], []
|
||||
|
||||
# No comparable baseline anywhere: a brand-new cohort. Make that loud and
|
||||
# machine-readable instead of an indistinguishable pass, so a cohort shift
|
||||
# (intended or accidental, e.g. an identity-field change) never silently
|
||||
# blinds the comparison — and never silently seeds a passing baseline.
|
||||
record["baseline_status"] = "initialized_new_cohort"
|
||||
record["comparator_status"] = STATUS_CALIBRATION_NEEDED
|
||||
print("=" * 72)
|
||||
print(f"CALIBRATION_NEEDED: no comparable baseline for {model} on "
|
||||
f"{record.get('gpu_type', 'unknown')}"
|
||||
f"{_format_identity_filters(identity_filters)}")
|
||||
print("Regression gating is INACTIVE for this record and it will NOT "
|
||||
"seed a baseline. Seed the cohort explicitly via the reseed "
|
||||
"workflow. If this cohort shift is unexpected, check the identity "
|
||||
"fields above.")
|
||||
print("=" * 72)
|
||||
return [], []
|
||||
|
||||
|
||||
def _compact_value(value: float | None, precision: int = 3) -> str:
|
||||
if value is None:
|
||||
return "n/a"
|
||||
@@ -486,7 +394,6 @@ def _build_summary_row(
|
||||
return {
|
||||
"model_id": record["model_id"],
|
||||
"gpu_type": record["gpu_type"],
|
||||
"comparator_status": record.get("comparator_status", STATUS_PASS),
|
||||
"baseline_n": len(baseline_records),
|
||||
"metrics": metric_values,
|
||||
"worst_regression_pct": worst_regression_pct,
|
||||
@@ -528,10 +435,7 @@ def _build_markdown_summary(
|
||||
else "none"
|
||||
)
|
||||
failing_metrics = ", ".join(row["failing_metrics"]) if row["failing_metrics"] else "none"
|
||||
status = row["comparator_status"]
|
||||
if row["failed"] and status == STATUS_PASS:
|
||||
# Baseline comparison passed but the fixed-threshold phase failed.
|
||||
status = "FAIL"
|
||||
status = "FAIL" if row["failed"] else "PASS"
|
||||
|
||||
lines.append(f"| {row['model_id']} | {row['gpu_type']} | "
|
||||
f"{row['baseline_n']} | "
|
||||
@@ -597,26 +501,46 @@ def main() -> int:
|
||||
if identity_filters is None:
|
||||
# Records without the full v2 identity block skip rolling-baseline
|
||||
# comparison entirely: only the static thresholds gate them and
|
||||
# they never become baseline eligible. Nothing was compared and
|
||||
# nothing failed, so the comparator verdict is PASS;
|
||||
# baseline_status keeps the skip machine-readable.
|
||||
# they never become baseline eligible.
|
||||
record["baseline_status"] = "skipped_missing_identity"
|
||||
record["comparator_status"] = STATUS_PASS
|
||||
failures: list[str] = []
|
||||
else:
|
||||
failures, baseline_records = _compare_record(record, metric_policies)
|
||||
baseline_records = load_records_for_model(
|
||||
TRACKING_ROOT,
|
||||
record["model_id"],
|
||||
record["gpu_type"],
|
||||
**identity_filters,
|
||||
last_n=5,
|
||||
successful_only=True,
|
||||
baseline_eligible_only=True,
|
||||
)
|
||||
|
||||
if not baseline_records:
|
||||
# A brand-new cohort has NO regression gating until history
|
||||
# accumulates — make that loud and machine-readable instead of
|
||||
# an indistinguishable pass, so a cohort shift (intended or
|
||||
# accidental, e.g. an identity-field change) never silently
|
||||
# blinds the comparison.
|
||||
record["baseline_status"] = "initialized_new_cohort"
|
||||
print("=" * 72)
|
||||
print(f"WARNING: NO BASELINE — initializing a NEW cohort for "
|
||||
f"{record['model_id']} on {record['gpu_type']}"
|
||||
f"{_format_identity_filters(identity_filters)}")
|
||||
print("Regression gating is INACTIVE for this cohort until "
|
||||
"baseline history accumulates. If this cohort shift is "
|
||||
"unexpected, check the identity fields above.")
|
||||
print("=" * 72)
|
||||
failures = []
|
||||
else:
|
||||
record["baseline_status"] = "compared"
|
||||
failures = _check_regressions(record, baseline_records, metric_policies)
|
||||
if static_threshold_failed:
|
||||
failures.append(f"{record['model_id']} fixed-threshold phase failed "
|
||||
f"(PERF_PYTEST_RC={os.environ.get('PERF_PYTEST_RC')})")
|
||||
|
||||
record["success"] = not failures
|
||||
# Only compared-and-PASS records may advance the rolling baseline: a
|
||||
# CALIBRATION_NEEDED record must never silently seed a new cohort, and
|
||||
# records without the v2 identity block never become eligible.
|
||||
record["baseline_eligible"] = (
|
||||
identity_filters is not None
|
||||
and record["comparator_status"] == STATUS_PASS
|
||||
and _is_baseline_eligible(record["run_source"], record["success"])
|
||||
identity_filters is not None and _is_baseline_eligible(record["run_source"], record["success"])
|
||||
)
|
||||
all_failures.extend(failures)
|
||||
|
||||
|
||||
@@ -421,196 +421,3 @@ def test_informational_metric_remains_visible_without_failing():
|
||||
assert row["metrics"]["throughput"]["regressed"] is False
|
||||
assert row["threshold_exceeded_metrics"] == ["throughput"]
|
||||
assert row["failing_metrics"] == []
|
||||
|
||||
|
||||
def _v2_raw_result(**overrides):
|
||||
raw = _raw_result()
|
||||
raw.update({
|
||||
"workload_id": "wan-t2v",
|
||||
"variant_id": "1.3b-sp2",
|
||||
"benchmark_version": 2,
|
||||
"recipe_fingerprint": "recipe-1",
|
||||
"hardware_profile_id": "hw-1",
|
||||
"software_profile_id": "sw-1",
|
||||
})
|
||||
raw.update(overrides)
|
||||
return raw
|
||||
|
||||
|
||||
def _v2_baseline_record(**overrides):
|
||||
record = {
|
||||
"gpu_type": "NVIDIA L40S",
|
||||
"workload_id": "wan-t2v",
|
||||
"variant_id": "1.3b-sp2",
|
||||
"benchmark_version": 2,
|
||||
"recipe_fingerprint": "recipe-1",
|
||||
"hardware_profile_id": "hw-1",
|
||||
"software_profile_id": "sw-1",
|
||||
"latency": 10.0,
|
||||
"throughput": 4.5,
|
||||
"memory": 10000.0,
|
||||
"timestamp": "2026-06-15T00:00:00+00:00",
|
||||
"success": True,
|
||||
"run_source": "scheduled_main",
|
||||
"baseline_eligible": True,
|
||||
}
|
||||
record.update(overrides)
|
||||
return record
|
||||
|
||||
|
||||
def _run_compare(monkeypatch, tmp_path, raw_result, baseline_records):
|
||||
"""Run compare_baseline.main() against a local tracking root.
|
||||
|
||||
Returns (exit_code, normalized_record, markdown_report).
|
||||
"""
|
||||
results_dir = tmp_path / "results"
|
||||
results_dir.mkdir()
|
||||
(results_dir / "perf_current.json").write_text(json.dumps(raw_result))
|
||||
|
||||
model_dir = tmp_path / "tracking" / raw_result["benchmark_id"]
|
||||
model_dir.mkdir(parents=True)
|
||||
for index, record in enumerate(baseline_records):
|
||||
(model_dir / f"rec{index}.json").write_text(json.dumps(record))
|
||||
|
||||
reports_dir = tmp_path / "reports"
|
||||
monkeypatch.setattr(compare_baseline, "RESULTS_DIR", str(results_dir))
|
||||
monkeypatch.setattr(compare_baseline, "TRACKING_ROOT", str(tmp_path / "tracking"))
|
||||
monkeypatch.setattr(compare_baseline, "PERF_REPORTS_DIR", str(reports_dir))
|
||||
monkeypatch.setattr(compare_baseline, "UPLOAD_POLICY", "never")
|
||||
monkeypatch.setattr(compare_baseline, "sync_from_hf", lambda *args, **kwargs: None)
|
||||
monkeypatch.setenv("PERF_RUN_SOURCE", "scheduled_main")
|
||||
monkeypatch.delenv("PERF_PYTEST_RC", raising=False)
|
||||
monkeypatch.delenv("GITHUB_STEP_SUMMARY", raising=False)
|
||||
|
||||
exit_code = compare_baseline.main()
|
||||
|
||||
normalized_paths = list((reports_dir / "results").glob("normalized_perf_*.json"))
|
||||
assert len(normalized_paths) == 1
|
||||
markdown = "\n".join(path.read_text() for path in reports_dir.glob("perf_*.md"))
|
||||
return exit_code, json.loads(normalized_paths[0].read_text()), markdown
|
||||
|
||||
|
||||
def test_main_pass_with_comparable_baseline(monkeypatch, tmp_path):
|
||||
exit_code, record, markdown = _run_compare(
|
||||
monkeypatch, tmp_path, _v2_raw_result(), [_v2_baseline_record()])
|
||||
|
||||
assert exit_code == 0
|
||||
assert record["comparator_status"] == "PASS"
|
||||
assert record["baseline_status"] == "compared"
|
||||
assert record["success"] is True
|
||||
# Scheduled-main PASS records keep advancing the rolling baseline.
|
||||
assert record["baseline_eligible"] is True
|
||||
assert "| PASS |" in markdown
|
||||
|
||||
|
||||
def test_main_regression_on_gated_metric_fails_ci(monkeypatch, tmp_path):
|
||||
exit_code, record, markdown = _run_compare(
|
||||
monkeypatch, tmp_path, _v2_raw_result(avg_generation_time_s=20.0),
|
||||
[_v2_baseline_record()])
|
||||
|
||||
assert exit_code == 1
|
||||
assert record["comparator_status"] == "REGRESSION"
|
||||
assert record["success"] is False
|
||||
assert record["baseline_eligible"] is False
|
||||
assert "| REGRESSION |" in markdown
|
||||
|
||||
|
||||
def test_main_missing_baseline_is_calibration_needed_and_does_not_seed(monkeypatch, tmp_path):
|
||||
exit_code, record, markdown = _run_compare(
|
||||
monkeypatch, tmp_path, _v2_raw_result(), [])
|
||||
|
||||
assert exit_code == 0
|
||||
assert record["comparator_status"] == "CALIBRATION_NEEDED"
|
||||
assert record["baseline_status"] == "initialized_new_cohort"
|
||||
assert record["success"] is True
|
||||
# Visible, but never silently seeds a passing baseline.
|
||||
assert record["baseline_eligible"] is False
|
||||
assert "| CALIBRATION_NEEDED |" in markdown
|
||||
|
||||
|
||||
def test_main_recipe_change_without_new_variant_is_recipe_mismatch(monkeypatch, tmp_path):
|
||||
exit_code, record, markdown = _run_compare(
|
||||
monkeypatch, tmp_path, _v2_raw_result(recipe_fingerprint="recipe-2"),
|
||||
[_v2_baseline_record()])
|
||||
|
||||
assert exit_code == 1
|
||||
assert record["comparator_status"] == "RECIPE_MISMATCH"
|
||||
assert record["success"] is False
|
||||
assert record["baseline_eligible"] is False
|
||||
assert "| RECIPE_MISMATCH |" in markdown
|
||||
|
||||
|
||||
def test_main_recipe_change_as_new_variant_is_calibration_needed(monkeypatch, tmp_path):
|
||||
exit_code, record, _ = _run_compare(
|
||||
monkeypatch, tmp_path,
|
||||
_v2_raw_result(variant_id="1.3b-sp2-r2", recipe_fingerprint="recipe-2"),
|
||||
[_v2_baseline_record()])
|
||||
|
||||
assert exit_code == 0
|
||||
assert record["comparator_status"] == "CALIBRATION_NEEDED"
|
||||
|
||||
|
||||
def test_main_v2_record_never_compares_against_v1_baselines(monkeypatch, tmp_path):
|
||||
legacy_v1_baseline = {
|
||||
"gpu_type": "NVIDIA L40S",
|
||||
# Would be a >100% latency regression if the comparator (wrongly)
|
||||
# matched the v2 record against v1 history.
|
||||
"latency": 1.0,
|
||||
"timestamp": "2026-06-15T00:00:00+00:00",
|
||||
"success": True,
|
||||
}
|
||||
|
||||
exit_code, record, _ = _run_compare(
|
||||
monkeypatch, tmp_path, _v2_raw_result(), [legacy_v1_baseline])
|
||||
|
||||
assert exit_code == 0
|
||||
assert record["comparator_status"] == "CALIBRATION_NEEDED"
|
||||
|
||||
|
||||
def test_main_legacy_v1_record_skips_comparison_with_pass_verdict(monkeypatch, tmp_path):
|
||||
legacy_v1_baseline = {
|
||||
"gpu_type": "NVIDIA L40S",
|
||||
# Would be a >100% latency regression if the comparator (wrongly)
|
||||
# compared the identity-less v1 record against this history.
|
||||
"latency": 1.0,
|
||||
"timestamp": "2026-06-15T00:00:00+00:00",
|
||||
"success": True,
|
||||
}
|
||||
|
||||
exit_code, record, markdown = _run_compare(
|
||||
monkeypatch, tmp_path, _raw_result(), [legacy_v1_baseline])
|
||||
|
||||
assert exit_code == 0
|
||||
assert record["comparator_status"] == "PASS"
|
||||
assert record["baseline_status"] == "skipped_missing_identity"
|
||||
assert record["baseline_eligible"] is False
|
||||
assert "| PASS |" in markdown
|
||||
|
||||
|
||||
def test_main_partial_identity_skips_comparison(monkeypatch, tmp_path):
|
||||
raw = _v2_raw_result()
|
||||
del raw["software_profile_id"]
|
||||
|
||||
exit_code, record, _ = _run_compare(monkeypatch, tmp_path, raw, [])
|
||||
|
||||
assert exit_code == 0
|
||||
assert record["comparator_status"] == "PASS"
|
||||
assert record["baseline_status"] == "skipped_missing_identity"
|
||||
assert record["success"] is True
|
||||
assert record["baseline_eligible"] is False
|
||||
|
||||
|
||||
def test_main_baseline_load_failure_is_infra_error_and_fails_ci(monkeypatch, tmp_path):
|
||||
def _boom(*_args, **_kwargs):
|
||||
raise OSError("hf store unavailable")
|
||||
|
||||
monkeypatch.setattr(compare_baseline, "load_records_for_model", _boom)
|
||||
|
||||
exit_code, record, markdown = _run_compare(
|
||||
monkeypatch, tmp_path, _v2_raw_result(), [])
|
||||
|
||||
assert exit_code == 1
|
||||
assert record["comparator_status"] == "INFRA_ERROR"
|
||||
assert record["success"] is False
|
||||
assert record["baseline_eligible"] is False
|
||||
assert "| INFRA_ERROR |" in markdown
|
||||
|
||||
@@ -0,0 +1,113 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""SSIM-based similarity test for daVinci-MagiHuman base text-to-AV.
|
||||
|
||||
Reference videos for this test are seeded separately via the
|
||||
`.agents/skills/seed-ssim-references/` skill on Modal L40S and uploaded
|
||||
to `FastVideo/ssim-reference-videos`. Until refs exist, the first run
|
||||
will fail downloading; run the seed skill once and commit the URLs.
|
||||
|
||||
Resolution + steps kept small enough for a CI budget; the full-quality
|
||||
variant falls back to the registered preset defaults.
|
||||
"""
|
||||
import os
|
||||
|
||||
import pytest
|
||||
|
||||
from fastvideo.api.sampling_param import SamplingParam
|
||||
from fastvideo.logger import init_logger
|
||||
from fastvideo.tests.ssim.inference_similarity_utils import (
|
||||
resolve_inference_device_reference_folder,
|
||||
run_text_to_video_similarity_test,
|
||||
)
|
||||
|
||||
logger = init_logger(__name__)
|
||||
|
||||
# 15B DiT + T5-Gemma 9B + Wan VAE + Stable-Audio VAE doesn't fit on a
|
||||
# single L40S (44 GB). Shard across 2 ranks via FSDP.
|
||||
REQUIRED_GPUS = 2
|
||||
|
||||
device_reference_folder = resolve_inference_device_reference_folder(logger)
|
||||
|
||||
# Umbrella HF repo holds all four variants under sibling subfolders;
|
||||
# `maybe_download_model` parses "org/repo/subfolder" and only fetches
|
||||
# the selected subfolder. Override via `MAGI_HUMAN_MODEL_PATH` to point
|
||||
# at a local converted_weights/ dir.
|
||||
_MAGI_HUMAN_MODEL_PATH = os.getenv(
|
||||
"MAGI_HUMAN_MODEL_PATH",
|
||||
"FastVideo/MagiHuman-Diffusers/base",
|
||||
)
|
||||
|
||||
MAGI_HUMAN_BASE_PARAMS = {
|
||||
"num_gpus": 2,
|
||||
"model_path": _MAGI_HUMAN_MODEL_PATH,
|
||||
# height/width/guidance_scale/seed/fps mirror the registered
|
||||
# `magi_human_base` preset defaults (see
|
||||
# `fastvideo/pipelines/basic/magi_human/presets.py::MAGI_HUMAN_BASE`)
|
||||
# so the SSIM test exercises the same code path as
|
||||
# `examples/inference/basic/basic_magi_human.py`. Only the budget
|
||||
# knobs (num_frames, num_inference_steps, sp_size) differ for CI fit.
|
||||
"height": 256,
|
||||
"width": 480,
|
||||
"num_frames": 26, # seconds=1 at fps=25 + 1; preset = 101
|
||||
"num_inference_steps": 8, # CI budget; preset = 32
|
||||
"guidance_scale": 5.0,
|
||||
"seed": 42,
|
||||
"sp_size": 2,
|
||||
"tp_size": 1,
|
||||
"fps": 25,
|
||||
}
|
||||
|
||||
try:
|
||||
_MAGI_HUMAN_FULL_DEFAULTS = SamplingParam.from_pretrained(_MAGI_HUMAN_MODEL_PATH)
|
||||
MAGI_HUMAN_FULL_PARAMS = {
|
||||
"num_gpus": MAGI_HUMAN_BASE_PARAMS["num_gpus"],
|
||||
"model_path": MAGI_HUMAN_BASE_PARAMS["model_path"],
|
||||
"height": _MAGI_HUMAN_FULL_DEFAULTS.height,
|
||||
"width": _MAGI_HUMAN_FULL_DEFAULTS.width,
|
||||
"num_frames": _MAGI_HUMAN_FULL_DEFAULTS.num_frames,
|
||||
"num_inference_steps": _MAGI_HUMAN_FULL_DEFAULTS.num_inference_steps,
|
||||
"guidance_scale": _MAGI_HUMAN_FULL_DEFAULTS.guidance_scale,
|
||||
"seed": _MAGI_HUMAN_FULL_DEFAULTS.seed,
|
||||
"sp_size": MAGI_HUMAN_BASE_PARAMS["sp_size"],
|
||||
"tp_size": MAGI_HUMAN_BASE_PARAMS["tp_size"],
|
||||
"fps": _MAGI_HUMAN_FULL_DEFAULTS.fps,
|
||||
}
|
||||
except Exception:
|
||||
# Model not registered / accessible on this machine — fall back to the
|
||||
# quick params as the full-quality map too; the test will skip anyway
|
||||
# when the model path is unavailable.
|
||||
MAGI_HUMAN_FULL_PARAMS = MAGI_HUMAN_BASE_PARAMS
|
||||
|
||||
|
||||
MAGI_HUMAN_MODEL_TO_PARAMS = {
|
||||
"MagiHuman-Base-Diffusers": MAGI_HUMAN_BASE_PARAMS,
|
||||
}
|
||||
FULL_QUALITY_MAGI_HUMAN_MODEL_TO_PARAMS = {
|
||||
"MagiHuman-Base-Diffusers": MAGI_HUMAN_FULL_PARAMS,
|
||||
}
|
||||
|
||||
MAGI_HUMAN_TEST_PROMPTS = [
|
||||
"A person sitting by a window, softly lit by afternoon sun, waving at "
|
||||
"the camera with a gentle smile.",
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("prompt", MAGI_HUMAN_TEST_PROMPTS)
|
||||
@pytest.mark.parametrize("attention_backend_name", ["FLASH_ATTN"])
|
||||
@pytest.mark.parametrize("model_id", list(MAGI_HUMAN_MODEL_TO_PARAMS.keys()))
|
||||
def test_magi_human_base_inference_similarity(
|
||||
prompt: str,
|
||||
attention_backend_name: str,
|
||||
model_id: str,
|
||||
) -> None:
|
||||
run_text_to_video_similarity_test(
|
||||
logger=logger,
|
||||
script_dir=os.path.dirname(os.path.abspath(__file__)),
|
||||
device_reference_folder=device_reference_folder,
|
||||
prompt=prompt,
|
||||
attention_backend_name=attention_backend_name,
|
||||
model_id=model_id,
|
||||
default_params_map=MAGI_HUMAN_MODEL_TO_PARAMS,
|
||||
full_quality_params_map=FULL_QUALITY_MAGI_HUMAN_MODEL_TO_PARAMS,
|
||||
min_acceptable_ssim=0.60,
|
||||
)
|
||||
Reference in New Issue
Block a user