The reference videos are used as part of e2e SSIM regression tests.
Wan inference coverage now lives in test_wan_t2v_similarity.py and
test_wan_i2v_similarity.py alongside the other model-specific SSIM files.
These tests compare newly generated videos against references to detect quality
regressions.
reference layout:
reference_videos/default/<GPU>_reference_videos/<model>/<backend>/...reference_videos/full_quality/<GPU>_reference_videos/<model>/<backend>/...
A40_reference_videos are generated on A40s and so on. (legacy default layout
<ssim_dir>/<GPU>_reference_videos/... is still read as fallback.)
Before SSIM tests run, missing reference videos are auto-downloaded from a
public HF repo (configured by FASTVIDEO_SSIM_REFERENCE_HF_REPO, default:
FastVideo/ssim-reference-videos).
Use the CLI:
python fastvideo/tests/ssim/reference_videos_cli.py --helppython fastvideo/tests/ssim/reference_videos_cli.py copy-local --helppython fastvideo/tests/ssim/reference_videos_cli.py copy-local --quality-tier default --device-folder H200_reference_videospython fastvideo/tests/ssim/reference_videos_cli.py copy-local --quality-tier full_quality --device-folder H200_reference_videospython fastvideo/tests/ssim/reference_videos_cli.py download --helppython fastvideo/tests/ssim/reference_videos_cli.py upload --helppython fastvideo/tests/ssim/reference_videos_cli.py download --quality-tier allpython fastvideo/tests/ssim/reference_videos_cli.py upload --quality-tier allpython fastvideo/tests/ssim/reference_videos_cli.py download --quality-tier full_quality --device-folder H200_reference_videospython fastvideo/tests/ssim/reference_videos_cli.py upload --quality-tier full_quality --device-folder H200_reference_videos
For upload, the tool reads HF token from HF_API_KEY /
HUGGINGFACE_HUB_TOKEN / HF_TOKEN and fails fast if none are set.
run pytest fastvideo/tests/ssim/ -vs --ssim-full-quality to use the
*_FULL_QUALITY_PARAMS configs (default run keeps the original shortened test
configs).
to override the HF reference repo at test time:
pytest fastvideo/tests/ssim/ -vs --ssim-reference-repo <org/repo>
to skip auto-download of missing refs:
pytest fastvideo/tests/ssim/ -vs --skip-ssim-reference-download
to bootstrap draft refs for a new model:
pytest fastvideo/tests/ssim/ -vs --ssim-bootstrap-mode
generated videos are written under:
- default params:
generated_videos/default/<GPU>_reference_videos/<model_id>/<ATTENTION_BACKEND>/ - full-quality params:
generated_videos/full_quality/<GPU>_reference_videos/<model_id>/<ATTENTION_BACKEND>/
HF repo layout mirrors quality + GPU split:
reference_videos/default/<GPU>_reference_videos/...reference_videos/full_quality/<GPU>_reference_videos/...
Draft bootstrap refs are uploaded under:
drafts/default/<GPU>_reference_videos/<model_id>/<ATTENTION_BACKEND>/...drafts/full_quality/<GPU>_reference_videos/<model_id>/<ATTENTION_BACKEND>/...
reference videos were generated on commit 4aeabbc629e0edf91477e80e795e7bb1823c71cb
causal videos were generated on commit b318063c0a
Adding a New SSIM Test
SSIM CI runs in one Modal instance (L40S:8). The orchestrator
(fastvideo/tests/modal/ssim_test.py) clones/builds/installs once, then
schedules multiple pytest subprocesses by GPU demand. Logs are printed in
deterministic order for failed tasks (file name, then model id), and
fail-fast is enabled: if one task fails, active tasks are terminated and the
full SSIM step fails.
The orchestrator auto-discovers every test_*.py file here, so no CI config
changes are needed when adding a new test. To declare how many GPUs your test
requires, add a module-level constant near the top of the file:
REQUIRED_GPUS = 2
If REQUIRED_GPUS is omitted, the test defaults to 1 GPU.
For files that define model maps (*_MODEL_TO_PARAMS), CI splits execution
into one subprocess per model id by setting FASTVIDEO_SSIM_MODEL_ID.
Bootstrapping New References
Normal SSIM runs are strict: missing references fail the test. For a new model
PR, add [new-model] to the PR title or set FASTVIDEO_SSIM_BOOTSTRAP_MODE=1
for the Buildkite job. CI then passes --ssim-bootstrap-mode to pytest.
In bootstrap mode, a missing pixel .mp4 or latent .pt reference is treated
as an expected draft-reference case after the generated artifact has been
written. The test uploads that generated artifact to the HF reference repo under
drafts/... and marks the case xfail. Bootstrap mode does not weaken normal
CI because it is opt-in only.
After reviewing a draft, promote it into the canonical reference layout:
python fastvideo/tests/ssim/reference_videos_cli.py promote-draft \
--quality-tier default \
--device-folder L40S_reference_videos \
--model-id <model_id>
Use --attention-backend <backend> to promote one backend folder. Promotion
refuses to overwrite existing canonical refs by default; pass --force only
when intentionally replacing reviewed references.
Generation Details
SSIM CI currently schedules work across the GPUs configured in
fastvideo/tests/modal/ssim_test.py.
Generation Parameters
FastHunyuan-diffusers: { "num_gpus": 2, "model_path": "data/FastHunyuan-diffusers", "height": 720, "width": 1280, "num_frames": 45, "num_inference_steps": 6, "guidance_scale": 1, "embedded_cfg_scale": 6, "flow_shift": 17, "seed": 1024, "sp_size": 2, "tp_size": 1, "vae_sp": true, "fps": 24 }
Wan2.1-T2V-1.3B-Diffusers: { "num_gpus": 2, "model_path": "Wan-AI/Wan2.1-T2V-1.3B-Diffusers", "height": 480, "width": 832, "num_frames": 45, "num_inference_steps": 20, "guidance_scale": 3, "embedded_cfg_scale": 6, "flow_shift": 7.0, "seed": 1024, "sp_size": 2, "tp_size": 1, "vae_sp": True, "fps": 24, "neg_prompt": "Bright tones, overexposed, static, blurred details, subtitles, style, works, paintings, images, static, overall gray, worst quality, low quality, JPEG compression residue, ugly, incomplete, extra fingers, poorly drawn hands, poorly drawn faces, deformed, disfigured, misshapen limbs, fused fingers, still picture, messy background, three legs, many people in the background, walking backwards", "text-encoder-precision": "fp32" }
Wan2.1-I2V-14B-480P-Diffusers: { "num_gpus": 2, "model_path": "Wan-AI/Wan2.1-I2V-14B-480P-Diffusers", "height": 480, "width": 832, "num_frames": 45, "num_inference_steps": 6, "guidance_scale": 5.0, "embedded_cfg_scale": 6, "flow_shift": 7.0, "seed": 1024, "sp_size": 2, "tp_size": 1, "vae_sp": True, "fps": 24, "neg_prompt": "Bright tones, overexposed, static, blurred details, subtitles, style, works, paintings, images, static, overall gray, worst quality, low quality, JPEG compression residue, ugly, incomplete, extra fingers, poorly drawn hands, poorly drawn faces, deformed, disfigured, misshapen limbs, fused fingers, still picture, messy background, three legs, many people in the background, walking backwards", "text-encoder-precision": "fp32" }
Text-to-Video Prompts
-
"Will Smith casually eats noodles, his relaxed demeanor contrasting with the energetic background of a bustling street food market. The scene captures a mix of humor and authenticity. Mid-shot framing, vibrant lighting."
-
"A lone hiker stands atop a towering cliff, silhouetted against the vast horizon. The rugged landscape stretches endlessly beneath, its earthy tones blending into the soft blues of the sky. The scene captures the spirit of exploration and human resilience. High angle, dynamic framing, with soft natural lighting emphasizing the grandeur of nature."
Image-to-Video Prompts
- "An astronaut hatching from an egg, on the surface of the moon, the darkness and depth of space realised in the background. High quality, ultrarealistic detail and breath-taking movie-like camera shot." Image path: "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg"
