# Testing In FastVideo This guide explains how to add and run tests in FastVideo. CI routing, slash-command mappings, and workflow ownership live in [CI/CD Architecture](ci_architecture.md). ## Test Types | Type | Location | Purpose | |---|---|---| | Unit tests | `fastvideo/tests/api`, `fastvideo/tests/dataset`, `fastvideo/tests/entrypoints`, `fastvideo/tests/workflow`, CPU-safe `fastvideo/tests/train` subsets | Validate individual functions, APIs, contracts, and lightweight workflows. | | Component tests | `fastvideo/tests/encoders`, `fastvideo/tests/transformers`, `fastvideo/tests/vaes` | Validate loading and basic behavior for model components. | | Golden gates | `fastvideo/tests/golden_gate` | Compare small, deterministic component outputs exactly against device/runtime-matched reference tensors. | | Train framework tests | `fastvideo/tests/train/models`, `fastvideo/tests/train/methods` | Exercise the new `fastvideo/train/` framework on real checkpoints and tiny synthetic batches. | | SSIM tests | `fastvideo/tests/ssim` | Compare generated videos against references to catch visual regressions. | | Training tests | `fastvideo/tests/training` | Validate legacy training loops, LoRA, distillation, self-forcing, and VSA behavior. | | Inference tests | `fastvideo/tests/inference` | Validate specialized inference paths such as LoRA inference and V-MoBA. | | Performance tests | `fastvideo/tests/performance` | Gate latency, throughput, peak memory, and stage timings. See [Performance Benchmarks](performance_benchmarks.md). | | Local performance behavior tests | `tests/local_tests/performance` | Validate performance config, policy, identity, result schemas, dashboards, and worker-log capture outside CI. | | Eval tests | `fastvideo/tests/eval` | Check eval metrics against pinned reference scores and assets. | | DreamVerse app tests | `apps/dreamverse` | Validate the DreamVerse backend, frontend, and mock-backed browser flows. | ## Running Tests Locally Start with the cheapest checks that cover the changed behavior, and stop on a failure before starting heavier dependent checks: 1. Run pre-commit on changed files and focused import, config, and contract tests. 2. Run the smallest matching component golden gate. Prefer one GPU, tiny fixed inputs, cached component weights, and direct tensor comparisons over renders. 3. Run focused component parity or default SSIM when the golden does not cover the changed behavior, such as VAE normalization or pipeline wiring. 4. Run full-quality renders or broad suites when required by the change, an explicit request, or CI policy, rather than on every edit. A golden must cover the component being changed. Wan has four small gates: | Gate | Boundary | |---|---| | `test_wan_t2v.py` | Dense transformer block 0 | | `test_wan_vae.py` | FP32 encode, BF16 decode, streaming, and cache reset | | `test_wan_causal.py` | Real block weights, cache append/rewrite, and sink eviction | | `test_wan_denoising.py` | Three real 1.3B DiT/UniPC steps with fixed prompt embeddings | These use an immutable Wan checkpoint revision and only download the required component. The trajectory gate needs no tokenizer, text encoder, VAE, or video reference. Weight-free stage tests also exercise every step of a 50-step UniPC loop, CFG caching, expert switching, conditioning layouts, and DMD RNG order. These checks do not replace independent Diffusers parity or end-to-end SSIM. Use the golden's matching GPU, dtype, backend, and runtime. A missing reference or environment mismatch is not a pass. Do not replace a reference with the candidate output just to clear a failure. For a relocation, the unchanged parent is the baseline; two imports of the same class are not numerical proof. FA4 fingerprints record the effective boolean switch. Legacy block references with `fastvideo_fa4: ` mean disabled and match an explicit `0`; enabling FA4 still fails the environment check. Other runtime pins and zero-tolerance tensor comparisons remain strict. The VAE and transformer CI lanes run their Wan component goldens first. Selected integration lanes wait for the golden lane in merge/full builds. See [CI/CD Architecture](ci_architecture.md) for direct-rerun and skip semantics. For Wan, one command enforces the local ordering and stops on failure: The initial contracts include `tests/api/test_wan_definitions.py`: all registered Wan aliases, local-manifest detector precedence, sampling/precision defaults, config isolation, and legacy serialized-config compatibility. These require no weights or Hub access; the current package import still needs its prepared runtime environment. ```bash bash scripts/validate_wan.sh vae # contracts, then the VAE golden bash scripts/validate_wan.sh dense parity # contracts, goldens, Diffusers parity bash scripts/validate_wan.sh all default # then focused T2V/I2V/causal SSIM ``` The second argument is an upper validation tier, not a reference override. Supply the GPU/backend/runtime and SSIM model/tier settings that match the reference. The script never updates references. Causal component coverage is a block-cache fingerprint, not independent full causal-pipeline parity. New named-tensor gates write a missing output to `*.candidate.pt` and fail; running candidate code again cannot turn it into an approved reference. Seed from unchanged, pushed source, verify two independent processes bit-for-bit, then review and publish only the new reference files. Preserve the baseline source SHA, test recipe SHA, checkpoint revision, runtime, and comparison receipt with the run artifacts. Never overwrite an existing video reference as a side effect of adding a tensor gate. Examples of focused checks: ```bash pytest fastvideo/tests/loader/test_wan_family_imports.py -q pytest fastvideo/tests/golden_gate/test_wan_t2v.py -q pytest fastvideo/tests/vaes/test_wan_vae.py -q ``` GPU-heavy suites need the right hardware, credentials, local caches, and sometimes custom kernels. Document those assumptions in new tests. The legacy VSA training regression selects a device-specific summary. Its GB200 reference was generated from unchanged upstream commit `d5253fd71055927f12aa6c4aa679b93a00238851` using two GB200 GPUs, Torch 2.12.0+cu130, CUDA 13.0 and FA2. Two independent upstream runs produced gradient norms 1.909 and 1.935 and losses 0.30861 and 0.30886. The reference JSON retains provenance; the H100 reference and all comparison tolerances remain unchanged. Candidate outputs must not be used to refresh this reference. ## SSIM Tests SSIM tests generate videos using specific models and parameters, then compare the output against reference videos with Structural Similarity Index Measure. Use them for pipeline-level visual regression coverage when output quality or generation behavior must be preserved. :::note Add enough prompts, seeds, backends, and parameter combinations to cover the behavior you want to protect, but keep runtime reasonable for CI. ::: ### Directory Structure ```text fastvideo/tests/ssim/ ├── reference_videos/ │ ├── default/ │ │ └── _reference_videos/ │ │ └── / │ │ └── / │ │ └── │ └── full_quality/ │ └── _reference_videos/ ├── generated_videos/ ├── reference_videos_cli.py ├── test_wan_t2v_similarity.py ├── test_wan_i2v_similarity.py └── ... ``` ### Adding An SSIM Test 1. Create or update a model-specific file, for example `test_wan_t2v_similarity.py`. 2. Define model parameters such as model path, dimensions, frame count, inference steps, guidance, seed, and GPU count. 3. Parametrize prompts, attention backends, and model variants where useful. 4. Generate the video with `VideoGenerator`. 5. Compare the generated video with the reference using the SSIM helpers. 6. Seed or update reference videos only after inspecting output quality. Example shape: ```python import pytest MY_MODEL_PARAMS = { "num_gpus": 1, "model_path": "organization/model-name", "height": 480, "width": 832, "num_frames": 45, "num_inference_steps": 20, } @pytest.mark.parametrize("prompt", TEST_PROMPTS) @pytest.mark.parametrize("attention_backend", ["FLASH_ATTN"]) def test_my_model_similarity(prompt, attention_backend): # Set backend, generate video, and compare against reference. ssim_values = compute_video_ssim_torchvision( reference_path, generated_path, use_ms_ssim=True, ) assert ssim_values[0] >= 0.98 ``` ### Reference Videos When a reference is missing, the test writes generated output under: ```text fastvideo/tests/ssim/generated_videos//_reference_videos ``` After inspecting the generated video, copy it into the matching reference tree: ```text fastvideo/tests/ssim/reference_videos//_reference_videos/// ``` The helper CLI can copy, upload, and download references: ```bash python fastvideo/tests/ssim/reference_videos_cli.py copy-local \ --quality-tier default \ --reference-dir fastvideo/tests/ssim/reference_videos/default/L40S_reference_videos python fastvideo/tests/ssim/reference_videos_cli.py upload --quality-tier all python fastvideo/tests/ssim/reference_videos_cli.py download \ --quality-tier full_quality \ --device-folder H200_reference_videos ``` ### Running SSIM Locally ```bash pytest fastvideo/tests/ssim/ -vs ``` Use a machine whose GPU and backend match the reference folder you are testing. ## Slurm CI Runs For SSIM Comment `/test ssim` on a pull request to run the canonical four-GPU SSIM lane on the Slinky Slurm cluster. `fastvideo/tests/ssim/ci_runner.py` discovers the suite without importing test modules, packs independent pytest processes across the four assigned GPUs, and stops the lane on the first failure. The change-aware `/merge` planner may run only the SSIM test basenames owned by the changed model family. Shared SSIM harness changes still select the complete lane. Independently, `main` runs the full SSIM matrix every Sunday at 05:00 UTC so infrequently touched model families retain periodic coverage. For a focused developer run, invoke pytest directly and optionally select one model from a parameterized test through `FASTVIDEO_SSIM_MODEL_ID`: ```bash pytest fastvideo/tests/ssim/test_wan_t2v_similarity.py -vs FASTVIDEO_SSIM_MODEL_ID=Wan2.1-T2V-1.3B-Diffusers \ pytest fastvideo/tests/ssim/test_wan_t2v_similarity.py -vs ``` The files under `fastvideo/tests/modal/` are retained only as a disabled manual rollback implementation. No active CI trigger invokes them. ### SSIM Bootstrap Mode Normal SSIM runs are strict: if a reference video or latent is missing, the test fails. For new-model PRs, CI can run SSIM in bootstrap mode so missing references are uploaded as draft artifacts for review instead of immediately blocking on a missing canonical reference. The active Slurm SSIM lane (`.buildkite/scripts/lanes/ssim.sh`) enables bootstrap mode only when `FASTVIDEO_SSIM_BOOTSTRAP_MODE=1` is set for the Buildkite job. A `[new-model]` PR title does not enable it on this path (only the retired Modal script `.buildkite/scripts/pr_test.sh` checked the title), so ask a maintainer to start the SSIM job with that variable set. Bootstrap mode passes `--ssim-bootstrap-mode` to pytest. When a generated artifact is available, the test uploads it under the `drafts/...` namespace in the SSIM reference repo and marks that case as expected-failed. After reviewing the draft, promote it into the canonical reference layout: ```bash python fastvideo/tests/ssim/reference_videos_cli.py promote-draft \ --quality-tier default \ --device-folder L40S_reference_videos \ --model-id ``` ## CI Integration FastVideo GPU CI is orchestrated by Buildkite and runs only on isolated Slinky Slurm workers. The main files are: | File | Purpose | |---|---| | `.buildkite/pipeline.yml` | Static, validated 20-lane Slurm test graph. | | `.github/scripts/plan_merge_ci.py` | Trusted path-to-lane and focused quality-test policy for `/merge`. | | `.buildkite/scripts/unit_test.sh`, `.buildkite/scripts/lanes/*.sh` | Repository-owned test payloads executed inside Slurm containers. | | `fastvideo/tests/ssim/ci_runner.py` | Four-GPU SSIM task discovery and scheduling. | | `.buildkite/scripts/pr_test.sh`, `fastvideo/tests/modal/*.py` | Dormant manual rollback path; rejected in Buildkite. | For exact tier membership, slash commands, runner isolation, and aggregate statuses, see [CI/CD Architecture](ci_architecture.md). ### Adding A New CI Test Category If a new test does not fit an existing lane: 1. Put the test payload in an executable `.buildkite/scripts/lanes/.sh`. 2. Add its static step to `.buildkite/pipeline.yml`, its changed-path ownership to `.github/scripts/plan_merge_ci.py`, and extend the CI contract tests. 3. Add the `/test` mapping in `.github/workflows/ci-slash-commands.yml`. 4. Coordinate the matching GPU, timeout, dependency, secret, and artifact policy in the private Slurm runner allowlist. 5. Document the new category in [CI/CD Architecture](ci_architecture.md) and add authoring notes here if contributors need them. When a test only extends an existing category, update that category's tests instead of adding a new CI lane.