SolitaryThinker
eb9d439bab
Wire text-to-video+sound end-to-end, completing the audio PR. - CFG velocity (`cosmos3_get_cfg_velocity`) + `Cosmos3DenoiseEngine` take an optional `sound_specs`: the denoise latent becomes a combined [vision | sound] flat vector; each step packs vision+sound, forwards the DiT, zeros the prediction on conditioning frames for both, and returns the concatenated velocity. The UniPC scheduler steps the joint vector. `Cosmos3SoundSpec` + `_split_sound_latent` added. - `Cosmos3DenoisingStage` t2vs path (gated on `COSMOS3_T2VS`): size the sound latent from the video duration (framework `create_placeholder_audio` + `get_latent_num_samples`), append sound noise, joint-denoise, split, AVAE-decode the sound latent, and set `batch.extra["audio"]` / `["audio_sample_rate"]` so the generator muxes a stereo 48 kHz AAC track into the mp4. Sound AVAE is lazy-loaded from `<model_path>/sound_tokenizer` (`_get_sound_vae`). - Parity (`test_cosmos3_sound_parity::test_t2vs_cfg_velocity_matches_framework`): the combined [vision|sound] sequential-CFG velocity vs a framework-DiT oracle — max abs diff = 0.0, mean abs diff = 0.0 across t2vs/i2vs cases. - Example `basic_cosmos3_t2vs_new_api.py`. Verified on B200 (real weights, 1280x704, 29f, 35 steps): coherent ocean/sunset video WITH a real stereo 48 kHz audio track (mean -10.2 dB, peak -0.4 dB; not silence). Full cosmos3 suite: 130 passed, 0 skipped.
Local Tests
Local-only parity, smoke, and component tests for FastVideo model ports. They compare FastVideo against the official reference implementations and are skipped in CI; run them locally on a single GPU (or CPU where noted).
For the CI-backed test suite, see fastvideo/tests/.
Layout
Tests are organized by model family, one directory per port. Each family
directory follows the layout produced by the
add-model-prep skill:
tests/local_tests/<family>/
├── README.md # reference assets, setup, test inventory
├── __init__.py
└── test_<family>_<component>_*.py
Families
| Family | Workload | Dir |
|---|---|---|
| Flux2 | T2I | flux2/ |
| Hunyuan GameCraft | T2V / I2V | gamecraft/ |
| GEN3C | T2V | gen3c/ |
| Kandinsky-5 | T2V | kandinsky5/ |
| LTX-2 | T2V (+ audio) | ltx2/ |
| Stable Diffusion 3.5 | T2I | sd35/ |
| Stable Audio Open 1.0 | T2A | stable_audio/ |
| Wan2.2 | I2V | wan22/ |
Running a family
pytest tests/local_tests/<family>/ -v -s
See each family's README.md for setup (clones, weights, env vars, gated
tokens) and the per-component parity coverage.
Adding a new family
Run the add-model-prep
skill from the FastVideo repo root. It scaffolds the family directory and
README.md and stages reference clones + weights. (In-progress ports may also
add a PORT_STATUS.md from the same skill template; remove it once the port
lands.)