SolitaryThinker
d6affff3a1
Activate the dormant sound MoT heads and pack the sound modality, completing the t2vs DiT path (audio component 2). DiT forward (`Cosmos3VFMTransformer`): - Encode sound: pack each [C,T] latent -> [T,C], `audio_proj_in` (= framework `sound2llm`) + `audio_modality_embed`, scatter timestep embeds onto noisy frames, scatter into the (full) gen split. Mirrors framework `_encode_sound`. - Decode sound: `audio_proj_out` (= `llm2sound`) on noisy sound hidden states -> `_unpack_sound` to per-sample [C,T] (framework `_decode_sound`). - Reuses the general `_scatter_timestep_embeds` (works for (T,1,1) shapes). Sound packing (`sequence_packing.py`): `Cosmos3SoundItem` + sound fields on `Cosmos3SampleInputs`/`Cosmos3PackedSequence`/`to_dit_kwargs`. Sound shares the vision "full" split (preserving the causal+full 2-split invariant), with `(T,1,1)` token shapes, a `(T,1)` condition mask, mse-loss/timestep bookkeeping per noisy frame, and 3D-MRoPE temporal positions starting at the vision temporal offset (parallel to vision; `start_frame_offset=0`, tcf=1; does not advance the offset). Mirrors framework `_pack_sound_tokens`. Parity (`test_cosmos3_sound_parity.py`, framework model+pack = oracle): - sound packing fields + `position_ids` exact vs framework `pack_input_sequence` (has_sound) across t2vs/i2vs cases. - `preds_vision` AND `preds_sound`: max abs diff = 0.0, mean abs diff = 0.0 (both native-pack and framework-pack inputs). Tiny framework builder gains a `sound_gen` flag. Full suite: 127 passed.
Local Tests
Local-only parity, smoke, and component tests for FastVideo model ports. They compare FastVideo against the official reference implementations and are skipped in CI; run them locally on a single GPU (or CPU where noted).
For the CI-backed test suite, see fastvideo/tests/.
Layout
Tests are organized by model family, one directory per port. Each family
directory follows the layout produced by the
add-model-prep skill:
tests/local_tests/<family>/
├── README.md # reference assets, setup, test inventory
├── __init__.py
└── test_<family>_<component>_*.py
Families
| Family | Workload | Dir |
|---|---|---|
| Flux2 | T2I | flux2/ |
| Hunyuan GameCraft | T2V / I2V | gamecraft/ |
| GEN3C | T2V | gen3c/ |
| Kandinsky-5 | T2V | kandinsky5/ |
| LTX-2 | T2V (+ audio) | ltx2/ |
| Stable Diffusion 3.5 | T2I | sd35/ |
| Stable Audio Open 1.0 | T2A | stable_audio/ |
| Wan2.2 | I2V | wan22/ |
Running a family
pytest tests/local_tests/<family>/ -v -s
See each family's README.md for setup (clones, weights, env vars, gated
tokens) and the per-component parity coverage.
Adding a new family
Run the add-model-prep
skill from the FastVideo repo root. It scaffolds the family directory and
README.md and stages reference clones + weights. (In-progress ports may also
add a PORT_STATUS.md from the same skill template; remove it once the port
lands.)