SolitaryThinker
54def005c4
Add the action (multi-embodiment world-model) modality. The Nano checkpoint ships action weights (action_gen=true, 32 embodiment domains); this activates them, bit-exact vs the framework. - DiT forward: action encode/decode mirroring sound but domain-aware — pack each [T, action_dim] latent with a per-token embodiment domain id, `action_proj_in` (= framework `action2llm`, a `DomainAwareLinear`) + `action_modality_embed` + timestep scatter; decode via `action_proj_out` (= `llm2action`) on noisy hidden states -> `_unpack_action`. Mirrors `_encode_action`/`_decode_action`. - Sound/action share the vision "full" split; sequence/flat order is [vision | action | sound] (matches the framework per-sample concat). - Packing: `Cosmos3ActionItem` + action fields. Action tokens are `(T,)`-shaped, domain-tagged, with 3D-MRoPE at the vision temporal offset and `start_frame_offset=1` (framework `_pack_action_tokens`). - CFG velocity / `Cosmos3DenoiseEngine` take optional `action_specs` (`Cosmos3ActionSpec`); the joint denoise machinery now handles action. Parity (`test_cosmos3_action_parity.py`, framework model+pack = oracle, 3 embodiment domains): - action packing fields + `position_ids` exact; - `preds_vision` + `preds_action` (forward): max abs diff = 0.0, mean = 0.0; - combined [vision|action] CFG velocity: max abs diff = 0.0, mean = 0.0. Action is processed bit-exactly at the model + packing + denoise-glue level. A full action2world real-weights run is gated on real robot-action input data (not available here); the bit-exact parity is the correctness proof. Full suite: 139 passed, 0 skipped.
Local Tests
Local-only parity, smoke, and component tests for FastVideo model ports. They compare FastVideo against the official reference implementations and are skipped in CI; run them locally on a single GPU (or CPU where noted).
For the CI-backed test suite, see fastvideo/tests/.
Layout
Tests are organized by model family, one directory per port. Each family
directory follows the layout produced by the
add-model-prep skill:
tests/local_tests/<family>/
├── README.md # reference assets, setup, test inventory
├── __init__.py
└── test_<family>_<component>_*.py
Families
| Family | Workload | Dir |
|---|---|---|
| Flux2 | T2I | flux2/ |
| Hunyuan GameCraft | T2V / I2V | gamecraft/ |
| GEN3C | T2V | gen3c/ |
| Kandinsky-5 | T2V | kandinsky5/ |
| LTX-2 | T2V (+ audio) | ltx2/ |
| Stable Diffusion 3.5 | T2I | sd35/ |
| Stable Audio Open 1.0 | T2A | stable_audio/ |
| Wan2.2 | I2V | wan22/ |
Running a family
pytest tests/local_tests/<family>/ -v -s
See each family's README.md for setup (clones, weights, env vars, gated
tokens) and the per-component parity coverage.
Adding a new family
Run the add-model-prep
skill from the FastVideo repo root. It scaffolds the family directory and
README.md and stages reference clones + weights. (In-progress ports may also
add a PORT_STATUS.md from the same skill template; remove it once the port
lands.)