nodes/hao-ai-lab-FastVideo
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

[docs] cosmos3 PR2: AVAE sound decoder done; remaining audio components

Browse Source
This commit is contained in:
SolitaryThinker
2026-07-21 15:45:38 -07:00
parent e741852b89
commit 876a64b3fd
2 changed files with 19 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/local_tests/cosmos3/AUDIO_PORT_PLAN.md
+18 -2
View File
@@ -55,5 +55,21 @@ Goal: text-to-video+sound (**t2vs**) — generate synchronized audio alongside v
## Status
- [x] Branch forked, framework audio path studied, plan written.
- [ ] D1 decision -> start AVAE port + parity harness.
- [ ] DiT sound pathway, packing, pipeline, AV infra.
- [x] D1: native port (user-chosen). D2: t2vs first.
- [x] **AVAE sound decoder (component 1) — DONE** (commit `5f81fb3d5`). Key
finding: the checkpoint is decoder-only in AutoencoderOobleck naming with
SnakeBeta + weight_g/v == FastVideo's native `OobleckVAE` decoder. Reused it
(+ `output_padding=stride%2` for the odd stride 5); `Cosmos3SoundVAE`
decoder-only wrapper; bit-exact parity vs the framework OobleckDecoder
(`test_cosmos3_avae_parity`); real 1.9 GB checkpoint strict-loads, decodes
[1,64,25] -> [1,2,48000] (1 s @ 48 kHz stereo).
- [ ] **DiT sound pathway (component 2)** — activate the dormant audio heads
(`audio_proj_in`/`audio_proj_out`/`audio_modality_embed`) in the native DiT
forward; port `pack_sound_latents` + sound token scatter/proj/modality-embed/
velocity from the framework MoT. Parity: extend the DiT harness with sound.
- [ ] **Sound sequence packing (component 3)** — sound modality in
`sequence_packing.py` (positions, attn mode, condition mask). Parity vs
framework `pack_input_sequence` with sound.
- [ ] **t2vs pipeline + AV mux (components 4-5)** — placeholder audio, joint
vision+sound denoise, AVAE-decode, mux into mp4 / save wav; example;
real-weights verification.
tests/local_tests/cosmos3/PORT_STATUS.md
+1 -1
View File
@@ -29,7 +29,7 @@
| scheduler | generic | reuse (flow-coerced) | framework `FlowUniPCMultistepScheduler` (`cosmos_framework/.../fm_solvers_unipc.py`; checkpoint ships diffusers-style config) | `model_index.json: scheduler`; `cosmos_framework/.../samplers/unipc.py:UniPCSampler` | FastVideo-native `UniPCMultistepScheduler` (flow config), coerced in `initialize_pipeline` | done | n/a | framework-parity DONE (`test_cosmos3_scheduler_parity`: timesteps bit-exact, sigmas ~1e-8, trajectory <~1e-6) | I003 (resolved) |
| text_tokenizer | tokenizer | reuse | transformers `Qwen2TokenizerFast` | `model_index.json: text_tokenizer` | reuse (tokenizer = allowed third-party) | not_started | passthrough | scaffold_skip (`test_cosmos3_tokenizer_chat_template`) | - |
| vision_encoder | encoder | port | transformers `Qwen3VLVisionModel` | `model_index.json: vision_encoder` | new encoder bucket OR documented lazy-wrapper | not_started | not_started | not_started | Q002 |
| sound_tokenizer | generic/vae | port | `Cosmos3AVAEAudioTokenizer` (model_type `autoencoder_v2`) | `model_index.json: sound_tokenizer` | new audio component | not_started | not_started | not_started | Q003 |
| sound_tokenizer | generic/vae | port (decode) | framework AVAE `LatentAutoEncoderV2` (`avae_utils`); checkpoint is decoder-only AutoencoderOobleck-named w/ SnakeBeta | `model_index.json: sound_tokenizer` | reuse FastVideo native `OobleckVAE` decoder + `Cosmos3SoundVAE` wrapper (`models/audio/cosmos3_avae.py`) | done (decode) | n/a | DECODE bit-exact vs framework (`test_cosmos3_avae_parity`); real ckpt strict-loads | PR2 (branch feat/cosmos3-audio) |
## Conversion State