diff --git a/tests/local_tests/cosmos3/AUDIO_PORT_PLAN.md b/tests/local_tests/cosmos3/AUDIO_PORT_PLAN.md index cfd2bc8bd..bf043be87 100644 --- a/tests/local_tests/cosmos3/AUDIO_PORT_PLAN.md +++ b/tests/local_tests/cosmos3/AUDIO_PORT_PLAN.md @@ -55,5 +55,21 @@ Goal: text-to-video+sound (**t2vs**) — generate synchronized audio alongside v ## Status - [x] Branch forked, framework audio path studied, plan written. -- [ ] D1 decision -> start AVAE port + parity harness. -- [ ] DiT sound pathway, packing, pipeline, AV infra. +- [x] D1: native port (user-chosen). D2: t2vs first. +- [x] **AVAE sound decoder (component 1) — DONE** (commit `5f81fb3d5`). Key + finding: the checkpoint is decoder-only in AutoencoderOobleck naming with + SnakeBeta + weight_g/v == FastVideo's native `OobleckVAE` decoder. Reused it + (+ `output_padding=stride%2` for the odd stride 5); `Cosmos3SoundVAE` + decoder-only wrapper; bit-exact parity vs the framework OobleckDecoder + (`test_cosmos3_avae_parity`); real 1.9 GB checkpoint strict-loads, decodes + [1,64,25] -> [1,2,48000] (1 s @ 48 kHz stereo). +- [ ] **DiT sound pathway (component 2)** — activate the dormant audio heads + (`audio_proj_in`/`audio_proj_out`/`audio_modality_embed`) in the native DiT + forward; port `pack_sound_latents` + sound token scatter/proj/modality-embed/ + velocity from the framework MoT. Parity: extend the DiT harness with sound. +- [ ] **Sound sequence packing (component 3)** — sound modality in + `sequence_packing.py` (positions, attn mode, condition mask). Parity vs + framework `pack_input_sequence` with sound. +- [ ] **t2vs pipeline + AV mux (components 4-5)** — placeholder audio, joint + vision+sound denoise, AVAE-decode, mux into mp4 / save wav; example; + real-weights verification. diff --git a/tests/local_tests/cosmos3/PORT_STATUS.md b/tests/local_tests/cosmos3/PORT_STATUS.md index 31571cc88..7c5eafc35 100644 --- a/tests/local_tests/cosmos3/PORT_STATUS.md +++ b/tests/local_tests/cosmos3/PORT_STATUS.md @@ -29,7 +29,7 @@ | scheduler | generic | reuse (flow-coerced) | framework `FlowUniPCMultistepScheduler` (`cosmos_framework/.../fm_solvers_unipc.py`; checkpoint ships diffusers-style config) | `model_index.json: scheduler`; `cosmos_framework/.../samplers/unipc.py:UniPCSampler` | FastVideo-native `UniPCMultistepScheduler` (flow config), coerced in `initialize_pipeline` | done | n/a | framework-parity DONE (`test_cosmos3_scheduler_parity`: timesteps bit-exact, sigmas ~1e-8, trajectory <~1e-6) | I003 (resolved) | | text_tokenizer | tokenizer | reuse | transformers `Qwen2TokenizerFast` | `model_index.json: text_tokenizer` | reuse (tokenizer = allowed third-party) | not_started | passthrough | scaffold_skip (`test_cosmos3_tokenizer_chat_template`) | - | | vision_encoder | encoder | port | transformers `Qwen3VLVisionModel` | `model_index.json: vision_encoder` | new encoder bucket OR documented lazy-wrapper | not_started | not_started | not_started | Q002 | -| sound_tokenizer | generic/vae | port | `Cosmos3AVAEAudioTokenizer` (model_type `autoencoder_v2`) | `model_index.json: sound_tokenizer` | new audio component | not_started | not_started | not_started | Q003 | +| sound_tokenizer | generic/vae | port (decode) | framework AVAE `LatentAutoEncoderV2` (`avae_utils`); checkpoint is decoder-only AutoencoderOobleck-named w/ SnakeBeta | `model_index.json: sound_tokenizer` | reuse FastVideo native `OobleckVAE` decoder + `Cosmos3SoundVAE` wrapper (`models/audio/cosmos3_avae.py`) | done (decode) | n/a | DECODE bit-exact vs framework (`test_cosmos3_avae_parity`); real ckpt strict-loads | PR2 (branch feat/cosmos3-audio) | ## Conversion State