diff --git a/.gitignore b/.gitignore index e1c63503e..4787eed47 100644 --- a/.gitignore +++ b/.gitignore @@ -39,6 +39,11 @@ logs/ official_weights/ converted_weights/ +# Cosmos3 local parity assets (symlinked from main worktree) +/official_weights/ +/converted_weights/ +/cosmos-framework + # SSIM test outputs fastvideo/tests/ssim/generated_videos/ **/.cache/** diff --git a/tests/local_tests/cosmos3/PORT_STATUS.md b/tests/local_tests/cosmos3/PORT_STATUS.md new file mode 100644 index 000000000..5761798fd --- /dev/null +++ b/tests/local_tests/cosmos3/PORT_STATUS.md @@ -0,0 +1,84 @@ +# Cosmos3 Port Status + +## Summary + +- model_family: `cosmos3` +- workload_types: `T2V, I2V, T2I` supported by `WorkloadType` today; full-omni target also needs audio (AV), VLM reasoning, and action-conditioning, which require framework extensions (Q002, Q003). +- official_ref: `https://github.com/NVIDIA/cosmos-framework` — diffusers backend `diffusers_cosmos3.pipeline.Cosmos3OmniDiffusersPipeline`; HF `nvidia/Cosmos3-Nano`. +- official_ref_dir: `cosmos-framework` (symlink -> `/home/william5lin/FastVideo/cosmos-framework`, commit `003d66d4`) +- hf_weights_path: `nvidia/Cosmos3-Nano` +- local_weights_dir: `official_weights/cosmos3` (symlink -> `/home/william5lin/FastVideo/official_weights/cosmos3`, 33 GiB / 67 files) +- source_layout: `diffusers` +- local_tests_readme: `tests/local_tests/cosmos3/README.md` + +## Current Phase + +- phase: `prep complete -> Phase 1 reference study` (resuming Tier-A branch; repointing reference vllm-omni -> official diffusers) +- status: `in_progress` +- owner: `orchestrator` +- last_updated: `2026-06-06` + +## Component Matrix + +| Component | Type | Reuse/Port | Official Definition | Official Instantiation | FastVideo Target | Prototype | Conversion | Parity | Open Issues | +|---|---|---|---|---|---|---|---|---|---| +| transformer | dit | port | `diffusers_cosmos3/transformer.py:Cosmos3OmniTransformer` (model_type `qwen3_vl_text`, MoT + MRoPE) | `model_index.json: transformer`; `cosmos_framework/model/vfm/mot/cosmos3_vfm_network.py`, `omni_mot_model.py` | `fastvideo/models/dits/cosmos3.py` (branch: `Cosmos3VFMTransformer`+`Cosmos3LanguageModel` — reconcile to `Cosmos3OmniTransformer`) | skeleton | not_started | scaffold_skip | I001 | +| vae | vae | reuse | diffusers `AutoencoderKLWan` | `model_index.json: vae` | reuse Wan VAE (`fastvideo/models/vaes/`, cf. `cosmos25wanvae.py`) | not_started | passthrough? | not_started | Q001 | +| scheduler | generic | reuse | diffusers `UniPCMultistepScheduler` | `model_index.json: scheduler` | reuse | not_started | n/a | scaffold_skip (`test_cosmos3_scheduler_default_parity`) | - | +| text_tokenizer | tokenizer | reuse | transformers `Qwen2TokenizerFast` | `model_index.json: text_tokenizer` | reuse (tokenizer = allowed third-party) | not_started | passthrough | scaffold_skip (`test_cosmos3_tokenizer_chat_template`) | - | +| vision_encoder | encoder | port | transformers `Qwen3VLVisionModel` | `model_index.json: vision_encoder` | new encoder bucket OR documented lazy-wrapper | not_started | not_started | not_started | Q002 | +| sound_tokenizer | generic/vae | port | `Cosmos3AVAEAudioTokenizer` (model_type `autoencoder_v2`) | `model_index.json: sound_tokenizer` | new audio component | not_started | not_started | not_started | Q003 | + +## Conversion State + +- conversion_script: `scripts/checkpoint_conversion/cosmos3_convert.py` (branch has it, 246 lines, built vs vllm-omni — repoint/verify vs diffusers checkpoint) +- converted_weights_dir: `converted_weights/cosmos3` (n/a while needs_conversion=no) +- source_layout: `diffusers` +- needs_conversion: `no` (HF already diffusers-format; verify FastVideo loaders consume directly) +- strict_load_status: `not_run` +- passthrough_components: `vae (AutoencoderKLWan), scheduler (UniPC), text_tokenizer (Qwen2)` likely passthrough +- retry_history: `none` + +## Parity Commands + +| Scope | Command | Last Result | Notes | +|---|---|---|---| +| Tier-A scaffold | `PYTHONPATH= python -m pytest tests/local_tests/cosmos3/ -q` | `15 skipped` (2026-06-06) | skip: env's editable `fastvideo` points to main worktree (lacks cosmos3); see E001 | +| component | `pytest tests/local_tests//test_cosmos3__parity.py -v -s` | `not_run` | after env activation + native prototypes | +| pipeline | `pytest tests/local_tests/pipelines/test_cosmos3_pipeline_parity.py -v -s` | `not_run` | | + +## Open Questions + +| ID | Question | Owner | Needed By Phase | Status | Resolution | +|---|---|---|---|---|---| +| Q001 | Does Cosmos3 VAE (`AutoencoderKLWan`) match FastVideo's existing Wan VAE config/instantiation exactly (z_dim, scale factors, latents_mean/std)? | orchestrator | 3 (reuse gate) | open | | +| Q002 | `vision_encoder` (`Qwen3VLVisionModel`): native port vs documented lazy-wrapper exception? Needed for I2V/reasoning. | user/orchestrator | 3 | open | | +| Q003 | `sound_tokenizer` (`Cosmos3AVAEAudioTokenizer`) + audio output requires `WorkloadType` AV + audio regression metric. | user | 0/10 | open | full-omni scope chosen 2026-06-06; infra extensions pending | + +## Issues And Blockers + +| ID | Phase | Component | Severity | Issue | Evidence | Owner | Status | Resolution | +|---|---|---|---|---|---|---|---|---| +| I001 | port | transformer | high | Branch DiT (`Cosmos3VFMTransformer`+`Cosmos3LanguageModel`) built vs vllm-omni #3454; official checkpoint loads `Cosmos3OmniTransformer` (diffusers shim). Class/structure reconciliation required. | `model_index.json`; `diffusers_cosmos3/transformer.py`; branch commit `52bb65f49` | orchestrator | open | | +| I002 | all | tests | medium | Tier-A conftest+tests mirror vllm-omni line-by-line (stubs, `vllm_omni...guardrails`). Must be repointed to `diffusers_cosmos3` / official structures. | `tests/local_tests/cosmos3/conftest.py` | orchestrator | open | | + +## Escape Hatches + +| ID | Phase | Decision Type | Question | Recommended Option | Status | Resolution | +|---|---|---|---|---|---|---| +| E001 | prep | dependency/env | Shared `fv-main` env has `fastvideo` editable-installed from the MAIN worktree; the cosmos3 worktree's `fastvideo` is not importable (PEP660 finder overrides PYTHONPATH), so Tier-A tests skip. How to activate the worktree's `fastvideo` for verification without disrupting ~24 other worktrees sharing the env? | Dedicated conda env for the cosmos3 worktree | open | | + +## Decisions + +| Date | Decision | Rationale | Impact | +|---|---|---|---| +| 2026-06-06 | Reference source of truth = official diffusers (`Cosmos3OmniDiffusersPipeline` + `cosmos-framework`/`diffusers-cosmos3`), not vllm-omni #3454 | Official weights now public & diffusers-format; the artifact users actually load | Repoint DiT/pipeline/conversion/tests off vllm-omni (I001, I002) | +| 2026-06-06 | Resume in worktree `/home/william5lin/FastVideo_cosmos3_port`; weights+reference symlinked (no copy) | Preserve 2,492 lines of Tier-A work; avoid 33 GB duplication | Verification needs worktree `fastvideo` active (E001) | +| 2026-06-06 | Scope = full omni (video + audio + reasoning + action) | User choice (revised from branch's original video-only scope) | Adds `vision_encoder`, `sound_tokenizer` ports + `WorkloadType` AV + audio metric | +| 2026-06-06 | Downloaded full 34.9 GB (33 GiB) `nvidia/Cosmos3-Nano` | Unblocks May-22 `PENDING` weight status (HF was 401, now public) | Real parity now possible | + +## Handoff Notes + +- Prep (weights/reference/env editable installs) done in MAIN worktree; symlinked into this worktree. Env installs (`diffusers-cosmos3`, `cosmos-framework`) are in shared `fv-main`. +- Next: resolve E001 (env), then Phase 1 reference study of `diffusers_cosmos3` pipeline/transformer, then Phase 3 reuse gate (VAE/scheduler/tokenizer) + component dispatch (transformer, vision_encoder, sound_tokenizer). +- diffusers 0.36.0 imports the shim OK; checkpoint saved with 0.37.1 — watch `from_pretrained` needs (bump within FastVideo's `diffusers>=0.33.1` pin if required). diff --git a/tests/local_tests/cosmos3/README.md b/tests/local_tests/cosmos3/README.md index 95a38eecf..c4be239a2 100644 --- a/tests/local_tests/cosmos3/README.md +++ b/tests/local_tests/cosmos3/README.md @@ -2,38 +2,70 @@ ## Overview -Cosmos3 Phase-1 prep tracks a future FastVideo port from the vllm-omni Cosmos3 integration in PR #3454. The current reference supports a single `Cosmos3OmniDiffusersPipeline` for text-to-video (T2V), image-to-video (I2V), and text-to-image (T2I). Follow-up Cosmos3 capabilities mentioned in the PR body, such as sound generation and action-generation modes, are out of scope for the initial FastVideo port. +This workspace tracks the FastVideo Cosmos3 port. Live port state, component matrix, +decisions, and blockers live in `PORT_STATUS.md`. + +- **Reference (2026-06-06): official NVIDIA `cosmos-framework` diffusers backend** — + `Cosmos3OmniDiffusersPipeline` from the `diffusers-cosmos3` shim — loading the + now-public `nvidia/Cosmos3-Nano` checkpoint. +- **Scope: full omni** — T2V / I2V / T2I, audio (sound generation), VLM reasoning, + and action-conditioning. +- The original Tier-A scaffold was written against vllm-omni PR #3454 before official + weights were public; it is being repointed to the diffusers reference (see I001/I002 + in `PORT_STATUS.md`). ## Reference code -- Reference checkout: `/home/william5lin/cosmos3-reference` -- Source PR: -- Pinned HEAD: `8536f5b1421f78c7df06af6d96fa195c1ceb6384` -- Key files: - - `vllm_omni/deploy/cosmos3.yaml` - - `vllm_omni/diffusion/models/cosmos3/pipeline_cosmos3.py` - - `vllm_omni/diffusion/models/cosmos3/transformer_cosmos3.py` - - `vllm_omni/diffusion/models/cosmos3/guardrails.py` - - `tests/diffusion/models/cosmos3/test_cosmos3_pipeline.py` - - `tests/diffusion/models/cosmos3/test_cosmos3_transformer.py` +Primary (official): + +- Local: `cosmos-framework/` (symlink -> `/home/william5lin/FastVideo/cosmos-framework`, + commit `003d66d4`); GitHub +- diffusers shim `cosmos-framework/packages/diffusers-cosmos3/diffusers_cosmos3/`: + - `pipeline.py` — `Cosmos3OmniDiffusersPipeline` + - `transformer.py` — `Cosmos3OmniTransformer` + - `sequence_packing.py` +- framework model code: `cosmos_framework/model/vfm/mot/cosmos3_vfm_network.py`, + `cosmos_framework/model/vfm/omni_mot_model.py` +- Installed editable in shared `fv-main`: `diffusers-cosmos3`, `cosmos-framework` + (both `--no-deps`). + +Original Tier-A reference (superseded, kept for diffing during repoint): + +- vllm-omni PR #3454 , pinned + `8536f5b1`, checkout `/home/william5lin/cosmos3-reference`. +- The current `conftest.py` + tests still mirror this suite line-by-line. ## Weight status -PENDING. Do not download weights during Phase 1. +DOWNLOADED (2026-06-06). `nvidia/Cosmos3-Nano` is now public and diffusers-format +(the 2026-05-22 `401` is resolved). -- Candidate serving/model id from the PR body: `nvidia/Cosmos3-Nano` -- Hugging Face API status on 2026-05-22: `401` for `https://huggingface.co/api/models/nvidia/Cosmos3-Nano`; API body reports `Invalid username or password.` -- Hugging Face author search `author=nvidia&search=Cosmos3`: empty list (`[]`). -- NGC API URL requested by handoff returned a Next.js 404 HTML page, not model metadata. +- Local: `official_weights/cosmos3/` (symlink -> main worktree; 33 GiB, 67 files, + `model_index.json` present) +- Source: `nvidia/Cosmos3-Nano`, default revision; `source_layout=diffusers`, + `needs_conversion=no` +- `model_index` class: `Cosmos3OmniDiffusersPipeline` (diffusers 0.37.1) +- Token: not required (public repo) -## Parity-test placeholder +Components (from `model_index.json`): `transformer` (`Cosmos3OmniTransformer`), +`vae` (`AutoencoderKLWan`), `scheduler` (`UniPCMultistepScheduler`), +`text_tokenizer` (`Qwen2TokenizerFast`), `vision_encoder` (`Qwen3VLVisionModel`), +`sound_tokenizer` (`Cosmos3AVAEAudioTokenizer`). -Phase 2 should add local parity tests here after the official weights become accessible and FastVideo component prototypes exist. Suggested first targets: +## Running the Tier-A scaffold -1. Transformer state-dict key/shape inventory versus `Cosmos3VFMTransformer`. -2. Scheduler/default-parameter parity for T2V, I2V, and T2I request modes. -3. Prompt metadata-template parity for duration/resolution and image-vs-video modalities. +```bash +PYTHONPATH=/home/william5lin/FastVideo_cosmos3_port \ + python -m pytest tests/local_tests/cosmos3/ -q +``` + +NOTE: as of 2026-06-06 these report `15 skipped` because the shared `fv-main` env's +editable `fastvideo` resolves to the MAIN worktree (a PEP660 finder overrides +`PYTHONPATH`), so the worktree's cosmos3 modules are not importable. Tracked as E001 +in `PORT_STATUS.md`. ## SSIM placeholder -No SSIM references seeded yet. Add SSIM coverage only after a FastVideo inference path can load resolved Cosmos3 weights and generate stable T2V/T2I/I2V outputs. +No SSIM references seeded yet. Add SSIM coverage only after a FastVideo inference path +can load the Cosmos3 weights and generate stable T2V/I2V/T2I outputs. Audio quality +uses a separate metric (not SSIM); see `PORT_STATUS.md` Q003.