nodes/hao-ai-lab-FastVideo
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

[misc]: cosmos3 resume: PORT_STATUS + README for official diffusers ref

Browse Source
This commit is contained in:
SolitaryThinker
2026-07-21 15:45:37 -07:00
parent 1abd7c95db
commit ee1521b41b
3 changed files with 143 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitignore
+5
View File
@@ -39,6 +39,11 @@ logs/
official_weights/
converted_weights/
# Cosmos3 local parity assets (symlinked from main worktree)
/official_weights/
/converted_weights/
/cosmos-framework
# SSIM test outputs
fastvideo/tests/ssim/generated_videos/
**/.cache/**
tests/local_tests/cosmos3/PORT_STATUS.md
+84
View File
@@ -0,0 +1,84 @@
# Cosmos3 Port Status
## Summary
- model_family: `cosmos3`
- workload_types: `T2V, I2V, T2I` supported by `WorkloadType` today; full-omni target also needs audio (AV), VLM reasoning, and action-conditioning, which require framework extensions (Q002, Q003).
- official_ref: `https://github.com/NVIDIA/cosmos-framework` — diffusers backend `diffusers_cosmos3.pipeline.Cosmos3OmniDiffusersPipeline`; HF `nvidia/Cosmos3-Nano`.
- official_ref_dir: `cosmos-framework` (symlink -> `/home/william5lin/FastVideo/cosmos-framework`, commit `003d66d4`)
- hf_weights_path: `nvidia/Cosmos3-Nano`
- local_weights_dir: `official_weights/cosmos3` (symlink -> `/home/william5lin/FastVideo/official_weights/cosmos3`, 33 GiB / 67 files)
- source_layout: `diffusers`
- local_tests_readme: `tests/local_tests/cosmos3/README.md`
## Current Phase
- phase: `prep complete -> Phase 1 reference study` (resuming Tier-A branch; repointing reference vllm-omni -> official diffusers)
- status: `in_progress`
- owner: `orchestrator`
- last_updated: `2026-06-06`
## Component Matrix
| Component | Type | Reuse/Port | Official Definition | Official Instantiation | FastVideo Target | Prototype | Conversion | Parity | Open Issues |
|---|---|---|---|---|---|---|---|---|---|
| transformer | dit | port | `diffusers_cosmos3/transformer.py:Cosmos3OmniTransformer` (model_type `qwen3_vl_text`, MoT + MRoPE) | `model_index.json: transformer`; `cosmos_framework/model/vfm/mot/cosmos3_vfm_network.py`, `omni_mot_model.py` | `fastvideo/models/dits/cosmos3.py` (branch: `Cosmos3VFMTransformer`+`Cosmos3LanguageModel` — reconcile to `Cosmos3OmniTransformer`) | skeleton | not_started | scaffold_skip | I001 |
| vae | vae | reuse | diffusers `AutoencoderKLWan` | `model_index.json: vae` | reuse Wan VAE (`fastvideo/models/vaes/`, cf. `cosmos25wanvae.py`) | not_started | passthrough? | not_started | Q001 |
| scheduler | generic | reuse | diffusers `UniPCMultistepScheduler` | `model_index.json: scheduler` | reuse | not_started | n/a | scaffold_skip (`test_cosmos3_scheduler_default_parity`) | - |
| text_tokenizer | tokenizer | reuse | transformers `Qwen2TokenizerFast` | `model_index.json: text_tokenizer` | reuse (tokenizer = allowed third-party) | not_started | passthrough | scaffold_skip (`test_cosmos3_tokenizer_chat_template`) | - |
| vision_encoder | encoder | port | transformers `Qwen3VLVisionModel` | `model_index.json: vision_encoder` | new encoder bucket OR documented lazy-wrapper | not_started | not_started | not_started | Q002 |
| sound_tokenizer | generic/vae | port | `Cosmos3AVAEAudioTokenizer` (model_type `autoencoder_v2`) | `model_index.json: sound_tokenizer` | new audio component | not_started | not_started | not_started | Q003 |
## Conversion State
- conversion_script: `scripts/checkpoint_conversion/cosmos3_convert.py` (branch has it, 246 lines, built vs vllm-omni — repoint/verify vs diffusers checkpoint)
- converted_weights_dir: `converted_weights/cosmos3` (n/a while needs_conversion=no)
- source_layout: `diffusers`
- needs_conversion: `no` (HF already diffusers-format; verify FastVideo loaders consume directly)
- strict_load_status: `not_run`
- passthrough_components: `vae (AutoencoderKLWan), scheduler (UniPC), text_tokenizer (Qwen2)` likely passthrough
- retry_history: `none`
## Parity Commands
| Scope | Command | Last Result | Notes |
|---|---|---|---|
| Tier-A scaffold | `PYTHONPATH=<worktree> python -m pytest tests/local_tests/cosmos3/ -q` | `15 skipped` (2026-06-06) | skip: env's editable `fastvideo` points to main worktree (lacks cosmos3); see E001 |
| component | `pytest tests/local_tests/<bucket>/test_cosmos3_<component>_parity.py -v -s` | `not_run` | after env activation + native prototypes |
| pipeline | `pytest tests/local_tests/pipelines/test_cosmos3_pipeline_parity.py -v -s` | `not_run` | |
## Open Questions
| ID | Question | Owner | Needed By Phase | Status | Resolution |
|---|---|---|---|---|---|
| Q001 | Does Cosmos3 VAE (`AutoencoderKLWan`) match FastVideo's existing Wan VAE config/instantiation exactly (z_dim, scale factors, latents_mean/std)? | orchestrator | 3 (reuse gate) | open | |
| Q002 | `vision_encoder` (`Qwen3VLVisionModel`): native port vs documented lazy-wrapper exception? Needed for I2V/reasoning. | user/orchestrator | 3 | open | |
| Q003 | `sound_tokenizer` (`Cosmos3AVAEAudioTokenizer`) + audio output requires `WorkloadType` AV + audio regression metric. | user | 0/10 | open | full-omni scope chosen 2026-06-06; infra extensions pending |
## Issues And Blockers
| ID | Phase | Component | Severity | Issue | Evidence | Owner | Status | Resolution |
|---|---|---|---|---|---|---|---|---|
| I001 | port | transformer | high | Branch DiT (`Cosmos3VFMTransformer`+`Cosmos3LanguageModel`) built vs vllm-omni #3454; official checkpoint loads `Cosmos3OmniTransformer` (diffusers shim). Class/structure reconciliation required. | `model_index.json`; `diffusers_cosmos3/transformer.py`; branch commit `52bb65f49` | orchestrator | open | |
| I002 | all | tests | medium | Tier-A conftest+tests mirror vllm-omni line-by-line (stubs, `vllm_omni...guardrails`). Must be repointed to `diffusers_cosmos3` / official structures. | `tests/local_tests/cosmos3/conftest.py` | orchestrator | open | |
## Escape Hatches
| ID | Phase | Decision Type | Question | Recommended Option | Status | Resolution |
|---|---|---|---|---|---|---|
| E001 | prep | dependency/env | Shared `fv-main` env has `fastvideo` editable-installed from the MAIN worktree; the cosmos3 worktree's `fastvideo` is not importable (PEP660 finder overrides PYTHONPATH), so Tier-A tests skip. How to activate the worktree's `fastvideo` for verification without disrupting ~24 other worktrees sharing the env? | Dedicated conda env for the cosmos3 worktree | open | |
## Decisions
| Date | Decision | Rationale | Impact |
|---|---|---|---|
| 2026-06-06 | Reference source of truth = official diffusers (`Cosmos3OmniDiffusersPipeline` + `cosmos-framework`/`diffusers-cosmos3`), not vllm-omni #3454 | Official weights now public & diffusers-format; the artifact users actually load | Repoint DiT/pipeline/conversion/tests off vllm-omni (I001, I002) |
| 2026-06-06 | Resume in worktree `/home/william5lin/FastVideo_cosmos3_port`; weights+reference symlinked (no copy) | Preserve 2,492 lines of Tier-A work; avoid 33 GB duplication | Verification needs worktree `fastvideo` active (E001) |
| 2026-06-06 | Scope = full omni (video + audio + reasoning + action) | User choice (revised from branch's original video-only scope) | Adds `vision_encoder`, `sound_tokenizer` ports + `WorkloadType` AV + audio metric |
| 2026-06-06 | Downloaded full 34.9 GB (33 GiB) `nvidia/Cosmos3-Nano` | Unblocks May-22 `PENDING` weight status (HF was 401, now public) | Real parity now possible |
## Handoff Notes
- Prep (weights/reference/env editable installs) done in MAIN worktree; symlinked into this worktree. Env installs (`diffusers-cosmos3`, `cosmos-framework`) are in shared `fv-main`.
- Next: resolve E001 (env), then Phase 1 reference study of `diffusers_cosmos3` pipeline/transformer, then Phase 3 reuse gate (VAE/scheduler/tokenizer) + component dispatch (transformer, vision_encoder, sound_tokenizer).
- diffusers 0.36.0 imports the shim OK; checkpoint saved with 0.37.1 — watch `from_pretrained` needs (bump within FastVideo's `diffusers>=0.33.1` pin if required).
tests/local_tests/cosmos3/README.md
+54 -22
View File
@@ -2,38 +2,70 @@
## Overview
Cosmos3 Phase-1 prep tracks a future FastVideo port from the vllm-omni Cosmos3 integration in PR #3454. The current reference supports a single `Cosmos3OmniDiffusersPipeline` for text-to-video (T2V), image-to-video (I2V), and text-to-image (T2I). Follow-up Cosmos3 capabilities mentioned in the PR body, such as sound generation and action-generation modes, are out of scope for the initial FastVideo port.
This workspace tracks the FastVideo Cosmos3 port. Live port state, component matrix,
decisions, and blockers live in `PORT_STATUS.md`.
- **Reference (2026-06-06): official NVIDIA `cosmos-framework` diffusers backend** —
`Cosmos3OmniDiffusersPipeline` from the `diffusers-cosmos3` shim — loading the
now-public `nvidia/Cosmos3-Nano` checkpoint.
- **Scope: full omni** — T2V / I2V / T2I, audio (sound generation), VLM reasoning,
and action-conditioning.
- The original Tier-A scaffold was written against vllm-omni PR #3454 before official
weights were public; it is being repointed to the diffusers reference (see I001/I002
in `PORT_STATUS.md`).
## Reference code
- Reference checkout: `/home/william5lin/cosmos3-reference`
- Source PR: <https://github.com/vllm-project/vllm-omni/pull/3454>
- Pinned HEAD: `8536f5b1421f78c7df06af6d96fa195c1ceb6384`
- Key files:
- `vllm_omni/deploy/cosmos3.yaml`
- `vllm_omni/diffusion/models/cosmos3/pipeline_cosmos3.py`
- `vllm_omni/diffusion/models/cosmos3/transformer_cosmos3.py`
- `vllm_omni/diffusion/models/cosmos3/guardrails.py`
- `tests/diffusion/models/cosmos3/test_cosmos3_pipeline.py`
- `tests/diffusion/models/cosmos3/test_cosmos3_transformer.py`
Primary (official):
- Local: `cosmos-framework/` (symlink -> `/home/william5lin/FastVideo/cosmos-framework`,
commit `003d66d4`); GitHub <https://github.com/NVIDIA/cosmos-framework>
- diffusers shim `cosmos-framework/packages/diffusers-cosmos3/diffusers_cosmos3/`:
- `pipeline.py` — `Cosmos3OmniDiffusersPipeline`
- `transformer.py` — `Cosmos3OmniTransformer`
- `sequence_packing.py`
- framework model code: `cosmos_framework/model/vfm/mot/cosmos3_vfm_network.py`,
`cosmos_framework/model/vfm/omni_mot_model.py`
- Installed editable in shared `fv-main`: `diffusers-cosmos3`, `cosmos-framework`
(both `--no-deps`).
Original Tier-A reference (superseded, kept for diffing during repoint):
- vllm-omni PR #3454 <https://github.com/vllm-project/vllm-omni/pull/3454>, pinned
`8536f5b1`, checkout `/home/william5lin/cosmos3-reference`.
- The current `conftest.py` + tests still mirror this suite line-by-line.
## Weight status
PENDING. Do not download weights during Phase 1.
DOWNLOADED (2026-06-06). `nvidia/Cosmos3-Nano` is now public and diffusers-format
(the 2026-05-22 `401` is resolved).
- Candidate serving/model id from the PR body: `nvidia/Cosmos3-Nano`
- Hugging Face API status on 2026-05-22: `401` for `https://huggingface.co/api/models/nvidia/Cosmos3-Nano`; API body reports `Invalid username or password.`
- Hugging Face author search `author=nvidia&search=Cosmos3`: empty list (`[]`).
- NGC API URL requested by handoff returned a Next.js 404 HTML page, not model metadata.
- Local: `official_weights/cosmos3/` (symlink -> main worktree; 33 GiB, 67 files,
`model_index.json` present)
- Source: `nvidia/Cosmos3-Nano`, default revision; `source_layout=diffusers`,
`needs_conversion=no`
- `model_index` class: `Cosmos3OmniDiffusersPipeline` (diffusers 0.37.1)
- Token: not required (public repo)
## Parity-test placeholder
Components (from `model_index.json`): `transformer` (`Cosmos3OmniTransformer`),
`vae` (`AutoencoderKLWan`), `scheduler` (`UniPCMultistepScheduler`),
`text_tokenizer` (`Qwen2TokenizerFast`), `vision_encoder` (`Qwen3VLVisionModel`),
`sound_tokenizer` (`Cosmos3AVAEAudioTokenizer`).
Phase 2 should add local parity tests here after the official weights become accessible and FastVideo component prototypes exist. Suggested first targets:
## Running the Tier-A scaffold
1. Transformer state-dict key/shape inventory versus `Cosmos3VFMTransformer`.
2. Scheduler/default-parameter parity for T2V, I2V, and T2I request modes.
3. Prompt metadata-template parity for duration/resolution and image-vs-video modalities.
```bash
PYTHONPATH=/home/william5lin/FastVideo_cosmos3_port \
python -m pytest tests/local_tests/cosmos3/ -q
```
NOTE: as of 2026-06-06 these report `15 skipped` because the shared `fv-main` env's
editable `fastvideo` resolves to the MAIN worktree (a PEP660 finder overrides
`PYTHONPATH`), so the worktree's cosmos3 modules are not importable. Tracked as E001
in `PORT_STATUS.md`.
## SSIM placeholder
No SSIM references seeded yet. Add SSIM coverage only after a FastVideo inference path can load resolved Cosmos3 weights and generate stable T2V/T2I/I2V outputs.
No SSIM references seeded yet. Add SSIM coverage only after a FastVideo inference path
can load the Cosmos3 weights and generate stable T2V/I2V/T2I outputs. Audio quality
uses a separate metric (not SSIM); see `PORT_STATUS.md` Q003.