diff --git a/.agents/skills/add-model-02-parity/SKILL.md b/.agents/skills/add-model-02-parity/SKILL.md index 5a96f34a5..8b4d5c3a3 100644 --- a/.agents/skills/add-model-02-parity/SKILL.md +++ b/.agents/skills/add-model-02-parity/SKILL.md @@ -158,6 +158,39 @@ Before `/add-model` handoff, each scaffolded test must be activated: [ ] Tolerance is justified for the component scope and kernel alignment. ``` +## Transformer CI Fingerprint Gate + +Official-vs-FastVideo parity establishes a transformer port. After that parity +passes, use a production-shaped one-layer forward as the normal package-level +CI regression gate when it is repeatably bit-exact on a pinned runner. + +- Instantiate the smallest native surface that covers every implementation and + config file exempted from media regression. Prefer a depth-one transformer + when top-level embedding, rotary, packing, normalization, and output code + must be covered; an isolated block is enough only when those paths are not + exempted. +- Materialize only the real checkpoint tensors used by that forward. Read the + checkpoint index and reuse an approved cache or fetch the minimum required + shards. A separate extracted fixture is optional, not a default requirement. +- Use fixed input tensor bytes and record their SHA-256. Do not rely only on a + random seed. +- Pin and assert the GPU model, PyTorch/CUDA build, dtype, attention backend, + TF32 policy, and deterministic-algorithm policy before comparing output. +- Hash the complete contiguous output bytes and hard-code the expected SHA-256 + together with shape and dtype. A mean, sum, or sampled element is not a + sufficient fingerprint. +- During initial seeding or an intentional reseed, run the corresponding + official and FastVideo layer on the same captured inputs and require exact + equality first. Normal CI then runs only FastVideo against the golden digest. +- Treat any golden change as a reviewed baseline update with fresh official + parity evidence; never update the digest merely to make CI green. + +This gate proves the selected one-layer forward is unchanged inside its pinned +execution envelope. Keep strict checkpoint loading and pipeline/media +regression gates for surfaces the forward does not exercise, such as registry +dispatch, external preprocessing, schedulers, CFG, VAE decode, and user-visible +output. + ## Component Parity Details Reference imports: diff --git a/.agents/skills/add-model-03-port-dit/SKILL.md b/.agents/skills/add-model-03-port-dit/SKILL.md index 028f6b346..c8e3cbfa2 100644 --- a/.agents/skills/add-model-03-port-dit/SKILL.md +++ b/.agents/skills/add-model-03-port-dit/SKILL.md @@ -110,5 +110,10 @@ choosing between incompatible official transformer definitions. ## Handoff +After real official parity passes, add the production-shaped one-layer CI +fingerprint described in `../add-model-02-parity/SKILL.md` when the pinned GPU +lane is repeatably bit-exact. This is the cheap regression gate; it does not +replace the official parity evidence used to seed it. + Return `../add-model/contracts/component_skill_handoff.md` following the common handoff rules in `../add-model/shared/component_skill_common.md`. diff --git a/tests/local_tests/cosmos3/PORT_STATUS.md b/tests/local_tests/cosmos3/PORT_STATUS.md index e85b33103..285af4bb0 100644 --- a/tests/local_tests/cosmos3/PORT_STATUS.md +++ b/tests/local_tests/cosmos3/PORT_STATUS.md @@ -1,152 +1,128 @@ # Cosmos3 Port Status -## Summary +## Current Scope -- model_family: `cosmos3` -- workload_types: `T2V, I2V, T2I` supported by `WorkloadType` today; full-omni target also needs audio (AV), VLM reasoning, and action-conditioning, which require framework extensions (Q002, Q003). -- official_ref: `https://github.com/NVIDIA/cosmos-framework` — diffusers backend `diffusers_cosmos3.pipeline.Cosmos3OmniDiffusersPipeline`; HF `nvidia/Cosmos3-Nano`. -- official_ref_dir: `cosmos-framework` (symlink -> `/home/william5lin/FastVideo/cosmos-framework`, commit `003d66d4`) -- hf_weights_path: `nvidia/Cosmos3-Nano` -- local_weights_dir: `official_weights/cosmos3` (symlink -> `/home/william5lin/FastVideo/official_weights/cosmos3`, 33 GiB / 67 files) -- source_layout: `diffusers` -- local_tests_readme: `tests/local_tests/cosmos3/README.md` +- Model family: `cosmos3` +- Registered checkpoint: `nvidia/Cosmos3-Nano` only +- Official parity oracle: +- Checkpoint layout: public diffusers-layout HF repository; no conversion or + FastVideo-owned weights repository is required +- Public registry workloads: T2V and I2V +- Additional implemented paths: T2I preset/example and opt-in T2VS generation +- Component-only paths: action conditioning and text/image reasoning have native + DiT/helper parity, but are not first-class public pipeline workloads +- Status: aggregate branch ready for review +- Last updated: 2026-07-21 -## Current Phase +The branch must not be described as full-omni end-to-end support. The native +model contains the sound, action, and reasoning surfaces needed to strict-load +and exercise the checkpoint, but public action/image-reasoning wiring and +generated-media/audio regression remain follow-ups. -- phase: `FULL OMNI SUPPORTED — every modality framework-parity verified bit-exact (suite 150 passed, 0 skipped). PR1 video core (T2V/I2V/T2I) + PR2 audio (t2vs) real-weights verified on B200; PR3 action (domain-aware) + PR4 reasoning (text + vision_encoder + deepstack reasoner) bit-exact. Branch chain: feat/cosmos3-tier-a-port (T2V) -> feat/cosmos3-i2v (I2V+T2I+flow_shift) -> feat/cosmos3-audio (t2vs) -> feat/cosmos3-action -> feat/cosmos3-reasoning. Optional follow-ups: real-weights action2world (needs robot-action data) + image-conditioned-reasoning prefill wiring (vision_encoder + get_rope_index, both proven).` -- status: `in_progress` -- owner: `orchestrator` -- last_updated: `2026-06-07` -- env: `fv-cosmos3` (conda clone of fv-main; `fastvideo` editable repointed to this worktree). Run tests from the worktree cwd with this env's python. -- branch: rebased onto `origin/main` @ `1c627a3f9` (was 33 behind, merge-base 2026-05-22); now 6 commits ahead; `fastvideo` imports clean; Tier-A `13 passed, 2 skipped`. +## Implementation Matrix -## Component Matrix +| Surface | Current support | Evidence / limitation | +|---|---|---| +| Video DiT | Native MoT implementation with production T2V/I2V pipeline | Framework parity and real-weight B200 runs recorded below | +| T2I | Pipeline dispatch, preset, and example | Not registered as a `WorkloadType` | +| Sound | Native DiT sound path plus lazy AVAE decode | Opt-in through `COSMOS3_T2VS`; no registered AV workload or audio-quality baseline | +| Action | Native domain-aware input/output path | Component parity only; pipeline does not build action specs | +| Reasoning | Native text/deepstack reasoner paths | Component parity only; public image-conditioned prefill wiring remains | +| VAE | Native Wan2.2 `AutoencoderKLWan` configuration | Framework parity recorded below | +| Scheduler | Native flow-configured UniPC | Framework timetable/sigma/trajectory parity recorded below | +| Tokenizer | Qwen2 tokenizer passthrough | Loaded from the checkpoint | +| Vision encoder | Qwen3-VL vision component parity | Not a required public pipeline module | +| Checkpoint loading | Direct diffusers-layout load | `cosmos3_convert.py` is a strict-load verifier, not a converter | -| Component | Type | Reuse/Port | Official Definition | Official Instantiation | FastVideo Target | Prototype | Conversion | Parity | Open Issues | -|---|---|---|---|---|---|---|---|---|---| -| transformer | dit | port | `diffusers_cosmos3/transformer.py:Cosmos3OmniTransformer` (model_type `qwen3_vl_text`, MoT + MRoPE) | `model_index.json: transformer`; `cosmos_framework/model/vfm/mot/cosmos3_vfm_network.py`, `omni_mot_model.py` | `fastvideo/models/dits/cosmos3.py` (branch: `Cosmos3VFMTransformer`+`Cosmos3LanguageModel` — reconcile to `Cosmos3OmniTransformer`) | skeleton | not_started | scaffold_skip | I001 | -| vae | vae | reuse | diffusers `AutoencoderKLWan` | `model_index.json: vae` | reuse Wan VAE (`fastvideo/models/vaes/`, cf. `cosmos25wanvae.py`) | not_started | passthrough? | not_started | Q001 | -| scheduler | generic | reuse (flow-coerced) | framework `FlowUniPCMultistepScheduler` (`cosmos_framework/.../fm_solvers_unipc.py`; checkpoint ships diffusers-style config) | `model_index.json: scheduler`; `cosmos_framework/.../samplers/unipc.py:UniPCSampler` | FastVideo-native `UniPCMultistepScheduler` (flow config), coerced in `initialize_pipeline` | done | n/a | framework-parity DONE (`test_cosmos3_scheduler_parity`: timesteps bit-exact, sigmas ~1e-8, trajectory <~1e-6) | I003 (resolved) | -| text_tokenizer | tokenizer | reuse | transformers `Qwen2TokenizerFast` | `model_index.json: text_tokenizer` | reuse (tokenizer = allowed third-party) | not_started | passthrough | scaffold_skip (`test_cosmos3_tokenizer_chat_template`) | - | -| vision_encoder | encoder | port | transformers `Qwen3VLVisionModel` | `model_index.json: vision_encoder` | new encoder bucket OR documented lazy-wrapper | not_started | not_started | not_started | Q002 | -| sound_tokenizer | generic/vae | port (decode) | framework AVAE `LatentAutoEncoderV2` (`avae_utils`); checkpoint is decoder-only AutoencoderOobleck-named w/ SnakeBeta | `model_index.json: sound_tokenizer` | reuse FastVideo native `OobleckVAE` decoder + `Cosmos3SoundVAE` wrapper (`models/audio/cosmos3_avae.py`) | done (decode) | n/a | DECODE bit-exact vs framework (`test_cosmos3_avae_parity`); real ckpt strict-loads | PR2 (branch feat/cosmos3-audio) | +## Verification -## Conversion State +| Scope | Result | Notes | +|---|---|---| +| Local framework component suite | `150 passed, 0 skipped` | Recorded on 2026-06-07 before the final aggregate rebase; this is not a fresh run against the fingerprint's newer official pin | +| Real-weight B200 inference | T2V, I2V, T2I, T2VS, text reasoning | Recorded on 2026-06-07; outputs were inspected and described as coherent/prompt-matching | +| Transformer fingerprint seed | `2 passed` | Modal L40S, 2026-07-21; T2V/T2VS/action decoder-layer outputs matched the pinned official framework layer bit-exact; config contract passed | +| Transformer fingerprint normal mode | `2 passed` | Modal L40S, 2026-07-21; T2V/T2VS/action/reasoning committed hashes passed without loading the official framework | +| Modal checkout unit tests | `12 passed` | Exact immutable Buildkite commit checkout and bounded retries | -- conversion_script: `scripts/checkpoint_conversion/cosmos3_convert.py` (branch has it, 246 lines, built vs vllm-omni — repoint/verify vs diffusers checkpoint) -- converted_weights_dir: `converted_weights/cosmos3` (n/a while needs_conversion=no) -- source_layout: `diffusers` -- needs_conversion: `no` (HF already diffusers-format; verify FastVideo loaders consume directly) -- strict_load_status: `not_run` -- passthrough_components: `vae (AutoencoderKLWan), scheduler (UniPC), text_tokenizer (Qwen2)` likely passthrough -- retry_history: `none` +Run the normal transformer fingerprint with: -## Parity Commands +```bash +CUBLAS_WORKSPACE_CONFIG=:4096:8 FASTVIDEO_FA4=0 \ + pytest fastvideo/tests/transformers/test_cosmos3_block_fingerprint.py -v -s +``` -| Scope | Command | Last Result | Notes | -|---|---|---|---| -| Tier-A scaffold | `cd && -m pytest tests/local_tests/cosmos3/ -q` | `13 passed, 2 skipped` (2026-06-06, post-rebase) | 2 skips: Cosmos3 tokenizer/_tokenize_prompt not yet wired on pipeline | -| component | `pytest tests/local_tests//test_cosmos3__parity.py -v -s` | `not_run` | after env activation + native prototypes | -| pipeline | `pytest tests/local_tests/pipelines/test_cosmos3_pipeline_parity.py -v -s` | `not_run` | | +The test requires the pinned Modal L40S execution envelope and a mounted cache +for `nvidia/Cosmos3-Nano`. -## Open Questions +## Transformer Fingerprint Baseline -| ID | Question | Owner | Needed By Phase | Status | Resolution | -|---|---|---|---|---|---| -| Q001 | Does Cosmos3 VAE (`AutoencoderKLWan`) match FastVideo's existing Wan VAE config/instantiation exactly (z_dim, scale factors, latents_mean/std)? | orchestrator | 3 (reuse gate) | open | | -| Q002 | `vision_encoder` (`Qwen3VLVisionModel`): native port vs documented lazy-wrapper exception? Needed for I2V/reasoning. | user/orchestrator | 3 | open | | -| Q003 | `sound_tokenizer` (`Cosmos3AVAEAudioTokenizer`) + audio output requires `WorkloadType` AV + audio regression metric. | user | 0/10 | open | full-omni scope chosen 2026-06-06; infra extensions pending | +- Checkpoint revision: `411f42a8fdfb8c5b2583cb8786e0938f49796eaa` +- Official framework revision: `ed8287fd7477113f8ac4f6b84290514d55cf0cdc` +- Container image digest: + `ghcr.io/hao-ai-lab/fastvideo/fastvideo-dev:py3.12-latest@sha256:5d0e99c518bb6ee06f5062f1e04e94846a3f327a67b3d78bb78e5e949b9f45f2` +- Hardware/software: NVIDIA L40S, PyTorch `2.12.0+cu126`, CUDA `12.6`, + BF16, MATH SDPA, TF32 disabled, deterministic algorithms enabled +- State loaded: 44 checkpoint tensors or slices, 408,486,912 parameters, + approximately 779.13 MiB in BF16 +- Environment SHA-256: + `ad3d22373224999f0a0520c96eb406335ad5b58bdf4fd2f3e4ec29de2f6e1dda` +- Config/structural contract SHA-256: + `5043ad8a281ddfd2595317c45943c2c06d71cf8012525db7efed1db7af7e31e3` +- Input SHA-256: + `4c2f1879c1847f65372fbb08df981bbb00e2e4b7ad6176c22db5515e69b2846f` +- Selected weights SHA-256: + `480b5a497bb0c9ef71edabeee62c057b2c54fce753362e8938d073d6bf7cfa62` +- T2V outputs SHA-256: + `2b15c03d936bace07acea7aba55fe850dd3cadc344b361dce8da68acd94f9cb7` +- T2VS outputs SHA-256: + `1bffb714b7f316665e0176ddfe2b930df70d6dae1f75b3f761e92b5b9d8c26dd` +- Action2world outputs SHA-256: + `4cf042b91e1a0c4131efbb454e23d679b73542abda9b36af18cd81bb694b1b34` +- Reasoning outputs SHA-256: + `112b9ecd765b0e1afc7d21d169b4f3e941901d975c27f59df6062dedef60bc35` -## Issues And Blockers +Normal CI runs separate depth-one T2V, T2VS, action2world, and deepstack-reason +forwards twice and checks exact output identity plus the committed hashes. +Seed/reseed mode additionally imports the pinned NVIDIA framework and compares +the captured generation decoder layer bit-for-bit. No extracted fixture or +separate HF upload is needed. -| ID | Phase | Component | Severity | Issue | Evidence | Owner | Status | Resolution | -|---|---|---|---|---|---|---|---|---| -| I001 | port | transformer | high | Branch DiT (`Cosmos3VFMTransformer`+`Cosmos3LanguageModel`) built vs vllm-omni #3454; official checkpoint loads `Cosmos3OmniTransformer` (diffusers shim). Class/structure reconciliation required. | `model_index.json`; `diffusers_cosmos3/transformer.py`; branch commit `52bb65f49` | orchestrator | resolved | DiT rewritten to checkpoint layout (single `layers` dual-pathway, BaseDiT-conformant); bit-identical framework parity (3d_rope + unified_3d_mrope), commits 59a4a571c/7c4633295 | -| I002 | all | tests | medium | Tier-A conftest+tests mirror vllm-omni line-by-line (stubs, `vllm_omni...guardrails`). Must be repointed to `diffusers_cosmos3` / official structures. | `tests/local_tests/cosmos3/conftest.py` | orchestrator | open | | -| I003 | inference | scheduler | high | First real-weights T2V was all-black: checkpoint `scheduler_config.json` sets `use_karras_sigmas=true`; vendored UniPC checks karras before `use_flow_sigmas` -> diffusion (beta) sigmas -> `scheduler.step` -> NaN latents. DiT/CFG velocity was clean. The scheduler had never been parity-tested vs the framework (`test_cosmos3_denoise_cfg_parity` used diffusers UniPC on both sides). | `result_latent` NaN at denoise step 0 (v_pred clean); ffprobe 3 KB black mp4 | orchestrator | resolved | Coerce loaded config to flow setup in `initialize_pipeline`; switch pipeline+tests to native UniPC (no diffusers at runtime); add `test_cosmos3_scheduler_parity` vs framework `FlowUniPCMultistepScheduler`; repoint denoise_cfg oracle to the framework scheduler. Commit 255311cf2 | +## Recorded Component Parity -## Escape Hatches +The following results were recorded from tiny CPU/FP32 models with framework +weights copied into the native implementations. They establish component +behavior, not current generated-media quality. -| ID | Phase | Decision Type | Question | Recommended Option | Status | Resolution | -|---|---|---|---|---|---|---| -| E001 | prep | dependency/env | Shared `fv-main` env has `fastvideo` editable-installed from the MAIN worktree; the cosmos3 worktree's `fastvideo` is not importable (PEP660 finder overrides PYTHONPATH), so Tier-A tests skip. How to activate the worktree's `fastvideo` for verification without disrupting ~24 other worktrees sharing the env? | Dedicated conda env for the cosmos3 worktree | resolved | Created fv-cosmos3 (clone of fv-main); repointed fastvideo editable to worktree; run from worktree cwd. Branch also rebased onto origin/main to fix stale import. | +| Component / pipeline | Test | Recorded result | +|---|---|---| +| Scheduler (UniPC flow) | `test_cosmos3_scheduler_parity` | timesteps exact; sigmas about `1e-8`; trajectory below about `1e-6` | +| DiT video path | `test_cosmos3_dit_parity_mrope` | exact | +| Sequence packing | `test_cosmos3_packing_parity` | exact | +| VAE (Wan2.2) | `test_cosmos3_vae_parity` | exact | +| Denoise / CFG velocity | `test_cosmos3_denoise_cfg_parity` | max below `1e-6` | +| Resolution flow shift | `test_cosmos3_flow_shift_parity` | exact | +| I2V conditioning | `test_cosmos3_i2v_conditioning_parity` | exact | +| AVAE decoder | `test_cosmos3_avae_parity` | exact | +| Sound DiT / packing / CFG | `test_cosmos3_sound_parity` | exact | +| Action DiT / packing / CFG | `test_cosmos3_action_parity` | exact | +| Text/deepstack reasoning | `test_cosmos3_reasoning_parity` | exact logits/forward and token-exact greedy generation | +| Vision encoder | `test_cosmos3_vision_encoder_parity` | exact | -## Decisions +## Remaining Gaps -| Date | Decision | Rationale | Impact | -|---|---|---|---| -| 2026-06-06 | Reference source of truth = official diffusers (`Cosmos3OmniDiffusersPipeline` + `cosmos-framework`/`diffusers-cosmos3`), not vllm-omni #3454 | Official weights now public & diffusers-format; the artifact users actually load | Repoint DiT/pipeline/conversion/tests off vllm-omni (I001, I002) | -| 2026-06-06 | Resume in worktree `/home/william5lin/FastVideo_cosmos3_port`; weights+reference symlinked (no copy) | Preserve 2,492 lines of Tier-A work; avoid 33 GB duplication | Verification needs worktree `fastvideo` active (E001) | -| 2026-06-06 | Scope = full omni (video + audio + reasoning + action) | User choice (revised from branch's original video-only scope) | Adds `vision_encoder`, `sound_tokenizer` ports + `WorkloadType` AV + audio metric | -| 2026-06-06 | Downloaded full 34.9 GB (33 GiB) `nvidia/Cosmos3-Nano` | Unblocks May-22 `PENDING` weight status (HF was 401, now public) | Real parity now possible | -| 2026-06-06 | Rebased branch onto origin/main (33 commits); resolved registry.py conflict by reconstructing from main + cosmos3 import/entry | Branch was stale; fastvideo failed to import (main removed MatrixGameI2V480PConfig) | Branch imports clean; Tier-A 13 passed/2 skipped | -| 2026-06-06 | Reference = cosmos_framework ONLY (full omni); diffusers shim dropped even for video | User directive (Phase 1 found diffusers __call__ is video-only; sound/action/reasoning live only in the framework) | Larger port; ref DiT = `Cosmos3VFMNetwork`/`Cosmos3VFMNetworkConfig` (not diffusers `Cosmos3OmniTransformer`); core model imports in fv-cosmos3 with light deps; TE only in optional dot_product_attention | +- Add Cosmos3 generated-media SSIM references once the desired public + T2V/I2V/T2I regression cases are selected. +- Add an audio-quality regression metric for T2VS; SSIM cannot cover audio. +- Decide whether to register T2I and an AV workload as public workload types. +- Wire action conditioning and image-conditioned reasoning into first-class + public pipeline requests if those are in product scope. +- Only Cosmos3-Nano is registered and tested; no other Cosmos3 size is claimed. -## Handoff Notes +## CI Decision -- Prep (weights/reference/env editable installs) done in MAIN worktree; symlinked into this worktree. Env installs (`diffusers-cosmos3`, `cosmos-framework`) are in shared `fv-main`. -- Next: resolve E001 (env), then Phase 1 reference study of `diffusers_cosmos3` pipeline/transformer, then Phase 3 reuse gate (VAE/scheduler/tokenizer) + component dispatch (transformer, vision_encoder, sound_tokenizer). -- diffusers 0.36.0 imports the shim OK; checkpoint saved with 0.37.1 — watch `from_pretrained` needs (bump within FastVideo's `diffusers>=0.33.1` pin if required). - -### PR1 (video core) progress — 2026-06-06 -- Arch config 1:1 with checkpoint, committed `9567efdf0`. -- Framework parity-reference harness committed `dd97efda3`: `tests/local_tests/cosmos3/test_cosmos3_reference_forward.py` builds a tiny `Cosmos3VFMNetwork` on CPU/float32 (SDPA monkeypatch; flash2/3/natten are CUDA-only) and forwards `packed_seq -> {last_hidden_state, preds_vision}`. 23 tests pass in fv-cosmos3. This is the ground-truth side for DiT parity. Run: `cd && -m pytest tests/local_tests/cosmos3/test_cosmos3_reference_forward.py -q`. -- THREE naming conventions to bridge: - 1. framework-native (`Cosmos3VFMNetwork`): `language_model.model.layers.{i}.self_attn.{q,k,v,o}_proj(+ _moe_gen)`, `{q,k}_norm(+_moe_gen)`, `mlp(+_moe_gen)`, `vae2llm`/`llm2vae`, `time_embedder.mlp.{0,2}`. - 2. diffusers checkpoint (on disk, what we load): `layers.{i}.self_attn.{to_q,to_k,to_v,to_out}` + `{add_q,add_k,add_v}_proj`/`to_add_out`, `{norm_q,norm_k,norm_added_q,norm_added_k}`, `mlp`/`mlp_moe_gen`, `proj_in`/`proj_out`, `time_embedder.linear_{1,2}`. - 3. FastVideo DiT (our choice). Conversion maps (2)->(3); the DiT parity test copies (1)->(3). -- BaseDiT signature is `__init__(self, config: DiTConfig, hf_config: dict)`; the branch `Cosmos3VFMTransformer` uses `fastvideo_args`/SimpleNamespace and does NOT conform — rewrite to conform + match the checkpoint key surface (single `layers` dual-pathway, not split language_model/gen_layers). -- Native layers (per cosmos2_5): `ReplicatedLinear`/`MLP`/`RMSNorm` (fastvideo.layers.*), `LocalAttention`/`DistributedAttention` (fastvideo.attention), `apply_rotary_emb` (use_real_unbind_dim=-2 for Cosmos). EntryClass at module bottom; class attrs bound from config; 3D-MRoPE has no reusable util — adapt Cosmos25RotaryPosEmbed. -- NEXT: write native `fastvideo/models/dits/cosmos3.py` + fastvideo-vs-framework forward parity test (copy framework weights into the FastVideo DiT, compare outputs), then conversion script (diffusers checkpoint -> FastVideo) + strict-load, then video pipeline/packing. - -### PR1 (video core) acceptance — real-weights E2E — 2026-06-07 -- First real-weights T2V (`examples/inference/basic/basic_cosmos3_new_api.py`, `COSMOS3_MODEL_PATH=official_weights/cosmos3`) ran mechanically but produced an all-black 3 KB mp4. Instrumenting the denoise loop showed `v_pred` clean at step 0 but `scheduler.step` -> NaN. Root cause I003: checkpoint `scheduler_config.json` is diffusers-style (`use_karras_sigmas=true`), and the vendored UniPC checks karras before `use_flow_sigmas` -> diffusion (beta) sigmas instead of flow sigmas -> NaN. The framework actually samples with `FlowUniPCMultistepScheduler` (pure flow: `shift` + `num_train_timesteps`). -- Fix (commit `255311cf2`): coerce the loaded scheduler to the flow setup in `Cosmos3OmniDiffusersPipeline.initialize_pipeline`; use FastVideo's native UniPC (not diffusers) in pipeline + tests. Added `test_cosmos3_scheduler_parity.py` (native UniPC flow-config vs framework `FlowUniPCMultistepScheduler`: timesteps bit-exact, sigmas ~1e-8, full trajectory <~1e-6 over shift in {10,3}, steps in {4,10,35}). Repointed `test_cosmos3_denoise_cfg_parity` oracle to the framework scheduler (it previously compared diffusers-vs-diffusers, so the scheduler was never checked against the framework). -- Also wired the remaining integration glue (registry alias `Cosmos3OmniTransformer`->`Cosmos3VFMTransformer`; `text_tokenizer`->TokenizerLoader; scheduler config param-filtering; DiT `materialize_non_persistent_buffers` + compute-dtype casts; packing device-move in `to_dit_kwargs`; empty text-preprocess). -- Verified: 1280x704, 29 frames, 35 steps on a single B200 -> coherent golden-retriever-in-meadow video matching the prompt (no NaNs; per-frame pixel std ~58; visible temporal motion). Full cosmos3 suite: 95 passed, 0 skipped. -- NEXT: PR2 audio (`sound_tokenizer` AVAE) / PR3 action / PR4 reasoning. Optional: I2V/T2I real-weights spot-checks; force-push branch (needs explicit OK). - -### PR1 (video core) — I2V real-weights — 2026-06-07 (branch feat/cosmos3-i2v) -- Forked `feat/cosmos3-i2v` off `feat/cosmos3-tier-a-port` (stacked, includes the T2V + scheduler fix). -- Studied the framework I2V path: `cosmos_framework.inference.vision.load_conditioning_image` (aspect-preserving resize + center crop + uint8 quantize -> `/127.5-1`) + `build_conditioned_video_batch` (frame 0 = image, remaining frames REPEAT the last conditioning frame -> static video), then VAE-encode; `condition_frame_indexes=[0]` (latent). Condition frames kept clean during sampling exactly as FastVideo already does: init noise `cond_mask*x0 + (1-cond_mask)*noise` (`omni_mot_model._prepare_inference_data`) + velocity zeroed `pred*(1-cond_mask)` each step (`_get_velocity`), no re-injection. -- Bug found + fixed (commit `bd8d604fb`): FastVideo's `_image_to_video_tensor` ZERO-filled the non-condition frames; the temporal Wan VAE (4x) makes latent frame 0 depend on several pixel frames, so zero-fill -> wrong conditioning latent. Rewrote it to repeat-fill + framework resize/crop/quantize. -- Parity: `test_cosmos3_i2v_conditioning_parity.py` vs framework `load_conditioning_image` + repeat-fill — bit-exact (max abs diff 0.0) across aspect/size/frame cases. Existing `test_cosmos3_denoise_cfg_parity` already covers the I2V cond-mask + velocity math (i2v case). -- Example: `examples/inference/basic/basic_cosmos3_i2v_new_api.py` (`InputConfig(image_path=...)`, default `assets/images/cyclist.jpg`). -- Verified on B200 (1280x704, 29f, 35 steps, real weights): output frame 0 reproduces the conditioning cyclist image; later frames show coherent forward motion down the trail following the prompt. Full suite 98 passed, 0 skipped. -- NEXT: optional T2I real-weights spot-check; then PR2 audio / PR3 action / PR4 reasoning. - -### PR1 (video core) — T2I real-weights + resolution-based flow_shift — 2026-06-07 (branch feat/cosmos3-i2v) -- Studied framework T2I: tokenization uses `vlm_config.use_system_prompt` which is `false` in the checkpoint (config.json:199) — matches FastVideo's hardcoded `use_system_prompt=False` for all modes (no divergence). Canonical T2I is 960x960 (inputs/omni/t2i.json), single-frame (num_frames=1). -- Bug found + fixed (commit `604dc2637`): the stage chose `flow_shift` by task (`3.0 if is_t2i else 10.0`), but the framework picks it purely by the named resolution bucket (`OmniSampleArgs._RESOLUTION_SHIFT_DEFAULTS`, 8B backbone: 256->3.0, 480->5.0, 720/768->10.0; model default resolution "720"). Task-based only matched T2V@720 / T2I@256 by luck; canonical T2I@960x960 is the "720" bucket -> 10.0, so `is_t2i->3.0` was wrong. Replaced with `_flow_shift_for_resolution(h,w)` (longest-side bucketing), applied to all tasks. -- Parity: `test_cosmos3_flow_shift_parity.py` checks the mapping vs framework `{VIDEO,IMAGE}_RES_SIZE_INFO` x `_RESOLUTION_SHIFT_DEFAULTS` (8B rows, 20 cases). Also hardened `_image_to_video_tensor` tensor branch to respect the [-1,1] convention (PIL path stays framework-exact). -- Example: `examples/inference/basic/basic_cosmos3_t2i_new_api.py` (num_frames=1, 960x960). -- Verified on B200 (real weights, 35 steps): coherent red-panda image matching the prompt, flow_shift=10.0. Full suite 118 passed, 0 skipped. -- Video core (T2V/I2V/T2I) is now complete and real-weights verified. NEXT: PR2 audio (sound_tokenizer AVAE + audio output) on a new stacked branch. - -## Full-omni parity summary (every component, max / mean abs diff vs framework) - -All run on CPU / float32 (tiny models, framework weights copied in; framework = -oracle). `tests/local_tests/cosmos3/`, suite: 150 passed, 0 skipped. - -| Component / pipeline | Test | max | mean | -|---|---|---|---| -| Scheduler (UniPC flow) | test_cosmos3_scheduler_parity | timesteps 0; sigmas ~1e-8; traj <~1e-6 | ~1e-7 | -| DiT (video, unified_3d_mrope) | test_cosmos3_dit_parity_mrope | 0.0 | 0.0 | -| Sequence packing (video) | test_cosmos3_packing_parity | 0.0 (exact) | 0.0 | -| VAE (Wan2.2) | test_cosmos3_vae_parity | 0.0 | 0.0 | -| Denoise / CFG velocity | test_cosmos3_denoise_cfg_parity | <1e-6 | <1e-7 | -| flow_shift (resolution) | test_cosmos3_flow_shift_parity | exact | exact | -| I2V conditioning (static-repeat) | test_cosmos3_i2v_conditioning_parity | 0.0 | 0.0 | -| AVAE sound decoder | test_cosmos3_avae_parity | 0.0 | 0.0 | -| DiT sound pathway + packing | test_cosmos3_sound_parity | 0.0 | 0.0 | -| t2vs CFG velocity | test_cosmos3_sound_parity | 0.0 | 0.0 | -| DiT action pathway + packing | test_cosmos3_action_parity | 0.0 | 0.0 | -| action CFG velocity | test_cosmos3_action_parity | 0.0 | 0.0 | -| Reasoner prefill logits (text) | test_cosmos3_reasoning_parity | 0.0 | 0.0 | -| Reasoner greedy generation | test_cosmos3_reasoning_parity | token-exact | - | -| Deepstack reasoner forward | test_cosmos3_reasoning_parity | 0.0 | 0.0 | -| vision_encoder (Qwen3-VL ViT) | test_cosmos3_vision_encoder_parity | 0.0 | 0.0 | - -Real-weights pipelines verified on B200 (`examples/inference/basic/basic_cosmos3*_new_api.py`): -T2V (1280x704), I2V (cyclist), T2I (960x960 red panda), t2vs (ocean + stereo -48kHz audio), text reasoning (greedy == framework). All coherent / prompt-matching. +Routine changes limited to `fastvideo/models/dits/cosmos3.py` and its DiT config +use the exact one-layer fingerprint instead of scheduling the full SSIM lane. +Changes to the fingerprint baseline itself, pipelines, sequence packing, +schedulers, encoders, VAEs, shared layers, dependencies, or the container still +schedule SSIM. The full end-to-end SSIM test is retained; it simply does not +need to run on every covered Cosmos3 DiT-only PR. diff --git a/tests/local_tests/cosmos3/README.md b/tests/local_tests/cosmos3/README.md index c4be239a2..f89e10444 100644 --- a/tests/local_tests/cosmos3/README.md +++ b/tests/local_tests/cosmos3/README.md @@ -5,35 +5,31 @@ This workspace tracks the FastVideo Cosmos3 port. Live port state, component matrix, decisions, and blockers live in `PORT_STATUS.md`. -- **Reference (2026-06-06): official NVIDIA `cosmos-framework` diffusers backend** — - `Cosmos3OmniDiffusersPipeline` from the `diffusers-cosmos3` shim — loading the - now-public `nvidia/Cosmos3-Nano` checkpoint. -- **Scope: full omni** — T2V / I2V / T2I, audio (sound generation), VLM reasoning, - and action-conditioning. +- **Reference: official NVIDIA `cosmos-framework`** — the framework is the parity + oracle for every modality; the public `nvidia/Cosmos3-Nano` repository supplies + the diffusers-layout checkpoint only. +- **Public scope:** registered T2V/I2V, plus a T2I path/preset and opt-in T2VS. + Action and reasoning are native component-parity surfaces, not first-class + public pipeline workloads. - The original Tier-A scaffold was written against vllm-omni PR #3454 before official - weights were public; it is being repointed to the diffusers reference (see I001/I002 - in `PORT_STATUS.md`). + weights were public; it was superseded by framework-based parity coverage. ## Reference code Primary (official): -- Local: `cosmos-framework/` (symlink -> `/home/william5lin/FastVideo/cosmos-framework`, - commit `003d66d4`); GitHub -- diffusers shim `cosmos-framework/packages/diffusers-cosmos3/diffusers_cosmos3/`: - - `pipeline.py` — `Cosmos3OmniDiffusersPipeline` - - `transformer.py` — `Cosmos3OmniTransformer` - - `sequence_packing.py` +- GitHub: +- The transformer fingerprint seed pins framework revision + `ed8287fd7477113f8ac4f6b84290514d55cf0cdc`. - framework model code: `cosmos_framework/model/vfm/mot/cosmos3_vfm_network.py`, `cosmos_framework/model/vfm/omni_mot_model.py` -- Installed editable in shared `fv-main`: `diffusers-cosmos3`, `cosmos-framework` - (both `--no-deps`). Original Tier-A reference (superseded, kept for diffing during repoint): - vllm-omni PR #3454 , pinned `8536f5b1`, checkout `/home/william5lin/cosmos3-reference`. -- The current `conftest.py` + tests still mirror this suite line-by-line. +- The remaining reference-specific helpers are retained only for local parity + tests; production code does not import that project. ## Weight status @@ -55,17 +51,30 @@ Components (from `model_index.json`): `transformer` (`Cosmos3OmniTransformer`), ## Running the Tier-A scaffold ```bash -PYTHONPATH=/home/william5lin/FastVideo_cosmos3_port \ - python -m pytest tests/local_tests/cosmos3/ -q +python -m pytest tests/local_tests/cosmos3/ -q ``` -NOTE: as of 2026-06-06 these report `15 skipped` because the shared `fv-main` env's -editable `fastvideo` resolves to the MAIN worktree (a PEP660 finder overrides -`PYTHONPATH`), so the worktree's cosmos3 modules are not importable. Tracked as E001 -in `PORT_STATUS.md`. +The recorded component-parity suite result is `150 passed, 0 skipped`; see +`PORT_STATUS.md` for the date, scope, and limitations of that evidence. ## SSIM placeholder -No SSIM references seeded yet. Add SSIM coverage only after a FastVideo inference path -can load the Cosmos3 weights and generate stable T2V/I2V/T2I outputs. Audio quality -uses a separate metric (not SSIM); see `PORT_STATUS.md` Q003. +No Cosmos3 SSIM references are seeded yet. Select and seed public T2V/I2V/T2I +cases before treating generated-media quality as CI-covered. Audio quality needs +a separate metric because SSIM cannot cover it. + +## Transformer CI fingerprint + +The routine transformer gate runs separate depth-one FastVideo T2V, T2VS, +action2world, and deepstack-reasoning forwards with production hidden, head, +MLP, latent, sound, and action dimensions. It reads 44 tensors or slices +(408,486,912 parameters, about 779.13 MiB in BF16) from the pinned +`nvidia/Cosmos3-Nano` checkpoint cache. It hashes the production/fingerprint +config contract, fixed inputs, selected weights, and every named output. + +The golden was seeded and verified on Modal L40S using the pinned CI image. Seed +mode also proved the captured FastVideo decoder-layer outputs for all three +generation cases bit-exact against the pinned NVIDIA framework layer. Normal +mode reran every case twice from the committed hashes and passed. No extracted +fixture or separate weights repository is required; SSIM remains the +end-to-end media gate for uncovered surfaces.