SolitaryThinker
6ca684539c
[misc]: simplify cosmos3 port (reuse embed, dedup prompt, hoist)
...
Behavior-preserving cleanup (95 tests green):
- reuse fastvideo.layers.visual_embedding.timestep_embedding in the DiT
- single-home COSMOS3_VIDEO_NEGATIVE_PROMPT in the lightweight presets
module; drop the unused duplicate from cosmos3_pipeline
- drop the write-only Cosmos3VisionSpec.clean_latent field + its kwarg
- hoist device = next(transformer.parameters()).device out of the per-pass
_run closure in cosmos3_get_cfg_velocity
2026-07-21 15:45:37 -07:00
SolitaryThinker
2f4191c88a
[bugfix] cosmos3: fix black-video UniPC config; wire E2E inference
...
The first real-weights Cosmos3-Nano T2V run produced an all-black video.
Root cause: the checkpoint's diffusers-style scheduler_config.json sets
use_karras_sigmas=true, and FastVideo's vendored UniPC checks karras
*before* use_flow_sigmas, so it built diffusion (beta) sigmas instead of
flow-matching sigmas -> scheduler.step diverged to NaN latents -> black
frames. The DiT/CFG velocity itself was clean.
The framework samples with FlowUniPCMultistepScheduler (pure flow:
shift + num_train_timesteps only). FastVideo's vendored UniPC is the same
algorithm, so the fix is config, not a port:
- Coerce the loaded scheduler to the flow setup in
Cosmos3OmniDiffusersPipeline.initialize_pipeline (kill karras/exp/beta
sigmas, force use_flow_sigmas + flow_prediction + final_sigmas_type=zero)
and rebuild the runtime scheduler from it.
- Use FastVideo's native UniPC (not diffusers) in the pipeline and tests,
honoring the "no diffusers at runtime" constraint.
Parity:
- Add test_cosmos3_scheduler_parity.py: FastVideo vendored UniPC (flow
config) vs the framework FlowUniPCMultistepScheduler. timesteps bit-exact,
sigmas within fp32 epsilon (~1e-8), full multi-step trajectory < ~1e-6
across shift in {10 (video), 3 (t2i)} and steps in {4,10,35}.
- Fix test_cosmos3_denoise_cfg_parity to use the framework scheduler as the
oracle (it previously compared diffusers-vs-diffusers, so the scheduler
was never actually checked against the framework).
Integration fixes to run end-to-end on real weights:
- registry: map checkpoint DiT class Cosmos3OmniTransformer ->
Cosmos3VFMTransformer.
- component_loader: route the "text_tokenizer" module to TokenizerLoader;
filter scheduler config to the class's __init__ params (robust to
diffusers schema drift like shift_terminal/sigma_min/sigma_max).
- DiT: materialize_non_persistent_buffers (recompute rotary_emb.inv_freq
after meta-device FSDP load) + cast vision latents / timestep embeds to
the compute dtype (no-ops in the fp32 parity tests).
- sequence_packing: move packed tensors to the model device in to_dit_kwargs.
- config: drop the unused identity text-preprocess (Cosmos3 has no text
encoder; the DiT tokenizes in the denoise stage).
Adds examples/inference/basic/basic_cosmos3_new_api.py (new VideoGenerator
API). Verified: 1280x704, 29 frames, 35 steps on B200 -> coherent video
matching the prompt (no NaNs, real pixel variance). Full cosmos3 parity
suite green.
2026-07-21 15:45:37 -07:00
SolitaryThinker
b8fbb30400
[feat] cosmos3: native DiT + framework parity (Cosmos3VFMTransformer)
2026-07-21 15:45:37 -07:00
SolitaryThinker
721bb8285b
[feat]: Cosmos3 DiT skeleton (Cosmos3VFMTransformer + Cosmos3LanguageModel)
...
Ports math (compute_mrope_position_ids_text/_vision, patchify, unpatchify) verbatim from vllm-omni transformer_cosmos3.py:113-177,1009-1036. Module tree (language_model.layers.*.self_attn.{q,k,v,o}_proj + gen_layers.*.cross_attention.* + vae2llm + llm2vae + time_embedder + norm_moe_gen) exposes the parameter names the checkpoint converter targets. All layer forward() raise NotImplementedError until weights publish; module instantiation + state-dict key tests pass.
2026-07-21 15:45:37 -07:00
Adriel Fung and SolitaryThinker
521dee0e82
[perf]: enable per-block torch.compile for LTX2 with persistent cache ( #1602 )
...
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com >
2026-07-20 15:59:13 -07:00
Junda Su
755a4e4470
[new-model] Add LingBot-Video Dense and MoE/refiner T2V inference ( #1595 )
2026-07-18 20:18:16 -07:00
Junda Su
d4bdd3621a
[new-model] Port LingBot-World-v2 ( #1579 )
2026-07-16 19:51:53 -07:00
6966f9e0bc
[fix] Z-Image ( #1236 ) draft port: rebase + strict-load contract + bf16 encoder parity + PORT_STATUS ( #1339 )
...
Co-authored-by: Mrinaal Dogra <mdogra@ucsd.edu >
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com >
2026-07-16 18:18:25 -07:00
Mac Lee
cae8fa18dc
[bugfix]: propagate Qwen2.5-VL visual dtype ( #1580 )
2026-07-13 18:37:11 -07:00
William Lin
ef15ea2391
[bugfix]: keep LTX2 rms_norm outputs bf16 under torch 2.12 autocast ( #1587 )
2026-07-13 16:04:21 -07:00
b063f8ca41
[feat] Fix FLUX.1-dev port: native RoPE, parity tests, SSIM reference ( #1321 )
...
Co-authored-by: Ishan Vaish <ivaish@ucsd.edu >
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com >
2026-07-13 01:49:42 -07:00
William Lin
e7fff0173a
[bugfix]: benchmark_weight_loading_comparison.py — iterate safe_open via .keys() ( #1378 )
2026-07-12 22:50:45 -07:00
Shreejith SG and H1yori233
d82abc271e
[feat] Add GLM-Image inference support ( #1030 )
...
Co-authored-by: H1yori233 <k1kong@ucsd.edu >
2026-07-12 22:42:42 -07:00
Guian Fang and SolitaryThinker
970409962f
[feat] Add AnyFlow any-step video distillation (pretrain + on-policy) ( #1371 )
...
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com >
2026-07-12 02:32:08 +00:00
Mingjia Huo and Claude Fable 5
4c08ffce49
[feat] World model training using third person games ( #1443 )
...
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
2026-07-10 05:09:07 +00:00
William Lin
8f47e85be0
[bugfix]: retry remote image downloads in load_image ( #1570 )
2026-07-09 07:33:24 -07:00
02e1143f22
[feat] Add Kandinsky-5 T2V/I2V pipeline support ( #1471 )
...
Co-authored-by: Aryan Kumar <aryan5v@users.noreply.github.com >
Co-authored-by: leffff <levnovitskiy@gmail.com >
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com >
2026-07-07 14:43:30 -07:00
Kaiqin Kong and SolitaryThinker
f037351146
[feat] Add Clean-history Teacher Forcing and Causal Consistency Distillation ( #1505 )
...
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com >
2026-07-07 08:13:11 +00:00
595f0ea60e
[feat] Add DreamX-World 5B Cam and AR pipelines ( #1538 )
...
Co-authored-by: Suckl <Suckl@users.noreply.github.com >
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com >
2026-07-07 06:16:18 +00:00
Mac Lee and SolitaryThinker
31aa115611
[bugfix]: preserve FSDP hooks for RMSNorm qk norms ( #1513 )
...
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com >
2026-07-05 04:15:49 +00:00
William Lin
b36d0ef085
[infra] Add DGX Spark and multi-architecture CUDA support ( #1447 )
2026-07-03 13:41:43 -07:00
William Lin
c4e108de78
[docs]: modernize documentation build ( #1503 )
2026-06-28 02:02:09 -07:00
Kaiqin Kong
16bf2eaf77
[feat] Relativistic RoPE re-indexing for long causal rollouts ( #1454 )
2026-06-28 01:43:00 -07:00
KyleNeverGivesUp and Claude Opus 4.8
3ccdec9798
[bugfix] Make enable_torch_compile_vae actually compile the Wan VAE ( #1498 )
...
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com >
2026-06-27 12:30:59 -07:00
Loay Rashid and SolitaryThinker
995a5fdf97
[bugfix] fixing denoising time in the fastwan script ( #1480 )
...
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com >
2026-06-23 10:01:23 -07:00
Kevin Lin
82ed9fe58d
[feat] QAD 5090: FP8 linear layer inference ( #1465 )
2026-06-22 18:25:06 -07:00
e60601df7f
[feat] QAD 5090: QAT training recipe — finetune + DMD distillation (12/12) ( #1462 )
...
Co-authored-by: William Lin <SolitaryThinker@users.noreply.github.com >
Co-authored-by: Loay Rashid <42599591+loaydatrain@users.noreply.github.com >
Co-authored-by: Kaiqin Kong <k1kong@ucsd.edu >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-06-17 15:18:52 -07:00
d922ab2cbc
[model] Flux2 Klein Port ( #1349 )
...
Co-authored-by: Gnav3852 <63612880+Gnav3852@users.noreply.github.com >
Co-authored-by: Mac Lee <macthecadillac@gmail.com >
2026-06-09 14:55:55 -07:00
2e35b0c6bd
[refactor]: linear/mlp FP4 path additions for Wan-2.1 (Attn-QAT 6/12) ( #1390 )
...
Co-authored-by: Peiyuan Zhang <a1286225768@gmail.com >
Co-authored-by: Matthew Noto <notomatthew31@gmail.com >
2026-06-08 14:51:57 -07:00
Shao Duan and SolitaryThinker
570607945c
[feat] dreamverse: sequence parallelism for serving ( #1424 )
...
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com >
2026-06-01 23:13:49 -07:00
Kaiqin Kong
3f24578139
[bugfix] LTX2: honor video_position_offset_sec in the DiT ( #1422 )
2026-06-01 16:59:15 -07:00
Kevin Lin
89fcf08378
[bugfix] Fix STFT dtype mismatch ( #1419 )
2026-05-31 20:56:07 -07:00
alexzms
84214c80bb
[feat] LTX-2.3 audio: BWE vocoder path ( #1398 )
2026-05-29 16:43:26 -07:00
alexzms and mergify[bot]
c2d7143c72
[feat] LTX-2.3 transformer support (config-gated extension of LTX-2) ( #1397 )
...
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-05-29 16:42:37 -07:00
Kaiqin Kong and SolitaryThinker
afdb6fbfa5
[feat] Add MatrixGame3.0 ( #1201 )
...
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com >
2026-05-27 21:07:01 +00:00
Junda Su
2c137931f3
[bugfix] Fix Dreamverse Modal compile warmup latency ( #1394 )
2026-05-26 17:58:38 -07:00
Wenxuan Tan and SolitaryThinker
af2ee9c78a
[feat] Optimize distributed weight loading in multi-node training ( #572 )
...
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com >
2026-05-21 16:59:48 +00:00
Mingjia Huo
63030cf6ec
[fix] Fix causal self-forcing attention settings ( #1355 )
2026-05-17 22:59:47 -07:00
Kaiqin Kong and mergify[bot]
773d44b875
[misc] Rename MatrixGame to MatrixGame2 ( #1357 )
...
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-05-17 22:57:03 -07:00
d2ffec5cce
[perf] Dreamverse 14/14: Add LTX2 profile speedups ( #1337 )
...
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com >
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com >
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com >
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com >
2026-05-15 18:10:47 -07:00
0958c344b8
[feat] Dreamverse 13/14: Activate LTX2 integration ( #1336 )
...
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com >
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com >
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com >
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com >
2026-05-15 15:35:05 -07:00
b4de5a9f1b
[feat] Dreamverse 12/14: Add LTX2 refine and upsampler support ( #1335 )
...
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com >
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com >
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com >
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com >
2026-05-15 12:21:40 -07:00
William Lin
424c643b6e
[feat]: MagiHuman pipeline stages (4/8) ( #1298 )
2026-05-12 14:59:21 -07:00
William Lin
de803cb250
[feat]: MagiHuman DiT (transformer) port + parity tests (3/8) ( #1297 )
2026-05-12 14:49:03 -07:00
William Lin
b1ddb1ba33
[feat]: T5-Gemma encoder for MagiHuman pipeline (2/8) ( #1296 )
2026-05-12 13:59:55 -07:00
Raghav K
9d0af28307
[feat] Add Cosmos 2.5 T2W training pipeline (LoRA + full fine-tune) ( #1227 )
2026-05-11 19:15:00 -07:00
d6dfe95466
[feat] FastVideo World Model Training ( #1179 )
...
Co-authored-by: mignonjia <mhuo@ucsd.edu >
Co-authored-by: alexzms <3036648523@qq.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-05-11 05:13:12 +00:00
f633e30ebb
[feat]: add LongCat bidirectional finetuning support ( #1244 )
...
Co-authored-by: Aryan Kumar <aryan5v@users.noreply.github.com >
Co-authored-by: alexzms <3036648523@qq.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-05-09 12:20:43 -07:00
William Lin
140bd1a6cf
[misc]: standardize install instructions on uv pip install ( #1279 )
2026-05-02 12:45:50 -07:00
William Lin
c85f6a477f
[docs] add hierarchical AGENTS.md per-directory guidance ( #1278 )
2026-05-02 03:28:18 -07:00
SolitaryThinker