SolitaryThinker
455cd83bed
[bugfix] cosmos3: fix black-video UniPC config; wire E2E inference
...
The first real-weights Cosmos3-Nano T2V run produced an all-black video.
Root cause: the checkpoint's diffusers-style scheduler_config.json sets
use_karras_sigmas=true, and FastVideo's vendored UniPC checks karras
*before* use_flow_sigmas, so it built diffusion (beta) sigmas instead of
flow-matching sigmas -> scheduler.step diverged to NaN latents -> black
frames. The DiT/CFG velocity itself was clean.
The framework samples with FlowUniPCMultistepScheduler (pure flow:
shift + num_train_timesteps only). FastVideo's vendored UniPC is the same
algorithm, so the fix is config, not a port:
- Coerce the loaded scheduler to the flow setup in
Cosmos3OmniDiffusersPipeline.initialize_pipeline (kill karras/exp/beta
sigmas, force use_flow_sigmas + flow_prediction + final_sigmas_type=zero)
and rebuild the runtime scheduler from it.
- Use FastVideo's native UniPC (not diffusers) in the pipeline and tests,
honoring the "no diffusers at runtime" constraint.
Parity:
- Add test_cosmos3_scheduler_parity.py: FastVideo vendored UniPC (flow
config) vs the framework FlowUniPCMultistepScheduler. timesteps bit-exact,
sigmas within fp32 epsilon (~1e-8), full multi-step trajectory < ~1e-6
across shift in {10 (video), 3 (t2i)} and steps in {4,10,35}.
- Fix test_cosmos3_denoise_cfg_parity to use the framework scheduler as the
oracle (it previously compared diffusers-vs-diffusers, so the scheduler
was never actually checked against the framework).
Integration fixes to run end-to-end on real weights:
- registry: map checkpoint DiT class Cosmos3OmniTransformer ->
Cosmos3VFMTransformer.
- component_loader: route the "text_tokenizer" module to TokenizerLoader;
filter scheduler config to the class's __init__ params (robust to
diffusers schema drift like shift_terminal/sigma_min/sigma_max).
- DiT: materialize_non_persistent_buffers (recompute rotary_emb.inv_freq
after meta-device FSDP load) + cast vision latents / timestep embeds to
the compute dtype (no-ops in the fp32 parity tests).
- sequence_packing: move packed tensors to the model device in to_dit_kwargs.
- config: drop the unused identity text-preprocess (Cosmos3 has no text
encoder; the DiT tokenizes in the denoise stage).
Adds examples/inference/basic/basic_cosmos3_new_api.py (new VideoGenerator
API). Verified: 1280x704, 29 frames, 35 steps on B200 -> coherent video
matching the prompt (no NaNs, real pixel variance). Full cosmos3 parity
suite green.
2026-07-20 12:19:45 -07:00
Junda Su
755a4e4470
[new-model] Add LingBot-Video Dense and MoE/refiner T2V inference ( #1595 )
2026-07-18 20:18:16 -07:00
Junda Su
d4bdd3621a
[new-model] Port LingBot-World-v2 ( #1579 )
2026-07-16 19:51:53 -07:00
6966f9e0bc
[fix] Z-Image ( #1236 ) draft port: rebase + strict-load contract + bf16 encoder parity + PORT_STATUS ( #1339 )
...
Co-authored-by: Mrinaal Dogra <mdogra@ucsd.edu >
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com >
2026-07-16 18:18:25 -07:00
b063f8ca41
[feat] Fix FLUX.1-dev port: native RoPE, parity tests, SSIM reference ( #1321 )
...
Co-authored-by: Ishan Vaish <ivaish@ucsd.edu >
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com >
2026-07-13 01:49:42 -07:00
Shreejith SG and H1yori233
d82abc271e
[feat] Add GLM-Image inference support ( #1030 )
...
Co-authored-by: H1yori233 <k1kong@ucsd.edu >
2026-07-12 22:42:42 -07:00
alexzms and SolitaryThinker
c096fda1eb
[docs] Add LTX-2.3 distilled inference run configs (t2v/i2v × 5+2/8+3 × resolutions) ( #1568 )
...
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com >
2026-07-09 18:53:12 +00:00
02e1143f22
[feat] Add Kandinsky-5 T2V/I2V pipeline support ( #1471 )
...
Co-authored-by: Aryan Kumar <aryan5v@users.noreply.github.com >
Co-authored-by: leffff <levnovitskiy@gmail.com >
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com >
2026-07-07 14:43:30 -07:00
595f0ea60e
[feat] Add DreamX-World 5B Cam and AR pipelines ( #1538 )
...
Co-authored-by: Suckl <Suckl@users.noreply.github.com >
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com >
2026-07-07 06:16:18 +00:00
William Lin
b36d0ef085
[infra] Add DGX Spark and multi-architecture CUDA support ( #1447 )
2026-07-03 13:41:43 -07:00
Loay Rashid and SolitaryThinker
995a5fdf97
[bugfix] fixing denoising time in the fastwan script ( #1480 )
...
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com >
2026-06-23 10:01:23 -07:00
Shao Duan
4d6ac89b43
[ci] eval: add metric regression + identity-invariant ci tests ( #1451 )
2026-06-23 01:27:34 -07:00
b2ade71467
[Bugfix] QAD 5090: Torch.compile and other optimizations (15/12) ( #1466 )
...
Co-authored-by: alexzms <3036648523@qq.com >
Co-authored-by: Kaiqin Kong <k1kong@ucsd.edu >
Co-authored-by: William Lin <SolitaryThinker@users.noreply.github.com >
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com >
2026-06-23 01:23:57 -07:00
Kevin Lin
82ed9fe58d
[feat] QAD 5090: FP8 linear layer inference ( #1465 )
2026-06-22 18:25:06 -07:00
c3b971488e
[docs] QAD 5090: Add NVFP4 + Attn-QAT inference example and how-to (9/12) ( #1458 )
...
Co-authored-by: William Lin <SolitaryThinker@users.noreply.github.com >
Co-authored-by: Loay Rashid <42599591+loaydatrain@users.noreply.github.com >
Co-authored-by: Kaiqin Kong <k1kong@ucsd.edu >
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com >
Co-authored-by: alexzms <26690162+alexzms@users.noreply.github.com >
2026-06-16 14:36:50 -07:00
alexzms
2a4fe697a6
[docs] LTX-2.3 distilled i2v: typed-API example (from_config + generate) ( #1448 )
2026-06-10 10:58:14 -07:00
alexzms
921db7479d
[perf] LTX-2.3 distilled i2v: drop max-autotune from compile kwargs ( #1445 )
2026-06-10 10:06:41 -07:00
Aryan Kumar and Aryan Kumar
7f539424cb
[feat]: add Lucy Edit inference scaffold ( #1363 )
...
Co-authored-by: Aryan Kumar <aryan5v@users.noreply.github.com >
2026-06-09 15:21:43 -07:00
d922ab2cbc
[model] Flux2 Klein Port ( #1349 )
...
Co-authored-by: Gnav3852 <63612880+Gnav3852@users.noreply.github.com >
Co-authored-by: Mac Lee <macthecadillac@gmail.com >
2026-06-09 14:55:55 -07:00
Kaiqin Kong and mergify[bot]
efcc245c2e
[feat] VLM as judge for WM ( #1429 )
...
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-06-05 04:25:54 +00:00
alexzms and mergify[bot]
922e7e0813
[docs] LTX-2.3 distilled i2v example with compile + timing breakdown ( #1430 )
...
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-06-05 01:26:50 +00:00
Kaiqin Kong and SolitaryThinker
afdb6fbfa5
[feat] Add MatrixGame3.0 ( #1201 )
...
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com >
2026-05-27 21:07:01 +00:00
Shao Duan
36682797a0
[feat] eval: input ergonomics + Evaluator features + bug fixes ( #1392 )
2026-05-26 13:59:45 -07:00
Shao Duan and abaghyangor
321d5112b4
[refactor] eval: consolidate FVD into common.fvd, remove benchmarks/fvd ( #1380 )
...
Co-authored-by: abaghyangor <abaghyangor@gmail.com >
2026-05-24 12:09:47 -07:00
Raghav K
68179cd752
[docs] Document enable_torch_compile (+ A/B example) ( #1366 )
2026-05-22 00:25:42 -07:00
e8597b7448
[Bugfix] FP4 FA4 installation fix ( #1367 )
...
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com >
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com >
2026-05-19 22:45:24 -07:00
Junda Su
72cb427cd9
[feat]: add FastLTX-2.3 Gradio demo package (draft) ( #1247 )
2026-05-17 23:03:16 -07:00
Kaiqin Kong and mergify[bot]
773d44b875
[misc] Rename MatrixGame to MatrixGame2 ( #1357 )
...
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-05-17 22:57:03 -07:00
Shao Duan and klhhhhh
6b2c731596
[feat] eval: add audio metrics ( #1352 )
...
Co-authored-by: klhhhhh <1412841649@qq.com >
2026-05-16 14:47:37 -07:00
d2ffec5cce
[perf] Dreamverse 14/14: Add LTX2 profile speedups ( #1337 )
...
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com >
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com >
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com >
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com >
2026-05-15 18:10:47 -07:00
Shao Duan
17f07bc313
[feat] eval: async VideoPool + metric streamlines ( #1320 )
2026-05-13 15:57:37 -07:00
b4458e5bae
feat: FP4 Flash Attention 4 for Blackwell GPUs ( #1221 )
...
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com >
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-05-12 17:04:39 -07:00
Shao Duan
f6396fb8c6
[feat] Add fastvideo.eval video evaluation suite ( #1305 )
2026-05-07 20:31:15 -07:00
William Lin
140bd1a6cf
[misc]: standardize install instructions on uv pip install ( #1279 )
2026-05-02 12:45:50 -07:00
William Lin
c77a76c6af
[feat] Stable Audio Open 1.0: T2A + A2A + RePaint inpainting (native) ( #1260 )
2026-05-01 00:07:11 -07:00
William Lin
145a3f166b
[feat] [4/n] Improve API: refactor sampling param and merge with presets ( #1234 )
2026-04-16 14:10:02 -07:00
William Lin
c591d6d2a6
[feat] [2/n] Improve API: add initial support in video_generator ( #1220 )
2026-04-06 10:33:54 -07:00
vishruthb
5789955bbe
[feat] add gen3c (cosmos-7b) model and pipeline support ( #1059 )
2026-04-01 11:42:02 +00:00
Zhang Peiyuan
cd1b7cf139
[Refactor] SP Mask --> original seq len; HunyuanVideo 1.5 does not need mask ( #1142 )
2026-03-04 11:44:47 +08:00
Zhang Peiyuan
fea556269b
[Misc] Fix memory leakage in VideoGenerator ( #1132 )
2026-02-26 19:51:32 -08:00
Zhang Peiyuan and Will Lin
fc80d95d7e
[Misc] Remove STA ( #1124 )
...
Co-authored-by: Will Lin <wlsaidhi@gmail.com >
2026-02-23 15:14:42 -08:00
Zhang Peiyuan
128d0b7fc5
[Misc] Remove Teacache ( #1121 )
2026-02-22 16:53:07 -08:00
10fc92dba5
Upstream LTX2 Training ( #1116 )
...
Co-authored-by: RandNMR73 <notomatthew31@gmail.com >
Co-authored-by: JerryZhou54 <zhouw.jerry2017@outlook.com >
Co-authored-by: Davids048 <jundasu@ucsd.edu >
Co-authored-by: Peiyuan Zhang <a1286225768@slurm-h200-204-239.slurm-compute.tenant-slurm.svc.cluster.local >
2026-02-21 16:06:55 -08:00
8e39ce38c9
[Feat] Native dit implementation for SD3.5 ( #1093 )
...
Co-authored-by: Ishan Vaish <vaish.ishan@gmail.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-02-18 10:19:40 +08:00
Mihir Jagtap
7a83e0e6fc
[feature] Add Hunyuan-GameCraft model support ( #1071 )
2026-02-14 08:07:44 +08:00
Shao Duan and Will Lin
7f795600c8
[bugfix] Fixed ltx2 base cfg guidance ( #1095 )
...
Co-authored-by: Will Lin <wlsaidhi@gmail.com >
2026-02-13 11:53:11 -08:00
Kaiqin Kong
31c0f1b341
[feat] Port LingBot-World-Base (Cam) ( #1081 )
2026-02-10 11:12:33 -08:00
William Lin
4bee0fa199
[misc] cleanup assets/ and demo/ ( #1091 )
2026-02-10 02:26:09 -08:00
530e6b8363
[Model] LTX 2 Base ( #1064 )
...
Co-authored-by: Davids048 <jundasu@ucsd.edu >
Co-authored-by: Will Lin <wlsaidhi@gmail.com >
2026-02-10 01:11:17 -08:00
Ishan and Jinzhe Pan
0aff68f51d
[Feat] Add Stable Diffusion 3.5 ( #1075 )
...
Co-authored-by: Jinzhe Pan <eigensystem1318@gmail.com >
2026-02-10 14:31:36 +08:00
SolitaryThinker