nodes/hao-ai-lab-FastVideo
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
915 Commits 336 Branches 8 Tags
455cd83bed8adf5cd62a89130cb07a64ca4a2b2c
Commit Graph
178 Commits
Author SHA1 Message Date
SolitaryThinker 455cd83bed [bugfix] cosmos3: fix black-video UniPC config; wire E2E inference 
The first real-weights Cosmos3-Nano T2V run produced an all-black video.
Root cause: the checkpoint's diffusers-style scheduler_config.json sets
use_karras_sigmas=true, and FastVideo's vendored UniPC checks karras
*before* use_flow_sigmas, so it built diffusion (beta) sigmas instead of
flow-matching sigmas -> scheduler.step diverged to NaN latents -> black
frames. The DiT/CFG velocity itself was clean.

The framework samples with FlowUniPCMultistepScheduler (pure flow:
shift + num_train_timesteps only). FastVideo's vendored UniPC is the same
algorithm, so the fix is config, not a port:

- Coerce the loaded scheduler to the flow setup in
  Cosmos3OmniDiffusersPipeline.initialize_pipeline (kill karras/exp/beta
  sigmas, force use_flow_sigmas + flow_prediction + final_sigmas_type=zero)
  and rebuild the runtime scheduler from it.
- Use FastVideo's native UniPC (not diffusers) in the pipeline and tests,
  honoring the "no diffusers at runtime" constraint.

Parity:
- Add test_cosmos3_scheduler_parity.py: FastVideo vendored UniPC (flow
  config) vs the framework FlowUniPCMultistepScheduler. timesteps bit-exact,
  sigmas within fp32 epsilon (~1e-8), full multi-step trajectory < ~1e-6
  across shift in {10 (video), 3 (t2i)} and steps in {4,10,35}.
- Fix test_cosmos3_denoise_cfg_parity to use the framework scheduler as the
  oracle (it previously compared diffusers-vs-diffusers, so the scheduler
  was never actually checked against the framework).

Integration fixes to run end-to-end on real weights:
- registry: map checkpoint DiT class Cosmos3OmniTransformer ->
  Cosmos3VFMTransformer.
- component_loader: route the "text_tokenizer" module to TokenizerLoader;
  filter scheduler config to the class's __init__ params (robust to
  diffusers schema drift like shift_terminal/sigma_min/sigma_max).
- DiT: materialize_non_persistent_buffers (recompute rotary_emb.inv_freq
  after meta-device FSDP load) + cast vision latents / timestep embeds to
  the compute dtype (no-ops in the fp32 parity tests).
- sequence_packing: move packed tensors to the model device in to_dit_kwargs.
- config: drop the unused identity text-preprocess (Cosmos3 has no text
  encoder; the DiT tokenizes in the denoise stage).

Adds examples/inference/basic/basic_cosmos3_new_api.py (new VideoGenerator
API). Verified: 1280x704, 29 frames, 35 steps on B200 -> coherent video
matching the prompt (no NaNs, real pixel variance). Full cosmos3 parity
suite green.
 2026-07-20 12:19:45 -07:00
William Lin 65f3b946b9 [feat] Add LTX-2 and LTX-2.3 fine-tuning to the modular trainer (#1624) 2026-07-20 11:58:39 -07:00
Zhang Peiyuan 191fcbf46c [feat] add qat docs (#1621) 2026-07-19 18:37:47 -07:00
Junda Su 755a4e4470 [new-model] Add LingBot-Video Dense and MoE/refiner T2V inference (#1595) 2026-07-18 20:18:16 -07:00
Zhang PeiyuanPeiyuan ZhangPeiyuan
 		9709b7513b [feat] Port NVFP4 QAT/QAD to modular train framework (#1619) 
Co-authored-by: Peiyuan Zhang <email>

Co-authored-by: Peiyuan <a>
 2026-07-18 15:01:09 -07:00
Junda Su d4bdd3621a [new-model] Port LingBot-World-v2 (#1579) 2026-07-16 19:51:53 -07:00
Raghav KMrinaal DograSolitaryThinker
 		6966f9e0bc [fix] Z-Image (#1236) draft port: rebase + strict-load contract + bf16 encoder parity + PORT_STATUS (#1339) 
Co-authored-by: Mrinaal Dogra <mdogra@ucsd.edu>
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com>
 2026-07-16 18:18:25 -07:00
Raghav KIshan VaishClaude Opus 4.8
 		b063f8ca41 [feat] Fix FLUX.1-dev port: native RoPE, parity tests, SSIM reference (#1321) 
Co-authored-by: Ishan Vaish <ivaish@ucsd.edu>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
 2026-07-13 01:49:42 -07:00
Shreejith SGandH1yori233 d82abc271e [feat] Add GLM-Image inference support (#1030) 
Co-authored-by: H1yori233 <k1kong@ucsd.edu>
 2026-07-12 22:42:42 -07:00
Guian FangandSolitaryThinker 970409962f [feat] Add AnyFlow any-step video distillation (pretrain + on-policy) (#1371) 
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com>
 2026-07-12 02:32:08 +00:00
Mingjia HuoandClaude Fable 5 4c08ffce49 [feat] World model training using third person games (#1443) 
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-07-10 05:09:07 +00:00
alexzmsandSolitaryThinker c096fda1eb [docs] Add LTX-2.3 distilled inference run configs (t2v/i2v × 5+2/8+3 × resolutions) (#1568) 
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com>
 2026-07-09 18:53:12 +00:00
Aryan KumarAryan KumarleffffSolitaryThinker
 		02e1143f22 [feat] Add Kandinsky-5 T2V/I2V pipeline support (#1471) 
Co-authored-by: Aryan Kumar <aryan5v@users.noreply.github.com>
Co-authored-by: leffff <levnovitskiy@gmail.com>
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com>
 2026-07-07 14:43:30 -07:00
Kaiqin KongandSolitaryThinker f037351146 [feat] Add Clean-history Teacher Forcing and Causal Consistency Distillation (#1505) 
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com>
 2026-07-07 08:13:11 +00:00
SucklSucklSolitaryThinker
 		595f0ea60e [feat] Add DreamX-World 5B Cam and AR pipelines (#1538) 
Co-authored-by: Suckl <Suckl@users.noreply.github.com>
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com>
 2026-07-07 06:16:18 +00:00
William Lin b36d0ef085 [infra] Add DGX Spark and multi-architecture CUDA support (#1447) 2026-07-03 13:41:43 -07:00
alexzms 8c23c86994 [docs]: add raw-video preprocess script for the QAD MixKit recipe (#1487) 2026-06-28 18:01:13 -07:00
William Lin 78d606a3a7 [feat]: make VSA tile cache configurable for training (#1444) 2026-06-28 02:18:31 -07:00
William Lin 6be280c914 [bugfix]: fix docs build (#1502) 2026-06-27 13:00:09 -07:00
Loay RashidandSolitaryThinker 995a5fdf97 [bugfix] fixing denoising time in the fastwan script (#1480) 
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com>
 2026-06-23 10:01:23 -07:00
Shao Duan 4d6ac89b43 [ci] eval: add metric regression + identity-invariant ci tests (#1451) 2026-06-23 01:27:34 -07:00
Loay RashidalexzmsKaiqin KongWilliam LinSolitaryThinker
 		b2ade71467 [Bugfix] QAD 5090: Torch.compile and other optimizations (15/12) (#1466) 
Co-authored-by: alexzms <3036648523@qq.com>
Co-authored-by: Kaiqin Kong <k1kong@ucsd.edu>
Co-authored-by: William Lin <SolitaryThinker@users.noreply.github.com>
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com>
 2026-06-23 01:23:57 -07:00
Kevin Lin 82ed9fe58d [feat] QAD 5090: FP8 linear layer inference (#1465) 2026-06-22 18:25:06 -07:00
alexzmsWilliam LinLoay RashidKaiqin Kongmergify[bot]
 		e60601df7f [feat] QAD 5090: QAT training recipe — finetune + DMD distillation (12/12) (#1462) 
Co-authored-by: William Lin <SolitaryThinker@users.noreply.github.com>
Co-authored-by: Loay Rashid <42599591+loaydatrain@users.noreply.github.com>
Co-authored-by: Kaiqin Kong <k1kong@ucsd.edu>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-06-17 15:18:52 -07:00
alexzmsWilliam LinLoay RashidKaiqin KongSolitaryThinkeralexzms
 		c3b971488e [docs] QAD 5090: Add NVFP4 + Attn-QAT inference example and how-to (9/12) (#1458) 
Co-authored-by: William Lin <SolitaryThinker@users.noreply.github.com>
Co-authored-by: Loay Rashid <42599591+loaydatrain@users.noreply.github.com>
Co-authored-by: Kaiqin Kong <k1kong@ucsd.edu>
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com>
Co-authored-by: alexzms <26690162+alexzms@users.noreply.github.com>
 2026-06-16 14:36:50 -07:00
Junda SuandPeiyuan Zhang 5854aec2ce [feat] Add Wan RL DiffusionNFT training (#1450) 
Co-authored-by: Peiyuan Zhang <a1286225768@gmail.com>
 2026-06-11 21:18:59 -07:00
alexzms 2a4fe697a6 [docs] LTX-2.3 distilled i2v: typed-API example (from_config + generate) (#1448) 2026-06-10 10:58:14 -07:00
alexzms 921db7479d [perf] LTX-2.3 distilled i2v: drop max-autotune from compile kwargs (#1445) 2026-06-10 10:06:41 -07:00
Aryan KumarandAryan Kumar 7f539424cb [feat]: add Lucy Edit inference scaffold (#1363) 
Co-authored-by: Aryan Kumar <aryan5v@users.noreply.github.com>
 2026-06-09 15:21:43 -07:00
William LinGnav3852Mac Lee
 		d922ab2cbc [model] Flux2 Klein Port (#1349) 
Co-authored-by: Gnav3852 <63612880+Gnav3852@users.noreply.github.com>
Co-authored-by: Mac Lee <macthecadillac@gmail.com>
 2026-06-09 14:55:55 -07:00
Kaiqin Kongandmergify[bot] efcc245c2e [feat] VLM as judge for WM (#1429) 
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-06-05 04:25:54 +00:00
alexzmsandmergify[bot] 922e7e0813 [docs] LTX-2.3 distilled i2v example with compile + timing breakdown (#1430) 
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-06-05 01:26:50 +00:00
Kaiqin KongandSolitaryThinker afdb6fbfa5 [feat] Add MatrixGame3.0 (#1201) 
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com>
 2026-05-27 21:07:01 +00:00
Shao Duan 36682797a0 [feat] eval: input ergonomics + Evaluator features + bug fixes (#1392) 2026-05-26 13:59:45 -07:00
Shao Duanandabaghyangor 321d5112b4 [refactor] eval: consolidate FVD into common.fvd, remove benchmarks/fvd (#1380) 
Co-authored-by: abaghyangor <abaghyangor@gmail.com>
 2026-05-24 12:09:47 -07:00
Raghav K 68179cd752 [docs] Document enable_torch_compile (+ A/B example) (#1366) 2026-05-22 00:25:42 -07:00
Wenxuan TanClaude Opus 4.6gemini-code-assist[bot]mergify[bot]SolitaryThinker
 		e8597b7448 [Bugfix] FP4 FA4 installation fix (#1367) 
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com>
 2026-05-19 22:45:24 -07:00
Junda Su 72cb427cd9 [feat]: add FastLTX-2.3 Gradio demo package (draft) (#1247) 2026-05-17 23:03:16 -07:00
Mingjia Huo 63030cf6ec [fix] Fix causal self-forcing attention settings (#1355) 2026-05-17 22:59:47 -07:00
Kaiqin Kongandmergify[bot] 773d44b875 [misc] Rename MatrixGame to MatrixGame2 (#1357) 
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-05-17 22:57:03 -07:00
Zhili Yangalexzmsmergify[bot]
 		1df513922f [feat] Add minimal LoRA finetuning support to the YAML training stack (#1242) 
Co-authored-by: alexzms <3036648523@qq.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-05-17 14:49:13 -07:00
Shao Duanandklhhhhh 6b2c731596 [feat] eval: add audio metrics (#1352) 
Co-authored-by: klhhhhh <1412841649@qq.com>
 2026-05-16 14:47:37 -07:00
Junda SuSolitaryThinkerMatthew NotoXOR-opZhang Peiyuan
 		d2ffec5cce [perf] Dreamverse 14/14: Add LTX2 profile speedups (#1337) 
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
 2026-05-15 18:10:47 -07:00
Shao Duan 17f07bc313 [feat] eval: async VideoPool + metric streamlines (#1320) 2026-05-13 15:57:37 -07:00
William Lin 325861fb99 [misc]: PR-1225 sync — housekeeping (1/12) (#1347) 2026-05-13 15:21:52 -07:00
Junda SuSolitaryThinkerMatthew NotoXOR-opZhang Peiyuan
 		0403c6f47e [infra] Dreamverse 09/14: Add Docker and launch scripts (#1332) 
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
 2026-05-12 18:01:11 -07:00
Wenxuan TanClaude Opus 4.6gemini-code-assist[bot]mergify[bot]
 		b4458e5bae feat: FP4 Flash Attention 4 for Blackwell GPUs (#1221) 
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-05-12 17:04:39 -07:00
Kaiqin Kongandmergify[bot] a790153705 [bugfix] MatrixGame2 SF distillation under gradient checkpointing (#1340) 
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-05-12 17:01:04 -07:00
Raghav K 9d0af28307 [feat] Add Cosmos 2.5 T2W training pipeline (LoRA + full fine-tune) (#1227) 2026-05-11 19:15:00 -07:00
Kaiqin Kongmignonjiaalexzmsmergify[bot]
 		d6dfe95466 [feat] FastVideo World Model Training (#1179) 
Co-authored-by: mignonjia <mhuo@ucsd.edu>
Co-authored-by: alexzms <3036648523@qq.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-05-11 05:13:12 +00:00