Kyle and Claude Opus 5
43644de268
[perf]: load pipeline components on demand and free them after their last stage
...
A pipeline materializes every component before the first stage runs, so peak
memory is the sum of all components even though no two are needed at the same
moment. The CPU offload flags cannot help with this: they act after loading,
and on a unified-memory device moving weights to the host frees nothing because
it is the same pool.
Add lazy_module_load, off by default. Heavy components become a LazyModule
proxy that loads on first use, and the pipeline installs a release hook on the
last stage that holds each one. Peak becomes the largest overlapping set
instead of the sum.
Measured on MiniMax-H3 r16, 121 GiB GB10, 1 GPU, 192x320, 4 steps. Peak CUDA
allocated is 57.7 GiB, reached during conditioning where the text encoder and
video VAE overlap. The four deferred components account for 96.3 GiB together,
which is what stays resident without the flag: text encoder 48.0, DiT 37.8,
video VAE 9.7, audio VAE 0.8. Generation completed and wrote a video.
Details worth flagging:
The release hook lives on PipelineStage.__call__, not in the pipeline stage
loop, so pipelines that override forward still free.
The proxy forwards __class__, so isinstance stays honest. A proxy answering
False to isinstance(module, FSDPModule) would take the wrong branch silently.
Self-returning methods hand back the proxy rather than the component. Stages
write self.vae = self.vae.to(device) in a dozen places; returning the component
there would replace the proxy with a reference the pipeline cannot release, and
the run would look normal while freeing nothing.
Releasing is a latency cost, never a correctness one: a released component
reloads on next access. Training keeps everything resident and warns if the
flag is set. If no stage holds a deferred component the pipeline warns rather
than silently doing nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-08-28 22:36:51 -07:00
ac48bb3cd1
[feat] Add MiniMax H3 MLX T2VA inference ( #1770 )
...
Co-authored-by: Aryan Kumar <aryank@Aryans-Mac-Studio.local >
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: CodeRabbit <noreply@coderabbit.ai >
2026-08-28 12:54:33 -07:00
KyleNeverGivesUp
e9bbaca07d
[perf] Disable every offload path on unified memory, unblocking MiniMax H3 generation on one GB10 ( #1715 )
2026-08-26 15:32:56 -07:00
Aryan Kumar and Aryan Kumar
6388db815b
[bugfix] FastMetal-QAD MLX support: refuse CUDA QAD trees, use packed mlx_dit config, stream loads ( #1736 ) ( #1758 )
...
Co-authored-by: Aryan Kumar <aryan5v@users.noreply.github.com >
2026-08-25 14:51:40 -07:00
William Lin
f9e3680f11
[perf] Align FastH3 optimized inference profile ( #1748 )
2026-08-23 02:01:03 -07:00
William Lin
c488fa1211
[perf] Add opt-in packed-varlen FA4 for MiniMax-H3 ( #1742 )
2026-08-22 17:16:47 -07:00
William Lin
d3cff517cd
[perf] Add opt-in regional fullgraph compile for DiT inference ( #1741 )
2026-08-22 12:14:39 -07:00
Aryan Kumar and Aryan Kumar
8537dcd6de
[feat]: Apple Silicon MLX runtime — INT8 Wan2.1 and Wan2.2 inference ( #1638 )
...
Co-authored-by: Aryan Kumar <aryan5v@users.noreply.github.com >
2026-08-18 15:08:00 -07:00
Kai
0653f8f3af
[new-model] Add V2A: native MMAudio inference pipeline ( #1622 )
2026-08-09 17:29:27 -07:00
Raghav K and SolitaryThinker
a2bfc7cdb2
[docs] DGX Spark (GB10) performance & tuning guide + reproduction examples ( #1631 )
...
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com >
2026-08-06 14:36:38 -07:00
Kaiqin Kong
e8b0e4c61e
[feat] Add MiniMax H3 ( #1674 )
2026-08-04 13:54:39 -07:00
William Lin
e8812bef0b
[docs]: batched docs cleanup (landing page, links, requirements, nav) ( #1644 )
2026-08-02 18:03:26 -07:00
William Lin
af2934dd6b
[feat]: FA4-FP4 ATTN_QAT_INFER on sm_100/sm_103 + NVFP4 weight purge ( #1647 )
2026-07-27 15:46:52 -07:00
William Lin
1801512818
[docs]: cover all registered models in the support matrix ( #1641 )
2026-07-27 11:13:53 -07:00
William Lin
69349dd2aa
[docs]: unify community links on the README Slack invite ( #1643 )
2026-07-25 17:26:58 -07:00
595f0ea60e
[feat] Add DreamX-World 5B Cam and AR pipelines ( #1538 )
...
Co-authored-by: Suckl <Suckl@users.noreply.github.com >
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com >
2026-07-07 06:16:18 +00:00
William Lin
a5a6d171e5
[attn] Make FA4 explicit opt-in via FASTVIDEO_FA4 and delete the runtime fallback machinery ( #1540 )
2026-07-04 14:51:00 -07:00
William Lin
b36d0ef085
[infra] Add DGX Spark and multi-architecture CUDA support ( #1447 )
2026-07-03 13:41:43 -07:00
Kevin Lin
82ed9fe58d
[feat] QAD 5090: FP8 linear layer inference ( #1465 )
2026-06-22 18:25:06 -07:00
c3b971488e
[docs] QAD 5090: Add NVFP4 + Attn-QAT inference example and how-to (9/12) ( #1458 )
...
Co-authored-by: William Lin <SolitaryThinker@users.noreply.github.com >
Co-authored-by: Loay Rashid <42599591+loaydatrain@users.noreply.github.com >
Co-authored-by: Kaiqin Kong <k1kong@ucsd.edu >
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com >
Co-authored-by: alexzms <26690162+alexzms@users.noreply.github.com >
2026-06-16 14:36:50 -07:00
Aryan Kumar and Aryan Kumar
7f539424cb
[feat]: add Lucy Edit inference scaffold ( #1363 )
...
Co-authored-by: Aryan Kumar <aryan5v@users.noreply.github.com >
2026-06-09 15:21:43 -07:00
KUAN-HAO HUANG and SolitaryThinker
019239690b
[perf] Add Adaptive Guidance (CFG gating) for stale-uncond reuse ( #1372 )
...
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com >
2026-05-30 13:58:55 -07:00
Kaiqin Kong and SolitaryThinker
afdb6fbfa5
[feat] Add MatrixGame3.0 ( #1201 )
...
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com >
2026-05-27 21:07:01 +00:00
Raghav K
68179cd752
[docs] Document enable_torch_compile (+ A/B example) ( #1366 )
2026-05-22 00:25:42 -07:00
e8597b7448
[Bugfix] FP4 FA4 installation fix ( #1367 )
...
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com >
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com >
2026-05-19 22:45:24 -07:00
Kaiqin Kong and mergify[bot]
773d44b875
[misc] Rename MatrixGame to MatrixGame2 ( #1357 )
...
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-05-17 22:57:03 -07:00
William Lin
325861fb99
[misc]: PR-1225 sync — housekeeping (1/12) ( #1347 )
2026-05-13 15:21:52 -07:00
b4458e5bae
feat: FP4 Flash Attention 4 for Blackwell GPUs ( #1221 )
...
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com >
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-05-12 17:04:39 -07:00
William Lin
140bd1a6cf
[misc]: standardize install instructions on uv pip install ( #1279 )
2026-05-02 12:45:50 -07:00
William Lin
145a3f166b
[feat] [4/n] Improve API: refactor sampling param and merge with presets ( #1234 )
2026-04-16 14:10:02 -07:00
William Lin
88a5a933ab
[feat] [3/n] Improve API: extend support to cli ( #1226 )
2026-04-14 15:20:47 -07:00
vishruthb
5789955bbe
[feat] add gen3c (cosmos-7b) model and pipeline support ( #1059 )
2026-04-01 11:42:02 +00:00
Ajay Anubolu
02c1c49b75
[CI] Add inference performance regression tests ( #1140 )
2026-03-07 08:26:54 +08:00
Jinzhe Pan
5190c1bb1e
[Doc] add doc for inference architecture ( #1147 )
2026-03-02 13:44:27 -08:00
Zhang Peiyuan
fea556269b
[Misc] Fix memory leakage in VideoGenerator ( #1132 )
2026-02-26 19:51:32 -08:00
Junda (David) Su
e315657066
[docs] [kernel] Migrate to uv ( #1127 )
2026-02-25 14:13:50 -08:00
Zhang Peiyuan and Will Lin
fc80d95d7e
[Misc] Remove STA ( #1124 )
...
Co-authored-by: Will Lin <wlsaidhi@gmail.com >
2026-02-23 15:14:42 -08:00
Zhang Peiyuan
128d0b7fc5
[Misc] Remove Teacache ( #1121 )
2026-02-22 16:53:07 -08:00
Zhang Peiyuan
03d9ce2edb
[Misc] Remove StepVideo ( #1118 )
2026-02-21 17:15:42 -08:00
6736dc06a5
Improve Docs ( #1112 )
...
Co-authored-by: Peiyuan Zhang <a1286225768@slurm-h200-204-227.slurm-compute.tenant-slurm.svc.cluster.local >
Co-authored-by: Peiyuan Zhang <a1286225768@slurm-login-0.slurm-login.tenant-slurm.svc.cluster.local >
2026-02-19 14:23:32 -08:00
William Lin
75e193a2c9
[core] Refactor and centralize our registry for models, pipelines, and sampling params ( #1066 )
2026-02-05 14:40:30 -08:00
XOR-op
38a6bd93d3
[chore]: update sageattn3 installation instructions ( #1050 )
2026-01-29 15:45:52 -08:00
XOR-op and Will Lin
e8c854bcf1
[docs] Offloading instruction ( #1022 )
...
Co-authored-by: Will Lin <wlsaidhi@gmail.com >
2026-01-26 13:40:28 -08:00
XOR-op
b7f69c2c1d
[feat!] Disable FSDP inference by default ( #1001 )
2026-01-13 14:20:05 -08:00
William Lin
e7f6db9bd1
[docs] Update docs and README ( #975 )
2026-01-04 14:59:19 -06:00
Qi Jia
5285116e73
[docs]: fix various broken links across the documentation ( #979 )
2026-01-01 20:02:39 -06:00
Mihir Jagtap
8f5712629f
[docs] port to mkdocs ( #855 )
2025-11-04 14:31:56 -08:00
