Compare commits

...
Author SHA1 Message Date
Satyam Srivastava 0ef47caf86 [bugfix]: select architecture-specific Blackwell kernel targets
- Detect GB200 and GB300 as 10.0a and 10.3a
- Build future GB200 CI images with 10.0a
- Add regression tests for detection, cache keys, and build arguments
2026-10-05 15:42:48 -07:00
Satyam Srivastava e71d01648c [feat]: add selectable Modal and GB200 Kubernetes GPU CI
- Preserve existing CI and add configurable GPU backend routing
- Limit execution to 2 active PRs, 4 GPUs per PR, and 8 GPUs total
- Add persistent admission, cancellation recovery, and backend statuses
- Document deployment and add infrastructure regression tests
2026-10-01 14:44:14 -07:00
e1f3904799 [kernel] sm_100a CUDA backward for VSA block-sparse attention, 128-token blocks (#1866)
Co-authored-by: Pengcheng Li <pengchengl@ptyche0203.ptyche.clusters.nvidia.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-29 16:57:22 -07:00
Junda Su 02f1ce11ae [misc] Move environment var reads into env registry and rename unprefixed variables (#1897) 2026-09-29 14:36:16 -07:00
Aryan KumarandAryan Kumar 7f03e03dc6 [docs]: FastH3 V2 cookbook serving and MLX install guide (#1884)
Co-authored-by: Aryan Kumar <aryan5v@users.noreply.github.com>
2026-09-29 13:03:18 -07:00
Junda Su e3b88bb12a [feat] Add a typed environment-variable registry, policy doc, and contract test (#1896) 2026-09-29 12:50:45 -07:00
Junda Su cb66acd400 [bugfix] Fix environment-variable bugs in LTX-2 debug logging, HF token lookup, and attention backend reads (#1895) 2026-09-29 11:48:31 -07:00
li-lizhe 442e2d2e18 [bugfix] fix(metrics): set SceneMetric device_map for any non-CPU device (#1817) 2026-09-28 10:24:00 -07:00
Max LI dd35763ad6 [bugfix] Fix MLX prompt enhancement sampler compatibility (#1891) 2026-09-27 14:53:25 -07:00
Keith e90be598e5 [perf] MiniMax H3: return uint8 frames from the decode worker (#1828) 2026-09-24 18:08:56 -07:00
Leleand武垚乐 ba5e81083c [bugfix] Preserve video frames when decoded audio is shorter (#1857)
Signed-off-by: 武垚乐 <wuyaole@mininglamp.com>
Co-authored-by: 武垚乐 <wuyaole@mininglamp.com>
2026-09-24 17:20:37 -07:00
YZJF 76ce9c7fd6 [bugfix] Reject mismatched model names on image generation routes (#1881) 2026-09-24 17:19:29 -07:00
Junda Su 08d99c089e [perf] Reuse cached VAE offload for H3 conditioning (#1882) 2026-09-24 17:18:39 -07:00
Jerry-XY 20751a21aa [bugfix] Keep the VSA-H3 tile buffer out of the autograd graph (#1858) 2026-09-24 17:16:49 -07:00
alanhuangyooandSolitaryThinker 9dd2a837f4 [bugfix] encoders: keep Qwen2.5-VL importable on transformers 5 (#1791)
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com>
2026-09-24 17:13:23 -07:00
alanhuangyoo 93aab45ac2 [bugfix] tests: include ltx2_3_base in local LTX-2 preset set (#1750) 2026-09-24 17:11:31 -07:00
Yogya Mehrotra 017ce6602d [bugfix] Fix dead NVFP4 QAT plumbing: bench imports, int8 error string, smooth_q crash (#1723) 2026-09-24 17:09:33 -07:00
Yogya Mehrotra a575055eec [perf] benchmark_attn_qat_train: device peak-TFLOPS table instead of silent RTX 5090 default (#1724) 2026-09-24 17:08:03 -07:00
Kyle Hu 81f3fec7fd [bugfix]: workers killed by a signal now log the reason (#1725) 2026-09-24 17:06:55 -07:00
Raghav K d265a454bf [bugfix]: Fall back when large pinned output allocation fails (#1759) 2026-09-24 17:05:59 -07:00
c100c66578 [bugfix]: isolate cancelled streaming requests from GPU result readers (#1848)
Co-authored-by: Gxj230958 <222823329+Gxj230958@users.noreply.github.com>
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com>
2026-09-24 17:04:59 -07:00
Aryan Kumar 8760eb7a06 [feat] Add FastH3 8-Step V2 MLX inference (#1863)
Keep the four-step uniform AdaLN cache, and use explicit DMD rungs only when the snapshot declares its own scheduler shifts.
2026-09-23 12:42:03 -07:00
Aryan Kumar 361f919c88 [feat]: add a narrow FastH3 8-step MLX ladder
Keep the four-step uniform AdaLN cache, and use explicit DMD rungs only when the snapshot declares its own scheduler shifts.
2026-09-23 11:06:55 -07:00
Aryan Kumar 8b5377aab2 Merge remote-tracking branch 'origin/main' into aryan/pr-1863-fasth3-8step 2026-09-23 10:52:20 -07:00
Yixing Wangandleo d995516da0 [bugfix] Restore page interaction after dismissing Create Job or successfully creating a job (#1869)
Co-authored-by: leo <yixingwang@YIXINGs-MacBook-Pro.local>
2026-09-21 12:03:54 -07:00
Keith f47ad3f5b7 [bugfix] fastvideo-kernel: install the Python + Triton package when the HIP toolchain cannot be configured (#1871) 2026-09-21 11:59:13 -07:00
Hexu ZhaoandClaude Opus 5 10bdf5e076 [bugfix] frozen offload: no host copy for an already resident module
`load` took the host copies before checking which tensors were missing from
the device, so with `vae_cpu_offload=False` — where the loader builds the VAEs
on CUDA and `unload` is never called — it pulled about 11 GB per rank back over
PCIe once and kept a pinned mirror of it for the life of the process, for a
module that never leaves the device.

It now asks what is absent first and returns early when nothing is. The
offloaded path is unchanged: weights still round-trip and come back bit
identical.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-20 22:46:13 -07:00
Hexu ZhaoandClaude Opus 5 3e26db40b0 [perf] H3 decode: pin the frozen VAEs' host copies
With the device-to-host copy gone, the remaining host-to-device copy is the
stage's cost, and from pageable memory it is staged through a bounce buffer at
roughly 2.6 GB/s instead of full PCIe speed. The host copies are made once, so
pinning them costs nothing per request.

It does cost the module's size in non-pageable host memory (10.4 GB for the
H3 video VAE, 0.6 GB for the audio VAE, per rank) for as long as the pipeline
is alive. It follows --pin-cpu-memory, which is on by default; --no-pin-cpu-
memory keeps the copies pageable and only loses the bandwidth. Note that this
flag did not already reach the VAE: its other use is FSDP's own CPUOffloadPolicy
and the VAE does not go through FSDP.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ATw7g6cxq9N2LtnMratsru
2026-09-20 22:46:13 -07:00
Hexu ZhaoandClaude Opus 5 c73dd0ab55 [perf] H3 decode: offload the frozen VAEs without the device-to-host copy
The decode stages move the 10.4 GB fp32 video VAE (and the 0.6 GB audio VAE)
with module.to(device) / module.to("cpu") on every request. The weights are
frozen -- the whole stage runs under no_grad -- so the copy back to the host
returns bytes the host already has.

Keep one host copy per tensor, made the first time the module is loaded, and
on unload point .data back at it instead of copying. One H2D per request, no
D2H at all; the module still lives on the host between requests. Where the old
path allocated a fresh host tensor per cycle and freed the previous one, this
reuses one buffer, so peak host memory can only go down.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ATw7g6cxq9N2LtnMratsru
2026-09-20 22:46:13 -07:00
430e52154e [bugfix]: Restore exact Qwen3-VL vision interpolation (#1737)
Co-authored-by: William Lin <8941107+SolitaryThinker@users.noreply.github.com>
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com>
2026-09-18 05:00:05 -07:00
vanch 384eee8aef feat(mlx): add FastH3-8-Step-V2 DMD schedule and thermal-safe MLX inference
- Support official 8-step DMD schedule from fastvideo_inference.json in MLX pipeline
- Implement MiniMaxH3SchedulerState.from_dmd_steps for exact sigma ladder calculation
- Add auto-detection of fastvideo_inference.json in checkpoint converter AdaLN precomputation
- Add inter_step_cooldown_s and per-step MLX graph/cache cleanup to prevent thermal throttling
- Support configurable VAE tile sizes to guarantee 256px tiled decode without grid artifacts
- Skip non-existent shard files gracefully in Qwen3-VL conditioner
2026-09-17 16:12:02 +08:00
KeithandSolitaryThinker c4824c7764 [bugfix] FP8: gate the ROCm _scaled_mm path on CDNA4 (gfx950) instead of the capability tuple (#1859)
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com>
2026-09-16 19:58:55 -07:00
Raghav K 9b0e57fe4b [ci] Make Dreamverse provider race test deterministic (#1729) 2026-09-15 14:13:56 -07:00
William Lin 0100218594 [feat] Support the FastH3 8-Step V2 checkpoint: checkpoint-defined shifts, explicit DMD schedule, new example (#1852) 2026-09-15 14:13:45 -07:00
li-lizhe 39718cd54d [bugfix] fix(cosmos): make AdaLayerNorm autocast device-agnostic (#1818) 2026-09-15 07:44:51 -07:00
IshanandSolitaryThinker 37d06a832f [feat] Add fastvideo serve configs for Wan CUDA models (#1801)
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com>
2026-09-14 18:23:35 -07:00
sudhirpol522 8839ba8d4d [bugfix] Add OpenAI-compatible image generation endpoint (#1840) 2026-09-14 17:55:56 -07:00
sudhirpol522andSolitaryThinker 61b91220c0 [bugfix] Validate image response format before generation (#1841)
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com>
2026-09-14 17:01:40 -07:00
Lele 316f3876c2 [bugfix]: allow MiniMax H3 frame padding at the 15-second limit
Accept the causal-VAE-aligned 362-frame bucket (15.083 s) for 15-second H3 requests.

- Hoist MINIMAX_H3_MIN/MAX_ALIGNED_FRAMES next to align_num_frames and reuse them in the CUDA stage, the MLX runtime, and the LoRA example so the bound is consistent across entry points.
- State the accepted frame range in the error messages.
- Cover seconds="15" -> 362 at OpenAI admission and the MLX resolve_geometry bound.
- Update the Spark/openai cookbook frame caps from 345 to 362.

Known gaps: no GPU-lane coverage for the 362 bucket (SSIM runs 124 frames; the golden gate is a fixed-geometry fingerprint), and explicit num_frames=360 still hits the pre-existing grid gate.
2026-09-14 16:54:41 -07:00
Yaegaki1Erika 614b59543c [bugfix]: respect serialized tensor dtype in parquet dataloader (#1843) 2026-09-14 16:34:25 -07:00
William Lin 1c14afd559 [docs]: refresh AGENTS.md maps and add Wan SP/I2V and CI test notes (#1845) 2026-09-14 16:06:53 -07:00
Yaegaki1Erika 9a3c45779c [bugfix]: fix Wan I2V DMD conditioning under sequence parallelism (#1844)
The pipeline pre-sharded only the image conditioning along the temporal axis while the noise input stayed full-length, so concatenation could not match for sp_world_size > 1. WanTransformer3DModel shards the flattened token sequence after patch embedding, so pass full-length mask+latent conditioning and let the transformer shard.

Also reads temporal_compression_ratio from the VAE config, simplifies the conditioning mask, concatenates in the transformer's (bs, c, t, h, w) layout, and adds unit coverage.

Co-authored-by: Yaegaki1Erika <70182590+Yaegaki1Erika@users.noreply.github.com>
2026-09-14 15:58:13 -07:00
Kyle Hu bfc9c01797 [feat]: convert the MiniMax H3 text encoder to NVFP4 (#1838) 2026-09-12 18:48:07 -07:00
Kyle Hu 3a3ad3d209 [feat]: NVFP4 text encoder for MiniMax H3 (#1837) 2026-09-12 18:31:59 -07:00
lpc0220andClaude Fable 5.1 aef4e9b3b1 [kernel] VSA kernel: one sm_100a / sm_103a image per listed arch; un-gate backward on sm_103a (#1833)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-12 16:49:36 -07:00
a943220c11 [bugfix]: drop dead h3_sequential_load from Spark FastH3 presets (#1831)
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Aryan Kumar <aryan5v@users.noreply.github.com>
2026-09-08 12:38:06 -07:00
William Lin 556ac7088e [refactor] Simplify Wan sampling and tests (#1825) 2026-09-07 15:57:19 -07:00
Junda Su e7456f1b75 Add H3 support into Dreamverse (#1800) 2026-09-07 13:09:44 -07:00
lpc0220 c993d7393e [kernel] sm_100a CUDA backward for VSA block-sparse attention (blk64) (#1819) 2026-09-06 21:00:18 -07:00
William Lin 7f83164233 [refactor] Move Wan VAE into the Wan package (#1824) 2026-09-05 18:43:55 -07:00
Junda Su 4e52f47d1e [feat] add MXFP8 support on H3 (#1796) 2026-09-05 17:08:58 -07:00
William Lin e19913f6e9 [refactor] Group Wan transformer and config (#1823) 2026-09-05 17:05:32 -07:00
William Lin 2413a57651 Disable old SSIM models (#1820) 2026-09-04 21:20:48 -07:00
SYLAR 7bb76b5ec9 [feat] Add native SM103a VSA support (#1812)
Signed-off-by: lishunyang12 <lishunyang12@163.com>
2026-09-03 15:00:28 -07:00
Aryan KumarandAryan Kumar 0bd19a976b [docs]: add one-Spark FastH3 cookbook runtime with a device-count row (#1811)
Co-authored-by: Aryan Kumar <aryan5v@users.noreply.github.com>
2026-09-02 10:05:59 -07:00
William Lin 40b93784d2 [docs]: add cookbook link to README (#1810) 2026-09-01 12:42:19 -07:00
Aryan Kumar 33d3478bad [docs] Announce local FastH3 support (#1809) 2026-09-01 12:28:04 -07:00
3d8ac9d14b [feat]: collapse cookbook recipe pages into an accordion layout, add … (#1805)
Co-authored-by: Vaish, Ishan <isvaish@UCSD.EDU>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-01 01:51:51 -07:00
aaef49bfc6 [feat]: run FastH3 across two DGX Sparks with Ray sequence parallel (#1803)
Co-authored-by: Kyle <shh075@ucsd.edu>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Satyam Srivastava <srivastavasatyam53@gmail.com>
Co-authored-by: Aryan Kumar <aryan5v@users.noreply.github.com>
2026-09-01 01:37:23 -07:00
Aryan KumarandAryan Kumar cf6a00b9be [feat]: add opt-in CUDA TAEH3 preview decode for FastH3 (#1795)
Co-authored-by: Aryan Kumar <aryan5v@users.noreply.github.com>
2026-08-31 23:34:56 -07:00
Shahrad ZomorrodiandShahrad Zomorrodi 1ae39562dd [bugfix] Write generated documentation as UTF-8 (#1797)
Co-authored-by: Shahrad Zomorrodi <264690209+shahradzomorrodi@users.noreply.github.com>
2026-08-31 22:45:03 -07:00
Aryan KumarandAryan Kumar 26064193e2 [feat] Add an H3 server cookbook and prompt playground (#1798)
Co-authored-by: Aryan Kumar <aryan5v@users.noreply.github.com>
2026-08-31 22:14:33 -07:00
William Lin 8446fc003e [docs]: add FastH3 Preview v1 news links (#1804) 2026-08-31 22:12:02 -07:00
Aryan KumarandAryan Kumar a28f2bab4b [feat] Add an optional MLX TAEH3 preview decoder (#1794)
Co-authored-by: Aryan Kumar <aryan5v@users.noreply.github.com>
2026-08-31 05:08:06 -07:00
Aryan KumarandAryan Kumar f82d8be4bf [perf] Sequential MiniMax H3 start with GPU-direct DiT load (#1793)
Co-authored-by: Aryan Kumar <aryan5v@users.noreply.github.com>
2026-08-31 04:20:21 -07:00
Aryan KumarandAryan Kumar 8e1775183e [perf] Speed up exact MiniMax H3 MLX inference (#1792)
Co-authored-by: Aryan Kumar <aryan5v@users.noreply.github.com>
2026-08-31 04:14:55 -07:00
Aryan KumarandAryan Kumar 620bc36dc4 [docs]: Cookbook catalog improvements (#1790)
Co-authored-by: Aryan Kumar <aryan5v@users.noreply.github.com>
2026-08-30 17:50:57 -07:00
Suhaan Khurana 29ff16ec96 [feat] Add MiniMax H3 MLX spatial fast mode (#1789) 2026-08-30 17:47:02 -07:00
Aryan KumarandAryan Kumar 8f9d76a80d [perf]: dispatch wide-M affine H3 MLX linears through dequant plus dense GEMM (#1788)
Co-authored-by: Aryan Kumar <aryan5v@users.noreply.github.com>
2026-08-30 14:42:01 -07:00
a4d9a75e2c [perf] Add MiniMax H3 MLX VSA and SIMD attention (#1776)
Co-authored-by: Aryan Kumar <aryan5v@users.noreply.github.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2026-08-30 13:44:54 -07:00
KyleNeverGivesUp b2db0c0a13 [ci]: seed stable GB10 grad-norm references (#1756) 2026-08-30 03:09:18 -07:00
Kevin Lin 6aa7d8a278 [misc] FastVideo Studio UI Additions (H3 Ref2V support) (#1783) 2026-08-30 03:06:47 -07:00
Aryan KumarandAryan Kumar ccc9014430 [docs] Add model-family inference cookbook (#1787)
Co-authored-by: Aryan Kumar <aryan5v@users.noreply.github.com>
2026-08-30 02:26:10 -07:00
William Lin a159b63c67 [bugfix] Harden OpenAI serving after post-merge review (#1782) 2026-08-28 22:29:24 -07:00
ac48bb3cd1 [feat] Add MiniMax H3 MLX T2VA inference (#1770)
Co-authored-by: Aryan Kumar <aryank@Aryans-Mac-Studio.local>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: CodeRabbit <noreply@coderabbit.ai>
2026-08-28 12:54:33 -07:00
William Lin 3987b9ddcd [feat] Align multimodal OpenAI serving APIs (#1781) 2026-08-28 10:09:58 -07:00
William Lin c7da2f5d60 [chore]: release v0.2.1 (#1778) 2026-08-28 02:03:18 -07:00
William Lin 39ae1decc0 [misc] pin fastvideo-kernel to exact 0.3.5 (#1777) 2026-08-28 02:02:27 -07:00
William Lin 1aed667377 [chore] release fastvideo-kernel 0.3.5 (#1775) 2026-08-27 23:18:59 -07:00
William Lin c1612ff397 [bugfix]: pin fastvideo-kernel to Torch 2.12.0 (#1774) 2026-08-27 23:13:54 -07:00
William Linandshaoxiongduan a534ba20a0 [feat] Add MiniMax H3 LoRA inference and preview launchers (#1771)
Co-authored-by: shaoxiongduan <shaoxiongduan@gmail.com>
2026-08-27 14:43:39 -07:00
KyleNeverGivesUp e9bbaca07d [perf] Disable every offload path on unified memory, unblocking MiniMax H3 generation on one GB10 (#1715) 2026-08-26 15:32:56 -07:00
KyleNeverGivesUp 9bfa585448 [perf]: stop holding the whole checkpoint during DiT load, unblocking MiniMax H3 on one GB10 (#1714) 2026-08-26 15:23:51 -07:00
Raghav K b2062556a9 [perf] VSA Triton: widen the autotune num_stages range (the optimum was outside it) (#1706) 2026-08-26 12:30:56 -07:00
KyleNeverGivesUp c9c5585758 [perf]: MiniMax H3 on GB10 - skip text encoder CPU offload on unified memory (5m49s to 30ms) (#1710) 2026-08-26 12:03:14 -07:00
William Lin 9212f4f218 [ci] make GPU validation change-aware (#1747) 2026-08-25 21:26:25 -07:00
Aryan KumarandAryan Kumar 6388db815b [bugfix] FastMetal-QAD MLX support: refuse CUDA QAD trees, use packed mlx_dit config, stream loads (#1736) (#1758)
Co-authored-by: Aryan Kumar <aryan5v@users.noreply.github.com>
2026-08-25 14:51:40 -07:00
lpc0220 7a4285189f [kernel] Route block-sparse VSA to the sm_100a forward behind FASTVIDEO_VSA_SM100A (opt-in) (#1754) 2026-08-24 15:26:56 -07:00
William Lin a837fe841a [docs] Update FastH3 README (#1749) 2026-08-23 05:05:58 -07:00
William Lin f9e3680f11 [perf] Align FastH3 optimized inference profile (#1748) 2026-08-23 02:01:03 -07:00
William Lin 98f761ec45 [bugfix] validation: inherit the trained denoising ladder (#1738) 2026-08-22 23:09:26 -07:00
Shao Duan c041318f2c [perf] Add fused NVLink all-to-all for Ulysses (#1740) 2026-08-22 23:06:24 -07:00
William Lin 604e0205a4 [perf] Keep odd MiniMax-H3 VSA tiles on sm100a (#1745) 2026-08-22 18:29:30 -07:00
William Lin 13213395b4 [perf] Parallelize MiniMax-H3 VAE over sequence ranks (#1744) 2026-08-22 18:05:31 -07:00
William Lin 46afee5998 [bugfix] Classify MiniMax-H3 inference controls in schema inventory (#1743) 2026-08-22 17:17:02 -07:00
William Lin c488fa1211 [perf] Add opt-in packed-varlen FA4 for MiniMax-H3 (#1742) 2026-08-22 17:16:47 -07:00
William Lin d3cff517cd [perf] Add opt-in regional fullgraph compile for DiT inference (#1741) 2026-08-22 12:14:39 -07:00
Junda Su 2f3d407406 [perf] Optimize MiniMax H3 VAE decoding (#1734) 2026-08-21 14:57:32 -07:00
Kaiqin Kong bcffa4026e [perf] Optimize MiniMax-H3 text encoder memory (#1732) 2026-08-21 14:57:06 -07:00
William Lin 6d6a10be7a [feat] FastVideo-Minimax-FastH3-Preview few-step example + 64-token-tile VSA-H3 inference path (#1731) 2026-08-21 12:40:09 -05:00
Kaiqin Kong 73dd105f3d [perf] Add opt-in MiniMax-H3 Sol-Engine fusions (#1735) 2026-08-21 12:39:28 -05:00
William LinandClaude Fable 5 56d4a6074f [bugfix] fastvideo-kernel: fix Triton block-sparse backward logit scaling (bf16 K pre-scaling) (#1730)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-21 04:50:06 -05:00
Shao Duan c4ad4227c0 [misc] MiniMax-H3: move the AdaLN converter into scripts/checkpoint_conversion (#1712) 2026-08-21 02:15:00 -05:00
KyleNeverGivesUpandSolitaryThinker a63ccce73d [docs]: add a maintained inference cookbook (#1290)
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com>
2026-08-21 01:53:44 -05:00
KyleNeverGivesUp 0462e1b0e7 [perf]: MiniMax H3 - build the Qwen3-VL encoder only as far as it is read (-13.7 GB) (#1711) 2026-08-20 23:19:25 -05:00
lpc0220 907f2100ec [kernel] sm_100a CUDA block-sparse VSA forward (Blackwell), 64- and 128-token blocks (#1719) 2026-08-20 23:16:25 -05:00
Kaiqin Kong e0a3db5651 [perf] Reduce MiniMax-H3 VAE peak memory (#1703) 2026-08-20 23:13:23 -05:00
Raghav K fca45bc8e1 [perf] Quantize frames to uint8 on-device before the post-decode D->H copy (#1362) 2026-08-20 21:55:55 -05:00
Aryan Kumar 86d639c848 [docs] Announce FastMetal-QAD (#1721) 2026-08-19 13:45:49 -07:00
00338aa9ca [perf] Add FA4 CuTe backward support for VSA-256 (#1639)
Co-authored-by: Hyunsung Lee <hyunsungl@sizigistudios.com>
Co-authored-by: alexzms <3036648523@qq.com>
2026-08-19 11:46:49 -07:00
679 changed files with 70951 additions and 9507 deletions
+99
View File
@@ -0,0 +1,99 @@
---
name: ci-runner
description: Work on FastVideo's Slurm-only, change-aware GPU CI lanes, static Buildkite graph, trusted ci-runner policy, lane scripts, and GB200 validation.
---
# Slinky Slurm CI lanes
FastVideo's `ci-runner` Buildkite queue is the control plane for all active
GPU CI. A private host-owned dispatcher leases GPUs from the Slinky Slurm tray
and runs the immutable PR SHA inside an isolated Enroot container. Buildkite
pipeline upload and Slurm submission occur on the login plane; every test
payload executes on Slurm compute.
The files under `fastvideo/tests/modal/` and `.buildkite/scripts/pr_test.sh`
are dormant rollback code. Never add an active Buildkite or slash-command
route to them. `pr_test.sh` must continue to reject Buildkite invocations.
The private operator bundle is deliberately outside this repository because
it contains site paths and credentials. See
`docs/contributing/ci_architecture.md`; this skill covers the repository half
and the coordination contract with that bundle.
## Invariants
- `.buildkite/pipeline.yml` contains exactly one static step for every active
GPU lane. Each step pins a unique key and label, a 90-minute timeout, the
trusted `/opt/fastvideo-ci-runner/run-ci` command (`run-unit` is the one
compatibility wrapper), step-level internal `TEST_TYPE`, and
`queue: "ci-runner"`.
- Active CI contains no `pr_test.sh` command, Modal invocation, default queue,
Buildkite plugin, `soft_fail`, or job-controlled artifact glob.
- The six Fastcheck lanes use `:microscope:` labels. Full-Suite-only lanes use
`:test_tube:` or `:bar_chart:` so direct reruns update the right aggregate.
- SSIM and vanilla training request all four GPUs. Keep both in the
`fastvideo/slinky/whole-tray` Buildkite concurrency group with a limit of one
so the second job does not consume an agent or command timeout while waiting
for the same tray.
- `/test full` schedules all twenty lanes. `/merge`, `ready`, and new pushes to
ready PRs use the trusted base-branch planner in
`.github/scripts/plan_merge_ci.py`: automatic Fastcheck remains the universal
six-lane baseline, and the merge build adds only path-relevant integration
lanes. Unknown source/build paths fail closed to all fourteen additive lanes.
The trusted uploader still normalizes and validates the complete static graph
before Buildkite evaluates its plan conditions.
- Focused merge builds may pass allowlisted golden-gate and SSIM test basenames.
The private host validates the lane plan and basenames before staging them,
and the in-container scripts validate them again. Direct `/test ssim`,
explicit `/test full`, and the weekly main-branch schedule run the complete
SSIM matrix.
- The trusted uploader serves exactly three entry pipelines:
`pr-fastcheck` for automatic PR builds, `ci` for slash-command/ready-label
API builds, and `fastvideo-performance-lane` for the weekly schedule. Keep
incoming GitHub webhook processing disabled on `ci` so it cannot duplicate
`pr-fastcheck` on every PR update.
- Test payloads live in `.buildkite/scripts/unit_test.sh` or executable
`.buildkite/scripts/lanes/<lane>.sh`. Backend policy (GPU count, extras,
secrets, kernel build, artifacts) stays in the agent-owned lane table.
- Tests must preserve an inherited `MASTER_PORT`. Packed containers share the
tray network namespace, so the private runner assigns a distinct port range
per GPU lease and the SSIM scheduler assigns task offsets within its range.
- The ARM64 runner image includes the pinned FA4 CuTe overlay validated on
GB200. Keep SSIM at `FASTVIDEO_FA4=1` because its references were seeded with
FA4; keep lanes with FA2 baselines at `FASTVIDEO_FA4=0`. A runner image change
must revalidate both the FA4 import and an actual GB200 forward kernel.
- `fastvideo/tests/ssim/ci_runner.py` is the active four-GPU SSIM scheduler.
New SSIM files are discovered through `REQUIRED_GPUS` and
`*_MODEL_TO_PARAMS`; do not wire them through the dormant Modal scheduler.
- The host policy fail-closes unknown tuples. A repository-side lane change is
inert until the operator updates the private lane table and uploader policy
in the same rollout.
## Adding or changing a lane
1. Read the closest `AGENTS.md` and the domain-specific testing guide.
2. Add or update the executable lane payload under `.buildkite/scripts/`.
Keep it deterministic and free of host-specific paths or credential fetches.
3. Add the static pipeline step and canonical `/test <name>` mapping. Keep the
`<name>-ci` alias only when compatibility requires it.
4. Add its source/test path ownership to `.github/scripts/plan_merge_ci.py`.
Prefer the narrowest correctness-preserving lane set; leave unknown paths
fail-closed. Extend `fastvideo/tests/contract/test_ci_test_collection.py`,
`test_merge_ci_plan.py`, and focused CPU-only scheduler/policy tests.
5. Coordinate the private lane row: GPU count (1-4), wall time, script, scope
pairs, step key, command, HF cache/token, tracking mode, extras, attention
backend policy, kernel policy, and artifact relay. Active training lanes
keep W&B offline and do not stage a W&B credential.
6. Update the trusted pipeline-uploader schema. A mismatch must reject the
pipeline rather than silently skip a lane.
7. Run `pre-commit run --files <changed paths>`, the planner's representative
diff matrix, contract tests, private driver tests, and a real GB200 canary.
Multi-GPU, hardware-reference, training, performance, and SSIM changes need
their own target-hardware evidence.
## Rollback
Rollback the Slurm routing/configuration change or pause the `ci-runner` queue.
Do not silently reactivate Modal. A manual Modal experiment requires the
explicit local opt-in documented in `ci_architecture.md`; returning it to
production CI needs a separate reviewed decision.
@@ -0,0 +1,76 @@
---
name: env-var-conventions
description: Add, read, rename, or remove an environment variable in FastVideo, or change the environment-variable policy. Use before touching fastvideo/envs.py, os.environ, os.getenv, or monkeypatch.setenv in fastvideo/, and when fastvideo/tests/contract/test_env_policy.py fails.
---
# Environment Variable Conventions
## Purpose
FastVideo registers its environment variables as typed fields in
`fastvideo/envs.py`. The policy that governs them is
`docs/contributing/env_vars.md`, and the contract test
`fastvideo/tests/contract/test_env_policy.py` enforces the policy in the unit
CI lane. This skill routes an environment-variable change through that policy.
The policy doc is the single source of the rules; read it instead of relying
on a summary here.
## Prerequisites
- Read `docs/contributing/env_vars.md` in full.
- Decide whether the setting belongs in an environment variable or an argument
(rule 5 in the policy doc). Settings that users change per deployment are
arguments; add them through `fastvideo/fastvideo_args.py` instead.
## Inputs
| Parameter | Required | Description |
| ---------- | -------- | -------------------------------------------------------------- |
| `change` | Yes | Add, read, rename, or remove a variable, or change the policy. |
| `variable` | Yes | The variable name, with the `FASTVIDEO_` prefix. |
## Steps
1. **Declare or edit the variable in `fastvideo/envs.py`.**
- Pick the field type and category that the policy doc lists.
- Write a description that states what the variable does and its units.
- To rename, keep the old name in `deprecated_names`. To remove, add the
name to `DEPRECATED_VARIABLES`. Update the uses in `examples/`,
`scripts/`, `docs/`, `apps/`, and the tests.
2. **Read the variable with `envs.NAME.get()` inside a function.**
- In tests, change the value with `envs.NAME.override(value)`.
- Do not call `os.environ`, `os.getenv`, or `monkeypatch.setenv` for a
FastVideo variable.
- To set a variable that another tool reads, call `envs.set_external`,
`envs.setdefault_external`, or `envs.unset_external`.
3. **Regenerate the table in the policy doc.**
- Run `python fastvideo/tests/contract/test_env_policy.py`.
4. **Run the contract test.**
- Run `pytest fastvideo/tests/contract/test_env_policy.py`.
- When the test reports a fixed known violation, delete or lower its entry
in `KNOWN_VIOLATIONS`. Never add an entry to `KNOWN_VIOLATIONS`.
5. **When the policy itself changes, update the policy doc and the contract
test in the same pull request.**
- The rules in `docs/contributing/env_vars.md`, the checks and allowlist in
`fastvideo/tests/contract/test_env_policy.py`, and this skill must agree.
## Outputs
- A registry entry in `fastvideo/envs.py` and call sites that use
`envs.NAME.get()`.
- A regenerated table in `docs/contributing/env_vars.md`.
- A passing `fastvideo/tests/contract/test_env_policy.py`.
## Example Usage
```
Add a FASTVIDEO_DEBUG_MY_STAGE switch that logs MyStage inputs.
```
## References
- `docs/contributing/env_vars.md`: the policy, the field types, and the
violation kinds that the contract test reports.
- `fastvideo/envs.py`: the registry.
- `fastvideo/tests/contract/test_env_policy.py`: the contract test,
`EXTERNAL_ALLOWLIST`, and `KNOWN_VIOLATIONS`.
@@ -1,6 +1,6 @@
---
name: reseed-ssim-references
description: Re-seed HF reference videos for a single existing SSIM test on Modal L40S. Always backs up current refs locally first, regenerates on Modal, pauses for the user to eyeball before-vs-after quality, then overwrites the targeted `<model_id>` subtree on `FastVideo/ssim-reference-videos` with `--force`. Use when an intentional code change (model port fix, attention backend swap, kernel upgrade, hyperparameter change) has invalidated existing refs and they need to be regenerated. Pairs with `seed-ssim-references`, which is for first-time seeding only.
description: Re-seed HF reference videos for a single existing SSIM test on Modal L40S. Always backs up current refs locally first, regenerates on Modal, pauses for the user to eyeball before-vs-after quality, then overwrites the targeted model subtree on `FastVideo/ssim-reference-videos` with `--force`. Use when an intentional code change (model port fix, attention backend swap, kernel upgrade, hyperparameter change) has invalidated existing refs and they need to be regenerated. Pairs with `seed-ssim-references`, which is for first-time seeding only.
---
# Re-seed SSIM Reference Videos
@@ -13,7 +13,7 @@ on HF — the old refs are overwritten — so the skill always:
1. Confirms intent with a one-liner the user has to type.
2. Downloads the existing refs as a local, timestamped backup.
3. Regenerates on Modal L40S (same code path that CI uses).
3. Regenerates through the manual legacy Modal L40S maintenance path.
4. Pauses for a side-by-side eyeball of backup vs new mp4s.
5. Uploads with `--force`, scoped to the single `--model-id`.
6. Reminds the user to keep the backup until the PR lands.
@@ -51,8 +51,9 @@ harder to recover from than failing closed.
Hardcoded:
- Modal GPU: **L40S** (matches CI; re-seeding from another SKU produces refs
that L40S CI cannot match).
- Modal GPU: **L40S**. This is a manual reference-maintenance target, not the
active Slurm CI compute path; changing the SKU also changes the historical
`L40S_reference_videos` contract.
- Quality tier: **`default`**. `full_quality` is a separate, deliberate
operation.
- HF repo: `FastVideo/ssim-reference-videos` (override via
+4 -2
View File
@@ -35,7 +35,8 @@ The skill is run **manually**, once per new test. Before invoking it, the user
has already sanity-tested the new test locally — it launches `VideoGenerator`
and writes an artefact without crashing (the missing-reference assertion at
the end is expected). The skill does not re-test locally; it goes straight
to Modal L40S (which is what CI uses).
to the manual legacy Modal L40S reference-maintenance target. Active CI runs
on the Slinky Slurm cluster and only consumes the resulting references.
## When to use
@@ -61,7 +62,8 @@ Prompt the user for it if they didn't supply it.
Everything else is fixed:
- Modal runner GPU: **L40S** (hardcoded in `fastvideo/tests/modal/ssim_test.py`).
- Modal maintenance GPU: **L40S** (hardcoded in
`fastvideo/tests/modal/ssim_test.py`; this is not the active CI compute path).
- Device folder: `L40S_reference_videos`.
- Quality tier: `default` (the tier CI runs). The `full_quality` tier is not
seeded by this skill.
+461 -528
View File
File diff suppressed because it is too large Load Diff
+5
View File
@@ -0,0 +1,5 @@
#!/usr/bin/env bash
# Canonical Slurm CI selection for the OpenAI-compatible API lane.
set -euo pipefail
exec pytest ./fastvideo/tests/entrypoints/test_openai_api_integration.py -vs
+5
View File
@@ -0,0 +1,5 @@
#!/usr/bin/env bash
# Canonical Slurm CI selection for the distillation-DMD lane.
set -euo pipefail
exec pytest ./fastvideo/tests/training/distill/test_distill_dmd.py -vs
+87
View File
@@ -0,0 +1,87 @@
#!/usr/bin/env bash
# DreamVerse needs a GPU for import-time device resolution, but it does not
# build or exercise fastvideo-kernel. A checksummed Node archive is installed
# in the disposable Slurm container because the shared CI image is
# Python/CUDA focused.
set -euo pipefail
node_version=v22.23.2
case $(uname -m) in
aarch64 | arm64)
node_arch=arm64
node_archive_sha256=013b59cfd2819703a6f4a14ab891fc46fc2a4e3f5bcd92de3fb4929b43e35b30
;;
x86_64 | amd64)
node_arch=x64
node_archive_sha256=b294a556e639d64338823920e5866c21c02741742d2e1529ee1a225c1ec9252a
;;
*)
echo "Unsupported architecture for DreamVerse Node runtime: $(uname -m)" >&2
exit 2
;;
esac
node_archive="node-${node_version}-linux-${node_arch}.tar.gz"
node_runtime_root=$(mktemp -d -t fastvideo-node.XXXXXX)
node_archive_path="${node_runtime_root}/${node_archive}"
node_install_dir="${node_runtime_root}/${node_archive%.tar.gz}"
curl --proto '=https' --tlsv1.2 --retry 5 --retry-all-errors \
--location --fail --silent --show-error \
"https://nodejs.org/dist/${node_version}/${node_archive}" \
--output "$node_archive_path"
printf '%s %s\n' "$node_archive_sha256" "$node_archive_path" | sha256sum --check --status
tar -xzf "$node_archive_path" -C "$node_runtime_root"
export PATH="${node_install_dir}/bin:${PATH}"
node --version
npm --version
export PYTHONPATH="$(pwd)/apps/dreamverse${PYTHONPATH:+:$PYTHONPATH}"
pytest apps/dreamverse/dreamverse/tests -q
cd apps/dreamverse/web
npm ci
npm run typecheck
npm test
machine_arch=$(uname -m)
if [[ $machine_arch =~ ^(aarch64|arm64)$ ]]; then
npx playwright install --with-deps chromium firefox
else
npx playwright install --with-deps chromium webkit firefox
fi
master_port=${MASTER_PORT:-7959}
BACKEND_PORT=${BACKEND_PORT:-$((master_port + 50))}
python -m uvicorn dreamverse.mock_server:app --host 127.0.0.1 --port "$BACKEND_PORT" &
mock_server_pid=$!
cleanup() {
kill "$mock_server_pid" 2>/dev/null || true
wait "$mock_server_pid" 2>/dev/null || true
}
trap cleanup EXIT INT TERM
for _ in {1..30}; do
curl -fsS "http://127.0.0.1:$BACKEND_PORT/healthz" && break
sleep 1
done
curl -fsS "http://127.0.0.1:$BACKEND_PORT/healthz"
if [[ $machine_arch =~ ^(aarch64|arm64)$ ]]; then
# Playwright WebKit traps before opening a page on Linux ARM64, and its
# bundled Chromium lacks the H.264/AAC codecs used by the fMP4 assertions.
# Firefox covers every flow, including streaming. Chromium and its mobile
# profile still cover all codec-independent UI behavior on GB200.
BACKEND_HOST=127.0.0.1 BACKEND_PORT="$BACKEND_PORT" CI=1 \
npm run e2e -- --project=firefox
BACKEND_HOST=127.0.0.1 BACKEND_PORT="$BACKEND_PORT" CI=1 \
npm run e2e -- \
--project=chromium \
--project=mobile-chromium \
--grep-invert='streams, plays, and surfaces a downloadable clip|starts a new project and switches back to the prior session|saved projects persist across a page reload'
else
BACKEND_HOST=127.0.0.1 BACKEND_PORT="$BACKEND_PORT" CI=1 \
npm run e2e -- \
--project=chromium \
--project=webkit \
--project=firefox \
--project=mobile-safari \
--project=mobile-chromium
fi
+5
View File
@@ -0,0 +1,5 @@
#!/usr/bin/env bash
# Canonical Slurm CI selection for the encoder lane.
set -euo pipefail
exec pytest ./fastvideo/tests/encoders -vs
+5
View File
@@ -0,0 +1,5 @@
#!/usr/bin/env bash
# Canonical Slurm CI selection for the evaluation lane.
set -euo pipefail
exec pytest ./fastvideo/tests/eval -vs
+35
View File
@@ -0,0 +1,35 @@
#!/usr/bin/env bash
# Canonical Slurm CI selection for the golden-gate lane. Environment (HF_HOME
# and authentication) is the runner's responsibility.
set -euo pipefail
golden_root=./fastvideo/tests/golden_gate
selected=${FASTVIDEO_GOLDEN_TEST_FILES-}
if [ -z "$selected" ]; then
if [ "${TEST_SCOPE:-}" = merge ]; then
echo "Missing FASTVIDEO_GOLDEN_TEST_FILES for merge scope" >&2
exit 2
fi
selected=all
fi
if [ "$selected" = all ]; then
exec pytest "$golden_root" -xvs
fi
[[ $selected =~ ^test_[a-z0-9_]+\.py(,test_[a-z0-9_]+\.py)*$ ]] || {
echo "Invalid FASTVIDEO_GOLDEN_TEST_FILES selection" >&2
exit 2
}
IFS=, read -r -a golden_files <<< "$selected"
golden_paths=()
for golden_file in "${golden_files[@]}"; do
golden_path="$golden_root/$golden_file"
[ -f "$golden_path" ] || {
echo "Selected golden test does not exist: $golden_file" >&2
exit 2
}
golden_paths+=("$golden_path")
done
exec pytest "${golden_paths[@]}" -xvs
+5
View File
@@ -0,0 +1,5 @@
#!/usr/bin/env bash
# Canonical Slurm CI selection for the LoRA-inference lane.
set -euo pipefail
exec pytest ./fastvideo/tests/inference/lora/test_lora_inference_similarity.py -vs
+5
View File
@@ -0,0 +1,5 @@
#!/usr/bin/env bash
# Canonical Slurm CI selection for the VMoBA-inference lane.
set -euo pipefail
exec python fastvideo/tests/inference/vmoba/test_vmoba_inference.py
+5
View File
@@ -0,0 +1,5 @@
#!/usr/bin/env bash
# Canonical Slurm CI selection for the custom-kernel lane.
set -euo pipefail
exec pytest fastvideo-kernel/tests/ -vs
+5
View File
@@ -0,0 +1,5 @@
#!/usr/bin/env bash
# Canonical Slurm CI selection for the LoRA-extraction lane.
set -euo pipefail
exec pytest ./fastvideo/tests/lora_extraction/test_lora_extraction.py -vs
+58
View File
@@ -0,0 +1,58 @@
#!/usr/bin/env bash
# Canonical Slurm performance lane. Reports are written outside the checkout
# so the trusted host driver can upload them after untrusted code exits.
set -uo pipefail
export PERFORMANCE_TRACKING_ROOT=/tmp/perf-tracking
export PERF_REPORTS_DIR=/workspace/artifacts/performance
mkdir -p "$PERF_REPORTS_DIR"
if [[ ${BUILDKITE_PULL_REQUEST:-false} =~ ^[1-9][0-9]*$ ]]; then
export PERF_RUN_SOURCE=pr
export PERF_UPLOAD_POLICY=pass
elif [ "${BUILDKITE_BRANCH:-}" = main ] \
&& { [ "${BUILDKITE_SOURCE:-}" = schedule ] || [ "${TEST_SCOPE:-}" = full ]; }; then
export PERF_RUN_SOURCE=scheduled_main
export PERF_UPLOAD_POLICY=always
elif [ "${TEST_SCOPE:-}" = direct ]; then
export PERF_RUN_SOURCE=unknown
export PERF_UPLOAD_POLICY=pass
else
export PERF_RUN_SOURCE=unknown
export PERF_UPLOAD_POLICY=never
fi
# Alternate GPU backends compare against references without publishing records.
# Their worker has read-only Hub credentials; publication is an operator task.
if [ "${FASTVIDEO_CI_LOCAL_ONLY:-0}" = 1 ]; then
export PERF_UPLOAD_POLICY=never
fi
nvidia-smi \
--query-gpu=index,timestamp,clocks.sm,clocks.max.sm,power.draw,power.limit,temperature.gpu \
--format=csv -l 10 > "$PERF_REPORTS_DIR/gpu_telemetry.csv" 2>/dev/null &
telemetry_pid=$!
cleanup() {
kill "$telemetry_pid" 2>/dev/null || true
wait "$telemetry_pid" 2>/dev/null || true
}
trap cleanup EXIT INT TERM
pytest ./fastvideo/tests/performance -vs
pytest_rc=$?
compare_rc=0
if [ "$pytest_rc" -eq 0 ] || [ "$PERF_UPLOAD_POLICY" = always ]; then
PERF_PYTEST_RC=$pytest_rc python ./fastvideo/tests/performance/compare_baseline.py
compare_rc=$?
fi
python ./fastvideo/tests/performance/dashboard.py || true
cp -f fastvideo/tests/performance/results/*.json "$PERF_REPORTS_DIR/" 2>/dev/null || true
echo "--- GPU telemetry (clocks.sm vs clocks.max.sm reveals capped hosts) ---"
cat "$PERF_REPORTS_DIR/gpu_telemetry.csv" || true
final_rc=$pytest_rc
if [ "$final_rc" -eq 0 ]; then
final_rc=$compare_rc
fi
exit "$final_rc"
+6
View File
@@ -0,0 +1,6 @@
#!/usr/bin/env bash
# Canonical Slurm CI selection for the self-forcing lane.
set -euo pipefail
export WANDB_MODE=offline
exec pytest ./fastvideo/tests/training/self-forcing/test_self_forcing.py -vs
+40
View File
@@ -0,0 +1,40 @@
#!/usr/bin/env bash
# Canonical four-GPU SSIM lane for the Slinky Slurm worker.
set -euo pipefail
args=()
if [ "${FASTVIDEO_SSIM_BOOTSTRAP_MODE:-0}" = 1 ]; then
args+=(--bootstrap-mode)
fi
selected=${FASTVIDEO_SSIM_TEST_FILES-}
if [ -z "$selected" ]; then
if [ "${TEST_SCOPE:-}" = merge ]; then
echo "Missing FASTVIDEO_SSIM_TEST_FILES for merge scope" >&2
exit 2
fi
selected=all
fi
if [ "$selected" != all ]; then
[[ $selected =~ ^test_[a-z0-9_]+\.py(,test_[a-z0-9_]+\.py)*$ ]] || {
echo "Invalid FASTVIDEO_SSIM_TEST_FILES selection" >&2
exit 2
}
IFS=, read -r -a ssim_files <<< "$selected"
for ssim_file in "${ssim_files[@]}"; do
args+=(--test-file "$ssim_file")
done
fi
# MoGe's utils3d dependency builds glcontext from source on ARM64. The current
# runner image predates the baked-in X11 headers below, so keep this guarded
# bootstrap until every deployed image digest contains libx11-dev.
if [ ! -f /usr/include/X11/Xlib.h ]; then
apt-get -o Acquire::Retries=5 update
apt-get -o Acquire::Retries=5 install -y --no-install-recommends libx11-dev
rm -rf /var/lib/apt/lists/*
fi
uv pip install git+https://github.com/microsoft/MoGe.git
uv pip install k_diffusion einops_exts alias_free_torch torchsde
exec python fastvideo/tests/ssim/ci_runner.py "${args[@]}"
+5
View File
@@ -0,0 +1,5 @@
#!/usr/bin/env bash
# Canonical Slurm CI selection for the modular training-framework lane.
set -euo pipefail
exec pytest ./fastvideo/tests/train/models ./fastvideo/tests/train/methods -vs
+6
View File
@@ -0,0 +1,6 @@
#!/usr/bin/env bash
# Canonical Slurm CI selection for the legacy vanilla-training lane.
set -euo pipefail
export WANDB_MODE=offline
exec pytest ./fastvideo/tests/training/Vanilla -srP
+6
View File
@@ -0,0 +1,6 @@
#!/usr/bin/env bash
# Canonical Slurm CI selection for the legacy LoRA-training lane.
set -euo pipefail
export WANDB_MODE=offline
exec pytest ./fastvideo/tests/training/lora/test_lora_training.py -srP
+6
View File
@@ -0,0 +1,6 @@
#!/usr/bin/env bash
# Canonical Slurm CI selection for the legacy VSA-training lane.
set -euo pipefail
export WANDB_MODE=offline
exec pytest ./fastvideo/tests/training/VSA -srP
+9
View File
@@ -0,0 +1,9 @@
#!/usr/bin/env bash
# Canonical Slurm CI selection for the transformer lane.
set -euo pipefail
# The existing block reference records an absent FASTVIDEO_FA4 (FA2). Keep
# that reference identity; the component lane also selects FA2 explicitly.
env -u FASTVIDEO_FA4 pytest ./fastvideo/tests/golden_gate/test_wan_t2v.py -xvs
pytest ./fastvideo/tests/golden_gate/test_wan_causal.py -xvs
exec pytest ./fastvideo/tests/transformers -vs
+6
View File
@@ -0,0 +1,6 @@
#!/usr/bin/env bash
# Canonical Slurm CI selection for the VAE lane.
set -euo pipefail
pytest ./fastvideo/tests/golden_gate/test_wan_vae.py -xvs
exec pytest ./fastvideo/tests/vaes -vs
+13
View File
@@ -1,6 +1,19 @@
#!/bin/bash
set -uo pipefail
# DORMANT ROLLBACK ONLY. Active CI is Slurm-only and pipeline.yml never calls
# this launcher. Refuse every Buildkite invocation even if a stale step or
# operator typo reaches this file; local rollback experiments require an
# explicit opt-in.
if [ -n "${BUILDKITE:-}" ]; then
echo "Legacy Modal CI is disabled; use the Slinky Slurm runner." >&2
exit 2
fi
if [ "${FASTVIDEO_ENABLE_LEGACY_MODAL_CI:-0}" != 1 ]; then
echo "Legacy Modal CI is dormant. Set FASTVIDEO_ENABLE_LEGACY_MODAL_CI=1 only for a manual rollback test." >&2
exit 2
fi
log() {
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1"
}
+26
View File
@@ -0,0 +1,26 @@
#!/usr/bin/env bash
set -euo pipefail
exec pytest \
./fastvideo/tests/api/ \
./fastvideo/tests/contract/ \
./fastvideo/tests/dataset/ \
./fastvideo/tests/workflow/ \
./fastvideo/tests/entrypoints/ \
./fastvideo/tests/loader/ \
./fastvideo/tests/pipelines/ \
./fastvideo/tests/platforms/ \
./fastvideo/tests/train/ \
./fastvideo/tests/stages/ \
./fastvideo/tests/ops/ \
./fastvideo/tests/worker/ \
./fastvideo/tests/training/test_trackers.py \
./fastvideo/tests/attention/test_sdpa_metadata_mask_contract.py \
./fastvideo/tests/attention/test_vsa_h3_tile_grad_safety.py \
./fastvideo/tests/modal/test_kernel_build_cache.py \
./fastvideo/tests/modal/test_pr_test.py \
./fastvideo/tests/modal/test_ssim_test.py \
--ignore=./fastvideo/tests/entrypoints/test_openai_api_integration.py \
--ignore=./fastvideo/tests/train/models \
--ignore=./fastvideo/tests/train/methods \
-vs
+2 -2
View File
@@ -8,10 +8,10 @@ PR TITLE: Must start with a type tag, e.g.:
MERGE WORKFLOW:
1. Ensure pre-commit passes and you have at least 1 approval
2. Comment /merge (or add the "ready" label) to enter the Merge Queue
3. Full Test Suite runs automatically on a staging branch → auto-merge on success
3. A path-aware merge gate runs only relevant integration tests → auto-merge on success
ON-DEMAND TESTING (write access required):
/test full — Full Test Suite /test ssim — SSIM regression
/test full — Explicit all-lane run /test ssim — Full SSIM regression
/test training — Training pipeline /test encoder — Encoder tests
/test transformer — Transformer tests /test vae — VAE tests
/test kernel — CUDA kernel tests /test unit — Unit tests
+10 -10
View File
@@ -1,14 +1,14 @@
#!/usr/bin/env bash
# Gate the expensive Buildkite full suite on the cheap GitHub checks.
# Gate the path-aware Buildkite merge plan on the cheap GitHub checks.
#
# Polls the workflow runs for the PR head commit and only exits 0 once the
# watched cheap workflows (pre-commit, docs build) have succeeded, so the
# 'ready' label cannot burn ~20 GPU lanes on a head that a cheap check has
# already doomed.
# 'ready' label cannot burn path-selected GPU lanes on a head that a cheap
# check has already doomed.
#
# Semantics:
# - watched run completed with a bad conclusion -> exit 1 (fail CLOSED:
# no full suite; the next push re-arms via the 'synchronize' trigger)
# no merge gate; the next push re-arms via the 'synchronize' trigger)
# - watched run cancelled -> still pending: the docs
# workflow's repo-global 'pages' concurrency group cancels runs superseded
# by unrelated pushes, so 'cancelled' is not a verdict on this PR
@@ -29,7 +29,7 @@ set -euo pipefail
: "${PR_NUMBER:?PR_NUMBER (pull request number) is required}"
: "${GITHUB_REPOSITORY:?GITHUB_REPOSITORY is required}"
# Workflow-level `name:` values that must be green before the full suite
# Workflow-level `name:` values that must be green before the merge gate
# may start. "Deploy Documentation" is path-filtered on PRs, so its run may
# legitimately never exist; pre-commit always runs, so it must appear.
WATCHED_NAMES='["pre-commit", "Deploy Documentation"]'
@@ -56,7 +56,7 @@ recheck_ready_label() {
if pr_json=$(gh_api "repos/${GITHUB_REPOSITORY}/pulls/${PR_NUMBER}" 2>/dev/null); then
if ! jq -e '[.labels[]?.name] | index("ready")' <<<"$pr_json" >/dev/null 2>&1; then
echo "::error::PR #${PR_NUMBER} no longer has the 'ready' label —" \
"NOT triggering the Buildkite full suite. Re-add the label to re-arm."
"NOT triggering the Buildkite merge gate. Re-add the label to re-arm."
exit 1
fi
else
@@ -84,7 +84,7 @@ while true; do
| map(.name) | join(", ")' <<<"$state")
if [ -n "$failed" ]; then
echo "::error::Cheap check(s) failed on ${PR_SHA}: ${failed}." \
"NOT triggering the Buildkite full suite. Push a fix (the 'ready'" \
"NOT triggering the Buildkite merge gate. Push a fix (the 'ready'" \
"label re-arms on every push), or re-run the failed check and then" \
"re-run this workflow."
exit 1
@@ -97,7 +97,7 @@ while true; do
if [ "$pending" -eq 0 ]; then
if [ -z "$missing" ]; then
recheck_ready_label
echo "All watched cheap checks are green — full suite may proceed."
echo "All watched cheap checks are green — merge gate may proceed."
exit 0
fi
case "$missing" in
@@ -119,14 +119,14 @@ while true; do
echo "::warning::GitHub API error querying workflow runs for ${PR_SHA} (attempt ${api_fails}/3)."
if [ "$api_fails" -ge 3 ]; then
recheck_ready_label
echo "::warning::FAILING OPEN: cannot query GitHub check status — triggering the full suite WITHOUT the cheap-check gate."
echo "::warning::FAILING OPEN: cannot query GitHub check status — triggering the merge gate WITHOUT the cheap-check gate."
exit 0
fi
fi
if [ "$elapsed" -ge "$MAX_WAIT_SECS" ]; then
recheck_ready_label
echo "::warning::FAILING OPEN: watched checks still pending after $(( MAX_WAIT_SECS / 60 )) min${missing:+ (never appeared: ${missing})} — triggering the full suite anyway."
echo "::warning::FAILING OPEN: watched checks still pending after $(( MAX_WAIT_SECS / 60 )) min${missing:+ (never appeared: ${missing})} — triggering the merge gate anyway."
exit 0
fi
sleep "$POLL_SECS"
+582
View File
@@ -0,0 +1,582 @@
#!/usr/bin/env python3
"""Select the additive GPU integration lanes needed by a PR diff.
Fastcheck is the universal six-lane baseline and is intentionally not repeated
here. This planner selects only the more expensive merge-gate lanes. Unknown
source/build paths fail closed to the complete integration set, while explicit
documentation and repository-metadata paths require no additional GPU work.
"""
from __future__ import annotations
import argparse
import fnmatch
import re
from dataclasses import dataclass, field
from pathlib import Path
from typing import TextIO
MERGE_LANES = (
"golden-gate",
"ssim",
"lora-inference",
"lora-extraction",
"training",
"distillation",
"self-forcing",
"lora-training",
"training-vsa",
"inference-vmoba",
"performance",
"api-server",
"train-framework",
"eval",
)
LANE_SCRIPT_TO_KEY = {
"api_server.sh": "api-server",
"distillation_dmd.sh": "distillation",
"eval.sh": "eval",
"golden_gate.sh": "golden-gate",
"inference_lora.sh": "lora-inference",
"inference_vmoba.sh": "inference-vmoba",
"lora_extraction.sh": "lora-extraction",
"performance.sh": "performance",
"self_forcing.sh": "self-forcing",
"ssim.sh": "ssim",
"train_framework.sh": "train-framework",
"training.sh": "training",
"training_lora.sh": "lora-training",
"training_vsa.sh": "training-vsa",
}
FASTCHECK_LANE_SCRIPTS = {
"dreamverse.sh",
"encoder.sh",
"kernel_tests.sh",
"transformer.sh",
"vae.sh",
}
LEGACY_TRAINING_LANES = (
"training",
"distillation",
"self-forcing",
"lora-training",
"training-vsa",
)
ALL_TRAINING_LANES = (*LEGACY_TRAINING_LANES, "train-framework")
SSIM_SMOKE_TESTS = (
"test_flux_t2i_similarity.py",
"test_wan_t2v_similarity.py",
)
SAFE_PATTERNS = (
"*.md",
"*.rst",
".agents/**",
".claude/**",
".codex/**",
".github/ISSUE_TEMPLATE/**",
".github/PULL_REQUEST_TEMPLATE.md",
".github/dependabot.yml",
".github/mergify.yml",
".github/scripts/**",
".github/workflows/**",
".buildkite/scripts/pre_commit.sh",
".git-blame-ignore-revs",
".gitattributes",
".gitignore",
".pre-commit-config.yaml",
"AGENTS.md",
"CITATION.cff",
"CODE_OF_CONDUCT.md",
"CONTRIBUTING.md",
"LICENSE",
"NOTICE",
"__init__.py",
"collect_env.py",
"SECURITY.md",
"assets/**",
"comfyui/**",
"docs/**",
"examples/**",
"mkdocs.yml",
"requirements-mkdocs.in",
"requirements-mkdocs.txt",
"scripts/**",
"tests/__init__.py",
"tests/local_tests/**",
)
ALL_IMPACT_PATTERNS = (
".buildkite/pipeline.yml",
"docker/**",
"pyproject.toml",
"requirements*.txt",
"setup.cfg",
"setup.py",
"uv.lock",
)
@dataclass(frozen=True)
class FamilyCoverage:
pattern: re.Pattern[str]
golden_tests: tuple[str, ...]
ssim_tests: tuple[str, ...]
FAMILY_COVERAGE = (
FamilyCoverage(
re.compile(r"(^|[/_.-])dreamx(_world)?([/_.-]|$)"),
("test_dreamx.py", ),
("test_dreamx_world_similarity.py", ),
),
FamilyCoverage(
re.compile(r"(^|[/_.-])flux[_-]?2([/_.-]|$)"),
("test_flux2_klein.py", ),
("test_flux2_similarity.py", ),
),
FamilyCoverage(
re.compile(r"(^|[/_.-])flux(?![_-]?2)([/_.-]|$)"),
("test_flux.py", ),
("test_flux_t2i_similarity.py", ),
),
FamilyCoverage(
re.compile(r"(^|[/_.-])(hunyuan)?gamecraft([/_.-]|$)"),
("test_gamecraft.py", ),
("test_gamecraft_similarity.py", ),
),
FamilyCoverage(
re.compile(r"(^|[/_.-])gen3c([/_.-]|$)"),
("test_gen3c.py", ),
("test_gen3c_similarity.py", ),
),
FamilyCoverage(
re.compile(r"(^|[/_.-])glm[_-]?image([/_.-]|$)"),
("test_glm_image.py", ),
("test_glm_image_similarity.py", ),
),
FamilyCoverage(
re.compile(r"(^|[/_.-])kandinsky[_-]?5([/_.-]|$)"),
("test_kandinsky5.py", ),
("test_kandinsky5_similarity.py", ),
),
FamilyCoverage(
re.compile(r"(^|[/_.-])lingbot([a-z0-9_-]*)([/_.-]|$)"),
("test_lingbot.py", ),
("test_lingbot_similarity.py", ),
),
FamilyCoverage(
re.compile(r"(^|[/_.-])longcat([/_.-]|$)"),
("test_longcat.py", ),
("test_longcat_similarity.py", ),
),
FamilyCoverage(
re.compile(r"(^|[/_.-])ltx[_-]?2([/_.-]|$)"),
("test_ltx2.py", ),
("test_ltx2_similarity.py", ),
),
FamilyCoverage(
re.compile(r"(^|[/_.-])matrixgame[_-]?2([/_.-]|$)"),
("test_matrixgame.py", ),
("test_matrixgame2_similarity.py", ),
),
FamilyCoverage(
re.compile(r"(^|[/_.-])matrixgame[_-]?3([/_.-]|$)"),
("test_matrixgame.py", ),
("test_matrixgame3_similarity.py", ),
),
FamilyCoverage(
re.compile(r"(^|[/_.-])minimax[_-]?h3([/_.-]|$)"),
("test_minimax_h3_t2v.py", ),
("test_minimax_h3_similarity.py", ),
),
FamilyCoverage(
re.compile(r"(^|[/_.-])sd[_-]?3([._-]?5)?([/_.-]|$)"),
("test_sd35.py", ),
("test_sd35_similarity.py", ),
),
FamilyCoverage(
re.compile(r"(^|[/_.-])stable[_-]?audio([/_.-]|$)"),
("test_stable_audio.py", ),
("test_stable_audio_similarity.py", ),
),
FamilyCoverage(
re.compile(r"(^|[/_.-])turbo(diffusion)?([/_.-]|$)"),
(),
("test_turbodiffusion_similarity.py", ),
),
FamilyCoverage(
re.compile(r"(^|[/_.-])wan(video|vae)?([/_.-]|$)"),
("test_wan_t2v.py", "test_wan_vae.py", "test_wan_causal.py", "test_wan_denoising.py"),
(
"test_causal_similarity.py",
"test_wan_i2v_similarity.py",
"test_wan_t2v_similarity.py",
),
),
FamilyCoverage(
re.compile(r"(^|[/_.-])z[_-]?image([/_.-]|$)"),
("test_zimage.py", ),
("test_zimage_similarity.py", ),
),
)
@dataclass
class MergePlan:
lanes: set[str] = field(default_factory=set)
golden_tests: set[str] = field(default_factory=set)
ssim_tests: set[str] = field(default_factory=set)
golden_all: bool = False
ssim_all: bool = False
reasons: list[str] = field(default_factory=list)
def add_lanes(self, *lanes: str, reason: str) -> None:
unknown = set(lanes) - set(MERGE_LANES)
if unknown:
raise ValueError(f"Unknown merge lanes: {sorted(unknown)}")
self.lanes.update(lanes)
self.reasons.append(reason)
def add_golden(self, tests: tuple[str, ...], reason: str) -> None:
self.add_lanes("golden-gate", reason=reason)
self.golden_tests.update(tests)
def add_ssim(self, tests: tuple[str, ...], reason: str) -> None:
self.add_lanes("ssim", reason=reason)
self.ssim_tests.update(tests)
def require_all(self, reason: str) -> None:
self.lanes.update(MERGE_LANES)
self.golden_all = True
self.ssim_all = True
self.reasons.append(reason)
def ordered_lanes(self) -> tuple[str, ...]:
return tuple(lane for lane in MERGE_LANES if lane in self.lanes)
def encoded_lanes(self) -> str:
lanes = self.ordered_lanes()
return "," + ",".join(lanes or ("none", )) + ","
def encoded_golden_tests(self) -> str:
if "golden-gate" not in self.lanes:
return "none"
if self.golden_all or not self.golden_tests:
return "all"
return ",".join(sorted(self.golden_tests))
def encoded_ssim_tests(self) -> str:
if "ssim" not in self.lanes:
return "none"
if self.ssim_all or not self.ssim_tests:
return "all"
return ",".join(sorted(self.ssim_tests))
def _matches_any(path: str, patterns: tuple[str, ...]) -> bool:
return any(fnmatch.fnmatchcase(path, pattern) for pattern in patterns)
def _family_coverage(path: str) -> tuple[set[str], set[str]]:
normalized = path.lower()
golden: set[str] = set()
ssim: set[str] = set()
for family in FAMILY_COVERAGE:
if family.pattern.search(normalized):
golden.update(family.golden_tests)
ssim.update(family.ssim_tests)
# Select the component actually touched, including compatibility paths.
# Family configs/pipeline wiring can affect all four Wan gates.
if re.search(r"(^|[/_.-])wan(video|vae)?([/_.-]|$)", normalized):
if (normalized.endswith(("/wan/vae.py", "/wan/vae_config.py", "/vaes/wanvae.py"))
or normalized.endswith("/wan/stages/conditioning.py")):
golden = {"test_wan_vae.py"}
elif normalized.endswith(("/wan/causal_transformer.py", "/dits/causal_wanvideo.py",
"/wan/stages/causal_denoising.py")):
golden = {"test_wan_causal.py"}
elif (normalized == "fastvideo/models/dits/wanvideo.py"
or normalized.endswith(("/wan/transformer.py", "/wan/stages/denoising.py", "/wan/stages/dmd.py"))):
golden = {"test_wan_t2v.py", "test_wan_denoising.py"}
return golden, ssim
def _select_output_coverage(plan: MergePlan, path: str) -> None:
golden, ssim = _family_coverage(path)
if golden:
plan.add_golden(tuple(sorted(golden)), reason=f"model-family golden coverage: {path}")
else:
plan.golden_all = True
plan.add_lanes("golden-gate", reason=f"shared output golden coverage: {path}")
if ssim:
plan.add_ssim(tuple(sorted(ssim)), reason=f"model-family SSIM coverage: {path}")
else:
plan.add_ssim(SSIM_SMOKE_TESTS, reason=f"shared output SSIM smoke coverage: {path}")
def classify_paths(paths: list[str]) -> MergePlan:
plan = MergePlan()
normalized_paths: list[str] = []
for raw_path in paths:
path = raw_path.strip()
while path.startswith("./"):
path = path[2:]
if path:
normalized_paths.append(path)
normalized_paths = sorted(set(normalized_paths))
if not normalized_paths:
plan.require_all("changed-file list was empty; failing closed")
return plan
for path in normalized_paths:
if path == "__FASTVIDEO_CI_PLAN_ALL__":
plan.require_all("changed-file API failed; failing closed")
continue
if path in {"requirements-mkdocs.in", "requirements-mkdocs.txt"}:
plan.reasons.append(f"documentation dependencies need no GPU integration: {path}")
continue
if _matches_any(path, ALL_IMPACT_PATTERNS):
plan.require_all(f"cross-cutting build/runtime surface: {path}")
continue
lane_script_prefix = ".buildkite/scripts/lanes/"
if path.startswith(lane_script_prefix):
script_name = Path(path).name
lane = LANE_SCRIPT_TO_KEY.get(script_name)
if lane is None:
if script_name in FASTCHECK_LANE_SCRIPTS:
plan.reasons.append(f"covered by automatic Fastcheck lane: {path}")
else:
plan.require_all(f"unknown lane script: {path}")
elif lane == "golden-gate":
plan.golden_all = True
plan.add_lanes(lane, reason=f"golden lane implementation: {path}")
elif lane == "ssim":
plan.ssim_all = True
plan.add_lanes(lane, reason=f"SSIM lane implementation: {path}")
else:
plan.add_lanes(lane, reason=f"lane implementation: {path}")
continue
if path.startswith("fastvideo/tests/golden_gate/"):
name = Path(path).name
if name.startswith("test_") and name.endswith(".py"):
plan.add_golden((name, ), reason=f"changed golden test: {path}")
elif name in {"AGENTS.md", "README.md"}:
plan.reasons.append(f"golden documentation only: {path}")
else:
plan.golden_all = True
plan.add_lanes("golden-gate", reason=f"shared golden harness/reference: {path}")
continue
if path.startswith("fastvideo/tests/ssim/"):
name = Path(path).name
if name.startswith("test_") and name.endswith(".py"):
plan.add_ssim((name, ), reason=f"changed SSIM test: {path}")
elif path.endswith((".py", ".json", ".pt", ".png", ".mp4")):
plan.ssim_all = True
plan.add_lanes("ssim", reason=f"shared SSIM harness/reference: {path}")
continue
if path.startswith("fastvideo/tests/performance/") or path.startswith(".buildkite/performance-benchmarks/"):
plan.add_lanes("performance", reason=f"performance coverage: {path}")
continue
if path.startswith(("fastvideo/performance/", "fastvideo/performance_dashboard/",
"apps/performance_dashboard/")):
plan.add_lanes("performance", reason=f"performance implementation: {path}")
continue
if path.startswith("fastvideo/benchmarks/"):
if "/mlx_" in path or Path(path).name.startswith("mlx_"):
plan.reasons.append(f"covered by the path-filtered macOS MLX workflow: {path}")
else:
plan.add_lanes("performance", reason=f"benchmark implementation: {path}")
continue
if path.startswith("fastvideo/tests/eval/") or path.startswith("fastvideo/eval/"):
plan.add_lanes("eval", reason=f"evaluation coverage: {path}")
continue
if path.startswith("fastvideo/third_party/eval/"):
plan.add_lanes("eval", reason=f"vendored evaluation implementation: {path}")
continue
if path.startswith("fastvideo/tests/lora_extraction/") or path.startswith("scripts/lora_extraction/"):
plan.add_lanes("lora-extraction", reason=f"LoRA extraction coverage: {path}")
continue
if path.startswith("fastvideo/tests/inference/lora/"):
plan.add_lanes("lora-inference", reason=f"LoRA inference coverage: {path}")
continue
if path.startswith("fastvideo/tests/inference/vmoba/"):
plan.add_lanes("inference-vmoba", reason=f"VMoBA inference coverage: {path}")
continue
if path.startswith(("fastvideo/dataset/", "fastvideo/workflow/", "fastvideo/pipelines/preprocess/",
"fastvideo/pipelines/training/")):
plan.add_lanes(*ALL_TRAINING_LANES, reason=f"shared data/training input surface: {path}")
continue
if path.startswith("fastvideo/tests/train/") or path.startswith("fastvideo/train/"):
plan.add_lanes("train-framework", reason=f"modular training coverage: {path}")
continue
if path.startswith("fastvideo/tests/training/"):
lowered = path.lower()
if "/vanilla/" in lowered:
plan.add_lanes("training", reason=f"vanilla training coverage: {path}")
elif "/distill/" in lowered:
plan.add_lanes("distillation", reason=f"distillation coverage: {path}")
elif "/self-forcing/" in lowered:
plan.add_lanes("self-forcing", reason=f"self-forcing coverage: {path}")
elif "/lora/" in lowered:
plan.add_lanes("lora-training", reason=f"LoRA training coverage: {path}")
elif "/vsa/" in lowered:
plan.add_lanes("training-vsa", reason=f"VSA training coverage: {path}")
else:
plan.add_lanes(*LEGACY_TRAINING_LANES, reason=f"shared legacy training coverage: {path}")
continue
if path.startswith("fastvideo/training/"):
lowered = path.lower()
if "self_forcing" in lowered:
plan.add_lanes("self-forcing", reason=f"self-forcing implementation: {path}")
elif "distill" in lowered:
plan.add_lanes("distillation", reason=f"distillation implementation: {path}")
elif "lora" in lowered:
plan.add_lanes("lora-training", reason=f"LoRA training implementation: {path}")
else:
plan.add_lanes(*LEGACY_TRAINING_LANES, reason=f"shared legacy training implementation: {path}")
continue
lowered = path.lower()
if "vmoba" in lowered and path.startswith(("fastvideo/", ".buildkite/")):
plan.add_lanes("inference-vmoba", reason=f"VMoBA implementation: {path}")
plan.add_golden(("test_wan_t2v.py", ), reason=f"VMoBA end-to-end coverage: {path}")
continue
if "lora" in lowered and path.startswith("fastvideo/"):
plan.add_lanes(
"lora-inference",
"lora-extraction",
"lora-training",
reason=f"shared LoRA implementation: {path}",
)
_select_output_coverage(plan, path)
continue
if path.startswith("fastvideo/entrypoints/") or path.startswith("fastvideo/api/"):
plan.add_lanes("api-server", reason=f"API/entrypoint integration: {path}")
if "openai" not in lowered and "/cli/" not in lowered:
_select_output_coverage(plan, path)
continue
if path.startswith("fastvideo/worker/"):
plan.add_lanes("api-server", reason=f"worker/API integration: {path}")
_select_output_coverage(plan, path)
continue
if path.startswith("fastvideo/distributed/"):
plan.add_lanes(
"training",
"train-framework",
reason=f"distributed runtime integration: {path}",
)
_select_output_coverage(plan, path)
continue
if path.startswith(("fastvideo/hooks/", "fastvideo/platforms/", "fastvideo/third_party/")):
_select_output_coverage(plan, path)
continue
if path.startswith(("fastvideo/models/", "fastvideo/pipelines/", "fastvideo/configs/",
"fastvideo/layers/", "fastvideo/attention/")):
_select_output_coverage(plan, path)
continue
if path in {
"fastvideo/fastvideo_args.py",
"fastvideo/forward_context.py",
"fastvideo/image_processor.py",
"fastvideo/registry.py",
"fastvideo/utils.py",
}:
_select_output_coverage(plan, path)
continue
if path.startswith("fastvideo/mlx_runtime/"):
plan.reasons.append(f"covered by the path-filtered macOS MLX workflow: {path}")
continue
if path.startswith("fastvideo/logging_utils/") or path in {
"fastvideo/__init__.py",
"fastvideo/envs.py",
"fastvideo/logger.py",
"fastvideo/profiler.py",
"fastvideo/version.py",
}:
plan.reasons.append(f"covered by automatic Fastcheck: {path}")
continue
if path.startswith(("fastvideo-kernel/", "csrc/")):
plan.add_golden(("test_wan_t2v.py", ), reason=f"kernel integration smoke: {path}")
plan.add_ssim(("test_wan_t2v_similarity.py", ), reason=f"kernel numerical smoke: {path}")
continue
if path.startswith("apps/dreamverse/"):
# DreamVerse is already one of the six automatic Fastcheck lanes.
plan.reasons.append(f"covered by automatic DreamVerse Fastcheck: {path}")
continue
if path.startswith("fastvideo/tests/"):
# The automatic unit/component Fastcheck lanes own the remaining
# package tests. Domain-specific expensive test roots were handled
# above.
plan.reasons.append(f"covered by automatic Fastcheck: {path}")
continue
if path in {".buildkite/scripts/unit_test.sh", ".buildkite/scripts/pr_test.sh"}:
plan.reasons.append(f"covered by automatic unit Fastcheck: {path}")
continue
if _matches_any(path, SAFE_PATTERNS):
plan.reasons.append(f"no additional GPU integration needed: {path}")
continue
plan.require_all(f"unclassified path; failing closed: {path}")
return plan
def _write_github_output(output: TextIO, plan: MergePlan) -> None:
output.write(f"merge_test_plan={plan.encoded_lanes()}\n")
output.write(f"merge_golden_tests={plan.encoded_golden_tests()}\n")
output.write(f"merge_ssim_tests={plan.encoded_ssim_tests()}\n")
output.write(f"merge_plan_label={','.join(plan.ordered_lanes()) or 'none'}\n")
def _write_summary(output: TextIO, plan: MergePlan) -> None:
output.write("## Change-aware merge test plan\n\n")
output.write("| Selection | Value |\n|---|---|\n")
output.write(f"| Additional Slurm lanes | `{','.join(plan.ordered_lanes()) or 'none'}` |\n")
output.write(f"| Golden tests | `{plan.encoded_golden_tests()}` |\n")
output.write(f"| SSIM tests | `{plan.encoded_ssim_tests()}` |\n\n")
output.write("Fastcheck remains the universal six-lane baseline.\n")
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--paths-file", type=Path, required=True)
parser.add_argument("--github-output", type=Path)
parser.add_argument("--summary-file", type=Path)
return parser.parse_args()
def main() -> int:
args = parse_args()
paths = args.paths_file.read_text(encoding="utf-8").splitlines()
plan = classify_paths(paths)
print(f"MERGE_TEST_PLAN={plan.encoded_lanes()}")
print(f"MERGE_GOLDEN_TESTS={plan.encoded_golden_tests()}")
print(f"MERGE_SSIM_TESTS={plan.encoded_ssim_tests()}")
for reason in plan.reasons:
print(f"- {reason}")
if args.github_output:
with args.github_output.open("a", encoding="utf-8") as output:
_write_github_output(output, plan)
if args.summary_file:
with args.summary_file.open("a", encoding="utf-8") as output:
_write_summary(output, plan)
return 0
if __name__ == "__main__":
raise SystemExit(main())
+1 -1
View File
@@ -53,7 +53,7 @@ PC_PENDING='{"name": "pre-commit", "id": 1, "status": "in_progress", "conclusion
DOCS_OK='{"name": "Deploy Documentation", "id": 2, "status": "completed", "conclusion": "success"}'
DOCS_BAD='{"name": "Deploy Documentation", "id": 2, "status": "completed", "conclusion": "failure"}'
DOCS_CANCELLED='{"name": "Deploy Documentation", "id": 2, "status": "completed", "conclusion": "cancelled"}'
OTHER='{"name": "Trigger Full Suite", "id": 3, "status": "in_progress", "conclusion": null}'
OTHER='{"name": "Trigger Merge Gate", "id": 3, "status": "in_progress", "conclusion": null}'
NULL_NAME='{"name": null, "id": 4, "status": "completed", "conclusion": "failure"}'
PC_OK_RERUN='{"name": "pre-commit", "id": 5, "status": "completed", "conclusion": "success"}'
@@ -190,6 +190,7 @@ jobs:
if: ${{ !inputs.push_by_digest }}
run: |
echo "✅ Python ${{ inputs.python_version }} image successfully built and pushed to ${{ steps.image.outputs.name }}:${{ inputs.tag_suffix }}-sha-${GITHUB_SHA::7}"
echo "Digest: ${{ steps.build-push.outputs.digest }}"
echo "To run tests with this image, manually trigger the 'Run Tests' workflow."
- name: Digest success message
+40 -25
View File
@@ -11,6 +11,7 @@ jobs:
if: >-
github.event.context == 'direct-test-completed'
&& github.event.state == 'success'
&& (vars.CI_GPU_BACKEND == '' || vars.CI_GPU_BACKEND == 'slurm')
runs-on: ubuntu-latest
steps:
- name: Check and update aggregate status
@@ -26,29 +27,48 @@ jobs:
per_page: 100,
});
const bkStatuses = data.statuses.filter(
s => s.context.startsWith('buildkite/ci/')
);
const FASTCHECK_PREFIX = 'buildkite/ci/microscope-';
// Buildkite derives the GitHub context prefix from the label emoji.
// Keep hard Full Suite lanes in test-tube/bar-chart namespaces and
// Fastcheck lanes in microscope so targeted reruns cannot clear the
// wrong aggregate status. Automatic PR jobs use pr-fastcheck while
// slash-command and Full Suite jobs use ci; normalize the suffix
// and keep the newest status for each logical lane.
const FASTCHECK_PREFIXES = [
'buildkite/pr-fastcheck/microscope-',
'buildkite/ci/microscope-',
];
const FULL_SUITE_PREFIXES = [
'buildkite/ci/test-tube-',
'buildkite/ci/bar-chart-',
];
const fastcheck = bkStatuses.filter(
s => s.context.startsWith(FASTCHECK_PREFIX)
);
const fullSuite = bkStatuses.filter(
s => FULL_SUITE_PREFIXES.some(p => s.context.startsWith(p))
);
function newestByLane(prefixes) {
const statuses = new Map();
for (const status of data.statuses) {
const prefix = prefixes.find(p => status.context.startsWith(p));
if (!prefix) continue;
const lane = status.context.slice(prefix.length);
const previous = statuses.get(lane);
if (!previous || Date.parse(status.updated_at) > Date.parse(previous.updated_at)) {
statuses.set(lane, status);
}
}
return statuses;
}
if (
fastcheck.length > 0
&& fastcheck.every(s => s.state === 'success')
) {
const fastcheck = newestByLane(FASTCHECK_PREFIXES);
const fullSuiteOnly = newestByLane(FULL_SUITE_PREFIXES);
const fastcheckPassed =
fastcheck.size === 6
&& [...fastcheck.values()].every(s => s.state === 'success');
const fullSuitePassed =
fastcheckPassed
&& fullSuiteOnly.size === 14
&& [...fullSuiteOnly.values()].every(s => s.state === 'success');
if (fastcheckPassed) {
core.info(
`All ${fastcheck.length} fastcheck tests passed — updating fastcheck-passed`
`All ${fastcheck.size} fastcheck tests passed — updating fastcheck-passed`
);
await github.rest.repos.createCommitStatus({
owner: context.repo.owner,
@@ -56,17 +76,13 @@ jobs:
sha,
state: 'success',
context: 'fastcheck-passed',
description:
`All ${fastcheck.length} fastcheck tests passed`,
description: `All ${fastcheck.size} fastcheck tests passed`,
});
}
if (
fullSuite.length > 0
&& fullSuite.every(s => s.state === 'success')
) {
if (fullSuitePassed) {
core.info(
`All ${fullSuite.length} full suite tests passed — updating full-suite-passed`
'All 20 full suite tests passed — updating full-suite-passed'
);
await github.rest.repos.createCommitStatus({
owner: context.repo.owner,
@@ -74,7 +90,6 @@ jobs:
sha,
state: 'success',
context: 'full-suite-passed',
description:
`All ${fullSuite.length} full suite tests passed`,
description: 'All 20 full suite tests passed',
});
}
@@ -0,0 +1,62 @@
name: Promote Selected GPU Backend Status
on:
status:
permissions:
statuses: write
concurrency:
group: gpu-ci-status-${{ github.event.sha }}-${{ vars.CI_GPU_BACKEND }}
cancel-in-progress: false
jobs:
promote:
if: >-
(vars.CI_GPU_BACKEND == 'modal' || vars.CI_GPU_BACKEND == 'vllm')
&& (github.event.context == format('gpu-ci/{0}/fastcheck-passed', vars.CI_GPU_BACKEND)
|| github.event.context == format('gpu-ci/{0}/full-suite-passed', vars.CI_GPU_BACKEND))
runs-on: ubuntu-latest
env:
SELECTED_BACKEND: ${{ vars.CI_GPU_BACKEND }}
steps:
- name: Mirror the selected backend's latest suite results
uses: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea # v7.0.1
with:
script: |
const backend = process.env.SELECTED_BACKEND;
if (!['modal', 'vllm'].includes(backend)) {
throw new Error('Unsupported selected GPU backend');
}
const sha = context.payload.sha;
// Read current state after entering the serialized workflow. A
// delayed event must not overwrite a newer failure with success.
const statuses = await github.paginate(github.rest.repos.listCommitStatusesForRef, {
owner: context.repo.owner,
repo: context.repo.repo,
ref: sha,
per_page: 100,
});
for (const suffix of ['fastcheck-passed', 'full-suite-passed']) {
const sourceContext = `gpu-ci/${backend}/${suffix}`;
const matches = statuses.filter(status => status.context === sourceContext);
matches.sort((a, b) =>
Date.parse(b.updated_at) - Date.parse(a.updated_at) || b.id - a.id
);
const latest = matches[0];
const state = latest ? latest.state : 'pending';
if (!['pending', 'success', 'failure', 'error'].includes(state)) {
throw new Error(`Unsupported status state for ${sourceContext}`);
}
await github.rest.repos.createCommitStatus({
owner: context.repo.owner,
repo: context.repo.repo,
sha,
context: suffix,
state,
description: latest
? `${backend} ${suffix}: ${state}`
: `Waiting for ${backend} ${suffix}`,
...(latest && latest.target_url ? {target_url: latest.target_url} : {}),
});
}
+22 -2
View File
@@ -8,6 +8,8 @@ on:
- "fastvideo/mlx_runtime/**"
- "fastvideo/tests/mlx/**"
- "fastvideo/tests/platforms/test_mps_vsa_error.py"
- "fastvideo/tests/platforms/test_cpu_sdpa.py"
- "fastvideo/platforms/cpu.py"
- "fastvideo/platforms/mps.py"
- "fastvideo/platforms/__init__.py"
- "fastvideo/__init__.py"
@@ -78,11 +80,19 @@ jobs:
fastvideo/tests/mlx/test_mlx_dit_parity.py \
fastvideo/tests/mlx/test_mlx_compile_parity.py \
fastvideo/tests/mlx/test_mlx_checkpoint.py \
fastvideo/tests/mlx/test_mlx_checkpoint_compat.py \
fastvideo/tests/mlx/test_mlx_affine_dq_gemm.py \
fastvideo/tests/mlx/test_mlx_minimax_h3_parity.py \
fastvideo/tests/mlx/test_mlx_minimax_h3_vsa.py \
fastvideo/tests/mlx/test_mlx_minimax_h3_vsa_regressions.py \
fastvideo/tests/mlx/test_mlx_minimax_h3_fast_mode.py \
fastvideo/tests/mlx/test_mlx_minimax_h3_fast_spatial.py \
fastvideo/tests/mlx/test_mlx_fastwan_benchmark.py \
fastvideo/tests/mlx/test_taehv_decode.py \
fastvideo/tests/mlx/test_frame_upsample.py \
fastvideo/tests/mlx/test_mlx_fast_spatial.py \
fastvideo/tests/mlx/test_mlx_refine.py \
fastvideo/tests/mlx/test_mlx_prompt_enhance.py \
fastvideo/tests/mlx/test_mlx_prompt_to_video_decode.py \
fastvideo/tests/mlx/test_mlx_wan22_prompt_cache_fingerprint.py \
fastvideo/tests/mlx/test_wan22_sample.py \
@@ -90,7 +100,8 @@ jobs:
fastvideo/tests/mlx/test_mlx_rife_interpolation.py::test_rife_download_unavailable_has_specific_error \
fastvideo/tests/mlx/test_mlx_rife_interpolation.py::test_rife_backend_regression_is_not_skip_eligible \
fastvideo/tests/platforms/test_mps_vsa_error.py \
-q
fastvideo/tests/platforms/test_cpu_sdpa.py \
-v -s -o faulthandler_timeout=120
# Same tests on MLX's CPU backend. Hosted macOS runners are scarce and
# slower to schedule; this Linux job gives fast PR signal on the identical
@@ -134,11 +145,19 @@ jobs:
fastvideo/tests/mlx/test_mlx_dit_parity.py \
fastvideo/tests/mlx/test_mlx_compile_parity.py \
fastvideo/tests/mlx/test_mlx_checkpoint.py \
fastvideo/tests/mlx/test_mlx_checkpoint_compat.py \
fastvideo/tests/mlx/test_mlx_affine_dq_gemm.py \
fastvideo/tests/mlx/test_mlx_minimax_h3_parity.py \
fastvideo/tests/mlx/test_mlx_minimax_h3_vsa.py \
fastvideo/tests/mlx/test_mlx_minimax_h3_vsa_regressions.py \
fastvideo/tests/mlx/test_mlx_minimax_h3_fast_mode.py \
fastvideo/tests/mlx/test_mlx_minimax_h3_fast_spatial.py \
fastvideo/tests/mlx/test_mlx_fastwan_benchmark.py \
fastvideo/tests/mlx/test_taehv_decode.py \
fastvideo/tests/mlx/test_frame_upsample.py \
fastvideo/tests/mlx/test_mlx_fast_spatial.py \
fastvideo/tests/mlx/test_mlx_refine.py \
fastvideo/tests/mlx/test_mlx_prompt_enhance.py \
fastvideo/tests/mlx/test_mlx_prompt_to_video_decode.py \
fastvideo/tests/mlx/test_mlx_wan22_prompt_cache_fingerprint.py \
fastvideo/tests/mlx/test_wan22_sample.py \
@@ -146,4 +165,5 @@ jobs:
fastvideo/tests/mlx/test_mlx_rife_interpolation.py::test_rife_download_unavailable_has_specific_error \
fastvideo/tests/mlx/test_mlx_rife_interpolation.py::test_rife_backend_regression_is_not_skip_eligible \
fastvideo/tests/platforms/test_mps_vsa_error.py \
-q
fastvideo/tests/platforms/test_cpu_sdpa.py \
-v -s -o faulthandler_timeout=120
+44
View File
@@ -0,0 +1,44 @@
name: Scheduled Full SSIM
on:
schedule:
- cron: "0 5 * * 0"
workflow_dispatch:
permissions:
contents: read
jobs:
trigger:
if: github.repository == 'hao-ai-lab/FastVideo'
runs-on: ubuntu-latest
steps:
- name: Trigger weekly full SSIM on Slinky Slurm
env:
BUILDKITE_API_TOKEN: ${{ secrets.BUILDKITE_API_TOKEN }}
SOURCE_SHA: ${{ github.sha }}
SOURCE_BRANCH: ${{ github.event.repository.default_branch }}
BK_ORG: ${{ vars.BUILDKITE_ORG_SLUG }}
BK_PIPELINE: ${{ vars.BUILDKITE_PIPELINE_SLUG }}
run: |
set -euo pipefail
curl -sS --fail-with-body -X POST \
"https://api.buildkite.com/v2/organizations/${BK_ORG}/pipelines/${BK_PIPELINE}/builds" \
-H "Authorization: Bearer $BUILDKITE_API_TOKEN" \
-H "Content-Type: application/json" \
--data-raw "$(jq -n \
--arg commit "$SOURCE_SHA" \
--arg branch "$SOURCE_BRANCH" \
'{
commit: $commit,
branch: $branch,
message: "Weekly full SSIM on Slinky Slurm",
ignore_pipeline_branch_filters: true,
env: {
TEST_SCOPE: "scheduled",
FULL_SUITE: "false",
TEST_TYPE: "ssim",
PR_NUMBER: "false",
PR_TITLE: "Scheduled full SSIM"
}
}')"
+12 -44
View File
@@ -33,7 +33,6 @@ jobs:
core.setOutput('has_write', String(hasWrite));
- name: Add ready label and react
id: label
if: steps.perm.outputs.has_write == 'true'
uses: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea # v7.0.1
with:
@@ -48,47 +47,6 @@ jobs:
comment_id: context.payload.comment.id,
content: 'rocket',
});
const { data: pr } = await github.rest.pulls.get({ owner, repo, pull_number: prNumber });
core.setOutput('pr_sha', pr.head.sha);
core.setOutput('pr_branch', pr.head.ref);
core.setOutput('pr_number', String(prNumber));
core.setOutput('pr_title', pr.title);
- name: Trigger Full Suite
if: steps.perm.outputs.has_write == 'true'
env:
BUILDKITE_API_TOKEN: ${{ secrets.BUILDKITE_API_TOKEN }}
PR_SHA: ${{ steps.label.outputs.pr_sha }}
PR_BRANCH: ${{ steps.label.outputs.pr_branch }}
PR_NUMBER: ${{ steps.label.outputs.pr_number }}
PR_TITLE: ${{ steps.label.outputs.pr_title }}
BK_ORG: ${{ vars.BUILDKITE_ORG_SLUG }}
BK_PIPELINE: ${{ vars.BUILDKITE_PIPELINE_SLUG }}
run: |
curl -sS --fail-with-body -X POST \
"https://api.buildkite.com/v2/organizations/${BK_ORG}/pipelines/${BK_PIPELINE}/builds" \
-H "Authorization: Bearer $BUILDKITE_API_TOKEN" \
-H "Content-Type: application/json" \
--data-raw "$(jq -n \
--arg commit "$PR_SHA" \
--arg branch "$PR_BRANCH" \
--arg message "Full Suite for PR #${PR_NUMBER} (via /merge)" \
--arg pr_title "$PR_TITLE" \
--argjson pr_id "$PR_NUMBER" \
'{
commit: $commit,
branch: $branch,
message: $message,
ignore_pipeline_branch_filters: true,
pull_request_id: $pr_id,
pull_request_base_branch: "main",
env: {
TEST_SCOPE: "full",
FULL_SUITE: "true",
PR_NUMBER: ($pr_id | tostring),
PR_TITLE: $pr_title
}
}')"
parse-command:
if: >-
@@ -129,7 +87,7 @@ jobs:
set -euo pipefail
TEST_NAME=$(echo "$COMMENT" | grep -oP '(?<=/test\s)\S+' | head -1 || true)
VALID="encoder vae transformer kernel unit dreamverse ssim golden-gate training lora-inference lora-training lora-extraction distillation self-forcing vsa vmoba performance api train-framework eval full fastcheck pre-commit"
VALID="encoder vae transformer kernel unit dreamverse ssim golden-gate training lora-inference lora-training lora-extraction distillation self-forcing vsa vmoba performance api train-framework eval unit-ci kernel-ci dreamverse-ci ssim-ci golden-gate-ci encoder-ci vae-ci transformer-ci lora-inference-ci lora-training-ci lora-extraction-ci training-ci distillation-ci self-forcing-ci vsa-ci vmoba-ci performance-ci api-ci train-framework-ci eval-ci full fastcheck pre-commit"
if [ -z "$TEST_NAME" ] || ! echo "$VALID" | grep -qw "$TEST_NAME"; then
echo "Unknown test: '$TEST_NAME'. Valid: $VALID"
exit 1
@@ -137,7 +95,17 @@ jobs:
declare -A MAP=(
[encoder]=encoder [vae]=vae [transformer]=transformer
[kernel]=kernel_tests [unit]=unit_test [dreamverse]=dreamverse_app
[kernel]=kernel_tests [unit]=unit_test [unit-ci]=unit_test_ci
[kernel-ci]=kernel_tests_ci [dreamverse-ci]=dreamverse_app_ci
[ssim-ci]=ssim_ci [vmoba-ci]=inference_vmoba_ci
[golden-gate-ci]=golden_gate_ci [training-ci]=training_ci
[encoder-ci]=encoder_ci [vae-ci]=vae_ci [transformer-ci]=transformer_ci
[lora-inference-ci]=inference_lora_ci [lora-training-ci]=training_lora_ci
[lora-extraction-ci]=lora_extraction_ci [distillation-ci]=distillation_dmd_ci
[self-forcing-ci]=self_forcing_ci [vsa-ci]=training_vsa_ci
[performance-ci]=performance_ci [api-ci]=api_server_ci
[train-framework-ci]=train_framework_ci [eval-ci]=eval_ci
[dreamverse]=dreamverse_app
[ssim]=ssim [golden-gate]=golden_gate [training]=training
[lora-inference]=inference_lora [lora-training]=training_lora
[lora-extraction]=lora_extraction
+66 -13
View File
@@ -1,4 +1,4 @@
name: Trigger Full Suite
name: Trigger Merge Gate
on:
pull_request_target:
@@ -10,7 +10,7 @@ permissions:
actions: read
concurrency:
group: full-suite-${{ github.event.pull_request.number }}
group: merge-gate-${{ github.event.pull_request.number }}
cancel-in-progress: false
jobs:
@@ -34,29 +34,72 @@ jobs:
});
const hasReady = pr.labels.some(l => l.name === 'ready');
core.setOutput('has_ready', String(hasReady));
if (!hasReady) core.info('No ready label — skipping Full Suite trigger.');
core.setOutput('changed_files', String(pr.changed_files));
if (!hasReady) core.info('No ready label — skipping merge-gate trigger.');
- name: Cancel previous Buildkite builds
if: steps.check.outputs.has_ready == 'true'
env:
BUILDKITE_API_TOKEN: ${{ secrets.BUILDKITE_API_TOKEN }}
PR_BRANCH: ${{ github.event.pull_request.head.ref }}
PR_NUMBER: ${{ github.event.pull_request.number }}
run: |
# Find running builds for this branch with TEST_SCOPE=full and cancel them
builds=$(curl -sS -H "Authorization: Bearer $BUILDKITE_API_TOKEN" \
"https://api.buildkite.com/v2/organizations/${{ vars.BUILDKITE_ORG_SLUG }}/pipelines/${{ vars.BUILDKITE_PIPELINE_SLUG }}/builds?branch=${PR_BRANCH}&state=running,scheduled" \
| jq -r '.[] | select(try (.env.TEST_SCOPE == "full") catch false) | .number')
# Match both branch and PR number: forks can reuse the same branch name.
builds=$(curl -sS --get -H "Authorization: Bearer $BUILDKITE_API_TOKEN" \
--data-urlencode "branch=$PR_BRANCH" \
--data-urlencode "state=running,scheduled" \
"https://api.buildkite.com/v2/organizations/${{ vars.BUILDKITE_ORG_SLUG }}/pipelines/${{ vars.BUILDKITE_PIPELINE_SLUG }}/builds" \
| jq -r --arg pr_number "$PR_NUMBER" \
'.[] | select((.env.TEST_SCOPE? == "merge") and (.env.PR_NUMBER? == $pr_number)) | .number')
for build_num in $builds; do
echo "Cancelling Buildkite build #$build_num"
curl -sS -X PUT -H "Authorization: Bearer $BUILDKITE_API_TOKEN" \
"https://api.buildkite.com/v2/organizations/${{ vars.BUILDKITE_ORG_SLUG }}/pipelines/${{ vars.BUILDKITE_PIPELINE_SLUG }}/builds/${build_num}/cancel"
done
# Checks out the BASE branch (default for pull_request_target), so PR
# authors cannot tamper with the gate script.
- name: Checkout gate script
# Check out the immutable BASE SHA: pull_request_target must never run a
# planner or gate script from the untrusted PR head.
- name: Checkout trusted merge planner
if: steps.check.outputs.has_ready == 'true'
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
ref: ${{ github.event.pull_request.base.sha }}
persist-credentials: false
- name: Collect changed paths
if: steps.check.outputs.has_ready == 'true'
env:
GH_TOKEN: ${{ github.token }}
PR_NUMBER: ${{ github.event.pull_request.number }}
EXPECTED_CHANGED_FILES: ${{ steps.check.outputs.changed_files }}
run: |
set -euo pipefail
changed_json="$RUNNER_TEMP/merge-changed-files.json"
changed_paths="$RUNNER_TEMP/merge-changed-paths.txt"
if gh api --paginate --slurp \
"repos/${GITHUB_REPOSITORY}/pulls/${PR_NUMBER}/files?per_page=100" \
> "$changed_json"; then
observed=$(jq '[.[][] | .filename] | unique | length' "$changed_json")
if [ "$observed" = "$EXPECTED_CHANGED_FILES" ]; then
jq -r '.[][] | .filename, (.previous_filename // empty)' "$changed_json" \
| sort -u > "$changed_paths"
else
echo "::warning::Changed-file API returned $observed of $EXPECTED_CHANGED_FILES paths; selecting all merge lanes."
echo '__FASTVIDEO_CI_PLAN_ALL__' > "$changed_paths"
fi
else
echo "::warning::Changed-file API failed; selecting all merge lanes."
echo '__FASTVIDEO_CI_PLAN_ALL__' > "$changed_paths"
fi
- name: Select minimal merge tests
id: plan
if: steps.check.outputs.has_ready == 'true'
run: |
python3 .github/scripts/plan_merge_ci.py \
--paths-file "$RUNNER_TEMP/merge-changed-paths.txt" \
--github-output "$GITHUB_OUTPUT" \
--summary-file "$GITHUB_STEP_SUMMARY"
- name: Wait for pre-commit and docs build
if: steps.check.outputs.has_ready == 'true'
@@ -66,7 +109,7 @@ jobs:
PR_NUMBER: ${{ github.event.pull_request.number }}
run: bash .github/scripts/gate_full_suite.sh
- name: Trigger Buildkite Full Suite
- name: Trigger Buildkite merge gate
if: steps.check.outputs.has_ready == 'true'
env:
BUILDKITE_API_TOKEN: ${{ secrets.BUILDKITE_API_TOKEN }}
@@ -76,6 +119,10 @@ jobs:
PR_TITLE: ${{ github.event.pull_request.title }}
BK_ORG: ${{ vars.BUILDKITE_ORG_SLUG }}
BK_PIPELINE: ${{ vars.BUILDKITE_PIPELINE_SLUG }}
MERGE_TEST_PLAN: ${{ steps.plan.outputs.merge_test_plan }}
MERGE_GOLDEN_TESTS: ${{ steps.plan.outputs.merge_golden_tests }}
MERGE_SSIM_TESTS: ${{ steps.plan.outputs.merge_ssim_tests }}
MERGE_PLAN_LABEL: ${{ steps.plan.outputs.merge_plan_label }}
run: |
curl -sS --fail-with-body -X POST \
"https://api.buildkite.com/v2/organizations/${BK_ORG}/pipelines/${BK_PIPELINE}/builds" \
@@ -84,8 +131,11 @@ jobs:
--data-raw "$(jq -n \
--arg commit "$PR_SHA" \
--arg branch "$PR_BRANCH" \
--arg message "Full Suite for PR #${PR_NUMBER}" \
--arg message "Merge gate [${MERGE_PLAN_LABEL}] for PR #${PR_NUMBER}" \
--arg pr_title "$PR_TITLE" \
--arg merge_test_plan "$MERGE_TEST_PLAN" \
--arg merge_golden_tests "$MERGE_GOLDEN_TESTS" \
--arg merge_ssim_tests "$MERGE_SSIM_TESTS" \
--argjson pr_id "$PR_NUMBER" \
'{
commit: $commit,
@@ -95,8 +145,11 @@ jobs:
pull_request_id: $pr_id,
pull_request_base_branch: "main",
env: {
TEST_SCOPE: "full",
TEST_SCOPE: "merge",
FULL_SUITE: "true",
MERGE_TEST_PLAN: $merge_test_plan,
MERGE_GOLDEN_TESTS: $merge_golden_tests,
MERGE_SSIM_TESTS: $merge_ssim_tests,
PR_NUMBER: ($pr_id | tostring),
PR_TITLE: $pr_title
}
+4 -4
View File
@@ -38,17 +38,17 @@ jobs:
**How our CI works:**
PRs run a two-tier CI system:
PRs run a three-tier CI system:
1. **Pre-commit** — formatting (yapf), linting (ruff), type checking (mypy). Runs immediately on every PR.
2. **Fastcheck** — core GPU tests (encoders, VAEs, transformers, kernels, unit tests). Runs automatically via Buildkite on relevant file changes (~10-15 min).
3. **Full Suite** — integration tests, training pipelines, SSIM regression. Runs only when a reviewer adds the `ready` label.
2. **Fastcheck** — six core GPU lanes run automatically via Buildkite (~10-15 min).
3. **Merge gate** — a reviewer adds `ready`; changed paths select only the relevant integration, training, golden, or SSIM coverage.
**Before your PR is reviewed:**
- [ ] `pre-commit run --all-files` passes locally
- [ ] You've added or updated tests for your changes
- [ ] The PR description explains what and why
If pre-commit fails, a bot comment will explain how to fix it. Fastcheck and Full Suite results appear in the Checks section below.
If pre-commit fails, a bot comment will explain how to fix it. Fastcheck and merge-gate results appear in the Checks section below.
**Useful links:**
- [Contributing Guide](https://hao-ai-lab.github.io/FastVideo/contributing/overview/)
+28
View File
@@ -13,6 +13,11 @@ on:
required: false
default: false
type: boolean
build_ci_runner_image:
description: 'Build the ARM64 CUDA 13 CI runner image (sm_100)'
required: false
default: false
type: boolean
# Auto-rebuild the CUDA images when a repository-controlled image input
# changes on main. This includes the trusted SM89 kernel artifact's source,
# metadata/key helper, ABI dependency metadata, and build orchestration.
@@ -198,6 +203,29 @@ jobs:
docker buildx imagetools create "${TAG_ARGS[@]}" "${IMAGE_REFS[@]}"
docker buildx imagetools inspect "${TAGS[0]}"
# The CI runner is ARM64 like DGX Spark, but targets sm_100a rather than sm_121.
# The architecture-specific target includes the GB200 VSA CUDA extensions.
# Publish a single-architecture variant so the self-hosted CI runner can reuse
# the exact prebuilt kernel instead of compiling it in every job.
build-ci-runner-image:
if: ${{ (github.event_name == 'push' && github.repository == 'hao-ai-lab/FastVideo') || github.event.inputs.build_ci_runner_image == 'true' }}
uses: ./.github/workflows/_template-build-image.yml
with:
python_version: '3.12'
dockerfile_path: docker/Dockerfile
tag_suffix: py3.12-cuda13.0.0-sm100
runner: ubuntu-24.04-arm
architecture: arm64
build_args: |
PYTHON_VERSION=3.12
CUDA_VERSION=13.0.0
UV_TORCH_BACKEND=cu130
TORCH_CUDA_ARCH_LIST=10.0a
CMAKE_BUILD_PARALLEL_LEVEL=1
FLASH_ATTN_WHEEL_TAG=cu130torch2.12
FLASH_ATTN_WHEEL_RELEASE_ARM64=https://github.com/mjun0812/flash-attention-prebuild-wheels/releases/download/v0.9.22
secrets: inherit
# Dreamverse matrix: {backend, UI} x {12.6.3, 13.0.0}, Python 3.12. Torch backend
# matches the base CUDA (cu126 / cu130). Keep these images amd64-only until the
# required FA4 dependency stack is available and validated on arm64.
+2
View File
@@ -6,6 +6,7 @@ on:
paths:
- 'docs/**'
- 'examples/**'
- 'scripts/inference/**'
- 'mkdocs.yml'
- 'requirements-mkdocs.in'
- 'requirements-mkdocs.txt'
@@ -16,6 +17,7 @@ on:
paths:
- 'docs/**'
- 'examples/**'
- 'scripts/inference/**'
- 'mkdocs.yml'
- 'requirements-mkdocs.in'
- 'requirements-mkdocs.txt'
+15 -8
View File
@@ -62,8 +62,9 @@ jobs:
cuda-version: '13.0.0'
torch-cuda-short: 'cu130'
platform:
# x86_64 builds the full cu126 + cu130 set (cu130 ships the consumer
# Blackwell sm_120a FP4 kernels).
# x86_64 builds the full cu126 + cu130 set. cu130 ships the
# data-center Blackwell sm_100a/sm_103a VSA and consumer sm_120a FP4
# kernels.
- os: ubuntu-22.04
arch: x86_64
wheel-plat: manylinux_2_35_x86_64
@@ -124,7 +125,7 @@ jobs:
- name: Install dependencies (GCC, Clang, CUDA Paths, Git)
run: |
sudo apt update
sudo apt install -y git patchelf gcc-11 g++-11 clang-11
sudo apt install -y git gcc-11 g++-11 clang-11
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-11 100 --slave /usr/bin/g++ g++ /usr/bin/g++-11
# Allow Git to Access Safe Directory
@@ -168,17 +169,18 @@ jobs:
# covers sm_120a; turbodiffusion covers sm_100a+sm_120a. The sm_100 FP4
# forward is the FA4 CuTe DSL path in the fastvideo package (PR #1221),
# JIT-compiled at runtime — not built into this wheel.
# * x86_64 cu130 = Hopper TK + consumer Blackwell sm_120a FP4.
# * x86_64 cu130 = Hopper TK + data-center Blackwell sm_100a/sm_103a VSA
# + consumer Blackwell sm_120a FP4.
# * x86_64 cu126 = Hopper TK only (older drivers; CUDA < 12.8 has no FP4).
# The per-arch split in CMakeLists pins the FP4 targets to sm_120a and builds
# the main extension for the full arch list. CMAKE_BUILD_PARALLEL_LEVEL caps
# Ninja so heavy CUTLASS/TK template TUs don't OOM the 16 GB runner (exit 143).
if [ "${{ matrix.platform.arch }}" = "aarch64" ]; then
export TORCH_CUDA_ARCH_LIST="10.0a;12.0a"
export TORCH_CUDA_ARCH_LIST="10.0a;10.3a;12.0a"
export CMAKE_ARGS="${CMAKE_ARGS:-} -DFASTVIDEO_KERNEL_BUILD_TK=OFF -DFASTVIDEO_KERNEL_BUILD_ATTN_QAT_INFER=ON"
export CMAKE_BUILD_PARALLEL_LEVEL=1
elif [ "${{ matrix.torch-cuda.torch-cuda-short }}" = "cu130" ]; then
export TORCH_CUDA_ARCH_LIST="9.0a;12.0a"
export TORCH_CUDA_ARCH_LIST="9.0a;10.0a;10.3a;12.0a"
export CMAKE_ARGS="${CMAKE_ARGS:-} -DFASTVIDEO_KERNEL_BUILD_TK=ON -DFASTVIDEO_KERNEL_BUILD_ATTN_QAT_INFER=ON -DCMAKE_CUDA_ARCHITECTURES=90a"
# A single FP4 TU (attn_qat_infer) can use ~8-12 GB on its own, so serialize.
export CMAKE_BUILD_PARALLEL_LEVEL=1
@@ -194,7 +196,11 @@ jobs:
python -m build --wheel --outdir dist
# Fix the wheel to be manylinux compliant
uv pip install --system auditwheel
# Ubuntu 22.04 ships patchelf 0.14.3, while current auditwheel
# requires at least 0.14.5. Use the stable PyPI binary on both
# x86_64 and aarch64 release runners.
uv pip install --system auditwheel patchelf==0.17.2.4
patchelf --version
# Point auditwheel at torch libs, but do not vendor them into the wheel.
TORCH_LIB_DIR=$(python - <<'PY'
import os
@@ -211,7 +217,8 @@ jobs:
--exclude libtorch.so \
--exclude libc10.so \
--exclude libc10_cuda.so \
--exclude libtorch_python.so
--exclude libtorch_python.so \
--exclude libnccl.so.2
# Move fixed wheels back to dist for upload consistency
rm dist/*.whl
mv fixed_dist/*.whl dist/
+5
View File
@@ -55,6 +55,8 @@ eggs/
# MkDocs documentation
site/
docs/assets/cookbook-serving.json
examples/serving/clients/node_modules/
docs/getting_started/examples/
docs/examples/
docs/inference/examples/
@@ -133,6 +135,9 @@ fastvideo/tests/ssim/reference_videos/**
!fastvideo/tests/ssim/reference_videos/**/*.mp4
!fastvideo/tests/ssim/reference_videos/**/*.png
# Local H3 MLX kernel / exactness benches (JSON, logs, frames, videos)
.kernel_bench/
# Editor logs and local Python version pins (accidentally committed)
*.nvimlog
.nvimlog
+1 -1
View File
@@ -9,7 +9,7 @@ exclude: |
tests/.*|
scripts/.*|
fastvideo/dataset/.*|
fastvideo/models/.*|
fastvideo/models/(?!wan/(config|vae_config|pipeline_config|definition|__init__)\.py$).*|
^apps/dreamverse/web/.*|
examples/.*|
\.agents/.*|
+4
View File
@@ -66,14 +66,18 @@ Local guidance lives next to the code. Read the in-scope file before editing:
| `fastvideo/AGENTS.md` | Core package map, public API, registry-driven model dispatch |
| `fastvideo/configs/AGENTS.md` | Arch + pipeline config dataclasses, `param_names_mapping` |
| `fastvideo/models/AGENTS.md` | DiT / VAE / encoder / scheduler / loader layout (pre-commit excluded) |
| `fastvideo/models/wan/AGENTS.md` | Wan family-local transformers, VAE, configs, and the SP sharding invariant |
| `fastvideo/layers/AGENTS.md` | Tensor-parallel linear/attention layer rules for ports |
| `fastvideo/attention/AGENTS.md` | Backend registry + env-var override |
| `fastvideo/pipelines/AGENTS.md` | Stage ABC, `basic/<model>/`, `preprocess/`, presets |
| `fastvideo/pipelines/basic/wan/AGENTS.md` | Wan sampling stages, first-frame conditioning, DMD/causal boundaries |
| `fastvideo/pipelines/basic/magi_human/AGENTS.md` | MagiHuman umbrella repo, lazy-loaded components, packing invariants |
| `fastvideo/training/AGENTS.md` | Legacy monolithic pipelines (frozen for existing models) |
| `fastvideo/train/AGENTS.md` | New modular trainer (methods × models × callbacks, YAML) |
| `fastvideo/tests/AGENTS.md` | Test taxonomy, conftest, pre-commit-excluded path |
| `fastvideo/tests/ssim/AGENTS.md` | GPU SSIM regression authoring + reference video sync |
| `scripts/checkpoint_conversion/AGENTS.md` | Adding a converter for a new HF/official checkpoint |
| `apps/dreamverse/AGENTS.md` | DreamVerse app structure and conventions |
## Critical: Two Training Stacks Coexist
+11 -7
View File
@@ -3,12 +3,16 @@
</div>
<p align="center">
| <a href="https://hao-ai-lab.github.io/FastVideo"><b>Documentation</b></a> | <a href="https://hao-ai-lab.github.io/FastVideo/inference/inference_quick_start/"><b> Quick Start</b></a> | <a href="https://github.com/hao-ai-lab/FastVideo/discussions/982" target="_blank"><b>Weekly Dev Meeting</b></a> | 🟣💬 <a href="https://join.slack.com/t/fastvideo/shared_invite/zt-3f4lao1uq-u~Ipx6Lt4J27AlD2y~IdLQ" target="_blank"> <b>Slack</b> </a> | 🟣💬 <a href="https://github.com/hao-ai-lab/FastVideo/discussions/1097" target="_blank"> <b> WeChat </b> </a> |
| <a href="https://hao-ai-lab.github.io/FastVideo"><b>Documentation</b></a> | <a href="https://haoailab.com/FastVideo/cookbook/"><b>Cookbook</b></a> | <a href="https://hao-ai-lab.github.io/FastVideo/inference/inference_quick_start/"><b> Quick Start</b></a> | <a href="https://github.com/hao-ai-lab/FastVideo/discussions/982" target="_blank"><b>Weekly Dev Meeting</b></a> | 🟣💬 <a href="https://join.slack.com/t/fastvideo/shared_invite/zt-3f4lao1uq-u~Ipx6Lt4J27AlD2y~IdLQ" target="_blank"> <b>Slack</b> </a> | 🟣💬 <a href="https://github.com/hao-ai-lab/FastVideo/discussions/1097" target="_blank"> <b> WeChat </b> </a> |
</p>
**FastVideo is a unified post-training and real-time inference framework for accelerated video generation.**
## NEWS
- `2026/09/15`: Release [FastH3 8-Step V2](https://huggingface.co/FastVideo/FastVideo-FastH3-8-Step-V2), an eight-forward data-free DMD2 checkpoint distilled from MiniMax-H3 with 80% Video Sparse Attention. Run it with `examples/inference/basic/basic_fasth3_8step.py` or the [FastH3 8-Step V2 recipe](https://haoailab.com/FastVideo/cookbook/minimax-h3/).
- `2026/09/01`: FastH3 now runs locally on Apple Silicon through MLX and on NVIDIA DGX Spark through CUDA 13, including two-Spark inference. Follow the [FastH3 recipes](https://haoailab.com/FastVideo/cookbook/minimax-h3/) and read the [Blog](https://haoailab.com/blogs/fasth3-local/).
- `2026/08/27`: [FastH3 Preview v1](https://haoailab.com/blogs/fasth3-preview/) is an open-weight 4-step sparse-distilled MiniMax-H3 model for synchronized video-and-audio generation, developed in collaboration with [Nuva Lab](https://nuvalab.ai/) and the [NVIDIA FastGen team](https://github.com/NVlabs/FastGen). Download the recommended [VSA / Data-Free weights](https://huggingface.co/FastVideo/FastVideo-FastH3-4-step-Preview-v1-VSA-DataFree), or see the [full FastH3 collection](https://huggingface.co/collections/FastVideo/fastvideo-fasth3).
- `2026/08/19`: FastVideo now supports MLX on Apple Silicon with [FastMetal-QAD](https://huggingface.co/collections/FastVideo/fastmetal), a family of 1.3B, 5B, and 14B models optimized for Mac. Follow the [MLX install guide](https://hao-ai-lab.github.io/FastVideo/getting_started/installation/mlx/) and read the [Blog](https://haoailab.com/blogs/fastmetal/).
- `2026/06/23`: Release FastWan-QAD: 5s of Video generated in 1.8s E2E. See the [FastWan-QAD models](https://huggingface.co/FastVideo/FastWan-QAD-FP8-1.3B), [Attn-QAT training guide](https://haoailab.com/FastVideo/training/attn_qat/), and [blog](https://haoailab.com/blogs/fastwan-qad/).
- `2026/03/17`: Release demo: Into the Dreamverse: Vibe Directing in FastVideo, check out the [Blog](https://haoailab.com/blogs/dreamverse/).
- `2026/03/13`: Release demo: Create a 5s 1080p Video in 4.5s with FastVideo on a Single GPU, check out the [Blog](https://haoailab.com/blogs/fastvideo_realtime_1080p/).
@@ -60,12 +64,12 @@ UV_TORCH_BACKEND=cu126 uv pip install fastvideo
```
Use `UV_TORCH_BACKEND=cu130` on CUDA 13. Apple silicon users should follow the
[MPS installation guide](https://hao-ai-lab.github.io/FastVideo/getting_started/installation/mps/).
[MLX install guide](https://hao-ai-lab.github.io/FastVideo/getting_started/installation/mlx/).
> **On an Apple Silicon Mac?** FastVideo runs FastWan text-to-video natively
> through an MLX runtime — a 5-second 480p clip generated locally, no cloud,
> no discrete GPU. Install with `uv pip install -e '.[mlx]'` and follow the
> [Apple Silicon guide](https://hao-ai-lab.github.io/FastVideo/getting_started/installation/mps/).
> **On an Apple Silicon Mac?** Install with `uv pip install -e '.[mlx]'` from
> a clone, then pick a recipe in the
> [cookbook](https://haoailab.com/FastVideo/cookbook/). See the
> [MLX install guide](https://hao-ai-lab.github.io/FastVideo/getting_started/installation/mlx/).
Please see our [docs](https://hao-ai-lab.github.io/FastVideo/getting_started/installation/) for more detailed installation instructions.
@@ -83,7 +87,7 @@ Install FastVideo (https://github.com/hao-ai-lab/FastVideo) into a fresh uv virt
https://hao-ai-lab.github.io/FastVideo/getting_started/installation/):
- NVIDIA GPU, x86_64 -> docs/getting_started/installation/gpu.md
- NVIDIA DGX Spark / GB10, aarch64, CUDA 13 -> docs/getting_started/installation/spark.md
- Apple Silicon, macOS -> docs/getting_started/installation/mps.md
- Apple Silicon, macOS -> docs/getting_started/installation/mlx.md
3. Use uv for every step. If a command fails, debug it and tell me what you changed.
4. Verify the result:
python -c "import fastvideo, torch; print('cuda', torch.cuda.is_available())"
+23 -2
View File
@@ -97,13 +97,33 @@ dreamverse-server --port 8009
dreamverse-mock-server --port 8009
```
### Run Dreamverse with FastH3
Select the VSA data-free FastH3 Preview profile when you start the backend:
```bash
DREAMVERSE_MODEL_ID=fast-h3 dreamverse-server --port 8009
```
The `fast-h3` profile uses four visible GPUs by default. It loads the `MiniMaxAI/MiniMax-H3` base checkpoint and the
`vsa-datafree/adapter_model.safetensors` adapter from
`FastVideo/FastVideo-FastH3-4-step-Preview-v1-LoRA`. Each request generates a 124-frame, 768×1344 video with
synchronized audio and five sigma-grid points. Dreamverse uses the last frame of each segment as first-frame
conditioning for the following segment.
Set `CUDA_VISIBLE_DEVICES` when you need to choose the four physical GPUs:
```bash
CUDA_VISIBLE_DEVICES=0,1,2,3 DREAMVERSE_MODEL_ID=fast-h3 dreamverse-server --port 8009
```
> **Expect a slow first boot.** With `torch.compile` and startup warmup enabled
> (the default), the backend compiles the segment 1 and segment 2 inference
> paths before it reports ready — this can take **tens of minutes on a cold
> cache**, regardless of how you deploy (local, server, Docker, or Modal).
> `/healthz` responds as soon as the process is up; `/readyz` stays `503` until
> warmup finishes. For a faster, uncompiled startup while testing, set
> `FASTVIDEO_ENABLE_STARTUP_WARMUP=0` before starting the backend.
> warmup finishes. To defer compilation until the first generated request while
> testing, set `FASTVIDEO_ENABLE_STARTUP_WARMUP=0` before starting the backend.
## Frontend Setup
@@ -219,6 +239,7 @@ selection, and mock-server behavior:
pytest apps/dreamverse/dreamverse/tests/test_config.py \
apps/dreamverse/dreamverse/tests/test_entrypoints.py \
apps/dreamverse/dreamverse/tests/test_gpu_pool.py \
apps/dreamverse/dreamverse/tests/test_minimax_h3_generation.py \
apps/dreamverse/dreamverse/tests/test_mock_server.py -q
```
+12 -1
View File
@@ -139,7 +139,18 @@ session.
- startup warmup
- user join/leave commands
- `USER_STEP` execution for each segment
- continuation state between segments
- generation-command routing and stream-result delivery
Model generation has a separate ownership boundary inside each GPU process:
- `apps/dreamverse/dreamverse/generation_worker.py` selects the backend that the active model profile declares and owns
the backend lifecycle.
- `apps/dreamverse/dreamverse/ltx2_generation.py` owns LTX-2 generator configuration, video and audio continuation, and
runtime LoRA application.
- `apps/dreamverse/dreamverse/minimax_h3_generation.py` owns the VSA data-free FastH3 adapter, FastH3 generator and
request configuration, and last-frame continuation through MiniMax H3 first-frame conditioning.
- `apps/dreamverse/dreamverse/generation_contracts.py` defines the decoded media and stream-trimming result that both
model backends return to `apps/dreamverse/dreamverse/gpu_pool.py`.
`apps/dreamverse/dreamverse/prompt_enhancer.py` manages:
@@ -1,6 +1,6 @@
"""Benchmark the LTX-2 generation pipeline driven by the dreamverse Python SDK path.
Mirrors how ``apps/dreamverse/dreamverse/video_generation.py`` constructs
Mirrors how ``apps/dreamverse/dreamverse/ltx2_generation.py`` constructs
``GeneratorConfig`` and calls ``VideoGenerator.generate()``, then
captures per-stage timings via the ``FASTVIDEO_STAGE_LOGGING=1`` log
hooks (same mechanism as ``FastVideo-internal/examples/inference/basic/
+21 -2
View File
@@ -1,5 +1,6 @@
import os
from pathlib import Path
from typing import cast
_REPO_ROOT = Path(__file__).resolve().parents[1]
_SERVER_ROOT = Path(__file__).resolve().parent
@@ -55,16 +56,34 @@ FRONTEND_STATIC_DIR_CANDIDATES = _resolve_frontend_static_dir_candidates()
MODEL_REGISTRY = {
"fast-ltx2": {
"name": "FastLTX2",
"generation_backend": "ltx2",
"default_sp_size": 1,
"model_path": "FastVideo/LTX2-Distilled-Diffusers",
"config_model_path": "FastVideo/LTX2-Distilled-Diffusers",
"lora_repo": "FastVideo/LTX2-OmniNFT-LoRA",
},
"fast-ltx23": {
"name": "FastLTX23",
"generation_backend": "ltx2",
"default_sp_size": 1,
"model_path": "FastVideo/LTX-2.3-Distilled-Diffusers",
"config_model_path": "FastVideo/LTX-2.3-Distilled-Diffusers",
"lora_repo": "FastVideo/LTX-2.3-OmniNFT-LoRA",
},
"fast-h3": {
"name": "FastH3",
"generation_backend": "minimax_h3",
"default_sp_size": 4,
"model_path": "MiniMaxAI/MiniMax-H3",
"adapter_repo": "FastVideo/FastVideo-FastH3-4-step-Preview-v1-LoRA",
"adapter_filename": "vsa-datafree/adapter_model.safetensors",
"attention_backend": "VIDEO_SPARSE_ATTN_H3",
"height": 768,
"width": 1344,
"num_frames": 124,
"num_inference_steps": 5,
"seed": 1000,
},
}
DEFAULT_MODEL_ID = "fast-ltx2"
@@ -171,7 +190,7 @@ def _optional_env(*names: str) -> str | None:
DEVTOOLS_ENABLED = _env_bool("FASTVIDEO_ENABLE_DEVTOOLS", False)
PROMPT_SAFETY_ENABLED = _env_bool("FASTVIDEO_ENABLE_PROMPT_SAFETY", False)
DREAMVERSE_MAX_AUTOTUNE = _env_bool("DREAMVERSE_MAX_AUTOTUNE", True)
DREAMVERSE_SP_SIZE = max(1, _env_int("DREAMVERSE_SP_SIZE", 1))
DREAMVERSE_SP_SIZE = max(1, _env_int("DREAMVERSE_SP_SIZE", cast(int, MODEL_CONFIG["default_sp_size"])))
DREAMVERSE_MODEL_PATH = (os.getenv("DREAMVERSE_MODEL_PATH", "").strip() or None)
if DREAMVERSE_MODEL_PATH:
@@ -213,7 +232,7 @@ def _resolve_lora_spec(spec: str) -> str | None:
if not spec:
return None
if spec.lower() == "omninft":
return MODEL_CONFIG.get("lora_repo")
return cast(str | None, MODEL_CONFIG.get("lora_repo"))
if spec.lower() in AVAILABLE_LORAS:
return AVAILABLE_LORAS[spec.lower()]["repo"]
return spec
@@ -0,0 +1,46 @@
"""Shared contract between DreamVerse generation backends and GPU workers."""
from __future__ import annotations
from dataclasses import dataclass
from typing import Any, Protocol
@dataclass
class StepResult:
"""Decoded media and stream-trimming metadata for one DreamVerse segment."""
frames: list
audio: Any
audio_sample_rate: int | None
timings: dict[str, float]
head_trim_frames: int
head_trim_audio_frames: int
class GenerationBackend(Protocol):
"""Model-owned generation operations used by one GPU worker process."""
def initialize(self, model_config: dict | None = None) -> None:
...
def shutdown(self) -> None:
...
def clear_conditioning(self) -> None:
...
def generate_step(
self,
prompt: str,
segment_idx: int,
image_path: str | None,
reset_conditioning: bool,
) -> StepResult:
...
def warmup(self, prompt: str) -> dict[str, float]:
...
def apply_lora_stack(self, stack: list[tuple[str, float]]) -> tuple[str | None, str | None]:
...
@@ -0,0 +1,96 @@
"""Select and own one model-specific generation backend per GPU process."""
from __future__ import annotations
from dreamverse.config import MODEL_CONFIG
from dreamverse.generation_contracts import GenerationBackend, StepResult
def _create_generation_backend(backend_name: str, gpu_id: int) -> GenerationBackend:
"""Construct the backend that owns the selected model family's behavior."""
if backend_name == "ltx2":
from dreamverse.ltx2_generation import LTX2GenerationBackend
return LTX2GenerationBackend(gpu_id)
if backend_name == "minimax_h3":
from dreamverse.minimax_h3_generation import MiniMaxH3GenerationBackend
return MiniMaxH3GenerationBackend(gpu_id)
raise ValueError(f"Unsupported DreamVerse generation backend: {backend_name!r}")
class VideoGenerationWorker:
"""Delegate GPU lifecycle and generation calls to the active model backend."""
def __init__(self, gpu_id: int):
self.gpu_id = gpu_id
self.model_config: dict = dict(MODEL_CONFIG)
self.backend_name: str | None = None
self.backend: GenerationBackend | None = None
def initialize(self, model_config: dict | None = None) -> None:
"""Load the requested model through its generation backend.
Model selection belongs here so the GPU process and streaming layers
use one stable media contract without importing model-specific code.
"""
requested_model_config = dict(model_config) if model_config is not None else dict(self.model_config)
backend_name = requested_model_config.get("generation_backend")
if not isinstance(backend_name, str) or not backend_name:
raise ValueError("DreamVerse model configuration requires `generation_backend`.")
candidate_backend = self.backend
if candidate_backend is None or self.backend_name != backend_name:
if candidate_backend is not None:
candidate_backend.shutdown()
candidate_backend = _create_generation_backend(backend_name, self.gpu_id)
try:
candidate_backend.initialize(requested_model_config)
except Exception:
try:
candidate_backend.shutdown()
except Exception as shutdown_error:
print(f"[GPU {self.gpu_id}] Backend cleanup after initialization failure: {shutdown_error}")
self.backend = None
self.backend_name = None
raise
self.model_config = requested_model_config
self.backend = candidate_backend
self.backend_name = backend_name
def _require_backend(self) -> GenerationBackend:
"""Return the initialized backend or fail before processing a command."""
if self.backend is None:
raise RuntimeError("Generation backend is not initialized.")
return self.backend
def shutdown(self) -> None:
"""Release model resources owned by the selected backend."""
if self.backend is not None:
self.backend.shutdown()
def clear_conditioning(self) -> None:
self._require_backend().clear_conditioning()
def generate_step(
self,
prompt: str,
segment_idx: int,
image_path: str | None,
reset_conditioning: bool,
) -> StepResult:
"""Generate one segment through the selected model backend."""
return self._require_backend().generate_step(
prompt,
segment_idx,
image_path,
reset_conditioning,
)
def warmup(self, prompt: str) -> dict[str, float]:
return self._require_backend().warmup(prompt)
def apply_lora_stack(self, stack: list[tuple[str, float]]) -> tuple[str | None, str | None]:
return self._require_backend().apply_lora_stack(stack)
+17 -10
View File
@@ -12,7 +12,7 @@ from enum import Enum
from multiprocessing import Process, Queue
from dreamverse.config import (
DEFAULT_MODEL_ID,
ACTIVE_MODEL_ID,
DREAMVERSE_SP_SIZE,
MODEL_REGISTRY,
STARTUP_WARMUP_ENABLED,
@@ -54,7 +54,7 @@ from dreamverse.worker_ipc import (
def _parse_requested_gpu_limit() -> int | None:
raw_value = os.getenv("FASTVIDEO_GPU_COUNT", "").strip().lower()
if not raw_value:
return 1
return DREAMVERSE_SP_SIZE
if raw_value == "all":
return None
try:
@@ -164,12 +164,12 @@ def gpu_worker_process(
os.environ["CUDA_VISIBLE_DEVICES"] = cuda_device
os.environ["FASTVIDEO_ATTENTION_BACKEND"] = "FLASH_ATTN"
from dreamverse.video_generation import VideoGenerationWorker
from dreamverse.generation_worker import VideoGenerationWorker
worker = VideoGenerationWorker(gpu_id)
def event_loop(first_cmd: Command = None):
"""Blocking event loop for LTX2; dispatches user commands."""
"""Block on generation commands after the model is initialized."""
print(f"[GPU {gpu_id}] Entering event loop")
def handle_command(cmd: Command):
@@ -435,7 +435,7 @@ class GPUSlot:
self._response_reader_task: asyncio.Task | None = None
self._active: bool = False
self._reader_lock: asyncio.Lock | None = None
self.current_model_id: str = DEFAULT_MODEL_ID
self.current_model_id: str | None = ACTIVE_MODEL_ID
self.shared_stream_buffer = None
self.shared_stream_buffer_size = SHARED_STREAM_BUFFER_BYTES
@@ -690,7 +690,7 @@ class GPUSlot:
async def join_user(self, user_id: str, model_id: str = None) -> JoinAck:
"""Add a user to this GPU."""
if model_id is None:
model_id = DEFAULT_MODEL_ID
model_id = ACTIVE_MODEL_ID
# Reload model if a different one is requested
if model_id != self.current_model_id and model_id in MODEL_REGISTRY:
@@ -705,16 +705,23 @@ class GPUSlot:
self.connected_users.clear()
model_config = MODEL_REGISTRY[model_id]
reload_response = await self._send_command(Command(CommandType.RELOAD_MODEL,
payload=ReloadModelPayload(model_config=model_config),
user_id="__reload__"),
timeout=600.0)
try:
reload_response = await self._send_command(Command(
CommandType.RELOAD_MODEL,
payload=ReloadModelPayload(model_config=model_config),
user_id="__reload__"),
timeout=600.0)
except Exception:
self.current_model_id = None
raise
match reload_response:
case ReloadAck():
pass
case WorkerError(message=msg):
self.current_model_id = None
raise RuntimeError(f"Model reload failed: {msg}")
case _:
self.current_model_id = None
raise RuntimeError(f"Unexpected reload response: "
f"{type(reload_response).__name__}")
@@ -1,9 +1,9 @@
"""LTX2 model lifecycle and continuation conditioning.
"""LTX-2 model lifecycle and continuation conditioning.
Runs inside a GPU worker subprocess. Owns the model, the audio
encoder, and the per-session continuation state carried across
segments. Callers must set ``os.environ["CUDA_VISIBLE_DEVICES"]``
before constructing ``VideoGenerationWorker`` — all ``fastvideo.*``
before constructing ``LTX2GenerationBackend`` — all ``fastvideo.*``
imports are deferred to method bodies so nothing touches CUDA at
module import time.
"""
@@ -14,9 +14,6 @@ import gc
import os
import re
import time
from dataclasses import dataclass
from typing import Any
import numpy as np
import torch
@@ -35,6 +32,7 @@ from dreamverse.config import (
DREAMVERSE_LORA_STACK,
_resolve_lora_spec,
)
from dreamverse.generation_contracts import StepResult
# Multi-frame decoded continuation defaults from
# examples/inference/basic/basic_ltx2_distilled_video_continuation.py.
@@ -80,22 +78,6 @@ def _reset_lora_registry(worker) -> dict:
return {"status": "lora_registry_reset"}
@dataclass
class StepResult:
"""Output of one generation step.
``head_trim_frames`` / ``head_trim_audio_frames`` are derived here
so downstream AV streaming never needs to import conditioning
constants.
"""
frames: list
audio: Any
audio_sample_rate: int | None
timings: dict
head_trim_frames: int
head_trim_audio_frames: int
class ContinuationState:
"""Per-session video + audio conditioning carried across segments."""
@@ -202,7 +184,7 @@ class ContinuationState:
self.audio_latents = latents.detach().clone().cpu()
class VideoGenerationWorker:
class LTX2GenerationBackend:
"""Single-GPU LTX2 generator with continuation state.
Caller must set ``os.environ["CUDA_VISIBLE_DEVICES"]`` before
@@ -0,0 +1,297 @@
"""FastH3 model lifecycle and first-frame continuation for DreamVerse."""
from __future__ import annotations
import gc
import os
import time
from typing import TYPE_CHECKING, Any
import numpy as np
import torch
from dreamverse.config import DREAMVERSE_SP_SIZE
from dreamverse.generation_contracts import StepResult
if TYPE_CHECKING:
from PIL.Image import Image
def _required_config_str(model_config: dict, field_name: str) -> str:
"""Read one required non-empty string from a DreamVerse model profile."""
value = model_config.get(field_name)
if not isinstance(value, str) or not value.strip():
raise ValueError(f"FastH3 model configuration requires `{field_name}`.")
return value.strip()
class MiniMaxH3GenerationBackend:
"""Run the VSA data-free FastH3 adapter and retain one continuation frame."""
def __init__(self, gpu_id: int):
self.gpu_id = gpu_id
self.generator: Any | None = None
self.model_config: dict = {}
self.continuation_image: Image | None = None
def _gpu_mem(self) -> str:
allocated_gib = torch.cuda.memory_allocated() / 1024**3
reserved_gib = torch.cuda.memory_reserved() / 1024**3
return f"alloc={allocated_gib:.2f}GiB, reserved={reserved_gib:.2f}GiB"
@staticmethod
def _configure_environment(attention_backend: str) -> None:
"""Apply the fixed boot-time switches from the FastH3 reference recipe."""
os.environ.update({
"FASTVIDEO_ATTENTION_BACKEND": attention_backend,
"FASTVIDEO_FA4": "1",
"FASTVIDEO_MINIMAX_H3_FUSIONS": "all",
"FASTVIDEO_VSA_SM100A": "0",
})
os.environ.pop("FASTVIDEO_INFERENCE_TORCH_COMPILE", None)
def initialize(self, model_config: dict | None = None) -> None:
"""Download the fixed Preview adapter and load the FastH3 generator.
The model profile owns the base checkpoint, adapter file, attention
backend, and generation geometry. The backend translates that profile
into FastVideo's typed generator configuration.
"""
if model_config is not None:
self.model_config = dict(model_config)
if not self.model_config:
raise ValueError("FastH3 initialization requires a model configuration.")
if self.generator is not None:
self.generator.shutdown()
self.generator = None
gc.collect()
torch.cuda.empty_cache()
self.clear_conditioning()
model_path = _required_config_str(self.model_config, "model_path")
adapter_repo = _required_config_str(self.model_config, "adapter_repo")
adapter_filename = _required_config_str(self.model_config, "adapter_filename")
attention_backend = _required_config_str(self.model_config, "attention_backend")
self._configure_environment(attention_backend)
from huggingface_hub import hf_hub_download
from fastvideo import VideoGenerator
from fastvideo.api import (
CompileConfig,
ComponentConfig,
EngineConfig,
GeneratorConfig,
OffloadConfig,
ParallelismConfig,
PipelineSelection,
)
adapter_path = hf_hub_download(repo_id=adapter_repo, filename=adapter_filename)
experimental = {
"attention_backend": attention_backend,
"inference_torch_compile": attention_backend == "FLASH_ATTN",
"vae_parallel_decode": True,
"vae_parallel_decode_strategy": "gather",
}
if attention_backend == "VIDEO_SPARSE_ATTN_H3":
experimental.update({
"VSA_sparsity": 0.9,
"VSA_tile_size": 64,
})
generator_config = GeneratorConfig(
model_path=model_path,
pipeline=PipelineSelection(
components=ComponentConfig(lora_path=adapter_path, lora_strength=1.0),
experimental=experimental,
),
engine=EngineConfig(
num_gpus=DREAMVERSE_SP_SIZE,
parallelism=ParallelismConfig(tp_size=1, sp_size=DREAMVERSE_SP_SIZE),
offload=OffloadConfig(
dit=False,
dit_layerwise=False,
text_encoder=True,
image_encoder=True,
vae=True,
pin_cpu_memory=True,
),
compile=CompileConfig(enabled=False, vae_enabled=True),
use_fsdp_inference=False,
),
)
print(f"[GPU {self.gpu_id}] Loading FastH3 model: {model_path}")
print(f"[GPU {self.gpu_id}] FastH3 adapter: {adapter_repo}/{adapter_filename}")
print(f"[GPU {self.gpu_id}] Before model load: {self._gpu_mem()}")
self.generator = VideoGenerator.from_config(generator_config)
print(f"[GPU {self.gpu_id}] FastH3 loaded: {self._gpu_mem()} (warmup pending)")
def shutdown(self) -> None:
"""Release the FastVideo generator and cached continuation image."""
self.clear_conditioning()
if self.generator is not None:
self.generator.shutdown()
self.generator = None
def clear_conditioning(self) -> None:
"""Release the first-frame image retained for the next segment."""
if self.continuation_image is not None:
self.continuation_image.close()
self.continuation_image = None
@staticmethod
def _load_rgb_image(image_path: str) -> Image:
"""Load an image into an independent RGB buffer with no open file handle."""
from PIL import Image
with Image.open(image_path) as image:
return image.convert("RGB").copy()
def _select_conditioning_image(
self,
segment_idx: int,
image_path: str | None,
reset_conditioning: bool,
) -> tuple[Image | None, bool]:
"""Select the initial upload or retained last frame for one segment."""
if reset_conditioning:
self.clear_conditioning()
if segment_idx > 1 and self.continuation_image is not None:
return self.continuation_image.copy(), True
if segment_idx > 1 and not reset_conditioning:
raise RuntimeError(f"FastH3 segment {segment_idx} requires a retained continuation frame.")
if segment_idx == 1 and image_path:
return self._load_rgb_image(image_path), False
return None, False
def _build_request(self, prompt: str, conditioning_image: Image | None):
"""Build the typed FastVideo request owned by the FastH3 profile."""
from fastvideo.api import GenerationRequest, InputConfig, OutputConfig, SamplingConfig
return GenerationRequest(
prompt=prompt,
negative_prompt="",
inputs=InputConfig(pil_image=conditioning_image),
sampling=SamplingConfig(
height=int(self.model_config["height"]),
width=int(self.model_config["width"]),
num_frames=int(self.model_config["num_frames"]),
fps=24,
num_inference_steps=int(self.model_config["num_inference_steps"]),
guidance_scale=1.0,
batch_cfg=False,
seed=int(self.model_config["seed"]),
),
output=OutputConfig(save_video=False, return_frames=True),
)
def _save_continuation_frame(self, frames: list) -> None:
"""Retain the last decoded frame as first-frame conditioning."""
from PIL import Image
self.clear_conditioning()
self.continuation_image = Image.fromarray(np.ascontiguousarray(frames[-1])).convert("RGB")
def generate_step(
self,
prompt: str,
segment_idx: int,
image_path: str | None,
reset_conditioning: bool,
) -> StepResult:
"""Generate one synchronized FastH3 segment and retain its last frame.
Later segments use MiniMax H3's first-frame-to-video path. The first
conditioned frame and its matching audio duration are trimmed before
streaming so adjacent segments do not duplicate media.
"""
if self.generator is None:
raise RuntimeError("FastH3 generator is not initialized.")
conditioning_image, uses_continuation = self._select_conditioning_image(
segment_idx,
image_path,
reset_conditioning,
)
request = self._build_request(prompt, conditioning_image)
started = time.perf_counter()
try:
result = self.generator.generate(request)
finally:
if conditioning_image is not None:
conditioning_image.close()
torch.cuda.synchronize()
generation_ms = (time.perf_counter() - started) * 1000.0
if isinstance(result, list):
raise RuntimeError("FastH3 returned multiple results for one DreamVerse segment.")
frames = result.frames
if not isinstance(frames, list) or not frames:
raise RuntimeError("FastH3 generation did not return decoded frames.")
audio = result.audio
audio_sample_rate = result.audio_sample_rate
if audio is not None and audio_sample_rate is None:
raise RuntimeError("FastH3 returned audio without an audio sample rate.")
save_started = time.perf_counter()
self._save_continuation_frame(frames)
save_conditioning_ms = (time.perf_counter() - save_started) * 1000.0
timings = {
"generation_ms": generation_ms,
"generation_time_ms": float(result.generation_time or 0.0) * 1000.0,
"save_conditioning_ms": save_conditioning_ms,
"e2e_latency_ms": (time.perf_counter() - started) * 1000.0,
}
trim_frames = 1 if uses_continuation else 0
print(f"[GPU {self.gpu_id}] FastH3 segment {segment_idx}: "
f"{len(frames)} frames, gen={generation_ms:.0f}ms, "
f"save_conditioning={save_conditioning_ms:.0f}ms, "
f"e2e={timings['e2e_latency_ms']:.0f}ms")
return StepResult(
frames=frames,
audio=audio,
audio_sample_rate=audio_sample_rate,
timings=timings,
head_trim_frames=trim_frames,
head_trim_audio_frames=trim_frames,
)
def warmup(self, prompt: str) -> dict[str, float]:
"""Compile the FastH3 text and first-frame paths before readiness."""
warmup_prompt = (prompt or "").strip()
if not warmup_prompt:
raise RuntimeError("Startup warmup prompt must be non-empty.")
print(f"[GPU {self.gpu_id}] FastH3 startup warmup starting "
"(synthetic segments: text-to-video, first-frame-to-video)")
started = time.perf_counter()
text_result = self.generate_step(
warmup_prompt,
segment_idx=1,
image_path=None,
reset_conditioning=True,
)
first_frame_result = self.generate_step(
warmup_prompt,
segment_idx=2,
image_path=None,
reset_conditioning=False,
)
total_ms = (time.perf_counter() - started) * 1000.0
self.clear_conditioning()
text_ms = float(text_result.timings.get("e2e_latency_ms", 0.0))
first_frame_ms = float(first_frame_result.timings.get("e2e_latency_ms", 0.0))
print(f"[GPU {self.gpu_id}] FastH3 startup warmup complete: "
f"text_to_video={text_ms:.0f}ms, "
f"first_frame_to_video={first_frame_ms:.0f}ms, "
f"total={total_ms:.0f}ms")
return {
"warmup_text_to_video_ms": text_ms,
"warmup_first_frame_to_video_ms": first_frame_ms,
"warmup_total_ms": total_ms,
}
def apply_lora_stack(self, stack: list[tuple[str, float]]) -> tuple[str | None, str | None]:
"""Reject runtime LoRA mutation because FastH3 uses one startup adapter."""
del stack
raise RuntimeError("FastH3 uses its fixed startup adapter and does not support runtime LoRA changes.")
@@ -30,7 +30,7 @@ from dreamverse.session_init_image import cleanup_session_init_image, persist_se
from dreamverse.worker_ipc import MediaChunk, MediaComplete, MediaInit
from dreamverse.config import (
DEFAULT_MODEL_ID,
ACTIVE_MODEL_ID,
GENERATION_SEGMENT_CAP,
PROMPT_AUTO_SLEEP_MS,
PROMPT_AUTO_TIMEOUT_MS,
@@ -264,7 +264,7 @@ class SessionController:
timeout_task = asyncio.create_task(session_timeout())
# Join the engine on this GPU.
await slot.join_user(client_id, model_id=DEFAULT_MODEL_ID)
await slot.join_user(client_id, model_id=ACTIVE_MODEL_ID)
# Notify client they're connected to a GPU.
await ws_send_json({
@@ -2,13 +2,14 @@ from __future__ import annotations
import importlib.util
from pathlib import Path
from types import ModuleType
import pytest
SERVER_DIR = Path(__file__).resolve().parents[1]
def _load_config_module():
def _load_config_module() -> ModuleType:
spec = importlib.util.spec_from_file_location(
"server_config_test_module",
SERVER_DIR / "config.py",
@@ -20,7 +21,7 @@ def _load_config_module():
return module
def _set_required_prompt_keys(monkeypatch):
def _set_required_prompt_keys(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setenv("CEREBRAS_API_KEY", "cerebras-key")
monkeypatch.setenv("GROQ_API_KEY", "groq-key")
@@ -150,3 +151,38 @@ def test_config_rejects_invalid_prompt_provider(monkeypatch):
with pytest.raises(RuntimeError, match="Invalid FASTVIDEO_PROMPT_PROVIDER"):
_load_config_module()
def test_config_registers_vsa_datafree_fasth3_profile(monkeypatch):
"""The FastH3 registry entry owns the complete fixed Preview recipe."""
_set_required_prompt_keys(monkeypatch)
module = _load_config_module()
assert module.MODEL_REGISTRY["fast-h3"] == {
"name": "FastH3",
"generation_backend": "minimax_h3",
"default_sp_size": 4,
"model_path": "MiniMaxAI/MiniMax-H3",
"adapter_repo": "FastVideo/FastVideo-FastH3-4-step-Preview-v1-LoRA",
"adapter_filename": "vsa-datafree/adapter_model.safetensors",
"attention_backend": "VIDEO_SPARSE_ATTN_H3",
"height": 768,
"width": 1344,
"num_frames": 124,
"num_inference_steps": 5,
"seed": 1000,
}
def test_config_uses_fasth3_sequence_parallel_default(monkeypatch):
"""Selecting FastH3 defaults DreamVerse to its four-GPU topology."""
_set_required_prompt_keys(monkeypatch)
monkeypatch.setenv("DREAMVERSE_MODEL_ID", "fast-h3")
monkeypatch.delenv("DREAMVERSE_SP_SIZE", raising=False)
module = _load_config_module()
assert module.ACTIVE_MODEL_ID == "fast-h3"
assert module.MODEL_CONFIG["generation_backend"] == "minimax_h3"
assert module.DREAMVERSE_SP_SIZE == 4
@@ -0,0 +1,9 @@
from dreamverse.generation_worker import _create_generation_backend
from dreamverse.ltx2_generation import LTX2GenerationBackend
def test_create_generation_backend_ltx2_module_import():
backend = _create_generation_backend("ltx2", gpu_id=3)
assert isinstance(backend, LTX2GenerationBackend)
assert backend.gpu_id == 3
@@ -63,6 +63,14 @@ def test_get_available_gpus_defaults_to_first_visible_device(monkeypatch):
assert gpu_pool.get_available_gpus() == [3]
def test_get_available_gpus_defaults_to_active_model_sequence_parallel_size(monkeypatch):
monkeypatch.setenv("CUDA_VISIBLE_DEVICES", "0,1,2,3,4")
monkeypatch.delenv("FASTVIDEO_GPU_COUNT", raising=False)
monkeypatch.setattr(gpu_pool, "DREAMVERSE_SP_SIZE", 4)
assert gpu_pool.get_available_gpus() == [0, 1, 2, 3]
def test_get_available_gpus_rejects_invalid_gpu_count(monkeypatch):
monkeypatch.delenv("CUDA_VISIBLE_DEVICES", raising=False)
monkeypatch.setenv("FASTVIDEO_GPU_COUNT", "zero")
@@ -71,6 +79,23 @@ def test_get_available_gpus_rejects_invalid_gpu_count(monkeypatch):
gpu_pool.get_available_gpus()
def test_join_user_failed_reload_marks_model_uninitialized(monkeypatch):
"""A failed model reload forces the next join to reload a model."""
slot = gpu_pool.GPUSlot(gpu_id=0, cuda_device="0")
slot.current_model_id = "fast-ltx2"
async def fake_send_command(command, timeout):
del command, timeout
return gpu_pool.WorkerError(user_id="__reload__", message="load failed")
monkeypatch.setattr(slot, "_send_command", fake_send_command)
with pytest.raises(RuntimeError, match="Model reload failed"):
asyncio.run(slot.join_user("client-id", model_id="fast-h3"))
assert slot.current_model_id is None
def test_send_command_raises_on_worker_death():
"""A worker that consumes a command and exits without replying must
surface as RuntimeError via sentinel detection, not after the long
@@ -92,9 +117,9 @@ def test_send_command_raises_on_worker_death():
ready = resp_q.get(timeout=30.0)
assert ready == "READY"
async def runner():
async def runner() -> None:
slot = gpu_pool.GPUSlot(gpu_id=0, cuda_device="0")
slot.process = proc
slot.process = proc # type: ignore[assignment]
slot.command_queue = cmd_q
slot.response_queue = resp_q
@@ -17,11 +17,11 @@ FORBIDDEN_PREFIXES = (
)
ALLOWED_INTERNAL_IMPORTS = {
(
"video_generation.py",
"ltx2_generation.py",
"fastvideo.models.audio.ltx2_audio_processing",
),
(
"video_generation.py",
"ltx2_generation.py",
"fastvideo.models.loader.component_loader",
),
}
@@ -0,0 +1,247 @@
from __future__ import annotations
import os
from types import SimpleNamespace
from typing import Any
import numpy as np
import pytest
import dreamverse.generation_worker as generation_worker
from dreamverse.minimax_h3_generation import MiniMaxH3GenerationBackend
FASTH3_MODEL_CONFIG = {
"name": "FastH3",
"generation_backend": "minimax_h3",
"default_sp_size": 4,
"model_path": "MiniMaxAI/MiniMax-H3",
"adapter_repo": "FastVideo/FastVideo-FastH3-4-step-Preview-v1-LoRA",
"adapter_filename": "vsa-datafree/adapter_model.safetensors",
"attention_backend": "VIDEO_SPARSE_ATTN_H3",
"height": 768,
"width": 1344,
"num_frames": 124,
"num_inference_steps": 5,
"seed": 1000,
}
class _RecordingGenerator:
"""Record typed requests and return small synchronized media fixtures."""
def __init__(self) -> None:
self.requests: list[Any] = []
self.conditioning_pixels: list[np.ndarray | None] = []
def generate(self, request):
"""Capture the request and return two tiny video frames with audio."""
self.requests.append(request)
conditioning_image = request.inputs.pil_image
self.conditioning_pixels.append(
None if conditioning_image is None else np.asarray(conditioning_image).copy())
frames = [
np.full((2, 3, 3), 10, dtype=np.uint8),
np.full((2, 3, 3), 20, dtype=np.uint8),
]
return SimpleNamespace(
frames=frames,
audio=np.zeros((2, 16), dtype=np.float32),
audio_sample_rate=44100,
generation_time=0.25,
)
def test_initialize_builds_vsa_datafree_fasth3_generator(monkeypatch):
"""Initialization translates the DreamVerse profile into typed FastVideo config."""
from fastvideo import VideoGenerator
captured = {}
fake_generator = SimpleNamespace(shutdown=lambda: None)
def fake_from_config(config):
captured["config"] = config
return fake_generator
def fake_download(**kwargs):
captured["download"] = kwargs
return f"/models/{kwargs['filename']}"
monkeypatch.setattr("huggingface_hub.hf_hub_download", fake_download)
monkeypatch.setattr(VideoGenerator, "from_config", fake_from_config)
monkeypatch.setattr("dreamverse.minimax_h3_generation.DREAMVERSE_SP_SIZE", 4)
monkeypatch.setenv("FASTVIDEO_ATTENTION_BACKEND", "test-attention")
monkeypatch.setenv("FASTVIDEO_FA4", "0")
monkeypatch.setenv("FASTVIDEO_MINIMAX_H3_FUSIONS", "0")
monkeypatch.setenv("FASTVIDEO_VSA_SM100A", "1")
monkeypatch.setenv("FASTVIDEO_INFERENCE_TORCH_COMPILE", "1")
backend = MiniMaxH3GenerationBackend(gpu_id=0)
monkeypatch.setattr(backend, "_gpu_mem", lambda: "alloc=0.00GiB, reserved=0.00GiB")
backend.initialize(FASTH3_MODEL_CONFIG)
config = captured["config"]
assert captured["download"] == {
"repo_id": "FastVideo/FastVideo-FastH3-4-step-Preview-v1-LoRA",
"filename": "vsa-datafree/adapter_model.safetensors",
}
assert config.model_path == "MiniMaxAI/MiniMax-H3"
assert config.pipeline.components.lora_path.endswith("vsa-datafree/adapter_model.safetensors")
assert config.pipeline.components.lora_strength == 1.0
assert config.pipeline.experimental == {
"attention_backend": "VIDEO_SPARSE_ATTN_H3",
"inference_torch_compile": False,
"vae_parallel_decode": True,
"vae_parallel_decode_strategy": "gather",
"VSA_sparsity": 0.9,
"VSA_tile_size": 64,
}
assert config.engine.num_gpus == 4
assert config.engine.parallelism.tp_size == 1
assert config.engine.parallelism.sp_size == 4
assert config.engine.offload.dit is False
assert config.engine.offload.dit_layerwise is False
assert config.engine.offload.text_encoder is True
assert config.engine.offload.vae is True
assert config.engine.compile.vae_enabled is True
assert config.engine.use_fsdp_inference is False
assert os.environ["FASTVIDEO_ATTENTION_BACKEND"] == "VIDEO_SPARSE_ATTN_H3"
assert os.environ["FASTVIDEO_FA4"] == "1"
assert os.environ["FASTVIDEO_MINIMAX_H3_FUSIONS"] == "all"
assert os.environ["FASTVIDEO_VSA_SM100A"] == "0"
assert "FASTVIDEO_INFERENCE_TORCH_COMPILE" not in os.environ
def test_initialize_selects_declared_generation_backend(monkeypatch):
"""The GPU worker constructs the backend that the active model profile declares."""
from unittest.mock import Mock
selected_backend = Mock()
monkeypatch.setattr(
generation_worker,
"_create_generation_backend",
lambda backend_name, gpu_id: selected_backend,
)
worker = generation_worker.VideoGenerationWorker(gpu_id=3)
worker.initialize(FASTH3_MODEL_CONFIG)
assert worker.backend_name == "minimax_h3"
assert worker.backend is selected_backend
selected_backend.initialize.assert_called_once_with(FASTH3_MODEL_CONFIG)
def test_initialize_failure_clears_backend_ownership(monkeypatch):
"""A failed family change leaves the GPU worker explicitly uninitialized."""
ltx_backend = SimpleNamespace(initialize=lambda config: None, shutdown=lambda: None)
def fail_initialize(config):
del config
raise RuntimeError("load failed")
fasth3_backend = SimpleNamespace(
initialize=fail_initialize,
shutdown=lambda: None,
)
backends = {
"ltx2": ltx_backend,
"minimax_h3": fasth3_backend,
}
monkeypatch.setattr(
generation_worker,
"_create_generation_backend",
lambda backend_name, gpu_id: backends[backend_name],
)
worker = generation_worker.VideoGenerationWorker(gpu_id=3)
worker.initialize({"generation_backend": "ltx2"})
with pytest.raises(RuntimeError, match="load failed"):
worker.initialize(FASTH3_MODEL_CONFIG)
assert worker.backend is None
assert worker.backend_name is None
assert worker.model_config == {"generation_backend": "ltx2"}
def test_generate_step_uses_last_frame_for_continuation(monkeypatch):
"""A later segment receives the prior segment's last decoded frame."""
backend = MiniMaxH3GenerationBackend(gpu_id=0)
backend.model_config = dict(FASTH3_MODEL_CONFIG)
backend.generator = _RecordingGenerator()
monkeypatch.setattr("dreamverse.minimax_h3_generation.torch.cuda.synchronize", lambda: None)
first_result = backend.generate_step(
"first prompt",
segment_idx=1,
image_path=None,
reset_conditioning=True,
)
second_result = backend.generate_step(
"second prompt",
segment_idx=2,
image_path=None,
reset_conditioning=False,
)
first_request = backend.generator.requests[0]
assert first_request.inputs.pil_image is None
assert first_request.negative_prompt == ""
assert first_request.sampling.height == 768
assert first_request.sampling.width == 1344
assert first_request.sampling.num_frames == 124
assert first_request.sampling.num_inference_steps == 5
assert first_request.sampling.fps == 24
assert first_request.sampling.guidance_scale == 1.0
assert first_request.sampling.batch_cfg is False
assert first_request.sampling.seed == 1000
assert first_request.output.save_video is False
assert first_request.output.return_frames is True
assert backend.generator.conditioning_pixels[1].tolist() == np.full((2, 3, 3), 20).tolist()
assert first_result.head_trim_frames == 0
assert first_result.head_trim_audio_frames == 0
assert second_result.head_trim_frames == 1
assert second_result.head_trim_audio_frames == 1
assert second_result.audio_sample_rate == 44100
def test_generate_step_reset_uses_text_to_video_path(monkeypatch):
"""Resetting continuation produces an unconditioned text-to-video request."""
backend = MiniMaxH3GenerationBackend(gpu_id=0)
backend.model_config = dict(FASTH3_MODEL_CONFIG)
backend.generator = _RecordingGenerator()
monkeypatch.setattr("dreamverse.minimax_h3_generation.torch.cuda.synchronize", lambda: None)
backend.generate_step("first prompt", 1, None, True)
reset_result = backend.generate_step("reset prompt", 2, None, True)
assert backend.generator.requests[-1].inputs.pil_image is None
assert reset_result.head_trim_frames == 0
assert reset_result.head_trim_audio_frames == 0
def test_generate_step_missing_continuation_frame(monkeypatch):
"""A later segment fails when no reset or retained frame defines its input."""
backend = MiniMaxH3GenerationBackend(gpu_id=0)
backend.model_config = dict(FASTH3_MODEL_CONFIG)
backend.generator = _RecordingGenerator()
with pytest.raises(RuntimeError, match="requires a retained continuation frame"):
backend.generate_step("later prompt", 2, None, False)
assert backend.generator.requests == []
def test_warmup_exercises_text_and_first_frame_paths(monkeypatch):
"""Warmup covers both request shapes used by a DreamVerse session."""
backend = MiniMaxH3GenerationBackend(gpu_id=0)
backend.model_config = dict(FASTH3_MODEL_CONFIG)
backend.generator = _RecordingGenerator()
monkeypatch.setattr("dreamverse.minimax_h3_generation.torch.cuda.synchronize", lambda: None)
timings = backend.warmup("warmup prompt")
assert backend.generator.conditioning_pixels[0] is None
assert backend.generator.conditioning_pixels[1] is not None
assert backend.continuation_image is None
assert "warmup_text_to_video_ms" in timings
assert "warmup_first_frame_to_video_ms" in timings
@@ -331,11 +331,11 @@ def test_rewrite_prompt_sequence_accepts_numbered_prose_output():
]
def test_enhance_prompt_prefers_cerebras_before_groq_fallback():
def test_enhance_prompt_uses_groq_when_it_returns_first():
enhancer = _build_staged_enhancer(
cerebras_payload=_chat_payload_with_content('{"prompt":"Cerebras prompt"}'),
groq_payload=_chat_payload_with_content('{"prompt":"Groq prompt"}'),
cerebras_delay_s=0.01,
cerebras_delay_s=0.08,
groq_delay_s=0.01,
)
@@ -346,12 +346,12 @@ def test_enhance_prompt_prefers_cerebras_before_groq_fallback():
assert result.fallback_used is False
assert result.error is None
assert result.provider == "cerebras"
assert result.provider == "groq"
assert result.model == "gpt-test"
assert result.prompt == "Cerebras prompt"
assert result.prompt == "Groq prompt"
assert enhancer.get_provider_success_counts() == {
"cerebras": 1,
"groq": 0,
"cerebras": 0,
"groq": 1,
}
+7 -7
View File
@@ -70,7 +70,7 @@
<mxCell id="dispatcher" value="command dispatcher&#xa;&#xa;gpu_worker_process() branches on&#xa;CommandType; asserts payload type&#xa;&#xa;INIT / WARMUP / RELOAD_MODEL&#xa;USER_JOIN / USER_STEP / USER_LEAVE&#xa;SHUTDOWN" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#ffe6cc;strokeColor=#d79b00;fontSize=11;align=left;spacingLeft=10;spacingTop=8;fontStyle=1;" parent="1" vertex="1">
<mxGeometry x="120" y="1120" width="240" height="120" as="geometry"/>
</mxCell>
<mxCell id="do_step" value="VideoGenerationWorker.generate_step()&#xa;video_generation.py:380&#xa;&#xa;reads + updates ContinuationState,&#xa;calls generator" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#e1d5e7;strokeColor=#9673a6;fontSize=11;align=left;spacingLeft=10;spacingTop=8;fontStyle=1;" parent="1" vertex="1">
<mxCell id="do_step" value="VideoGenerationWorker.generate_step()&#xa;ltx2_generation.py:380&#xa;&#xa;reads + updates ContinuationState,&#xa;calls generator" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#e1d5e7;strokeColor=#9673a6;fontSize=11;align=left;spacingLeft=10;spacingTop=8;fontStyle=1;" parent="1" vertex="1">
<mxGeometry x="460" y="1120" width="240" height="120" as="geometry"/>
</mxCell>
<mxCell id="stream_av" value="stream_fmp4()&#xa;av_streaming.py:121&#xa;&#xa;trims overlap, pipes to ffmpeg,&#xa;publishes StreamInit / StreamChunk /&#xa;StreamComplete via callback" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#b1d8d7;strokeColor=#23445d;fontSize=11;align=left;spacingLeft=10;spacingTop=8;fontStyle=1;" parent="1" vertex="1">
@@ -79,13 +79,13 @@
<mxCell id="Ot8BU52QTIb4EhyRSe7I-2" value="" style="edgeStyle=none;html=1;" parent="1" source="generator" target="Ot8BU52QTIb4EhyRSe7I-1" edge="1">
<mxGeometry relative="1" as="geometry"/>
</mxCell>
<mxCell id="generator" value="VideoGenerator (fastvideo)&#xa;&#xa;LTX2 DiT + refine upsampler&#xa;FP4 quant, torch.compile&#xa;&#xa;owned by VideoGenerationWorker&#xa;video_generation.py:211" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#e1d5e7;strokeColor=#9673a6;fontSize=11;" parent="1" vertex="1">
<mxCell id="generator" value="VideoGenerator (fastvideo)&#xa;&#xa;LTX2 DiT + refine upsampler&#xa;FP4 quant, torch.compile&#xa;&#xa;owned by VideoGenerationWorker&#xa;ltx2_generation.py:211" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#e1d5e7;strokeColor=#9673a6;fontSize=11;" parent="1" vertex="1">
<mxGeometry x="460" y="1300" width="240" height="100" as="geometry"/>
</mxCell>
<mxCell id="ffmpeg" value="ffmpeg subprocess&#xa;&#xa;libx264 / *_nvenc&#xa;fragmented mp4" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#ffffff;strokeColor=#d79b00;fontSize=11;" parent="1" vertex="1">
<mxGeometry x="800" y="1300" width="260" height="100" as="geometry"/>
</mxCell>
<mxCell id="caches" value="ContinuationState&#xa;video_generation.py:89&#xa;&#xa;• video_images: list[PIL.Image]&#xa;• audio_latents: torch.Tensor (CPU)&#xa;&#xa;carried across segments" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#e1d5e7;strokeColor=#9673a6;fontSize=11;align=left;spacingLeft=10;spacingTop=8;" parent="1" vertex="1">
<mxCell id="caches" value="ContinuationState&#xa;ltx2_generation.py:89&#xa;&#xa;• video_images: list[PIL.Image]&#xa;• audio_latents: torch.Tensor (CPU)&#xa;&#xa;carried across segments" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#e1d5e7;strokeColor=#9673a6;fontSize=11;align=left;spacingLeft=10;spacingTop=8;" parent="1" vertex="1">
<mxGeometry x="120" y="1300" width="240" height="100" as="geometry"/>
</mxCell>
<mxCell id="e_cp" value="acquire" style="edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;strokeColor=#6c8ebf;endArrow=classic;fontSize=11;exitX=0.5;exitY=1;exitDx=0;exitDy=0;entryX=0.5;entryY=0;entryDx=0;entryDy=0;" parent="1" source="client" target="pool" edge="1">
@@ -250,7 +250,7 @@
<mxPoint x="690" y="880"/>
</Array>
</mxCell>
<mxCell id="legend" value="Legend&#xa;&#xa;■ blue client / external&#xa;■ green main-process pool/slot&#xa; (methods — italic label)&#xa;■ yellow containers (routing state)&#xa;■ red IPC primitives (mp.Queue, mp.RawArray)&#xa;&#xa;Worker subprocess modules:&#xa;■ orange gpu_pool.py (dispatcher)&#xa;■ lavender video_generation.py&#xa;■ teal av_streaming.py&#xa;■ gray worker_ipc.py (shared types)&#xa;&#xa;Flow:&#xa; client → pool → slot&#xa; → _send_command(_tagged) → command_queue&#xa; → dispatcher → generate_step()&#xa; → stream_fmp4() → ffmpeg&#xa; → shared_buf + response_queue&#xa; → _response_reader → futures / stream_queues&#xa; → client awaits (via main.py AV loop)" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#f5f5f5;strokeColor=#999999;fontSize=11;align=left;spacingLeft=10;spacingTop=8;" parent="1" vertex="1">
<mxCell id="legend" value="Legend&#xa;&#xa;■ blue client / external&#xa;■ green main-process pool/slot&#xa; (methods — italic label)&#xa;■ yellow containers (routing state)&#xa;■ red IPC primitives (mp.Queue, mp.RawArray)&#xa;&#xa;Worker subprocess modules:&#xa;■ orange gpu_pool.py (dispatcher)&#xa;■ lavender ltx2_generation.py&#xa;■ teal av_streaming.py&#xa;■ gray worker_ipc.py (shared types)&#xa;&#xa;Flow:&#xa; client → pool → slot&#xa; → _send_command(_tagged) → command_queue&#xa; → dispatcher → generate_step()&#xa; → stream_fmp4() → ffmpeg&#xa; → shared_buf + response_queue&#xa; → _response_reader → futures / stream_queues&#xa; → client awaits (via main.py AV loop)" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#f5f5f5;strokeColor=#999999;fontSize=11;align=left;spacingLeft=10;spacingTop=8;" parent="1" vertex="1">
<mxGeometry x="39" y="-200" width="270" height="380" as="geometry"/>
</mxCell>
<mxCell id="Ot8BU52QTIb4EhyRSe7I-1" value="FastVideo video_generator" style="whiteSpace=wrap;html=1;fontSize=11;fillColor=#e1d5e7;strokeColor=#9673a6;rounded=1;" parent="1" vertex="1">
@@ -389,10 +389,10 @@
<mxCell id="cw2" value="from fastvideo.entrypoints.video_generator import VideoGenerator&#xa;from fastvideo.models.dits.ltx2 import DEFAULT_LTX2_AUDIO_*&#xa;&#xa;** Dreamverse reaches into fastvideo internals here **" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#ffe0b2;strokeColor=#d79b00;fontSize=10;align=left;spacingLeft=8;fontFamily=monospace;" parent="1" vertex="1">
<mxGeometry x="675" y="695" width="550" height="60" as="geometry"/>
</mxCell>
<mxCell id="cw3" value="on Command(INIT):&#xa; VideoGenerationWorker.initialize() (video_generation.py:247)&#xa; maybe_download_model(model_id)&#xa; VideoGenerator.from_pretrained(path, FP4Config, PipelineConfig)&#xa; load audio VAE, resolve refine upsampler&#xa; resp_q.put(InitAck(success=True))" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#ffffff;strokeColor=#d79b00;fontSize=10;align=left;spacingLeft=8;fontFamily=monospace;" parent="1" vertex="1">
<mxCell id="cw3" value="on Command(INIT):&#xa; VideoGenerationWorker.initialize() (ltx2_generation.py:247)&#xa; maybe_download_model(model_id)&#xa; VideoGenerator.from_pretrained(path, FP4Config, PipelineConfig)&#xa; load audio VAE, resolve refine upsampler&#xa; resp_q.put(InitAck(success=True))" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#ffffff;strokeColor=#d79b00;fontSize=10;align=left;spacingLeft=8;fontFamily=monospace;" parent="1" vertex="1">
<mxGeometry x="675" y="765" width="550" height="95" as="geometry"/>
</mxCell>
<mxCell id="cw4" value="on Command(WARMUP) with WarmupPayload:&#xa; VideoGenerationWorker.warmup(payload.prompt) (video_generation.py:518)&#xa; two synthetic segments prime caches + torch.compile&#xa; resp_q.put(WarmupComplete(timings=...))" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#ffffff;strokeColor=#d79b00;fontSize=10;align=left;spacingLeft=8;fontFamily=monospace;" parent="1" vertex="1">
<mxCell id="cw4" value="on Command(WARMUP) with WarmupPayload:&#xa; VideoGenerationWorker.warmup(payload.prompt) (ltx2_generation.py:518)&#xa; two synthetic segments prime caches + torch.compile&#xa; resp_q.put(WarmupComplete(timings=...))" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#ffffff;strokeColor=#d79b00;fontSize=10;align=left;spacingLeft=8;fontFamily=monospace;" parent="1" vertex="1">
<mxGeometry x="675" y="870" width="550" height="55" as="geometry"/>
</mxCell>
<mxCell id="cw5" value="enter main worker loop → waits for JOIN_USER / USER_STEP / LEAVE" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#c8e6c9;strokeColor=#388e3c;fontSize=11;fontStyle=1;fontFamily=monospace;" parent="1" vertex="1">
@@ -534,7 +534,7 @@
<mxPoint x="1040" y="1610" as="targetPoint"/>
</mxGeometry>
</mxCell>
<mxCell id="dm11a" value="10a. worker runs:&#xa;VideoGenerationWorker.generate_step()&#xa; (video_generation.py:380)&#xa; → generator.generate_video()&#xa; → updates ContinuationState&#xa;then stream_fmp4() (av_streaming.py:121)&#xa; → ffmpeg (rawvideo+wav → fmp4)" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#ffe0b2;strokeColor=#d79b00;fontSize=10;align=left;spacingLeft=8;fontFamily=monospace;" parent="1" vertex="1">
<mxCell id="dm11a" value="10a. worker runs:&#xa;VideoGenerationWorker.generate_step()&#xa; (ltx2_generation.py:380)&#xa; → generator.generate_video()&#xa; → updates ContinuationState&#xa;then stream_fmp4() (av_streaming.py:121)&#xa; → ffmpeg (rawvideo+wav → fmp4)" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#ffe0b2;strokeColor=#d79b00;fontSize=10;align=left;spacingLeft=8;fontFamily=monospace;" parent="1" vertex="1">
<mxGeometry x="955" y="1640" width="180" height="70" as="geometry"/>
</mxCell>
<mxCell id="dm11" value="10b. resp_q.put(MediaInit / MediaChunk / MediaComplete / StepComplete)" style="endArrow=classic;html=1;strokeColor=#b85450;fontSize=10;labelBackgroundColor=#ffffff;" parent="1" edge="1">
File diff suppressed because one or more lines are too long

Before

Width:  |  Height:  |  Size: 85 KiB

After

Width:  |  Height:  |  Size: 85 KiB

+20 -4
View File
@@ -9,6 +9,7 @@ from __future__ import annotations
import contextlib
import logging
import json
import sqlite3
import threading
from pathlib import Path
@@ -46,8 +47,13 @@ DEFAULT_SETTINGS: dict[str, Any] = {
def _sqlite_row_get(row: sqlite3.Row, key: str, default: Any) -> Any:
"""Like dict.get for sqlite3.Row (Row has no .get on Python 3.10)."""
return row[key] if key in row else default # noqa: SIM401
"""Like dict.get for sqlite3.Row (Row has no .get on Python 3.10).
NOTE: `key in row` tests Row *values*, not column names, so the membership
check has to go through .keys() -- otherwise every lookup falls back to the
default and jobs restored from the database lose their stored fields.
"""
return row[key] if key in row.keys() else default # noqa: SIM401, SIM118
def _get_db_path(data_dir: Path) -> Path:
@@ -83,6 +89,9 @@ def _migrate_db(conn: sqlite3.Connection) -> None:
_add_column_if_missing(conn, "jobs", "fps", "INTEGER", "24")
_add_column_if_missing(conn, "jobs", "workload_type", "TEXT", "'t2v'")
_add_column_if_missing(conn, "jobs", "image_path", "TEXT", "''")
_add_column_if_missing(conn, "jobs", "name", "TEXT", "''")
_add_column_if_missing(conn, "jobs", "last_image_path", "TEXT", "''")
_add_column_if_missing(conn, "jobs", "references_json", "TEXT", "''")
_add_column_if_missing(conn, "jobs", "job_type", "TEXT", "'inference'")
_add_column_if_missing(conn, "jobs", "data_path", "TEXT", "''")
_add_column_if_missing(conn, "jobs", "max_train_steps", "INTEGER", "1000")
@@ -242,7 +251,8 @@ class Database:
self._execute(
"""
INSERT INTO jobs (
id, model_id, prompt, workload_type, image_path, job_type, status,
id, model_id, name, prompt, workload_type, image_path,
last_image_path, references_json, job_type, status,
created_at, started_at, finished_at, error, output_path, log_file_path,
num_inference_steps, num_frames, height, width, guidance_scale,
guidance_rescale, fps, seed, num_gpus, dit_cpu_offload,
@@ -254,14 +264,17 @@ class Database:
dmd_use_vsa, dmd_vsa_sparsity, dmd_denoising_steps,
real_score_guidance_scale,
generator_update_interval, real_score_model_path, fake_score_model_path
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
""",
(
job["id"],
job["model_id"],
job.get("name", ""),
job["prompt"],
job.get("workload_type", "t2v"),
job.get("image_path", ""),
job.get("last_image_path", ""),
json.dumps(job.get("references") or []),
job.get("job_type", "inference"),
job["status"],
job["created_at"],
@@ -540,9 +553,12 @@ def _row_to_job(row: sqlite3.Row) -> dict[str, Any]:
result = {
"id": row["id"],
"model_id": row["model_id"],
"name": _sqlite_row_get(row, "name", "") or "",
"prompt": row["prompt"],
"workload_type": _sqlite_row_get(row, "workload_type", "t2v"),
"image_path": _sqlite_row_get(row, "image_path", "") or "",
"last_image_path": _sqlite_row_get(row, "last_image_path", "") or "",
"references": _sqlite_row_get(row, "references_json", "") or "",
"job_type": _sqlite_row_get(row, "job_type", "inference"),
"status": row["status"],
"created_at": row["created_at"],
@@ -0,0 +1,63 @@
import { expect, test } from '@playwright/test';
import { skipWithoutMock } from './helpers';
test.describe('create job interactions', () => {
skipWithoutMock();
for (const jobType of ['inference', 'finetuning', 'distillation']) {
test(`${jobType} remains interactive after repeated dialog dismissals`, async ({ page }) => {
await page.goto(`/${jobType}`);
const trigger = page.getByRole('button', { name: 'Create Job', exact: true });
const dialog = page.getByRole('dialog');
// Exercise both dismissal paths and reopen without reloading the page.
for (const closeWithEscape of [false, true]) {
await trigger.click();
await page.getByRole('menuitem').first().click();
await expect(dialog).toBeVisible();
if (closeWithEscape) {
await page.keyboard.press('Escape');
} else {
await dialog.getByRole('button', { name: 'Close', exact: true }).click();
}
await expect(dialog).toBeHidden();
await expect(page.locator('body')).toHaveCSS('pointer-events', 'auto');
await expect(trigger).toBeFocused();
}
await page.getByRole('link', { name: 'Datasets', exact: true }).click();
await expect(page).toHaveURL(/\/datasets$/);
});
}
test('preserves keyboard menu dismissal and dialog focus trapping', async ({ page }) => {
await page.goto('/inference');
const trigger = page.getByRole('button', { name: 'Create Job', exact: true });
await trigger.focus();
await page.keyboard.press('Enter');
const firstItem = page.getByRole('menuitem').first();
await expect(firstItem).toBeFocused();
await page.keyboard.press('Escape');
await expect(page.getByRole('menu')).toBeHidden();
await expect(trigger).toBeFocused();
await expect(page.locator('body')).toHaveCSS('pointer-events', 'auto');
await page.keyboard.press('Enter');
await expect(firstItem).toBeFocused();
await page.keyboard.press('Enter');
const dialog = page.getByRole('dialog');
await expect(dialog).toBeVisible();
await expect(dialog.getByLabel('Name (optional)')).toBeFocused();
// Shift+Tab from the first field wraps to Close, then Tab wraps back.
await page.keyboard.press('Shift+Tab');
await expect(dialog.getByRole('button', { name: 'Close', exact: true })).toBeFocused();
await page.keyboard.press('Tab');
await expect(dialog.getByLabel('Name (optional)')).toBeFocused();
await page.keyboard.press('Escape');
await expect(dialog).toBeHidden();
await expect(trigger).toBeFocused();
await expect(page.locator('body')).toHaveCSS('pointer-events', 'auto');
});
});
+18 -2
View File
@@ -1,6 +1,6 @@
import { expect, test } from '@playwright/test';
import { skipWithoutMock } from './helpers';
import { API_BASE, skipWithoutMock } from './helpers';
/**
* Create-job flow: open the Create Job modal on /inference, fill the prompt
@@ -10,7 +10,8 @@ import { skipWithoutMock } from './helpers';
test.describe('create inference job', () => {
skipWithoutMock();
test('creates a T2V job and shows it in the queue', async ({ page }) => {
test('creates a T2V job and starts it without refreshing', async ({ page, request }) => {
await request.put(`${API_BASE}/settings`, { data: { autoStartJob: false } });
await page.goto('/inference');
// The trigger opens a real menu on click, so this path works for touch,
@@ -38,5 +39,20 @@ test.describe('create inference job', () => {
// Modal closes and the queue refreshes with the newly created job.
await expect(dialog).toBeHidden();
await expect(page.getByText(prompt)).toBeVisible();
await expect(page.locator('body')).toHaveCSS('pointer-events', 'auto');
const card = page.getByRole('article').filter({ hasText: prompt });
await expect(card.getByText('pending', { exact: true })).toBeVisible();
const started = page.waitForResponse((response) =>
response.url().startsWith(`${API_BASE}/jobs/`) &&
response.url().endsWith('/start') &&
response.request().method() === 'POST',
);
await card.getByRole('button', { name: 'Start', exact: true }).click();
expect((await started).ok()).toBe(true);
await expect(card.getByText('running', { exact: true })).toBeVisible();
await page.getByRole('link', { name: 'Datasets', exact: true }).click();
await expect(page).toHaveURL(/\/datasets$/);
});
});
+239 -58
View File
@@ -10,7 +10,9 @@ from __future__ import annotations
import atexit
import collections
import contextlib
import copy
import enum
import json
import logging
import logging.handlers
import multiprocessing as mp
@@ -123,10 +125,13 @@ class LogBufferHandler(logging.Handler):
class Job:
id: str
model_id: str
prompt: str
name: str = ""
prompt: str = ""
workload_type: str = "t2v"
job_type: str = "inference"
image_path: str = ""
last_image_path: str = ""
references: list[dict[str, Any]] = field(default_factory=list)
status: JobStatus = JobStatus.PENDING
created_at: float = field(default_factory=time.time)
started_at: float | None = None
@@ -145,6 +150,7 @@ class Job:
negative_prompt: str = ""
num_gpus: int = 1
dit_cpu_offload: bool = False
dit_layerwise_offload: bool = False
text_encoder_cpu_offload: bool = False
vae_cpu_offload: bool = False
image_encoder_cpu_offload: bool = False
@@ -180,10 +186,13 @@ class Job:
return {
"id": self.id,
"model_id": self.model_id,
"name": self.name,
"prompt": self.prompt,
"workload_type": self.workload_type,
"job_type": self.job_type,
"image_path": self.image_path,
"last_image_path": self.last_image_path,
"references": self.references,
"status": self.status.value,
"created_at": self.created_at,
"started_at": self.started_at,
@@ -202,6 +211,7 @@ class Job:
"negative_prompt": self.negative_prompt,
"num_gpus": self.num_gpus,
"dit_cpu_offload": self.dit_cpu_offload,
"dit_layerwise_offload": self.dit_layerwise_offload,
"text_encoder_cpu_offload": self.text_encoder_cpu_offload,
"vae_cpu_offload": self.vae_cpu_offload,
"image_encoder_cpu_offload": self.image_encoder_cpu_offload,
@@ -232,6 +242,75 @@ class Job:
}
MINIMAX_H3_REF2VA_PIPELINE = "MiniMaxH3Ref2VAModularPipeline"
def _build_h3_references(raw: list[dict[str, Any]]) -> list[Any]:
"""Turn the API's reference dicts into MiniMaxH3Reference objects.
Imported lazily so the API server starts without pulling in fastvideo.
"""
from fastvideo.pipelines.basic.minimax_h3 import MiniMaxH3Reference
built = []
for i, ref in enumerate(raw):
source = (ref or {}).get("source")
if not source:
raise ValueError(f"reference {i} has no source")
if not os.path.isfile(source):
raise ValueError(f"reference {i} source not found: {source}")
kwargs: dict[str, Any] = {
"source": source,
"media_type": (ref.get("media_type") or "image"),
}
for opt in ("soundtrack", "fps", "sample_rate"):
if ref.get(opt) not in (None, ""):
kwargs[opt] = ref[opt]
built.append(MiniMaxH3Reference(**kwargs))
return built
JOB_LOG_FILENAME = "out.log"
def _job_log_path(output_dir: str, job_id: str) -> str:
"""Each job's log lives beside its outputs: <output_dir>/<job_id>/out.log."""
return os.path.join(output_dir, job_id, JOB_LOG_FILENAME)
def _decode_references(value: Any) -> list[dict[str, Any]]:
"""Reference lists round-trip through the DB as JSON text."""
if not value:
return []
if isinstance(value, list):
return list(value)
try:
decoded = json.loads(value)
except (TypeError, ValueError):
logger.warning("Could not decode stored references: %r", value)
return []
return list(decoded) if isinstance(decoded, list) else []
def _generator_is_alive(generator: Any) -> bool:
"""True if the generator's worker processes are all still running.
A cached VideoGenerator holds a MultiprocExecutor whose workers are separate
processes; nothing notices when they exit. Probing `proc.is_alive()` is what
the executor itself uses during shutdown. Anything unexpected in the object
graph is treated as alive so a probe failure can never wedge the cache.
"""
executor = getattr(generator, "executor", None)
workers = getattr(executor, "workers", None)
if not workers:
return True
try:
return all(w.proc.is_alive() for w in workers)
except Exception:
logger.debug("Worker liveness probe failed", exc_info=True)
return True
class JobRunner:
"""Manages video generation jobs, their execution, and generator caching."""
@@ -276,7 +355,7 @@ class JobRunner:
"""Populate job's log buffer from its log file if it exists."""
path = job.log_file_path
if not path:
path = os.path.join(self.log_dir, f"{job.id}.log")
path = _job_log_path(self.output_dir, job.id)
if not os.path.isfile(path):
return
try:
@@ -314,10 +393,13 @@ class JobRunner:
job = Job(
id=row["id"],
model_id=row["model_id"],
name=row.get("name", "") or "",
prompt=row["prompt"],
workload_type=row.get("workload_type", "t2v"),
job_type=row.get("job_type", "inference"),
image_path=row.get("image_path", "") or "",
last_image_path=row.get("last_image_path", "") or "",
references=_decode_references(row.get("references")),
data_path=row.get("data_path", "") or "",
max_train_steps=row.get("max_train_steps", 1000),
train_batch_size=row.get("train_batch_size", 1),
@@ -350,6 +432,7 @@ class JobRunner:
negative_prompt=row.get("negative_prompt", "") or "",
num_gpus=row.get("num_gpus", 1),
dit_cpu_offload=row.get("dit_cpu_offload", False),
dit_layerwise_offload=row.get("dit_layerwise_offload", False),
text_encoder_cpu_offload=row.get("text_encoder_cpu_offload", False),
vae_cpu_offload=row.get("vae_cpu_offload", False),
image_encoder_cpu_offload=row.get("image_encoder_cpu_offload", False),
@@ -394,9 +477,12 @@ class JobRunner:
job_id: str,
model_id: str,
prompt: str,
name: str = "",
workload_type: str = "t2v",
job_type: str = "inference",
image_path: str = "",
last_image_path: str = "",
references: list[dict[str, Any]] | None = None,
data_path: str = "",
max_train_steps: int = 1000,
train_batch_size: int = 1,
@@ -422,6 +508,7 @@ class JobRunner:
num_gpus: int = 1,
negative_prompt: str = "",
dit_cpu_offload: bool = False,
dit_layerwise_offload: bool = False,
text_encoder_cpu_offload: bool = False,
vae_cpu_offload: bool = False,
image_encoder_cpu_offload: bool = False,
@@ -435,10 +522,13 @@ class JobRunner:
job = Job(
id=job_id,
model_id=model_id,
name=(name or "").strip(),
prompt=prompt.strip(),
workload_type=workload_type or "t2v",
job_type=job_type or "inference",
image_path=image_path or "",
last_image_path=last_image_path or "",
references=list(references or []),
data_path=data_path or "",
max_train_steps=max_train_steps,
train_batch_size=train_batch_size,
@@ -464,6 +554,7 @@ class JobRunner:
negative_prompt=negative_prompt or "",
num_gpus=num_gpus,
dit_cpu_offload=dit_cpu_offload,
dit_layerwise_offload=dit_layerwise_offload,
text_encoder_cpu_offload=text_encoder_cpu_offload,
vae_cpu_offload=vae_cpu_offload,
image_encoder_cpu_offload=image_encoder_cpu_offload,
@@ -521,6 +612,84 @@ class JobRunner:
logger.info("Deleted job %s", job.id)
return True
CONFIG_FIELDS: tuple[str, ...] = (
"model_id",
"name",
"prompt",
"workload_type",
"job_type",
"image_path",
"last_image_path",
"references",
"negative_prompt",
"num_inference_steps",
"num_frames",
"height",
"width",
"guidance_scale",
"guidance_rescale",
"fps",
"seed",
"num_gpus",
"dit_cpu_offload",
"dit_layerwise_offload",
"text_encoder_cpu_offload",
"vae_cpu_offload",
"image_encoder_cpu_offload",
"use_fsdp_inference",
"enable_torch_compile",
"vsa_sparsity",
"tp_size",
"sp_size",
"data_path",
"max_train_steps",
"train_batch_size",
"learning_rate",
"num_latent_t",
"validation_dataset_file",
"lora_rank",
"dmd_use_vsa",
"dmd_vsa_sparsity",
"dmd_denoising_steps",
"real_score_guidance_scale",
"generator_update_interval",
"real_score_model_path",
"fake_score_model_path",
)
def duplicate_job(self, job_id: str, new_job_id: str) -> Job:
"""Create a new pending job with an existing job's configuration.
Runtime state (status, timings, logs, outputs) is not carried over.
"""
with self._jobs_lock:
source = self._jobs.get(job_id)
if source is None:
raise ValueError(f"Job {job_id} not found")
config = {f: copy.deepcopy(getattr(source, f)) for f in self.CONFIG_FIELDS}
return self.create_job(job_id=new_job_id, **config)
#: Editable exactly when startable: the same set start_job() accepts.
EDITABLE_STATUSES = (JobStatus.PENDING, JobStatus.FAILED, JobStatus.STOPPED)
def update_job_config(self, job_id: str, updates: dict[str, Any]) -> Job:
"""Edit the configuration of a job that has not produced a result."""
with self._jobs_lock:
job = self._jobs.get(job_id)
if job is None:
raise ValueError(f"Job {job_id} not found")
if job.status not in self.EDITABLE_STATUSES:
allowed = ", ".join(s.value for s in self.EDITABLE_STATUSES)
raise ValueError(f"Job is {job.status.value}; only {allowed} jobs can be edited. "
"Duplicate it instead.")
unknown = set(updates) - set(self.CONFIG_FIELDS)
if unknown:
raise ValueError(f"Not editable: {', '.join(sorted(unknown))}")
for field_name, value in updates.items():
setattr(job, field_name, value)
self._save_job(job)
return job
def start_job(self, job_id: str) -> Job:
"""Start (or restart) a pending / stopped / failed job.
@@ -623,6 +792,8 @@ class JobRunner:
workload_type: str,
num_gpus: int,
dit_cpu_offload: bool = False,
dit_layerwise_offload: bool = False,
override_pipeline_cls_name: str | None = None,
text_encoder_cpu_offload: bool = False,
vae_cpu_offload: bool = False,
image_encoder_cpu_offload: bool = False,
@@ -638,6 +809,10 @@ class JobRunner:
workload_type,
num_gpus,
dit_cpu_offload,
dit_layerwise_offload,
# Ref2VA loads different DiT weights (transformer_ref), so the
# override must key the cache or a t2v/i2v generator gets reused.
override_pipeline_cls_name,
text_encoder_cpu_offload,
vae_cpu_offload,
image_encoder_cpu_offload,
@@ -650,8 +825,21 @@ class JobRunner:
# Generators are cached by model_id and configuration parameters
with self._generators_lock:
if cache_key in self._generators:
return self._generators[cache_key]
cached = self._generators.get(cache_key)
if cached is not None:
if _generator_is_alive(cached):
return cached
# Workers can exit while a generator sits idle in the cache;
# reusing it fails every later job with the same config.
logger.warning(
"Cached generator for %s has dead workers; reloading.",
model_id,
)
self._generators.pop(cache_key, None)
try:
cached.shutdown()
except Exception:
logger.debug("Shutdown of the dead generator failed", exc_info=True)
# Import lazily so starting the server is fast even without a GPU.
from fastvideo import VideoGenerator
@@ -677,6 +865,11 @@ class JobRunner:
gen = VideoGenerator.from_pretrained(
model_id,
workload_type=workload_type,
num_gpus=num_gpus,
dit_layerwise_offload=dit_layerwise_offload,
**({
"override_pipeline_cls_name": override_pipeline_cls_name
} if override_pipeline_cls_name else {}),
dit_cpu_offload=dit_cpu_offload,
text_encoder_cpu_offload=text_encoder_cpu_offload,
vae_cpu_offload=vae_cpu_offload,
@@ -706,10 +899,9 @@ class JobRunner:
def _run_training_job(self, job: Job):
"""Run a finetuning, distillation, or LoRA job via subprocess."""
buf = job._log_buf
os.makedirs(self.log_dir, exist_ok=True)
job.log_file_path = os.path.join(self.log_dir, f"{job.id}.log")
job_output_dir = os.path.join(self.output_dir, job.id)
os.makedirs(job_output_dir, exist_ok=True)
job.log_file_path = _job_log_path(self.output_dir, job.id)
if not job.data_path or not os.path.isdir(job.data_path):
job.status = JobStatus.FAILED
@@ -827,8 +1019,8 @@ class JobRunner:
def _run_inference_job(self, job: Job):
buf = job._log_buf
os.makedirs(self.log_dir, exist_ok=True)
job.log_file_path = os.path.join(self.log_dir, f"{job.id}.log")
os.makedirs(os.path.join(self.output_dir, job.id), exist_ok=True)
job.log_file_path = _job_log_path(self.output_dir, job.id)
# Add file handler to persist logs
file_handler = logging.FileHandler(job.log_file_path, mode='w', encoding='utf-8')
@@ -875,62 +1067,45 @@ class JobRunner:
buf.phase = "loading model"
logger.info("Loading model...")
# Run generator creation in a background thread so we
# can poll _stop_event while the (potentially slow)
# model download / load is in progress.
_gen_result: list[Any] = []
_gen_error: list[BaseException] = []
# The generator MUST be created on this thread: building it spawns
# the executor's worker processes, and they are torn down if the
# creating thread exits. Running it in a helper thread (to poll
# _stop_event during load) made every collective_rpc fail with
# ConnectionResetError.
if job._stop_event.is_set():
job.status = JobStatus.STOPPED
job.finished_at = time.time()
self._save_job(job)
logger.warning("Job %s stopped before model loading", job.id)
buf.phase = "stopped"
return
def _load_generator() -> None:
try:
gen = self._get_or_create_generator(
job.model_id,
job.workload_type,
job.num_gpus,
dit_cpu_offload=job.dit_cpu_offload,
text_encoder_cpu_offload=(job.text_encoder_cpu_offload),
vae_cpu_offload=job.vae_cpu_offload,
image_encoder_cpu_offload=(job.image_encoder_cpu_offload),
use_fsdp_inference=job.use_fsdp_inference,
enable_torch_compile=(job.enable_torch_compile),
vsa_sparsity=job.vsa_sparsity,
tp_size=job.tp_size,
sp_size=job.sp_size,
log_queue=log_queue,
)
_gen_result.append(gen)
except BaseException as exc:
_gen_error.append(exc)
loader = threading.Thread(
target=_load_generator,
daemon=True,
generator = self._get_or_create_generator(
job.model_id,
job.workload_type,
job.num_gpus,
dit_cpu_offload=job.dit_cpu_offload,
dit_layerwise_offload=job.dit_layerwise_offload,
override_pipeline_cls_name=(MINIMAX_H3_REF2VA_PIPELINE if job.references else None),
text_encoder_cpu_offload=(job.text_encoder_cpu_offload),
vae_cpu_offload=job.vae_cpu_offload,
image_encoder_cpu_offload=(job.image_encoder_cpu_offload),
use_fsdp_inference=job.use_fsdp_inference,
enable_torch_compile=(job.enable_torch_compile),
vsa_sparsity=job.vsa_sparsity,
tp_size=job.tp_size,
sp_size=job.sp_size,
log_queue=log_queue,
)
loader.start()
while loader.is_alive():
if job._stop_event.is_set():
job.status = JobStatus.STOPPED
job.finished_at = time.time()
self._save_job(job)
logger.warning(
"Job %s stopped during model loading",
job.id,
)
buf.phase = "stopped"
return
loader.join(timeout=0.5)
if _gen_error:
raise _gen_error[0]
generator = _gen_result[0]
buf.phase = "generating"
logger.info("Starting generation for job %s (model=%s)", job.id, job.model_id)
# Without a name FastVideo derives the filename from the prompt.
safe_name = re.sub(r'[\\/:*?"<>|]+', "", job.name).strip().strip(".")
output_target = (os.path.join(job_output_dir, f"{safe_name[:80]}.mp4") if safe_name else job_output_dir)
gen_kwargs: dict[str, Any] = {
"prompt": job.prompt,
"output_path": job_output_dir,
"output_path": output_target,
"save_video": True,
"num_inference_steps": job.num_inference_steps,
"num_frames": job.num_frames,
@@ -945,6 +1120,12 @@ class JobRunner:
}
if job.image_path:
gen_kwargs["image_path"] = job.image_path
if job.references:
gen_kwargs["references"] = _build_h3_references(job.references)
if job.last_image_path:
# _prepare_fl2va requires a PIL image, not a path.
from PIL import Image as _PILImage
gen_kwargs["last_image"] = _PILImage.open(job.last_image_path)
generator.generate_video(**gen_kwargs)
buf.phase = "saving"
@@ -977,7 +1158,7 @@ class JobRunner:
except Exception as exception:
error_msg = str(exception)
logger.error("Critical error in job thread: %s", error_msg)
logger.exception("Critical error in job thread: %s", error_msg)
job.status = JobStatus.FAILED
job.error = f"Critical error ({type(exception).__name__}): {error_msg}"
job.finished_at = time.time()
@@ -1,15 +1,20 @@
# SPDX-License-Identifier: Apache-2.0
"""Request model for creating a job."""
from typing import Any
from pydantic import BaseModel
class CreateJobRequest(BaseModel):
model_id: str
name: str = ""
prompt: str
workload_type: str = "t2v"
job_type: str = "inference"
image_path: str = ""
last_image_path: str = ""
references: list[dict[str, Any]] | None = None
data_path: str = ""
max_train_steps: int = 1000
train_batch_size: int = 1
@@ -28,6 +33,7 @@ class CreateJobRequest(BaseModel):
seed: int = 1024
num_gpus: int = 1
dit_cpu_offload: bool = False
dit_layerwise_offload: bool = False
text_encoder_cpu_offload: bool = False
vae_cpu_offload: bool = False
image_encoder_cpu_offload: bool = False
+868 -722
View File
File diff suppressed because it is too large Load Diff
+1 -10
View File
@@ -17,20 +17,11 @@
"start:all": "concurrently --kill-others-on-fail \"npm:start:api\" \"npm:start:web\""
},
"dependencies": {
"@radix-ui/react-dialog": "^1.1.0",
"@radix-ui/react-dropdown-menu": "^2.1.24",
"@radix-ui/react-label": "^2.1.8",
"@radix-ui/react-scroll-area": "^1.2.10",
"@radix-ui/react-select": "^2.2.6",
"@radix-ui/react-separator": "^1.1.8",
"@radix-ui/react-slider": "^1.2.0",
"@radix-ui/react-slot": "^1.2.4",
"@radix-ui/react-switch": "^1.1.0",
"@radix-ui/react-tabs": "^1.1.0",
"class-variance-authority": "^0.7.1",
"clsx": "^2.1.1",
"lucide-react": "^0.577.0",
"next": "15.5.18",
"radix-ui": "^1.6.7",
"react": "^19.1.0",
"react-dom": "^19.1.0",
"sonner": "^2.0.7",
+92 -2
View File
@@ -18,6 +18,7 @@ import argparse
import contextlib
import logging
import os
import re
import shutil
import signal
import time
@@ -116,7 +117,30 @@ def list_models(workload_type: str | None = None) -> list[dict[str, Any]]:
return _available_models
def _safe_upload_name(filename: str | None, ext: str) -> str:
"""A filesystem-safe version of the client's filename, keeping it readable.
Uploads live under a per-file uuid directory, so the basename does not have
to be unique -- only safe. Keeping the original name means the path stays
self-describing wherever it travels: the database, job logs, and payloads
copied back out to the API.
"""
stem = os.path.basename(filename or "").rsplit(".", 1)[0]
stem = re.sub(r"[^A-Za-z0-9._-]+", "_", stem).strip("._-")
return f"{stem[:80] or 'upload'}{ext}"
def _upload_destination(ext: str, filename: str | None) -> str:
"""<upload_dir>/<uuid4>/<safe original name><ext>"""
directory = os.path.join(upload_dir, uuid.uuid4().hex)
os.makedirs(directory, exist_ok=True)
return os.path.join(directory, _safe_upload_name(filename, ext))
ALLOWED_IMAGE_EXTENSIONS = {".png", ".jpg", ".jpeg", ".webp", ".bmp"}
ALLOWED_VIDEO_EXTENSIONS = {".mp4", ".mov", ".mkv", ".webm", ".avi"}
ALLOWED_AUDIO_EXTENSIONS = {".wav", ".mp3", ".flac", ".m4a", ".ogg"}
ALLOWED_MEDIA_EXTENSIONS = (ALLOWED_IMAGE_EXTENSIONS | ALLOWED_VIDEO_EXTENSIONS | ALLOWED_AUDIO_EXTENSIONS)
@app.post("/api/upload-image")
@@ -136,8 +160,7 @@ async def upload_image(file: Annotated[UploadFile, File()], ) -> dict[str, str]:
f"{', '.join(ALLOWED_IMAGE_EXTENSIONS)}"),
)
os.makedirs(upload_dir, exist_ok=True)
unique_name = f"{uuid.uuid4().hex}{ext}"
dest_path = os.path.join(upload_dir, unique_name)
dest_path = _upload_destination(ext, file.filename)
try:
contents = await file.read()
with open(dest_path, "wb") as f:
@@ -150,6 +173,47 @@ async def upload_image(file: Annotated[UploadFile, File()], ) -> dict[str, str]:
return {"path": os.path.abspath(dest_path)}
@app.post("/api/upload-media")
async def upload_media(file: Annotated[UploadFile, File()], ) -> dict[str, str]:
"""Upload an image, video or audio file for Ref2VA references.
Returns the absolute path plus the media_type MiniMax-H3 expects, so the
caller does not have to re-derive it from the extension.
"""
global upload_dir # noqa: PLW0603
if not upload_dir:
raise HTTPException(
status_code=503,
detail="Upload directory not configured",
)
ext = Path(file.filename or "").suffix.lower()
if ext not in ALLOWED_MEDIA_EXTENSIONS:
raise HTTPException(
status_code=400,
detail=(f"Invalid file type. Allowed: "
f"{', '.join(sorted(ALLOWED_MEDIA_EXTENSIONS))}"),
)
if ext in ALLOWED_VIDEO_EXTENSIONS:
media_type = "video"
elif ext in ALLOWED_AUDIO_EXTENSIONS:
media_type = "audio"
else:
media_type = "image"
os.makedirs(upload_dir, exist_ok=True)
dest_path = _upload_destination(ext, file.filename)
try:
contents = await file.read()
with open(dest_path, "wb") as f:
f.write(contents)
except OSError as e:
raise HTTPException(
status_code=500,
detail=f"Failed to save upload: {e}",
) from e
return {"path": os.path.abspath(dest_path), "media_type": media_type}
ALLOWED_VIDEO_EXTENSIONS = {".mp4", ".webm", ".avi", ".mov", ".mkv"}
@@ -282,10 +346,13 @@ def create_job(req: CreateJobRequest) -> dict[str, Any]:
job = job_runner.create_job(
job_id=str(uuid.uuid4()),
model_id=req.model_id,
name=req.name or "",
prompt=req.prompt,
workload_type=req.workload_type or "t2v",
job_type=job_type,
image_path=req.image_path or "",
last_image_path=req.last_image_path or "",
references=req.references or [],
data_path=data_path,
max_train_steps=req.max_train_steps,
train_batch_size=req.train_batch_size,
@@ -304,6 +371,7 @@ def create_job(req: CreateJobRequest) -> dict[str, Any]:
seed=req.seed,
num_gpus=req.num_gpus,
dit_cpu_offload=req.dit_cpu_offload,
dit_layerwise_offload=req.dit_layerwise_offload,
text_encoder_cpu_offload=req.text_encoder_cpu_offload,
vae_cpu_offload=req.vae_cpu_offload,
image_encoder_cpu_offload=req.image_encoder_cpu_offload,
@@ -336,6 +404,28 @@ def create_job(req: CreateJobRequest) -> dict[str, Any]:
return job.to_dict()
@app.post("/api/jobs/{job_id}/duplicate", status_code=201)
def duplicate_job(job_id: str) -> dict[str, Any]:
"""Create a new pending job with the same configuration as an existing one."""
try:
job = job_runner.duplicate_job(job_id, str(uuid.uuid4()))
except ValueError as e:
raise HTTPException(status_code=404, detail=str(e)) from e
return job.to_dict()
@app.patch("/api/jobs/{job_id}")
def update_job(job_id: str, updates: dict[str, Any]) -> dict[str, Any]:
"""Edit a pending job's configuration. Started jobs cannot be edited."""
try:
job = job_runner.update_job_config(job_id, updates)
except ValueError as e:
detail = str(e)
status = 404 if "not found" in detail else 400
raise HTTPException(status_code=status, detail=detail) from e
return job.to_dict()
@app.post("/api/jobs/{job_id}/start")
def start_job(job_id: str) -> dict[str, Any]:
"""Start (or restart) a pending / stopped / failed job."""
@@ -1,24 +1,26 @@
import { render, screen } from '@testing-library/react';
import { render, screen, waitFor, within } from '@testing-library/react';
import userEvent from '@testing-library/user-event';
import { describe, expect, it, vi } from 'vitest';
import { beforeEach, describe, expect, it, vi } from 'vitest';
import CreateJobButton from './CreateJobButton';
import { getDatasets, getModels } from '@/lib/api';
vi.mock('./CreateJobModal', () => ({
default: ({
isOpen,
workloadType,
}: {
isOpen: boolean;
workloadType: string;
}) =>
isOpen ? (
<div role="dialog" data-workload-type={workloadType}>
Create job form
</div>
) : null,
vi.mock('@/lib/api', () => ({
createJob: vi.fn(),
getModels: vi.fn(),
getDatasets: vi.fn(),
uploadImage: vi.fn(),
getSettings: vi.fn(),
updateSettings: vi.fn(),
}));
beforeEach(() => {
vi.mocked(getModels).mockResolvedValue([
{ id: 'wan/t2v-1.3b', label: 'Wan T2V' },
]);
vi.mocked(getDatasets).mockResolvedValue([]);
});
describe('CreateJobButton', () => {
it('opens the workload menu on click and selects an item', async () => {
const user = userEvent.setup();
@@ -27,10 +29,9 @@ describe('CreateJobButton', () => {
await user.click(screen.getByRole('button', { name: 'Create Job' }));
await user.click(screen.getByRole('menuitem', { name: /I2V/i }));
expect(screen.getByRole('dialog')).toHaveAttribute(
'data-workload-type',
'i2v',
);
expect(
screen.getByRole('dialog', { name: 'New Inference Job (I2V)' }),
).toBeInTheDocument();
});
it('opens and operates the workload menu from the keyboard', async () => {
@@ -45,9 +46,42 @@ describe('CreateJobButton', () => {
expect(firstItem).toHaveFocus();
await user.keyboard('{Enter}');
expect(screen.getByRole('dialog')).toHaveAttribute(
'data-workload-type',
't2v',
expect(
screen.getByRole('dialog', { name: 'New Inference Job (T2V)' }),
).toBeInTheDocument();
await user.keyboard('{Escape}');
await waitFor(() =>
expect(screen.queryByRole('dialog')).not.toBeInTheDocument(),
);
await waitFor(() =>
expect(document.body.style.pointerEvents).not.toBe('none'),
);
expect(trigger).toHaveFocus();
});
it.each(['inference', 'finetuning', 'distillation'] as const)(
'restores page interaction after closing the real %s dialog',
async (jobType) => {
const user = userEvent.setup();
render(<CreateJobButton jobType={jobType} />);
const trigger = screen.getByRole('button', { name: 'Create Job' });
// Keep the real Dialog mounted: mocking it hides conflicting Radix layers.
for (let attempt = 0; attempt < 2; attempt++) {
await user.click(trigger);
await user.click(screen.getAllByRole('menuitem')[0]);
const dialog = screen.getByRole('dialog');
await user.click(
within(dialog).getByRole('button', { name: 'Close' }),
);
await waitFor(() =>
expect(screen.queryByRole('dialog')).not.toBeInTheDocument(),
);
await waitFor(() =>
expect(document.body.style.pointerEvents).not.toBe('none'),
);
expect(trigger).toHaveFocus();
}
},
);
});
@@ -2,7 +2,7 @@
import * as React from 'react';
import { ChevronDown } from 'lucide-react';
import * as DropdownMenu from '@radix-ui/react-dropdown-menu';
import { DropdownMenu } from 'radix-ui';
import CreateJobModal from '@/components/jobs/CreateJobModal';
import { Button } from '@/components/ui/button';
@@ -16,6 +16,7 @@ interface CreateJobButtonProps {
export default function CreateJobButton({ jobType }: CreateJobButtonProps) {
const options = WORKLOAD_OPTIONS[jobType] ?? [];
const triggerRef = React.useRef<HTMLButtonElement>(null);
const [modalOpen, setModalOpen] = React.useState(false);
const [workloadType, setWorkloadType] = React.useState(
@@ -36,7 +37,7 @@ export default function CreateJobButton({ jobType }: CreateJobButtonProps) {
<>
<DropdownMenu.Root>
<DropdownMenu.Trigger asChild>
<Button type="button" className="gap-1.5">
<Button ref={triggerRef} type="button" className="gap-1.5">
Create Job
<ChevronDown className="size-3.5 opacity-85" aria-hidden />
</Button>
@@ -66,6 +67,11 @@ export default function CreateJobButton({ jobType }: CreateJobButtonProps) {
<CreateJobModal
isOpen={modalOpen}
onClose={() => setModalOpen(false)}
onCloseAutoFocus={(event) => {
// This dialog opens from a menu item, so it has no DialogTrigger.
event.preventDefault();
triggerRef.current?.focus();
}}
onSuccess={handleSuccess}
jobType={jobType}
workloadType={workloadType}
@@ -22,30 +22,60 @@ import { useStore } from '@/hooks/useStore';
import { defaultOptionsStore } from '@/stores/defaultOptions';
import {
createJob,
updateJob,
getDatasets,
getModels,
uploadImage,
uploadMedia,
type CreateJobRequest,
type Model,
} from '@/lib/api';
import { getDefaultModelForWorkload } from '@/lib/defaultOptions';
import { WORKLOAD_OPTIONS } from '@/lib/jobConfig';
import type { JobType } from '@/lib/types';
import {
H3_MAX_REFERENCES,
labelReferences,
referencePromptSeed,
validateReferences,
type H3Reference,
} from '@/lib/h3References';
import {
EMPTY_H3_PROMPT_FIELDS,
H3_PROMPT_SECTIONS,
H3_SECTION_HINTS,
H3_SECTION_LABELS,
isEmptyPromptFields,
parseH3Prompt,
serializeH3Prompt,
type H3PromptFields,
} from '@/lib/h3Prompt';
import { jobToFormFields, type JobLike } from '@/lib/jobToFields';
export interface CreateJobModalProps {
isOpen: boolean;
onClose: () => void;
onCloseAutoFocus?: React.ComponentProps<
typeof DialogContent
>['onCloseAutoFocus'];
onSuccess: () => void;
jobType: JobType;
workloadType: string;
/** When set, the modal edits this pending job instead of creating a new one. */
editingJob?: JobLike | null;
/** Show the configuration without allowing changes (started/finished jobs). */
readOnly?: boolean;
}
export default function CreateJobModal({
isOpen,
onClose,
onCloseAutoFocus,
onSuccess,
jobType,
workloadType,
editingJob,
readOnly = false,
}: CreateJobModalProps) {
const { options } = useStore(defaultOptionsStore);
@@ -55,8 +85,22 @@ export default function CreateJobModal({
const [models, setModels] = React.useState<Model[]>([]);
const [modelId, setModelId] = React.useState('');
const [name, setName] = React.useState('');
const [prompt, setPrompt] = React.useState('');
const [imagePath, setImagePath] = React.useState('');
const [lastImagePath, setLastImagePath] = React.useState('');
const [references, setReferences] = React.useState<H3Reference[]>([]);
const [isUploadingReference, setIsUploadingReference] = React.useState(false);
const [referenceError, setReferenceError] = React.useState<string | null>(null);
const [promptFields, setPromptFields] = React.useState<H3PromptFields>(
EMPTY_H3_PROMPT_FIELDS,
);
const [useGuidedPrompt, setUseGuidedPrompt] = React.useState(true);
const [lastImageFileName, setLastImageFileName] = React.useState('');
const [isUploadingLastImage, setIsUploadingLastImage] = React.useState(false);
const [lastImageUploadError, setLastImageUploadError] = React.useState<
string | null
>(null);
const [imageFileName, setImageFileName] = React.useState('');
const [isUploadingImage, setIsUploadingImage] = React.useState(false);
const [negativePrompt, setNegativePrompt] = React.useState('');
@@ -70,12 +114,50 @@ export default function CreateJobModal({
const [seed, setSeed] = React.useState(1024);
const [numGpus, setNumGpus] = React.useState(1);
const [ditCpuOffload, setDitCpuOffload] = React.useState(false);
const [ditLayerwiseOffload, setDitLayerwiseOffload] = React.useState(false);
const [textEncoderCpuOffload, setTextEncoderCpuOffload] =
React.useState(false);
const [vaeCpuOffload, setVaeCpuOffload] = React.useState(false);
const [imageEncoderCpuOffload, setImageEncoderCpuOffload] =
React.useState(false);
const [useFsdpInference, setUseFsdpInference] = React.useState(false);
// H3 is the only registered model with an end frame or references.
const supportsLastImage = modelId.toLowerCase().includes('minimax-h3');
const usingReferences = supportsLastImage && references.length > 0;
// JobCard re-renders on every job-list poll, so `editingJob` is a fresh
// object each time. Effects must depend on these, never on the object.
const editingJobId = editingJob?.id ?? null;
const editingJobModelId = editingJob?.model_id ?? null;
// Layerwise offload and FSDP compete for the DiT weights and FastVideoArgs
// silently picks a winner (fastvideo_args.py:859); resolve it visibly here.
// dit_cpu_offload is deliberately not interlocked -- it is a modifier, not a
// competing strategy.
const handleDitLayerwiseOffloadChange = React.useCallback((next: boolean) => {
setDitLayerwiseOffload(next);
if (next) {
setUseFsdpInference(false);
}
}, []);
const handleUseFsdpInferenceChange = React.useCallback((next: boolean) => {
setUseFsdpInference(next);
if (next) {
setDitLayerwiseOffload(false);
}
}, []);
const handleNumGpusChange = React.useCallback((next: number) => {
setNumGpus(next);
if (next > 1) {
// Dropping back to one GPU leaves FSDP alone: single-GPU FSDP is a
// valid way to reach its CPU offload (docs/inference/offloading.md).
setUseFsdpInference(true);
setDitLayerwiseOffload(false);
}
}, []);
const [enableTorchCompile, setEnableTorchCompile] = React.useState(false);
const [vsaSparsity, setVsaSparsity] = React.useState(0);
const [tpSize, setTpSize] = React.useState(-1);
@@ -126,6 +208,47 @@ export default function CreateJobModal({
const justOpened = isOpen && !justOpenedRef.current;
justOpenedRef.current = isOpen;
if (!justOpened) return;
if (editingJob) {
// Must not fall through to the defaults below: a partially-seeded
// form silently edits values the user never saw.
const f = jobToFormFields(editingJob);
setModelId(f.modelId);
setName(f.name);
setPrompt(f.prompt);
setNegativePrompt(f.negativePrompt);
setImagePath(f.imagePath);
setImageFileName(f.imagePath.split('/').pop() ?? '');
setLastImagePath(f.lastImagePath);
setLastImageFileName(f.lastImagePath.split('/').pop() ?? '');
setReferences(f.references);
setPromptFields(f.promptFields ?? EMPTY_H3_PROMPT_FIELDS);
setUseGuidedPrompt(f.promptFields !== null);
setNumInferenceSteps(f.numInferenceSteps);
setNumFrames(f.numFrames);
setHeight(f.height);
setWidth(f.width);
setGuidanceScale(f.guidanceScale);
setGuidanceRescale(f.guidanceRescale);
setFps(f.fps);
setSeed(f.seed);
setNumGpus(f.numGpus);
setDitCpuOffload(f.ditCpuOffload);
setDitLayerwiseOffload(f.ditLayerwiseOffload);
setTextEncoderCpuOffload(f.textEncoderCpuOffload);
setVaeCpuOffload(f.vaeCpuOffload);
setImageEncoderCpuOffload(f.imageEncoderCpuOffload);
setUseFsdpInference(f.useFsdpInference);
setEnableTorchCompile(f.enableTorchCompile);
setVsaSparsity(f.vsaSparsity);
setTpSize(f.tpSize);
setSpSize(f.spSize);
setReferenceError(null);
setModelLoadError(null);
setImageUploadError(null);
setLastImageUploadError(null);
setSubmitError(null);
return;
}
const opts = options;
setNumInferenceSteps(opts.numInferenceSteps);
setNumFrames(workloadType === 't2i' ? 1 : opts.numFrames);
@@ -137,6 +260,7 @@ export default function CreateJobModal({
setSeed(opts.seed);
setNumGpus(opts.numGpus);
setDitCpuOffload(opts.ditCpuOffload);
setDitLayerwiseOffload(opts.ditLayerwiseOffload ?? false);
setTextEncoderCpuOffload(opts.textEncoderCpuOffload);
setVaeCpuOffload(opts.vaeCpuOffload);
setImageEncoderCpuOffload(opts.imageEncoderCpuOffload);
@@ -151,8 +275,16 @@ export default function CreateJobModal({
inferenceWorkload as 't2v' | 'i2v' | 't2i',
),
);
setName('');
setImagePath('');
setImageFileName('');
setLastImagePath('');
setLastImageFileName('');
setLastImageUploadError(null);
setReferences([]);
setReferenceError(null);
setPromptFields(EMPTY_H3_PROMPT_FIELDS);
setUseGuidedPrompt(true);
setSelectedDatasetId('');
setSelectedValidationDatasetId('');
setModelLoadError(null);
@@ -168,7 +300,7 @@ export default function CreateJobModal({
setRealScoreModelPath('');
setFakeScoreModelPath('');
}
}, [isOpen, workloadType, inferenceWorkload, options]);
}, [isOpen, workloadType, inferenceWorkload, options, editingJobId]);
// Load the models available for this workload.
React.useEffect(() => {
@@ -188,7 +320,16 @@ export default function CreateJobModal({
opts,
inferenceWorkload as 't2v' | 'i2v' | 't2i',
);
const chosen = ids.includes(defaultId) ? defaultId : (list[0]?.id ?? '');
// When editing, the job's own model wins over the workload default --
// this resolves after the seeding effect, so choosing a default here
// would silently swap the model out from under the user.
const editedId = editingJobModelId;
const chosen =
editedId && ids.includes(editedId)
? editedId
: ids.includes(defaultId)
? defaultId
: (list[0]?.id ?? '');
setModelId(chosen);
if (workloadType === 'dmd_t2v') {
setRealScoreModelPath(chosen);
@@ -210,7 +351,7 @@ export default function CreateJobModal({
return () => {
stale = true;
};
}, [isOpen, inferenceWorkload, workloadType]);
}, [isOpen, inferenceWorkload, workloadType, editingJobModelId]);
// Training jobs need a dataset; load the ready datasets when relevant.
React.useEffect(() => {
@@ -262,6 +403,106 @@ export default function CreateJobModal({
}
}
async function handleLastImageChange(
e: React.ChangeEvent<HTMLInputElement>,
) {
const file = e.target.files?.[0];
if (!file) {
setLastImagePath('');
setLastImageFileName('');
setLastImageUploadError(null);
return;
}
setIsUploadingLastImage(true);
setLastImageFileName(file.name);
setLastImageUploadError(null);
try {
const { path } = await uploadImage(file);
setLastImagePath(path);
} catch (error) {
console.error('Failed to upload end image:', error);
setLastImagePath('');
setLastImageFileName('');
setLastImageUploadError(
error instanceof Error
? `${error.message}. Choose the image again to retry.`
: 'The image could not be uploaded. Choose it again to retry.',
);
} finally {
setIsUploadingLastImage(false);
}
}
async function handleAddReference(
e: React.ChangeEvent<HTMLInputElement>,
) {
const file = e.target.files?.[0];
e.target.value = ''; // allow re-picking the same file
if (!file) return;
setIsUploadingReference(true);
setReferenceError(null);
try {
const { path, media_type } = await uploadMedia(file);
const next: H3Reference[] = [
...references,
{
id: `${Date.now()}-${file.name}`,
source: path,
media_type,
fileName: file.name,
},
];
setReferences(next);
setReferenceError(validateReferences(next));
} catch (error) {
console.error('Failed to upload reference:', error);
setReferenceError(
error instanceof Error ? error.message : 'The file could not be uploaded.',
);
} finally {
setIsUploadingReference(false);
}
}
function removeReference(id: string) {
const next = references.filter((r) => r.id !== id);
setReferences(next);
setReferenceError(validateReferences(next));
}
function seedPromptFields() {
setPromptFields({
...EMPTY_H3_PROMPT_FIELDS,
...referencePromptSeed(references),
});
setUseGuidedPrompt(true);
}
function setPromptField(section: string, value: string) {
setPromptFields((prev) => ({ ...prev, [section]: value }));
}
// Switching between the guided fields and the raw editor keeps whatever was
// typed: serialize on the way out, parse back on the way in.
function toggleGuidedPrompt() {
if (useGuidedPrompt) {
if (!isEmptyPromptFields(promptFields)) {
setPrompt(serializeH3Prompt(promptFields));
}
setUseGuidedPrompt(false);
} else {
const parsed = parseH3Prompt(prompt);
if (parsed) setPromptFields(parsed);
setUseGuidedPrompt(true);
}
}
function clearLastImage() {
setLastImagePath('');
setLastImageFileName('');
setLastImageUploadError(null);
}
function clearImage() {
setImagePath('');
setImageFileName('');
@@ -271,7 +512,16 @@ export default function CreateJobModal({
async function handleSubmit(e: React.FormEvent<HTMLFormElement>) {
e.preventDefault();
if (isInference && workloadType === 'i2v' && !imagePath) return;
if (isInference && workloadType === 'i2v' && !imagePath && !usingReferences)
return;
if (usingReferences && validateReferences(references)) return;
if (
usingReferences &&
useGuidedPrompt &&
isEmptyPromptFields(promptFields) &&
!prompt.trim()
)
return;
// Send the dataset id; the backend resolves it to the on-disk media dir.
const effectiveDataPath = selectedDatasetId ?? '';
if (!isInference && !selectedDatasetId) return;
@@ -285,14 +535,35 @@ export default function CreateJobModal({
try {
const payload: CreateJobRequest = {
model_id: modelId,
prompt,
name: name.trim(),
prompt:
usingReferences && useGuidedPrompt && !isEmptyPromptFields(promptFields)
? serializeH3Prompt(promptFields)
: prompt,
workload_type: workloadType,
job_type: effectiveJobType,
...(isInference
? {
...(workloadType === 'i2v' && imagePath
// Ref2VA and the FL2VA keyframes are mutually exclusive:
// _prepare_ref2va rejects image_path/last_image_path outright
// when references are present.
...(workloadType === 'i2v' && !usingReferences && imagePath
? { image_path: imagePath }
: {}),
...(workloadType === 'i2v' &&
supportsLastImage &&
!usingReferences &&
lastImagePath
? { last_image_path: lastImagePath }
: {}),
...(workloadType === 'i2v' && supportsLastImage && references.length
? {
references: references.map((r) => ({
source: r.source,
media_type: r.media_type,
})),
}
: {}),
negative_prompt: negativePrompt,
num_inference_steps: numInferenceSteps,
num_frames: numFrames,
@@ -304,6 +575,7 @@ export default function CreateJobModal({
seed,
num_gpus: numGpus,
dit_cpu_offload: ditCpuOffload,
dit_layerwise_offload: ditLayerwiseOffload,
text_encoder_cpu_offload: textEncoderCpuOffload,
vae_cpu_offload: vaeCpuOffload,
image_encoder_cpu_offload: imageEncoderCpuOffload,
@@ -334,7 +606,14 @@ export default function CreateJobModal({
: {}),
}),
};
await createJob(payload);
if (editingJob) {
await updateJob(
editingJob.id,
payload as unknown as Record<string, unknown>,
);
} else {
await createJob(payload);
}
onSuccess();
onClose();
} catch (err) {
@@ -356,9 +635,9 @@ export default function CreateJobModal({
const workloadLabel =
WORKLOAD_OPTIONS[jobType]?.find((o) => o.type === workloadType)?.label ?? '';
const title = `New ${jobType.charAt(0).toUpperCase() + jobType.slice(1)} Job${
workloadLabel ? ` (${workloadLabel})` : ''
}`;
const title = `${readOnly ? 'View' : editingJob ? 'Edit' : 'New'} ${
jobType.charAt(0).toUpperCase() + jobType.slice(1)
} Job${workloadLabel ? ` (${workloadLabel})` : ''}`;
return (
<Dialog
@@ -369,6 +648,7 @@ export default function CreateJobModal({
>
<DialogContent
className="max-h-[90vh] w-[90vw] max-w-[850px] overflow-y-auto"
onCloseAutoFocus={onCloseAutoFocus}
onEscapeKeyDown={(e) => {
if (isSubmitting) e.preventDefault();
}}
@@ -385,6 +665,23 @@ export default function CreateJobModal({
autoComplete="off"
className="flex flex-col gap-3.5"
>
{/* disabled cascades to every control inside; display:contents
keeps the parent's flex layout. */}
<fieldset
disabled={readOnly}
style={{ display: 'contents' }}
className="contents"
>
<FieldRow htmlFor="modal-name" label="Name (optional)">
<Input
id="modal-name"
value={name}
onChange={(e) => setName(e.target.value)}
placeholder="Shown on the job card and used for the output filename"
disabled={isSubmitting}
/>
</FieldRow>
<FieldRow htmlFor="modal-modelId" label="Model">
<NativeSelect
id="modal-modelId"
@@ -429,12 +726,12 @@ export default function CreateJobModal({
type="file"
accept=".png,.jpg,.jpeg,.webp,.bmp"
onChange={handleImageChange}
disabled={isSubmitting || isUploadingImage}
disabled={isSubmitting || isUploadingImage || usingReferences}
aria-describedby={
imageUploadError ? 'modal-image-error' : undefined
}
aria-invalid={imageUploadError ? true : undefined}
required
required={!usingReferences}
className="h-auto py-2 file:mr-3 file:cursor-pointer file:rounded-md file:border-0 file:bg-secondary file:px-2 file:py-1 file:text-sm file:text-secondary-foreground"
/>
{imageFileName && (
@@ -462,24 +759,169 @@ export default function CreateJobModal({
</FieldRow>
)}
<FieldRow
htmlFor="modal-prompt"
label={isInference ? 'Prompt' : 'Description'}
>
<Textarea
id="modal-prompt"
value={prompt}
onChange={(e) => setPrompt(e.target.value)}
rows={isInference ? 3 : 2}
placeholder={
isInference
? 'A curious raccoon peers through a vibrant field of yellow sunflowers…'
: 'Brief description of this training job…'
}
required
disabled={isSubmitting}
/>
</FieldRow>
{isInference && workloadType === 'i2v' && supportsLastImage && (
<FieldRow htmlFor="modal-last-image" label="End Frame (optional)">
<Input
id="modal-last-image"
type="file"
accept=".png,.jpg,.jpeg,.webp,.bmp"
onChange={handleLastImageChange}
disabled={isSubmitting || isUploadingLastImage}
aria-describedby={
lastImageUploadError ? 'modal-last-image-error' : undefined
}
aria-invalid={lastImageUploadError ? true : undefined}
className="h-auto py-2 file:mr-3 file:cursor-pointer file:rounded-md file:border-0 file:bg-secondary file:px-2 file:py-1 file:text-sm file:text-secondary-foreground"
/>
{lastImageFileName && (
<span className="mt-0.5 text-xs text-muted-foreground">
{isUploadingLastImage ? 'Uploading…' : lastImageFileName} ·{' '}
<button
type="button"
onClick={clearLastImage}
disabled={isSubmitting || isUploadingLastImage}
className="text-accent-blue underline-offset-2 hover:underline disabled:cursor-not-allowed disabled:opacity-50"
>
Clear
</button>
</span>
)}
{lastImageUploadError && (
<p
id="modal-last-image-error"
role="alert"
className="text-sm text-destructive"
>
{lastImageUploadError}
</p>
)}
</FieldRow>
)}
{isInference && workloadType === 'i2v' && supportsLastImage && (
<FieldRow htmlFor="modal-reference" label="References (Ref2VA)">
<Input
id="modal-reference"
type="file"
accept=".png,.jpg,.jpeg,.webp,.bmp,.mp4,.mov,.mkv,.webm,.avi,.wav,.mp3,.flac,.m4a,.ogg"
onChange={handleAddReference}
disabled={
isSubmitting ||
isUploadingReference ||
references.length >= H3_MAX_REFERENCES
}
className="h-auto py-2 file:mr-3 file:cursor-pointer file:rounded-md file:border-0 file:bg-secondary file:px-2 file:py-1 file:text-sm file:text-secondary-foreground"
/>
{isUploadingReference && (
<span className="mt-0.5 text-xs text-muted-foreground">
Uploading…
</span>
)}
{references.length > 0 && (
<ul className="mt-1 flex list-none flex-col gap-1 p-0">
{references.map((reference, index) => (
<li
key={reference.id}
className="flex items-center gap-2 text-xs text-muted-foreground"
>
<code className="font-mono text-accent-blue">
{labelReferences(references)[index]}
</code>
<span className="truncate">{reference.fileName}</span>
<button
type="button"
onClick={() => removeReference(reference.id)}
disabled={isSubmitting}
className="ml-auto text-accent-blue underline-offset-2 hover:underline disabled:cursor-not-allowed disabled:opacity-50"
>
Remove
</button>
</li>
))}
</ul>
)}
{references.length > 0 && (
<button
type="button"
onClick={seedPromptFields}
disabled={isSubmitting}
className="mt-1 self-start text-xs text-accent-blue underline-offset-2 hover:underline disabled:cursor-not-allowed disabled:opacity-50"
>
Fill prompt sections from references
</button>
)}
{referenceError && (
<p role="alert" className="mt-0.5 text-xs text-destructive">
{referenceError}
</p>
)}
<span className="mt-0.5 text-xs text-muted-foreground">
Ref2VA replaces the keyframes: up to 9 images, 3 videos, 3 audio
(12 total). Audio needs at least one image or video.
</span>
</FieldRow>
)}
{usingReferences && useGuidedPrompt ? (
/* Six-section format from the model's reference prompt guide. */
<>
{H3_PROMPT_SECTIONS.map((section) => (
<FieldRow
key={section}
htmlFor={`modal-prompt-${section}`}
label={H3_SECTION_LABELS[section]}
>
<Textarea
id={`modal-prompt-${section}`}
value={promptFields[section]}
onChange={(e) => setPromptField(section, e.target.value)}
rows={section === 'detailed_description' ? 5 : 2}
placeholder={H3_SECTION_HINTS[section]}
disabled={isSubmitting}
/>
</FieldRow>
))}
<button
type="button"
onClick={toggleGuidedPrompt}
disabled={isSubmitting}
className="self-start text-xs text-accent-blue underline-offset-2 hover:underline disabled:cursor-not-allowed disabled:opacity-50"
>
Edit as raw prompt
</button>
</>
) : (
<>
<FieldRow
htmlFor="modal-prompt"
label={isInference ? 'Prompt' : 'Description'}
>
<Textarea
id="modal-prompt"
value={prompt}
onChange={(e) => setPrompt(e.target.value)}
rows={isInference ? 3 : 2}
placeholder={
isInference
? 'A curious raccoon peers through a vibrant field of yellow sunflowers…'
: 'Brief description of this training job…'
}
required={!(usingReferences && useGuidedPrompt)}
disabled={isSubmitting}
/>
</FieldRow>
{usingReferences && (
<button
type="button"
onClick={toggleGuidedPrompt}
disabled={isSubmitting}
className="self-start text-xs text-accent-blue underline-offset-2 hover:underline disabled:cursor-not-allowed disabled:opacity-50"
>
Edit as prompt sections
</button>
)}
</>
)}
{isInference && (
<FieldRow htmlFor="modal-negative-prompt" label="Negative Prompt">
@@ -849,6 +1291,13 @@ export default function CreateJobModal({
onChange={setDitCpuOffload}
disabled={isSubmitting}
/>
<ToggleRow
id="modal-dit-layerwise-offload"
label="DiT Layerwise Offload"
checked={ditLayerwiseOffload}
onChange={handleDitLayerwiseOffloadChange}
disabled={isSubmitting}
/>
<ToggleRow
id="modal-text-encoder-cpu-offload"
label="Text Encoder CPU Offload"
@@ -860,7 +1309,7 @@ export default function CreateJobModal({
id="modal-use-fsdp-inference"
label="Use FSDP Inference"
checked={useFsdpInference}
onChange={setUseFsdpInference}
onChange={handleUseFsdpInferenceChange}
disabled={isSubmitting}
/>
<ToggleRow
@@ -891,7 +1340,7 @@ export default function CreateJobModal({
max={8}
step={1}
value={numGpus}
onChange={setNumGpus}
onChange={handleNumGpusChange}
disabled={isSubmitting}
/>
<NumberRow
@@ -906,6 +1355,8 @@ export default function CreateJobModal({
</details>
)}
</fieldset>
<div className="flex flex-col items-start gap-2">
{submitError && (
<p role="alert" className="text-sm text-destructive">
@@ -914,14 +1365,22 @@ export default function CreateJobModal({
)}
<Button
type="submit"
hidden={readOnly}
disabled={
readOnly ||
isSubmitting ||
isUploadingImage ||
!!modelLoadError ||
!!datasetLoadError
}
>
{isSubmitting ? 'Creating…' : 'Create Job'}
{isSubmitting
? editingJob
? 'Saving…'
: 'Creating…'
: editingJob
? 'Save Changes'
: 'Create Job'}
</Button>
</div>
</form>
@@ -3,11 +3,13 @@
import * as React from 'react';
import { Timer } from 'lucide-react';
import CreateJobModal from '@/components/jobs/CreateJobModal';
import { Badge, type BadgeProps } from '@/components/ui/badge';
import { Button } from '@/components/ui/button';
import { useStore } from '@/hooks/useStore';
import {
deleteJob,
duplicateJob,
downloadJobVideo,
startJob,
stopJob,
@@ -62,6 +64,8 @@ export default function JobCard({ job, onJobUpdated }: JobCardProps) {
const isSelected = activeJobId === job.id;
const [isLoading, setIsLoading] = React.useState(false);
const [isEditing, setIsEditing] = React.useState(false);
const [isViewing, setIsViewing] = React.useState(false);
const [currentTime, setCurrentTime] = React.useState(() => Date.now());
const elapsedTime = computeElapsed(job, currentTime);
@@ -103,6 +107,21 @@ export default function JobCard({ job, onJobUpdated }: JobCardProps) {
}
}
async function handleDuplicate(e: React.MouseEvent) {
e.preventDefault();
e.stopPropagation();
if (isLoading) return;
setIsLoading(true);
try {
await duplicateJob(job.id);
onJobUpdated?.();
} catch (err) {
alert(err instanceof Error ? err.message : 'Failed to duplicate job');
} finally {
setIsLoading(false);
}
}
async function handleDelete(e: React.MouseEvent) {
e.preventDefault();
e.stopPropagation();
@@ -156,16 +175,23 @@ export default function JobCard({ job, onJobUpdated }: JobCardProps) {
>
<span className="flex flex-wrap items-center justify-between gap-2">
<span className="text-[0.95rem] font-semibold text-foreground">
{job.model_id}
{job.name?.trim() || job.model_id}
</span>
<Badge variant={BADGE_VARIANTS[job.status] ?? 'secondary'}>
{job.status}
</Badge>
</span>
<span className="max-w-full overflow-hidden text-ellipsis whitespace-nowrap text-sm text-muted-foreground">
{job.prompt}
{job.name?.trim() ? `${job.model_id} · ${job.prompt}` : job.prompt}
</span>
<span className="flex flex-wrap items-center gap-4 text-xs text-muted-foreground">
{/* Short job id; logs and output dirs are keyed on the full UUID. */}
<span
className="font-mono text-muted-foreground/80"
title={job.id}
>
{job.id.slice(0, 8)}
</span>
{job.job_type === 'inference' ? (
<>
<span>{job.num_frames} frames</span>
@@ -226,6 +252,51 @@ export default function JobCard({ job, onJobUpdated }: JobCardProps) {
Download Video
</Button>
)}
{!(
job.status === 'pending' ||
job.status === 'failed' ||
job.status === 'stopped'
) && (
<Button
size="sm"
variant="outline"
onClick={(e) => {
e.preventDefault();
e.stopPropagation();
setIsViewing(true);
}}
disabled={isLoading}
title="View this job's configuration"
>
View
</Button>
)}
{(job.status === 'pending' ||
job.status === 'failed' ||
job.status === 'stopped') && (
<Button
size="sm"
variant="outline"
onClick={(e) => {
e.preventDefault();
e.stopPropagation();
setIsEditing(true);
}}
disabled={isLoading}
title="Edit this job's configuration"
>
Edit
</Button>
)}
<Button
size="sm"
variant="outline"
onClick={handleDuplicate}
disabled={isLoading}
title="Create a new pending job with this configuration"
>
Duplicate
</Button>
<Button
size="sm"
variant="destructive"
@@ -235,6 +306,30 @@ export default function JobCard({ job, onJobUpdated }: JobCardProps) {
Delete
</Button>
</div>
{isViewing && (
<CreateJobModal
isOpen
readOnly
editingJob={job}
jobType={(job.job_type ?? 'inference') as never}
workloadType={job.workload_type ?? 't2v'}
onClose={() => setIsViewing(false)}
onSuccess={() => setIsViewing(false)}
/>
)}
{isEditing && (
<CreateJobModal
isOpen
editingJob={job}
jobType={(job.job_type ?? 'inference') as never}
workloadType={job.workload_type ?? 't2v'}
onClose={() => setIsEditing(false)}
onSuccess={() => {
setIsEditing(false);
onJobUpdated?.();
}}
/>
)}
</article>
);
}
@@ -153,9 +153,16 @@ export default function JobDetailsSidebar({
}}
>
<div className="flex items-center justify-between border-b border-border px-5 py-4">
<h2 className="m-0 text-base font-semibold text-foreground">
Job Details
</h2>
<div className="min-w-0">
<h2 className="m-0 text-base font-semibold text-foreground">
Job Details
</h2>
{/* Full job id: keys ~/h3_studio_logs/<id>.log, the output directory
and every API route, so make it selectable for copy/paste. */}
<code className="mt-0.5 block select-all truncate font-mono text-xs text-muted-foreground">
{job.id}
</code>
</div>
<div className="flex items-center gap-2">
<Button
type="button"
@@ -1,5 +1,7 @@
import * as React from 'react';
import { render, screen } from '@testing-library/react';
import { describe, expect, it } from 'vitest';
import userEvent from '@testing-library/user-event';
import { describe, expect, it, vi } from 'vitest';
import { Button } from './button';
import { Input } from './input';
@@ -8,6 +10,24 @@ import { Slider } from './slider';
import { Switch } from './switch';
describe('shared control accessibility', () => {
it('forwards refs and click handlers to the asChild button', async () => {
const user = userEvent.setup();
const ref = React.createRef<HTMLButtonElement>();
const onClick = vi.fn();
render(
<Button asChild ref={ref} onClick={onClick}>
<button type="button">Slotted action</button>
</Button>,
);
const button = screen.getByRole('button', { name: 'Slotted action' });
expect(screen.getAllByRole('button')).toHaveLength(1);
expect(ref.current).toBe(button);
await user.click(button);
expect(onClick).toHaveBeenCalledTimes(1);
expect(button).toHaveFocus();
});
it('keeps button, input, and select targets at least 44px tall', () => {
render(
<>
@@ -1,7 +1,7 @@
"use client";
import * as React from "react";
import { Slot } from "@radix-ui/react-slot";
import { Slot } from "radix-ui";
import { cva, type VariantProps } from "class-variance-authority";
import { cn } from "@/lib/utils";
@@ -37,7 +37,7 @@ export interface ButtonProps extends React.ButtonHTMLAttributes<HTMLButtonElemen
}
const Button = React.forwardRef<HTMLButtonElement, ButtonProps>(({ className, variant, size, asChild = false, ...props }, ref) => {
const Comp = asChild ? Slot : "button";
const Comp = asChild ? Slot.Root : "button";
return <Comp className={cn(buttonVariants({ variant, size, className }))} ref={ref} {...props} />;
});
Button.displayName = "Button";
@@ -1,7 +1,7 @@
'use client';
import * as React from 'react';
import * as DialogPrimitive from '@radix-ui/react-dialog';
import { Dialog as DialogPrimitive } from 'radix-ui';
import { X } from 'lucide-react';
import { cn } from '@/lib/utils';
@@ -1,7 +1,7 @@
'use client';
import * as React from 'react';
import * as LabelPrimitive from '@radix-ui/react-label';
import { Label as LabelPrimitive } from 'radix-ui';
import { cva, type VariantProps } from 'class-variance-authority';
import { cn } from '@/lib/utils';
@@ -1,7 +1,7 @@
'use client';
import * as React from 'react';
import * as ScrollAreaPrimitive from '@radix-ui/react-scroll-area';
import { ScrollArea as ScrollAreaPrimitive } from 'radix-ui';
import { cn } from '@/lib/utils';
@@ -1,7 +1,7 @@
'use client';
import * as React from 'react';
import * as SelectPrimitive from '@radix-ui/react-select';
import { Select as SelectPrimitive } from 'radix-ui';
import { Check, ChevronDown, ChevronUp } from 'lucide-react';
import { cn } from '@/lib/utils';
@@ -1,7 +1,7 @@
'use client';
import * as React from 'react';
import * as SeparatorPrimitive from '@radix-ui/react-separator';
import { Separator as SeparatorPrimitive } from 'radix-ui';
import { cn } from '@/lib/utils';
@@ -1,7 +1,7 @@
'use client';
import * as React from 'react';
import * as SliderPrimitive from '@radix-ui/react-slider';
import { Slider as SliderPrimitive } from 'radix-ui';
import { cn } from '@/lib/utils';
@@ -1,7 +1,7 @@
'use client';
import * as React from 'react';
import * as SwitchPrimitives from '@radix-ui/react-switch';
import { Switch as SwitchPrimitives } from 'radix-ui';
import { cn } from '@/lib/utils';
@@ -1,7 +1,7 @@
'use client';
import * as React from 'react';
import * as TabsPrimitive from '@radix-ui/react-tabs';
import { Tabs as TabsPrimitive } from 'radix-ui';
import { cn } from '@/lib/utils';
+61
View File
@@ -56,6 +56,8 @@ export function getJobVideoUrl(jobId: string): string {
export interface CreateJobRequest {
model_id: string;
/** Optional label; the card and output filename fall back to the prompt. */
name?: string;
prompt: string;
workload_type?: string;
job_type?: JobType;
@@ -152,6 +154,65 @@ export async function updateSettings(
return response.json();
}
export type MediaType = "image" | "video" | "audio";
/**
* Upload an image, video or audio file for a MiniMax-H3 Ref2VA reference.
* The server derives media_type from the extension and returns it, so callers
* do not have to duplicate that mapping.
*/
/** Create a new pending job with the same configuration as an existing one. */
export async function duplicateJob(jobId: string): Promise<{ id: string }> {
const baseApiUrl = getApiBaseUrl();
const response = await fetch(`${baseApiUrl}/jobs/${jobId}/duplicate`, {
method: "POST",
});
if (!response.ok) {
const err = await response
.json()
.catch(() => ({ detail: "Duplicate failed" }));
throw new Error(err.detail || "Duplicate failed");
}
return response.json();
}
/** Edit a pending job's configuration. Started jobs are rejected by the API. */
export async function updateJob(
jobId: string,
updates: Record<string, unknown>,
): Promise<unknown> {
const baseApiUrl = getApiBaseUrl();
const response = await fetch(`${baseApiUrl}/jobs/${jobId}`, {
method: "PATCH",
headers: { "Content-Type": "application/json" },
body: JSON.stringify(updates),
});
if (!response.ok) {
const err = await response.json().catch(() => ({ detail: "Update failed" }));
throw new Error(err.detail || "Update failed");
}
return response.json();
}
export async function uploadMedia(
file: File,
): Promise<{ path: string; media_type: MediaType }> {
const baseApiUrl = getApiBaseUrl();
const formData = new FormData();
formData.append("file", file);
const response = await fetch(`${baseApiUrl}/upload-media`, {
method: "POST",
body: formData,
});
if (!response.ok) {
const err = await response
.json()
.catch(() => ({ detail: "Upload failed" }));
throw new Error(err.detail || "Upload failed");
}
return response.json();
}
export async function uploadImage(file: File): Promise<{ path: string }> {
const baseApiUrl = getApiBaseUrl();
const formData = new FormData();
@@ -16,6 +16,7 @@ export interface DefaultOptions {
seed: number;
numGpus: number;
ditCpuOffload: boolean;
ditLayerwiseOffload: boolean;
textEncoderCpuOffload: boolean;
vaeCpuOffload: boolean;
imageEncoderCpuOffload: boolean;
@@ -44,6 +45,7 @@ export const DEFAULT_OPTIONS: DefaultOptions = {
seed: 1024,
numGpus: 1,
ditCpuOffload: false,
ditLayerwiseOffload: false,
textEncoderCpuOffload: false,
vaeCpuOffload: false,
imageEncoderCpuOffload: false,
@@ -0,0 +1,65 @@
import { describe, expect, it } from "vitest";
import {
EMPTY_H3_PROMPT_FIELDS,
H3_PROMPT_SECTIONS,
isEmptyPromptFields,
parseH3Prompt,
serializeH3Prompt,
type H3PromptFields,
} from "@/lib/h3Prompt";
const filled: H3PromptFields = {
subject_definitions: "<Subject 1> is the dog in <Picture 1>.",
summary: "[reference generation] The target video shows <Subject 1>.",
retention_analysis: "<Subject 1> (appears in [Shot 1]): fully_preserved - fur retained.",
detailed_description: "[Shot 1] A medium shot establishes <Subject 1>.",
overall_soundscape: "Room tone throughout.",
non_diegetic_music: "N/A",
};
describe("serializeH3Prompt", () => {
it("emits sections in order, flush left, blank-line separated", () => {
const out = serializeH3Prompt(filled);
expect(out).toBe(
[
"subject_definitions:\n<Subject 1> is the dog in <Picture 1>.",
"summary:\n[reference generation] The target video shows <Subject 1>.",
"retention_analysis:\n<Subject 1> (appears in [Shot 1]): fully_preserved - fur retained.",
"detailed_description:\n[Shot 1] A medium shot establishes <Subject 1>.",
"overall_soundscape:\nRoom tone throughout.",
"non_diegetic_music:\nN/A",
].join("\n\n"),
);
// content is never indented
expect(out).not.toMatch(/\n {2}\S/);
});
it("fills blank sections with N/A rather than dropping them", () => {
const out = serializeH3Prompt({ ...EMPTY_H3_PROMPT_FIELDS, summary: "x" });
for (const s of H3_PROMPT_SECTIONS) expect(out).toContain(`${s}:`);
expect(out).toContain("non_diegetic_music:\nN/A");
});
});
describe("parseH3Prompt", () => {
it("round-trips a serialized prompt", () => {
expect(parseH3Prompt(serializeH3Prompt(filled))).toEqual(filled);
});
it("keeps multi-line section bodies", () => {
const p = parseH3Prompt("summary:\nline one\nline two\n\ndetailed_description:\nd");
expect(p?.summary).toBe("line one\nline two");
expect(p?.detailed_description).toBe("d");
});
it("returns null for a plain prompt", () => {
expect(parseH3Prompt("a toy car drives into a plush dog")).toBeNull();
});
});
describe("isEmptyPromptFields", () => {
it("detects blank vs filled", () => {
expect(isEmptyPromptFields(EMPTY_H3_PROMPT_FIELDS)).toBe(true);
expect(isEmptyPromptFields(filled)).toBe(false);
});
});
+94
View File
@@ -0,0 +1,94 @@
/**
* MiniMax-H3 full-reference prompt sections. Serialization follows the worked
* example in the model's VIDEO_PROMPT_WRITING_GUIDE_ref_en.md: `section_name:`
* on its own line, content flush left, one blank line between sections.
*/
export const H3_PROMPT_SECTIONS = [
'subject_definitions',
'summary',
'retention_analysis',
'detailed_description',
'overall_soundscape',
'non_diegetic_music',
] as const;
export type H3PromptSection = (typeof H3_PROMPT_SECTIONS)[number];
export type H3PromptFields = Record<H3PromptSection, string>;
export const EMPTY_H3_PROMPT_FIELDS: H3PromptFields = {
subject_definitions: '',
summary: '',
retention_analysis: '',
detailed_description: '',
overall_soundscape: '',
non_diegetic_music: '',
};
export const H3_SECTION_LABELS: Record<H3PromptSection, string> = {
subject_definitions: 'Subject definitions',
summary: 'Summary',
retention_analysis: 'Retention analysis',
detailed_description: 'Detailed description',
overall_soundscape: 'Overall soundscape',
non_diegetic_music: 'Non-diegetic music',
};
/** Per-section guidance, condensed from the guide's rules for each section. */
export const H3_SECTION_HINTS: Record<H3PromptSection, string> = {
subject_definitions:
'One line per <Subject N>. A subject may draw on several references, e.g. "<Subject 2> is the dog in <Picture 2>, <Picture 3>, and <Picture 4>."',
summary:
'One paragraph, starting with a [task type] prefix: reference generation, keyframe completion, video editing, video continuation, audio reuse, audio reference. Combine with " + ".',
retention_analysis:
'Per subject: "<Subject 1> (appears in [Shot 1], [Shot 2]): fully_preserved - what is retained."',
detailed_description:
'Playback order, using [Shot N] markers, timestamps, (S1) speaker tags and <d>[English] dialogue</d>.',
overall_soundscape: 'Ambience and physical sounds. N/A if none.',
non_diegetic_music:
'Background music audible only to the audience. N/A if none.',
};
/** True when every section is blank. */
export function isEmptyPromptFields(fields: H3PromptFields): boolean {
return H3_PROMPT_SECTIONS.every((s) => !fields[s].trim());
}
/**
* Join the sections into the prompt string the model is given. Blank sections
* become "N/A" rather than being dropped, matching the guide's example.
*/
export function serializeH3Prompt(fields: H3PromptFields): string {
return H3_PROMPT_SECTIONS.map((section) => {
const body = fields[section].trim() || 'N/A';
return `${section}:\n${body}`;
}).join('\n\n');
}
/**
* Split a serialized prompt back into sections, so switching between the
* guided fields and the raw editor does not lose work. Returns null when the
* text is not in section format (a plain prompt, say).
*/
export function parseH3Prompt(text: string): H3PromptFields | null {
const fields = { ...EMPTY_H3_PROMPT_FIELDS };
const headings = new Set<string>(H3_PROMPT_SECTIONS);
let current: H3PromptSection | null = null;
let found = false;
for (const line of text.split('\n')) {
const heading = line.trim().replace(/:$/, '');
if (line.trim().endsWith(':') && headings.has(heading)) {
current = heading as H3PromptSection;
found = true;
continue;
}
if (current) fields[current] += (fields[current] ? '\n' : '') + line;
}
if (!found) return null;
for (const section of H3_PROMPT_SECTIONS) {
fields[section] = fields[section].trim();
}
return fields;
}
@@ -0,0 +1,70 @@
import { describe, expect, it } from "vitest";
import {
labelReferences,
referencePromptSeed,
validateReferences,
type H3Reference,
} from "@/lib/h3References";
const ref = (media_type: H3Reference["media_type"], n: number): H3Reference => ({
id: `${media_type}-${n}`,
source: `/tmp/${media_type}${n}`,
media_type,
fileName: `${media_type}${n}`,
});
describe("labelReferences", () => {
it("numbers each media type independently, in list order", () => {
expect(
labelReferences([ref("image", 1), ref("video", 1), ref("image", 2)]),
).toEqual(["<Picture 1>", "<Video 1>", "<Picture 2>"]);
});
});
describe("validateReferences", () => {
it("accepts an empty list and a normal mix", () => {
expect(validateReferences([])).toBeNull();
expect(validateReferences([ref("image", 1), ref("audio", 1)])).toBeNull();
});
it("rejects audio-only lists", () => {
expect(validateReferences([ref("audio", 1)])).toMatch(/paired/);
});
it("enforces the per-type caps", () => {
const videos = [1, 2, 3, 4].map((n) => ref("video", n));
expect(validateReferences(videos)).toMatch(/At most 3 video/);
});
it("enforces the overall cap", () => {
const many = Array.from({ length: 13 }, (_, i) => ref("image", i));
expect(validateReferences(many)).toMatch(/At most 12/);
});
});
describe("referencePromptSeed", () => {
it("cites the real labels and never indents content", () => {
const seed = referencePromptSeed([ref("image", 1), ref("video", 1)]);
expect(seed.subject_definitions).toContain("<Picture 1>");
expect(seed.subject_definitions).toContain("<Video 1>");
for (const value of Object.values(seed)) {
expect(value).not.toMatch(/^ {2}\S/m);
}
});
it("uses the guide's retention_analysis form", () => {
const seed = referencePromptSeed([ref("image", 1)]);
expect(seed.retention_analysis).toMatch(/fully_preserved - /);
});
it("starts the summary with a bracketed task type", () => {
const seed = referencePromptSeed([ref("image", 1)]);
expect(seed.summary).toMatch(/^\[[a-z +]+\]/);
});
it("describes audio references separately", () => {
const seed = referencePromptSeed([ref("image", 1), ref("audio", 1)]);
expect(seed.subject_definitions).toContain("<Audio 1>");
expect(seed.retention_analysis).toContain("<Audio 1>: reference -");
});
});
@@ -0,0 +1,90 @@
/**
* MiniMax-H3 Ref2VA reference helpers. Labels mirror the per-type counters in
* `build_ref2va_presentation`, so what the user sees is what the model is shown.
*/
import type { MediaType } from "@/lib/api";
export interface H3Reference {
/** stable key for React lists */
id: string;
source: string;
media_type: MediaType;
fileName: string;
}
/** Per-media-type caps enforced by validate_references (reference.py). */
export const H3_REFERENCE_LIMITS: Record<MediaType, number> = {
image: 9,
video: 3,
audio: 3,
};
export const H3_MAX_REFERENCES = 12;
const LABEL_FOR: Record<MediaType, string> = {
image: "Picture",
video: "Video",
audio: "Audio",
};
/** Label each reference the way the pipeline will, e.g. "<Picture 2>". */
export function labelReferences(refs: H3Reference[]): string[] {
const counts: Record<MediaType, number> = { image: 0, video: 0, audio: 0 };
return refs.map((ref) => {
counts[ref.media_type] += 1;
return `<${LABEL_FOR[ref.media_type]} ${counts[ref.media_type]}>`;
});
}
/** Human-readable reason the list is invalid, or null when it is acceptable. */
export function validateReferences(refs: H3Reference[]): string | null {
if (refs.length === 0) return null;
if (refs.length > H3_MAX_REFERENCES) {
return `At most ${H3_MAX_REFERENCES} references (have ${refs.length}).`;
}
const counts: Record<MediaType, number> = { image: 0, video: 0, audio: 0 };
for (const ref of refs) counts[ref.media_type] += 1;
for (const type of Object.keys(counts) as MediaType[]) {
if (counts[type] > H3_REFERENCE_LIMITS[type]) {
return `At most ${H3_REFERENCE_LIMITS[type]} ${type} references (have ${counts[type]}).`;
}
}
if (counts.audio === refs.length) {
return "Audio references must be paired with at least one image or video.";
}
return null;
}
/** Seed the guided prompt fields from the current reference list. */
export function referencePromptSeed(
refs: H3Reference[],
): Record<string, string> {
const labels = labelReferences(refs);
const visual = labels.filter((l) => !l.startsWith("<Audio"));
const audio = labels.filter((l) => l.startsWith("<Audio"));
const subjects = visual.map(
(label, i) =>
`<Subject ${i + 1}> is the subject from ${label}; describe appearance and distinguishing features.`,
);
for (const label of audio) {
subjects.push(`${label} is the audio reference; describe what it provides.`);
}
const retention = visual.map(
(_, i) =>
`<Subject ${i + 1}> (appears in [Shot 1]): fully_preserved - what is retained.`,
);
for (const label of audio) {
retention.push(`${label}: reference - how it guides the audio.`);
}
return {
subject_definitions: subjects.join("\n"),
summary: "[reference generation] Describe the target video and each reference's role.",
retention_analysis: retention.join("\n"),
detailed_description:
"[Shot 1] Describe composition, subjects, environment, lighting, action and camera movement, saying where each reference takes effect.",
overall_soundscape: "",
non_diegetic_music: "",
};
}
@@ -0,0 +1,74 @@
import { describe, expect, it } from "vitest";
import { jobToFormFields, referenceFileName, type JobLike } from "@/lib/jobToFields";
const job: JobLike = {
id: "abc",
model_id: "MiniMaxAI/MiniMax-H3",
prompt: "subject_definitions:\n<Subject 1> is a dog.\n\nsummary:\n[reference generation] x",
workload_type: "i2v",
references: [
{ source: "/uploads/9f/wukong_source.mp4", media_type: "video" },
{ source: "/uploads/2a/MonkeyKing_0.jpg", media_type: "image" },
],
num_frames: 141,
height: 768,
width: 1344,
guidance_scale: 1.0,
guidance_rescale: 0.1,
seed: 0,
num_gpus: 4,
use_fsdp_inference: true,
};
describe("jobToFormFields", () => {
it("carries the settings that differ from form defaults", () => {
const f = jobToFormFields(job);
expect(f.numFrames).toBe(141);
expect(f.height).toBe(768);
expect(f.width).toBe(1344);
expect(f.guidanceScale).toBe(1.0);
expect(f.guidanceRescale).toBe(0.1);
expect(f.seed).toBe(0); // 0 must survive, not fall back to 1024
expect(f.numGpus).toBe(4);
expect(f.useFsdpInference).toBe(true);
});
it("does not let falsy-but-valid values fall through to defaults", () => {
const f = jobToFormFields({ ...job, seed: 0, vsa_sparsity: 0, guidance_rescale: 0 });
expect(f.seed).toBe(0);
expect(f.vsaSparsity).toBe(0);
expect(f.guidanceRescale).toBe(0);
});
it("rebuilds the reference list with readable names", () => {
const f = jobToFormFields(job);
expect(f.references).toHaveLength(2);
expect(f.references[0].fileName).toBe("wukong_source.mp4");
expect(f.references[1].media_type).toBe("image");
expect(new Set(f.references.map((r) => r.id)).size).toBe(2);
});
it("splits a six-section prompt back into fields", () => {
const f = jobToFormFields(job);
expect(f.promptFields?.subject_definitions).toBe("<Subject 1> is a dog.");
expect(f.promptFields?.summary).toBe("[reference generation] x");
});
it("returns null promptFields for a plain prompt", () => {
const f = jobToFormFields({ ...job, prompt: "a toy car" });
expect(f.promptFields).toBeNull();
expect(f.prompt).toBe("a toy car");
});
it("handles a job with no references", () => {
const f = jobToFormFields({ ...job, references: null });
expect(f.references).toEqual([]);
});
});
describe("referenceFileName", () => {
it("takes the basename", () => {
expect(referenceFileName("/a/b/c.mp4")).toBe("c.mp4");
expect(referenceFileName("c.mp4")).toBe("c.mp4");
});
});
@@ -0,0 +1,119 @@
/**
* Map a persisted job back onto the create-job form's fields. Every value the
* form seeds from defaults must be covered here, or edit mode silently shows
* defaults for whatever is missing.
*/
import type { H3Reference } from "@/lib/h3References";
import { parseH3Prompt, type H3PromptFields } from "@/lib/h3Prompt";
export interface JobLike {
id: string;
model_id: string;
name?: string;
prompt: string;
workload_type?: string;
job_type?: string;
image_path?: string;
last_image_path?: string;
references?: { source: string; media_type: string }[] | null;
negative_prompt?: string;
num_inference_steps?: number;
num_frames?: number;
height?: number;
width?: number;
guidance_scale?: number;
guidance_rescale?: number;
fps?: number;
seed?: number;
num_gpus?: number;
dit_cpu_offload?: boolean;
dit_layerwise_offload?: boolean;
text_encoder_cpu_offload?: boolean;
vae_cpu_offload?: boolean;
image_encoder_cpu_offload?: boolean;
use_fsdp_inference?: boolean;
enable_torch_compile?: boolean;
vsa_sparsity?: number;
tp_size?: number;
sp_size?: number;
}
export interface JobFormFields {
modelId: string;
name: string;
workloadType: string;
jobType: string;
prompt: string;
negativePrompt: string;
imagePath: string;
lastImagePath: string;
references: H3Reference[];
promptFields: H3PromptFields | null;
numInferenceSteps: number;
numFrames: number;
height: number;
width: number;
guidanceScale: number;
guidanceRescale: number;
fps: number;
seed: number;
numGpus: number;
ditCpuOffload: boolean;
ditLayerwiseOffload: boolean;
textEncoderCpuOffload: boolean;
vaeCpuOffload: boolean;
imageEncoderCpuOffload: boolean;
useFsdpInference: boolean;
enableTorchCompile: boolean;
vsaSparsity: number;
tpSize: number;
spSize: number;
}
/** Uploads keep their original basename, so this is the display name. */
export function referenceFileName(source: string): string {
return source.split("/").filter(Boolean).pop() ?? source;
}
export function jobToFormFields(job: JobLike): JobFormFields {
const refs: H3Reference[] = (job.references ?? []).map((r, i) => ({
id: `${job.id}-${i}`,
source: r.source,
media_type: r.media_type as H3Reference["media_type"],
fileName: referenceFileName(r.source),
}));
return {
modelId: job.model_id,
name: job.name ?? "",
workloadType: job.workload_type ?? "t2v",
jobType: job.job_type ?? "inference",
prompt: job.prompt ?? "",
negativePrompt: job.negative_prompt ?? "",
imagePath: job.image_path ?? "",
lastImagePath: job.last_image_path ?? "",
references: refs,
// null when the prompt is not in six-section form; the caller then keeps
// the raw editor rather than silently dropping content into fields.
promptFields: parseH3Prompt(job.prompt ?? ""),
numInferenceSteps: job.num_inference_steps ?? 50,
numFrames: job.num_frames ?? 81,
height: job.height ?? 480,
width: job.width ?? 832,
guidanceScale: job.guidance_scale ?? 5.0,
guidanceRescale: job.guidance_rescale ?? 0.0,
fps: job.fps ?? 24,
seed: job.seed ?? 1024,
numGpus: job.num_gpus ?? 1,
ditCpuOffload: job.dit_cpu_offload ?? false,
ditLayerwiseOffload: job.dit_layerwise_offload ?? false,
textEncoderCpuOffload: job.text_encoder_cpu_offload ?? false,
vaeCpuOffload: job.vae_cpu_offload ?? false,
imageEncoderCpuOffload: job.image_encoder_cpu_offload ?? false,
useFsdpInference: job.use_fsdp_inference ?? false,
enableTorchCompile: job.enable_torch_compile ?? false,
vsaSparsity: job.vsa_sparsity ?? 0,
tpSize: job.tp_size ?? -1,
spSize: job.sp_size ?? -1,
};
}
+1
View File
@@ -5,6 +5,7 @@ export type JobType = "inference" | "finetuning" | "distillation";
export interface Job {
id: string;
model_id: string;
name?: string;
prompt: string;
job_type?: JobType;
workload_type?: string;
@@ -0,0 +1,99 @@
# SPDX-License-Identifier: Apache-2.0
"""Duplicating a job's config, and editing one that has not started."""
import uuid
import pytest
from fastvideo_studio.database import Database
from fastvideo_studio.job_runner import JobRunner, JobStatus
@pytest.fixture
def runner(tmp_path):
return JobRunner(
output_dir=str(tmp_path / "out"),
log_dir=str(tmp_path / "logs"),
database=Database(tmp_path / "t.db"),
)
def _make(runner, **over):
kwargs = dict(
job_id=str(uuid.uuid4()),
model_id="MiniMaxAI/MiniMax-H3",
prompt="p",
workload_type="i2v",
num_frames=141,
guidance_scale=1.0,
num_gpus=4,
references=[{"source": "/x/clip.mp4", "media_type": "video"}],
)
kwargs.update(over)
return runner.create_job(**kwargs)
def test_duplicate_copies_config_but_not_runtime_state(runner):
src = _make(runner)
src.status = JobStatus.COMPLETED
src.output_path = "/x/out.mp4"
dup = runner.duplicate_job(src.id, str(uuid.uuid4()))
assert dup.id != src.id
assert dup.status is JobStatus.PENDING
assert dup.output_path is None
for field in ("model_id", "prompt", "workload_type", "num_frames",
"guidance_scale", "num_gpus", "references"):
assert getattr(dup, field) == getattr(src, field)
def test_duplicate_deep_copies_references(runner):
src = _make(runner)
dup = runner.duplicate_job(src.id, str(uuid.uuid4()))
dup.references[0]["source"] = "/changed"
assert src.references[0]["source"] == "/x/clip.mp4"
def test_duplicate_unknown_job(runner):
with pytest.raises(ValueError, match="not found"):
runner.duplicate_job("nope", str(uuid.uuid4()))
def test_edit_pending_job(runner):
job = _make(runner)
updated = runner.update_job_config(job.id, {"num_frames": 192, "seed": 7})
assert updated.num_frames == 192
assert updated.seed == 7
@pytest.mark.parametrize("status", [JobStatus.FAILED, JobStatus.STOPPED])
def test_edit_allows_restartable_jobs(runner, status):
"""Editable exactly when startable: neither has produced an output."""
job = _make(runner)
job.status = status
assert runner.update_job_config(job.id, {"seed": 7}).seed == 7
@pytest.mark.parametrize("status", [JobStatus.COMPLETED, JobStatus.RUNNING])
def test_edit_rejects_jobs_with_or_producing_a_result(runner, status):
job = _make(runner)
job.status = status
with pytest.raises(ValueError, match="can be edited"):
runner.update_job_config(job.id, {"seed": 7})
def test_edit_rejects_unknown_field(runner):
job = _make(runner)
with pytest.raises(ValueError, match="Not editable"):
runner.update_job_config(job.id, {"status": "completed"})
def test_name_is_carried_by_duplicate(runner):
src = _make(runner, name="wukong swap v2")
dup = runner.duplicate_job(src.id, str(uuid.uuid4()))
assert dup.name == "wukong swap v2"
def test_name_is_editable(runner):
job = _make(runner, name="a")
assert runner.update_job_config(job.id, {"name": "b"}).name == "b"
@@ -0,0 +1,36 @@
# SPDX-License-Identifier: Apache-2.0
"""Uploaded files keep a readable basename under a unique directory."""
from __future__ import annotations
import pytest
from fastvideo_studio.server import _safe_upload_name
@pytest.mark.parametrize(
("filename", "ext", "expected"),
[
("wukong_source.mp4", ".mp4", "wukong_source.mp4"),
("MonkeyKing_0.jpg", ".jpg", "MonkeyKing_0.jpg"),
("my clip (final).mp4", ".mp4", "my_clip_final.mp4"),
("../../etc/passwd.png", ".png", "passwd.png"),
("/abs/path/frame.png", ".png", "frame.png"),
("émoji✨.png", ".png", "moji.png"),
("", ".png", "upload.png"),
(None, ".png", "upload.png"),
("...", ".png", "upload.png"),
],
)
def test_safe_upload_name(filename, ext, expected):
assert _safe_upload_name(filename, ext) == expected
def test_long_names_are_capped():
out = _safe_upload_name("x" * 300 + ".png", ".png")
assert out == "x" * 80 + ".png"
def test_no_path_separators_survive():
for bad in ("a/b.png", "a\\b.png", "../x.png"):
assert "/" not in _safe_upload_name(bad, ".png")
assert "\\" not in _safe_upload_name(bad, ".png")
+27 -13
View File
@@ -68,7 +68,7 @@ ARG FLASH_ATTN_WHEEL_RELEASE_ARM64=https://github.com/mjun0812/flash-attention-p
# cutlass-4.4 `cute.core.ThrMma` API, which crashes on the cutlass-dsl 4.5 that
# flashinfer/quack pull in. After the wheel install we overlay this cutlass-4.5-safe
# upstream cute (flash-attn-4) so the image runs FA4 instead of the FA2 fallback.
ARG FA4_CUTE_REF=82d6441eec5d4dfec120153db2c0145ae855a083
ARG FA4_CUTE_REF=14c377950125c70b7a9dabf9c561fca53715ac7d
# Provided automatically by BuildKit/buildx (e.g. "amd64" / "arm64") and used to
# select the prebuilt flash-attn wheel. Empty under a plain `docker build` without
@@ -89,11 +89,29 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
zsh \
vim \
curl \
ffmpeg \
libgl1 \
libglib2.0-0 \
libx11-dev \
gcc-11 \
g++-11 \
clang-11 \
cmake \
pkg-config \
build-essential \
libssl-dev \
&& rm -rf /var/lib/apt/lists/*
# Rust toolchain: some dependencies only ship sdists on aarch64 and need cargo
# to build. The dormant legacy Modal image layers the identical apt set +
# rustup on top of this image (fastvideo/tests/modal/pr_test.py); baking both
# here keeps its manual rollback path reproducible without changing the Slurm
# runner's package surface.
RUN set -o pipefail && \
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain stable --profile minimal && \
/root/.cargo/bin/cargo --version && /root/.cargo/bin/rustc --version
ENV PATH=/root/.cargo/bin:${PATH}
# Set up C++20 compilers for ThunderKittens
RUN update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-11 100 --slave /usr/bin/g++ g++ /usr/bin/g++-11
@@ -138,6 +156,7 @@ RUN --mount=type=cache,target=/opt/uv/cache \
source /opt/venv/bin/activate && \
uv pip install --upgrade pip && \
uv pip install --excludes docker/uv-excludes ".[dev]" && \
python -c "import cv2; print('OpenCV', cv2.__version__)" && \
PYTAG=cp$(echo "${PYTHON_VERSION}" | tr -d .) && \
case "${TARGETARCH:-amd64}" in \
amd64) \
@@ -169,26 +188,21 @@ RUN --mount=type=cache,target=/opt/uv/cache \
# flash_attn/__init__.py), so FA2/varlen/bert_padding stay from the install above;
# rmtree clears the wheel's stale cute files first to avoid an install conflict.
# Then verify both survive so a broken overlay fails the build instead of shipping
# an FA2-less image. x86 only: the FA4 stack (quack-kernels etc.) is unvalidated on
# arm64 / GB10 (sm_121), so there we skip the overlay; FA4 is opt-in
# (FASTVIDEO_FA4=1) and errors if set without the overlay, so leave it unset on
# arm64 and the image runs FA3/FA2 as usual.
# an FA2-less image. The pinned stack is validated on ARM64 GB200 (sm_100) as well
# as x86; FA4 remains opt-in through FASTVIDEO_FA4=1 so lanes with FA2 baselines
# keep their existing numerics.
RUN --mount=type=cache,target=/opt/uv/cache \
source $HOME/.local/bin/env && \
source /opt/venv/bin/activate && \
if [ "${TARGETARCH}" = "arm64" ]; then \
echo "Skipping FA4 cute overlay on arm64 (FA4 stack unvalidated there; do not set FASTVIDEO_FA4)"; \
else \
python -c "import glob, shutil; [shutil.rmtree(d, ignore_errors=True) for d in glob.glob('/opt/venv/lib/python*/site-packages/flash_attn/cute')]" && \
uv pip install "flash-attn-4 @ git+https://github.com/Dao-AILab/flash-attention.git@${FA4_CUTE_REF}#subdirectory=flash_attn/cute" && \
python -c "import flash_attn; assert hasattr(flash_attn, 'flash_attn_func'), 'FA2 was clobbered by the cute overlay'; import flash_attn.cute; print('FA2 + FA4 cute OK')"; \
fi
python -c "import glob, shutil; [shutil.rmtree(d, ignore_errors=True) for d in glob.glob('/opt/venv/lib/python*/site-packages/flash_attn/cute')]" && \
uv pip install "flash-attn-4 @ git+https://github.com/Dao-AILab/flash-attention.git@${FA4_CUTE_REF}#subdirectory=flash_attn/cute" && \
python -c "import flash_attn; assert hasattr(flash_attn, 'flash_attn_func'), 'FA2 was clobbered by the cute overlay'; import flash_attn.cute; print('FA2 + FA4 cute OK')"
COPY . .
# Build immutable FastVideo kernel wheels for the published image. The requested
# architecture remains installed for normal image users; amd64 images also carry
# an SM89 artifact so the predominant L40S Modal lanes can reuse it exactly.
# an SM89 artifact for L40S users and the dormant legacy rollback path.
ARG FASTVIDEO_KERNEL_PREBUILT_DIR=/opt/fastvideo-kernel-prebuilt
RUN --mount=type=cache,target=/opt/uv/cache \
source $HOME/.local/bin/env && \
+842
View File
@@ -0,0 +1,842 @@
{
"version": 11,
"recipes": [
{
"id": "fastwan21-t2v",
"family": "wan",
"stage": "inference",
"task": "Text to video",
"label": "FastWan2.1 1.3B (distilled + VSA)",
"summary": "Generate a video in three denoising steps with the distilled FastWan2.1 1.3B checkpoint and video sparse attention.",
"model": "FastVideo/FastWan2.1-T2V-1.3B-Diffusers",
"source": "scripts/inference/inference_wan_VSA_DMD_1_3B.yaml",
"command": "FASTVIDEO_ATTENTION_BACKEND=VIDEO_SPARSE_ATTN fastvideo generate --config scripts/inference/inference_wan_VSA_DMD_1_3B.yaml",
"gpu_types": ["NVIDIA"],
"hardware": {"gpu_count": 1, "evidence": "source-configured"},
"evidence": "Source-backed",
"expected_artifact": "MP4 videos under outputs_video_dmd_1.3B/"
},
{
"id": "wan22-t2v",
"family": "wan",
"stage": "inference",
"task": "Text to video",
"label": "Wan2.2 A14B",
"summary": "The maintained high-capacity Wan2.2 text-to-video example with CPU offload settings encoded in its checked-in Python source.",
"model": "Wan-AI/Wan2.2-T2V-A14B-Diffusers",
"source": "examples/inference/basic/basic_wan2_2.py",
"command": "python examples/inference/basic/basic_wan2_2.py",
"gpu_types": ["NVIDIA"],
"hardware": {"gpu_count": 2, "evidence": "source-configured"},
"evidence": "Source-backed",
"expected_artifact": "MP4 videos under video_samples_wan2_2_14B_t2v/"
},
{
"id": "wan21-i2v",
"family": "wan",
"stage": "inference",
"task": "Image to video",
"label": "Wan2.1 14B 480P",
"summary": "Animate an input image at 480P using the maintained Wan2.1 YAML configuration and its recorded offload settings.",
"model": "Wan-AI/Wan2.1-I2V-14B-480P-Diffusers",
"source": "scripts/inference/inference_wan_i2v.yaml",
"command": "fastvideo generate --config scripts/inference/inference_wan_i2v.yaml",
"gpu_types": ["NVIDIA"],
"hardware": {"gpu_count": 2, "evidence": "source-configured"},
"evidence": "Source-backed"
},
{
"id": "wan22-ti2v",
"family": "wan",
"stage": "inference",
"task": "Text or image to video",
"label": "Wan2.2 TI2V 5B",
"summary": "Use one maintained 5B checkpoint for text-to-video or add an image input to switch the same recipe to image-to-video.",
"model": "Wan-AI/Wan2.2-TI2V-5B-Diffusers",
"source": "examples/inference/basic/basic_wan2_2_ti2v.py",
"command": "python examples/inference/basic/basic_wan2_2_ti2v.py",
"gpu_types": ["NVIDIA"],
"hardware": {"gpu_count": 1, "evidence": "source-configured"},
"evidence": "Source-backed",
"expected_artifact": "MP4 videos under video_samples_wan2_2_5B_ti2v/"
},
{
"id": "fastmetal-1-3b-mlx",
"family": "wan",
"stage": "inference",
"task": "Text to video",
"label": "FastMetal 1.3B",
"summary": "Run the released FastMetal 1.3B QAD checkpoint through FastVideo's native Apple Silicon MLX path.",
"model": "FastVideo/FastMetal-1.3B-QAD",
"source": "examples/inference/basic/mlx_wan_prompt_to_video.py",
"command": "hf download FastVideo/FastMetal-1.3B-QAD --local-dir ./FastMetal-1.3B-QAD\npython examples/inference/basic/mlx_wan_prompt_to_video.py --model-root ./FastMetal-1.3B-QAD --mlx-checkpoint ./FastMetal-1.3B-QAD --height 480 --width 832 --num-frames 81 --prompt \"A bird's-eye view of a misty forest valley at dawn.\" --output-path ./outputs/fastmetal_1_3b.mp4",
"gpu_types": ["Apple Silicon"],
"hardware": {
"platform": "mlx",
"accelerator": "Apple M4 Max",
"system_memory": "36 GB unified memory",
"minimum_memory": "16 GB+ unified memory",
"peak_memory": "3.87 GiB peak MLX memory",
"evidence": "validated",
"evidence_url": "https://github.com/hao-ai-lab/FastVideo/pull/1638"
},
"evidence": "Verified",
"expected_artifact": "MP4 video at outputs/fastmetal_1_3b.mp4",
"modes": ["T2V", "temporal --fast", "spatial --fast-spatial", "two-pass --refine"],
"limitations": [
"Native MLX FastMetal T2V. Add --fast for temporal RIFE, --fast-spatial to denoise at half resolution, or --refine for a two-pass upsample. basic_mps.py is the older PyTorch MPS demo."
]
},
{
"id": "fastmetal-5b-mlx",
"family": "wan",
"stage": "inference",
"task": "Text to video",
"label": "FastMetal 5B",
"summary": "Run the released Wan2.2 5B FastMetal checkpoint as MLX T2V with MLX DiT denoising and MLX TAEHV decode. The CUDA Wan2.2 TI2V 5B recipe is the image-capable path.",
"model": "FastVideo/FastMetal-5B-QAD",
"source": "examples/inference/basic/mlx_wan22_generate.py",
"command": "hf download FastVideo/FastMetal-5B-QAD --local-dir ./FastMetal-5B-QAD\npython examples/inference/basic/mlx_wan22_generate.py --mlx-checkpoint ./FastMetal-5B-QAD --text-encoder-root ./FastMetal-5B-QAD --vae-root ./FastMetal-5B-QAD/vae --height 704 --width 1280 --num-frames 81 --prompt \"A cinematic portrait with soft neon lighting and smooth camera motion.\" --output-path ./outputs/fastmetal_5b.mp4",
"gpu_types": ["Apple Silicon"],
"hardware": {
"platform": "mlx",
"accelerator": "Apple M4 Max",
"system_memory": "36 GB unified memory",
"minimum_memory": "16 GB+ unified memory",
"peak_memory": "9.34 GiB peak MLX memory",
"evidence": "validated",
"evidence_url": "https://github.com/hao-ai-lab/FastVideo/pull/1638"
},
"evidence": "Verified",
"expected_artifact": "MP4 video at outputs/fastmetal_5b.mp4",
"modes": ["T2V", "temporal --fast", "spatial --fast-spatial", "two-pass --refine"],
"limitations": [
"The checked-in MLX example is T2V. Image-to-video is not in mlx_wan22_generate.py. Add --fast, --fast-spatial, or --refine on the same script."
]
},
{
"id": "fastmetal-14b-mlx",
"family": "wan",
"stage": "inference",
"task": "Text to video",
"label": "FastMetal 14B",
"summary": "Run the released 14B FastMetal QAD checkpoint through the same Apple Silicon MLX entrypoint as the 1.3B release.",
"model": "FastVideo/FastMetal-14B-QAD",
"source": "examples/inference/basic/mlx_wan_prompt_to_video.py",
"command": "hf download FastVideo/FastMetal-14B-QAD --local-dir ./FastMetal-14B-QAD\npython examples/inference/basic/mlx_wan_prompt_to_video.py --model-root ./FastMetal-14B-QAD --mlx-checkpoint ./FastMetal-14B-QAD --height 480 --width 832 --num-frames 81 --prompt \"A wide cinematic landscape at sunrise.\" --output-path ./outputs/fastmetal_14b.mp4",
"gpu_types": ["Apple Silicon"],
"hardware": {
"platform": "mlx",
"accelerator": "Apple M4 Max",
"system_memory": "36 GB unified memory",
"minimum_memory": "36 GB+ unified memory",
"peak_memory": "21.68 GiB peak MLX memory",
"evidence": "validated",
"evidence_url": "https://github.com/hao-ai-lab/FastVideo/pull/1638"
},
"evidence": "Verified",
"expected_artifact": "MP4 video at outputs/fastmetal_14b.mp4",
"modes": ["T2V", "temporal --fast", "spatial --fast-spatial", "two-pass --refine"],
"limitations": [
"Native MLX FastMetal T2V on 36 GB+ unified memory. Same --fast, --fast-spatial, and --refine flags as the 1.3B script."
]
},
{
"id": "turbodiffusion-wan21-1-3b-t2v",
"family": "turbodiffusion",
"stage": "inference",
"task": "Text to video",
"label": "TurboWan2.1 1.3B",
"summary": "A TurboDiffusion-accelerated Wan2.1 1.3B text-to-video run from its maintained single-GPU example.",
"model": "loayrashid/TurboWan2.1-T2V-1.3B-Diffusers",
"source": "examples/inference/basic/basic_turbodiffusion.py",
"command": "python examples/inference/basic/basic_turbodiffusion.py",
"gpu_types": ["NVIDIA"],
"hardware": {"gpu_count": 1, "evidence": "source-configured"},
"evidence": "Source-backed",
"expected_artifact": "MP4 videos under video_samples_turbodiffusion/",
"related": ["turbodiffusion-wan21-14b-t2v", "turbowan22-i2v"]
},
{
"id": "turbodiffusion-wan21-14b-t2v",
"family": "turbodiffusion",
"stage": "inference",
"task": "Text to video",
"label": "TurboWan2.1 14B",
"summary": "TurboDiffusion acceleration applied to the 14B Wan2.1 text-to-video checkpoint; the checked-in source is configured for two GPUs.",
"model": "loayrashid/TurboWan2.1-T2V-14B-Diffusers",
"source": "examples/inference/basic/basic_turbodiffusion_14b.py",
"command": "python examples/inference/basic/basic_turbodiffusion_14b.py",
"gpu_types": ["NVIDIA"],
"hardware": {"gpu_count": 2, "evidence": "source-configured"},
"evidence": "Source-backed",
"expected_artifact": "MP4 videos under video_samples_turbodiffusion_14B/",
"related": ["turbodiffusion-wan21-1-3b-t2v", "turbowan22-i2v"]
},
{
"id": "turbowan22-i2v",
"family": "turbodiffusion",
"stage": "inference",
"task": "Image to video",
"label": "TurboWan2.2 A14B",
"summary": "A one-to-four-step image-to-video path using TurboDiffusion and the SLA attention backend from its maintained example.",
"model": "loayrashid/TurboWan2.2-I2V-A14B-Diffusers",
"source": "examples/inference/basic/basic_turbodiffusion_i2v.py",
"command": "python examples/inference/basic/basic_turbodiffusion_i2v.py",
"gpu_types": ["NVIDIA"],
"hardware": {"gpu_count": 2, "evidence": "source-configured"},
"evidence": "Source-backed",
"related": ["turbodiffusion-wan21-14b-t2v"]
},
{
"id": "ltx2-distilled-t2v",
"family": "ltx2",
"stage": "inference",
"task": "Text to video",
"label": "LTX-2 distilled",
"summary": "The distilled LTX-2 text-to-video checkpoint with audio, from its maintained example. The source is configured for four GPUs.",
"model": "FastVideo/LTX2-Distilled-Diffusers",
"source": "examples/inference/basic/basic_ltx2_distilled.py",
"command": "python examples/inference/basic/basic_ltx2_distilled.py",
"gpu_types": ["NVIDIA"],
"hardware": {"gpu_count": 4, "evidence": "source-configured"},
"evidence": "Source-backed",
"expected_artifact": "MP4 (video with audio) at outputs_video/ltx2_basic/output_ltx2_distilled_t2v.mp4",
"related": ["ltx23-base-t2v"]
},
{
"id": "ltx23-base-t2v",
"family": "ltx2",
"stage": "inference",
"task": "Text to video",
"label": "LTX-2 base (1088p)",
"summary": "Base LTX-2 text-to-video at 1088x1920 using FastVideo default sampling for LTX2 base. The example loads a community Diffusers mirror of the base checkpoint; registered aliases include Lightricks/LTX-2 and FastVideo/LTX2-Diffusers.",
"model": "Davids048/LTX2-Base-Diffusers",
"source": "examples/inference/basic/basic_ltx2.py",
"command": "python examples/inference/basic/basic_ltx2.py",
"gpu_types": ["NVIDIA"],
"hardware": {"gpu_count": 1, "evidence": "source-configured"},
"evidence": "Source-backed",
"expected_artifact": "MP4 (video with audio) at outputs_video/ltx2_basic/output_ltx2_base_t2v_1088_1920_1.1.mp4",
"limitations": ["The maintained example loads the Davids048/LTX2-Base-Diffusers community mirror rather than a Lightricks upstream ID."],
"related": ["ltx2-distilled-t2v"]
},
{
"id": "hy15-t2v-480p",
"family": "hunyuan",
"stage": "inference",
"task": "Text to video",
"label": "HunyuanVideo 1.5 480P",
"summary": "HunyuanVideo 1.5 text-to-video at 480P with CPU offload enabled in the checked-in source for smaller GPUs.",
"model": "hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-480p_t2v",
"source": "examples/inference/basic/basic_hy15.py",
"command": "python examples/inference/basic/basic_hy15.py",
"gpu_types": ["NVIDIA"],
"hardware": {"gpu_count": 1, "evidence": "source-configured"},
"evidence": "Source-backed",
"expected_artifact": "MP4 videos under video_samples_hy15/",
"related": ["hy15-1080p-upscale"]
},
{
"id": "hy15-1080p-upscale",
"family": "hunyuan",
"stage": "inference",
"task": "Text to video (upscaled)",
"label": "HunyuanVideo 1.5 1080P upscale",
"summary": "Run HunyuanVideo 1.5 through the 480p to 720p to 1080p upscale chain in one maintained script.",
"model": "weizhou03/HunyuanVideo-1.5-Diffusers-1080p-2SR",
"source": "examples/inference/basic/basic_hy15_1080p.py",
"command": "python examples/inference/basic/basic_hy15_1080p.py",
"gpu_types": ["NVIDIA"],
"hardware": {"gpu_count": 1, "evidence": "source-configured"},
"evidence": "Source-backed",
"expected_artifact": "MP4 videos under video_samples_hy15_1080p/",
"related": ["hy15-t2v-480p"]
},
{
"id": "cosmos25-t2w",
"family": "cosmos",
"stage": "inference",
"task": "Text to world",
"label": "Cosmos Predict 2.5 2B",
"summary": "Generate a navigable world video from a text prompt with Cosmos Predict 2.5 2B on a single GPU.",
"model": "KyleShao/Cosmos-Predict2.5-2B-Diffusers",
"source": "examples/inference/basic/basic_cosmos2_5_t2w.py",
"command": "python examples/inference/basic/basic_cosmos2_5_t2w.py",
"gpu_types": ["NVIDIA"],
"hardware": {"gpu_count": 1, "evidence": "source-configured"},
"evidence": "Source-backed"
},
{
"id": "kandinsky5-t2v-lite-sft",
"family": "kandinsky5",
"stage": "inference",
"task": "Text to video",
"label": "Kandinsky 5.0 T2V Lite SFT",
"summary": "Kandinsky 5.0 text-to-video (Lite SFT variant) from the maintained example; alternative Lite/Pro checkpoints are listed in the source.",
"model": "kandinskylab/Kandinsky-5.0-T2V-Lite-sft-5s-Diffusers",
"source": "examples/inference/basic/basic_kandinsky5_t2v.py",
"command": "python examples/inference/basic/basic_kandinsky5_t2v.py",
"gpu_types": ["NVIDIA"],
"hardware": {
"platform": "cuda",
"gpu_count": 1,
"accelerator": "NVIDIA B200",
"evidence": "validated",
"evidence_url": "https://github.com/hao-ai-lab/FastVideo/pull/1471"
},
"evidence": "Verified",
"expected_artifact": "MP4 videos under video_samples_kandinsky5_t2v/",
"related": ["kandinsky5-i2v-pro-distilled"]
},
{
"id": "kandinsky5-i2v-pro-distilled",
"family": "kandinsky5",
"stage": "inference",
"task": "Image to video",
"label": "Kandinsky 5.0 I2V Pro distilled",
"summary": "Animate an input image with Kandinsky 5.0 I2V Pro (distilled) on a single GPU.",
"model": "kandinskylab/Kandinsky-5.0-I2V-Pro-distilled-5s-Diffusers",
"source": "examples/inference/basic/basic_kandinsky5_i2v.py",
"command": "python examples/inference/basic/basic_kandinsky5_i2v.py",
"gpu_types": ["NVIDIA"],
"hardware": {
"platform": "cuda",
"gpu_count": 1,
"accelerator": "NVIDIA B200",
"peak_memory": "10,365.89 MB peak GPU memory",
"evidence": "validated",
"evidence_url": "https://github.com/hao-ai-lab/FastVideo/pull/1471"
},
"evidence": "Verified",
"expected_artifact": "MP4 videos under video_samples_kandinsky5_i2v/",
"related": ["kandinsky5-t2v-lite-sft"]
},
{
"id": "flux2-klein-t2i",
"family": "flux",
"stage": "inference",
"task": "Text to image",
"label": "FLUX.2 Klein 4B",
"summary": "Generate an image in four denoising steps with the distilled FLUX.2 Klein checkpoint.",
"model": "black-forest-labs/FLUX.2-klein-4B",
"source": "examples/inference/basic/basic_flux2_klein.py",
"command": "python examples/inference/basic/basic_flux2_klein.py",
"gpu_types": ["NVIDIA"],
"hardware": {"gpu_count": 1, "evidence": "source-configured"},
"evidence": "Source-backed",
"expected_artifact": "PNG image at outputs/flux2/flux2_klein.png",
"related": ["flux2-dev-t2i"]
},
{
"id": "flux2-dev-t2i",
"family": "flux",
"stage": "inference",
"task": "Text to image",
"label": "FLUX.2 dev",
"summary": "Full FLUX.2 dev text-to-image with embedded guidance and the Mistral3 text encoder, from its maintained example.",
"model": "black-forest-labs/FLUX.2-dev",
"source": "examples/inference/basic/basic_flux2.py",
"command": "python examples/inference/basic/basic_flux2.py",
"gpu_types": ["NVIDIA"],
"hardware": {"gpu_count": 1, "evidence": "source-configured"},
"evidence": "Source-backed",
"expected_artifact": "PNG image at outputs/flux2/flux2.png",
"related": ["flux2-klein-t2i"]
},
{
"id": "flux1-dev-t2i",
"family": "flux",
"stage": "inference",
"task": "Text to image",
"label": "FLUX.1 dev",
"summary": "FLUX.1 dev text-to-image through the Diffusers-backed pipeline. The example defaults to a local weights directory, so this recipe passes the Hugging Face ID explicitly.",
"model": "black-forest-labs/FLUX.1-dev",
"source": "examples/inference/basic/basic_flux_dev.py",
"command": "python examples/inference/basic/basic_flux_dev.py --model-path black-forest-labs/FLUX.1-dev",
"gpu_types": ["NVIDIA"],
"hardware": {"gpu_count": 1, "evidence": "source-configured"},
"evidence": "Source-backed",
"expected_artifact": "PNG images under outputs/flux_dev/samples/",
"limitations": ["FLUX.1 is loadable by ID but registers no model_family in fastvideo/registry.py; it is grouped under FLUX for documentation only."]
},
{
"id": "glm-image-t2i",
"family": "glm_image",
"stage": "inference",
"task": "Text to image",
"label": "GLM-Image",
"summary": "GLM-Image text-to-image generation from its maintained example.",
"model": "zai-org/GLM-Image",
"source": "examples/inference/basic/basic_glm_image.py",
"command": "python examples/inference/basic/basic_glm_image.py",
"gpu_types": ["NVIDIA"],
"hardware": {"gpu_count": 1, "evidence": "source-configured"},
"evidence": "Source-backed",
"expected_artifact": "PNG image at image_output/landscape.png",
"related": ["glm-image-edit"]
},
{
"id": "glm-image-edit",
"family": "glm_image",
"stage": "inference",
"task": "Image editing",
"label": "GLM-Image editing",
"summary": "Edit an input image with an instruction prompt using GLM-Image, from its maintained editing example.",
"model": "zai-org/GLM-Image",
"source": "examples/inference/basic/edit_glm_image.py",
"command": "python examples/inference/basic/edit_glm_image.py",
"gpu_types": ["NVIDIA"],
"hardware": {"gpu_count": 1, "evidence": "source-configured"},
"evidence": "Source-backed",
"expected_artifact": "PNG image at image_output/edited.png (input: assets/images/couple.jpg)",
"related": ["glm-image-t2i"]
},
{
"id": "zimage-turbo-t2i",
"family": "zimage",
"stage": "inference",
"task": "Text to image",
"label": "Z-Image Turbo",
"summary": "Z-Image Turbo text-to-image on a single GPU from its maintained example.",
"model": "Tongyi-MAI/Z-Image-Turbo",
"source": "examples/inference/basic/basic_zimage.py",
"command": "python examples/inference/basic/basic_zimage.py",
"gpu_types": ["NVIDIA"],
"hardware": {"gpu_count": 1, "evidence": "source-configured"},
"evidence": "Source-backed",
"expected_artifact": "PNG image at outputs/zimage/zimage_turbo.png"
},
{
"id": "sd35-medium-t2i",
"family": "sd35",
"stage": "inference",
"task": "Text to image",
"label": "Stable Diffusion 3.5 Medium",
"summary": "Stable Diffusion 3.5 Medium text-to-image over a small built-in prompt set, from its maintained example.",
"model": "stabilityai/stable-diffusion-3.5-medium",
"source": "examples/inference/basic/basic_sd35_t2i.py",
"command": "python examples/inference/basic/basic_sd35_t2i.py",
"gpu_types": ["NVIDIA"],
"hardware": {"gpu_count": 1, "evidence": "source-configured"},
"evidence": "Source-backed",
"expected_artifact": "PNG images under outputs/sd35/samples/"
},
{
"id": "minimax-h3-t2v",
"family": "minimax_h3",
"stage": "inference",
"task": "Text to video (with audio)",
"label": "MiniMax H3 T2VA",
"summary": "Generate synchronized video and stereo audio from a structured text prompt with the full MiniMax H3 checkpoint.",
"model": "MiniMaxAI/MiniMax-H3",
"source": "examples/inference/basic/basic_minimax_h3_t2v.py",
"command": "python examples/inference/basic/basic_minimax_h3_t2v.py --prompt \"(S1) A presenter says <d>[English] FastVideo runs MiniMax H3.</d>\"",
"gpu_types": ["NVIDIA"],
"hardware": {"platform": "cuda", "gpu_count": 4, "evidence": "source-configured"},
"evidence": "Source-backed",
"expected_artifact": "MP4 with synchronized audio at outputs/minimax_h3_t2v/minimax_h3_t2v.mp4",
"modes": ["T2VA"],
"knobs": [
{"key": "num_gpus", "label": "GPUs", "hint": "Sequence-parallel degree", "flag": "--num-gpus", "options": [1, 2, 4, 8], "default": 4}
],
"limitations": ["The checked-in example defaults to four-way sequence parallelism. It does not record a GPU model or memory requirement."]
},
{
"id": "fasth3-preview-cuda",
"group": "fasth3-preview",
"group_label": "FastH3 V1",
"group_task": "4-step text to video + audio",
"family": "minimax_h3",
"stage": "inference",
"task": "Few-step text to video (with audio)",
"label": "FastH3 V1 on CUDA",
"summary": "Run FastH3 V1 with four DiT forwards, trained H3 sparse attention, compiled decode, and synchronized audio.",
"model": "FastVideo/FastVideo-Minimax-FastH3-Preview-v0.2",
"source": "examples/inference/basic/basic_fasth3.py",
"serving": {
"source": "examples/serving/openai_fasth3.yaml",
"install": "UV_TORCH_BACKEND=cu130 uv pip install -e \".[fasth3]\""
},
"command": "UV_TORCH_BACKEND=cu130 uv pip install -e \".[fasth3]\"\npython examples/inference/basic/basic_fasth3.py --prompt \"(S1) A presenter says <d>[English] FastVideo runs FastH3.</d>\" --profile all",
"gpu_types": ["NVIDIA"],
"hardware": {
"platform": "cuda",
"gpu_count": 4,
"accelerator": "NVIDIA GB200",
"evidence": "validated",
"evidence_url": "https://github.com/hao-ai-lab/FastVideo/pull/1731"
},
"evidence": "Verified",
"expected_artifact": "Warmup and measured MP4 files under outputs/fasth3/",
"modes": ["T2VA", "4-step FastH3"],
"knobs": [
{"key": "num_gpus", "label": "GPUs", "hint": "Sequence-parallel degree", "flag": "--num-gpus", "options": [1, 2, 4, 8], "default": 4},
{"key": "video_decode_backend", "label": "VAE decode", "hint": "Fidelity vs. speed", "flag": "--video-decode-backend", "options": [{"value": "h3-vae", "label": "Full H3 VAE"}, {"value": "taeh3", "label": "TAEH3 preview"}], "default": "h3-vae"}
],
"limitations": ["The default all profile is the measured GB200 performance route and can change floating-point operation order. Use --profile strict --no-inference-torch-compile for the eager strict route."]
},
{
"id": "fasth3-preview-mlx",
"serving": {
"source": "examples/serving/mlx_fasth3.yaml",
"install": "uv pip install -e \".[mlx]\"",
"prepare": "hf download FastVideo/FastVideo-Minimax-FastH3-Preview-v0.2 --local-dir ./FastH3-Preview-v0.2\npython scripts/checkpoint_conversion/convert_minimax_h3_mlx.py --model-root ./FastH3-Preview-v0.2/transformer --out ./FastH3-MLX --formats \"int6\""
},
"group": "fasth3-preview",
"group_label": "FastH3 V1",
"group_task": "4-step text to video + audio",
"family": "minimax_h3",
"stage": "inference",
"task": "Few-step text to video (with audio)",
"label": "FastH3 V1 on MLX",
"summary": "Run FastH3 V1 on Apple Silicon with a locally converted INT6 DiT, streamed Qwen3-VL conditioning, and native MLX video and audio VAEs.",
"model": "FastVideo/FastVideo-Minimax-FastH3-Preview-v0.2",
"source": "examples/inference/basic/mlx_fasth3.py",
"command": "hf download FastVideo/FastVideo-Minimax-FastH3-Preview-v0.2 --local-dir ./FastH3-Preview-v0.2\npython scripts/checkpoint_conversion/convert_minimax_h3_mlx.py --model-root ./FastH3-Preview-v0.2/transformer --out ./FastH3-MLX --formats \"int6\"\npython examples/inference/basic/mlx_fasth3.py --model-root ./FastH3-Preview-v0.2 --mlx-checkpoint ./FastH3-MLX/int6 --prompt \"(S1) A presenter says <d>[English] FastVideo runs FastH3.</d>\" --height 480 --width 832 --num-frames 124 --seed 2026 --output-path ./outputs/fasth3_int6.mp4",
"gpu_types": ["Apple Silicon"],
"hardware": {
"platform": "mlx",
"accelerator": "Apple M4 Max",
"system_memory": "36 GB unified memory",
"peak_memory": "19.63 GiB peak MLX memory during denoising",
"evidence": "validated",
"evidence_url": "https://github.com/hao-ai-lab/FastVideo/pull/1770"
},
"evidence": "Verified",
"expected_artifact": "MP4 with H.264 video and stereo AAC audio at outputs/fasth3_int6.mp4",
"modes": ["T2VA", "temporal --fast", "spatial --fast-spatial", "opt-in VSA"],
"knobs": [
{"key": "video_decode_backend", "label": "VAE decode", "hint": "Fidelity vs. speed", "flag": "--video-decode-backend", "options": [{"value": "h3-vae", "label": "Full H3 VAE"}, {"value": "taeh3", "label": "TAEH3 preview"}], "default": "h3-vae"}
],
"limitations": [
"The MLX path supports T2VA, optional temporal --fast, optional spatial --fast-spatial, and opt-in VSA on --include-vsa checkpoints. FL2VA, Ref2VA, and two-pass refinement are not wired."
]
},
{
"id": "fasth3-preview-spark",
"group": "fasth3-preview",
"group_label": "FastH3 V1",
"group_task": "4-step text to video + audio",
"family": "minimax_h3",
"stage": "inference",
"task": "Few-step text to video (with audio)",
"label": "FastH3 V1 on one DGX Spark",
"summary": "Run FastH3 V1 on one GB10 with Triton VSA, FA4 off, and lazy module load. Height, width, frames, and steps in the YAML are examples.",
"model": "FastVideo/FastVideo-FastH3-4-step-Preview-v1-VSA-DataFree",
"source": "examples/inference/basic/basic_fasth3_spark.yaml",
"serving": {
"source": "examples/serving/openai_fasth3_spark.yaml",
"install": "UV_TORCH_BACKEND=cu130 uv pip install -e .",
"env": "FASTVIDEO_VSA_SM100A=0 FASTVIDEO_FA4=0 FASTVIDEO_ATTENTION_BACKEND=VIDEO_SPARSE_ATTN_H3"
},
"command": "FASTVIDEO_VSA_SM100A=0 FASTVIDEO_FA4=0 FASTVIDEO_ATTENTION_BACKEND=VIDEO_SPARSE_ATTN_H3 FASTVIDEO_STAGE_LOGGING=1 fastvideo generate --config examples/inference/basic/basic_fasth3_spark.yaml",
"gpu_types": ["NVIDIA"],
"hardware": {
"platform": "cuda",
"device": "spark",
"gpu_count": 1,
"evidence": "source-configured"
},
"evidence": "Source-backed",
"expected_artifact": "MP4 under outputs/fasth3_spark/",
"modes": ["T2VA", "1-Spark"],
"limitations": [
"Install from the DGX Spark guide, not the generic CUDA extra. GB10 has no FA4 / sm_100a VSA kernel; keep FASTVIDEO_FA4=0 and FASTVIDEO_VSA_SM100A=0.",
"Legal num_frames values are 17n+5, capped at 345 (15 s). Native 16:9 sizes include 832x480 and 1344x768.",
"Lazy module load reloads Qwen3-VL and the DiT between phases of each request. Do not pass --no-lazy-module-load on this box.",
"A 345-frame request on one Spark can OOM. Prefer 124 or 243 frames, TAEH3 decode, or two Sparks over QSFP."
]
},
{
"id": "fasth3-spark-pair",
"group": "fasth3-preview",
"group_label": "FastH3 V1",
"group_task": "4-step text to video + audio",
"family": "minimax_h3",
"stage": "inference",
"task": "Few-step text to video (with audio)",
"label": "FastH3 V1 on two DGX Sparks",
"summary": "Run one FastH3 clip across two GB10s with Ray sequence parallel over QSFP RoCE. Sequential load and lazy module load stay on because SP replicates the DiT on each node.",
"model": "FastVideo/FastVideo-FastH3-4-step-Preview-v1-VSA-DataFree",
"source": "examples/inference/basic/basic_fasth3_spark_pair.yaml",
"command": "source examples/inference/optimizations/spark_pair_env.sh && FASTVIDEO_VSA_SM100A=0 FASTVIDEO_FA4=0 FASTVIDEO_ATTENTION_BACKEND=VIDEO_SPARSE_ATTN_H3 FASTVIDEO_VAE_PARALLEL_DECODE=1 fastvideo generate --config examples/inference/basic/basic_fasth3_spark_pair.yaml",
"gpu_types": ["NVIDIA"],
"hardware": {
"platform": "cuda",
"device": "spark",
"gpu_count": 2,
"accelerator": "NVIDIA GB10 (DGX Spark pair)",
"evidence": "validated",
"evidence_url": "https://github.com/hao-ai-lab/FastVideo/pull/1803"
},
"evidence": "Verified",
"expected_artifact": "MP4 under outputs/fasth3_spark_pair/",
"modes": ["T2VA", "2-Spark SP"],
"limitations": [
"Requires a two-node Ray cluster on the QSFP interconnect. There is no cookbook server for this path; use Python / generate.",
"Height, width, frames, and steps in the YAML are examples. Edit them or pass CLI flags. See docs/getting_started/installation/spark_pair.md."
]
},
{
"id": "fasth3-8step-v2-cuda",
"group": "fasth3-8step-v2",
"group_label": "FastH3 V2",
"group_task": "8-step text to video + audio",
"family": "minimax_h3",
"stage": "inference",
"task": "Few-step text to video (with audio)",
"label": "FastH3 V2 on CUDA",
"summary": "Run FastH3 V2, the eight-forward checkpoint (video/audio shifts 10/3, VSA 0.8, 64-token tiles) with the trained DMD ladder loaded from the checkpoint's fastvideo_inference.json.",
"model": "FastVideo/FastVideo-FastH3-8-Step-V2",
"source": "examples/inference/basic/basic_fasth3_8step.py",
"serving": {
"source": "examples/serving/openai_fasth3_8step.yaml",
"install": "UV_TORCH_BACKEND=cu130 uv pip install -e \".[fasth3]\""
},
"command": "UV_TORCH_BACKEND=cu130 uv pip install -e \".[fasth3]\"\npython examples/inference/basic/basic_fasth3_8step.py --prompt \"(S1) A presenter says <d>[English] FastVideo runs FastH3.</d>\" --profile strict --no-inference-torch-compile --no-compile-vae",
"gpu_types": ["NVIDIA"],
"hardware": {
"platform": "cuda",
"gpu_count": 4,
"accelerator": "NVIDIA GB200",
"evidence": "validated",
"evidence_url": "https://github.com/hao-ai-lab/FastVideo/pull/1852"
},
"evidence": "Verified",
"expected_artifact": "Warmup and measured MP4 files under outputs/fasth3_8step/",
"modes": ["T2VA", "8-step FastH3"],
"knobs": [
{"key": "num_gpus", "label": "GPUs", "hint": "Sequence-parallel degree", "flag": "--num-gpus", "options": [1, 2, 4, 8], "default": 4},
{"key": "video_decode_backend", "label": "VAE decode", "hint": "Fidelity vs. speed", "flag": "--video-decode-backend", "options": [{"value": "h3-vae", "label": "Full H3 VAE"}, {"value": "taeh3", "label": "TAEH3 preview"}], "default": "h3-vae"}
],
"limitations": [
"Nine sigma-grid points (eight transformer forwards) are fixed by the checkpoint's trained ladder; the example rejects any other --steps.",
"Validated with the eager strict route (--profile strict --no-inference-torch-compile --no-compile-vae). The compiled all profile has not been measured for this checkpoint.",
"T2AV only; no FL2VA/Ref2VA distillation and no matching LoRA. V2 uses eight forwards rather than V1's four."
]
},
{
"id": "fasth3-8step-v2-mlx",
"serving": {
"source": "examples/serving/mlx_fasth3_8step.yaml",
"install": "uv pip install -e \".[mlx]\"",
"prepare": "hf download FastVideo/FastVideo-FastH3-8-Step-V2 --local-dir ./FastH3-8-Step-V2\npython scripts/checkpoint_conversion/convert_minimax_h3_mlx.py --model-root ./FastH3-8-Step-V2/transformer --out ./FastH3-8-Step-V2-MLX --formats \"int8\" --include-vsa"
},
"group": "fasth3-8step-v2",
"group_label": "FastH3 V2",
"group_task": "8-step text to video + audio",
"family": "minimax_h3",
"stage": "inference",
"task": "Few-step text to video (with audio)",
"label": "FastH3 V2 on MLX",
"summary": "Run FastH3 V2 on Apple Silicon with a locally converted INT8 DiT, the checkpoint's DMD contract ladder, and trained VSA (sparsity 0.8, 64-token tiles).",
"model": "FastVideo/FastVideo-FastH3-8-Step-V2",
"source": "examples/inference/basic/mlx_fasth3_8step.py",
"command": "hf download FastVideo/FastVideo-FastH3-8-Step-V2 --local-dir ./FastH3-8-Step-V2\npython scripts/checkpoint_conversion/convert_minimax_h3_mlx.py --model-root ./FastH3-8-Step-V2/transformer --out ./FastH3-8-Step-V2-MLX --formats \"int8\" --include-vsa\npython examples/inference/basic/mlx_fasth3_8step.py --model-root ./FastH3-8-Step-V2 --mlx-checkpoint ./FastH3-8-Step-V2-MLX/int8 --prompt \"(S1) A presenter says <d>[English] FastVideo runs FastH3.</d>\" --height 480 --width 832 --num-frames 124 --seed 2026 --output-path ./outputs/fasth3_8step_int8.mp4",
"gpu_types": ["Apple Silicon"],
"hardware": {
"platform": "mlx",
"accelerator": "Apple M4 Max",
"system_memory": "36 GB unified memory",
"peak_memory": "27.39 GiB peak MLX memory during denoising",
"evidence": "validated",
"evidence_url": "https://github.com/hao-ai-lab/FastVideo/pull/1863"
},
"evidence": "Verified",
"expected_artifact": "MP4 with H.264 video and stereo AAC audio at outputs/fasth3_8step_int8.mp4",
"modes": ["T2VA", "8-step FastH3", "trained VSA"],
"knobs": [
{"key": "video_decode_backend", "label": "VAE decode", "hint": "Fidelity vs. speed", "flag": "--video-decode-backend", "options": [{"value": "h3-vae", "label": "Full H3 VAE"}, {"value": "taeh3", "label": "TAEH3 preview"}], "default": "h3-vae"}
],
"limitations": [
"Convert with --include-vsa. mlx_fasth3_8step.py turns VSA on (sparsity 0.8, tile 64). A dense export fails at configure_vsa.",
"--steps 8 or 9 both run the eight trained forwards. Reuse the preview VAE, audio VAE, text encoder, and tokenizer if those directories already exist.",
"The MLX path supports T2VA only. FL2VA, Ref2VA, and two-pass refinement are not wired."
]
},
{
"id": "minimax-h3-fl2va",
"family": "minimax_h3",
"stage": "inference",
"task": "First/last frame to video (with audio)",
"label": "MiniMax H3 FL2VA",
"summary": "Animate a first frame, optionally guide the final frame, and generate synchronized audio with the full MiniMax H3 checkpoint.",
"model": "MiniMaxAI/MiniMax-H3",
"source": "examples/inference/basic/basic_minimax_h3_fl2va.py",
"command": "python examples/inference/basic/basic_minimax_h3_fl2va.py --image path/to/first-frame.png --prompt \"(S1) The subject turns toward the camera and says <d>[English] Hello.</d>\"",
"gpu_types": ["NVIDIA"],
"hardware": {"platform": "cuda", "gpu_count": 4, "evidence": "source-configured"},
"evidence": "Source-backed",
"expected_artifact": "MP4 with synchronized audio at outputs/minimax_h3_fl2va/minimax_h3_fl2va.mp4",
"modes": ["FL2VA"],
"knobs": [
{"key": "num_gpus", "label": "GPUs", "hint": "Sequence-parallel degree", "flag": "--num-gpus", "options": [1, 2, 4, 8], "default": 4}
],
"limitations": ["Pass --last-image to constrain the final frame. The checked-in source defaults to four GPUs."]
},
{
"id": "minimax-h3-ref2va",
"family": "minimax_h3",
"stage": "inference",
"task": "Reference media to video (with audio)",
"label": "MiniMax H3 Ref2VA",
"summary": "Condition H3 on an ordered reference video and optional audio reference, then generate a new synchronized video and audio result.",
"model": "MiniMaxAI/MiniMax-H3",
"source": "examples/inference/basic/basic_minimax_h3_ref2va.py",
"command": "python examples/inference/basic/basic_minimax_h3_ref2va.py --reference-video path/to/reference.mp4 --prompt \"Create a new scene that preserves the reference identity and motion language.\"",
"gpu_types": ["NVIDIA"],
"hardware": {"platform": "cuda", "gpu_count": 4, "evidence": "source-configured"},
"evidence": "Source-backed",
"expected_artifact": "MP4 with synchronized audio at outputs/minimax_h3_ref2va/minimax_h3_ref2va.mp4",
"modes": ["Ref2VA"],
"knobs": [
{"key": "num_gpus", "label": "GPUs", "hint": "Sequence-parallel degree", "flag": "--num-gpus", "options": [1, 2, 4, 8], "default": 4}
],
"limitations": ["Pass --reference-audio for an additional audio reference. The checked-in source defaults to four GPUs."]
},
{
"id": "fasth3-lora-preview",
"family": "minimax_h3",
"stage": "inference",
"task": "LoRA-adapted few-step video (with audio)",
"label": "FastH3 LoRA Preview",
"summary": "Apply a FastH3 preview adapter at load time while keeping the shared four-forward performance profile and synchronized audio output.",
"model": "MiniMaxAI/MiniMax-H3",
"source": "examples/inference/basic/basic_fasth3_lora_preview.py",
"command": "python examples/inference/basic/basic_fasth3_lora_preview.py --lora-path path/to/adapter.safetensors --prompt \"(S1) A presenter says <d>[English] This is an adapted Fast H3 run.</d>\"",
"gpu_types": ["NVIDIA"],
"hardware": {
"platform": "cuda",
"gpu_count": 4,
"accelerator": "NVIDIA B200",
"evidence": "validated",
"evidence_url": "https://github.com/hao-ai-lab/FastVideo/pull/1771"
},
"evidence": "Verified",
"expected_artifact": "Warmup and measured MP4 files under outputs/fasth3_lora_preview/",
"modes": ["T2VA", "FastH3 LoRA"],
"knobs": [
{"key": "num_gpus", "label": "GPUs", "hint": "Sequence-parallel degree", "flag": "--num-gpus", "options": [1, 2, 4, 8], "default": 4},
{"key": "video_decode_backend", "label": "VAE decode", "hint": "Fidelity vs. speed", "flag": "--video-decode-backend", "options": [{"value": "h3-vae", "label": "Full H3 VAE"}, {"value": "taeh3", "label": "TAEH3 preview"}], "default": "h3-vae"}
],
"limitations": ["Supply a compatible FastH3 adapter. The script infers dense or VSA attention from the adapter payload unless you override it."]
},
{
"id": "longcat-t2v",
"family": "longcat",
"stage": "inference",
"task": "Text to video",
"label": "LongCat Video T2V",
"summary": "LongCat Video text-to-video at 480p (50 steps), with distilled and 720p refinement passes included in the same maintained script.",
"model": "FastVideo/LongCat-Video-T2V-Diffusers",
"source": "examples/inference/basic/basic_longcat_t2v.py",
"command": "python examples/inference/basic/basic_longcat_t2v.py",
"gpu_types": ["NVIDIA"],
"hardware": {"gpu_count": 1, "evidence": "source-configured"},
"evidence": "Source-backed",
"expected_artifact": "MP4 videos under outputs_video/longcat_t2v_basic/, longcat_t2v_distill/, and longcat_t2v_refine_720p/",
"related": ["longcat-i2v"]
},
{
"id": "longcat-i2v",
"family": "longcat",
"stage": "inference",
"task": "Image to video",
"label": "LongCat Video I2V",
"summary": "LongCat Video image-to-video with optional distilled and refinement passes, from its maintained example.",
"model": "FastVideo/LongCat-Video-I2V-Diffusers",
"source": "examples/inference/basic/basic_longcat_i2v.py",
"command": "python examples/inference/basic/basic_longcat_i2v.py",
"gpu_types": ["NVIDIA"],
"hardware": {"gpu_count": 1, "evidence": "source-configured"},
"evidence": "Source-backed",
"expected_artifact": "MP4 videos under outputs_video/longcat_i2v_basic/ and longcat_i2v_distill/",
"related": ["longcat-t2v"]
},
{
"id": "stable-audio-open-t2a",
"family": "stable_audio",
"stage": "inference",
"task": "Text to audio",
"label": "Stable Audio Open 1.0",
"summary": "Six-second text-to-audio generation with Stable Audio Open 1.0 from its maintained example; duration and steps are documented knobs in the source.",
"model": "FastVideo/stable-audio-open-1.0-Diffusers",
"source": "examples/inference/basic/basic_stable_audio.py",
"command": "python examples/inference/basic/basic_stable_audio.py",
"gpu_types": ["NVIDIA"],
"hardware": {
"platform": "cuda",
"gpu_count": 1,
"accelerator": "NVIDIA B200",
"evidence": "validated",
"evidence_url": "https://github.com/hao-ai-lab/FastVideo/pull/1260"
},
"evidence": "Verified",
"expected_artifact": "WAV audio at outputs_audio/stable_audio_basic/output_stable_audio.wav",
"limitations": ["Must load the FastVideo converted Diffusers repo; upstream stabilityai monolithic checkpoints are not loader-compatible (see scripts/checkpoint_conversion/stable_audio_to_diffusers.py)."],
"related": ["stable-audio-small-t2a"]
},
{
"id": "stable-audio-small-t2a",
"family": "stable_audio",
"stage": "inference",
"task": "Text to audio",
"label": "Stable Audio Open Small",
"summary": "The smaller Stable Audio Open variant with its own shorter training window, from its maintained example.",
"model": "FastVideo/stable-audio-open-small-Diffusers",
"source": "examples/inference/basic/basic_stable_audio_small.py",
"command": "python examples/inference/basic/basic_stable_audio_small.py",
"gpu_types": ["NVIDIA"],
"hardware": {"gpu_count": 1, "evidence": "source-configured"},
"evidence": "Source-backed",
"related": ["stable-audio-open-t2a"]
},
{
"id": "mmaudio-v2a",
"family": "mmaudio",
"stage": "inference",
"task": "Video/Text to audio",
"label": "MMAudio large 44k v2",
"summary": "Add synchronized audio to a video (or from a prompt) with MMAudio large 44k v2. The example reads the model path from MMAUDIO_MODEL_PATH; this recipe passes the converted Hugging Face repo explicitly.",
"model": "FastVideo/MMAudio-large-44k-v2-Diffusers",
"source": "examples/inference/basic/basic_mmaudio.py",
"command": "MMAUDIO_MODEL_PATH=FastVideo/MMAudio-large-44k-v2-Diffusers python examples/inference/basic/basic_mmaudio.py",
"gpu_types": ["NVIDIA"],
"hardware": {"gpu_count": 1, "evidence": "source-configured"},
"evidence": "Source-backed",
"limitations": ["The upstream checkpoint must be converted to Diffusers layout via scripts/checkpoint_conversion/convert_mmaudio_to_diffusers.py unless loaded from the FastVideo converted repo as done here."]
},
{
"id": "matrix-game-2",
"family": "matrixgame",
"stage": "inference",
"task": "Interactive world",
"label": "Matrix Game 2.0",
"summary": "Generate an interactive-world sequence from the maintained Matrix Game 2.0 example.",
"model": "FastVideo/Matrix-Game-2.0-Base-Distilled-Diffusers",
"source": "examples/inference/basic/basic_matrixgame2.py",
"command": "python examples/inference/basic/basic_matrixgame2.py",
"gpu_types": ["NVIDIA"],
"hardware": {"gpu_count": 1, "evidence": "source-configured"},
"evidence": "Source-backed",
"related": ["matrix-game-3-i2w"]
},
{
"id": "matrix-game-3-i2w",
"family": "matrixgame",
"stage": "inference",
"task": "Interactive world",
"label": "Matrix Game 3.0",
"summary": "Drive Matrix Game 3.0 from an input image plus prompt at 720p, three steps, from its maintained example.",
"model": "FastVideo/Matrix-Game-3.0-Base-Distilled-Diffusers",
"source": "examples/inference/basic/basic_matrixgame3.py",
"command": "python examples/inference/basic/basic_matrixgame3.py",
"gpu_types": ["NVIDIA"],
"hardware": {"gpu_count": 1, "evidence": "source-configured"},
"evidence": "Source-backed",
"expected_artifact": "MP4 videos under video_samples_matrixgame3/",
"related": ["matrix-game-2"]
}
]
}
+727
View File
@@ -0,0 +1,727 @@
(() => {
const PATTERN_CHARS = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789";
const PATTERN_LENGTH = 900;
const SCRAMBLE_MS = 140;
const loadRecipes = (url) =>
fetch(url).then((response) => {
if (!response.ok) throw new Error(`HTTP ${response.status}`);
return response.json();
});
const generatePattern = (length) => {
const chars = new Array(length);
for (let index = 0; index < length; index += 1) {
chars[index] = PATTERN_CHARS.charAt((Math.random() * PATTERN_CHARS.length) | 0);
}
return chars.join("");
};
const motionQuery = () =>
window.matchMedia("(hover: hover) and (pointer: fine) and (prefers-reduced-motion: no-preference)");
const initEvervault = (root) => {
root.querySelectorAll("[data-evervault]").forEach((visual) => {
if (visual.dataset.evervaultReady) return;
visual.dataset.evervaultReady = "true";
const noise = visual.querySelector("[data-cookbook-pattern]");
if (!noise) return;
const query = motionQuery();
if (!query.matches) return;
let rect = null;
let pointerX = 0;
let pointerY = 0;
let frame = 0;
let lastScramble = 0;
let patternReady = false;
const paint = (scramble) => {
visual.style.setProperty("--mouse-x", `${pointerX}px`);
visual.style.setProperty("--mouse-y", `${pointerY}px`);
if (scramble) noise.textContent = generatePattern(PATTERN_LENGTH);
frame = 0;
};
const onEnter = () => {
rect = visual.getBoundingClientRect();
if (!patternReady) {
noise.textContent = generatePattern(PATTERN_LENGTH);
patternReady = true;
}
};
const onMove = (event) => {
if (!rect) rect = visual.getBoundingClientRect();
pointerX = event.clientX - rect.left;
pointerY = event.clientY - rect.top;
if (frame) return;
frame = window.requestAnimationFrame(() => {
const now = performance.now();
const scramble = now - lastScramble >= SCRAMBLE_MS;
if (scramble) lastScramble = now;
paint(scramble);
});
};
const onLeave = () => {
if (frame) window.cancelAnimationFrame(frame);
frame = 0;
rect = null;
pointerX = 0;
pointerY = 0;
visual.style.setProperty("--mouse-x", "50%");
visual.style.setProperty("--mouse-y", "50%");
};
visual.addEventListener("mouseenter", onEnter);
visual.addEventListener("mousemove", onMove);
visual.addEventListener("mouseleave", onLeave);
});
};
let familyPopstate = null;
const bindFamilyPopstate = () => {
if (bindFamilyPopstate.bound) return;
bindFamilyPopstate.bound = true;
window.addEventListener("popstate", () => {
if (typeof familyPopstate === "function") familyPopstate();
});
};
const groupIdFor = (recipe) => recipe.group || recipe.id;
const gpuCountLabel = (hardware, runtimeId) => {
const count = hardware?.gpu_count;
if (count == null) return "";
if (runtimeId === "spark") return `${count} Spark${count === 1 ? "" : "s"}`;
return `${count} GPU${count === 1 ? "" : "s"}`;
};
const knobsFor = (recipe) => recipe.knobs || [];
const knobOptions = (knob) =>
knob.options.map((option) =>
option !== null && typeof option === "object" ? option : { value: option, label: String(option) },
);
const knobDefaultLabel = (knob) => {
const match = knobOptions(knob).find((option) => option.value === knob.default);
return match ? match.label : String(knob.default);
};
// Knob flags are always shown explicitly in the displayed command, even at
// their default value, so the command stays copy-pasteable and precise
// about what it runs -- not just "trust the script's own default".
const appendKnobFlags = (commandText, knobs, knobValues) => {
const flags = knobs
.filter((knob) => knobValues[knob.key] !== undefined)
.map((knob) => `${knob.flag} ${knobValues[knob.key]}`);
if (!flags.length) return commandText;
const lines = commandText.split("\n");
lines[lines.length - 1] = `${lines[lines.length - 1]} ${flags.join(" ")}`;
return lines.join("\n");
};
const runtimeFor = (recipe) => {
const platform = recipe.hardware?.platform || "cuda";
if (platform === "mlx") {
return {
id: "mlx",
label: "Apple Silicon · MLX",
hint:
recipe.hardware?.minimum_memory ||
[recipe.hardware?.accelerator, recipe.hardware?.system_memory].filter(Boolean).join(" · ") ||
"Memory not recorded",
};
}
if (platform === "mps") {
return {
id: "mps",
label: "Apple Silicon · PyTorch MPS",
hint: recipe.hardware?.minimum_memory || recipe.hardware?.system_memory || "Memory not recorded",
};
}
const hardware = recipe.hardware || {};
if (hardware.device === "spark") {
return {
id: "spark",
label: "NVIDIA DGX Spark",
hint: "GB10 · 128 GB unified memory",
};
}
return {
id: "cuda",
label: "NVIDIA CUDA",
hint: hardware.accelerator
? `${hardware.accelerator} · ${gpuCountLabel(hardware, "cuda")}`
: `${gpuCountLabel(hardware, "cuda")} configured · GPU model not recorded`,
};
};
const runtimeSummary = (recipe) => {
const runtime = runtimeFor(recipe);
const hardware = recipe.hardware || {};
if (runtime.id === "mlx") {
return [hardware.accelerator || "Apple Silicon", hardware.system_memory || hardware.minimum_memory, "MLX"]
.filter(Boolean)
.join(" · ");
}
if (runtime.id === "mps") {
return [hardware.accelerator || "Apple Silicon", hardware.system_memory || hardware.minimum_memory, "PyTorch MPS"]
.filter(Boolean)
.join(" · ");
}
if (runtime.id === "spark") {
return [hardware.accelerator || "NVIDIA GB10", gpuCountLabel(hardware, "spark")].filter(Boolean).join(" · ");
}
if (hardware.accelerator) return [hardware.accelerator, gpuCountLabel(hardware, runtime.id)].join(" · ");
return `NVIDIA CUDA · ${gpuCountLabel(hardware, "cuda")} configured · GPU model and VRAM not recorded`;
};
const renderHardwareEvidence = (container, badge, recipe) => {
const hardware = recipe.hardware || {};
const runtime = runtimeFor(recipe);
const isValidated = hardware.evidence === "validated";
container.classList.toggle("cookbook-hardware-state--verified", isValidated);
container.classList.toggle("cookbook-hardware-state--source", !isValidated);
badge.classList.toggle("cookbook-badge--verified", isValidated);
badge.classList.toggle("cookbook-badge--configured", !isValidated);
badge.textContent = isValidated ? "Recorded run" : "Source config";
const heading = document.createElement("strong");
heading.textContent = isValidated ? "Recorded hardware" : "Source configuration";
const details = document.createElement("span");
if (isValidated) {
const recorded = [
hardware.accelerator,
runtime.id === "mlx" || runtime.id === "mps" ? hardware.system_memory : gpuCountLabel(hardware, runtime.id),
].filter(Boolean);
const statements = [`${recorded.join(" · ")}.`];
if (hardware.minimum_memory) statements.push(`Documented minimum: ${hardware.minimum_memory}.`);
if (hardware.peak_memory) statements.push(`Measured: ${hardware.peak_memory}.`);
if (!hardware.minimum_memory) statements.push("This recorded device is not a minimum requirement.");
details.textContent = ` ${statements.join(" ")}`;
} else if (runtime.id === "spark") {
details.textContent = ` NVIDIA DGX Spark · ${gpuCountLabel(hardware, "spark")}. GB10 has no FA4 / sm_100a VSA kernel; keep Triton VSA and FA4 off.`;
} else if (runtime.id === "cuda") {
details.textContent = ` NVIDIA CUDA · ${gpuCountLabel(hardware, "cuda")}. The source does not record the GPU model or VRAM.`;
} else {
details.textContent = ` ${runtime.label}. The source does not record a device or memory requirement.`;
}
container.replaceChildren(heading, details);
if (hardware.evidence_url) {
const evidenceLink = document.createElement("a");
evidenceLink.href = hardware.evidence_url;
evidenceLink.textContent = "View run evidence";
evidenceLink.setAttribute("aria-label", `View recorded hardware evidence for ${recipe.label}`);
container.append(" ", evidenceLink);
}
};
const compactLifecycle = (root) => {
const lifecycle = root.querySelector(".cookbook-lifecycle");
if (!lifecycle || lifecycle.dataset.compact) return;
lifecycle.dataset.compact = "true";
const stages = [...lifecycle.querySelectorAll(".cookbook-lifecycle__stage")];
const active = stages.find((stage) => stage.classList.contains("cookbook-lifecycle__stage--active"));
const planned = stages
.filter((stage) => stage !== active)
.map((stage) => stage.childNodes[0]?.textContent?.trim())
.filter(Boolean);
const summary = document.createElement("span");
summary.className = "cookbook-lifecycle__summary";
summary.textContent = `Next: ${planned.join(", ")}`;
lifecycle.replaceChildren(...(active ? [active] : []), summary);
};
const initFamilyBuilder = async (root) => {
const family = root.dataset.family;
if (!family) return;
compactLifecycle(root);
const modelOptions = root.querySelector("[data-cookbook-model-options]");
const hardwareOptions = root.querySelector("[data-cookbook-hardware-options]");
const description = root.querySelector("[data-cookbook-description]");
const label = root.querySelector("[data-cookbook-label]");
const model = root.querySelector("[data-cookbook-model]");
const task = root.querySelector("[data-cookbook-task]");
const hardwareValue = root.querySelector("[data-cookbook-gpus]");
const artifact = root.querySelector("[data-cookbook-artifact]");
const evidenceCell = root.querySelector("[data-cookbook-evidence]");
const source = root.querySelector("[data-cookbook-source]");
const modelLink = root.querySelector("[data-cookbook-model-link]");
const command = root.querySelector("[data-cookbook-command]");
const status = root.querySelector("[data-cookbook-status]");
const hardwareState = root.querySelector("[data-cookbook-hardware-state]");
const hardwareBadge = root.querySelector("[data-cookbook-hardware-badge]");
const count = root.querySelector("[data-cookbook-count]");
const result = root.querySelector(".cookbook-result");
const commandBlock = root.querySelector(".cookbook-command");
const servingPanel = root.querySelector("[data-cookbook-serving]");
const usage = root.querySelector("[data-cookbook-usage]");
const servingAvailability = root.querySelector("[data-cookbook-serving-availability]");
const knobsContainer = root.querySelector("[data-cookbook-knobs]");
const deviceRow = root.querySelector("[data-cookbook-device-row]");
const deviceOptions = root.querySelector("[data-cookbook-device-options]");
const deviceCaption = root.querySelector("[data-cookbook-device-caption]");
modelOptions.setAttribute("aria-label", "Recipe");
hardwareOptions.setAttribute("aria-label", "Runtime");
let recipes;
try {
({ recipes } = await loadRecipes(root.dataset.recipes));
} catch (error) {
if (status) status.textContent = "Recipes could not be loaded. Use the maintained examples link below.";
console.error("Failed to load FastVideo cookbook recipes", error);
return;
}
if (!root.isConnected) return;
const familyRecipes = recipes.filter((recipe) => recipe.family === family);
if (!familyRecipes.length) return;
let servingProfiles = {};
let servingLoadFailed = false;
if (servingPanel && familyRecipes.some((recipe) => recipe.serving)) {
try {
const dataUrl = new URL(root.dataset.recipes, document.baseURI);
servingProfiles = await loadRecipes(new URL("cookbook-serving.json", dataUrl));
} catch (error) {
servingLoadFailed = true;
console.error("Failed to load FastVideo serving profiles", error);
}
}
const byId = new Map(familyRecipes.map((recipe) => [recipe.id, recipe]));
const groups = new Map();
familyRecipes.forEach((recipe) => {
const groupId = groupIdFor(recipe);
if (!groups.has(groupId)) groups.set(groupId, []);
groups.get(groupId).push(recipe);
});
if (count) count.textContent = `${familyRecipes.length} maintained recipes`;
modelOptions.replaceChildren();
groups.forEach((groupRecipes, groupId) => {
const representative = groupRecipes[0];
const option = document.createElement("button");
option.type = "button";
option.dataset.recipeGroup = groupId;
option.setAttribute("aria-pressed", "false");
const optionLabel = document.createElement("strong");
optionLabel.textContent = representative.group_label || representative.label;
const optionTask = document.createElement("span");
optionTask.textContent = representative.group_task || representative.task;
option.append(optionLabel, optionTask);
modelOptions.append(option);
});
const query = new URLSearchParams(window.location.search);
const requestedRecipe = query.get("recipe");
const defaultRecipeId = byId.has(root.dataset.defaultRecipe) ? root.dataset.defaultRecipe : familyRecipes[0].id;
let selectedRecipeId = requestedRecipe && byId.has(requestedRecipe) ? requestedRecipe : defaultRecipeId;
let selectedGroupId = groupIdFor(byId.get(selectedRecipeId));
let renderedRuntimeGroup = null;
// Keep previously shared local/openai links working after renaming workflows.
const workflow = (value) => ["local", "python"].includes(value) ? "python" : "server";
let usagePreference = workflow(query.get("use"));
let selectedClient = ["python", "javascript", "curl"].includes(query.get("client")) ? query.get("client") : "curl";
const clientDetails = servingPanel?.querySelector(".cookbook-serving__code");
if (clientDetails && query.has("client")) clientDetails.open = true;
const knobDefs = new Map();
familyRecipes.forEach((recipe) => knobsFor(recipe).forEach((knob) => {
if (!knobDefs.has(knob.key)) knobDefs.set(knob.key, knob);
}));
const knobValues = {};
knobDefs.forEach((knob, key) => {
const fromQuery = query.get(key);
const validValues = knobOptions(knob).map((option) => String(option.value));
const useQueryValue = fromQuery !== null && validValues.includes(fromQuery);
const raw = useQueryValue ? fromQuery : knob.default;
knobValues[key] = typeof knob.default === "number" ? Number(raw) : raw;
});
const renderKnobs = (recipe, hidden) => {
if (!knobsContainer) return;
const knobs = knobsFor(recipe);
const renderedKeys = [...knobsContainer.querySelectorAll("[data-knob-row]")].map((row) => row.dataset.knobRow);
if (renderedKeys.join(",") !== knobs.map((knob) => knob.key).join(",")) {
knobsContainer.replaceChildren();
knobs.forEach((knob) => {
const row = document.createElement("div");
row.className = "cookbook-selection-row";
row.dataset.knobRow = knob.key;
const labelWrap = document.createElement("div");
labelWrap.className = "cookbook-selection-row__label";
const strongLabel = document.createElement("strong");
strongLabel.textContent = knob.label;
const hintLabel = document.createElement("span");
hintLabel.textContent = knob.hint || "";
labelWrap.append(strongLabel, hintLabel);
const grid = document.createElement("div");
grid.className = "cookbook-option-grid cookbook-option-grid--hardware";
grid.setAttribute("role", "group");
grid.setAttribute("aria-label", knob.label);
knobOptions(knob).forEach((option) => {
const optionButton = document.createElement("button");
optionButton.type = "button";
optionButton.dataset.knobKey = knob.key;
optionButton.dataset.knobValue = String(option.value);
optionButton.setAttribute("aria-pressed", "false");
const optionLabel = document.createElement("strong");
optionLabel.textContent = option.label;
optionButton.append(optionLabel);
grid.append(optionButton);
});
row.append(labelWrap, grid);
knobsContainer.append(row);
});
}
knobsContainer.hidden = hidden || knobs.length === 0;
knobsContainer.querySelectorAll("button[data-knob-key]").forEach((optionButton) => {
const selected = String(knobValues[optionButton.dataset.knobKey]) === optionButton.dataset.knobValue;
optionButton.classList.toggle("cookbook-option--selected", selected);
optionButton.setAttribute("aria-pressed", String(selected));
});
};
const recipesForRuntime = (runtimeId, groupRecipes) =>
groupRecipes.filter((item) => runtimeFor(item).id === runtimeId);
const uniqueRuntimeIds = (groupRecipes) => {
const ids = [];
groupRecipes.forEach((item) => {
const id = runtimeFor(item).id;
if (!ids.includes(id)) ids.push(id);
});
return ids;
};
const pickRecipeForRuntime = (runtimeId, preferredCount, groupRecipes) => {
const siblings = recipesForRuntime(runtimeId, groupRecipes);
if (!siblings.length) return null;
if (preferredCount != null) {
const match = siblings.find((item) => item.hardware?.gpu_count === preferredCount);
if (match) return match;
}
return siblings.find((item) => item.hardware?.gpu_count === 1) || siblings[0];
};
const renderRuntimeOptions = () => {
const groupRecipes = groups.get(selectedGroupId) || [];
const runtimeIds = uniqueRuntimeIds(groupRecipes);
const renderedIds = [...hardwareOptions.querySelectorAll("[data-runtime-id]")].map((option) => option.dataset.runtimeId);
if (renderedIds.join(",") === runtimeIds.join(",")) return;
hardwareOptions.replaceChildren();
runtimeIds.forEach((runtimeId) => {
const representative = pickRecipeForRuntime(runtimeId, 1, groupRecipes);
const runtime = runtimeFor(representative);
const option = document.createElement("button");
option.type = "button";
option.dataset.runtimeId = runtime.id;
option.setAttribute("aria-pressed", "false");
const optionLabel = document.createElement("strong");
optionLabel.textContent = runtime.label;
const optionHint = document.createElement("span");
optionHint.textContent = runtime.hint;
option.append(optionLabel, optionHint);
hardwareOptions.append(option);
});
};
const renderDeviceOptions = (recipe) => {
if (!deviceRow || !deviceOptions) return;
const groupRecipes = groups.get(selectedGroupId) || [];
const runtime = runtimeFor(recipe);
const siblings = recipesForRuntime(runtime.id, groupRecipes)
.slice()
.sort((left, right) => (left.hardware?.gpu_count || 0) - (right.hardware?.gpu_count || 0));
const show = siblings.length > 1;
deviceRow.hidden = !show;
if (deviceCaption) {
deviceCaption.textContent = runtime.id === "spark" ? "1 Spark or a QSFP pair" : "GPU count for this runtime";
}
if (!show) {
deviceOptions.replaceChildren();
return;
}
const renderedIds = [...deviceOptions.querySelectorAll("[data-recipe-id]")].map((option) => option.dataset.recipeId);
if (renderedIds.join(",") !== siblings.map((item) => item.id).join(",")) {
deviceOptions.replaceChildren();
siblings.forEach((candidate) => {
const option = document.createElement("button");
option.type = "button";
option.dataset.recipeId = candidate.id;
option.setAttribute("aria-pressed", "false");
const optionLabel = document.createElement("strong");
optionLabel.textContent = gpuCountLabel(candidate.hardware, runtime.id);
const optionHint = document.createElement("span");
optionHint.textContent = runtime.id === "spark" && candidate.hardware?.gpu_count === 2
? "Ray sequence parallel over QSFP"
: runtime.id === "spark"
? "One GB10, local process"
: `${gpuCountLabel(candidate.hardware, runtime.id)} configured`;
option.append(optionLabel, optionHint);
deviceOptions.append(option);
});
}
deviceOptions.querySelectorAll("button").forEach((option) => {
const selected = option.dataset.recipeId === recipe.id;
option.classList.toggle("cookbook-option--selected", selected);
option.setAttribute("aria-pressed", String(selected));
});
};
let notes = root.querySelector("[data-cookbook-notes]");
if (!notes) {
notes = document.createElement("aside");
notes.className = "cookbook-recipe-notes";
notes.dataset.cookbookNotes = "";
notes.hidden = true;
result.insertBefore(notes, commandBlock);
}
const render = ({ groupChanged = false, historyMode = "replace" } = {}) => {
if (!root.isConnected) return;
if (!byId.has(selectedRecipeId)) selectedRecipeId = defaultRecipeId;
let recipe = byId.get(selectedRecipeId);
if (groupChanged || groupIdFor(recipe) !== selectedGroupId) {
const currentRuntime = runtimeFor(recipe).id;
const currentCount = recipe.hardware?.gpu_count;
const groupRecipes = groups.get(selectedGroupId) || [];
recipe = pickRecipeForRuntime(currentRuntime, currentCount, groupRecipes) || groupRecipes[0];
selectedRecipeId = recipe.id;
}
selectedGroupId = groupIdFor(recipe);
if (renderedRuntimeGroup !== selectedGroupId) {
renderRuntimeOptions();
renderedRuntimeGroup = selectedGroupId;
}
renderDeviceOptions(recipe);
const runtime = runtimeFor(recipe);
const profile = servingPanel && servingProfiles[recipe.id];
const useServer = Boolean(profile && usagePreference === "server");
// The measured local profile and the server config have separate evidence.
const activeRecipe = useServer ? { ...recipe, hardware: profile.hardware, evidence: "Source-backed" } : recipe;
const knobs = knobsFor(recipe);
renderKnobs(recipe, useServer);
if (usage) {
usage.querySelectorAll("[data-cookbook-mode]").forEach((option) => {
const selected = option.dataset.cookbookMode === (useServer ? "server" : "python");
option.disabled = option.dataset.cookbookMode === "server" && !profile;
option.classList.toggle("cookbook-option--selected", selected);
option.setAttribute("aria-pressed", String(selected));
});
servingAvailability.textContent = profile
? "The playground and the OpenAI Python client share one server process. Both workflows can run on your own machine."
: servingLoadFailed
? "Server examples could not be loaded. Open the H3 server guide below, or use Python directly."
: "This recipe uses Python directly. FastH3 V1 and FastH3 V2 can also run a local server for the playground and the OpenAI Python client.";
servingPanel.hidden = !useServer;
commandBlock.hidden = useServer;
root.querySelector("[data-cookbook-python-note]").hidden = useServer;
}
if (useServer) {
const isMLX = profile.runtime === "mlx";
const isSpark = runtime.id === "spark";
servingPanel.querySelector("[data-cookbook-server-lifetime]").textContent = isMLX
? "Start once, then change prompts in the playground or your app. MLX reuses its pipeline and prompt cache, but loads and releases model components between phases to limit unified-memory use. It does not keep all weights resident."
: isSpark
? "Start once, then change prompts in the playground or your app. On a DGX Spark, lazy module load still reloads Qwen3-VL and the DiT between phases of each request, so later prompts are not a free hot cache."
: "Start once, then change prompts in the playground or your app. CUDA requests reuse the loaded model. The Python SDK can also reuse a generator within one process.";
servingPanel.querySelector("[data-cookbook-install-guide]").href = isMLX
? "../../getting_started/installation/mlx/"
: isSpark
? "../../getting_started/installation/spark/"
: "../../getting_started/installation/gpu/";
servingPanel.querySelector("[data-cookbook-prepare]").hidden = !profile.prepare;
servingPanel.querySelector("[data-cookbook-server-prepare]").textContent = profile.prepare;
servingPanel.querySelector("[data-cookbook-server-install]").textContent = profile.install;
servingPanel.querySelector("[data-cookbook-server-command]").textContent = profile.command;
servingPanel.querySelector("[data-cookbook-health-command]").textContent = profile.health_command;
servingPanel.querySelector("[data-cookbook-playground]").href = profile.playground_url;
const client = profile.clients[selectedClient];
const filename = client.source.split("/").pop();
servingPanel.querySelector("[data-cookbook-client-install]").textContent = client.install;
const clientCode = servingPanel.querySelector("[data-cookbook-client-code]");
clientCode.className = `language-${selectedClient === "curl" ? "bash" : selectedClient}`;
clientCode.textContent = client.code;
servingPanel.querySelector("[data-cookbook-client-filename]").textContent = filename;
servingPanel.querySelector("[data-cookbook-client-source]").href = `https://github.com/hao-ai-lab/FastVideo/blob/main/${client.source}`;
const runner = { python: "python", javascript: "node", curl: "bash" }[selectedClient];
servingPanel.querySelector("[data-cookbook-client-run]").textContent = `Save as ${filename} and run ${runner} ${filename}. The MP4 is saved with the job ID as its filename.`;
servingPanel.querySelectorAll("[data-cookbook-client]").forEach((option) => {
option.setAttribute("aria-pressed", String(option.dataset.cookbookClient === selectedClient));
});
}
modelOptions.querySelectorAll("button").forEach((option) => {
const selected = option.dataset.recipeGroup === selectedGroupId;
option.classList.toggle("cookbook-option--selected", selected);
option.setAttribute("aria-pressed", String(selected));
});
hardwareOptions.querySelectorAll("button").forEach((option) => {
const selected = option.dataset.runtimeId === runtime.id;
option.classList.toggle("cookbook-option--selected", selected);
option.setAttribute("aria-pressed", String(selected));
});
description.textContent = useServer
? `${recipe.group_label || recipe.label} generates video with audio. Start the local server, then use the playground or the OpenAI Python client. This profile uses the checked-in ${runtime.label} configuration.`
: recipe.summary;
label.textContent = useServer ? `${recipe.group_label || recipe.label} · Server` : recipe.label;
model.textContent = recipe.model;
task.textContent = recipe.task;
hardwareValue.textContent = runtimeSummary(activeRecipe);
if (artifact) artifact.textContent = useServer ? "MP4 with audio" : recipe.expected_artifact || "Not yet documented for this recipe.";
if (evidenceCell) {
evidenceCell.textContent = activeRecipe.evidence || "Source-backed";
evidenceCell.classList.toggle("cookbook-badge--verified", activeRecipe.evidence === "Verified");
evidenceCell.classList.toggle("cookbook-badge--source-backed", activeRecipe.evidence !== "Verified");
}
source.href = `https://github.com/hao-ai-lab/FastVideo/blob/main/${useServer ? profile.source : recipe.source}`;
source.textContent = useServer ? "View server configuration" : "Open example source";
modelLink.href = `https://huggingface.co/${recipe.model}`;
command.textContent = useServer ? recipe.command : appendKnobFlags(recipe.command, knobs, knobValues);
renderHardwareEvidence(hardwareState, hardwareBadge, activeRecipe);
const knobCaveats = useServer ? [] : knobs
.filter((knob) => String(knobValues[knob.key]) !== String(knob.default))
.map((knob) => `${knob.label} is set away from its recorded default (${knobDefaultLabel(knob)}). ` +
"The script accepts this value, but it has not been benchmarked here.");
const limitations = [...(useServer ? [
profile.runtime === "mlx"
? "This MLX server config has no recorded hardware run. Measurements from the Python recipe are not server memory requirements. Only text-to-video/audio is wired; reference inputs and fast modes are not exposed here."
: runtime.id === "spark"
? "This Spark server config has no recorded serving benchmark. Lazy module load reloads Qwen3-VL and the DiT between phases of each request. Compilation of the DiT is disabled."
: "This server config has no recorded serving benchmark. Compilation is disabled, unlike the measured Python performance profile.",
`${profile.sampling.width} × ${profile.sampling.height} · ${profile.sampling.num_frames} frames · ${profile.sampling.fps} fps. The server supplies these defaults; the client sends the model and prompt.`,
"Generation is serialized. Job metadata is held in memory and is lost when the server restarts.",
] : recipe.limitations || []), ...knobCaveats];
notes.replaceChildren();
notes.hidden = limitations.length === 0;
if (limitations.length) {
const notesHeading = document.createElement("strong");
notesHeading.textContent = "Know before you run";
const notesList = document.createElement("ul");
limitations.forEach((item) => {
const listItem = document.createElement("li");
listItem.textContent = item;
notesList.append(listItem);
});
notes.append(notesHeading, notesList);
}
const nextQuery = new URLSearchParams(window.location.search);
nextQuery.set("recipe", recipe.id);
nextQuery.set("runtime", runtime.id);
const deviceSiblings = recipesForRuntime(runtime.id, groups.get(selectedGroupId) || []);
if (deviceSiblings.length > 1 && recipe.hardware?.gpu_count != null) {
nextQuery.set("gpus", String(recipe.hardware.gpu_count));
} else {
nextQuery.delete("gpus");
}
knobDefs.forEach((knob, key) => {
if (knobs.some((activeKnob) => activeKnob.key === key)) nextQuery.set(key, String(knobValues[key]));
else nextQuery.delete(key);
});
if (usage) {
nextQuery.set("use", useServer ? "server" : "python");
if (useServer && clientDetails?.open) nextQuery.set("client", selectedClient);
else nextQuery.delete("client");
}
const nextUrl = `${window.location.pathname}?${nextQuery.toString()}${window.location.hash}`;
if (historyMode === "push") window.history.pushState({}, "", nextUrl);
else if (historyMode === "replace") window.history.replaceState({}, "", nextUrl);
const modeSummary = useServer ? " with a persistent server" : "";
status.textContent = `${recipe.label} selected for ${runtime.label}${modeSummary}.`;
};
modelOptions.addEventListener("click", (event) => {
const option = event.target.closest("button[data-recipe-group]");
if (!option) return;
selectedGroupId = option.dataset.recipeGroup;
render({ groupChanged: true, historyMode: "push" });
});
hardwareOptions.addEventListener("click", (event) => {
const option = event.target.closest("button[data-runtime-id]");
if (!option) return;
const groupRecipes = groups.get(selectedGroupId) || [];
const currentCount = byId.get(selectedRecipeId)?.hardware?.gpu_count;
const nextRecipe = pickRecipeForRuntime(option.dataset.runtimeId, currentCount, groupRecipes);
if (!nextRecipe) return;
selectedRecipeId = nextRecipe.id;
selectedGroupId = groupIdFor(nextRecipe);
render({ historyMode: "push" });
});
deviceOptions?.addEventListener("click", (event) => {
const option = event.target.closest("button[data-recipe-id]");
if (!option) return;
selectedRecipeId = option.dataset.recipeId;
selectedGroupId = groupIdFor(byId.get(selectedRecipeId));
render({ historyMode: "push" });
});
usage?.addEventListener("click", (event) => {
const option = event.target.closest("button[data-cookbook-mode]");
if (!option || option.disabled) return;
usagePreference = option.dataset.cookbookMode;
render({ historyMode: "push" });
});
servingPanel?.addEventListener("click", (event) => {
const option = event.target.closest("button[data-cookbook-client]");
if (!option) return;
selectedClient = option.dataset.cookbookClient;
render({ historyMode: "push" });
});
knobsContainer?.addEventListener("click", (event) => {
const option = event.target.closest("button[data-knob-key]");
if (!option) return;
const knob = knobDefs.get(option.dataset.knobKey);
knobValues[option.dataset.knobKey] = typeof knob.default === "number"
? Number(option.dataset.knobValue) : option.dataset.knobValue;
render({ historyMode: "push" });
});
render();
familyPopstate = () => {
if (!root.isConnected) return;
const nextQuery = new URLSearchParams(window.location.search);
const nextRecipe = nextQuery.get("recipe");
selectedRecipeId = nextRecipe && byId.has(nextRecipe) ? nextRecipe : defaultRecipeId;
selectedGroupId = groupIdFor(byId.get(selectedRecipeId));
usagePreference = workflow(nextQuery.get("use"));
selectedClient = ["python", "javascript", "curl"].includes(nextQuery.get("client")) ? nextQuery.get("client") : "curl";
if (clientDetails) clientDetails.open = nextQuery.has("client");
knobDefs.forEach((knob, key) => {
const fromQuery = nextQuery.get(key);
const validValues = knobOptions(knob).map((option) => String(option.value));
if (fromQuery !== null && validValues.includes(fromQuery)) {
knobValues[key] = typeof knob.default === "number" ? Number(fromQuery) : fromQuery;
}
});
render({ historyMode: "none" });
};
bindFamilyPopstate();
};
const init = () => {
initEvervault(document);
document.querySelectorAll("[data-cookbook][data-family]").forEach((root) => {
if (root.dataset.initialized) return;
root.dataset.initialized = "true";
initFamilyBuilder(root);
});
};
if (window.document$) window.document$.subscribe(init);
else document.addEventListener("DOMContentLoaded", init);
})();

Some files were not shown because too many files have changed in this diff Show More