Compare commits

..
Author SHA1 Message Date
Satyam Srivastava b3a9874fc8 Merge branch 'main' into test-hf-sync 2026-05-01 11:38:11 -07:00
Satyam Srivastava d657cbbf17 Update regression gatekeep to 5% 2026-05-01 11:05:33 -07:00
William Lin 9801037c3d [refactor] tests/local_tests: organize by model family (#1269) 2026-05-01 01:49:54 -07:00
alexzmsandmergify[bot] 74d09b0efd [misc] cleanup: grad-norm asserts, dead offload file, callback names (#1268)
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-05-01 01:16:13 -07:00
alexzms 38dc8820ac [ci] add CPU unit tests for train callback system in fastvideo.train (#1267) 2026-05-01 00:53:29 -07:00
William Lin c77a76c6af [feat] Stable Audio Open 1.0: T2A + A2A + RePaint inpainting (native) (#1260) 2026-05-01 00:07:11 -07:00
alexzms d14d5aadea [feat] Cosmos 2.5 training support in fastvideo.train (#1224) 2026-05-01 01:15:02 +00:00
alexzms 4c915b7742 [ci] add CPU unit tests for train checkpoint utilities in fastvideo.train (#1265) 2026-04-29 18:55:39 +00:00
Satyam Srivastava 48534ef4de [ci] Use median instead of mean to detect regressions. 2026-04-28 01:19:19 -07:00
Satyam Srivastava 1116f514be Refactor upload performance metrics 2026-04-28 01:07:27 -07:00
Satyam Srivastava d451e61749 Abstract hf code and fix plot bug 2026-04-27 23:12:42 -07:00
alexzms 9a8bbe18fa [bugfix]: fix SP deadlock in negative prompt encoding during training (#1178) 2026-04-28 01:06:49 +00:00
alexzms ea25441ef0 [ci] add CPU unit tests for fastvideo.train load_run_config (#1264) 2026-04-28 01:06:18 +00:00
48957fcde1 [bugfix] Fix modal remote functions crash container on sys exit in CI remote functions (#1261)
Co-authored-by: Satyam Srivastava <satyam53@Mac.lan1>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-04-27 21:50:55 +00:00
Mook 7b872cc41e [Perf] Skip bool-mask round-trip in block-sparse VSA attention (#1243) 2026-04-26 15:14:37 -07:00
Satyam Srivastava 3ff4a8d2d2 [ci] Refactor pr_test.sh post run hooks 2026-04-26 12:26:24 -07:00
Satyam Srivastava 9343d4cdf4 [bug] Fix modal remote functions crash container on sys.exit(0) during PR tests 2026-04-26 11:18:21 -07:00
Satyam Srivastava 66fb3d1e79 [ci] Fix buildkite agent annotate and upload 2026-04-26 02:36:45 -07:00
alexzms 37418946c8 [docs]: clarify real_score_guidance_scale CFG parameterization (#1256) 2026-04-26 16:38:00 +08:00
William Lin 95fd29e0cb [feat] Streaming WebSocket server skeleton (single generator + fMP4) (#1251) 2026-04-26 00:33:49 -07:00
Satyam Srivastava aca850cef2 [ci] Fix plotly bug 2026-04-25 20:17:49 -07:00
Satyam Srivastava 1c79779956 Debug changes 2026-04-25 18:15:09 -07:00
Satyam Srivastava eee03527ed [ci] Add plot render for performance tracking 2026-04-25 17:39:41 -07:00
Satyam Srivastava 1eb8541094 Test with different run configs 2026-04-25 15:53:36 -07:00
Junda Suandmergify[bot] e17cd2633c [bugfix]: normalize uint8 pil_image in I2V VAE encoding (#1249)
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-04-24 09:16:01 +00:00
William Lin e0dc5f2b0c [feat] Add typed LTX-2 continuation state and streaming session store (#1250) 2026-04-24 01:28:07 -07:00
Satyam Srivastava 69c214d13a Test: HF performance sync logic 2026-04-23 16:06:04 -07:00
Satyam Srivastava 0341481aa7 [ci] Upload Perf. Regression Results to HF_Repo 2026-04-23 15:51:26 -07:00
William Lin 70ee5d230c [feat] [6/n] Improve API: LTX-2 public preset + asset wiring + gpu_pool translation (#1239) 2026-04-23 11:36:45 -07:00
Satyam Srivastava d1c3fdd187 [ci] Perf CI Run Schedule
Make Full Performance CI run only on main branch in X days and not on every PR.
This schedule will be created in the Buildkite.
2026-04-22 21:12:54 -07:00
Satyam Srivastava 980e8d933e Add CI Performance Regression Tracking Changes 2026-04-22 18:11:13 -07:00
William Lin 24ced500f5 [test] add LTX-2 distilled T2V SSIM regression test (#1240) 2026-04-21 12:03:38 -07:00
William Lin 4ddcdf541f [feat] [5.5/n] Improve API: streaming server config surface + serve dispatch (#1238) 2026-04-17 15:36:21 -07:00
William Lin 0e3529869c [feat] [5/n] Improve API: wire ServeConfig.default_request into OpenAI serving (#1237) 2026-04-17 13:26:18 -07:00
William Lin e1e0d91c00 [misc] small cleanup for API handling (#1235) 2026-04-16 16:21:21 -07:00
William Lin 145a3f166b [feat] [4/n] Improve API: refactor sampling param and merge with presets (#1234) 2026-04-16 14:10:02 -07:00
William Lin 88a5a933ab [feat] [3/n] Improve API: extend support to cli (#1226) 2026-04-14 15:20:47 -07:00
William Lin c591d6d2a6 [feat] [2/n] Improve API: add initial support in video_generator (#1220) 2026-04-06 10:33:54 -07:00
Kun Linandmergify[bot] 65dff806a8 [bugfix]Fixing Lora distillation training distributed checkpointing bug (#1192)
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-04-06 02:20:26 +00:00
KUAN-HAO HUANGandmergify[bot] b85f0f4c2a [perf]: Eliminate CPU-GPU synchronization bottlenecks in training pipeline (#1217)
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-04-06 02:03:46 +00:00
William Lin 76c62d7a00 [feat] [1/n] API improvements: add intial files for new fastvideo public API (#1218) 2026-04-05 18:13:19 -07:00
f6e65ff668 [Feature] Add BSA (Bidirectional Sparse Attention) inference backend (#1174)
Co-authored-by: Satyam Srivastava <satyam53@Mac.lan1>
Co-authored-by: Satyam Srivastava <satyam53@Satyams-MacBook-Air.local>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-04-05 05:00:33 +00:00
mergify[bot] c220aa8000 [ci](mergify): upgrade configuration to current format (#1216)
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-04-04 23:09:17 +00:00
Jinzhe PanandDarren Sadr 4713fc17ed [feat] Job Runner UI (#1189)
Co-authored-by: Darren Sadr <darrensadr@gmail.com>
2026-04-02 16:07:24 -07:00
vishruthb 5789955bbe [feat] add gen3c (cosmos-7b) model and pipeline support (#1059) 2026-04-01 11:42:02 +00:00
Jinzhe Pan 2ad84a3b78 [ci] Use update instead of rebase for auto branch sync (#1215) 2026-04-01 19:16:59 +08:00
Jinzhe Pan 12d699cd78 [ci] Add direct test retry with check overwrite and aggregate status refresh (#1214) 2026-04-01 17:21:28 +08:00
Jinzhe Pan 34f14ded21 [ci] Use pull_request_target for Full Suite trigger (#1213) 2026-04-01 03:01:07 +08:00
Jinzhe Pan 71d1ab411f [ci] Fix jq crash when Buildkite build env is null (#1212) 2026-04-01 02:35:01 +08:00
Jinzhe Pan 805e487773 [ci] Ignore legacy reference videos when checking for HF download (#1211) 2026-04-01 02:12:09 +08:00
Jinzhe Pan 8803b4547e [ci] Add retry for flaky tests and fix stale SSIM references (#1210) 2026-04-01 01:11:49 +08:00
Jinzhe Pan 3b3806b3f6 [ci] Fix /merge to directly trigger Full Suite + simplify rebase conditions (#1209) 2026-03-31 23:17:09 +08:00
Jinzhe Pan 38d962e89d [ci] Remove Mergify ready-label race condition (#1208) 2026-03-31 20:59:13 +08:00
Jinzhe Pan 3966a365d0 [ci] Add statuses:write permission for /test pre-commit (#1207) 2026-03-31 20:33:18 +08:00
Jinzhe Pan d73fd14af0 [ci] Post pre-commit status to PR commit SHA (#1206) 2026-03-31 20:21:21 +08:00
Jinzhe Pan a87cc89916 [ci] Trigger pre-commit on /test slash commands (#1205) 2026-03-31 20:12:57 +08:00
Jinzhe Pan 81fd80c8ee [ci] Add TEST_SCOPE routing for clean single-test execution (#1203) 2026-03-31 19:40:59 +08:00
Jinzhe Pan ff22439f28 [ci] Fix fork PR checkout for /test and Full Suite triggers (#1202) 2026-03-31 13:42:13 +08:00
Jinzhe Pan de0de04212 [ci] Replace Merge Queue with auto-merge — reduce CI complexity (#1200) 2026-03-31 10:09:33 +08:00
mergify[bot] 7f2c3e1f64 [ci](mergify): upgrade configuration to current format (#1194)
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-03-31 00:47:48 +08:00
Jinzhe Pan ab55e57c22 [ci] Fix Merge Queue requeue and draft PR pre-commit skip (#1197) 2026-03-30 22:35:13 +08:00
Jinzhe Pan 46f6b43a53 [ci] Fix Merge Queue immediate dequeue (#1196) 2026-03-30 21:33:45 +08:00
Jinzhe Pan 833a33b663 [ci] CI follow-up: gate checks, issue label unification, draft PR skip (#1193) 2026-03-30 20:19:46 +08:00
Jinzhe Pan 9ea1307cd4 [ci] Add approval and pre-commit checks to merge protections (#1190)
## Summary

Follow-up to #1187. Two small changes:

1. **Merge Protections expanded** — adds `#approved-reviews-by>=1` and `check-success~=pre-commit` to `merge_protections` so the Mergify check shows a unified requirements checklist on every PR (title format + approval + pre-commit), instead of only showing the title format.

2. **Buildkite pipeline comment fix** — updates the outdated Full Suite section comment from "Triggered by adding the 'ready' label via GitHub Actions → Buildkite API" to reflect the new Merge Queue trigger path.
2026-03-30 05:14:49 +00:00
Jinzhe Pan be35003cb1 [ci] Merge Queue, label system overhaul, and slash commands (2/2) (#1187) 2026-03-30 08:22:35 +08:00
Jinzhe Pan 26bd4db253 [ci] CI infrastructure cleanup and workflow reorganization (1/2) (#1186) 2026-03-29 17:01:09 -07:00
Jinzhe PanandWill Lin e294ca011c [feat]: overhaul SSIM test infrastructure — partition scheduling, helper migration, CI fixes (#1185)
Co-authored-by: Will Lin <wlsaidhi@gmail.com>
2026-03-29 23:59:25 +00:00
alexzms 2085a4fc4a [bugfix]: fix VAE temporal tiling blend corruption in tiled_encode (#1181) 2026-03-29 23:12:42 +00:00
Jinzhe Pan c0c8e39c04 Revert "[feat] Job Runner UI" (#1188) 2026-03-29 16:46:27 +08:00
Darren f72618dafb [feat] Job Runner UI (#1172) 2026-03-29 16:17:56 +08:00
alexzms b3edfacdd8 [bugfix]: fix I2V preprocessing crash for models without CLIP (Wan2.2 I2V) (#1184) 2026-03-28 10:28:50 +08:00
alexzms 30129a3350 [misc]: reorganize training configs and add documentation (#1177) 2026-03-26 16:38:31 -07:00
jaisurya27 4d49f7b0aa Kandinsky5 lite dit clean (#1088) 2026-03-26 08:07:11 +08:00
alexzms 71bfc13d75 [feat]: add HunyuanVideo model plugin for fastvideo/train framework (#1175) 2026-03-24 16:49:51 -07:00
Kaiqin Kong 74db6e18d1 [misc] update action loading in validation and preprocess (#1143) 2026-03-24 15:10:03 -07:00
Kaiqin Kong 7d263c6a36 [bugfix] self-forcing train/validation step mismatch (#1173) 2026-03-20 00:52:45 -07:00
Zhang Peiyuan 454c32d1d1 Update README.md 2026-03-17 14:26:07 -07:00
Jinzhe Pan d1240b9238 [CI] add contributor interaction automation (#1170) 2026-03-17 12:05:40 +08:00
Hao Zhang 4105094fa5 [docs] Update README with realtime demo announcement (#1169) 2026-03-13 15:52:02 -07:00
alexzms f036469d3d [feat]: Knowledge Distillation training method for ODE-init (KDMethod + KDCausalMethod) (#1166) 2026-03-11 20:43:24 -07:00
alexzms 14261bc98c [feat] pre-commit support 120 col num (#1167) 2026-03-11 20:19:30 -07:00
alexzms d92858659d [feat] Self-Forcing methods in refactored training infra (#1164) 2026-03-09 20:20:59 -07:00
1a383f3f66 [refactor] train v1 clean up
Co-authored-by: alexzms <3036648523@qq.com>
Co-authored-by: Peiyuan Zhang <a1286225768@slurm-h200-204-215.slurm-compute.tenant-slurm.svc.cluster.local>
2026-03-09 18:58:56 -07:00
alexzmsandPeiyuan Zhang bc27a032c5 [feat] Refactor training framework into fastvideo/train (#1159)
Co-authored-by: Peiyuan Zhang <a1286225768@gmail.com>
2026-03-09 15:16:42 -07:00
alexzms 2b13e117f0 [Feat] Add causal Wan pipeline with multi-step denoising (#1161) 2026-03-08 13:32:04 -07:00
Junda Chen 99c166c381 feat: Building agent friendly repo (#1151) 2026-03-07 17:46:29 -08:00
XOR-op 95066245db [misc] FlashAttention 4 support (#1114) 2026-03-07 16:53:43 -08:00
Jinzhe Panandgemini-code-assist[bot] 6dcaac768b [CI] PR template (#1157)
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2026-03-07 11:51:28 -08:00
Ajay Anubolu 02c1c49b75 [CI] Add inference performance regression tests (#1140) 2026-03-07 08:26:54 +08:00
Zhang Peiyuan cd1b7cf139 [Refactor] SP Mask --> original seq len; HunyuanVideo 1.5 does not need mask (#1142) 2026-03-04 11:44:47 +08:00
Ajay Anubolu e63b7d8ac4 [Feat] Added OpenAI-compatible API server and benchmark script (#1109) 2026-03-02 17:12:32 -05:00
Jinzhe Pan 5190c1bb1e [Doc] add doc for inference architecture (#1147) 2026-03-02 13:44:27 -08:00
Darren 2cb3bba658 [bugfix]: fix a bug where collect_env was not running properly... (#1145) 2026-03-02 10:57:29 -08:00
Jinzhe Pan f9e1c46c3c [CI][Feat] launch 2 instance to run ssim (#1137) 2026-03-01 01:49:29 -08:00
Peiyuan Zhang e1eda47589 remove temporal frame adjustment 2026-02-27 20:47:16 +00:00
Zhang Peiyuan d902967208 Py/fix sp (#1138) 2026-02-27 12:14:44 +08:00
Zhang Peiyuan fea556269b [Misc] Fix memory leakage in VideoGenerator (#1132) 2026-02-26 19:51:32 -08:00
William Lin 69dd3c68f6 [bugfix] fix matrix game kv indexing and CI (#1135) 2026-02-26 01:24:42 -08:00
Jinzhe Pan 5433f6e80b [fix] preprocessing issue (#1134) 2026-02-25 21:49:52 -08:00
Junda (David) Su e315657066 [docs] [kernel] Migrate to uv (#1127) 2026-02-25 14:13:50 -08:00
Zhang Peiyuan f8d9a0c57f [misc] fix hunyuan (#1125) 2026-02-25 08:26:29 +08:00
Jinzhe PanandWill Lin fa6d276925 [Feat] Improved CI (#1119)
Co-authored-by: Will Lin <wlsaidhi@gmail.com>
2026-02-24 12:53:02 -08:00
Zhang PeiyuanandWill Lin fc80d95d7e [Misc] Remove STA (#1124)
Co-authored-by: Will Lin <wlsaidhi@gmail.com>
2026-02-23 15:14:42 -08:00
Shao DuanandSolitaryThinker 37cab18780 [bugfix] Added ltx2 guidance missing modulation term (#1100)
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com>
2026-02-23 14:22:31 -08:00
Zhang Peiyuan 128d0b7fc5 [Misc] Remove Teacache (#1121) 2026-02-22 16:53:07 -08:00
Matthew Noto 8092f02e6d small refactor in post-processing to improve efficiency (#1123) 2026-02-22 16:45:44 -08:00
Zhang Peiyuan 03d9ce2edb [Misc] Remove StepVideo (#1118) 2026-02-21 17:15:42 -08:00
10fc92dba5 Upstream LTX2 Training (#1116)
Co-authored-by: RandNMR73 <notomatthew31@gmail.com>
Co-authored-by: JerryZhou54 <zhouw.jerry2017@outlook.com>
Co-authored-by: Davids048 <jundasu@ucsd.edu>
Co-authored-by: Peiyuan Zhang <a1286225768@slurm-h200-204-239.slurm-compute.tenant-slurm.svc.cluster.local>
2026-02-21 16:06:55 -08:00
6736dc06a5 Improve Docs (#1112)
Co-authored-by: Peiyuan Zhang <a1286225768@slurm-h200-204-227.slurm-compute.tenant-slurm.svc.cluster.local>
Co-authored-by: Peiyuan Zhang <a1286225768@slurm-login-0.slurm-login.tenant-slurm.svc.cluster.local>
2026-02-19 14:23:32 -08:00
William Lin 8c002c62af [misc] add hy-world link to readme (#1113) 2026-02-18 12:01:10 -08:00
Darren 7061313d04 [bugfix] get_torch_device and other device calls were being made on non-cuda platforms (#1107) 2026-02-18 11:43:46 -08:00
Zhang Peiyuanandgemini-code-assist[bot] 76d3ba69e0 [Misc] clean up VSA finetuning examples. (#1111)
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2026-02-18 11:37:20 -08:00
8e39ce38c9 [Feat] Native dit implementation for SD3.5 (#1093)
Co-authored-by: Ishan Vaish <vaish.ishan@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-18 10:19:40 +08:00
Darren d4bd8bf2c0 Update README.md (#1110) 2026-02-17 14:54:34 -08:00
Darren e83d7bc50c [bugfix] fix import PreTrainedModel in stepllm.py (#1108) 2026-02-16 21:45:08 -08:00
Jinzhe Pan ff3d5aff75 [Fix] hunyuan postprecessing issue (#1104) 2026-02-15 12:26:35 -08:00
XOR-op 959dbcc8a2 [perf] causal MatrixGame optimization (#1078) 2026-02-15 09:51:00 +08:00
William Lin 36bf37e9ba [bugfix] Fix failed kernel publish and SFT regressions (#1103) 2026-02-14 16:19:17 -08:00
Mihir Jagtap 7a83e0e6fc [feature] Add Hunyuan-GameCraft model support (#1071) 2026-02-14 08:07:44 +08:00
William Lin 8be1313b86 [kernel] add torch 2.10 to package build matrix (#1099) 2026-02-13 13:03:05 -08:00
alexzms d925ad05f3 [bugfix] fastvideo-kernel: fix VSA Triton padding NaNs and support q/kv length mismatch (#1094) 2026-02-13 12:39:49 -08:00
Shao DuanandWill Lin 7f795600c8 [bugfix] Fixed ltx2 base cfg guidance (#1095)
Co-authored-by: Will Lin <wlsaidhi@gmail.com>
2026-02-13 11:53:11 -08:00
William Lin ec16b6b01d [misc] update wechat group link (#1098) 2026-02-13 01:19:11 -08:00
Kaiqin Kong 31c0f1b341 [feat] Port LingBot-World-Base (Cam) (#1081) 2026-02-10 11:12:33 -08:00
William Lin 4bee0fa199 [misc] cleanup assets/ and demo/ (#1091) 2026-02-10 02:26:09 -08:00
530e6b8363 [Model] LTX 2 Base (#1064)
Co-authored-by: Davids048 <jundasu@ucsd.edu>
Co-authored-by: Will Lin <wlsaidhi@gmail.com>
2026-02-10 01:11:17 -08:00
Jinzhe Pan 9ab2725db1 [ci] CI Transformer Tests (#1089) 2026-02-10 01:08:59 -08:00
IshanandJinzhe Pan 0aff68f51d [Feat] Add Stable Diffusion 3.5 (#1075)
Co-authored-by: Jinzhe Pan <eigensystem1318@gmail.com>
2026-02-10 14:31:36 +08:00
ad58f802f3 [Feat] Port LTX2 trainer (#1074)
Co-authored-by: Davids048 <jundasu@ucsd.edu>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
2026-02-09 17:32:57 -08:00
Wei Zhou 04fa356ee3 [Misc] [Training] Fixed a bunch of bugs in current training pipeline (#1084) 2026-02-09 16:01:05 -08:00
Matthew Notoandgemini-code-assist[bot] f9c076fe2b [misc] add AGENTS.md file (#1085)
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2026-02-09 01:03:49 -08:00
XOR-op f76efe798e [bugfix]: _compile_conditions regression (#1077) 2026-02-06 18:57:43 -08:00
Zhang Peiyuan 09f455233e [misc] readme small fix (#1076) 2026-02-06 16:57:48 -08:00
XOR-op aea300f690 [perf]: use CUDA IPC in multiproc executor to avoid serialization overhead (#1061) 2026-02-06 19:43:18 -05:00
Hao Zhang b92219f6a6 more fix and relocate STA arguments to pipeline config (#1073) 2026-02-06 13:38:35 -08:00
Jinzhe Pan a321b95a8a [Fix] remove video ratio limitation (#1069) 2026-02-05 20:33:08 -08:00
Wei Zhou 98308db7e0 [Feature] [Hy1.5] Support HY1.5 super-resolution pipeline for 1080p videos (#1046) 2026-02-05 16:39:20 -08:00
Hao Zhang c1e18f6722 Some minor fixes (#1068) 2026-02-05 16:28:00 -08:00
William Lin 75e193a2c9 [core] Refactor and centralize our registry for models, pipelines, and sampling params (#1066) 2026-02-05 14:40:30 -08:00
XOR-op d6e0a7d0dd [refactor] Action module (#1065) 2026-02-05 13:56:35 -08:00
William Linandgemini-code-assist[bot] 7fc5f241da [misc] Fix naming instruction in runpod.md (#1067)
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2026-02-05 13:45:49 -08:00
Mingjia Huo aae48a7e90 [feat] HYworld VAE with cache (#1057) 2026-02-05 04:18:17 -08:00
William Lin 88f38eb0f4 [misc] upgrade torch to 2.10 (#1048) 2026-02-05 04:15:22 -08:00
Shao Duan d750b463dc Added Sequence Parallelism for LTX-2 Distilled (#1036) 2026-02-04 15:25:30 -08:00
KyleShaoandWill Lin e10b26a3d8 [feat] Add Cosmos 2.5 I2W/V2W support (staged pipeline + examples) (#1021)
Co-authored-by: Will Lin <wlsaidhi@gmail.com>
2026-02-04 14:19:30 -08:00
XOR-op 74636ba246 [chore]: use higher precision timestamp in logging (#1062) 2026-02-04 13:22:59 -08:00
Wei Zhou 7e2f3f14e7 [Bugfix] [Wan I2V] Fix CLIP Image encoder config (#1063) 2026-02-04 13:20:19 -08:00
Kaiqin Kong caa1c402ba [bugfix] Double Normalization in Preprocessing Dataset (#1055) 2026-01-31 11:11:59 -08:00
XOR-op 38a6bd93d3 [chore]: update sageattn3 installation instructions (#1050) 2026-01-29 15:45:52 -08:00
alexzms b867ef7e7c [SP Sharding] Fix SP loss sharding on token axis (thw) with padding; add distributed correctness tests (#1045)
Fixes the sequence parallel sharding on t, now SP shards on t*h*w
2026-01-27 22:53:04 -08:00
William Lin 3ae58c277a [docs] Update design overview and add agents tutorial (#1044) 2026-01-27 15:56:58 -08:00
Kaiqin Kong 0c6862ca55 [feature] Add Matrix Game 2.0 training (#1017)
The CI tests are quite unstable, but since multiple CI tests indicates that each individual tests are passed, I think we can merge this.
2026-01-26 19:21:43 -08:00
XOR-opandWill Lin e8c854bcf1 [docs] Offloading instruction (#1022)
Co-authored-by: Will Lin <wlsaidhi@gmail.com>
2026-01-26 13:40:28 -08:00
William Lin 06860e96fe [docs] Update runpod instructions (#1043) 2026-01-26 13:13:54 -08:00
Matthew Noto 1b503554d1 [bugfix] fix torchvision import (#1039) 2026-01-24 22:37:14 -08:00
Shreejith SGandgemini-code-assist[bot] 351ceb7c59 [bugfix]: handle architectural differences while lora extraction (#1035)
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2026-01-24 15:42:28 -08:00
KyleShao 10875e0d7b [bugfix] Fix NCCL all_gather contiguity + correct ParallelTiledVAE decode tiling threshold (#1037) 2026-01-24 15:37:35 -08:00
alexzms 1eaae8a10b [ci] Increase ci test error threshold (#1038) 2026-01-24 15:36:10 -08:00
Mingjia Huo 59e00f6164 [feat] Add HY-World1.5-Bidirectional-480P-I2V (#1027)
VAE requires further improvement, will raise PR in near future.
2026-01-23 14:18:04 -08:00
745cc05b10 [bugfix] Allow update timesteps for hy1.5 model. (#1033)
Co-authored-by: Davids048 <jundasu@ucsd.edu>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2026-01-22 22:04:53 -08:00
William Lin c5dc244871 [bugfix] add omegaconf as dep. (#1032) 2026-01-22 11:59:28 -08:00
alexzms dbf3917bf4 [fastvideo-kernel] replace map to index with Triton implementation + add vsa benchmark (#1029) 2026-01-22 11:35:02 -08:00
XOR-op 050f189c95 fix: SP for hunyuanvideo 1.5 (#1026) 2026-01-21 14:40:06 -08:00
Shao DuanandWill Lin 029216029f Added LTX-2 Distilled T2V Generation (#1016)
Co-authored-by: Will Lin <wlsaidhi@gmail.com>
2026-01-21 14:11:39 -08:00
alexzmsandWilliam Lin 31f44110b5 [kernel] [bugfix] [ci] bump v0.2.4. Fix STA output handling, TurboDiffusion CUDA norm dtypes for fastvideo-kernel unit tests. (#1020)
Co-authored-by: William Lin <SolitaryThinker@users.noreply.github.com>
2026-01-19 17:42:42 -08:00
William Lin 21f3ce6577 [kernel] Fix fastvideo-kernel release workflow (#1019) 2026-01-17 15:57:46 -08:00
XOR-op 785d123e36 [feat] Hooks API and layerwise offloading for all DiTs (#1006) 2026-01-17 11:22:02 -08:00
William Lin d58c551c11 [chore] release fastvideo-kernel 0.2.3 (#1018) 2026-01-17 02:24:23 -08:00
alexzms 560628709c [Bug Fix] Add autograd wrapper for block-sparse attention in fastvideo-kernel + fix CMake extension linking (#1015) 2026-01-16 21:16:43 -08:00
William Lin 0f53b51e6c [CI] Fix OOM issues in ssim tests (#1011) 2026-01-16 21:15:20 -08:00
alexzmsandWill Lin 06093a9c4e [CI] SSIM tests optimization: load all model weights from Modal persistent Volume (#958)
Co-authored-by: Will Lin <wlsaidhi@gmail.com>
2026-01-16 11:50:43 -08:00
KyleShao dbddfab6d2 [feat] Introduce Cosmos 2.5 Text2World pipeline (#974) 2026-01-15 15:09:05 -08:00
William Lin 7188170277 [misc] [bugfix] unpin 'av' in pyproject (#1009) 2026-01-13 15:40:46 -08:00
XOR-op b7f69c2c1d [feat!] Disable FSDP inference by default (#1001) 2026-01-13 14:20:05 -08:00
Loay Rashid 23a4531491 [CI] Fixed Turbodiffusion I2V CI (#1002) 2026-01-13 01:08:58 -08:00
William Linandgemini-code-assist[bot] 7d52ad0118 [ci] temporarily disable turbodiffusion ssim test (#1000)
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2026-01-08 15:43:50 -08:00
Will Lin 4d7bf35fa3 Revert "dit"
This reverts commit a6a9c9ca07.
2026-01-07 03:22:48 -08:00
Will Lin a6a9c9ca07 dit 2026-01-07 03:18:48 -08:00
f4704847c2 [bugfix] Add configs for TurboDiffusion T2V/I2V models (#993)
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Will Lin <wlsaidhi@gmail.com>
2026-01-06 16:36:45 -06:00
Shreejith SGandWill Lin d9c996310b [docs]: add LoRA extraction utilities documentation (#992)
Co-authored-by: Will Lin <wlsaidhi@gmail.com>
2026-01-06 16:36:26 -06:00
Shao Duan d6651afd2e [examples] Added longcat-video python api examples (#994) 2026-01-06 15:03:42 -06:00
William Lin cf67618cad [chore] release 0.1.7 (real) (#980) 2026-01-05 15:47:05 -06:00
William Lin 2f0a2b3c57 [misc] add pin_cpu_memory false for RTX 4090 (#990) 2026-01-05 15:45:35 -06:00
Loay Rashid e7748d9952 [feat] add Turbodiffusion I2V pipeline (#984) 2026-01-05 15:41:23 -06:00
William Lin 8eb3140b2f [misc] pin fastvideo-kernel in .toml file (#989) 2026-01-05 13:42:32 -06:00
Shao Duan d6ddcea682 Add LongCat-Video I2V and Video Continuation (Base, Distillation and Refinement) Support to FastVideo (#953) 2026-01-04 22:20:09 -06:00
William Lin 3559ba2377 [chore] update wechat QR code (#988) 2026-01-04 21:59:38 -06:00
William Lin 61e63ea0d7 [chore] release fastvideo-kernel 0.2.2 (#986) 2026-01-04 21:21:06 -06:00
William Lin 4ce4ac4734 [ci] increase ssim and lora inference test timeout (#985) 2026-01-04 15:08:20 -06:00
William Lin e7f6db9bd1 [docs] Update docs and README (#975) 2026-01-04 14:59:19 -06:00
Ohm-Rishabh d83f45a6a0 Layer offloading (#966) 2026-01-03 21:46:00 -08:00
XOR-op dd91542cd1 [feat] Support text encoder weight override and quantization (#983) 2026-01-03 15:33:33 -06:00
Kaiqin Kong 581e8115fe [feat] support Matrix-Game 2.0 streaming generation (#957) 2026-01-02 19:14:38 -06:00
Loay Rashid dea69cf651 [New Model] Turbodiffusion (#971) 2026-01-02 17:55:56 -06:00
XOR-op 60ac6537df [feat] Support absmax style quantization for FP8 (#981) 2026-01-02 16:00:18 -06:00
Qi Jia 5285116e73 [docs]: fix various broken links across the documentation (#979) 2026-01-01 20:02:39 -06:00
William Lin 40ce2d72f5 [kernel] add turbodiffusion kernels (#972) 2025-12-30 04:23:10 -06:00
William Lin 704bc9aaf9 [misc] Add util script to create diffuser HF repo from custom component weights (#970) 2025-12-29 19:38:30 -06:00
RoyWangandroywang de264fcc99 [fix]: fix STA trition kernel for AMD RDNA archs (#969)
Co-authored-by: roywang <roywang@amd.com>
2025-12-29 14:25:32 -06:00
RoyWangandroywang 7b952e4673 [fix]: fix fastvideo-kernel Rocm build and Dockerfile for Rocm (#968)
Co-authored-by: roywang <roywang@amd.com>
2025-12-29 14:24:46 -06:00
551b2d2048 [fix]: fix sliding_tile_attn with sdpa(without flash_attn) (#967)
Co-authored-by: roywang <roywang@amd.com>
Co-authored-by: Will Lin <wlsaidhi@gmail.com>
2025-12-29 14:19:40 -06:00
Ketaki Tank 7bfaf82fd7 [feat] Add new feature extractors for fvd (#954) 2025-12-27 05:08:26 -06:00
William Lin 9cd6a86b95 [chore] release v0.1.7 (#955) 2025-12-27 05:05:37 -06:00
William Lin 16e9552778 [kernel] Fix docker release build for kernel (#965) 2025-12-26 21:42:55 -06:00
William Lin 87f8a2782d [docs] refactor attention docs (#964) 2025-12-26 15:21:49 -06:00
William Lin cbbb09d7b8 [kernel] Release fastvideo-kernel v0.2.1 (#963) 2025-12-26 13:59:03 -06:00
William LinandShreejithSG 2f6230abcf [kernel] Reorg and fix fastvideo-kernel (#962)
Co-authored-by: ShreejithSG <shreejithsg@gmail.com>
2025-12-26 01:50:24 -06:00
Shreejith SGandWilliam Lin f8bfc76015 feat: consolidate attention kernels into unified fastvideo-kernel package (#946)
Co-authored-by: William Lin <SolitaryThinker@users.noreply.github.com>
2025-12-24 01:39:51 -06:00
alexzmsandShao Duan 8f1e6c3336 Add LongCat T2V (Base, Distillation and Refinement) Support to FastVideo (#883)
Co-authored-by: Shao Duan <shaoxiongduan@gmail.com>
2025-12-23 01:11:18 -06:00
William Lin 8e7d2e7879 [bugfix] [dmd2] allow dmd2 simulate_student_forward to use text-only dataset (#951) 2025-12-23 00:41:31 -06:00
William Lin 6ab2870942 [rocm] Add rocm fastvideo docker image (#952) 2025-12-22 18:21:04 -06:00
RoyWang e0ad145152 [feat] add sliding_tile attention triton kernel and ROCM support (#916) 2025-12-22 18:02:51 -06:00
Matthew Noto da04d08426 [docs] small fixes (#947) 2025-12-22 15:23:55 -06:00
Wei Zhou 1f70032af5 [New Model] Hunyuan1.5 (#943) 2025-12-21 00:57:52 -06:00
William Lin 7f71994653 [misc] Allow manual override of Pipeline class through override_pipeline_cls_name (#945) 2025-12-20 14:39:17 -06:00
Loay Rashid 2bb3349da1 [bugfix] Added VSA Padding logic (#944) 2025-12-20 14:29:11 -06:00
Kaiqin Kong 8fe1689968 [feat] Add Matrix-Game 2.0 (#938) 2025-12-20 14:09:12 -06:00
Loay Rashid e53730f324 [docs] Minor Fixes (#942) 2025-12-19 16:48:16 -06:00
Loay Rashid 7a4fe9086a [feat] Support sequence packing and shard after pachification for USP (#894) 2025-12-19 16:19:46 -06:00
Ohm-Rishabh d277361aae [misc] add schedule configurations to pytorch profiler (#934) 2025-12-18 01:45:23 -06:00
alexzms 734a54e7a9 [ci]: Use pre-built docker image & skip VSA compilation (#939) 2025-12-16 23:14:11 -08:00
alexzms 91364982df [Feature] Support for Variable Q/KV Sequence Lengths in VSA ThunderKittens kernel (#911) 2025-12-16 20:08:15 -08:00
William Lin 50145e4fcb [CI] Fix CI tests (#935) 2025-12-16 04:59:43 -08:00
William Lin 4112507e99 [misc] upgrade pytorch version to 2.9.0 (#928) 2025-12-15 04:12:43 -08:00
William Lin 424fc2b4ae [bugfix] [lora] [distillation] Fix lora distillation bug (#933) 2025-12-15 04:12:02 -08:00
William Lin e6066223e6 [bugfix] [VSA] [distillation] Various bugfixes for VSA and distillation and nightly tests (#932) 2025-12-12 16:51:54 -08:00
William Lin b6fa3d24d8 [misc] update wechat image (#931) 2025-12-11 21:22:21 -08:00
Ketaki Tank 55c2e7cd76 [feat] Add fvd implementation (#923) 2025-12-11 19:06:19 -08:00
Tuyabei 5a549af823 [bugfix] [VSA] Fix block_size computation in backward kernel (#925) 2025-12-10 14:36:40 -08:00
Shreejith SG 92fb660c2e Add LoRA extraction, verification, and comparison scripts (#865) 2025-12-08 16:07:58 -08:00
William Lin 3ff640b2e6 [bigfix] [distillation] Fix DMD inference pipeline noise initialization shape (#921) 2025-12-08 13:00:48 -08:00
William Lin c722429ab5 [docs] fix testing.md visibility (#920) 2025-12-08 00:44:53 -08:00
KyleShaoandKyleS1016 e04a192de6 [feat]: add COSMOS 2.5 DiT implementation (#897)
Co-authored-by: KyleS1016 <kyle.s@gmicloud.ai>
2025-12-07 21:48:32 -08:00
William Lin c9ca6d1298 [docs] add docs for ssim testing (#918) 2025-12-06 18:20:04 -08:00
Wenxuan TanandSolitaryThinker 754292c419 Use assert_close in tests (#429)
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com>
2025-12-06 18:18:25 -08:00
Qi Jia 0082bc66fc fix: correct mp backend GPU assignment on multi-GPU systems (#912) 2025-11-30 23:00:22 -08:00
Ohm-Rishabh 8b1937422e [feat] training mfu calculation scripts (#871) 2025-11-27 16:54:17 -08:00
fb6cbf23e6 Fix the docs (#905)
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: William Lin <SolitaryThinker@users.noreply.github.com>
2025-11-27 00:37:03 -08:00
Mihir Jagtap c8fdd5ed7b [docs] modified the .github/workflows/docs.yml file to include path filtering (#906) 2025-11-26 17:34:21 -08:00
Loay Rashid 1c19a6a00c [Bugfix] Minor bugfixes (#889) 2025-11-26 17:20:45 -08:00
William Lin d44409c704 [CI] fix VSA training CI (#900) 2025-11-24 17:47:59 -08:00
Zhang Peiyuan 5d1c7852b7 + Awesome work using FastVideo or our research projects (#898) 2025-11-23 22:22:27 -08:00
Wenxuan Tan 77a211d006 [misc] Update wechat link (#893) 2025-11-20 19:59:05 -08:00
Wei Zhou bef8169bb1 [Feat] [I2V] resize all image sizes to below 480*832 (#890) 2025-11-20 00:08:36 -08:00
William Lin 681f1583f9 [readme] update link to inference code (#887) 2025-11-19 13:24:13 -08:00
e3b4564d5a [feat] Add inference for MoE SF (#880)
Co-authored-by: RandNMR73 <notomatthew31@gmail.com>
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com>
2025-11-19 13:16:24 -08:00
Shao Duan c0d03fc43d [bugfix] [lora] [CI] Fix LoRA alpha scaling factor & Fix LoRA Inference CI (#870) 2025-11-19 01:02:01 -08:00
Wei Zhou 404ee8538e [Bugfix] [DMD Distillation] Each rank should have its own timestep sampled (#885) 2025-11-18 14:03:25 -08:00
Shao Duan e57ac59462 Fix mp worker busy loop to handle all string RPC methods (#881) 2025-11-16 13:26:44 -08:00
Mihir Jagtap 8c55fdaf7e [docs] add favicon (#878) 2025-11-15 13:44:16 -08:00
Y-aang c30779184f fix: incorrect dv in vsa Triton kernel causing test_vsa error (#879) 2025-11-14 22:00:39 -08:00
William Lin 9d188c0b6c [misc] update wechat and slack invite links (#875) 2025-11-12 23:03:56 -08:00
Mihir Jagtap 9dd7c54221 [docs] Update Home Readme.md with fixed links (#873) 2025-11-12 13:32:44 -08:00
William Lin 62b95d8287 [feat] prepare for wan2.2 SF (#861) 2025-11-04 18:06:48 -08:00
Kaiqin Kong fdf21702f5 [Docs] add diagrams to docs (#863) 2025-11-04 16:29:07 -08:00
Ohm-Rishabh 2972fc9449 Improve FSDP loading with size-based filtering (#853) 2025-11-04 15:31:07 -08:00
Mihir Jagtap 8f5712629f [docs] port to mkdocs (#855) 2025-11-04 14:31:56 -08:00
Kevin Lin 436c701b9f [bugfix] Add Cosmos2 sampling params to registry (#862) 2025-11-02 00:09:17 -07:00
Kevin Lin 543fea88e3 [Feature] Add Cosmos2 i2v pipeline (#837) 2025-10-30 20:03:57 -07:00
Kaiqin Kong bdec816b31 move STA_configuration.py to fastvideo/attention/backends (#856) 2025-10-29 13:54:13 -07:00
William Lin 2cd2e57d2e [ci] fix causal ssim test (#848) 2025-10-26 19:33:07 -07:00
William Linandainsley 9370234294 [feat] Add gradio local inference demo (#847)
Co-authored-by: ainsley <jzhang2765@wisc.edu>
2025-10-26 07:01:33 -07:00
Jinzhe Pan 50da62e722 [bugfix] always force spawn instead of fork (#852) 2025-10-23 16:36:50 -07:00
William Lin 4f3e8751db [bugfix] [misc] Use training_state_checkpointing_steps in scripts/ (#846) 2025-10-19 20:20:53 -07:00
Jinzhe PanandXingyu Long f4c58894d9 [Feat] add ray support (#838)
Co-authored-by: Xingyu Long <xingyulong97@gmail.com>
2025-10-16 23:17:54 -07:00
Ohm-Rishabh 01c94ef385 [feat] unified trainer logging (#841) 2025-10-16 23:16:16 -07:00
Zhang Peiyuan 2415226d25 Update WeChat Link 2025-10-13 21:02:46 -07:00
Jiali Chen 404314d00f [Feature]Add video-to-video (V2V) pipeline (#829) 2025-10-12 21:53:05 -07:00
zyang6andkiritorl 87489f0872 Add wan2.1 functionality support for Ascend NPU platform (#810)
Co-authored-by: kiritorl <1021709528@qq.com>
2025-10-09 16:25:08 -07:00
Zhang Peiyuan 9ce7c8039e Update Wechat link 2025-10-06 15:01:19 -07:00
William Lin e1e25e95f9 [feature] Add torch profiler (#827) 2025-10-06 07:59:46 -07:00
William Lin 490bde90e1 [bugfix] Allow overriding dit checkpoint for inference and Lower VSA LR in example scripts (#831) 2025-10-05 01:44:49 -07:00
dc7596b973 [self-forcing][8/n] Self-Forcing For Wan2.2-A14B + torch.compile training and distillation support (#818)
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2025-10-02 15:01:45 -07:00
William Lin 335afa4457 [bugfix] Use training_state_checkpointing_steps instead of checkpointing_steps (#821) 2025-09-28 15:22:43 -07:00
Yongqi Chen 3f77a6805a [Feature]Update count trainable param for FSDP2 (#820) 2025-09-28 15:22:04 -07:00
RandNMR73 13d0aae706 Add Sage Attention 3 Backend (#815) 2025-09-24 15:11:38 -07:00
William Lin 404cbf4f3c [self-forcing] [6/n] Add Ode Init training (#811) 2025-09-22 17:58:19 -07:00
William Lin 958ffec844 [bugfix] Update learning rates for sparse distillation recipe (#812) 2025-09-22 12:07:03 -07:00
31f000d1cc [self-forcing] [5/n] Add Self-Forcing distillation pipeline (#808)
Co-authored-by: RandNMR73 <notomatthew31@gmail.com>
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com>
2025-09-20 19:32:10 -07:00
Yongqi Chen cd32b3e02f Update example files and readme (#809) 2025-09-20 18:15:59 -07:00
Zhang Peiyuan bf27908095 Update WeChat Link 2025-09-20 14:16:20 -07:00
William Lin c5f9ea53b2 [self-forcing] [4/n] Preprocessing for collecting ODE trajectory (#788) 2025-09-15 17:54:42 -07:00
William Lin d32a7184da [bugfix] Wan2.2 Boundary ratio (#804) 2025-09-15 11:17:35 -07:00
Wenxuan Tanandgemini-code-assist[bot] 2930abe456 [Bugfix] Fix VMoba requirements (#802)
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2025-09-14 18:28:52 -07:00
William Lin b93ef4289d [bugfix] Fix empty PipelineConfigs for Wan2.2 A14B (#800) 2025-09-13 17:31:38 -07:00
401bdbd316 [self-forcing] [3/n] Text embed only preprocessing (#797)
Co-authored-by: RandNMR73 <notomatthew31@gmail.com>
Co-authored-by: JerryZhou54 <zhouw.jerry2017@outlook.com>
Co-authored-by: kevin314 <kevin.lin.cs1@gmail.com>
2025-09-13 14:03:53 -07:00
William Lin 1048d79cf8 [bugfix] pin gradio version and set current_vsa_sparsity in TrainingPipeline (#798) 2025-09-11 17:04:47 -07:00
1e8406162d [bugfix] Fix delta calculation (#796)
Co-authored-by: zbchu2 <zbchu2@iflytek.com>
Co-authored-by: William Lin <SolitaryThinker@users.noreply.github.com>
2025-09-11 16:31:23 -07:00
William Lin 03edd35c83 [preprocessing] [self-forcing] [2/n] Improve preprocessing and add ode trajectory dataset schema (#794) 2025-09-10 17:33:57 -07:00
William LinandRandNMR73 ac11127397 [Self-forcing] [1/n] Handle extra dim in time embedding and add timestep warping (#792)
Co-authored-by: RandNMR73 <notomatthew31@gmail.com>
2025-09-09 02:52:02 -07:00
Eric LiangandEricLiang e028dcc7c0 [Backend][Vmoba] Add implementation of VMoba (#778)
Co-authored-by: EricLiang <https://github.com/EricLina>
2025-09-08 23:53:25 -07:00
Wenxuan Tanandgemini-code-assist[bot] 076f45c1ee [Feature] Support Lora for DMD (#755)
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2025-09-08 14:18:21 -07:00
85eb7265db fix: lora_B init zeros (#781)
Co-authored-by: zbchu2 <zbchu2@iflytek.com>
Co-authored-by: Wenxuan Tan <wenxuan.tan@wisc.edu>
2025-09-05 22:56:52 -07:00
William Lin d3ceb67e66 [misc] Update Slack invite link (#786) 2025-09-05 12:16:18 -07:00
Zhang Peiyuan 7ac153a5ca Update WeChat Link 2025-09-05 11:40:47 -07:00
William Lin d1e7aa0abd [CI] Add ssim test for causal inference (#784) 2025-09-05 01:23:01 -07:00
William Lin 2d846c55a1 [misc] Improve text encoding stage (#774) 2025-09-04 17:51:27 -07:00
Jinzhe Pan b318063c0a [Preprocess][Fix] video quality issue (#773) 2025-09-03 20:47:33 -07:00
Jinzhe Pan 4aa307be55 [Preprocess][Feat] support torchvision to load video in new preprocessing (#761) 2025-09-01 23:37:01 -07:00
William Lin 055e52e5ea [misc] [VSA] [STA] fix tk_root in setup.py for VSA and STA (#772) 2025-08-29 01:13:37 -07:00
William Lin 7d2069596b [bugfix] [VSA] [STA] Fix MANIFEST.in for VSA and STA; Move tk into both directories (#771) 2025-08-29 00:51:05 -07:00
William Lin c45009c9a4 [bugfix] fix STA install setup.py import (#770) 2025-08-28 23:02:53 -07:00
William LinandPeiyuan Zhang b91020b407 [VSA] [STA] Fix directory structure for pypi publishing (#769)
Co-authored-by: Peiyuan Zhang <a1286225768@gmail.com>
2025-08-28 22:34:03 -07:00
William Lin 2dcc5ea4f6 [chore] Release 0.1.6 (#768) 2025-08-28 20:56:21 -07:00
Wei ZhouandSolitaryThinker 359151d9a0 [Feature] Add wan2.2 5b i2v (#760)
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com>
2025-08-28 18:15:59 -07:00
Wei ZhouandSolitaryThinker ce67cd3729 [Feat] Support Self-Forcing's Causal Inference for Wan2.1 T2V 1.3B (#766)
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com>
2025-08-28 16:47:49 -07:00
Zhang Peiyuan 7c554e5da8 Update Community Link (#765) 2025-08-27 16:12:47 -07:00
William Lin 663ea33ff1 [bugfix] Fix wrong HF model string for FastWan2.2 5B (#763) 2025-08-26 22:05:40 -07:00
William Lin 3ef04f1654 [misc] [docs] Various fixes for logging and docs (#758) 2025-08-23 21:13:50 -07:00
Jinzhe Pan 0eced76a41 [Feat][Preprocess] support multi-gpus (#753) 2025-08-23 11:34:42 +08:00
Jinzhe Pan 3ab6470d1a [Feat][Preprocess] support merged dataset (#752) 2025-08-22 15:29:33 -07:00
Wenxuan Tan 989a03532c Optionally use unmerged weights for inference (#745) 2025-08-22 15:20:31 -07:00
William Lin fa15369a02 [bugfix] Check that model_index.json module is in required_modules list before removing (#756) 2025-08-22 14:36:44 -07:00
Zhang Peiyuan 78a9cb88d8 [Fix] fix seed in dmd denoising loop (#736) 2025-08-21 18:06:16 -07:00
Peng Xiaoand肖鹏 a0bff12746 [bugfix] [dmd] Align backward simulation with dmd2 sample back (#744)
Co-authored-by: 肖鹏 <xiaopeng1@aishi.ai>
2025-08-20 22:25:33 -07:00
William Lin 98f2af94e5 [bugfix] Missing Docker file for cuda12.9 (#750) 2025-08-20 15:34:31 -07:00
William Lin 46f7b6d574 [Docker] add 12.9 docker image and also fix py3.10 and py3.11 dockerfile (#749) 2025-08-20 15:31:15 -07:00
Jinzhe Pan 911a6a6a35 [Feat][Preprocessing] i2v preprocessing workflow (#737) 2025-08-14 20:47:25 -07:00
Zhang Peiyuan 38c7949d5c Update WeChat group link (#739) 2025-08-14 15:03:35 -07:00
Jinzhe Pan 7e7a0dba9d feat: preprocess validation dataset only when exist (#734) 2025-08-12 02:16:31 -07:00
Zhang Peiyuan f62e210ae6 Fix vsa backward gQ (#735) 2025-08-11 21:43:13 -07:00
William Lin 6ceb4942a0 [bugfix] [dmd] Fix backward simulation and also naming in wan_i2v_dmd_pipeline (#731) 2025-08-10 21:13:30 -07:00
William LinandRandNMR73 8cae5e4708 [feature] add Gradio live serving demo code (#727)
Co-authored-by: RandNMR73 <notomatthew31@gmail.com>
2025-08-10 15:34:03 -07:00
William Lin 2a773fa34e [bugfix] [distill] remove i2v validation schema import in distill (#728) 2025-08-09 20:47:42 -07:00
Wenxuan Tan 5357f63327 Fix LoRA load from training checkpoint (#719) 2025-08-09 20:46:00 -05:00
William Lin 60f61c8101 [bugfix] fix pyproject install and VSA precision test (#726) 2025-08-08 18:45:03 -07:00
Jiali Chen 3d75ba8251 update version selection for VSA workflow (#725) 2025-08-08 13:05:16 -07:00
Wenxuan Tan 6c6bcd914d Remove all empty_cache (#713) 2025-08-07 22:50:38 -07:00
Jiali Chen f79b08de81 add cicd workflow for publishing VSA kernel (#723) 2025-08-07 18:53:05 -07:00
Jinzhe Pan f2bc037fff [Fix] training pipeline pin_cpu_memory issue (#692) 2025-08-07 02:31:20 -07:00
Jinzhe Pan 86604a684b [3/3][Preprocess] add preprocessing workflows (#645) 2025-08-07 01:49:07 -07:00
Zhang Peiyuan 47bd1e0178 [Misc] change installation logic of vsa (#721) 2025-08-06 21:54:09 -07:00
Wei ZhouandSolitaryThinker c41305ad18 [Feat] Add Wan2.2 14B MoE (#688)
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com>
2025-08-06 20:31:03 -07:00
Zhang Peiyuan 98ce9034f0 [Chore] Include our demo in the readme. (#720) 2025-08-06 19:29:40 -07:00
William Lin 0ceff110da [chore] Release 0.1.5 (#717) 2025-08-06 13:07:52 -07:00
Yongqi Chen 1d018acb3e [Feature]Add Data-free distillation readme (#710) 2025-08-05 14:27:39 -04:00
Yongqi Chen 7d8cf38dbe Fix typo (#709) 2025-08-04 20:21:21 -07:00
Yongqi Chen 8d483fe4aa [Bugfix] Fix neg_prompt bug when training from local cp (#708) 2025-08-04 15:54:06 -07:00
Zhang Peiyuan c1191250bf Add WeChat group link (#707) 2025-08-04 15:19:01 -07:00
Wenxuan Tan 4b7266349a [misc] Remove allow_tf32 in scripts (#705) 2025-08-04 15:37:56 -05:00
Yongqi Chen 22f9b7681f [Feature]Update Wan2.2+DMD doc example (#706) 2025-08-04 16:14:22 -04:00
Yongqi Chen 589d32cc39 [Feature] Update Readme and scripts (#703) 2025-08-04 15:02:32 -04:00
Hao Zhang 89199837db Update readme pre-release (#704) 2025-08-04 11:54:31 -07:00
William Lin d6ebaf1b49 [Docs] Fix README (#701) 2025-08-04 11:27:27 -07:00
Yongqi Chen fac927777c [Feature] Update readme (#702) 2025-08-04 14:27:18 -04:00
William Lin ecbd697dae [misc] Readme fixes (#699) 2025-08-04 10:20:57 -07:00
Yongqi Chen 7d4acef64d [Feature] Update sparse distill readme and doc (#700) 2025-08-04 10:16:20 -07:00
William Lin 9f0ce517cf [Docs] Update README and docs for FastWan (#698) 2025-08-04 09:05:18 -07:00
Yongqi Chen c718e56b0d [Feature] Remove unused args (#695) 2025-08-03 23:01:54 -04:00
Yongqi ChenandSolitaryThinker b65f0316d1 [Feature] Add Wan2.2 DMD example files; Update lr scheduler (#694)
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com>
2025-08-03 22:55:01 -04:00
William Lin 8d8bcb76b0 [config] Add config for FastWan2.2 ti2v 5B (#693) 2025-08-03 19:09:30 -07:00
Yongqi ChenandSolitaryThinker 5f42748ed1 [Feature] Add Wan2.2-TI2V-5B Sparse Distill (#690)
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com>
2025-08-03 01:29:57 -04:00
Yongqi Chen c9005045dc [Feature[[Readme] Add VSA/DMD doc (#673) 2025-08-02 02:35:46 -04:00
Wenxuan Tan 6c81befc87 [Feature] Optionally enable torch compile (#684) 2025-08-01 20:17:40 -07:00
Yongqi Chen dfe0b288e1 [Bugfix] Add i2v vae loading (#686) 2025-08-01 23:15:37 -04:00
Wenxuan Tanandgemini-code-assist[bot] 31200fbb83 [Misc] Fix training scripts (#683)
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2025-08-01 15:54:31 -05:00
Yongqi Chen 9185978c55 [Bugfix] Fix multi-gpu training lr_scheduler (#682) 2025-08-01 15:54:03 -04:00
MartinPernus fcba463553 [Bugfix] fix _normalize_dit_input (#681) 2025-08-01 05:10:27 -04:00
Yongqi Chen 2c53d3eecf [Feature]Add DMD visualization for debugging (#674) 2025-07-31 05:54:47 -04:00
Zhang Peiyuan 516ecd374a [Misc] Update examples/ and other misc (#672) 2025-07-30 19:10:27 -07:00
Wei Zhou 3b1b54a74d Modify args to make sure the scripts are runnable on 4090 (#671) 2025-07-30 14:55:08 -07:00
Yongqi Chen 6914e7c904 [Bugfix]Fix DMD pipeline registry (#670) 2025-07-30 13:21:35 -07:00
Sopiko Kurdadze 5452369749 [Feature] [Inference]Add ROCm platform support for single-gpu inference (#669) 2025-07-30 12:57:02 -07:00
Yongqi Chen a113311e77 [Bugfix][Training]Fix Wan2.2 training vae config issue (#668) 2025-07-30 12:24:45 -07:00
Kevin Lin 44da97da92 [chore] Release 0.1.4 (#667) 2025-07-30 01:02:36 -07:00
Yongqi Chen f759980a58 [Feature]Add VSA slurm training example scripts (#666) 2025-07-30 01:27:54 -04:00
Zhang Peiyuan 37e0f8c236 [BUG] Fix distillation + vsa (#665) 2025-07-29 19:48:11 -07:00
Kevin Lin 51711d5906 [ComfyUI] Add __init__.py for node discovery (#663) 2025-07-29 18:21:09 -07:00
William LinandJerryZhou54 6375223b16 [Feature] Add wan2.2 5B T2V (#658)
Co-authored-by: JerryZhou54 <zhouw.jerry2017@outlook.com>
2025-07-29 17:16:03 -07:00
Yongqi Chen 4cb046768d [Feature]Add DMD distillation training resume checkpoint; Update DMD CI test (#662) 2025-07-29 19:06:05 -04:00
Yongqi Chen 3322542444 [Feature] Add DMD CI test (#661) 2025-07-29 03:14:00 -04:00
Yongqi Chen 65f707354b [Bugfix]Fix mdoel inference checkpoint saving when enabling HSDP (#660) 2025-07-28 22:39:06 -07:00
Yongqi Chen 109e2e7e9d [Bugfix]Fix DMD wan pipeline (#659) 2025-07-28 21:50:44 -07:00
Zhang Peiyuan cbc3a6bb9d [Feat] Support VSA with any resolution. (#650) 2025-07-28 20:14:40 -07:00
Yongqi Chen 2fa8d4ae6d [Feature][Distill]Add 14B 480p T2V distill example scripts (#655) 2025-07-28 18:36:31 -04:00
Jinzhe Pan 7b6c8aee99 [2/3][Preprocess] refactor pipeline registry & file structure (#639) 2025-07-27 23:30:17 -07:00
Yongqi Chen 6284eaa363 [Feature][Distill]Add DMD+VSA joint training example (#654) 2025-07-27 18:18:01 -04:00
Yongqi Chen 636524e87f [Feature] Add Wan-14B-T2V-VSA CLI inference; add master port args (#653) 2025-07-27 07:13:44 -04:00
Yongqi Chen 202b2f3972 [Feature] Ignore [union-attr] and [override] mypy check and remove from training (#652) 2025-07-27 04:34:38 -04:00
Yongqi Chen 247fe273d8 [Feature] Add DMD T2V training pipeline (#651) 2025-07-27 03:35:51 -04:00
William Lin cb320dfa3a [bugfix] VideoGenerator improperly extracts output_video_name (#649) 2025-07-26 19:46:26 -07:00
Kevin Lin d8bb5abc46 [CI] Fix ComfyUI publisher ID (#648) 2025-07-25 19:02:52 -07:00
Kevin Lin cc703eca51 [CI] Add publish workflow for ComfyUI (#647) 2025-07-25 18:39:05 -07:00
William Lin 81c9df629c [core] Add offloading for vae and image encoder and rename offloading args (#643) 2025-07-25 17:55:03 -07:00
Yongqi Chen d3c0c52208 [Feature] Add prompt_txt support for CLI inference; Add DMD CLI inference (#646) 2025-07-25 19:44:10 -04:00
William Lin 744e0555c0 [misc] Use FASTVIDEO_STAGE_LOGGING for perf timing of stage (#644) 2025-07-25 16:20:53 -07:00
Jinzhe Pan 3a38f7dfdc [1/3][Preprocess] refactor preprocessing configs (#638) 2025-07-25 14:27:12 -07:00
William Lin f572319bd9 [Feature] Remove V1 folder (#642) 2025-07-24 22:43:12 -07:00
Wenxuan TanandWei Feng 48528f468c [Feature] Multi-lora inference (#640)
Co-authored-by: Wei (Will) Feng <134637289+weifengpy@users.noreply.github.com>
2025-07-24 21:01:28 -07:00
Yongqi ChenandSolitaryThinker 4264a80ca9 [Feature] Add DMD inference pipeline (#637)
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com>
2025-07-24 21:00:47 -07:00
William Lin 8573d4f05e [Docs] Docs update for Training and MPS (#641) 2025-07-24 19:23:58 -07:00
Wenxuan Tan 210a733515 [Bugfix] Fix LoRA trainable params and training ckpt loading (#630) 2025-07-23 20:01:40 -07:00
William Lin 0aef0e6f63 [bugfix] Fix preprocessing pipelines and nightly tests (#633) 2025-07-22 22:44:25 -07:00
Kevin Lin dd022ad9be [CI] Fix CI for pull request targets other than main (#632) 2025-07-22 21:00:03 -07:00
William Lin 832ad61e5b [bugfix] fa3 no longer returns lse (#631) 2025-07-22 18:31:34 -07:00
Wenxuan Tan 9419c04ee3 Fix lora train steps (#627) 2025-07-21 23:29:30 -05:00
Zhang Peiyuanandroot a37b39d83c Py/add triton block sparse (#593)
Co-authored-by: root <a1286225768@gmail,com>
2025-07-17 16:50:40 -07:00
Wenxuan Tan bb8c769c8e [LoRA] Support v1 LoRA training (#576) 2025-07-17 15:52:09 -05:00
William Lin 576c214f28 [v0] Remove V0 code (#621) 2025-07-15 22:05:16 -07:00
RandNMR73 b79d1fc15b video gen working on apple silicon (addressed issues from prior pr) (#595) 2025-07-15 22:04:27 -07:00
William Lin eb66e1c18d [chore] release 0.1.2 (#622) 2025-07-15 14:42:54 -07:00
William Lin 616d43c1cf [bugfix] [training] use separate generator for validation (#610) 2025-07-15 13:19:38 -07:00
Wenxuan Tan 7244a4b27f [CI] Add LoRA inference tests (#546) 2025-07-15 15:06:44 -05:00
Yongqi Chen 7e5ebb4582 [Feature][Training]Update example fine-tuning scripts to enable gradient checkpointing (#618) 2025-07-15 11:22:56 -07:00
Wenxuan Tan 65ed588570 Set encoder TP size to 1 by default (#569) 2025-07-09 17:44:24 -05:00
Wenxuan Tan 14adfe2edc Remove all unnecessary torch.cuda.empty_cache (#606) 2025-07-09 16:45:20 -05:00
William Lin 6198c6a640 [docs] update dev guide runpod image to py3.12 (#602) 2025-07-07 19:56:07 -05:00
William Lin e6b71b531b [docs] Update slack invite (#601) 2025-07-07 14:20:42 -05:00
William Lin ae1d112c6a [bugfix] [training] fix deadlock in latent datasets and init error in multi-node training (#598) 2025-07-06 01:34:15 -05:00
William Lin bf4de1f38f [chore] Upgrade min Python version from 3.8 to 3.10 (#597) 2025-07-04 22:08:31 -05:00
William Lin 66fdcc8e76 [Training] Use inference pipeline for training validation (#585) 2025-07-04 17:23:55 -05:00
Wenxuan Tan ed1e8d6bad [Feature] Offload all text encoders by default (#594) 2025-07-03 19:30:14 -05:00
Kevin Lin b9423ca3f8 Add ComfyUI custom node for inference (#596) 2025-07-03 16:09:03 -05:00
Wenxuan Tan ad16289871 [LoRA] Fix lora merge weights (#579) 2025-07-02 00:12:52 -05:00
Wenxuan Tan 2a41da1e6b Fix VAE precisions (#588) 2025-07-01 14:49:04 -05:00
William Lin 32133171da [chore] Release 0.1.1 (#592) 2025-07-01 01:43:12 -05:00
Kevin Lin 19674c6f29 [CI] Fix fork builds (#590) 2025-07-01 01:03:34 -05:00
Yongqi Chen 508afb7002 [docs] Update Readme (#591) 2025-06-30 23:48:45 -05:00
Yongqi Chen 288ea88105 [Feat][Training] Rename weight conversion function and update gradient checkpoint in scripts (#589) 2025-07-01 00:20:02 -04:00
Jinzhe Pan eb0f1318f3 [Feat] activation checkpointing (#584) 2025-06-30 15:24:29 -05:00
William Lin ce9b5910cc [Training] add caption to validation log (#582) 2025-06-30 02:42:17 -05:00
William Lin d0e5a6214a [misc] [training] Add --video_length_tolerance_range 10 to preprocessing scripts (#581) 2025-06-30 02:21:22 -05:00
Wenxuan Tan 834562b2db [CI] Fix pre-commit CI (#578) 2025-06-29 16:52:29 -05:00
Wei (Will) Feng 060cc7b9ba fully_shard usage on RMSNorm (#577) 2025-06-29 16:35:24 -05:00
Yongqi Chen 6c58a5ba62 [Bugfix]Fix VSA sp for training/inference (#574) 2025-06-29 13:44:33 -05:00
William Lin 48d9f61f86 [ci] [misc] fix training test threshold (#573) 2025-06-28 22:17:18 -05:00
William Lin 5f938b5844 [Revert] "[Feature] Load weights from distributed" (#571) 2025-06-28 20:55:14 -05:00
Wenxuan Tan 74da2a7370 Fix CLIP config (#568) 2025-06-28 19:01:23 -05:00
Kevin Lin 580d6dfe1f [CI] Add tests to Modal (#562) 2025-06-28 14:02:16 -05:00
Wenxuan Tan 344e43006a [CI] Fix SSIM and transformers CI (#564) 2025-06-28 00:26:20 -05:00
Wenxuan Tan c5155b256e [Feature] Load weights from distributed (#470) 2025-06-27 22:52:40 -05:00
William Lin e005c7f3ac [Docs] [Training] add readme for example training (#563) 2025-06-27 14:50:42 -05:00
Yongqi Chen ff5a79ef60 [Feature][Inference] Add VSA inference script (#561) 2025-06-27 02:19:23 -05:00
William Lin ab01dc4ba5 [Feature] [Training] Add i2v training (#559) 2025-06-27 01:56:50 -05:00
William Lin 285a950c1b [CI] fix vae and ssim tests (#557) 2025-06-26 23:53:01 -05:00
William Lin 46a0a85d85 [Training] Fixes SP for training; Improve Datasets and schema (#555) 2025-06-26 21:13:28 -05:00
Yongqi Chen 4aeabbc629 [Feature][Training] Add cfg rate for dataset loader (#556) 2025-06-26 18:22:37 -04:00
Wenxuan Tan 949bb5c835 [CI] Fix CI checks (#553) 2025-06-25 14:07:51 -05:00
Wenxuan Tan aab74c1271 [Kernel] Remove all syncs from STA & VSA kernels (#517) 2025-06-23 13:13:09 -07:00
Yongqi Chen f89d86944f [Feature][Training]Add diffusers format checkpoint saving for inference (#542) 2025-06-22 01:23:41 -04:00
William Lin 8741d204a5 [Training] Refactor and improve validation datasets (#539) 2025-06-21 17:58:35 -07:00
Wenxuan Tan cdc85f58a8 [chore] Bump torch to 2.7.1 to support Blackwell (#483) 2025-06-20 22:10:56 -07:00
William Lin 0262d2f089 [misc] [training] Reorganize training pipeline (#533) 2025-06-20 20:42:25 -07:00
William Lin 62c0343465 [bugfix] [VSA] Fix layernorm type for VSA Wan2.1 TransformerBlock (#534) 2025-06-20 00:24:51 -07:00
William Lin 1e1a023fb0 [bugfix] Fix stage validator for multi text encoder models (#535) 2025-06-19 22:49:16 -07:00
William Lin 1d2517ad8e [misc] Remove gradient checking code (#532) 2025-06-18 23:29:25 -07:00
William Lin d41186cb4a [Feat] Add Stage input and output verification (#523) 2025-06-18 23:29:11 -07:00
78e0c7eec9 Specify cu128 Pytorch installation (#530)
Co-authored-by: Edenzzzz <wtan45@wisc.edu>
Co-authored-by: Wenxuan Tan <wenxuan.tan@wisc.edu>
2025-06-18 20:02:50 -05:00
Wenxuan Tan 1c41a94b62 [Refactor] Move dict_to_3d_list under utils (#507) 2025-06-18 13:34:37 -07:00
Yongqi Chen 2e66aafe20 [Bugfix][Readme]Fix readme website bugs and add VSA finetune docs (#531) 2025-06-17 22:48:29 -07:00
Yongqi ChenandWill Lin 55074bda76 [CI] Add STA-inference/VSA-training test (#527)
Co-authored-by: Will Lin <wlsaidhi@gmail.com>
2025-06-17 21:13:06 -07:00
William Lin de65bec2b7 [Ci] add sta and vsa install to docker image (#528) 2025-06-17 18:09:48 -07:00
Yongqi Chen 7664dd0de3 [Bugfix][Inference]Fix envs.attn_backend (#525) 2025-06-17 18:38:06 -05:00
William Linandkevin314 019a88ced4 [CI][bugfix] Use new 3.12 docker image (#526)
Co-authored-by: kevin314 <kevin.lin.cs1@gmail.com>
2025-06-17 15:37:08 -07:00
Kevin Lin 72de11abcc [CI] Add current PR test workflow to Buildkite/Modal (#512) 2025-06-17 13:29:22 -07:00
Kevin Lin d71a4ebffc [CI] Update Docker image to flash-attn 2.8.0 / CUDA 12.8 (#524) 2025-06-16 17:48:23 -07:00
William Lin 1089ab43bf [bugfix] [Training] use diffusers fp32layernorm for wan2.1 (#490) 2025-06-15 22:45:48 -07:00
William Lin 97d4b984c9 [misc] [ci] fix e2e preprocess+training data path (#521) 2025-06-14 22:37:51 -07:00
Wenxuan Tan 2a8953d74d [Refactor] Fix attn backend selection not correctly setting env variable (#516) 2025-06-15 00:04:54 -05:00
Yongqi Chen 8801b10da7 [Bugfix][Preprocess]fix mini dataset name (#520) 2025-06-14 22:03:22 -07:00
William Lin 6b413f2ec4 [CI] [Training] drop negative prompt in validation dataset and CI test for preprocess + training overfit (#519) 2025-06-14 18:50:17 -07:00
Yongqi Chen 28b72694aa [Feature][Preprocess]Add Readme doc for preprocess (#518) 2025-06-14 20:41:13 -04:00
Yongqi Chen 4afb0cfe4f [Feature][Training]vsa for t2v training ready (#513) 2025-06-14 01:08:00 -04:00
Zhang Peiyuan 3eec1281cf [misc] Fix preprocessing and dataloader extra padding (#514) 2025-06-13 15:15:33 -07:00
Wenxuan Tan 0660489e38 [CI] Restrict training CI to v1 (#508) 2025-06-12 15:26:05 -07:00
Zhang Peiyuan dd871a17bf fix logging (#509) 2025-06-12 15:24:12 -07:00
dc11529862 [Refactor][Configurations] clean config orgnization (#505)
Co-authored-by: Peiyuan Zhang <a1286225768@gmail.com>
Co-authored-by: Will Lin <wlsaidhi@gmail.com>
2025-06-12 13:27:08 -07:00
Zhang Peiyuan ffabf85e31 [feat] Add parquet iterable dataset. (#506) 2025-06-12 04:30:56 -04:00
William Lin c0026ca5ba [CI] [Training] Initial e2e small training test (#504) 2025-06-11 13:53:36 -07:00
Zhang Peiyuan 0f2bbe71ac [misc] rename dp_size to hdsp_replicate_dim (#491) 2025-06-10 16:36:56 -07:00
Yongqi ChenandJerryZhou54 2a46902ecb [Feature][VSA]Update STA publish workflow (#498)
Co-authored-by: JerryZhou54 <zhouw.jerry2017@outlook.com>
2025-06-10 19:33:34 -04:00
Zhang Peiyuan 66012d3a4c [Feat][Dataloader] 1/n Refactor parquet map-style dataloader (#492) 2025-06-10 16:00:13 -07:00
William Lin f666b9de41 [misc] Add missing license headers (#499) 2025-06-10 14:25:32 -07:00
Yongqi Chen 7e3c073b55 [Feature] Adding VSA inference (#478) 2025-06-10 16:03:53 -04:00
Wei Zhou a6aa21bd07 [bugfix][Cli Inference] Resolve runtime errors when running fastvideo generate (#495) 2025-06-10 02:50:21 -04:00
Wenxuan Tan 6519b57aab [chore] Fix main pre-commit CI failure (#494) 2025-06-10 00:54:37 -05:00
Wei Zhou 675aea6ece [bugfix][Cli Inference] Resolve runtime errors when running fastvideo generate (#493) 2025-06-09 19:49:34 -07:00
Zhang Peiyuan 46e7a15e0d [misc] Improve distributed related env variables and setup (#487) 2025-06-08 09:14:48 -07:00
Yongqi ChenandJerryZhou54 e4f702d7ec [Bug] Fix multi gpus issues in v1 scripts (#489)
Co-authored-by: JerryZhou54 <zhouw.jerry2017@outlook.com>
2025-06-07 21:55:49 -07:00
Wenxuan Tan bb68fcc809 Revert "Add torch.compile for all small ops" (#484) 2025-06-07 07:21:17 -05:00
Wenxuan Tan b392e6a874 Add torch.compile for all small ops (#432) 2025-06-06 21:10:42 -07:00
Zhang PeiyuanandWill Lin 0991003905 [bugfix] [misc] fix denoising stage init; rename distributed env function; fix logging. (#481)
Co-authored-by: Will Lin <wlsaidhi@gmail.com>
2025-06-06 20:01:10 -07:00
Zhang PeiyuanandWill Lin 8f8ce6d9e1 [bugfix] [training] Add negative prompt to preprocessing and validation (#479)
Co-authored-by: Will Lin <wlsaidhi@gmail.com>
2025-06-06 11:21:21 -07:00
KevinzzandBrianChen1129 e3d0cbe185 [STA] Implement mask search for V1's Wan2.1 (#415)
Co-authored-by: BrianChen1129 <yongqich@umich.edu>
2025-06-05 15:03:31 -04:00
eeecho d5ec468d43 [V0] [Distill] support distill in V0 for wan (#444) 2025-06-04 16:20:14 -07:00
Yongqi Chen e55fa6e5dc [Preprocess] I2V dataset (#473) 2025-06-04 16:19:20 -07:00
Wenxuan Tan 61b6ddeee1 [Issue template] Move env report to the end for readability (#476) 2025-06-04 16:18:22 -07:00
William Lin a9a000f45d [bugfix] fix bz >1 for training (#477) 2025-06-04 16:03:39 -07:00
Wenxuan Tan 66b8b8561e [LoRA] Support V1 LoRA inference (#451) 2025-06-04 16:19:29 -05:00
Wenxuan Tan 6684872616 [misc] Find unused port in distributed init (#475) 2025-06-04 14:15:12 -05:00
Wenxuan Tan 7f654e3332 [misc] Polish V1 training code (#469) 2025-06-03 22:02:03 -05:00
William Lin 8631c1b806 [Training] Support Multi-Node training with FSDP + SP (#459) 2025-06-03 17:40:22 -07:00
Wei Zhouand“BrianChen1129” 5357e12b5a Update v1 inference scripts (#467)
Co-authored-by: “BrianChen1129” <yongqich@umich.edu>
2025-06-03 02:54:12 -04:00
Kevin Lin bdfdf1dfee [Training] Add distributed checkpointing (#458) 2025-06-02 15:15:45 -07:00
Wei Zhou d156461785 Fix WanVideo (#461) 2025-06-01 23:08:11 -07:00
Yongqi Chen 6edf113838 [Misc] Bring back mask files under asset/ and update new Wan mask strategy file (#462) 2025-06-01 19:46:11 -07:00
William Lin dcf7738cbc [Misc] disable cast_forward_inputs (#460) 2025-05-31 22:06:29 -07:00
Wenxuan Tan b2ebaaf865 [Misc] Remove InferenceEngine (#455) 2025-05-31 11:12:05 -07:00
Wenxuan Tan 7768bb80f6 misc: add remote pdb for debugging workers (#456) 2025-05-30 20:44:27 -05:00
William Lin a335811869 [Training] [8/n] SP Training (#450) 2025-05-29 17:02:26 -07:00
William LinandZihang-He 357b0533fe [Training] [7/n] gradient clipping (#449)
Co-authored-by: Zihang-He <z6he@ucsd.edu>
2025-05-29 15:07:29 -07:00
William Lin 2ec3732758 [Training] [6/n]Mixed precision training (#448) 2025-05-29 14:34:28 -07:00
Wei Zhouand“BrianChen1129” a004408a93 [Training] [0/n] Add preprocessing pipeline (#442)
Co-authored-by: “BrianChen1129” <yongqich@umich.edu>
2025-05-29 14:30:09 -07:00
007e237e69 [Training] [5/n] Add single gpu training pipeline (#447)
Co-authored-by: JerryZhou54 <zhouw.jerry2017@outlook.com>
Co-authored-by: Wei Zhou <69577934+JerryZhou54@users.noreply.github.com>
Co-authored-by: Kevin Lin <42618777+kevin314@users.noreply.github.com>
Co-authored-by: “BrianChen1129” <yongqich@umich.edu>
2025-05-29 11:49:46 -07:00
Yongqi Chen 8e18dc9f71 Update STA mask strategy downloading (#445) 2025-05-28 12:49:22 -07:00
7ab32539af [Training] [1/n] Add latent datasets (#438)
Co-authored-by: Wei Zhou <wzhou322@gatech.edu>
Co-authored-by: JerryZhou54 <zhouw.jerry2017@outlook.com>
Co-authored-by: “BrianChen1129” <yongqich@umich.edu>
2025-05-28 11:01:52 -07:00
William Lin 6ef8fcb61d [Training] [4/n] add training save checkpoint (#441) 2025-05-27 17:53:53 -07:00
William Lin 016e24da63 [Training] [3/n] Add training args and dependencies (#440) 2025-05-27 17:53:39 -07:00
William Lin 85b8717545 [Training] [2/n] add bwd for all2all and all_gather (#439) 2025-05-27 14:27:54 -07:00
Wenxuan Tan 657fd745e1 misc: Trigger transformers CI for layers and attention code change (#434) 2025-05-27 11:43:23 -07:00
applesaucethebunandBrayden Zhong 12647457a7 [Misc] Small fixes to Torch code (#395)
Co-authored-by: Brayden Zhong <b8zhong@uwaterloo.ca>
2025-05-23 14:40:24 -07:00
Kevin Lin 298f74f956 Set device for encode (#420) 2025-05-23 14:19:45 -07:00
Wenxuan Tan ee8babb298 Unify env report script in issue template (#423) 2025-05-23 14:19:18 -07:00
Wenxuan Tan 60295cc03f Use version.py (#424) 2025-05-23 14:17:33 -07:00
William Lin 1572e13b6e [Tests] don't run 3.10 and 3.11 for SSIM (#427) 2025-05-23 12:59:50 -07:00
Wenxuan Tan a157275b4c Fix version number (#422) 2025-05-22 12:37:34 -07:00
William Lin c4dbe7dac3 [bug] fix bs > 1 (#418) 2025-05-21 21:07:01 -07:00
Kevin Lin d39591108e Fulfill worker response on interrupt (#417) 2025-05-21 20:59:11 -07:00
William Lin ace6e971e5 [V1] Remove vLLM dependency (#413) 2025-05-18 00:57:28 -07:00
William Lin b4f6758253 [Teacache] allow None for forward_context batch when using teacache (#412) 2025-05-17 18:43:14 -07:00
William Lin 535d29b392 [Docs] Fix image (#407) 2025-05-12 14:30:05 -07:00
William Lin b4255517e0 [Docs] Add CLI docs (#406) 2025-05-12 14:16:48 -07:00
William Lin 6eeb60613f Release 0.1.0 (#405) 2025-05-12 11:54:23 -07:00
William Lin 53d2c7791f [V1] Update where num_frame rounding is done (#403) 2025-05-12 11:52:49 -07:00
William Lin 53cb693dca [V1] Docs Update (#402) 2025-05-12 11:52:09 -07:00
Kevin Lin 6f72d24876 [CLI] Default to pipeline config (#401) 2025-05-12 00:22:53 -07:00
William Lin d1459e9976 [V1] Update README (#400) 2025-05-11 22:32:03 -07:00
William Lin 59ab481eb1 release 0.0.5 (#399) 2025-05-11 16:23:11 -07:00
William Lin 0cf001986a [Docs] More docs update (#394) 2025-05-11 16:19:47 -07:00
applesaucethebunandBrayden Zhong 51956369a5 [Misc] Replace instances of time.time() with time.perf_counter() (#396)
Co-authored-by: Brayden Zhong <b8zhong@uwaterloo.ca>
2025-05-11 15:56:39 -07:00
Kevin Lin 4b0970cbbf [CI] Set volume_size required to false (#398) 2025-05-11 15:55:34 -07:00
Kevin Lin 94bf47a572 [CI] Set default disk size (#397) 2025-05-11 15:34:17 -07:00
River (Zihang He)andWill Lin 1a3ac9074b Zihang stepvideo v1 (#389)
Co-authored-by: Will Lin <wlsaidhi@gmail.com>
2025-05-11 14:41:42 -07:00
Kevin Lin fb0581d5b0 [CLI] Update cli to support new api/model config (#384) 2025-05-11 13:50:33 -07:00
Kevin Lin 6c74ab4132 [CI] Use python 3.10/3.11 for SSIM test (#392) 2025-05-10 13:48:29 -07:00
William Lin 9a91021c56 [Docs] Add collect_env.py and various docs update (#393) 2025-05-10 13:42:42 -07:00
Wei Zhou 51c94d6a73 [Misc] Small Fixes & Features (#390) 2025-05-07 15:40:49 -07:00
William Lin dba38dbc03 Release 0.0.4 (#388) 2025-05-07 02:40:42 -07:00
William Lin 2034cc3c4f [misc] Improve worker cleanup (#387) 2025-05-06 22:45:22 -07:00
William Lin c69afce2f6 [Docs] Update for V1 (#381) 2025-05-06 16:34:44 -07:00
Wei Zhou b08e758eb3 Small Fixes & Features (#378) 2025-05-06 15:52:34 -07:00
William Lin 3f3462d7ce Cleanup Teacache params (#386) 2025-05-06 15:52:05 -07:00
William Lin c9c47dd89c Add Teacache to V1 (#371) 2025-05-06 11:58:56 -07:00
William Lin 9c4ef7c2f1 [Lint] fix (#382) 2025-05-06 00:42:55 -07:00
Kevin Lin f25eb4b905 [CI] Add write permissions to build-image workflow (#379) 2025-05-05 19:21:03 -07:00
Kevin Lin 048d55ccbb [CI] Add new images for different Python versions (#377) 2025-05-05 10:28:33 -07:00
Wei Zhou a271c55fe4 Fix FSDP issues when using cpu_offload flag (#376) 2025-05-02 15:54:52 -07:00
William Lin f663ae0d8a change gradio example to use model configs (#375) 2025-05-02 10:42:03 -07:00
William Lin c0911aa3dd release 0.0.3 (#374) 2025-05-02 03:50:41 -07:00
William Lin 5f59687ae7 Fix model config for python 3.11+ (#373) 2025-05-02 03:18:56 -07:00
Wei Zhou 5adbc81cdc Refactor encoder (#370) 2025-05-02 02:41:12 -07:00
Kevin Lin f26d5c37c1 Update SSIM tests to use new API (#369) 2025-05-01 04:14:24 -07:00
Wei Zhou 6a4ef42378 [V1] Model config (#358) 2025-04-30 14:48:22 -07:00
William Lin f1098c77dc [Attn] Add SageAttention Backend (#366) 2025-04-28 12:40:07 -06:00
William Lin 0405b618f8 [Docs] Docs for design and adding new pipeline (#363) 2025-04-24 01:04:10 -07:00
Kevin Lin eac79b753f [V1] Worker improvements/cleanup (#361) 2025-04-22 00:32:43 -07:00
William Lin 4d58cf20d0 chore: Release FastVideo 0.0.2 and update python requirements (#360) 2025-04-21 14:10:48 -07:00
Kevin LinandWill Lin 52c93ecc9d [V1] Gradio demo with new API (#357)
Co-authored-by: Will Lin <wlsaidhi@gmail.com>
2025-04-19 18:14:24 -07:00
William Lin 42d63166ac [V1] Process aware logging; improve logging msg (#356) 2025-04-19 15:03:29 -07:00
William Lin 6db20345a2 [V1] Worker cleanup; Logging clean up; enables isort again (#355) 2025-04-18 19:26:47 -07:00
William Lin ad27ea596c [sta] release 0.0.4 (#354) 2025-04-18 14:54:40 -07:00
William Lin 9aadb4bf8c [1/n] [v1] Add Worker abstractions for User API (#336) 2025-04-18 14:38:46 -07:00
Kevin Lin bd941df271 [Docs] Fix developer guide images (#353) 2025-04-17 22:32:19 -07:00
Yongqi Chen 8a73876d3b add STA to Wan v1 (#349) 2025-04-17 16:35:19 -07:00
Kevin Lin 1483a1138a [CLI] Fix duplicate --num-gpus (#352) 2025-04-17 13:01:48 -07:00
Wei Zhou 5e243d8292 Default to using original WanVAE's encoding/decoding algorithm (#351) 2025-04-17 13:00:25 -07:00
Kevin Lin b0c66d3200 [CI] Docker image improvements (#350) 2025-04-17 12:27:10 -07:00
Wei ZhouandWill Lin c86da2c736 [core] Pipeline config (#343)
Co-authored-by: Will Lin <wlsaidhi@gmail.com>
2025-04-15 15:43:10 -07:00
Kevin Lin bae2a19dcf [CI] Add manual trigger to sta-publish and fastvideo-publish (#346) 2025-04-15 15:39:11 -07:00
Kevin Lin 057686f59d [CI] Free up runner disk for sta-publish (#345) 2025-04-15 15:27:09 -07:00
William Lin 67da56628b [STA] Sta release 0.0.3 (#344) 2025-04-15 13:06:03 -07:00
Zhang PeiyuanandSolitaryThinker 2325adffa2 Add STA to V1 (#312)
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com>
2025-04-15 02:39:58 -07:00
Kevin Lin 20cf836ef1 [CI] Support custom Docker image (#342) 2025-04-14 20:08:53 -07:00
William Lin 2bf69b6f92 [Docs] Initial examples setup and more docs (#332) 2025-04-14 16:53:57 -07:00
William Lin 13583f5ffb [Model] Remove RMSNorm's forward_native hardcode from Wan (#339) 2025-04-14 16:46:43 -07:00
William LinandJerryZhou54 008ee2099a V1 wan rebased (#335)
Co-authored-by: JerryZhou54 <zhouw.jerry2017@outlook.com>
2025-04-11 15:34:54 -07:00
Kevin Lin 137f61f2fe Port tests to v1 (#333) 2025-04-11 01:40:01 -07:00
William Lin ccb262974e [Docs] Add dev guide and doc building CI (#330) 2025-04-09 13:09:00 -07:00
Kevin Lin 30966e3bc9 [CI] Set allowedCudaVersions (#329) 2025-04-09 10:16:05 -07:00
William Lin 7b4272d6b7 [Docs] Fix doc lint (#325) 2025-04-09 10:14:53 -07:00
William Linandkevin314 15553f7706 [CI] Use pre-commit to run linter (#321)
Co-authored-by: kevin314 <kevin.lin.cs1@gmail.com>
2025-04-08 11:38:47 -07:00
William LinandPorridgeSwim 60eeea50bb [Docs] Initial Docs Build (#322)
Co-authored-by: PorridgeSwim <yz3883@columbia.edu>
2025-04-08 11:38:36 -07:00
Kevin Linandkevin314 927b3a40b9 [CI] Add manual triggers for PR workflow (#320)
Co-authored-by: kevin314 <kevin.lin.cs1@gmail.com>
2025-04-07 14:25:39 -07:00
William Linandkevin314 c64f826ae2 Add torch sdpa backend to ssim test (#316)
Co-authored-by: kevin314 <kevin.lin.cs1@gmail.com>
2025-04-07 13:50:46 -07:00
Zhang Peiyuan 55c1040f0b Fix sdpa (#315) 2025-04-06 12:00:34 -07:00
Kevin Linandkevin314 8a3e7aa761 Add ssim test (#314)
Co-authored-by: kevin314 <kevin.lin.cs1@gmail.com>
2025-04-05 17:35:45 -07:00
You ZhouandWill Lin 4324c1c21d refactor the env setup and install of fastvideo (#309)
Co-authored-by: Will Lin <wlsaidhi@gmail.com>
2025-04-04 16:33:03 -07:00
Kevin Linandkevin314 2c342ee37f [CI] Add test workflow improvements (#311)
Co-authored-by: kevin314 <kevin.lin.cs1@gmail.com>
2025-04-02 23:14:05 -07:00
Kevin Linandkevin314 708201f531 Set up text encoder tests to work with pytest and Github Actions (#302)
Co-authored-by: kevin314 <kevin.lin.cs1@gmail.com>
2025-04-01 17:56:21 -07:00
1fee098f10 [do not merge] Rebased refactor (#270)
Signed-off-by: <>
Co-authored-by: William Lin <SolitaryThinker@users.noreply.github.com>
Co-authored-by: Will Lin <wlsaidhi@gmail.com>
Co-authored-by: Zhou, Wei <wzhou322@gatech.edu>
Co-authored-by: Kevin Lin <42618777+kevin314@users.noreply.github.com>
Co-authored-by: kevin314 <kevin.lin.cs1@gmail.com>
Co-authored-by: JerryZhou54 <69577934+JerryZhou54@users.noreply.github.com>
Co-authored-by: Yongqi Chen <144848849+BrianChen1129@users.noreply.github.com>
Co-authored-by: Peiyuan Zhang <m2deng@ucsd.edu>
2025-03-29 17:43:47 -05:00
You Zhou 8a77cf22c9 Establish cicd workflow to build and publish FastVideo and STA Kernel (#227) 2025-03-11 20:27:36 -07:00
Yongqi ChenandPeiyuan Zhang d869d90d12 fix training mask strategy issue (#248)
Co-authored-by: Peiyuan Zhang <a1286225768@gmail.com>
2025-03-05 20:00:16 -08:00
Zhang Peiyuan 554ee17de5 [BUG] update cfg bug? (#223) 2025-02-27 16:02:44 -08:00
Yongqi ChenandPeiyuan Zhang 0be4fc62c9 fix train/distill issue (#215)
Co-authored-by: Peiyuan Zhang <a1286225768@gmail.com>
2025-02-25 08:11:17 -08:00
Yongqi ChenandPeiyuan Zhang 1e08893546 Added multi-GPU support for Hunyuan STA (#211)
Co-authored-by: Peiyuan Zhang <a1286225768@gmail.com>
2025-02-21 14:16:28 -08:00
Zhang Peiyuan 09ab452610 Update STA README.md (#206) 2025-02-20 22:26:26 -08:00
Yongqi ChenandPeiyuan Zhang e768b5ec5b Update readme (#202)
Co-authored-by: Peiyuan Zhang <a1286225768@gmail.com>
2025-02-20 13:16:25 -08:00
Zhang Peiyuan 59ec42f40e [FIX] Make STA optinal (#204) 2025-02-20 13:09:50 -08:00
rlsu9 5ae5b247b3 [FIX] fix isort format (#203) 2025-02-20 12:20:20 -08:00
ead6c62be4 [Feat] Add STA for StepVideo (#200)
Co-authored-by: rlsu9 <r3su@ucsd.edu>
Co-authored-by: BrianChen1129 <yongqich@umich.edu>
2025-02-20 11:33:58 -08:00
Yongqi ChenandPeiyuan Zhang 6805eaa06c [bug]: fix ori hunyuan inference issue (#199)
Co-authored-by: Peiyuan Zhang <a1286225768@gmail.com>
2025-02-19 14:18:15 -08:00
Zhang Peiyuan c39a15551c Update typo (#198) 2025-02-18 19:34:45 -08:00
Zhang Peiyuan e6dda263b0 Update Cite (#195) 2025-02-18 21:01:46 -05:00
Zhang Peiyuan f9482d113c update env (#194) 2025-02-18 20:45:08 -05:00
rlsu9 a3ec969397 [feat]: fix readme demo and add video to readme (#191) 2025-02-18 17:46:32 -05:00
Yongqi ChenandPeiyuan Zhang 76a12cc8a1 Infer sta tea with torch.compile (#190)
Co-authored-by: Peiyuan Zhang <a1286225768@gmail.com>
2025-02-18 11:29:36 -08:00
Yongqi ChenandPeiyuan Zhang ac490399c6 fix kernel issue (#185)
Co-authored-by: Peiyuan Zhang <a1286225768@gmail.com>
2025-02-16 21:35:56 -08:00
Yongqi ChenandPeiyuan Zhang 9ea39cee57 Add STA and teacache forward (#184)
Co-authored-by: Peiyuan Zhang <a1286225768@gmail.com>
2025-02-15 16:22:01 -08:00
Zhang Peiyuanandrlsu9 52e6e612a2 add sliding tile attn (#182)
Co-authored-by: rlsu9 <r3su@ucsd.edu>
2025-02-15 15:44:34 -08:00
Hangliang Ding 9aebc4ada1 Create config.yml (#152) 2025-01-20 20:11:01 -08:00
Yongqi Chen b53cf7425c Lora README update (#155) 2025-01-18 12:30:53 -08:00
Zhang Peiyuan d9ce056901 [typo] 2025-01-13 20:05:57 -08:00
Brian Chen 218449c54d adding hunyuan hf (support lora finetuning); unified hunyuan hf inference with quantization (#135) 2025-01-13 19:47:42 -08:00
Hangliang Ding 221958bcde Update README.md (#131) 2025-01-08 09:02:40 -08:00
Yuzhou Nieand“Peiyuan Zhang” 4a1f1e35bb add parallel for vae decoding (#134)
Co-authored-by: “Peiyuan Zhang” <a1286225768@gmail.com>
2025-01-07 17:14:21 -08:00
rlsu9 e0e05f97f2 [feat]: Add tests for FastVideo (#127) 2025-01-06 12:27:39 -08:00
Zhang Peiyuan dd75ee8509 [Fix] Save CK, Dataset bug fix (#125) 2024-12-31 22:19:10 -08:00
rlsu9 0aed1868df [feat]: Add format auto fixer to main branch (#124) 2024-12-31 15:23:17 -08:00
Hangliang Ding d467c7cd35 [Minor] Adding issue template. (#114) 2024-12-25 21:50:57 -08:00
Zhang Peiyuanandrlsu9 88b2583c2c [feat]:Single 4090 inference for fasthunyuan (#104)
Co-authored-by: rlsu9 <r3su@ucsd.edu>
2024-12-25 12:40:16 -08:00
rlsu9 a730e43d5f Update README.md layout 2024-12-19 13:36:43 -08:00
Brian Chen edf116fa46 fix lora checkpoint saving issue (#97) 2024-12-19 08:42:59 -08:00
Luis Catacora de3cefb5e5 Add Replicate demo and API (#93) 2024-12-18 19:56:09 -08:00
Hangliang Ding e087e85e09 Adding Development plan 2024-12-18 16:46:14 +08:00
Your Name e1b998b6ef merge 2024-12-17 12:48:16 -08:00
rlsu9 fb49c93dbc Update README.md 2024-12-17 12:29:03 -08:00
rlsu9 172f4802b4 Update README.md 2024-12-17 12:28:08 -08:00
rlsu9 24e57fafc9 Update README.md 2024-12-17 12:26:17 -08:00
Your Name 6debd46482 merge docs 2024-12-17 12:20:42 -08:00
rlsu9 f7dc36f7ea Update README.md 2024-12-17 12:13:33 -08:00
Brian Chen a0fb954f56 Update README.md
fix typo
2024-12-17 15:09:10 -05:00
rlsu9 053106922c Update README.md 2024-12-17 11:43:49 -08:00
a57122c519 Rlsu lora readme (#86)
Co-authored-by: rlsu9 <r3su@ucsd.edu>
Co-authored-by: rlsu9 <147024991+rlsu9@users.noreply.github.com>
2024-12-17 11:37:07 -08:00
Zhang Peiyuanandrlsu9 b393570e45 Update README (#85)
Co-authored-by: rlsu9 <r3su@ucsd.edu>
2024-12-16 17:06:14 -08:00
Zhang Peiyuanandrlsu9 285635e8c0 Clean up (#84)
Co-authored-by: rlsu9 <r3su@ucsd.edu>
2024-12-15 20:33:11 -08:00
Zhang Peiyuanandrlsu9 58cfd71b5e Cleanup
Co-authored-by: rlsu9 <r3su@ucsd.edu>
2024-12-15 17:03:29 -08:00
Hangliang Dingandrlsu9 3bf892b6ab update release readme (#81)
Co-authored-by: rlsu9 <r3su@ucsd.edu>
2024-12-15 22:24:13 +08:00
Zhang Peiyuan 85639d1101 [feat] add hunyuan adv (#79) 2024-12-13 11:52:57 -08:00
Zhang Peiyuanandforeverpiano 6ab2263f3a [Feat] Add HunyuanVideo (#78)
Co-authored-by: foreverpiano <pianoqwz@qq.com>
2024-12-12 14:14:09 -08:00
Zhang Peiyuanandrlsu9 b421c2e183 Cleanup (#77)
Co-authored-by: rlsu9 <r3su@ucsd.edu>
2024-12-12 14:04:02 -08:00
Zhang Peiyuan de1e8d868e Cleanup (#75) 2024-12-06 20:56:06 -08:00
Brian ChenandBrianChenn1129 98b92be25e add web demo (#73)
Co-authored-by: BrianChenn1129 <yonqgich@umich>
2024-12-06 09:47:22 -08:00
Zhang Peiyuanandforeverpiano 8d41d505fe [cleanup] (#72)
Co-authored-by: foreverpiano <pianoqwz@gmail.com>
2024-12-05 21:02:10 -08:00
Brian Chen 8cfdf58a17 [Feat] HF Lora
yongqich@umich.edu
2024-12-05 18:50:53 -08:00
Zhang Peiyuan cf15594055 [Feat & Debug] fix uncond; multi guidance validaiton; multiphase schedule; linear range (#64)
typo

update

update

typo

[Debug] Typo (#65)

debug gradient accumulation loss

update gitignore

typo

typo

update scripts

update experiment 10

new script

typo

update

update

update

update

wandb offline and dir

runlong

update

update

update

update

update

update

update

fix finetune code bug

update

update

update

add l2

update
2024-11-30 16:43:24 -08:00
Zhang Peiyuan ce95c2df29 [Feat] EMA Distill; Distributed validation (#63)
update
2024-11-29 21:46:40 -08:00
Zhang Peiyuan d417e4c7c4 [Feat] Refactor GAN; State saving & Resume; Experiments script (#59)
typo

add upload command

add

add ema_transform

add ema transformer

remove harcode

ok

ema

remove hardcode

update env

update script; no sp

revert to sp=4, sp bs=2, full shard

add aws efo env

distributed validation

readme

distributed validation

add gupload

linear range; fix uncond; multi guidance validation

add script
2024-11-28 21:09:31 -08:00
Zhang Peiyuan 2a70d05b4f [Feat] Training precision (#57) 2024-11-27 16:25:07 -08:00
Zhang Peiyuan 03187fd83a [Feat][Debug] linear quadratic distill; HF precision bug (#56) 2024-11-27 15:13:20 -08:00
Zhang Peiyuan 8a128ad815 [Fix] Squeeze bug (#55) 2024-11-26 13:29:03 -08:00
Zhang Peiyuan 13f665e455 [Feat] PCM Distill; Refactor FM logit to be compatible with all SD3/Flux scheduler. (#54) 2024-11-26 12:26:24 -08:00
rlsu9andRunlong 94ba0ab6ea [feat]: Add batchy data preprocess (#53)
Co-authored-by:Runlong <rlsu9@ucsd.edu>
2024-11-25 21:52:36 -08:00
rlsu9andrunlong 5a5d0ef1a0 [feat]: Add Image-Video Mixture training to main repo (#50)
Co-authored-by: runlong <r3su@ucsd.edu>
2024-11-24 14:36:45 -08:00
Zhang Peiyuan 45e4adca4d [Fix]: Resolve config bug and seed (#51)
Co-authored-by: Peiyuan Zhang <a1286225768@gmail.com>
2024-11-15 20:29:48 -08:00
Zhang Peiyuanandforeverpiano 7106eadffc [feat]: Add LADD (#45)
Co-authored-by: Peiyuan Zhang <a1286225768@gmail.com>
Co-authored-by: foreverpiano <pianoqwz@gmail.com>
2024-11-15 19:28:41 -08:00
Yongqi ChenandPeiyuan Zhang 2f3a8661bf [feat]: add lr scheduler; precision bug fix; add naive dataloader resume (#49)
Co-authored-by: Peiyuan Zhang <a1286225768@gmail.com>
2024-11-14 22:57:38 -05:00
Yongqi ChenandPeiyuan Zhang 6d0082c1a9 [Feat] Lora resume (#48)
Co-authored-by: Peiyuan Zhang <a1286225768@gmail.com>
2024-11-13 01:30:58 -05:00
Zhang PeiyuanandYongqi Chen 3d9189571a [feat]: Add lora (#47)
Co-authored-by: Yongqi Chen <144848849+BrianChen1129@users.noreply.github.com>
2024-11-11 13:12:14 -08:00
Zhang Peiyuan b042e321a1 [Refactor] Switch to FSDP (#42) 2024-11-09 15:46:25 -08:00
rlsu9 44bda9f8a3 [feat]: Add adaptive fps dataloader and remove redundant code (#41)
No checkout mochi
2024-11-06 19:52:48 -08:00
rlsu9 035ba5f5cf [feat]: Add vae encoder embedded generator to main (#30) 2024-11-06 11:58:15 -08:00
Zhang Peiyuan 52ba538e8e [feat]: Add validation logging with SP (#36) 2024-11-05 14:46:20 -08:00
Zhang Peiyuanandforeverpiano f0bc297260 [Feat] Sequence Parallel (#31)
Co-authored-by: foreverpiano <pianoqwz@gmail.com>
2024-11-05 08:13:01 -08:00
rlsu9 7413b1dd5f [feat]: update data preprocess 2024-10-31 00:13:05 +00:00
Peiyuan Zhang 8ec82cbc37 Generate synthetic dataset
Delete Open Sora Plan Modeling

amend log validation

Change name to fast video

Remove OSP modeling

clean up name changing

remove files

 Deleted unnecessary files

commit first

commit first

training

ok

Update generate_synthetic.sh and deepspeed_zero2_config.yaml

14

debug

overfitting ....

debugging ..

Debug successful!

Add zero3

OK

update optimizer

load

small bug

random seed args

sp enable & still has bug

rename

update inference sp code / can run / still has bug / don't output normal mp4

switch to deepspeed dummyoptim

fix some bugs; still output green

SP inference done!

fix typos in readme and latent dataset debug file
2024-10-28 00:44:02 +00:00
Peiyuan Zhang 54e74aec3f Mochi Inferenfce & Diffusers
typo

Original pipeline
2024-10-27 22:57:14 +00:00
Peiyuan Zhang c1c276b616 Refactor OpenSora sample_t2v.py and update download_hf 2024-10-26 21:59:22 +00:00
Peiyuan Zhang 8e5fa4d383 remove vae loss 2024-10-26 19:20:04 +00:00
Peiyuan Zhang 06ff1c912f Remove files in causalvae 2024-10-26 19:06:46 +00:00
Peiyuan Zhang 857f5df51b normalize 255; vae reconstruct 2024-10-26 18:57:07 +00:00
Peiyuan Zhang 18c5ca131d Add mochi download & Change output dir 2024-10-26 18:08:50 +00:00
Peiyuan Zhang c16625242e Delete merge_data.txt 2024-10-26 18:00:16 +00:00
Peiyuan Zhang fcc45701c9 Remove unused adaptor files 2024-10-26 17:59:04 +00:00
Peiyuan Zhang 6c87e003aa Remove unused arguments in train_t2v_diffusers.py 2024-10-26 17:54:43 +00:00
Peiyuan Zhang 23181aac96 Update T5Base 2024-10-26 17:53:51 +00:00
Peiyuan Zhang 26b65a8baf Update PyTorch installation command 2024-10-26 17:36:23 +00:00
Peiyuan Zhang 25498a9d85 Update model path and cache directory 2024-10-25 10:21:31 +00:00
Peiyuan Zhang afb3ac9fc1 Update t2v_debug_multi.sh with video_length_tolerance_range and dataloader_num_workers 2024-10-25 09:58:41 +00:00
Peiyuan Zhang 3d385600e9 update version 2024-10-25 09:22:16 +00:00
Peiyuan Zhang 5680039dbe Add environment setup and training instructions to README.md
Update dependencies; Setup code for debugging

Delete unused files and code

Remove all npu code

Update torchvision imports

Update EMA model and t2v_debug.sh script

Delete npu related stuff and remove inpaint module

Remove compress kv

Fix warning with dataset handling and model loading

Update PyTorch index URLs and video length tolerance range

Remove UDIT and inpaint

fix typo for dataset download

 include pretrained open-sora

Add pretrained model for OpenSoraT2V-ROPE-L

Update max height and width for video processing
2024-10-24 17:47:34 +00:00
jzhang38 2c1eb3b0e2 Remove NPU related code and update training process 2024-10-24 02:36:18 +00:00
jzhang38 5e2e3ab06a Remove unused scripts and update TODO list 2024-10-24 02:27:03 +00:00
jzhang38 f5ca624aff Delete unnecessary files 2024-10-24 02:23:33 +00:00
d1ea86d351 Initial Commit based on Open-Sora-Plan-V1.2
Former-commit-id: 7a3dccf2c738eaf535e62996211c8bc5f3de5683

Update README.md

Former-commit-id: aee63a3a8422e029f1550e47825ca4beb916874b

Update README.md

Former-commit-id: fb46c80c1a2663c09849472e93b2da6cd068a98c

Update README.md

Former-commit-id: 5fd1e6afe1277ccffdfeaf507477d944a6a35f03

Update README.md

Former-commit-id: 1cd7d8b5df34ab9f61d9346b2647b643f78a8f5d

Update README.md

Former-commit-id: e27cc5d49a19e9580ba1a29d948051b918aede74

Update README.md

Former-commit-id: 0ff9314bddef1dfbbf429726ed4240b823b221b3

Update README.md

Former-commit-id: f295f4623448dd4ffa9be747a5cc6733cf48c663

Create Contribution_Guidelines.md

Former-commit-id: 897dbd63dc6f53095e1dda3f10a86dd40e4bd263

Update README.md

Former-commit-id: c652acd1fc5a003b4d6eefcfb83dfb310f5b5e81

Rename data.md to Data.md

Former-commit-id: 27484f7a2d8634bd1cd9781a2f10afa7f304f5cb

Update README.md

Former-commit-id: 351be764d8648ac60cf170c3fe4e6ed9c544c90e

reformat code

Former-commit-id: ce23b280fbb419c08347ebcbf1ed4075bdf9af60

Update requirements.txt

As in previous step,the three packages have been installed, these packages should be deleted here because it can not be installed by " pip install -r requirements.txt" command and will abort the command

Former-commit-id: 0655fa28cf8bb761f1a3d2def62f44197b257b2c

Update README.md

Former-commit-id: 1dd9c945be2a170d87d568bb00e1ec30e7be592b

Update README.md

Former-commit-id: 6ecaba8e03d7c7f99c068a57c923f11c5456a8c9

Update Contribution_Guidelines.md

Former-commit-id: dc87e459ca5a8a090800331111118acb65214f65

Update README.md

Former-commit-id: be7ecc3d56f7289052fc66d535038e7111f7ac84

Update README.md

Former-commit-id: 164e76cf496b1157eebea299ed58ca6c1799d70d

Update README.md

Former-commit-id: 3e3d674134498512b1ef1b0dd72d6e19afd6ef65

[feat]: frame_interpolation

Former-commit-id: 711fc4bba80d5f051f712e3460e2c0685117cf68

Update README.md

Former-commit-id: 1436d8feea7f82d3d747c632a68c47b3ea74258f

fix_readme_requirements

Former-commit-id: eb5a5c69d12cc412c12a4fea7f00baafd02701ca

Update README.md

Former-commit-id: c5a79c7e993d9e6a77c6c70efa1c0ac2bb3a063b

add latte

Former-commit-id: 01ba10e5d65c9d2d957472fbe7958e1b07a971e9

Update README.md

Former-commit-id: 437bc5c68b5c7a85865ad97d544c4818be45cb52

first commit

Former-commit-id: ec3481bc35a8120d519082dc76585428f0896470

[refactor] reformat videogpt, support training videogpt on accelerate

Former-commit-id: 75385ed47fe60008b61b1bcf89f61ea696d095bc

[refactor] adapt to old training code

Former-commit-id: 358d362ecbc100b21ea02895cb0426cf5b386a72

Delete LICENSE.txt

Former-commit-id: b10d8b2c6620e3a8412527bd03ea0adc26c981a6

Create LICENSE

Former-commit-id: 369fe085e91e55b9be6165da4a1eb7f9c0cf5a7f

Update README.md

Former-commit-id: f1aaeca19bc9e4304961c8c903f9bf03cb42fd9f

Update README.md

Former-commit-id: 0f2bdd2bcd819d528f2a41a96ba93f5c3c04dc8c

Update README.md

Former-commit-id: 10ee64c122f28efea39fc1a4542d278683a58ce1

option to use rebased linear attention

Former-commit-id: f1b39b34df5fbdef14ef3f6a0e6d0972a8570445

support for ring attention

Former-commit-id: 0533c73b99217e88da121bd0f68e0949bc88016e

add vae and fix bug in dit

Former-commit-id: 2dd4254c22ce671d5cd5e7d22a2386b7660fbdc4

Update README.md

Former-commit-id: 898d35f15ce0fd124cd48078f0a8af2764bf55d1

support latte training

Former-commit-id: d471a9414842f422d1200c2fd3f3ddfcf9e97fd6

Update README.md

Former-commit-id: 9fd88eb8ac5122b0999ccb0071f195aaeb0979a0

Update README.md

Former-commit-id: 398cfcd7b19cf1ba827eb49fc11223f72fa93a85

Update README.md

Former-commit-id: 673659495b7503ba402fbcb6c46204499761b4cf

[fix] fix a bug when using ucf101_stride4x4x4

Former-commit-id: 9911403c688321db731450d9383bc02923a4be2d

[refactor] remove test.ipynb

Former-commit-id: e73c3a9e0d41e51a0fd755e57739424213a76312

[fix] fix a bug when using ucf101_stride4x4x4

Former-commit-id: f541e723d65ea44ae4560c68908123c7e6781bbc

[refactor] rename

Former-commit-id: ec317605d24223527b8c2d4b4d2ee876f86abbaf

update

add RGT

Former-commit-id: a491111e48cec588095381eb70ffca94600ca4f7

add sample script

Former-commit-id: 9ce847651bb6dba28c51ce5807e222ae2902a78a

[refactor] configuration default values

Former-commit-id: d7ca7685d6a8c6d015082dda241bc98ab156770d

[refactor] added more methods in abstract classes

Former-commit-id: a535e3a06eb766bebe7baa3d9df64b04d04d40e6

Update README.md

Former-commit-id: 59a595a1e31b156500a944f244afec40988e872d

[fix]: disable gradient computation to save GPU memory when reconstructing a video

Former-commit-id: b7bf763e7f8800959ed82bb89f41684db216dd8e

support accelerate training

Former-commit-id: b32671c9991b9a9db66cb148822ccb3e8be7dc11

Update train.sh

Former-commit-id: 6c4957d715eaa489691be21dd0abdb9309660e0d

Update train.sh

Former-commit-id: 66aa3709f80d89a30d0a57c5e033ce1ae46f998f

Update README.md

Former-commit-id: 604a29d1de43e0fcb90ada4dd97f5f080e37caea

Update README.md

Former-commit-id: d097e40998a3396f74c2bfbcf26d6ace8080d0b6

Update README.md

Former-commit-id: 1a7178a61547c33a744d032302fb76ee0a3c6671

denorm fun

Former-commit-id: 05707a07b14295f5e1932d0c10bb84a20a0cd328

Update README.md

Former-commit-id: b3f6bb7f252505efb0ffd8195003c79206fac313

refine readme.

Former-commit-id: 391d9511b72aa9c09c7ad8992ae5df8c89c88c63

complement emoji.

Former-commit-id: 50679cc482d8ecf4e3083c0f5a791f195a58a855

Update README.md

Former-commit-id: 29f24a819fe2d4593c48afe6ca990659784bd776

[feat]: add sit model

Former-commit-id: 2eda854d6c25254b71d0105a3ed092953a9d6f68

fix: remove replicated diffusion module in sit

Former-commit-id: 997cb049185421d303b14d59ab6c0603f1d1a66f

fix: rename sit model filename

Former-commit-id: f3b2cd7dcf644f680616eee0571db90478fede23

feat: Incorporating SiT sample

Former-commit-id: 91e2931e412f8c96ef748cb2322ee956309fb6aa

feat: update sample

Former-commit-id: e7dc492b9b4764db28aa31cc1eede914052b4ca6

attn_mask with bf16

Former-commit-id: d9667a8c15a5344e3c777197fd6badec15dcddbd

Update README.md

Former-commit-id: a1cc0b26a3b156d7708e8df0cfe42aad33b1c946

Update README.md

Former-commit-id: 4179ed075e68d48094655c38dbde784836fd6e0f

Update README.md

Former-commit-id: 36e69275598b0cff38fdc5006926e422cda069b3

[fix]: disable gradient computation to save GPU memory when reconstructing a video

Former-commit-id: 06da17790b53cf593a0c264dda17c76f68d2a541

Update README.md

Former-commit-id: 8e9658ed993380abc43b518fcbac21a02d8ba9ba

Update README.md

Former-commit-id: 8b553a4477c64b5452f8424ca6987c0fb8c111ce

Update README.md

Former-commit-id: 0898e4b917f5e245c4394118511bcb7639973817

Update README.md

Former-commit-id: 4a893ff6177d59219b6755c4a89764ef54fc7f62

Update README.md

Former-commit-id: 89f9fc8eed3ca8cc2434ec227acbb845bfbcff82

[refactor] rename train_videogpt.sh

Former-commit-id: e2ea2caacaf5be046f48cf50a53ccafa7a3cc351

[docs] Modify README.md and add VQVAE documentation.

Former-commit-id: a5bfe70f8a8998db969d9c4938fda123aa153bc0

fixed dit forward

Former-commit-id: 4cd537fab12231f7471107762b02b36217ca42d8

videogpt inference

Former-commit-id: c5d617450a8271ad3b8a5e8f76ee2d77a822e31b

Update README.md

Former-commit-id: 4d83f1feab7d6872aca1c6b41be95950ec3cf465

Update README.md

Former-commit-id: da5507e570a27becab754ba24a0671d669b7e08e

Update README.md

Former-commit-id: 4a04e71636dabccbef446623674f3a514395b7f9

Update README.md

Former-commit-id: 2a286e5aab48aa68e7d3f04df23ec1cdfe71c6de

Update README.md

Former-commit-id: f4c4b3b4b1ce3c1d0b56ba5a109cd6cc57da900d

Update README.md

Former-commit-id: 7af6b82b29c265e53e9c48477083b4cff0e0a3d3

Update README.md

Former-commit-id: 7bc544af55c0a987d0df426848cb06cb330fd52b

Update README.md

Former-commit-id: 2a68a1aa399a53f58246df4f9f64dfdcdd130bcf

[docs] Modify README.md to fix translation error in name "CloseGPT"

Former-commit-id: dd36043df4b0408e592a82f9046f6d654a5375c3

[docs] Modify README.md to fix other grammatical errors

Former-commit-id: f0de27fcc38bb4dab0eeafac225a2924080a3218

Update README.md

Former-commit-id: a881ff5198e1125032418f4f02fd95d2179ffb3a

Update README.md

Former-commit-id: 8a191abc1903ad5998fe83149c9c9ef5c8226b9e

Update README.md

Former-commit-id: 3b4eb14dcd8862b8251b72f30cba523a5dc1def8

Update README.md

Former-commit-id: e458366e47d4b13737a71bbf866e20ec10961ade

init pos embed

Former-commit-id: 28f8076136ec214c66039ead7547c9b25cfe7bfc

Update README.md

Former-commit-id: a1d1f07c83eb0c03d93b023b5c9162a6a3c26cd9

Update README.md

Former-commit-id: deddba1f03bf697fb1252a0ce4fdb5df5bcae8de

Update README.md

Former-commit-id: 076e6e70e717888f24265d624c6a7a8ffa2f5d1b

Update README.md

Former-commit-id: 7cd398721e556db6199f1d06c7a0d1de2a9720dd

[updata] updata video super resolution

updata video super resolution

Former-commit-id: 82397545859791d112d6f0569e33fe23437563df

train script

Former-commit-id: 3fd1ccf099c2deed20af50bf9556e2fc3b7a4382

Delete opensora/models/super_resolution/placeholder

Former-commit-id: 8dd5a6a3c85dc28989ff95e6ad41be7fdb0f0811

Update Data.md

Former-commit-id: 9531e33666f842125b331cb266eb26a3a03a4009

Update Data.md

Former-commit-id: 86be6b2bf5a9be1030bb3af661a2ec85e710a5c3

commit

Former-commit-id: 81359cf67248b6c594ff79f121a8a38558c1cc70

Update README.md

Former-commit-id: c4d7aafa3ceb6e4d48bf85b402ae2b2ce9394e53

Update README.md

Former-commit-id: bca7070fac79d5ae5c37830cd55ca1e81a289197

Update README.md

Former-commit-id: b37da2202e730a4930a194a80a703b591a535adf

Update __init__.py

Former-commit-id: 03a45aaf60d294b7a68d8fc44132d3d671e4472b

Update README.md

Former-commit-id: 97fe914aaa2b9bd7c2ff5d4239381631c47e8356

Update README.md

Former-commit-id: c6b1ea8f1ef01096aa8c3e5272bc9fb255b5532b

Delete scripts/train_vqvae.sh

Former-commit-id: 952a4ce3a3b0da159d599ff3a0a052265f48c2dd

update script

Former-commit-id: 43f539b68af150c45040d79918a84de1dce47c33

feature dataset

Former-commit-id: 9683caae8648735552270b871de22fc540ba2b55

update train.py

Former-commit-id: 844173711342f2dcc021294e87971ca39854bfe4

del use_fp16

Former-commit-id: 5424c0112e04b56d1df30ac76cbc104e848406a3

update sample.py

Former-commit-id: 10d8621051d1510270c6a55abe686eac86297170

fix extract feature

Former-commit-id: d4dce5ec1aaa058e452a41f15cd35fe1713f0b78

Update README.md

Former-commit-id: 70f6ebf421f0830a51dcc2838c2c3783cfb67376

[Updata] updata the video super resolution

updata the run.py and add README.md

Former-commit-id: 1e6d6ace7a0963149e228813801210b022aceacf

add CI for docker autobuild

Former-commit-id: c4e2d5ff1dce8662a42f1ed6a7786855d40173f7

[refactor]: add docker development support

Former-commit-id: fded3baff7915c0cfc9cec3765c271259359831c

update docker scripts and configs

Former-commit-id: 4567e6368f1bf413d8f6319a97a852b8289ca770

Update README.md

Former-commit-id: 99b0066895af072a8887737f2b603ec7fc893ccd

extract script

Former-commit-id: 4139f2f68ae09695302ce49afdb7318df83cabfd

[docs]:update the eval_code for calculate the FVD, clip_score, ssim, lpips, psnr

Former-commit-id: 1ee226c338126fbdc5178020b12271b980a7ce0c

[docs]:update the eval_code for calculate the FVD, clip_score, ssim, lpips, psnr

Former-commit-id: b504623d7ec2ac0b508530d21ffc88978f8a4f71

[docs]: update the evaluation

Former-commit-id: 02e3789b74daa036fbbff6d13dfc7e98372260ed

Update latte.py

revise the xformers inputs

Former-commit-id: 60db66c912192c3fbafe55ebe16157514b884e5b

Fix the memory error bug when loading metadata pickle file of vqvae dataset.

Former-commit-id: 97ab243bd12915cea7aa2deecfac4b99417f78a3

Update README.md

Former-commit-id: 8f7b41118a3f491d1d5b5789c53790c46b023191

Update pyproject.toml

Former-commit-id: 36da5d0e950625e629aebe954f89f92827397d0d

Update README.md

Former-commit-id: 826cc5e9162888bed82413088c6be662c278176b

Update README.md

Former-commit-id: 88e9f21cfc8ec775dbc9a65f740043e91e26bd91

Update README.md

Former-commit-id: 38e0e290e7a5909706c2cce2e4dd047647134fb8

[docs]:update the eval code

Former-commit-id: 937f2fc87d34ab9d4724cb95a6457b54a6bf4ebc

[docs]:update the eval code

Former-commit-id: a8c9e6cb9e39a9e929dab34077280a8d80d85573

flash-attn train and sample

Former-commit-id: e336565bd5ff2aaf4d0b4ad5f42732e89cd216b3

[refactor] add static type checking

Former-commit-id: 12c4240dc28888c61f1f79243627e7cdb445c840

Update README.md

Former-commit-id: 513883df1ad853c9a5e7457c364227261df17a83

Fix einops bug: module 'keras.backend' has no attribute 'is_tensor'

Former-commit-id: 849f8ed815295bac78c3c10d319e6731aa26ca02

[feat] Add 2D RoPE

Former-commit-id: ccae80d339e34fb23706827565b5095ae2c2f320

Update README.md

Former-commit-id: 6630c22baa2340a8b55badd61472daab7177bc63

xformers and flashattn

Former-commit-id: 002cad95253f7da36344237ce8c8410ffb517711

safe dit

Former-commit-id: f407cab7dc0e53257972d430e873f4fd77bb039d

[fix] 2d RoPE init

Former-commit-id: ff5931bae6efa94675b9ef6bc25f1b76217d128e

fixed attn_mask in flash-attn

Former-commit-id: 5e2504f429a342b75bc7a9004cee7b3f75582ee3

typo

Former-commit-id: 9e87366e906ed19726c8905afb27b5df4d4e1512

Decouple mixed_precision and gradient_accumulation to command args.

Former-commit-id: 903d473d1432eb492a87efe184d03439f643b92d

Fix mixed_precision bugs of latte.

Former-commit-id: d70a551972e04d95444d46cdbe2bfb16ad099fcb

Refactor latte.

Former-commit-id: 304534269f105578097bc16ab497636ad59fd9b2

Refactor latte.

Former-commit-id: 7408fd0ee16247afc3d4ba43b8c8e7dfb1091a97

Support deepspeed zero2 and zero2_offload for latte training.

Former-commit-id: c87e95dd357e6fa6d6fb2dcfee781bc49c039e13

Add some comments.

Former-commit-id: d5b5bd7743edde6a63dc74457bded0f8c56be1c2

Fix typos.

Former-commit-id: c10330d373466f3c1ab455bfb391569b9175bc3a

deepspeed

Former-commit-id: 6179a4b0e7d06caed4201a27679abc64615febe6

Update README.md

Former-commit-id: 582070d4f97c4a4398dcb9a6699f4ab6e980c812

Add base config.

Former-commit-id: d37c551a0421d6b73dee26060b517c46d8217e4f

train 1080p video

Former-commit-id: 74625142ef892f9e7fa459b2f98f2f7ae5b36d6a

Update README.md

Former-commit-id: 87636c1dc2275c4da77c6368a0f4425bea15dc12

Update README.md

Former-commit-id: 059d9e863958cfd679dbb8d365de6cd933e284fc

Update pyproject.toml

Former-commit-id: 5c379b6e4515ad75b89cdebbc5cd490c9a821d78

Update README.md

Former-commit-id: 7a62dec060c91b02ca556451266e6f3ef28e03e7

clean script

Former-commit-id: 02c61f63a8539a31516b92f5bb7c2d2e428ca7d7

Update README.md

Former-commit-id: 6d41cfe906f97dbed1cea8c9ed06a1f6dc84308e

Update README.md

Former-commit-id: 64d147863368aeef479ad25f2e5463fdcb76de46

Update README.md

Former-commit-id: 2808edf11005e04aa730971a786d1a4e3bca88b5

Update README.md

Former-commit-id: 1f94f2721f35670aeae8dbfb7c0a5abd06cd2aee

[feat]: Caption Refiner

Caption Refiner for Video Caption

Former-commit-id: 97c16bccb9979acf19fb91ae8d7b84ac4a3b4d97

[BUG]: Shorten the refined caption

Shorten the refined caption using GPT-3.5 summary

Former-commit-id: bb159f0d5fa928258475b2058983397ceb91af9d

Fix resolution typos.

Former-commit-id: 3ad55d28eaf13f9c8fd23912d2691965737527e9

clean dit

Former-commit-id: 117ec4196a6f2cd83da223b993b14e28f7a693f8

Update README.md

Former-commit-id: abb8ed57a29b1f66adec66d854429f36f9199540

Update README.md

Former-commit-id: c2c64665d4f281e264d87c20cc1cfd38e7e5f168

[fix]: correct attention_mode to attention-mode

Former-commit-id: 9957e2eebb2166b02600c6052fb54908f5d12bad

Update README.md

Former-commit-id: 266f9f97ea8f327fdefb2ca959c23d08b7084137

Update README.md

Former-commit-id: dd00e04220ff5232c5c4503a095e409471476a51

Update README.md

Former-commit-id: 5717aab628fceba311e284d1bdddeacc4c43c7c5

Create train_256.sh

Former-commit-id: bf62151617feb49663f8dfe0c78aa01f41e62092

Update README.md

Former-commit-id: c5eea8d3492872255a6a34d422230509d1c3531c

[feat]: use accelerate on multi-node

Former-commit-id: 0b12fc31f1d2a63f9da38821174ab63d9b3a2d72

Update README.md

Former-commit-id: 0d7dd6233811ec3c8741affb436d8004d8c676b8

Update sample.py

Former-commit-id: 6e09fc266a626f465109abc199d9fb452657929e

Update interpolation.py

Former-commit-id: e10e3416433e5f396aa13e7d8beaf451ff47a383

Add files via upload

Former-commit-id: b707b1968fa59d85891040604e7d54bef61b4549

Update and rename Frame Interpolation.md to readme.md

Former-commit-id: 87bdf1ebecb002c086497b3c47acb3f3cb2aaf34

Update readme.md

Former-commit-id: e287f94838007f2eda434a46bcd250099b627097

Support deepspeed for videogpt training.

Former-commit-id: 1f3dfe038845d91b81b0ecc9b6f3cabb06eed5d5

Fix config typos.

Former-commit-id: 455973b57bda19379d41e59aa418421fd269281e

Add hidden_size argument for videogpt config.

Former-commit-id: 1be9f22767fe9c80ad90cb17184751b97aff9012

Add deepspeed script for videogpt training.

Former-commit-id: 3bda2a05cb789fe785b2355a613385b3eb7abbbc

use fp16 temporarily.

Former-commit-id: 93381968e10aae7bbec23ec1ef40e7cb311fddf0

update videogpt.

Former-commit-id: 2e211563c684a8b6ef62dfc0c4f1897224dfb58d

Fix deepspeed zero loss bug.

Former-commit-id: e24af415bc5d6fd38d72a24d6312f4e65d3627dd

Can't fix zero loss bug.

Former-commit-id: 1f32f9d889743b7961b34b823b276bfd03974191

use imageio for write video which has better compatibility.

Former-commit-id: 005119dc9f78ea2a611d22a53772fd59d97d7139

Update training arguments of videogpt training.

Former-commit-id: c9660af891c0dae1f51a1075ef5b2df012f65e06

Update README.md

Former-commit-id: 8ec7524f4f171e650d7a25cc88ed29f7b5017454

Update train_256.sh

Former-commit-id: a2dfc9a8c70845446e731c88acd222e9b2db9122

Update README.md

Former-commit-id: 7e74356a3279b02d062bab9c9f361f61eb756179

Update README.md

Former-commit-id: cf1c56c0058288427c1dfd651c8c0e9a981c5653

refactor dit for supporting deepspeed.

Former-commit-id: e1909745cb05ea2f28f3a391c194f7656593a236

Fix typos.

Former-commit-id: 6add5ee7ef39a973e91b7a7c1b3a4b0980130b88

Update README.md

Former-commit-id: 2d5b6815f95e94a98847b8a0376473f01bb15d14

[bug] fix quantization loss

Former-commit-id: 111fcf55f969ffe4c9be284a317ae5712570a527

del t2v

Former-commit-id: 910398ca297be8a188d59c59c121857666774af1

Update sample.py

Former-commit-id: eb906691abec030d7aa2078b63fc9c7d678662ed

Update sample.sh

Former-commit-id: b149a300f1aff53d9ab6e66a91cc2bcd37900347

Update README.md

Former-commit-id: 7b96fd89eaf3743a4a9fbc995e6a2a791c734f2e

del t2v

Former-commit-id: 5daa2952997a29e74e2b5d235001d6d9dabf9ff6

[fix]: fix sample.sh

Former-commit-id: 09916dfcfb0b80a4217016919e0394a2cb665cfd

update scripts

Former-commit-id: 8b2a794095f2deb9b1e8090486cb8242bf81031d

Update pyproject.toml

Former-commit-id: 7fb921440b608e47781d2f541b2a8d893a48193e

Update README.md

Former-commit-id: 34e529fd02712d4466ab2bc1a63f9bd69976b6c3

Update README.md

Former-commit-id: d97edd0e714a237a54f191dcebb6fd39f73185d6

resume and compress kv

Former-commit-id: 0af7d58f88c1b65b4089da6f3e5c70456614da7c

refactor and fix resume bug

Former-commit-id: 1b1136671b9a7e0b15b70168e2c44dbff04a22e2

Update train.py

Former-commit-id: 783faf5cd86c148a713ea986ceed8527b108f2b8

Update train_t2v.py

Former-commit-id: d25b39ffac4474abd40b551fe0e51c421cb7bf5e

Update README.md

Former-commit-id: 6729039f059a23bf71faa944e2498657ef2c7442

[docs]: fix VQVAE script path

Former-commit-id: d5f5c681b8c50622a84256881d193d5803ff3c28

Update pyproject.toml

Former-commit-id: 5637656a79faba1b397b56918c42022cc209e731

[feat] add casual vqvae

Former-commit-id: 89087970b267908c8667984aba42b1ade2505467

Update __init__.py

Former-commit-id: 7c8af2909c4549e96bc0893d09fcc53d71fb6cc7

Create clip.py

Former-commit-id: e3397c567efcfeef6f58dfef2eb4251bedbbf33a

Update clip.py

Former-commit-id: 1ee53d152630b6acc69369be33287abfb143224d

train with image

Former-commit-id: 209b9a8ad2f19a50e46f9b7ec68dd781898e491a

support attention_mode

Former-commit-id: c72f354fbfd948e8cd4b0cc63c157bc4688c7665

Update README.md

Former-commit-id: 9065d12ee867c76452ce370fb71cc83bdc01470b

Update README.md

Former-commit-id: 95b28706d257d7769049abe30a7121b60e39a53f

t2v attention_mode

Former-commit-id: 4fd8bd87e0d0a98d5d0a3201d5db9a5ba055b894

Update README.md

Former-commit-id: 859c0f60dac1562b8a6bbb72b793a4709dbc3597

Update Data.md

Former-commit-id: d1c72e3209769d0204efb8e7d705bdf4f22b82ea

train t2v feature

Former-commit-id: 7f16b162cae0e3533c8ba9aa03186a846d9e02bf

scripts

Former-commit-id: 047046f684c2831e68e0cf256cb46fc3790cda33

add causalvae

Former-commit-id: ac936276fd3213ee1815dfe096cd64cecb601428

Create causalvideovae.md

Former-commit-id: 98414c2ad7dd39f740440b2d9059140f93492e83

Rename causalvideovae.md to CausalVideoVAE.md

Former-commit-id: 0a78e4193fcaec4662b7c6465a9d80643b354cd6

Update README.md

Former-commit-id: 8d70a890fc4191bfc98c3aca6dd3c8a70534fb3d

Update README.md

Former-commit-id: a41674f5ef71ced59b374226dddfa09c15a99f25

Update CausalVideoVAE.md

Former-commit-id: 1cd665f699632e9dc9357379225c085d9c1278d5

Update README.md

Former-commit-id: 90d6d298eaee6255098b342b31444e7a347fc50e

Update README.md

Former-commit-id: f294a0a28f288ae592e7de004320d3eb5e48e945

Update README.md

Former-commit-id: 02b0422fd745b4b7ec091cc6ee81d1d237245fe6

[feat] eval

Former-commit-id: d2c3cf5ef8d18feb7b0544d68d9796656e90ba8d

fix training bug

Former-commit-id: 09a6283ddd9ab11479f30324de788bd7415702ae

update model

Former-commit-id: e4e24650376d9dce974b290752efb416d5061983

update vae

Former-commit-id: 24298df5d374d5aaa5696f6200d4e23ae887715f

fix videovae trainer

Former-commit-id: 79e1feb412b6eeead3306f7f0d55a01f9b2a543e

Update CausalVideoVAE.md

Former-commit-id: fcde4d0ff783448c2006d6f4751a0e4f9cc23d0f

Update README.md

Former-commit-id: 2d9ae56309717a91e0690bd5c354fe3cbe5d5ed6

Update __init__.py

Former-commit-id: 2bdc29d86ebff3d6f29c38ded57c7b0dd1e6bbdb

update sample

Former-commit-id: 116726b9d565c689496123d4b6e62b98400536a6

tile conv

Former-commit-id: 5133615bcba221c4b86ae4e8b09b82ee5d34c66d

fix reshape bugs in AttnBlock3D

Former-commit-id: 299622e168a6cbee14ef2056c47cae722c20e005

tile only2d

Former-commit-id: 7f31ad03fdbaac97aea51a98bd29dc7327b700c9

released v1.0.0

Former-commit-id: 8df5897a4ffc341ced45faf617687fb8ebf1286c

sample pipeline

Former-commit-id: 8964b16c0dd65445a20050c9a0e82d909b55eaff

add tile training

Former-commit-id: 216079dc925beefe1ec722a4e87e46a4e1ca56be

update train script

Former-commit-id: ac16f77d19488cd46130587369cd7a5ea378052b

refactor vae to hf

Former-commit-id: a938d3f7f962709f7ecff5e9d49c9bf1ce40d7fa

release

Former-commit-id: 52fe2ac6610661d6a13a08390c84a8237f37986a

v1.0.0

Former-commit-id: 93f44f3b425c090d9bc92687a9c50a26b4f0d7c8

clean

Former-commit-id: 56ed87088e13cdd93929e2ceedbfae693e2f013a

update gradio demo

Former-commit-id: 80c2cc0bcc2a7532bc2440df16c1f811b0b30516

Update and rename CausalVideoVAE.md to Report-v1.0.0.md

Former-commit-id: 3b848f8de33cf4f0ed802428e9ad31d2fa26de63

Update README.md

Former-commit-id: e6fb5d864ac8af8214cdc89c71b42f296dda14e7

Update README.md

Former-commit-id: a61200bea2eea4fd22bd8095f192fef4c22c49b4

Update README.md

Former-commit-id: a2ec90a19eacb099cb9f36c1b5dd6e24be307fae

Add files via upload

Former-commit-id: 1ae8512b7846e9faafdb38ca532898d7f1bdcce2

Add files via upload

Former-commit-id: 292cbcfa4e4f384fe6958badc27d3067173119f9

Update Report-v1.0.0.md

Former-commit-id: 92d906b340f5970a77377586d7295f46b5de99ec

Update README.md

Former-commit-id: 037d7d3e084d7ff426347db49c4b210a2ee3927d

Update README.md

Former-commit-id: 7d4118566b77e09de4a3615612a5b368c76c293e

Update README.md

Former-commit-id: 3210474a46845ae3a6c0bde7fb6c7f9845866628

Add files via upload

Former-commit-id: a232ec2d98b245a9b502a2c49fc4f468a86fbd64

Update README.md

Former-commit-id: d5ab411b2e2fd5d6abeda500895e3339167572ab

Update README.md

Former-commit-id: 30f61958e221c3d41c4aa5917f4511e8c0482cf9

Update README.md

Former-commit-id: f8cb7f38751d7a6ef586c293fb769f5d60ad5c59

Update Report-v1.0.0.md

Former-commit-id: bd9aa7991212d98a6224b855a3bfb2c12feb5950

Update Report-v1.0.0.md

Former-commit-id: aea1f4a82a525d3e6e1d1a1bb7631326b294cdcc

Update README.md

Former-commit-id: f709d75e7a7d8a08ec60132aa906f6b880af8e58

Add files via upload

Former-commit-id: 9a207435d9efac0b19771df5d2288160af1bdd6e

Update README.md

Former-commit-id: 9bae31c73ea902581e1ee94cfd9012efdf78d349

Update README.md

Former-commit-id: 9f926abb08f156807091dd044c133788649a6bac

Update Report-v1.0.0.md

Former-commit-id: 7cec40420601f7da57449586508e50119bd02a61

Update README.md

Former-commit-id: ce2bbf82045041495ed448b46d5f4bdba84aa740

Fix some compatibility issues

Former-commit-id: 942d8598b28516369780c3519607929869b427b7

Fix bugs in inference

Former-commit-id: c251a86ac7c91d6f2bb12ae645628e9cf4b83d3d

fix bugs in inference

Former-commit-id: e17c7a7b7704920cb86894419bf573d0fb0c06fa

Update README.md

Former-commit-id: c1492211ced3cdbec93ef28678a1467245e122e9

Update README.md

Former-commit-id: c4a976e9b93536299b2e350454c780fec54fbe95

Update README.md

Former-commit-id: bbf3cef43a129a25a80ce134a8895796e24353c6

Update README.md

Former-commit-id: 5731e810e2efb7ca7eda1aa8954eb7fd260f4250

Update README.md

Former-commit-id: 1b54444249fa91b29334b8fc364fb1978c25fb2d

Update README.md

Former-commit-id: 9c0f82ae9953b56f2540e9c7ada021e4e1bc6165

Update Report-v1.0.0.md

Former-commit-id: 38b779022418cc41cdf9fc508e461f2f6299fd60

Update Report-v1.0.0.md

Former-commit-id: 6ebe2c62c0904142026d3796027e95ce43afe16a

Update README.md

Former-commit-id: 2873d0eaedfad392c627a714448118949d7b7e13

Update Report-v1.0.0.md

Former-commit-id: 400f708a080f0799e02697bb8e66d8430ffb1f56

Update Report-v1.0.0.md

Former-commit-id: 37f181f34f64ca1141a1345b7c1f2af283be1cad

Update README.md

Former-commit-id: dc923fe2413b3be31d8d88e7dd83a931fcbf9458

Update README.md

Former-commit-id: 828be4267f0795969293e4a321e4e0bfa8e1f1e6

clean

Former-commit-id: a7f2c1c8b2243587cedd6e152e64298842287095

Update pyproject.toml

Former-commit-id: bb037420b75303b58486ba79da9180b2bf63cf88

Update Data.md

Former-commit-id: d9a5c8fbd36a34b7baf720bc27d69604ae5c9d57

add causalvae doc

Former-commit-id: 4e6d9e038c8edb81f8402834bd01eeeb823924f9

Update Train_And_Eval_CausalVideoVAE.md

Former-commit-id: 589657b089930379bb75f3d49063ebb72671efc6

Update Train_And_Eval_CausalVideoVAE.md

Former-commit-id: 156739c35b5dda594138b2e3b1fd6ec99556af23

Update README.md

Former-commit-id: 567ade18047a9cc81648997c5d9287fa8900614d

更新 README.md

Former-commit-id: 42607786f3f3d0eba9336f774b0a2ae076cce793

Update README.md

Former-commit-id: 88e7f2062c494c199caff468f5f14f3165f763de

Create Report-v1.0.0-cn.md

Former-commit-id: d484754e080bd65f6b35619b840040581fa6b69c

Update README.md

Former-commit-id: e386ec112337271bb9f55ea6ec872eaba94aff50

Update README.md

Former-commit-id: c437a18dbec7ac6e02287565a84d75f96664ee99

Update README.md

Former-commit-id: 0cc27a5b065b61f37d44dc2c90ba2782075eefb2

Update Report-v1.0.0-cn.md

Former-commit-id: d5cf6aed7c1673562a6d2022c3fd7f7b16006cfd

Update Data.md

Former-commit-id: 15defc1a3846335fd68e18fa9b8ffbe38e3e7100

Update Report-v1.0.0.md

Former-commit-id: 3c8eb75dc2045154e8654f860b1fabd7e597094b

Update Report-v1.0.0-cn.md

Former-commit-id: 0edac24fcaa447a534755bf3aac1f2027a491c06

Update Report-v1.0.0-cn.md

Former-commit-id: 59edce70edbaed957ba40f1d95b4a5050ece12bf

Update Report-v1.0.0-cn.md

Former-commit-id: 83124efeaa5a8cd528a15760bdf88acc403142bd

Add files via upload

Former-commit-id: 943dd9a063baaa05d7f0d0bd52cbde35cf2b3d98 [formerly 5542359738243a8e948a8f08ed4e8dedf6b4b818]
Former-commit-id: fbee07f6fef2ac86bc15ff66f2e2a1a68b64ed01

Update README.md

Former-commit-id: 6fd60c73b6ae69d6ad9dc14c7371bd9bff086b44 [formerly 2ed5d5b7bdcd88044da26ca94237c50af7ab47dd]
Former-commit-id: 443c333353963bc1ec64a4946afbe5e17f9b5a87

Update README.md

Former-commit-id: 54bad83bf054a2ff574199931121c686b4e28008 [formerly 06bb190a39a9b98c8de5e385fc304a86cc8c9fb2]
Former-commit-id: 23e8a235d6828d9ab7c02ef59d40fe13a46fa4b1

Update README.md

Former-commit-id: c48eba0bb15d594007809d4cc605dc46ba029cec [formerly 7deb8a0decf7a9757af4475e3029d415d78d185f]
Former-commit-id: 7f43bb2f1d9308bd08154171d78e9b36e00a837f

Update README.md

Former-commit-id: 68eaf2e6b00266cf0858b33f9b4d09681efbd2bc [formerly d8dcad250ed8357964ab8ee570b9973b45e29fcb]
Former-commit-id: 75f51b3d84203be3672f6b07bb3c70ba0ae5556c

Update README.md

Former-commit-id: be00a17729ac889493f63fd1d649054d7980ca42 [formerly 9b4d8dcfc8e6e2abe4ca727e01561d27d1ba3968]
Former-commit-id: f1b6f5013b99d0d288af0f05c46cb8300d79a8aa

Update train_videoae_65x256x256.sh

Former-commit-id: b2f5cfe72452dfb369aba166e4a101dfcf5758c9 [formerly 557b78ab38a61d7520e91044137f818e50ec24cb]
Former-commit-id: 0e46c9f261e7f5fd4d08e3d7735c17c7975b0e72

Update train_videoae_65x512x512.sh

Former-commit-id: 8e40db106efa2fd879cca42bd9ce2d9b79d777f8 [formerly 3e11781b045a838a5aa98ea75e274198f2b26a44]
Former-commit-id: f3b0804a7cf063e7ff04ee57dccee0320c1f32a5

fix using pretrained bug

Former-commit-id: 2dfbb0683b2f3bd1a1d619e5e4fabb5bd4c92d79 [formerly 27f352e822f732cb351d64ac5726c5df3b62c1bb]
Former-commit-id: 722d60ae5ef28499c47292e0a4807a87b9658b01

Update train_t2v_feature.py

Former-commit-id: 4a4f7ea2366f1dff5669d6d4834238ebd790e6e2 [formerly ca7d3158fb01f8e7228f833cea8e6741be127c2a]
Former-commit-id: 389f0e7dc66826026046b8e12d7a9f8692d32e42

[docs] update docs and train.sh

Former-commit-id: 1d9358e84d12863681b396c7621ae7244caa4dc2 [formerly b801491889b3abad1211225d7f07af3ae0389895]
Former-commit-id: 9e6a59ae2f246b50ace0a033cbd6548660241204

[fix] fix bug in #185

Former-commit-id: d74efbd0195380ca383598c06d3d8185dd9cc9bc [formerly 0f2687266bc2271e11b2f06edfe514f466e57b0f]
Former-commit-id: cb6bb234befc49b3870c5690653469a0714890c5

[docs] add evaluation detail

Former-commit-id: c15a717cc61159e5f6adafb718237ab578ac81e7 [formerly 9e92772776fa4dd18772c7fec939a42ad6dfbc0a]
Former-commit-id: 98c2a38af5064f27888d1b102c9fded87d2420d5

[docs] update train.sh

Former-commit-id: eeb00ac046692bc97a064ec1b47c994436798ab1 [formerly 6d0fd85e1b13a7824b8181ba9ccebec2a3c76d40]
Former-commit-id: 1e8e669ad9967aeeeb7622bc04a0759f348cb26f

Delete assets/we_want_you.jpg

Update README.md

fix typo

Fix typos

line 24: `github -> GitHub`
line 54: `a -> an`
line 85: `re-organizes -> re-organize, modulizes -> modulize`
line 87: `opened -> open` (modified according to the changelogs of other dates)
line 94: `a -> an`

[fix]: Fix variable naming errors

Update README.md

[docs] update inference example.

[feat] add time chunk inference

[docs] fix typo

[refactor] fix hardcode

Update LICENSE

Update Report-v1.0.0.md

Update pyproject.toml

add 2drope and dynamic training

refactor dynamic training

read image from folder

img training

img training

support newvae

vae temporal tiling

abs and rope

compress kv and rope pi

fix rope with compress

update dataset

mask loss

multi-data

fix dataset

train vis

update

fix mask

5.15

prepare release

prepare scripts

fix demo

5.27

update prompt

Update README.md

Update README.md

Create Report-v1.1.0.md

Update README.md

Update README.md

Update README.md

Update README.md

Update README.md

Update README.md

Update README.md

Update README.md

Update README.md

Update README.md

Update README.md

Update LICENSE

Update Report-v1.1.0.md

solve typos

Update pyproject.toml

Update README.md

Update dataset_utils.py

Update modeling_latte.py

Update train_t2v.py

Update train_t2v.py

Update t2v_datasets.py

fix wrong code

Update LICENSE

fix train bug

fix vis

[docs] update CausalVideoVAE docs

Update README.md

fix the bug

Update gradio_web_server.py

Update README.md

Update gradio_utils.py

Update sample_video_513.sh

Rename sample_video_513.sh to sample_video_221.sh

Update README.md

Update Report-v1.1.0.md

release v1.2.0

Update README.md

Update Report-v1.2.0.md

Update README.md

Update pyproject.toml

Update Report-v1.2.0.md

Update Report-v1.2.0.md

Update Report-v1.2.0.md

Update t2v_datasets.py

Update Report-v1.2.0.md

Update Report-v1.2.0.md

Update README.md

Update Report-v1.2.0.md

Update Report-v1.2.0.md

Update README.md

fix sample

Delete opensora/train/train_t2v_diffusers_lora.py

lora bug

Update README.md

Update Report-v1.2.0.md

Update train_t2v_diffusers.py

release Open-Sora Plan v1.2.0 i2v (#389)

* gitignore

* inpaint

* Create condition_image_path.txt

* Rename condition_image_path.txt to condition_images_path.txt

* Update sample_inpaint.py

* Update sample_inpaint.sh

* inpaint

* fix bug

* Update README.md

* inpainting

* release v1.2.0 i2v

* release v1.2.0 i2v

* Update Report-v1.2.0.md

* Update pipeline_inpaint_sp.py

* Update sample_inpaint_ddp.py

* Update sample_inpaint_sp.py

* Update sample_inpaint.py

* Grammar Error Correction

* Grammar Error Correction

---------

Co-authored-by: LinB203 <2267330597@qq.com>

Update Report-v1.2.0.md

add 29x480p link

Update README.md

Update train_inpaint.sh

do not set seed

[fix] fix the path typo for google/mt5-xxl in gradio_web_server.py (#405)

* [fix] fix the path typo for google/mt5-xxl in gradio_web_server.py

* [fix] fix the issue: the cache of pretrained mt5-xxl weights is inconsistent.

---------

Co-authored-by: guangyi <guangyi.liu@mbz-h100-029.core42.ai>
2024-03-04 14:06:39 +00:00
1254 changed files with 214637 additions and 11555 deletions
+94
View File
@@ -0,0 +1,94 @@
# Agent Infrastructure — Status Dashboard
Developer-maintained overview of all agent components and their maturity.
Use this to understand what exists, how complete it is, and how much to trust it.
_Last synced: 2026-03-02_
> To resync this dashboard, use the workflow: `.agents/workflows/sync-dashboard.md`
---
## Summary
| Category | Total | ✅ Ready | 🟡 Draft | 🔴 Stub | Trust |
|----------|-------|---------|---------|---------|-------|
| Skills | 8 | 0 | 8 | 0 | Low — newly created, untested |
| Workflows (SOPs) | 4 | 0 | 4 | 0 | Low — newly created, untested |
| Memory files | 4 | 1 | 3 | 0 | Medium — codebase_map is solid |
| Lessons | 0 | — | — | — | N/A — empty |
| Exploration logs | 0 | — | — | — | N/A — empty |
---
## Skills (`.agents/skills/`)
| Skill | File | Status | Trust | Tested | Notes |
|-------|------|--------|-------|--------|-------|
| Launch Experiment | `launch-experiment.md` | 🟡 Draft | Low | ❌ | Needs dry-run validation |
| Monitor Experiment | `monitor-experiment.md` | 🟡 Draft | Low | ❌ | Requires W&B API access to test |
| Summarize Run | `summarize-run.md` | 🟡 Draft | Low | ❌ | Pattern from existing test infra |
| Log Experiment | `log-experiment.md` | 🟡 Draft | Low | ❌ | Journal formatting only |
| Evaluate Video Quality | `evaluate-video-quality.md` | 🟡 Draft | Low | ❌ | SSIM section most mature |
| Index Related Work | `index-related-work.md` | 🟡 Draft | Low | ❌ | Schema defined, no entries yet |
| Search Related Work | `search-related-work.md` | 🟡 Draft | Low | ❌ | Depends on indexed entries |
| Skill Template | `SKILL_TEMPLATE.md` | ✅ Ready | High | ✅ | Meta-template, stable |
### Trust Level Definitions
- **High**: Tested in production, validated against real experiments
- **Medium**: Logic is sound, partially tested or based on existing patterns
- **Low**: Newly written, not yet validated
- **None**: Placeholder only
---
## Workflows / SOPs (`.agents/workflows/`)
| Workflow | File | Status | Trust | Tested | Notes |
|----------|------|--------|-------|--------|-------|
| Experiment Lifecycle | `experiment-lifecycle.md` | 🟡 Draft | Low | ❌ | End-to-end flow, untested |
| Evaluation Development | `evaluation-development.md` | 🟡 Draft | Low | ❌ | Metric dev process |
| Experiment Journaling | `experiment-journaling.md` | 🟡 Draft | Low | ❌ | Journaling cadence |
| Lesson Capture | `lesson-capture.md` | 🟡 Draft | Low | ❌ | Post-experiment reflection |
| Sync Dashboard | `sync-dashboard.md` | 🟡 Draft | Low | ❌ | This dashboard's updater |
---
## Memory (`.agents/memory/`)
| File | Status | Trust | Notes |
|------|--------|-------|-------|
| `codebase_map.md` | ✅ Ready | High | Synthesized from full repo research |
| `experiment_journal.md` | 🟡 Draft | Medium | Schema defined, no entries yet |
| `evaluation_registry.md` | 🟡 Draft | Medium | SSIM/loss metrics documented |
| `related_work/README.md` | 🟡 Draft | Medium | Schema defined, no entries yet |
---
## Lessons (`.agents/lessons/`)
| File | Category | Severity | Notes |
|------|----------|----------|-------|
_No lessons captured yet._
---
## Exploration Logs (`.agents/exploration/`)
| File | Status | Topic | Notes |
|------|--------|-------|-------|
_No exploration logs yet._
---
## What to Do Next
1. **Validate skills**: Run a minimal training experiment using the
`experiment-lifecycle` SOP to test `launch-experiment` → `monitor-experiment`
→ `summarize-run` end-to-end.
2. **Index first related work**: Use `index-related-work` to add at least one
paper (e.g., the Self-Forcing paper used in the codebase).
3. **Capture first lesson**: After the validation run, capture any findings.
4. **Promote to Ready**: As each skill/SOP is tested, update its status here.
+46
View File
@@ -0,0 +1,46 @@
# Exploration Logs
This directory holds draft procedures and investigation notes for tasks that
don't yet have a standardized skill or SOP. Each exploration should follow this
template.
## When to Create an Exploration Log
- You are working on a task with no existing skill or workflow.
- You are experimenting with a new metric, training technique, or tool.
- You want to document findings before they are promoted to a standard.
## File Naming
`<topic-slug>.md` — e.g., `fvd-metric-investigation.md`
## Template
```markdown
# Exploration Log: <Topic>
## Status: draft | under_review | promoted | abandoned
## Context
<Why this exploration is needed — link to experiment or task if applicable.>
## Progress
- [ ] Step 1: ...
- [ ] Step 2: ...
## Findings
<What you have learned so far.>
## Mistakes / Dead Ends
<What didn't work and why — these become lessons.>
## Proposed Standardization
<If this works, describe the skill/SOP/workflow to create.>
```
## Lifecycle
1. **Create** during exploration mode.
2. **Update** as you make progress.
3. **Promote**: If findings are solid, create a skill in `.agents/skills/` or an SOP in `.agents/workflows/`.
4. **Archive mistakes**: Move failures into `.agents/lessons/`.
+48
View File
@@ -0,0 +1,48 @@
# Lessons Learned Database
This directory stores documented mistakes, unexpected behaviors, and their fixes.
Each lesson is a permanent record that helps agents and humans avoid repeating
past errors.
## When to Create a Lesson
- An experiment failed for a non-obvious reason.
- A configuration or hyperparameter choice led to wasted compute.
- A porting, data, or infrastructure issue was discovered and resolved.
- A workaround was needed for a known framework/library bug.
## File Naming
`<YYYY-MM-DD>_<short-slug>.md` — e.g., `2026-03-02_lr-too-high-for-lora.md`
## Template
```markdown
---
date: <ISO-8601>
experiment: <reference to experiment_journal.md entry, if applicable>
category: hyperparameter | data | infrastructure | evaluation | porting | other
severity: critical | important | minor
---
# <Short Descriptive Title>
## What Happened
<Description of the problem and its symptoms.>
## Root Cause
<Analysis of why it happened.>
## Fix / Workaround
<What resolved the issue.>
## Prevention
<How to avoid this in the future — updated skills, SOPs, or checks.>
```
## Usage
- Before starting a task, **search this directory** for relevant lessons.
- After completing or failing a task, **check if a new lesson should be created**.
- Periodically review lessons for **patterns** — recurring themes may warrant
a new skill, SOP, or codebase fix.
+129
View File
@@ -0,0 +1,129 @@
# FastVideo-WorldModel — Codebase Map
High-level structural index for agent orientation. Updated 2026-03-08.
## Repository Layout
```
FastVideo-WorldModel/
├── fastvideo/ # Core Python package
│ ├── models/ # Model implementations
│ │ ├── dits/ # DiT transformers (wanvideo, ltx2, ...)
│ │ ├── vaes/ # VAE models
│ │ ├── encoders/ # Text/image encoders (T5, CLIP)
│ │ ├── schedulers/ # Noise schedulers
│ │ ├── upsamplers/ # Super-resolution models
│ │ ├── audio/ # Audio models
│ │ └── loader/ # Component loaders for HF repos
│ ├── configs/ # Configuration system
│ │ ├── models/ # Arch configs + param_names_mapping
│ │ ├── pipelines/ # Pipeline wiring
│ │ └── sample/ # Default sampling parameters
│ ├── pipelines/ # End-to-end pipelines
│ │ ├── basic/ # Per-model pipelines (wan/, ltx2/, ...)
│ │ └── stages/ # Reusable pipeline stages
│ ├── train/ # Refactored training framework (YAML-driven, preferred)
│ │ ├── trainer.py # Main training loop coordinator
│ │ ├── entrypoint/ # Training entrypoint (train.py) + checkpoint conversion
│ │ ├── methods/ # Training algorithms (FineTune, DFSFT, DMD2, SelfForcing)
│ │ │ ├── base.py # TrainingMethod ABC
│ │ │ ├── fine_tuning/ # FineTuneMethod, DiffusionForcingSFTMethod
│ │ │ └── distribution_matching/ # DMD2Method, SelfForcingMethod
│ │ ├── models/ # Per-role model wrappers (ModelBase, CausalModelBase)
│ │ │ └── wan/ # WanModel, WanCausalModel
│ │ ├── callbacks/ # Composable hooks (grad_clip, ema, validation)
│ │ └── utils/ # Config, builder, checkpoint, optimizer, tracking
│ ├── training/ # Legacy training infrastructure (being phased out)
│ │ ├── trackers.py # W&B tracker (BaseTracker → WandbTracker)
│ │ ├── training_utils.py # Checkpointing, grad clipping, state dicts
│ │ ├── training_pipeline.py # Base training pipeline
│ │ ├── wan_training_pipeline.py # Wan T2V training
│ │ ├── wan_i2v_training_pipeline.py # Wan I2V training
│ │ ├── distillation_pipeline.py # Distillation base
│ │ ├── wan_distillation_pipeline.py # Wan distillation
│ │ ├── self_forcing_distillation_pipeline.py # Self-forcing distill
│ │ ├── ltx2_training_pipeline.py # LTX-2 training
│ │ └── matrixgame_training_pipeline.py # MatrixGame training
│ ├── attention/ # Attention backends
│ ├── distributed/ # Sequence/tensor parallel utilities
│ ├── layers/ # Tensor-parallel layers
│ ├── tests/ # Package-level tests
│ │ ├── training/ # Training regression tests (W&B summary comparison)
│ │ ├── ssim/ # SSIM visual regression tests
│ │ ├── encoders/ # Encoder parity tests
│ │ └── modal/ # Modal CI test runner
│ └── registry.py # Unified config registry
├── fastvideo-kernel/ # CUDA/custom kernels (separate build: ./build.sh)
├── scripts/ # Utility scripts
│ ├── distill/ # Distillation launch scripts
│ ├── inference/ # Inference scripts
│ ├── checkpoint_conversion/ # Weight conversion tools
│ ├── finetune/ # Finetune scripts
│ └── preprocess/ # Data preprocessing
├── examples/ # Ready-to-run examples
│ ├── training/ # Training examples (finetune/, consistency_finetune/)
│ ├── distill/ # Distillation examples
│ ├── inference/ # Inference examples
│ └── dataset/ # Dataset examples
├── docs/ # MkDocs documentation source
│ ├── design/overview.md # Architecture overview
│ ├── training/ # Training guides
│ └── contributing/ # Contributor guides + coding_agents.md
├── tests/ # Top-level tests (local_tests/)
├── AGENTS.md # Agent coding guidelines
└── .agents/ # Agent infrastructure (you are here)
```
## Key Training Entrypoints
### New framework (`fastvideo/train/`) — preferred
| Method | Config Example | Launch Pattern |
|--------|---------------|----------------|
| FineTune (Wan) | `examples/train/finetune_wan2.1_t2v_1.3B_vsa_*.yaml` | `torchrun -m fastvideo.train.entrypoint.train --config <yaml>` |
| DFSFT (Wan causal) | `examples/train/dfsft_wan_causal_t2v_1.3B.yaml` | `torchrun -m fastvideo.train.entrypoint.train --config <yaml>` |
| DMD2 distillation | `examples/train/distill_wan2.1_t2v_1.3B_dmd2.yaml` | `torchrun -m fastvideo.train.entrypoint.train --config <yaml>` |
| Self-Forcing | `examples/train/self_forcing_wan_causal_t2v_1.3B.yaml` | `torchrun -m fastvideo.train.entrypoint.train --config <yaml>` |
### Legacy pipelines (`fastvideo/training/`) — being phased out
| Pipeline | Entrypoint | Launch Pattern |
|----------|-----------|----------------|
| Wan T2V finetune | `fastvideo/training/wan_training_pipeline.py` | `torchrun --nproc_per_node N` |
| Wan I2V finetune | `fastvideo/training/wan_i2v_training_pipeline.py` | `torchrun --nproc_per_node N` |
| Wan distillation (DMD) | `fastvideo/training/wan_distillation_pipeline.py` | `torchrun --nproc_per_node N` |
| Self-forcing distill | `fastvideo/training/wan_self_forcing_distillation_pipeline.py` | `torchrun --nproc_per_node N` |
| LTX-2 finetune | `fastvideo/training/ltx2_training_pipeline.py` | `torchrun --nproc_per_node N` |
| MatrixGame | `fastvideo/training/matrixgame_training_pipeline.py` | `torchrun --nproc_per_node N` |
## W&B Integration
- **Tracker classes**: `fastvideo/training/trackers.py`
- `WandbTracker` — logs metrics, videos, timing
- `SequentialTracker` — fan-out to multiple trackers
- `DummyTracker` — no-op for offline/test
- **Run summary location**: `<output_dir>/tracker/wandb/latest-run/files/wandb-summary.json`
- **Reference summaries**: `fastvideo/tests/training/*/` (e.g., `a40_reference_wandb_summary.json`)
- **Environment**: `WANDB_API_KEY`, `WANDB_BASE_URL`, `WANDB_MODE`
## Critical Environment Variables
| Variable | Purpose |
|----------|---------|
| `WANDB_API_KEY` | W&B authentication |
| `WANDB_MODE` | `online` / `offline` |
| `FASTVIDEO_ATTENTION_BACKEND` | `FLASH_ATTN` / `TORCH_SDPA` |
| `TOKENIZERS_PARALLELISM` | Set `false` to avoid fork warnings |
| `HF_HOME` | HuggingFace cache directory |
## Build & Test Commands
```bash
uv pip install -e .[dev] # Editable install
pre-commit run --all-files # Lint/format/spell
pytest tests/ # Top-level tests
pytest fastvideo/tests/ -v # Package tests
pytest fastvideo/tests/training/Vanilla -srP # Training loss regression
pytest fastvideo/tests/ssim/ -vs # SSIM visual regression
cd fastvideo-kernel && ./build.sh # Build kernels
```
@@ -0,0 +1,327 @@
# Evaluation Metrics Registry
Living catalog of all evaluation metrics for FastVideo-WorldModel video quality
assessment. Each metric includes a detailed explanation, implementation status,
usage instructions, and interpretation guide.
_Last updated: 2026-03-02_
---
## Metric Summary
| Metric | Category | Status | Location | Trust |
|--------|----------|--------|----------|-------|
| **FVD** | Distribution | ✅ Implemented | `benchmarks/fvd/` | High |
| **SSIM** | Reference | ✅ Implemented | `fastvideo/tests/ssim/` | High |
| **LPIPS** | Perceptual | ✅ Implemented | `scripts/lora_extraction/` | Medium |
| **Loss trajectory** | Training signal | ✅ Implemented | W&B `train_loss` | Medium |
| **Grad norm stability** | Training signal | ✅ Implemented | W&B `grad_norm` | Medium |
| **GameWorld Score** | Multi-dim benchmark | 🟡 External | Matrix-Game repo | Low |
| **Human preference** | Gold standard | 🔴 Manual | N/A | Highest |
---
## Implemented Metrics
### FVD — Fréchet Video Distance
**Category**: Distribution-level quality metric
**Status**: ✅ Fully implemented in `benchmarks/fvd/`
**Trust**: High — standard protocol, I3D feature extractor
#### What It Measures
FVD measures the distance between the **distribution** of generated videos and
a distribution of real/reference videos. It works by:
1. Extracting spatiotemporal features from both real and generated video sets
using a pretrained **I3D** (Inflated 3D ConvNet) model.
2. Modeling each set of features as a multivariate Gaussian (mean + covariance).
3. Computing the **Fréchet distance** between the two Gaussians.
Lower FVD = generated videos are more statistically similar to real videos.
#### Why It Matters
- FVD is the **de facto standard** for benchmarking video generation models.
- It captures both **visual quality** (are individual frames realistic?) and
**temporal coherence** (do frames flow naturally?).
- Matrix-Game 2.0, Open-Sora, and most video generation papers report FVD.
#### Limitations
- Requires a **large sample set** (standard protocol uses 2048 videos) to
produce stable statistics. Small sample sizes yield noisy results.
- Measures **distributional similarity**, not per-video quality. A model could
have low FVD by generating a diverse set of "roughly okay" videos.
- The I3D model was trained on Kinetics-400 (human actions). It may be less
sensitive to domain-specific artifacts in non-human-action videos (e.g.,
driving, game environments).
- Does not directly measure text-video alignment or action controllability.
#### How to Use
```python
# Programmatic
from benchmarks.fvd import compute_fvd_with_config, FVDConfig
config = FVDConfig.fvd2048_16f() # Standard: 2048 videos, 16 frames
results = compute_fvd_with_config('data/real/', 'outputs/gen/', config)
print(f"FVD: {results['fvd']:.2f}")
```
```bash
# CLI
python -m benchmarks.fvd.cli \
--real-path data/real/ \
--gen-path outputs/gen/ \
--protocol fvd2048_16f
```
**Preset protocols**:
| Protocol | Videos | Frames | Use Case |
|----------|--------|--------|----------|
| `fvd2048_16f` | 2048 | 16 | Standard benchmark (papers) |
| `fvd2048_128f` | 2048 | 128 | Long video evaluation |
| `quick_test` | 100 | 16 | Fast dev iteration |
**Feature extractors**: `i3d` (default, standard), `clip`, `videomae`
#### Interpretation
| FVD Range | Interpretation |
|-----------|---------------|
| < 100 | Excellent — near-real quality |
| 100–300 | Good — competitive with SOTA |
| 300–600 | Fair — noticeable gap from real |
| > 600 | Poor — significant quality issues |
> FVD values are dataset-dependent. Always compare against baselines evaluated
> on the same real video distribution.
---
### SSIM — Structural Similarity Index
**Category**: Per-frame reference comparison
**Status**: ✅ Implemented in `fastvideo/tests/ssim/`
**Trust**: High — used in CI regression tests
#### What It Measures
SSIM compares two images (or video frames) based on three components:
1. **Luminance**: brightness similarity
2. **Contrast**: dynamic range similarity
3. **Structure**: spatial pattern similarity
The final score is a value in [0, 1] where 1.0 = identical.
#### Why It Matters
- Used as a **regression guard** in CI: ensures model updates don't degrade
visual output below a threshold.
- More perceptually meaningful than raw pixel MSE.
- Fast to compute — suitable for automated testing.
#### Limitations
- Requires a **pixel-aligned reference** video. Cannot compare videos with
different seeds, prompts, or angles.
- Operates **per-frame** — does not capture temporal coherence.
- Insensitive to some perceptual artifacts (color shifts, high-frequency noise).
#### How to Use
```bash
pytest fastvideo/tests/ssim/ -vs
```
#### Interpretation
| SSIM Range | Quality |
|------------|---------|
| > 0.90 | Excellent — very close to reference |
| 0.80–0.90 | Good — acceptable for most uses |
| 0.70–0.80 | Fair — noticeable differences |
| < 0.70 | Poor — significant divergence |
---
### LPIPS — Learned Perceptual Image Patch Similarity
**Category**: Per-frame perceptual distance
**Status**: ✅ Implemented in `scripts/lora_extraction/lora_inference_comparison.py`
**Trust**: Medium — available but only used for LoRA comparison currently
#### What It Measures
LPIPS uses a pretrained neural network (AlexNet by default) to extract
deep features from two images and computes the distance between them in
feature space. Unlike SSIM, LPIPS correlates much more strongly with
**human perceptual judgments**.
Lower LPIPS = more perceptually similar.
#### Why It Matters
- Best available automated proxy for **human visual judgments** at the frame
level.
- Captures semantic and structural differences that SSIM misses (e.g., texture
changes, minor recoloring).
- Used for validating LoRA merge quality.
#### Limitations
- Per-frame metric — no temporal awareness.
- Requires reference video (paired comparison only).
- Slightly slower than SSIM due to neural network forward pass.
#### How to Use
```bash
python scripts/lora_extraction/lora_inference_comparison.py \
--base merged_model \
--ft path/to/finetuned \
--adapter NONE \
--output-dir results \
--prompt "A cat" \
--compute-lpips
```
#### Interpretation
| LPIPS Range | Quality |
|-------------|---------|
| < 0.10 | Excellent — nearly indistinguishable |
| 0.10–0.20 | Good — minor perceptual differences |
| 0.20–0.40 | Fair — noticeable differences |
| > 0.40 | Poor — clearly different |
---
### Loss Trajectory
**Category**: Training signal proxy
**Status**: ✅ Active (from W&B `train_loss`)
**Trust**: Medium — proxy, not direct quality measure
#### What It Measures
Tracks the training loss over time. A healthy training run shows:
- **Decreasing loss** over the first hundreds of steps.
- **Stable gradient norms** (no wild spikes).
- **Consistent step times** (no infrastructure issues).
#### Why It Matters
- Cheapest evaluation signal — available in real-time from W&B.
- Critical for the **30-minute quality check** workflow.
- At later training stages (when loss becomes meaningful), trajectory shape
can predict final model quality.
#### Context: How This Evolves
The team's experience shows evaluation signals change during a project:
- **Early stage**: Loss may be flat or meaningless → focus on SSIM & visual
inspection instead.
- **Mid stage**: Loss starts decreasing → trajectory shape becomes useful.
- **Late stage**: Loss is meaningful → can compare trajectories across runs.
This dynamic is a key insight from the team's workflow: don't over-rely on
loss early; don't ignore it late.
---
### Grad Norm Stability
**Category**: Training health diagnostic
**Status**: ✅ Active (from W&B `grad_norm`)
**Trust**: Medium — diagnostic, not quality metric
#### What It Measures
The magnitude of gradients during training. Stable grad norms indicate
healthy optimization. Spikes or NaN values indicate training instability.
#### Alert Thresholds
| Condition | Meaning |
|-----------|---------|
| Stable ~0.3–0.5 | Normal training |
| Single spike > 3× average | Possible bad batch, monitor |
| NaN or Inf | 🔴 Training has diverged — stop run |
| Increasing trend | Learning rate may be too high |
---
## External Benchmarks
### GameWorld Score Benchmark (Matrix-Game)
**Category**: Multi-dimensional evaluation framework for interactive world models
**Status**: 🟡 External — not implemented in-repo
**Source**: [Matrix-Game 1.0 benchmark](https://github.com/SkyworkAI/Matrix-Game), used in [Matrix-Game 2.0 paper](https://arxiv.org/abs/2508.13009)
#### What It Measures
A comprehensive benchmark examining **four critical capabilities**:
| Dimension | What It Evaluates | Example Signals |
|-----------|-------------------|-----------------|
| **Visual quality** | Frame-level realism, absence of artifacts | Color fidelity, sharpness, coherence |
| **Temporal quality** | Smoothness across frames, motion consistency | Jitter, flickering, temporal aliasing |
| **Action controllability** | Response to input actions (keyboard/mouse) | Action delay, correctness, smoothness |
| **Physical rule understanding** | Adherence to physics (gravity, collision) | Object persistence, plausible motion |
#### Context from Matrix-Game 2.0
- Evaluation uses **597-frame composite action sequences** over 32 Minecraft
scenes and 16 wild scenes.
- Action controllability assessment is **Minecraft-specific** — cannot be
directly applied to wild/general scenes.
- The paper notes that models that "collapse" to static frames can
paradoxically score higher on consistency metrics — beware of this confound.
#### Relevance to FastVideo
- Matrix-Game 2.0 is built on SkyReels-V2/Wan2.1 architecture — **same model
family as FastVideo**.
- Their distillation uses DMD-based Self-Forcing — **same technique** as our
`self_forcing_distillation_pipeline.py`.
- GameWorld Score dimensions are a useful framework for thinking about world
model quality even outside gaming contexts.
---
## Human Preference Evaluation
**Category**: Gold-standard quality assessment
**Status**: 🔴 Manual process — no automated implementation
**Priority**: **Highest** — this is the most important evaluation signal
**Trust**: Highest — but expensive
### What It Measures
Human evaluators compare generated videos and rate them on dimensions like:
- Overall quality and realism
- Temporal coherence and smoothness
- Prompt adherence / action correctness
- Absence of artifacts
#### Why It's the Most Important Metric
All automated metrics are **proxies** for human judgment. They can be gamed
or may miss artifacts that humans easily notice. Human preference is the
ultimate ground truth for video generation quality.
#### Cost & Practicality
| Approach | Cost | Scale | When to Use |
|----------|------|-------|-------------|
| Internal team review | Low | ~10–50 videos | Every major checkpoint |
| Crowdsource (MTurk, Scale) | Medium | 100+ videos | Pre-release validation |
| A/B preference test | Medium | Pairs | Comparing two model versions |
#### Recommended Protocol
1. Sample 10–20 videos from the model at a checkpoint.
2. Include diverse prompts (easy + hard, short + long).
3. Have 2–3 evaluators score each video 1–5 on: quality, coherence, fidelity.
4. Record scores in the experiment journal.
---
## Metrics NOT Used
| Metric | Reason |
|--------|--------|
| ~~CLIP-Score~~ | Not used by the team. Measures text-image alignment using CLIP embeddings, but not well-suited for video temporal quality. |
| Inception Score (IS) | Less informative than FVD for video; primarily an image metric. |
| PSNR | Pixel-level metric; less perceptually meaningful than SSIM/LPIPS. |
---
## Adding a New Metric
Follow the SOP: `.agents/workflows/evaluation-development.md`
1. Prototype in `.agents/exploration/`
2. Validate on known-good and known-bad samples
3. Add to this registry
4. Update the `evaluate-video-quality` skill
@@ -0,0 +1,21 @@
# Experiment Journal
Living log of all experiments. Each entry captures what was tried, the result,
and any insights. Newest entries go at the top.
_No experiments logged yet. Use the `log-experiment` skill to add entries._
<!-- TEMPLATE — copy and fill for each new experiment:
## [YYYY-MM-DD] Experiment: <name>
- **Hypothesis**: <what you expected to learn>
- **Config**: model=..., lr=..., sp_size=..., gpus=..., script=...
- **W&B run**: <run_id or URL>
- **Duration**: <total wall time>
- **Key metrics**: loss=..., step_time=..., grad_norm=...
- **Checkpoint**: <path>
- **Insight**: <what was learned>
- **Status**: running | completed | failed | abandoned
- **Related lessons**: `.agents/lessons/<filename>.md`
-->
+4
View File
@@ -0,0 +1,4 @@
{"name": "codebase-map", "description": "High-level structural index of the FastVideo-WorldModel repository", "path": "codebase-map/README.md", "status": "ready", "trust": "high"}
{"name": "evaluation-registry", "description": "Catalog of all evaluation metrics with detailed explanations, implementation status, and usage guides", "path": "evaluation-registry/README.md", "status": "draft", "trust": "medium"}
{"name": "experiment-journal", "description": "Living log of all experiments with hypotheses, configs, metrics, and insights", "path": "experiment-journal/README.md", "status": "draft", "trust": "medium"}
{"name": "related-work", "description": "Index of related papers, repos, and blog posts with structured comparisons to FastVideo", "path": "related-work/README.md", "status": "draft", "trust": "low"}
+34
View File
@@ -0,0 +1,34 @@
# Related Work Index
Each file in this directory is a structured summary of a related paper, repo,
or blog post relevant to FastVideo-WorldModel training.
## File Format
Each file is named `<slug>.md` and follows this structure:
```markdown
---
title: <paper/repo title>
source: <URL or citation>
type: paper | repo | blog
date_indexed: <ISO-8601>
tags: [world-model, distillation, evaluation, reward-shaping, ...]
---
## Summary
<1-2 paragraph summary of the work.>
## Key Differences from FastVideo
- <Bullet points comparing their approach to ours.>
## Actionable Insights
- <What we could adopt or adapt.>
```
## How to Add New Entries
Use the `index-related-work` skill, or manually create a file following the
template above.
_No related work indexed yet._
+76
View File
@@ -0,0 +1,76 @@
# Agent Onboarding — FastVideo-WorldModel
Welcome, agent. This is the **master onboarding** guide. Follow the steps below,
then check if a **domain-specific onboarding** exists for your task.
## Domain-Specific Onboarding
If your task falls into one of these areas, read the specialized guide **after**
completing the general steps below:
| Domain | Guide | When to Use |
|--------|-------|-------------|
| **WorldModel Training** | `worldmodel-training/README.md` | Training, finetuning, distillation, experiment management |
---
## Step 1: Understand the Codebase
Read these files to build your context:
| Priority | File | What you learn |
|----------|------|----------------|
| 1 | `AGENTS.md` | Coding guidelines, build/test commands, PR conventions |
| 2 | `docs/design/overview.md` | Architecture: models, pipelines, configs, registry |
| 3 | `fastvideo/train/` | Refactored training framework (YAML-driven, modular methods/models/callbacks) |
| 4 | `docs/training/overview.md` | Training data flow and preprocessing |
| 5 | `docs/training/finetune.md` | Training arguments, parallelism, LoRA, validation |
| 6 | `docs/contributing/coding_agents.md` | How to add model pipelines with agent assistance |
## Step 2: Discover Available Resources
Read these two index files to see what skills and memory modules exist:
- **`.agents/skills/index.jsonl`** — catalog of all agent skills (name + description)
- **`.agents/memory/index.jsonl`** — catalog of all memory modules (name + description)
Each entry has a `path` field pointing to the full content. Only load the
full README.md for modules relevant to your current task.
## Step 3: Check for Existing Skills & SOPs
Before writing new code or procedures:
1. **Skills**: Read `.agents/skills/index.jsonl` — find a matching skill by description.
2. **Workflows/SOPs**: Browse `.agents/workflows/` — step-by-step procedures for common tasks.
3. **Lessons**: Browse `.agents/lessons/` — known pitfalls and their fixes.
If a skill or SOP exists for your task, **use it**. If not, you are in **exploration mode** — see Step 4.
## Step 4: Exploration Mode
If no existing skill/SOP covers your task:
1. Document your progress in `.agents/exploration/<topic>.md` using the template in `.agents/exploration/README.md`.
2. At the end of your session, reflect:
- **What worked** → propose a new skill or SOP in the exploration log.
- **What failed** → create a lesson in `.agents/lessons/`.
3. Flag the exploration log for human review.
## Quick Reference
```
.agents/
├── ONBOARDING.md ← you are here
├── STATUS.md ← dashboard: completeness & trust of all components
├── skills/ ← reusable agent skills
├── workflows/ ← SOPs and procedures
├── memory/ ← persistent context (folder per topic + index.jsonl)
│ ├── index.jsonl
│ ├── codebase-map/
│ ├── experiment-journal/
│ ├── evaluation-registry/
│ └── related-work/
├── lessons/ ← mistakes and fixes
└── exploration/ ← draft procedures
```
@@ -0,0 +1,302 @@
# WorldModel Training — Agent Onboarding
Specialized onboarding for agents working on FastVideo-WorldModel training,
distillation, and evaluation. Read the master onboarding (`.agents/onboarding/README.md`)
first, then come here.
---
## Domain Context
FastVideo-WorldModel trains **interactive world models** — video generation systems
that respond to user actions (keyboard/mouse) in real-time. The architecture is
based on **Wan2.1** (SkyReels-V2) DiT models with causal attention for
auto-regressive streaming generation.
**Key techniques you will work with:**
- Full finetuning and LoRA on Wan / LTX-2 / MatrixGame models
- DMD-based distillation (few-step generation)
- Self-Forcing distillation (causal streaming)
- Diffusion-Forcing SFT (DFSFT) for causal models
- VSA (Variable Sparsity Acceleration) for efficient training
---
## Training Code: Two Generations
### New modular framework: `fastvideo/train/` (preferred)
The refactored training code uses a **YAML-only config-driven** architecture
with composable methods, per-role models, and a callback system. All new
training work should use this framework.
### Legacy pipelines: `fastvideo/training/` (deprecated)
The old monolithic pipeline classes (`WanTrainingPipeline`,
`DistillationPipeline`, etc.) still exist but are being phased out. The new
framework imports select utilities from `fastvideo/training/` for backward
compatibility (EMA, gradient clipping, checkpoint wrappers).
---
## Essential Reading (Training-Specific)
Read these **in order** before touching any training code:
| # | File | What You Learn |
|---|------|----------------|
| 1 | `docs/training/overview.md` | Training data flow: raw video → text embeddings + video latents → training |
| 2 | `docs/training/finetune.md` | Training arguments, parallelism (SP/TP), LoRA, validation settings |
| 3 | `docs/training/data_preprocess.md` | How to preprocess datasets into the expected format |
| 4 | `docs/design/overview.md` | Architecture: models, pipelines, configs, registry |
---
## New Training Framework (`fastvideo/train/`)
### Architecture Overview
```
fastvideo/train/
├── __init__.py → exports Trainer
├── trainer.py → main training loop coordinator
├── entrypoint/
│ ├── train.py → YAML-only training entrypoint
│ └── dcp_to_diffusers.py → checkpoint conversion utility
├── methods/ → training algorithms (TrainingMethod ABC)
│ ├── base.py → TrainingMethod base class
│ ├── fine_tuning/
│ │ ├── finetune.py → FineTuneMethod (supervised finetuning)
│ │ └── dfsft.py → DiffusionForcingSFTMethod (causal)
│ ├── distribution_matching/
│ │ ├── dmd2.py → DMD2Method (distribution matching distill)
│ │ └── self_forcing.py → SelfForcingMethod (causal streaming)
│ ├── knowledge_distillation/ → (stub, not yet implemented)
│ └── consistency_model/ → (stub, not yet implemented)
├── models/ → per-role model instances
│ ├── base.py → ModelBase & CausalModelBase (ABC)
│ └── wan/
│ ├── wan.py → WanModel (non-causal)
│ └── wan_causal.py → WanCausalModel (causal streaming)
├── callbacks/ → training hooks & monitoring
│ ├── callback.py → Callback base class + CallbackDict
│ ├── grad_clip.py → GradNormClipCallback
│ ├── ema.py → EMACallback (shadow weights)
│ └── validation.py → ValidationCallback (sampling + eval)
└── utils/ → configuration, building, checkpointing
├── builder.py → build_from_config() (config → runtime)
├── checkpoint.py → CheckpointManager (DCP-based)
├── config.py → load_run_config() (YAML → RunConfig)
├── training_config.py → TypedConfig dataclasses
├── optimizer.py → build_optimizer_and_scheduler()
├── instantiate.py → resolve_target() + instantiate()
├── tracking.py → build_tracker() (W&B, etc.)
├── dataloader.py → dataloader utilities
├── module_state.py → apply_trainable()
└── moduleloader.py → load_module_from_path()
```
### Key Concepts
**TrainingMethod** (`methods/base.py`): Abstract base class for all training
algorithms. Owns role models (student, teacher, critic), manages checkpoint
state, and defines the training step interface.
**ModelBase** (`models/base.py`): Per-role model wrapper. Each role (student,
teacher, critic) gets its own `ModelBase` instance owning a `transformer` and
`noise_scheduler`. `CausalModelBase` extends this for streaming models.
**Callback system** (`callbacks/`): Composable hooks for gradient clipping,
EMA, validation, etc. Configured via YAML, dispatched by `CallbackDict`.
**Config system** (`utils/config.py`, `utils/training_config.py`): YAML files
are parsed into typed `RunConfig` dataclass trees. Models and methods use
`_target_` fields for instantiation (similar to Hydra).
### Training Flow
```
run_training_from_config(config_path)
→ load_run_config() # YAML → RunConfig
→ init_distributed() # TP/SP setup
→ build_from_config() # instantiate models, method, dataloader
→ Trainer.run() # main loop:
├─ callbacks.on_train_start()
├─ checkpoint_manager.maybe_resume()
├─ for step in range(max_steps):
│ ├─ method.single_train_step(batch)
│ ├─ method.backward()
│ ├─ callbacks.on_before_optimizer_step()
│ ├─ method.optimizers_schedulers_step()
│ ├─ tracker.log(metrics, step)
│ ├─ callbacks.on_training_step_end()
│ └─ checkpoint_manager.maybe_save(step)
├─ callbacks.on_train_end()
└─ checkpoint_manager.save_final()
```
### Training Methods
| Method | Class | Use Case |
|--------|-------|----------|
| **FineTune** | `FineTuneMethod` | Single-role supervised finetuning |
| **DFSFT** | `DiffusionForcingSFTMethod` | Diffusion-forcing SFT with inhomogeneous timesteps |
| **DMD2** | `DMD2Method` | Multi-role distribution matching distillation (student + teacher + critic) |
| **Self-Forcing** | `SelfForcingMethod` | Extends DMD2 for causal student rollouts |
### Launching Training (New Framework)
Training is launched via `torchrun` with a single YAML config:
```bash
torchrun --nproc_per_node <N_GPUS> \
-m fastvideo.train.entrypoint.train \
--config examples/train/<config>.yaml
```
### Example YAML Configs
| Config | Method | Description |
|--------|--------|-------------|
| `examples/train/finetune_wan2.1_t2v_1.3B_vsa_phase3.4_0.9sparsity.yaml` | FineTune | Wan 1.3B finetuning with VSA sparsity |
| `examples/train/distill_wan2.1_t2v_1.3B_dmd2.yaml` | DMD2 | Wan 1.3B distillation (student + teacher + critic) |
| `examples/train/dfsft_wan_causal_t2v_1.3B.yaml` | DFSFT | Causal Wan 1.3B diffusion-forcing SFT |
| `examples/train/self_forcing_wan_causal_t2v_1.3B.yaml` | Self-Forcing | Causal streaming distillation |
### Checkpointing (New Framework)
**CheckpointManager** (`utils/checkpoint.py`) saves via `torch.distributed.checkpoint`:
```
output_dir/
└─ checkpoint-{step}/
├─ dcp/ # DCP state dict
├─ config.json # resolved training config
└─ .fastvideo_metadata.json
```
Checkpoint state includes: role model weights, per-role optimizers/schedulers,
CUDA RNG state, and callback state (e.g., EMA shadow weights).
### Config Structure
A YAML config defines the full training pipeline:
```yaml
models:
student:
_target_: fastvideo.train.models.wan.WanModel
model_path: ...
trainable: true
teacher: # optional, for distillation
_target_: fastvideo.train.models.wan.WanModel
model_path: ...
trainable: false
method:
_target_: fastvideo.train.methods.fine_tuning.FineTuneMethod
# method-specific params...
training:
distributed: { num_gpus: 8, tp_size: 1, sp_size: 8 }
data: { data_path: ..., batch_size: 1 }
optimizer: { lr: 1e-5, lr_scheduler: constant_with_warmup }
loop: { max_train_steps: 1000 }
checkpoint: { output_dir: ./outputs }
tracker: { trackers: [wandb], project_name: ... }
callbacks:
grad_clip:
_target_: fastvideo.train.callbacks.GradNormClipCallback
max_grad_norm: 1.0
validation:
_target_: fastvideo.train.callbacks.ValidationCallback
validation_steps: 100
```
---
## Legacy Training Pipelines (`fastvideo/training/`)
> **Note:** Use the new `fastvideo/train/` framework for new work. This section
> is retained for reference on existing pipelines not yet migrated.
| Pipeline | Entrypoint | Use Case |
|----------|-----------|----------|
| Wan T2V finetune | `fastvideo/training/wan_training_pipeline.py` | Standard text-to-video finetune / LoRA |
| Wan I2V finetune | `fastvideo/training/wan_i2v_training_pipeline.py` | Image-to-video (first frame conditioned) |
| MatrixGame finetune | `fastvideo/training/matrixgame_training_pipeline.py` | Action-conditioned world model |
| LTX-2 finetune | `fastvideo/training/ltx2_training_pipeline.py` | LTX-2 architecture finetuning |
| Wan DMD distillation | `fastvideo/training/wan_distillation_pipeline.py` | Few-step distillation via DMD |
| Self-Forcing distill | `fastvideo/training/wan_self_forcing_distillation_pipeline.py` | Causal streaming distillation |
---
## Key Infrastructure
### W&B Integration
- **Tracker**: `fastvideo/training/trackers.py` — `WandbTracker` class
- **New framework tracker**: `fastvideo/train/utils/tracking.py` — `build_tracker()`
- **Env vars**: `WANDB_API_KEY`, `WANDB_BASE_URL`, `WANDB_MODE`
### Parallelism
- **SP** (Sequence Parallel): splits video frames across GPUs — `sp_size: N`
- **TP** (Tensor Parallel): splits model layers across GPUs — `tp_size: N`
- Typical configs: SP=2–8, TP=1–2
---
## Evaluation (for training runs)
Read `.agents/memory/evaluation-registry/README.md` for the full metric catalog.
**Quick summary for training agents:**
| Metric | When to Use | Trust |
|--------|-------------|-------|
| **Loss trajectory** | Every run, real-time from W&B | Medium |
| **SSIM** | When comparing against reference outputs | High |
| **FVD** | For benchmarking model quality (`benchmarks/fvd/`) | High |
| **LPIPS** | LoRA merge validation | Medium |
| **Human preference** | Major checkpoints | Highest |
---
## Common Workflows
| Task | Skill / SOP |
|------|-------------|
| Launch a training run | `.agents/skills/launch-experiment/SKILL.md` |
| Monitor a running experiment | `.agents/skills/monitor-experiment/SKILL.md` |
| Summarize final results | `.agents/skills/summarize-run/SKILL.md` |
| Full experiment lifecycle | `.agents/workflows/experiment-lifecycle.md` |
| Capture lessons from failures | `.agents/workflows/lesson-capture.md` |
---
## World Model–Specific Concepts
### Action Injection (MatrixGame)
The MatrixGame pipeline adds **action modules** to each DiT block, enabling
frame-level mouse/keyboard input conditioning. The action sequence is injected
per-frame alongside the latent video tokens.
### Causal Architecture
For streaming generation, the model uses **causal attention** (each frame only
attends to previous frames). This enables auto-regressive chunk-by-chunk
generation — critical for real-time interactive world models.
### Self-Forcing Distillation
A **data-free** distillation method where the student model is trained to
generate coherent video sequences by being forced to use its own previous
outputs (rather than ground-truth) as context. This produces models robust to
their own error accumulation during long auto-regressive generation.
### DMD Distillation (Distribution Matching Distillation)
Reduces inference steps from ~50 to 3–4 by training a student model to match
the output distribution of the teacher model. Uses a critic network to estimate
distribution divergence.
### Diffusion-Forcing SFT (DFSFT)
Supervised finetuning with **inhomogeneous timesteps** across chunks — each
chunk in a causal sequence can have a different noise level, training the model
to handle mixed-fidelity contexts.
+96
View File
@@ -0,0 +1,96 @@
#!/usr/bin/env bash
# Sync .agents/skills/ into .claude/skills/ via per-skill symlinks.
#
# Why: Claude Code only scans .claude/skills/ and ~/.claude/skills/ for
# user-invocable skills (no skillsPath config exists — see
# https://code.claude.com/docs/en/skills.md). This repo's skills live
# in .agents/skills/ so they travel with the repo and stay under git.
# Run this once after cloning (or after adding/removing a skill) to
# expose them to Claude Code without maintaining a parallel tree.
#
# Usage:
# .agents/scripts/sync-skills.sh
#
# Idempotent and safe to re-run. Prunes stale symlinks whose source
# has been removed from .agents/skills/. Leaves hand-written
# .claude/skills/<name>/ directories untouched (only symlinks are
# managed).
set -euo pipefail
REPO_ROOT="$(git -C "$(dirname "$0")" rev-parse --show-toplevel)"
SRC_DIR="$REPO_ROOT/.agents/skills"
DST_DIR="$REPO_ROOT/.claude/skills"
if [[ ! -d "$SRC_DIR" ]]; then
echo "Error: $SRC_DIR does not exist." >&2
exit 1
fi
mkdir -p "$DST_DIR"
linked=0
unchanged=0
skipped=0
pruned=0
link_skill() {
local name="$1"
local src="$SRC_DIR/$name"
local dst="$DST_DIR/$name"
# Relative target keeps symlinks portable across clones.
local rel="../../.agents/skills/$name"
if [[ -L "$dst" ]]; then
if [[ "$(readlink "$dst")" == "$rel" ]]; then
unchanged=$((unchanged + 1))
return
fi
rm "$dst"
elif [[ -e "$dst" ]]; then
echo "Skipped (not a symlink): .claude/skills/$name" >&2
skipped=$((skipped + 1))
return
fi
ln -s "$rel" "$dst"
echo "Linked: .claude/skills/$name -> $rel"
linked=$((linked + 1))
}
prune_stale() {
local link="$1"
local target
target="$(readlink "$link")"
case "$target" in
../../.agents/skills/*) ;;
*) return ;;
esac
local name="${target##*/}"
if [[ ! -d "$SRC_DIR/$name" ]]; then
rm "$link"
echo "Pruned stale: .claude/skills/$(basename "$link")"
pruned=$((pruned + 1))
fi
}
for src in "$SRC_DIR"/*/; do
[[ -d "$src" ]] || continue
name="$(basename "$src")"
# Only treat directories that actually contain a SKILL.md as skills.
[[ -f "$src/SKILL.md" ]] || continue
link_skill "$name"
done
shopt -s nullglob
for link in "$DST_DIR"/*; do
[[ -L "$link" ]] || continue
prune_stale "$link"
done
shopt -u nullglob
printf "\nSummary: %d linked, %d unchanged, %d pruned" "$linked" "$unchanged" "$pruned"
if [[ "$skipped" -gt 0 ]]; then
printf ", %d skipped (non-symlink collision)" "$skipped"
fi
printf "\n"
+57
View File
@@ -0,0 +1,57 @@
---
name: <skill-name>
description: <one-line description — Codex uses this for implicit invocation matching>
---
# <Skill Name>
## Purpose
<Why this skill exists and when to use it.>
## Prerequisites
- <What must be true before using this skill>
## Inputs
| Parameter | Required | Description |
|-----------|----------|-------------|
| `param1` | Yes | ... |
## Steps
1. **Step 1 title**
- Detail...
2. **Step 2 title**
- Detail...
## Outputs
- <What this skill produces>
## Example Usage
```
<Example invocation or prompt snippet>
```
## References
- <Links to relevant files in the codebase>
---
## Folder Structure
Each skill lives in its own directory under `.agents/skills/`:
```
.agents/skills/<skill-name>/
├── SKILL.md # Required: instructions + metadata (this file)
├── scripts/ # Optional: executable helper scripts
├── references/ # Optional: documentation, papers
└── assets/ # Optional: templates, resources
```
After creating a new skill, add an entry to `.agents/skills/index.jsonl`:
```json
{"name": "<skill-name>", "description": "<description>", "path": "<skill-name>/SKILL.md", "status": "draft", "trust": "low"}
```
@@ -0,0 +1,128 @@
---
name: evaluate-video-quality
description: Evaluate generated video quality using available metrics (SSIM, loss trajectory, caption consistency)
---
# Evaluate Video Quality
## Purpose
Assess the quality of videos generated by a training run. Combines multiple
signals to give a holistic quality assessment. This skill is **evolving** —
new metrics will be added as they are developed.
## Prerequisites
- Generated videos available locally or via W&B artifacts.
- For SSIM: reference videos from official implementations.
- For caption consistency: LLM access (optional, stub for now).
## Inputs
| Parameter | Required | Description |
|-----------|----------|-------------|
| `video_paths` | Yes | List of paths to generated videos |
| `reference_paths` | No | Paths to reference videos (for SSIM) |
| `prompts` | No | Prompts used to generate videos (for caption check) |
| `loss_summary` | No | Path to W&B summary JSON (for loss trajectory) |
| `metrics` | No | Which metrics to run (default: all available) |
## Available Metrics
Check `.agents/memory/evaluation-registry/README.md` for the current catalog.
### SSIM (Active)
Leverages the existing infrastructure in `fastvideo/tests/ssim/`.
```bash
pytest fastvideo/tests/ssim/ -vs --video-path <generated> --reference-path <reference>
```
Or use the SSIM utility directly:
```python
from fastvideo.tests.ssim.ssim_utils import compute_ssim
score = compute_ssim(generated_video, reference_video)
# score > 0.85 is typically "acceptable"
```
**Interpretation**:
| SSIM Range | Quality |
|------------|---------|
| > 0.90 | Excellent — very close to reference |
| 0.80–0.90 | Good — acceptable for most uses |
| 0.70–0.80 | Fair — noticeable differences |
| < 0.70 | Poor — significant quality issues |
### Loss Trajectory (Active)
Analyze the loss curve shape from W&B summary:
```python
import json
with open(loss_summary_path) as f:
summary = json.load(f)
final_loss = summary["train_loss"]
runtime = summary["_runtime"]
steps = summary["_step"]
```
**Early-stage heuristics** (first 500 steps):
- Loss should be decreasing (even slightly).
- Grad norm should be stable (no wild oscillations).
- If loss is flat or increasing, flag for review.
### Caption Consistency (Draft — Not Yet Calibrated)
Use an LLM to evaluate whether the video content matches the input prompt.
```
Prompt: "A golden retriever playing in the snow"
Video: <path>
Score the video on:
1. Object presence (is there a golden retriever?)
2. Action accuracy (is it playing?)
3. Environment match (is there snow?)
4. Overall coherence (does it look natural?)
Each 1-5, total /20.
```
> ⚠️ This metric is in **draft** status. Results should not be treated as
> ground truth until calibrated against human judgments.
## Steps
1. **Identify available metrics** — Check `.agents/memory/evaluation-registry/README.md`.
2. **Run each metric** — Collect scores.
3. **Aggregate** — Produce a combined quality report.
4. **Log** — Update the experiment journal with quality results.
## Outputs
```markdown
## Video Quality Report: <experiment_name>
| Metric | Score | Threshold | Status |
|--------|-------|-----------|--------|
| SSIM (avg) | 0.87 | > 0.80 | ✅ Pass |
| Loss trajectory | decreasing | decreasing | ✅ Pass |
| Caption consistency | 16/20 | > 14/20 | ✅ Pass |
### Per-Video Scores
| Video | SSIM | Caption |
|-------|------|---------|
| video_001.mp4 | 0.89 | 17/20 |
| video_002.mp4 | 0.85 | 15/20 |
```
## References
- `fastvideo/tests/ssim/` — SSIM test infrastructure
- `fastvideo/tests/training/Vanilla/test_training_loss.py` — loss comparison
- `.agents/memory/evaluation-registry/README.md` — metric catalog
## Changelog
| Date | Change |
|------|--------|
| 2026-03-02 | Initial version with SSIM, loss trajectory, caption consistency stub |
@@ -0,0 +1,94 @@
---
name: index-related-work
description: Ingest a paper or repository into the related work index
---
# Index Related Work
## Purpose
Create a structured summary of a related paper, repository, or blog post and
add it to `.agents/memory/related-work/` for future reference. This builds the
agent's knowledge base for making informed decisions about training, evaluation,
and architecture choices.
## Prerequisites
- Access to the paper/repo (URL, PDF, or local clone).
## Inputs
| Parameter | Required | Description |
|-----------|----------|-------------|
| `source` | Yes | URL, citation, or local path |
| `type` | Yes | `paper`, `repo`, or `blog` |
| `tags` | No | List of tags (default: inferred from content) |
## Steps
### 1. Extract key information
For **papers**: Read abstract, method section, experimental setup, and results.
For **repos**: Read README, key source files, and training scripts.
For **blogs**: Read the full post.
Focus on:
- What problem does it solve?
- What architecture/technique is used?
- How does it relate to FastVideo's approach?
### 2. Create the index entry
Write to `.agents/memory/related-work/<slug>.md`:
```markdown
---
title: <title>
source: <URL or citation>
type: paper | repo | blog
date_indexed: <ISO-8601>
tags: [world-model, distillation, evaluation, ...]
---
## Summary
<1-2 paragraph summary.>
## Key Differences from FastVideo
- <comparison points>
## Actionable Insights
- <what we could adopt or adapt>
```
### 3. Update the catalog
If `.agents/memory/related-work/_catalog.md` exists, append the new entry.
If not, create it:
```markdown
# Related Work Catalog
| Slug | Title | Type | Tags | Date |
|------|-------|------|------|------|
| <slug> | <title> | <type> | <tags> | <date> |
```
## Outputs
- New file in `.agents/memory/related-work/<slug>.md`.
- Updated catalog.
## Example Usage
```
Index the Self-Forcing paper:
source: https://arxiv.org/abs/2406.xxxxx
type: paper
tags: [world-model, self-forcing, distillation]
```
## References
- `.agents/memory/related-work/README.md` — schema documentation
## Changelog
| Date | Change |
|------|--------|
| 2026-03-02 | Initial version |
+8
View File
@@ -0,0 +1,8 @@
{"name": "launch-experiment", "description": "Generate and execute a training launch command for FastVideo models", "path": "launch-experiment/SKILL.md", "status": "draft", "trust": "low"}
{"name": "monitor-experiment", "description": "Poll a running W&B training run for progress and emit structured alerts", "path": "monitor-experiment/SKILL.md", "status": "draft", "trust": "low"}
{"name": "summarize-run", "description": "Extract a W&B run summary into a structured experiment report", "path": "summarize-run/SKILL.md", "status": "draft", "trust": "low"}
{"name": "log-experiment", "description": "Append or update an experiment entry in the experiment journal", "path": "log-experiment/SKILL.md", "status": "draft", "trust": "low"}
{"name": "evaluate-video-quality", "description": "Evaluate generated video quality using available metrics (SSIM, loss trajectory, caption consistency)", "path": "evaluate-video-quality/SKILL.md", "status": "draft", "trust": "low"}
{"name": "index-related-work", "description": "Ingest a paper or repository into the related work index", "path": "index-related-work/SKILL.md", "status": "draft", "trust": "low"}
{"name": "search-related-work", "description": "Query the related work index for relevant papers, repos, or comparisons", "path": "search-related-work/SKILL.md", "status": "draft", "trust": "low"}
{"name": "seed-ssim-references", "description": "Run a new or updated fastvideo/tests/ssim/ test on Modal, pull generated videos, and upload them to FastVideo/ssim-reference-videos so the test has a regression baseline", "path": "seed-ssim-references/SKILL.md", "status": "draft", "trust": "low"}
+127
View File
@@ -0,0 +1,127 @@
---
name: launch-experiment
description: Generate and execute a training launch command for FastVideo models
---
# Launch Experiment
## Purpose
Construct a fully-specified `torchrun` training command for a FastVideo model
given a target pipeline, dataset, and hyperparameter overrides. This skill
automates the boilerplate of setting environment variables, picking the right
entrypoint, and applying defaults from the closest example script.
## Prerequisites
- The repo is cloned and `fastvideo` is installed (`uv pip install -e .[dev]`).
- Dataset is preprocessed (see `docs/training/data_preprocess.md`).
- `WANDB_API_KEY` is set in the environment (or `WANDB_MODE=offline` for local).
- GPU resources are available (multi-GPU requires NCCL).
## Inputs
| Parameter | Required | Description |
|-----------|----------|-------------|
| `pipeline` | Yes | Training pipeline type: `finetune`, `distill-dmd`, `self-forcing`, `lora`, `consistency` |
| `model` | Yes | Model family: `wan-t2v-1.3B`, `wan-i2v-14B`, `ltx2`, `matrixgame` |
| `data_path` | Yes | Path to preprocessed dataset (parquet) |
| `num_gpus` | Yes | Number of GPUs |
| `overrides` | No | Dict of hyperparameter overrides (any CLI arg) |
| `output_dir` | No | Output directory (default: `outputs/<model>_<pipeline>`) |
| `run_name` | No | W&B run name (default: auto-generated) |
## Steps
### 1. Identify the training entrypoint
| Pipeline | Entrypoint |
|----------|-----------|
| `finetune` (Wan T2V) | `fastvideo/training/wan_training_pipeline.py` |
| `finetune` (Wan I2V) | `fastvideo/training/wan_i2v_training_pipeline.py` |
| `finetune` (LTX-2) | `fastvideo/training/ltx2_training_pipeline.py` |
| `finetune` (MatrixGame) | `fastvideo/training/matrixgame_training_pipeline.py` |
| `distill-dmd` | `fastvideo/training/wan_distillation_pipeline.py` |
| `self-forcing` | `fastvideo/training/wan_self_forcing_distillation_pipeline.py` |
### 2. Resolve default hyperparameters
Find the closest example script in `examples/training/` for the model:
| Model | Example Script Directory |
|-------|-------------------------|
| `wan-t2v-1.3B` | `examples/training/finetune/wan_t2v_1.3B/crush_smol/` |
| `wan-i2v-14B` | `examples/training/finetune/wan_i2v_14B_480p/crush_smol/` |
| `ltx2` | `examples/training/finetune/ltx2/` |
| `matrixgame` | `examples/training/finetune/MatrixGame2.0/` |
| `distill-dmd` | `scripts/distill/v1_distill_dmd_wan.sh` |
Read the script to extract default values for:
- `--learning_rate`, `--train_batch_size`, `--sp_size`, `--tp_size`
- `--num_latent_t`, `--num_height`, `--num_width`, `--num_frames`
- `--gradient_accumulation_steps`, `--max_train_steps`
- `--mixed_precision`, `--weight_decay`, `--max_grad_norm`
- `--validation_steps`, `--validation_sampling_steps`
### 3. Set environment variables
```bash
export WANDB_API_KEY="${WANDB_API_KEY}"
export WANDB_BASE_URL="https://api.wandb.ai"
export FASTVIDEO_ATTENTION_BACKEND=FLASH_ATTN
export TOKENIZERS_PARALLELISM=false
export TRITON_CACHE_DIR=/tmp/triton_cache
```
### 4. Construct the torchrun command
```bash
torchrun --nnodes 1 --nproc_per_node <num_gpus> \
<entrypoint> \
--pretrained_model_name_or_path <model_hf_id> \
--data_path "<data_path>" \
--output_dir "<output_dir>" \
--wandb_run_name "<run_name>" \
--tracker_project_name "<project_name>" \
--log_validation \
<...all hyperparameters...>
```
### 5. Log to experiment journal
After launching, append an entry to `.agents/memory/experiment-journal/README.md`:
```markdown
## [YYYY-MM-DD] Experiment: <run_name>
- **Hypothesis**: <user-provided or auto-generated>
- **Config**: model=<model>, lr=<lr>, sp_size=<sp>, gpus=<n>, script=<entrypoint>
- **W&B run**: <pending — will be updated by monitor skill>
- **Status**: running
```
## Outputs
- A ready-to-execute shell command.
- An experiment journal entry.
## Example Usage
```
Launch a Wan T2V 1.3B finetune on 4 GPUs with lr=5e-5 and max_train_steps=1000:
pipeline: finetune
model: wan-t2v-1.3B
data_path: data/crush_smol_preprocessed/
num_gpus: 4
overrides:
learning_rate: 5e-5
max_train_steps: 1000
```
## References
- `examples/training/finetune/wan_t2v_1.3B/crush_smol/finetune_t2v.sh`
- `scripts/distill/v1_distill_dmd_wan.sh`
- `docs/training/finetune.md` (training arguments table)
- `fastvideo/training/trackers.py` (tracker initialization)
## Changelog
| Date | Change |
|------|--------|
| 2026-03-02 | Initial version |
+87
View File
@@ -0,0 +1,87 @@
---
name: log-experiment
description: Append or update an experiment entry in the experiment journal
---
# Log Experiment
## Purpose
Create or update an entry in `.agents/memory/experiment-journal/README.md` to maintain
a living record of all experiments and their outcomes.
## Prerequisites
- `.agents/memory/experiment-journal/README.md` exists.
## Inputs
| Parameter | Required | Description |
|-----------|----------|-------------|
| `name` | Yes | Experiment name / identifier |
| `hypothesis` | No | What you expected to learn |
| `config` | Yes | Key config: model, lr, sp_size, gpus, script |
| `wandb_run` | No | W&B run ID or URL |
| `duration` | No | Total wall time |
| `metrics` | No | Key metrics dict (loss, step_time, grad_norm) |
| `checkpoint` | No | Path to checkpoint |
| `insight` | No | What was learned |
| `status` | Yes | `running`, `completed`, `failed`, `abandoned` |
| `lessons` | No | Paths to related lesson files |
## Steps
### 1. Check for existing entry
Search `.agents/memory/experiment-journal/README.md` for an entry with the same name.
If found, update it instead of creating a duplicate.
### 2. Format the entry
```markdown
## [YYYY-MM-DD] Experiment: <name>
- **Hypothesis**: <hypothesis or "N/A">
- **Config**: model=<model>, lr=<lr>, sp_size=<sp>, gpus=<n>, script=<script>
- **W&B run**: <wandb_run or "pending">
- **Duration**: <duration or "in progress">
- **Key metrics**: loss=<loss>, step_time=<step_time>, grad_norm=<grad_norm>
- **Checkpoint**: <checkpoint or "N/A">
- **Insight**: <insight or "pending">
- **Status**: <status>
- **Related lessons**: <lessons or "none">
```
### 3. Insert at the top of the journal
New entries go at the top of the file (after the header), so the most recent
experiments are always visible first.
### 4. Warn on duplicates
If a similar experiment name exists with `status: completed`, warn that this
may be a repeat. If it's `status: running`, assume this is an update.
## Outputs
- Updated `.agents/memory/experiment-journal/README.md`.
## Example Usage
```
Log a completed experiment:
name: wan-t2v-finetune-lr5e5-sp4
config: model=wan-t2v-1.3B, lr=5e-5, sp_size=4, gpus=4
wandb_run: fastvideo/training/run_abc123
duration: 2h 15m
metrics: {loss: 0.065, step_time: 2.3, grad_norm: 0.35}
checkpoint: outputs/wan_finetune/checkpoint-1000
insight: LR 5e-5 converges 30% faster than 1e-5 with no quality loss
status: completed
```
## References
- `.agents/memory/experiment-journal/README.md` — journal file
- `.agents/workflows/experiment-lifecycle.md` — when to log
## Changelog
| Date | Change |
|------|--------|
| 2026-03-02 | Initial version |
+134
View File
@@ -0,0 +1,134 @@
---
name: monitor-experiment
description: Poll a running W&B training run for progress and emit structured alerts
---
# Monitor Experiment
## Purpose
Continuously (or on-demand) check a running experiment's W&B metrics and emit
alerts for anomalies. Supports the "30-minute quality check" paradigm: after
the first 30 minutes of a long training run, produce a checkpoint quality
report before committing more resources.
## Prerequisites
- `WANDB_API_KEY` is set in the environment.
- The experiment is actively logging to W&B (not in `WANDB_MODE=offline`).
- For offline mode: read from local `wandb-summary.json` instead.
## Inputs
| Parameter | Required | Description |
|-----------|----------|-------------|
| `run_id` | Yes* | W&B run ID (e.g., `entity/project/run_id`) |
| `output_dir` | Yes* | Local output directory (for offline mode fallback) |
| `poll_interval` | No | Seconds between polls (default: 60) |
| `alert_on` | No | List of alert conditions to enable (default: all) |
\* One of `run_id` or `output_dir` is required.
## Steps
### 1. Connect to the run
**Online mode** (preferred):
```python
import wandb
api = wandb.Api()
run = api.run("<run_id>")
```
**Offline fallback**:
```python
import json
summary_path = f"{output_dir}/tracker/wandb/latest-run/files/wandb-summary.json"
with open(summary_path) as f:
summary = json.load(f)
```
### 2. Track key metrics
| Metric | W&B Key | Description |
|--------|---------|-------------|
| Training loss | `train_loss` | Primary training loss |
| Gradient norm | `grad_norm` | Gradient magnitude |
| Step time | `step_time` | Wall-clock seconds per step |
| Learning rate | `learning_rate` | Current LR |
| Avg step time | `avg_step_time` | Running average step time |
| Validation videos | `validation_videos_*` | Generated validation samples |
### 3. Evaluate alert conditions
| Alert | Condition | Severity |
|-------|-----------|----------|
| **Loss spike** | `current_loss > 3 × rolling_avg_loss` | 🔴 Critical |
| **NaN/Inf gradient** | `grad_norm` is NaN or Inf | 🔴 Critical |
| **Step time regression** | `step_time > 2 × baseline_step_time` | 🟡 Warning |
| **No progress** | No new W&B logs for > 10 minutes | 🟡 Warning |
| **Loss plateau** | Loss change < 1% over last 100 steps | 🟢 Info |
### 4. Emit structured status
Output format (agent-consumable):
```json
{
"run_id": "...",
"step": 500,
"metrics": {
"train_loss": 0.078,
"grad_norm": 0.41,
"step_time": 2.5,
"learning_rate": 1e-6
},
"alerts": [
{"type": "loss_spike", "severity": "critical", "message": "Loss jumped to 0.45 (avg: 0.08)"}
],
"status": "running"
}
```
### 5. 30-Minute Quality Check
After the first 30 minutes of wall-clock time:
1. Summarize the loss curve shape (decreasing? at what rate?).
2. Check if validation videos have been generated.
3. Report step count, loss at start vs. current, and estimated time to completion.
4. Produce a go/no-go recommendation.
```markdown
## 30-Minute Check: <run_name>
- **Steps completed**: 150
- **Loss**: 0.12 → 0.08 (↓ 33%)
- **Grad norm**: stable at ~0.4
- **Step time**: 2.5s/step (consistent)
- **Validation videos**: 5 generated at step 100
- **Recommendation**: ✅ Continue — loss is decreasing normally
```
## Outputs
- Structured JSON status updates.
- Alert messages for anomalous conditions.
- 30-minute checkpoint quality report.
## Example Usage
```
Monitor W&B run "fastvideo/Wan_distillation/abc123":
run_id: fastvideo/Wan_distillation/abc123
poll_interval: 120
alert_on: [loss_spike, nan_gradient, step_time_regression]
```
## References
- `fastvideo/training/trackers.py` — `WandbTracker` implementation
- `fastvideo/tests/training/Vanilla/test_training_loss.py` — how summaries are compared
- `fastvideo/tests/training/Vanilla/a40_reference_wandb_summary.json` — reference summary format
## Changelog
| Date | Change |
|------|--------|
| 2026-03-02 | Initial version |
@@ -0,0 +1,82 @@
---
name: search-related-work
description: Query the related work index for relevant papers, repos, or comparisons
---
# Search Related Work
## Purpose
Search through `.agents/memory/related-work/` to find indexed papers, repos,
or blog posts relevant to a query. Use this when you need to understand how
other work compares to FastVideo's approach, or when looking for techniques
to adopt.
## Prerequisites
- The related work index has entries (`.agents/memory/related-work/*.md`).
## Inputs
| Parameter | Required | Description |
|-----------|----------|-------------|
| `query` | Yes | Natural language query |
| `tags` | No | Filter by tags (e.g., `[distillation, evaluation]`) |
| `type` | No | Filter by type (`paper`, `repo`, `blog`) |
## Steps
### 1. Search the index
Use grep-based search through `.agents/memory/related-work/`:
```bash
# Search by content
grep -rl "<query>" .agents/memory/related-work/
# Search by tags (in frontmatter)
grep -l "tags:.*<tag>" .agents/memory/related-work/*.md
```
### 2. Rank results
For each matching file:
1. Read the file.
2. Score relevance to the query based on:
- Title match
- Tag match
- Content match (summary, differences, insights)
3. Return top results.
### 3. Format output
```markdown
## Related Work Search: "<query>"
### 1. <Title> (relevance: high)
- **Source**: <URL>
- **Tags**: <tags>
- **Key insight**: <most relevant excerpt>
- **File**: `.agents/memory/related-work/<slug>.md`
### 2. <Title> (relevance: medium)
...
```
## Outputs
- Ranked list of relevant related work entries with excerpts.
## Example Usage
```
Search for work related to video quality evaluation metrics:
query: "video generation quality evaluation metrics"
tags: [evaluation]
```
## References
- `.agents/memory/related-work/README.md` — index schema
## Changelog
| Date | Change |
|------|--------|
| 2026-03-02 | Initial version |
@@ -0,0 +1,250 @@
---
name: seed-ssim-references
description: Seed HF reference videos for a single newly-added SSIM test. Runs the test on Modal L40S, downloads the generated mp4s via `modal volume get`, pauses for the user to eyeball quality, then uploads only that test's files to `FastVideo/ssim-reference-videos`. Use when a new `fastvideo/tests/ssim/test_*_similarity.py` has just been added and has no references on HF yet.
---
# Seed SSIM Reference Videos
## Purpose
A brand-new SSIM test in `fastvideo/tests/ssim/` fails forever until its
reference videos exist on the HF dataset (`FastVideo/ssim-reference-videos`).
This skill:
1. Runs the test on Modal's L40S pool to generate the videos.
2. Downloads them to the local repo via `modal volume get`.
3. Pauses so the user can eyeball the mp4s and confirm quality.
4. Uploads only the new test's files to HF, with a guard that refuses to
overwrite anything already present.
The skill is run **manually**, once per new test. Before invoking it, the user
has already sanity-tested the new test locally — it launches `VideoGenerator`
and writes an mp4 without crashing. The skill does not re-test locally; it
goes straight to Modal L40S (which is what CI uses).
## When to use
- A new `test_*_similarity.py` file has been added in `fastvideo/tests/ssim/`
and the HF dataset has no `reference_videos/default/L40S_reference_videos/<model_id>/`
subtree for it yet.
## When not to use
- Regular CI runs — once refs exist, `pytest fastvideo/tests/ssim/` downloads
them automatically.
- Re-seeding an existing test. That requires `--force` on the upload step, and
is out of scope here; treat as a separate, deliberate operation.
## Inputs
The skill has **one required input**: the path to the new SSIM test file.
Prompt the user for it if they didn't supply it.
| Parameter | Required | Description |
|-----------|----------|-------------|
| `test_file` | Yes | e.g. `fastvideo/tests/ssim/test_ltx2_similarity.py`. The skill's first action is to ask for this if missing. |
Everything else is fixed:
- Modal runner GPU: **L40S** (hardcoded in `fastvideo/tests/modal/ssim_test.py`).
- Device folder: `L40S_reference_videos`.
- Quality tier: `default` (the tier CI runs). The `full_quality` tier is not
seeded by this skill.
- HF repo: `FastVideo/ssim-reference-videos` (dataset).
- Multi-model test files: all model ids in `*_MODEL_TO_PARAMS` are seeded
together; the Modal run produces one mp4 per (model, prompt, backend) and
the upload scopes by `--model-id`, looping if there is more than one.
## Prerequisites
The user has confirmed:
- `modal` CLI authenticated.
- `HF_API_KEY` (or `HUGGINGFACE_HUB_TOKEN` / `HF_TOKEN`) exported with write
access to `FastVideo/ssim-reference-videos`.
- The test file runs locally end-to-end (generates an mp4; SSIM assertion
failure due to missing reference is expected and fine).
Fail fast if the token env var is missing.
## Steps
### 1. Ask for the test file
If the user didn't name one, ask: *"Which SSIM test file do you want to seed
references for? (e.g. `fastvideo/tests/ssim/test_ltx2_similarity.py`)"*.
Validate:
- Path exists and matches `fastvideo/tests/ssim/test_*_similarity.py`.
- File defines a `*_MODEL_TO_PARAMS` dict — grep it to extract the set of
model ids. Those ids drive step 5.
If either check fails, stop and tell the user what's wrong.
### 2. Run the test on Modal L40S
Pick a subdir name so repeated runs don't collide:
```bash
SHORT_COMMIT=$(git rev-parse --short=12 HEAD)
TIMESTAMP=$(date -u +%Y%m%d_%H%M%S)
SUBDIR="${TIMESTAMP}_${SHORT_COMMIT}"
```
Then launch the Modal run:
```bash
modal run fastvideo/tests/modal/ssim_test.py \
--git-repo="$(git config --get remote.origin.url)" \
--git-commit="$(git rev-parse HEAD)" \
--hf-api-key="$HF_API_KEY" \
--test-files="<test_file>" \
--sync-generated-to-volume \
--generated-volume-subdir="$SUBDIR" \
--skip-reference-download \
--no-fail-fast
```
Flag rationale:
- `--skip-reference-download`: no refs exist yet, so conftest must not try to
pull them.
- `--no-fail-fast`: lets the test finish generation before `_assert_similarity`
raises `FileNotFoundError: Reference video folder does not exist`. The
expected failure is what we want — the mp4 has already been written.
- `--sync-generated-to-volume` + `--generated-volume-subdir`: copies the
generated mp4s to the `hf-model-weights` Modal volume under
`ssim_generated_videos/default/<SUBDIR>/generated_videos/` so we can pull
them locally.
The Modal run will end with a nonzero exit (expected) and print a
`modal volume get hf-model-weights ssim_generated_videos/default/<SUBDIR>/generated_videos ./generated_videos_modal/default`
command. Capture that `<SUBDIR>` — you need it for step 3.
### 3. Download generated videos locally
```bash
modal volume get --force hf-model-weights \
ssim_generated_videos/default/"$SUBDIR"/generated_videos \
./generated_videos_modal/default
```
`--force` is required when the parent `./generated_videos_modal/default`
already exists; without it, `modal volume get` errors with `[Errno 21] Is a
directory`. Safe to pass on the first run too.
After this, the mp4s live at
`./generated_videos_modal/default/generated_videos/L40S_reference_videos/<model_id>/<backend>/<prompt>.mp4`.
The extra `generated_videos/` level comes from the volume layout in
`_sync_generated_videos_to_volume` (`ssim_test.py`) — the command copies
`<repo>/fastvideo/tests/ssim/generated_videos/<tier>` to
`ssim_generated_videos/<tier>/<SUBDIR>/generated_videos/`, and `modal volume
get` preserves that trailing `generated_videos/` segment.
### 4. PAUSE — user reviews quality
Print the list of downloaded mp4s and their paths, then stop. Tell the user:
> "Generated videos downloaded to `./generated_videos_modal/default/generated_videos/L40S_reference_videos/`. Please open them and confirm the quality looks correct. Reply **`upload`** to continue, or anything else to abort."
Do not proceed until the user explicitly says `upload`. If they abort, leave
everything on disk so they can inspect further — no cleanup.
### 5. Copy into the local reference layout
Scoped copy — only the new test's mp4s. Loop over each `<model_id>` extracted
in step 1:
```bash
python fastvideo/tests/ssim/reference_videos_cli.py copy-local \
--quality-tier default \
--device-folder L40S_reference_videos \
--generated-dir ./generated_videos_modal/default/generated_videos/L40S_reference_videos
```
(The `--generated-dir` points at the device-folder root inside the
downloaded tree; `copy-local` walks all `<model>/<backend>/*.mp4`
underneath it. Since the Modal run was scoped to a single test file via
`--test-files`, only that test's model(s) are present — so the copy is
implicitly per-test.)
Result: `fastvideo/tests/ssim/reference_videos/default/L40S_reference_videos/<model_id>/<backend>/<prompt>.mp4`.
### 6. Upload to HF — scoped per model_id, with overwrite guard
For each `<model_id>`:
```bash
python fastvideo/tests/ssim/reference_videos_cli.py upload \
--quality-tier default \
--device-folder L40S_reference_videos \
--model-id "<model_id>"
```
The upload command:
- Uploads **only** `reference_videos/default/L40S_reference_videos/<model_id>/`.
- **Refuses** if any file already exists at that path on HF (this is the
guard — seeding a new test should never clobber existing refs). To override,
the user must re-run with `--force`. If the guard fires, stop and report
exactly which files exist; do not silently `--force`.
Reads the HF token from `HF_API_KEY` / `HUGGINGFACE_HUB_TOKEN` / `HF_TOKEN`.
### 7. Report success
List what was uploaded (paths in repo) and remind the user to push any
related code changes. Do **not** auto-verify by re-running Modal — the user
can run `pytest fastvideo/tests/ssim/<test_file>` later to confirm end-to-end;
it will auto-download the refs they just uploaded.
## Failure modes and how to handle them
- **`HF_API_KEY` unset.** Stop before step 2. The Modal run needs it (passed
via `--hf-api-key`), and step 6 needs it for upload.
- **Modal run fails before generation.** No mp4s on the volume — nothing to
download. Fix the test locally (`pytest fastvideo/tests/ssim/<test_file>`)
and retry from step 2.
- **`./generated_videos_modal/default/L40S_reference_videos/` missing after
`modal volume get`.** The run didn't produce videos (most likely the test
crashed before writing, or `REQUIRED_GPUS` exceeded the partition capacity
— see Modal logs).
- **Upload guard fires (files already exist).** The test name / model id
collides with something already on HF. Verify the user actually wants to
replace existing refs; if so, re-run the upload with `--force`. If not,
rename the model id in `*_MODEL_TO_PARAMS` and re-seed.
- **Quality looks wrong in step 4.** Abort. The mp4s stay on disk for
inspection. The fix is usually in the test's params (resolution, steps,
seed) — edit the test, then re-run the skill.
## Design notes (for future skill maintainers)
- The skill deliberately runs on Modal, **not** locally, because the CI
runner is L40S. Seeding from a different GPU SKU produces refs that CI's
L40S runs can't match (SSIM drifts across SKUs).
- The skill is default-tier only. `full_quality` refs are seeded by a
separate, deliberate operation — they double runtime and aren't what CI
gates on.
- The overwrite guard in `reference_videos_cli.py upload` is default-on
specifically because this skill exists. Re-seeding is a distinct operation
that requires explicit `--force`.
## References
- `fastvideo/tests/modal/ssim_test.py` — Modal orchestrator; see
`--sync-generated-to-volume`, `--generated-volume-subdir`,
`--skip-reference-download`, `--no-fail-fast`.
- `fastvideo/tests/ssim/reference_videos_cli.py` — `copy-local`, `upload`
(with `--model-id`, `--force`), `download`, `ensure` subcommands.
- `fastvideo/tests/ssim/README.md` — reference layout, HF repo conventions.
- `fastvideo/tests/ssim/inference_similarity_utils.py` —
`run_text_to_video_similarity_test` + `_build_init_kwargs`: what each test
config passes to `VideoGenerator.from_pretrained`.
## Changelog
| Date | Change |
|------|--------|
| 2026-04-17 | Initial version (Modal sync-to-volume flow). |
| 2026-04-21 | Rewrite: single-test scope, explicit user-review pause, per-`model_id` upload, HF overwrite guard. Dropped `scripts/seed_ssim.sh`. |
| 2026-04-21 | Post-first-run fixes: `modal volume get` needs `--force` when parent exists; download tree has an extra `generated_videos/` level so `--generated-dir` must reflect it. |
+137
View File
@@ -0,0 +1,137 @@
---
name: summarize-run
description: Extract a W&B run summary into a structured experiment report
---
# Summarize Run
## Purpose
After a training run completes (or at any checkpoint), extract key metrics from
the W&B run summary and produce a structured markdown report. Supports both
online (W&B API) and offline (local `wandb-summary.json`) modes.
## Prerequisites
- Run has completed or reached a checkpoint with a saved summary.
- For online: `WANDB_API_KEY` set in environment.
- For offline: access to `<output_dir>/tracker/wandb/latest-run/files/wandb-summary.json`.
## Inputs
| Parameter | Required | Description |
|-----------|----------|-------------|
| `run_id` | Yes* | W&B run ID for online access |
| `output_dir` | Yes* | Local output dir for offline access |
| `reference_run` | No | Path to reference `wandb-summary.json` for comparison |
| `experiment_name` | No | Name for the journal entry (default: from W&B) |
\* One of `run_id` or `output_dir` is required.
## Steps
### 1. Load run summary
**Online**:
```python
import wandb
api = wandb.Api()
run = api.run("<run_id>")
summary = dict(run.summary)
config = dict(run.config)
```
**Offline** (existing codebase pattern from `fastvideo/tests/training/`):
```python
import json
summary_path = f"{output_dir}/tracker/wandb/latest-run/files/wandb-summary.json"
with open(summary_path) as f:
summary = json.load(f)
```
### 2. Extract key fields
| Field | Source | Description |
|-------|--------|-------------|
| `train_loss` | `summary["train_loss"]` | Final training loss |
| `avg_step_time` | `summary["avg_step_time"]` | Average seconds per step |
| `step_time` | `summary["step_time"]` | Last step time |
| `grad_norm` | `summary["grad_norm"]` | Final gradient norm |
| `learning_rate` | `summary["learning_rate"]` | Final LR |
| `_step` | `summary["_step"]` | Total steps completed |
| `_runtime` | `summary["_runtime"]` | Total wall-clock seconds |
| `validation_videos_*` | `summary[key]` | Validation video artifacts |
### 3. Compare against reference (optional)
Follow the pattern in `fastvideo/tests/training/Vanilla/test_training_loss.py`:
```python
# Fields to compare
compare_fields = ["train_loss", "grad_norm", "avg_step_time"]
tolerance = 0.05 # 5% relative tolerance
for field in compare_fields:
ref_val = reference_summary[field]
cur_val = summary[field]
diff_pct = abs(cur_val - ref_val) / abs(ref_val) * 100
status = "✅" if diff_pct < tolerance * 100 else "⚠️"
print(f"{status} {field}: {cur_val:.4f} (ref: {ref_val:.4f}, diff: {diff_pct:.1f}%)")
```
### 4. Generate report
```markdown
# Run Summary: <experiment_name>
| Metric | Value | Reference | Diff |
|--------|-------|-----------|------|
| Train Loss | 0.0788 | 0.0800 | -1.5% ✅ |
| Avg Step Time | 2.81s | 2.80s | +0.4% ✅ |
| Grad Norm | 0.408 | 0.410 | -0.5% ✅ |
| Total Steps | 500 | — | — |
| Wall Time | 23m 30s | — | — |
## Configuration
- Model: Wan-AI/Wan2.1-T2V-1.3B-Diffusers
- Learning Rate: 1e-6
- Batch Size: 1
- GPUs: 8 × (SP=1, TP=1)
- Mixed Precision: bf16
## Validation Videos
<list of validation video paths if available>
## Notes
<any observations or anomalies>
```
### 5. Update experiment journal
Append or update the experiment's entry in `.agents/memory/experiment-journal/README.md`
with the final metrics and status.
## Outputs
- Structured markdown report.
- Updated experiment journal entry.
## Example Usage
```
Summarize the run in output directory "outputs/wan_finetune":
output_dir: outputs/wan_finetune
reference_run: fastvideo/tests/training/Vanilla/a40_reference_wandb_summary.json
experiment_name: wan-t2v-finetune-lr1e6
```
## References
- `fastvideo/tests/training/Vanilla/test_training_loss.py` — reference comparison pattern
- `fastvideo/tests/training/Vanilla/a40_reference_wandb_summary.json` — example summary
- `fastvideo/tests/training/lora/test_lora_training.py` — LoRA summary comparison
- `fastvideo/training/trackers.py` — tracker summary generation
## Changelog
| Date | Change |
|------|--------|
| 2026-03-02 | Initial version |
@@ -0,0 +1,54 @@
---
description: How to develop, validate, and register a new evaluation metric
---
# Evaluation Development SOP
Standard procedure for adding new video quality evaluation metrics to the
FastVideo agent toolkit.
## When to Use
- You need a metric that doesn't exist in `.agents/memory/evaluation-registry/README.md`.
- An existing metric needs significant changes to its methodology.
- You're exploring a new evaluation approach.
## Steps
### 1. Research
- Search `.agents/memory/related-work/` for existing evaluation approaches.
- Check the `evaluation_registry.md` for current metrics and their limitations.
- Review literature: FVD, CLIP-Score, human preference, etc.
### 2. Prototype
- Write a standalone script in `.agents/exploration/<metric-name>.md`.
- Keep it simple: one script, minimal dependencies.
- Test on a few known-good and known-bad video samples.
### 3. Validate
- **Known-good test**: Metric should score high on reference-quality videos.
- **Known-bad test**: Metric should score low on degraded/unrelated videos.
- **Sensitivity test**: Small quality differences should produce meaningful
score differences.
- Document thresholds and their justification.
### 4. Register
Update `.agents/memory/evaluation-registry/README.md`:
- Add the metric with status `Active`.
- Document location, thresholds, and trust level.
### 5. Integrate
Update `.agents/skills/evaluate-video-quality.md`:
- Add the new metric as a section.
- Include code examples and interpretation guide.
### 6. Document
- Move the exploration log content into the skill.
- Clean up the exploration file or mark it as `promoted`.
- If anything went wrong during development, create a lesson.
@@ -0,0 +1,47 @@
---
description: When and how to log experiments in the experiment journal
---
# Experiment Journaling SOP
Ensures every experiment is properly recorded with context and outcomes.
## When to Log
**Always.** Every experiment — even quick tests — should be journaled.
## Steps
### 1. Before Launch — Create Draft Entry
Use the `log-experiment` skill with `status: running`:
- Include hypothesis and config.
- Leave metrics, duration, and insight blank.
### 2. After 30-Minute Check — Update with Initial Metrics
Update the entry with:
- Current loss and its trajectory direction.
- Step time.
- Number of validation videos generated.
- Preliminary go/no-go assessment.
### 3. On Completion — Fill Final Entry
Update the entry with `status: completed`:
- Final loss, grad norm, avg step time.
- Total duration and steps.
- Checkpoint path.
- Key insight.
### 4. On Failure — Document Failure Mode
Update the entry with `status: failed`:
- What went wrong (OOM, NaN, crash, etc.).
- At what step the failure occurred.
- Create a lesson in `.agents/lessons/` for non-trivial failures.
### 5. Cross-Reference
- Link related lessons: `**Related lessons**: .agents/lessons/<filename>.md`
- Link related experiments: if this is a follow-up, reference the prior entry.
+87
View File
@@ -0,0 +1,87 @@
---
description: End-to-end experiment lifecycle from hypothesis to lessons learned
---
# Experiment Lifecycle SOP
Standard operating procedure for running ML training experiments on
FastVideo-WorldModel. Every experiment should follow this flow.
## Overview
```
Plan → Launch → Monitor → Summarize → Journal → Reflect
```
## Steps
### 1. Plan the Experiment
Before launching:
- [ ] Define a clear **hypothesis** (what you expect to learn).
- [ ] Select the **model** and **pipeline** type (finetune, distill, lora, etc.).
- [ ] Prepare the **dataset** (preprocessed into parquet format).
- [ ] Review existing experiments in `.agents/memory/experiment-journal/README.md` for related work.
- [ ] Check `.agents/lessons/` for known pitfalls with this configuration.
- [ ] Document the plan in the experiment journal as a draft entry.
### 2. Launch the Experiment
Use the `launch-experiment` skill:
- Provide: pipeline, model, data_path, num_gpus, and any hyperparameter overrides.
- The skill generates the `torchrun` command and creates a journal entry.
- Verify the command looks correct before executing.
Reference: `.agents/skills/launch-experiment.md`
### 3. Monitor the Experiment
Use the `monitor-experiment` skill:
- Provide the W&B run ID (or output_dir for offline).
- Monitor alerts: loss spikes, NaN gradients, step time regressions.
- At the **30-minute mark**: perform the quality check.
- Is loss decreasing?
- Are validation videos reasonable?
- Is step time consistent?
- **Decision point**: Continue or abort based on the 30-min check.
Reference: `.agents/skills/monitor-experiment.md`
### 4. Summarize the Run
After completion (or at any checkpoint), use the `summarize-run` skill:
- Extract final metrics from W&B summary.
- Compare against reference runs if available.
- Generate a structured report.
Reference: `.agents/skills/summarize-run.md`
### 5. Update the Experiment Journal
Use the `log-experiment` skill to update the journal entry:
- Fill in final metrics, duration, checkpoint paths.
- Record the key insight learned.
- Set status to `completed`, `failed`, or `abandoned`.
Reference: `.agents/skills/log-experiment.md`
### 6. Reflect and Capture Lessons
After every experiment:
- **What went right?** → Note in the journal insight field.
- **What went wrong?** → Create a lesson in `.agents/lessons/`:
- Use the template in `.agents/lessons/README.md`.
- Cross-reference the experiment journal entry.
- **What was surprising?** → Consider creating an exploration log if this
warrants further investigation.
Reference: `.agents/workflows/lesson-capture.md`
## Validation Criteria
This SOP is validated when an agent can:
1. Follow steps 1–6 end-to-end for a minimal training run
(e.g., `examples/training/finetune/wan_t2v_1.3B/crush_smol/finetune_t2v.sh`
with `--max_train_steps 5`).
2. Produce a complete experiment journal entry.
3. Generate a run summary report.
+71
View File
@@ -0,0 +1,71 @@
---
description: Post-experiment reflection to capture lessons learned
---
# Lesson Capture SOP
Systematic procedure for turning experiment outcomes into persistent knowledge.
## When to Use
After **every** completed or failed experiment. Even successful experiments
can yield lessons (e.g., "LR 5e-5 works better than 1e-5 for LoRA").
## Steps
### 1. Review the Experiment
Read the experiment journal entry. Ask:
- Did anything go wrong?
- Was anything surprising?
- Did anything take longer than expected?
- Was a workaround needed?
### 2. Decide: Lesson or Not?
| Situation | Action |
|-----------|--------|
| Something broke | Create a lesson (category: `infrastructure` or `data`) |
| Hyperparameter choice mattered | Create a lesson (category: `hyperparameter`) |
| Porting issue found | Create a lesson (category: `porting`) |
| Evaluation metric was misleading | Create a lesson (category: `evaluation`) |
| Everything went smoothly | No lesson needed, but note in the journal insight |
### 3. Create the Lesson File
In `.agents/lessons/`, create `<YYYY-MM-DD>_<short-slug>.md`:
```markdown
---
date: <ISO-8601>
experiment: <journal entry reference>
category: hyperparameter | data | infrastructure | evaluation | porting
severity: critical | important | minor
---
# <Short Descriptive Title>
## What Happened
<description>
## Root Cause
<analysis>
## Fix / Workaround
<resolution>
## Prevention
<how to avoid in future>
```
### 4. Cross-Reference
- Update the experiment journal entry with a link to the lesson file.
- If a similar lesson already exists, add a reference or update it.
### 5. Periodic Pattern Review
Every ~10 lessons, scan for patterns:
- Multiple lessons in the same category → consider a new skill or SOP.
- Repeated mistakes → strengthen the relevant SOP with a checklist item.
- Infrastructure issues → propose a codebase fix.
+67
View File
@@ -0,0 +1,67 @@
---
description: Synchronize the STATUS.md dashboard by scanning .agents/ directories
---
# Sync Dashboard
Updates `.agents/STATUS.md` by scanning the skills, workflows, memory, lessons,
and exploration directories to reflect what actually exists on disk.
## When to Use
- After adding, removing, or renaming any file in `.agents/`.
- Periodically (e.g., at end of each conversation session).
- When the dashboard feels out of date.
## Steps
### 1. Scan directories
List all files in each directory:
```bash
echo "=== Skills ==="
ls -1 .agents/skills/*.md 2>/dev/null | grep -v SKILL_TEMPLATE
echo "=== Workflows ==="
ls -1 .agents/workflows/*.md 2>/dev/null
echo "=== Memory ==="
ls -1 .agents/memory/*.md 2>/dev/null
ls -1 .agents/memory/related-work/*.md 2>/dev/null | grep -v README
echo "=== Lessons ==="
ls -1 .agents/lessons/*.md 2>/dev/null | grep -v README
echo "=== Exploration ==="
ls -1 .agents/exploration/*.md 2>/dev/null | grep -v README
```
### 2. Compare with STATUS.md
For each file found:
- If it's in STATUS.md → leave it (preserve status/trust/tested fields).
- If it's NOT in STATUS.md → add it with status `🔴 Stub`, trust `None`, tested `❌`.
For each entry in STATUS.md:
- If the file no longer exists → mark it as `❌ Removed` or delete the row.
### 3. Update counts
Recalculate the summary table at the top:
- Count files per category.
- Count by status (Ready, Draft, Stub).
### 4. Update timestamp
Set `_Last synced: <current date>_` at the top of STATUS.md.
### 5. Review
Read through the updated STATUS.md for accuracy. Flag anything that looks wrong.
## Notes
- Do NOT change trust levels during sync — those are set manually after testing.
- Do NOT change status during sync — status changes require actual validation.
- This workflow only handles structural sync (file existence), not content review.
@@ -0,0 +1,46 @@
{
"benchmark_id": "wan-t2v-1.3b-2gpu",
"description": "Wan2.1 T2V 1.3B inference performance",
"model": {
"model_path": "Wan-AI/Wan2.1-T2V-1.3B-Diffusers",
"model_short_name": "Wan2.1-T2V-1.3B"
},
"init_kwargs": {
"num_gpus": 2,
"flow_shift": 7.0,
"sp_size": 2,
"tp_size": 1,
"vae_sp": true,
"vae_tiling": true,
"text_encoder_precisions": ["fp32"]
},
"generation_kwargs": {
"height": 480,
"width": 832,
"num_frames": 45,
"num_inference_steps": 4,
"guidance_scale": 3,
"embedded_cfg_scale": 6,
"seed": 1024,
"fps": 24,
"neg_prompt": "Bright tones, overexposed, static, blurred details, subtitles, style, works, paintings, images, static, overall gray, worst quality, low quality, JPEG compression residue, ugly, incomplete, extra fingers, poorly drawn hands, poorly drawn faces, deformed, disfigured, misshapen limbs, fused fingers, still picture, messy background, three legs, many people in the background, walking backwards"
},
"test_prompts": [
"Will Smith casually eats noodles, his relaxed demeanor contrasting with the energetic background of a bustling street food market. The scene captures a mix of humor and authenticity. Mid-shot framing, vibrant lighting."
],
"run_config": {
"num_warmup_runs": 2,
"num_measurement_runs": 5,
"required_gpus": 2
},
"thresholds": {
"L40S": {
"max_generation_time_s": 34.0,
"max_peak_memory_mb": 11000.0
},
"default": {
"max_generation_time_s": 120.0,
"max_peak_memory_mb": 30000.0
}
}
}
+438
View File
@@ -0,0 +1,438 @@
env:
IMAGE_VERSION: "py3.12-latest"
BUILDKITE_CLEAN_CHECKOUT: true
notify:
- github_commit_status:
context: "fastcheck-passed"
if: build.env("TEST_SCOPE") == "fastcheck" || build.env("TEST_SCOPE") == null
- github_commit_status:
context: "full-suite-passed"
if: build.env("TEST_SCOPE") == "full"
- github_commit_status:
context: "direct-test-completed"
if: build.env("TEST_SCOPE") == "direct"
steps:
# ============================================================
# Direct test: triggered by /test <name> slash command.
# Labels match fastcheck/full-suite counterparts so the GitHub
# check status overwrites the original failed check.
# Only ONE step executes per build (gated by TEST_TYPE).
# ============================================================
# --- Fastcheck-scope direct tests ---
- label: ":microscope: Encoder Tests"
if: build.env("TEST_SCOPE") == "direct" && build.env("TEST_TYPE") == "encoder"
command: "timeout 90m .buildkite/scripts/pr_test.sh"
retry:
automatic:
- exit_status: 128
limit: 3
- exit_status: -1
limit: 2
agents:
queue: "default"
- label: ":microscope: VAE Tests"
if: build.env("TEST_SCOPE") == "direct" && build.env("TEST_TYPE") == "vae"
command: "timeout 90m .buildkite/scripts/pr_test.sh"
retry:
automatic:
- exit_status: 128
limit: 3
- exit_status: -1
limit: 2
agents:
queue: "default"
- label: ":microscope: Transformer Tests"
if: build.env("TEST_SCOPE") == "direct" && build.env("TEST_TYPE") == "transformer"
command: "timeout 90m .buildkite/scripts/pr_test.sh"
retry:
automatic:
- exit_status: 128
limit: 3
- exit_status: -1
limit: 2
agents:
queue: "default"
- label: ":microscope: Kernel Tests"
if: build.env("TEST_SCOPE") == "direct" && build.env("TEST_TYPE") == "kernel_tests"
command: "timeout 90m .buildkite/scripts/pr_test.sh"
retry:
automatic:
- exit_status: 128
limit: 3
- exit_status: -1
limit: 2
agents:
queue: "default"
- label: ":microscope: Unit Tests"
if: build.env("TEST_SCOPE") == "direct" && build.env("TEST_TYPE") == "unit_test"
command: "timeout 90m .buildkite/scripts/pr_test.sh"
retry:
automatic:
- exit_status: 128
limit: 3
- exit_status: -1
limit: 2
agents:
queue: "default"
# --- Full-suite-scope direct tests ---
- label: ":bar_chart: SSIM Tests"
if: build.env("TEST_SCOPE") == "direct" && build.env("TEST_TYPE") == "ssim"
command: "timeout 90m .buildkite/scripts/pr_test.sh"
retry:
automatic:
- exit_status: 128
limit: 3
- exit_status: -1
limit: 2
- exit_status: 1
limit: 2
agents:
queue: "default"
- label: ":test_tube: LoRA Inference Tests"
if: build.env("TEST_SCOPE") == "direct" && build.env("TEST_TYPE") == "inference_lora"
command: "timeout 90m .buildkite/scripts/pr_test.sh"
retry:
automatic:
- exit_status: 128
limit: 3
- exit_status: -1
limit: 2
agents:
queue: "default"
- label: ":test_tube: Training Tests"
if: build.env("TEST_SCOPE") == "direct" && build.env("TEST_TYPE") == "training"
command: "timeout 90m .buildkite/scripts/pr_test.sh"
retry:
automatic:
- exit_status: 128
limit: 3
- exit_status: -1
limit: 2
agents:
queue: "default"
- label: ":test_tube: Distillation DMD Tests"
if: build.env("TEST_SCOPE") == "direct" && build.env("TEST_TYPE") == "distillation_dmd"
command: "timeout 90m .buildkite/scripts/pr_test.sh"
retry:
automatic:
- exit_status: 128
limit: 3
- exit_status: -1
limit: 2
agents:
queue: "default"
- label: ":test_tube: Self-Forcing Tests"
if: build.env("TEST_SCOPE") == "direct" && build.env("TEST_TYPE") == "self_forcing"
command: "timeout 90m .buildkite/scripts/pr_test.sh"
retry:
automatic:
- exit_status: 128
limit: 3
- exit_status: -1
limit: 2
agents:
queue: "default"
- label: ":test_tube: LoRA Training Tests"
if: build.env("TEST_SCOPE") == "direct" && build.env("TEST_TYPE") == "training_lora"
command: "timeout 90m .buildkite/scripts/pr_test.sh"
retry:
automatic:
- exit_status: 128
limit: 3
- exit_status: -1
limit: 2
- exit_status: 1
limit: 2
agents:
queue: "default"
- label: ":test_tube: Training Tests VSA"
if: build.env("TEST_SCOPE") == "direct" && build.env("TEST_TYPE") == "training_vsa"
command: "timeout 90m .buildkite/scripts/pr_test.sh"
retry:
automatic:
- exit_status: 128
limit: 3
- exit_status: -1
limit: 2
- exit_status: 1
limit: 2
agents:
queue: "default"
- label: ":test_tube: Inference Tests VMoBA"
if: build.env("TEST_SCOPE") == "direct" && build.env("TEST_TYPE") == "inference_vmoba"
command: "timeout 90m .buildkite/scripts/pr_test.sh"
retry:
automatic:
- exit_status: 128
limit: 3
- exit_status: -1
limit: 2
agents:
queue: "default"
- label: ":test_tube: Performance Tests"
if: build.env("TEST_SCOPE") == "direct" && build.env("TEST_TYPE") == "performance"
command: "timeout 90m .buildkite/scripts/pr_test.sh"
retry:
automatic:
- exit_status: 128
limit: 3
- exit_status: -1
limit: 2
agents:
queue: "default"
- label: ":test_tube: API Server Tests"
if: build.env("TEST_SCOPE") == "direct" && build.env("TEST_TYPE") == "api_server"
command: "timeout 90m .buildkite/scripts/pr_test.sh"
retry:
automatic:
- exit_status: 128
limit: 3
- exit_status: -1
limit: 2
agents:
queue: "default"
# ============================================================
# Fastcheck: Runs on every PR (~10-15 min parallel)
# Core component validation: encoders, VAEs, transformers,
# CUDA kernels, and unit tests.
# ============================================================
- label: "Trigger Fastcheck"
if: build.env("TEST_SCOPE") == "fastcheck" || build.env("TEST_SCOPE") == null
retry:
automatic:
- exit_status: 128
limit: 3
- exit_status: -1
limit: 2
plugins:
- monorepo-diff#v1.4.0:
diff: 'git fetch origin "${BUILDKITE_PULL_REQUEST_BASE_BRANCH:-main}" && git diff --name-only "origin/${BUILDKITE_PULL_REQUEST_BASE_BRANCH:-main}...HEAD"'
watch:
- path:
- "fastvideo/models/encoders/**"
- "fastvideo/models/loader/**"
- "fastvideo/tests/encoders/**"
- "pyproject.toml"
- "docker/Dockerfile.python3.12"
config:
command: "timeout 20m .buildkite/scripts/pr_test.sh"
label: ":microscope: Encoder Tests"
env:
- TEST_TYPE=encoder
agents:
queue: "default"
- path:
- "fastvideo/models/vaes/**"
- "fastvideo/models/loader/**"
- "fastvideo/tests/vaes/**"
- "pyproject.toml"
- "docker/Dockerfile.python3.12"
config:
command: "timeout 20m .buildkite/scripts/pr_test.sh"
label: ":microscope: VAE Tests"
env:
- TEST_TYPE=vae
agents:
queue: "default"
- path:
- "fastvideo/models/dits/**"
- "fastvideo/models/loader/**"
- "fastvideo/tests/transformers/**"
- "fastvideo/layers/**"
- "fastvideo/attention/**"
- "pyproject.toml"
- "docker/Dockerfile.python3.12"
config:
command: "timeout 15m .buildkite/scripts/pr_test.sh"
label: ":microscope: Transformer Tests"
env:
- TEST_TYPE=transformer
agents:
queue: "default"
- path:
- "fastvideo-kernel/**"
- "pyproject.toml"
- "docker/Dockerfile.python3.12"
config:
command: "timeout 15m .buildkite/scripts/pr_test.sh"
label: ":microscope: Kernel Tests"
env:
- TEST_TYPE=kernel_tests
agents:
queue: "default"
- path:
- "fastvideo/**"
- ".buildkite/**"
- ".github/**"
- "pyproject.toml"
- "docker/Dockerfile.python3.12"
config:
command: "timeout 15m .buildkite/scripts/pr_test.sh"
label: ":microscope: Unit Tests"
env:
- TEST_TYPE=unit_test
agents:
queue: "default"
# ============================================================
# Full Suite: Runs when TEST_SCOPE=full
# Triggered by adding the 'ready' label (via ci-trigger-full-suite.yml)
# or on-demand via /test full slash command.
# Includes integration tests, SSIM regression, training pipelines,
# and performance benchmarks.
# ============================================================
- label: "Trigger Full Suite"
if: build.env("TEST_SCOPE") == "full"
retry:
automatic:
- exit_status: 128
limit: 3
- exit_status: -1
limit: 2
plugins:
- monorepo-diff#v1.4.0:
diff: 'git fetch origin "${BUILDKITE_PULL_REQUEST_BASE_BRANCH:-main}" && git diff --name-only "origin/${BUILDKITE_PULL_REQUEST_BASE_BRANCH:-main}...HEAD"'
watch:
- path:
- "fastvideo/**/*.py"
- "pyproject.toml"
- "docker/Dockerfile.python3.12"
config:
command: "timeout 90m .buildkite/scripts/pr_test.sh"
label: ":bar_chart: SSIM Tests"
env:
- TEST_TYPE=ssim
retry:
automatic:
- exit_status: 1
limit: 2
agents:
queue: "default"
- path:
- "fastvideo/tests/lora/**"
- "fastvideo/models/loader/**"
- "fastvideo/tests/transformers/**"
- "fastvideo/pipelines/**"
- "fastvideo/layers/lora/**"
- "pyproject.toml"
- "docker/Dockerfile.python3.12"
config:
command: "timeout 20m .buildkite/scripts/pr_test.sh"
label: ":test_tube: LoRA Inference Tests"
env:
- TEST_TYPE=inference_lora
agents:
queue: "default"
- path:
- "fastvideo/**"
- "pyproject.toml"
- "docker/Dockerfile.python3.12"
config:
command: "timeout 15m .buildkite/scripts/pr_test.sh"
label: ":test_tube: Training Tests"
env:
- TEST_TYPE=training
agents:
queue: "default"
- path:
- "fastvideo/training/*distillation_pipeline.py"
- "pyproject.toml"
- "docker/Dockerfile.python3.12"
config:
command: "timeout 15m .buildkite/scripts/pr_test.sh"
label: ":test_tube: Distillation DMD Tests"
env:
- TEST_TYPE=distillation_dmd
agents:
queue: "default"
- path:
- "fastvideo/training/*self_forcing_distillation_pipeline.py"
- "fastvideo/tests/training/self-forcing/**"
- "pyproject.toml"
- "docker/Dockerfile.python3.12"
config:
command: "timeout 15m .buildkite/scripts/pr_test.sh"
label: ":test_tube: Self-Forcing Tests"
env:
- TEST_TYPE=self_forcing
agents:
queue: "default"
- path:
- "fastvideo/**"
- "pyproject.toml"
- "docker/Dockerfile.python3.12"
config:
command: "timeout 15m .buildkite/scripts/pr_test.sh"
label: ":test_tube: LoRA Training Tests"
env:
- TEST_TYPE=training_lora
retry:
automatic:
- exit_status: 1
limit: 2
agents:
queue: "default"
- path:
- "fastvideo/**"
- "fastvideo-kernel/**"
- "pyproject.toml"
- "docker/Dockerfile.python3.12"
config:
command: "timeout 15m .buildkite/scripts/pr_test.sh"
label: ":test_tube: Training Tests VSA"
env:
- TEST_TYPE=training_vsa
retry:
automatic:
- exit_status: 1
limit: 2
agents:
queue: "default"
- path:
- "fastvideo-kernel/**"
- "fastvideo/attention/backends/vmoba.py"
- "pyproject.toml"
- "docker/Dockerfile.python3.12"
config:
command: "timeout 15m .buildkite/scripts/pr_test.sh"
label: ":test_tube: Inference Tests VMoBA"
env:
- TEST_TYPE=inference_vmoba
agents:
queue: "default"
- path:
- "fastvideo/models/dits/**"
- "fastvideo/pipelines/**"
- "fastvideo/attention/**"
- "fastvideo/layers/**"
- "fastvideo/worker/**"
- "fastvideo/entrypoints/**"
- "fastvideo/tests/performance/**"
- ".buildkite/performance-benchmarks/**"
- "pyproject.toml"
- "docker/Dockerfile.python3.12"
config:
command: "timeout 30m .buildkite/scripts/pr_test.sh"
label: ":test_tube: Performance Tests"
env:
- TEST_TYPE=performance
agents:
queue: "default"
- path:
- "fastvideo/entrypoints/openai/**"
- "fastvideo/entrypoints/cli/serve.py"
- "fastvideo/tests/entrypoints/test_openai_api_integration.py"
- "pyproject.toml"
- "docker/Dockerfile.python3.12"
config:
command: "timeout 30m .buildkite/scripts/pr_test.sh"
label: ":test_tube: API Server Tests"
env:
- TEST_TYPE=api_server
agents:
queue: "default"
+222
View File
@@ -0,0 +1,222 @@
#!/bin/bash
set -uo pipefail
log() {
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1"
}
log "=== Starting Modal test execution ==="
# Change to the project directory
cd "$(dirname "$0")/../.."
PROJECT_ROOT=$(pwd)
log "Project root: $PROJECT_ROOT"
# Install Modal if not available
if ! python3 -m modal --version &> /dev/null; then
log "Modal not found, installing..."
python3 -m pip install modal
# Verify installation
if ! python3 -m modal --version &> /dev/null; then
log "Error: Failed to install modal. Please install it manually."
exit 1
fi
fi
log "modal version: $(python3 -m modal --version)"
# Set up Modal authentication using Buildkite secrets
log "Setting up Modal authentication from Buildkite secrets..."
MODAL_TOKEN_ID=$(buildkite-agent secret get modal_token_id)
MODAL_TOKEN_SECRET=$(buildkite-agent secret get modal_token_secret)
# Retrieve other secrets
WANDB_API_KEY=$(buildkite-agent secret get wandb_api_key)
HF_API_KEY=$(buildkite-agent secret get hf_api_key)
if [ -n "$MODAL_TOKEN_ID" ] && [ -n "$MODAL_TOKEN_SECRET" ]; then
log "Retrieved Modal credentials from Buildkite secrets"
python3 -m modal token set --token-id "$MODAL_TOKEN_ID" --token-secret "$MODAL_TOKEN_SECRET" --profile buildkite-ci --activate --verify
if [ $? -eq 0 ]; then
log "Modal authentication successful"
else
log "Error: Failed to set Modal credentials"
exit 1
fi
else
log "Error: Could not retrieve Modal credentials from Buildkite secrets."
log "Please ensure 'modal_token_id' and 'modal_token_secret' secrets are set in Buildkite."
exit 1
fi
MODAL_TEST_FILE="fastvideo/tests/modal/pr_test.py"
MODAL_SSIM_TEST_FILE="fastvideo/tests/modal/ssim_test.py"
if [ -z "${TEST_TYPE:-}" ]; then
log "Error: TEST_TYPE environment variable is not set"
exit 1
fi
log "Test type: $TEST_TYPE"
EFFECTIVE_PR=${BUILDKITE_PULL_REQUEST:-false}
if [ "$EFFECTIVE_PR" = "false" ] && [ -n "${PR_NUMBER:-}" ]; then
EFFECTIVE_PR=$PR_NUMBER
fi
MODAL_ENV="BUILDKITE_REPO=$BUILDKITE_REPO BUILDKITE_COMMIT=$BUILDKITE_COMMIT BUILDKITE_PULL_REQUEST=$EFFECTIVE_PR BUILDKITE_BRANCH=${BUILDKITE_BRANCH:-} TEST_SCOPE=${TEST_SCOPE:-} IMAGE_VERSION=$IMAGE_VERSION"
POST_RUN_HOOK=""
upload_performance_artifacts() {
SHORT_SHA=${BUILDKITE_COMMIT:0:7}
LOCAL_DIR="downloaded_reports"
_download_reports() {
log "Downloading perf_reports/ from Modal Volume..."
mkdir -p "$LOCAL_DIR"
if ! modal volume get hf-model-weights "perf_reports/" "$LOCAL_DIR"; then
log "Error: Failed to download perf_reports/ from Modal Volume."
return 1
fi
}
_upload_dashboard() {
local target
target=$(find "$LOCAL_DIR" -name "dashboard_*${SHORT_SHA}*" | head -n 1)
log "TARGET dashboard: '$target'"
if [ -n "$target" ]; then
log "Found dashboard: $target. Uploading to Buildkite..."
buildkite-agent artifact upload "$target"
buildkite-agent annotate --style info --context "perf-dashboard" < "$target"
else
log "Warning: Could not find a dashboard file matching $SHORT_SHA"
fi
}
_upload_perf_summary() {
local target
target=$(find "$LOCAL_DIR" -name "perf_*${SHORT_SHA}*" | head -n 1)
log "TARGET perf summary: '$target'"
if [ -n "$target" ]; then
log "Found perf summary: $target. Uploading to Buildkite..."
buildkite-agent artifact upload "$target"
buildkite-agent annotate --style info --context "perf-summary" < "$target"
else
log "Warning: Could not find a perf summary file matching $SHORT_SHA"
fi
}
_cleanup_modal_volume() {
log "Cleaning up perf_reports/ from Modal Volume..."
if modal volume rm hf-model-weights "perf_reports/" --recursive; then
log "Successfully deleted perf_reports/ from Modal Volume."
else
log "Warning: Failed to delete perf_reports/ from Modal Volume. Manual cleanup may be required."
fi
}
_cleanup_local() {
log "Cleaning up local download directory..."
rm -rf "$LOCAL_DIR"
}
# --- Main flow ---
_download_reports || { _cleanup_local; return 1; }
_upload_dashboard
_upload_perf_summary
_cleanup_modal_volume
_cleanup_local
}
case "$TEST_TYPE" in
"encoder")
log "Running encoder tests..."
MODAL_COMMAND="$MODAL_ENV HF_API_KEY=$HF_API_KEY python3 -m modal run $MODAL_TEST_FILE::run_encoder_tests"
;;
"vae")
log "Running VAE tests..."
MODAL_COMMAND="$MODAL_ENV HF_API_KEY=$HF_API_KEY python3 -m modal run $MODAL_TEST_FILE::run_vae_tests"
;;
"transformer")
log "Running transformer tests..."
MODAL_COMMAND="$MODAL_ENV HF_API_KEY=$HF_API_KEY python3 -m modal run $MODAL_TEST_FILE::run_transformer_tests"
;;
"ssim")
log "Running SSIM tests..."
MODAL_COMMAND="$MODAL_ENV HF_API_KEY=$HF_API_KEY python3 -m modal run $MODAL_SSIM_TEST_FILE::run_ssim_tests"
;;
"training")
log "Running training tests..."
MODAL_COMMAND="$MODAL_ENV WANDB_API_KEY=$WANDB_API_KEY python3 -m modal run $MODAL_TEST_FILE::run_training_tests"
;;
"training_lora")
log "Running LoRA training tests..."
MODAL_COMMAND="$MODAL_ENV WANDB_API_KEY=$WANDB_API_KEY python3 -m modal run $MODAL_TEST_FILE::run_training_lora_tests"
;;
"training_vsa")
log "Running training VSA tests..."
MODAL_COMMAND="$MODAL_ENV WANDB_API_KEY=$WANDB_API_KEY python3 -m modal run $MODAL_TEST_FILE::run_training_tests_VSA"
;;
"kernel_tests")
log "Running kernel tests..."
MODAL_COMMAND="$MODAL_ENV python3 -m modal run $MODAL_TEST_FILE::run_kernel_tests"
;;
"inference_lora")
log "Running LoRA tests..."
MODAL_COMMAND="$MODAL_ENV python3 -m modal run $MODAL_TEST_FILE::run_inference_lora_tests"
;;
"distillation_dmd")
log "Running distillation DMD tests..."
MODAL_COMMAND="$MODAL_ENV WANDB_API_KEY=$WANDB_API_KEY python3 -m modal run $MODAL_TEST_FILE::run_distill_dmd_tests"
;;
# run_inference_tests_vmoba
"self_forcing")
log "Running self-forcing tests..."
MODAL_COMMAND="$MODAL_ENV WANDB_API_KEY=$WANDB_API_KEY python3 -m modal run $MODAL_TEST_FILE::run_self_forcing_tests"
;;
"inference_vmoba")
log "Running V-MoBA inference tests..."
MODAL_COMMAND="$MODAL_ENV python3 -m modal run $MODAL_TEST_FILE::run_inference_tests_vmoba"
;;
"unit_test")
log "Running unit tests..."
MODAL_COMMAND="$MODAL_ENV python3 -m modal run $MODAL_TEST_FILE::run_unit_test"
;;
"lora_extraction")
log "Running LoRA extraction tests..."
MODAL_COMMAND="$MODAL_ENV HF_API_KEY=$HF_API_KEY python3 -m modal run $MODAL_TEST_FILE::run_lora_extraction_tests"
;;
"performance")
log "Running performance tests on Modal..."
MODAL_COMMAND="$MODAL_ENV HF_API_KEY=$HF_API_KEY python3 -m modal run $MODAL_TEST_FILE::run_performance_tests"
POST_RUN_HOOK="upload_performance_artifacts"
;;
"api_server")
log "Running API server integration tests..."
MODAL_COMMAND="$MODAL_ENV HF_API_KEY=$HF_API_KEY python3 -m modal run $MODAL_TEST_FILE::run_api_server_tests"
;;
*)
log "Error: Unknown test type: $TEST_TYPE"
exit 1
;;
esac
log "Executing: $MODAL_COMMAND"
eval "$MODAL_COMMAND"
TEST_EXIT_CODE=$?
if [ $TEST_EXIT_CODE -eq 0 ]; then
log "Modal test completed successfully"
else
log "Error: Modal test failed with exit code: $TEST_EXIT_CODE"
fi
if [ -n "$POST_RUN_HOOK" ]; then
log "Executing post-run hook: $POST_RUN_HOOK"
"$POST_RUN_HOOK"
fi
log "=== Test execution completed with exit code: $TEST_EXIT_CODE ==="
exit $TEST_EXIT_CODE
+40
View File
@@ -0,0 +1,40 @@
#!/bin/bash
set -uo pipefail
log() {
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1"
}
log "=== Starting pre-commit checks ==="
cd "$(dirname "$0")/../.."
PROJECT_ROOT=$(pwd)
log "Project root: $PROJECT_ROOT"
if ! python3 -m pre_commit --version &> /dev/null; then
log "pre-commit not found, installing..."
python3 -m pip install --user pre-commit==4.0.1
if ! python3 -m pre_commit --version &> /dev/null; then
log "Error: Failed to install pre-commit."
exit 1
fi
fi
log "Pre-commit version: $(python3 -m pre_commit --version)"
log "Installing/updating pre-commit hooks..."
python3 -m pre_commit install --install-hooks
log "Running pre-commit checks on all files..."
python3 -m pre_commit run --all-files
PRE_COMMIT_EXIT_CODE=$?
if [ $PRE_COMMIT_EXIT_CODE -eq 0 ]; then
log "Pre-commit checks completed successfully"
else
log "Error: Pre-commit checks failed with exit code: $PRE_COMMIT_EXIT_CODE"
fi
log "=== Pre-commit checks completed with exit code: $PRE_COMMIT_EXIT_CODE ==="
exit $PRE_COMMIT_EXIT_CODE
+29
View File
@@ -0,0 +1,29 @@
name: 🐞 Bug report
description: Create a report to help us reproduce and fix the bug
title: "[Bug] "
labels: ['Bug']
body:
- type: textarea
attributes:
label: Describe the bug
description: A clear and concise description of what the bug is.
validations:
required: true
- type: textarea
attributes:
label: Reproduction
description: |
What command or script did you run? Which **model** are you using?
placeholder: |
A placeholder for the command.
validations:
required: true
- type: textarea
attributes:
label: Environment
description: |
Please share your environment with us. You can run the command **python collect_env.py** and copy-paste its output below.
placeholder: FastVideo version, platform, python version, cuda version...
validations:
required: true
@@ -0,0 +1,17 @@
name: 🚀 Feature request
description: Suggest an idea for this project
title: "[Feature] "
body:
- type: textarea
attributes:
label: Motivation
description: |
A clear and concise description of the motivation of the feature.
validations:
required: true
- type: textarea
attributes:
label: Related resources
description: |
If there is an official code release or third-party implementations, please also provide the information here, which would be very helpful.
+56
View File
@@ -0,0 +1,56 @@
name: 💬 Request for comments (RFC).
description: Ask for feedback on major architectural changes or design choices.
title: "[RFC]: "
labels: ["RFC"]
body:
- type: markdown
attributes:
value: >
#### Please take a look at previous [RFCs](https://github.com/hao-ai-lab/FastVideo/issues?q=label%3ARFC+sort%3Aupdated-desc) for reference.
- type: textarea
attributes:
label: Motivation.
description: >
The motivation of the RFC.
validations:
required: true
- type: textarea
attributes:
label: Proposed Change.
description: >
The proposed change of the RFC.
validations:
required: true
- type: textarea
attributes:
label: Feedback Period.
description: >
The feedback period of the RFC. Usually at least one week.
validations:
required: false
- type: textarea
attributes:
label: CC List.
description: >
The list of people you want to CC.
validations:
required: false
- type: textarea
attributes:
label: Any Other Things.
description: >
Any other things you would like to mention.
validations:
required: false
- type: markdown
attributes:
value: >
Thanks for contributing 🎉!
- type: checkboxes
id: askllm
attributes:
label: Before submitting a new issue...
options:
- label: Make sure you already searched for relevant issues.
required: true
+1
View File
@@ -0,0 +1 @@
blank_issues_enabled: false
+62
View File
@@ -0,0 +1,62 @@
<!--
PR TITLE: Must start with a type tag, e.g.:
[feat] Add new model [bugfix] Fix VAE tiling [refactor] Restructure pipeline
[perf] Optimize kernel [ci] Update tests [docs] Add guide
[misc] Cleanup configs [new-model] Port Flux2
MERGE WORKFLOW:
1. Ensure pre-commit passes and you have at least 1 approval
2. Comment /merge (or add the "ready" label) to enter the Merge Queue
3. Full Test Suite runs automatically on a staging branch → auto-merge on success
ON-DEMAND TESTING (write access required):
/test full — Full Test Suite /test ssim — SSIM regression
/test training — Training pipeline /test encoder — Encoder tests
/test transformer — Transformer tests /test vae — VAE tests
/test kernel — CUDA kernel tests /test unit — Unit tests
See docs/contributing/pull_requests.md for all 17 test commands
-->
## Purpose
<!-- What does this PR do? Link the related issue if applicable. -->
Fixes #
## Changes
<!-- Describe your changes concisely. What approach did you take? -->
-
## Test Plan
<!-- How did you verify your changes? Paste exact commands and output. -->
```bash
# Commands you ran
```
## Test Results
<!-- Paste test output, before/after comparisons, or SSIM scores for model changes. -->
<details>
<summary>Test output</summary>
```
# Paste output here
```
</details>
## Checklist
- [ ] I ran `pre-commit run --all-files` and fixed all issues
- [ ] I added or updated tests for my changes
- [ ] I updated documentation if needed
- [ ] I considered GPU memory impact of my changes
**For model/pipeline changes, also check:**
- [ ] I verified SSIM regression tests pass
- [ ] I updated the support matrix if adding a new model
+316
View File
@@ -0,0 +1,316 @@
merge_protections:
- name: PR merge requirements
if:
- base = main
success_conditions:
- "title~=(?i)^\\[(feat|feature|bugfix|fix|refactor|perf|ci|doc|docs|misc|chore|kernel|new.?model)\\]"
- "#approved-reviews-by>=1"
- check-success~=pre-commit
- check-success=fastcheck-passed
- check-success=full-suite-passed
pull_request_rules:
# ============================================================
# Type labels (from PR title prefix)
# ============================================================
- name: "label type: feat"
conditions:
- "title~=(?i)^\\[(feat|feature)\\]"
- -closed
actions:
label:
add: ["type: feat"]
- name: "label type: bugfix"
conditions:
- "title~=(?i)^\\[(bug)?fix\\]"
- -closed
actions:
label:
add: ["type: bugfix"]
- name: "label type: refactor"
conditions:
- "title~=(?i)^\\[refactor\\]"
- -closed
actions:
label:
add: ["type: refactor"]
- name: "label type: perf"
conditions:
- "title~=(?i)^\\[perf\\]"
- -closed
actions:
label:
add: ["type: perf"]
- name: "label type: ci"
conditions:
- "title~=(?i)^\\[ci\\]"
- -closed
actions:
label:
add: ["type: ci"]
- name: "label type: docs"
conditions:
- "title~=(?i)^\\[(doc|docs)\\]"
- -closed
actions:
label:
add: ["type: docs"]
- name: "label type: misc"
conditions:
- "title~=(?i)^\\[(misc|chore)\\]"
- -closed
actions:
label:
add: ["type: misc"]
- name: "label type: new-model"
conditions:
- "title~=(?i)^\\[new.?model\\]"
- -closed
actions:
label:
add: ["type: new-model"]
# ============================================================
# Scope labels (from changed files)
# ============================================================
- name: "label scope: training"
conditions:
- or:
- files~=^fastvideo/train/
- files~=^fastvideo/training/
- files~=^fastvideo/distillation/
- files~=^examples/train/
- files~=^examples/training/
- files~=^examples/distill/
- -closed
actions:
label:
add: ["scope: training"]
- name: "label scope: inference"
conditions:
- or:
- files~=^fastvideo/pipelines/basic/
- files~=^fastvideo/pipelines/stages/
- files~=^fastvideo/pipelines/samplers/
- files~=^fastvideo/entrypoints/
- files~=^fastvideo/worker/
- files~=^fastvideo/api/sampling_param
- files~=^fastvideo/configs/pipelines/
- files~=^examples/inference/
- -closed
actions:
label:
add: ["scope: inference"]
- name: "label scope: attention"
conditions:
- files~=^fastvideo/attention/
- -closed
actions:
label:
add: ["scope: attention"]
- name: "label scope: kernel"
conditions:
- or:
- files~=^fastvideo-kernel/
- files~=^csrc/
- -closed
actions:
label:
add: ["scope: kernel"]
- name: "label scope: data"
conditions:
- or:
- files~=^fastvideo/dataset/
- files~=^fastvideo/pipelines/preprocess/
- files~=^examples/preprocessing/
- -closed
actions:
label:
add: ["scope: data"]
- name: "label scope: infra"
conditions:
- or:
- files~=^\.github/
- files~=^\.buildkite/
- files~=^fastvideo/tests/
- files~=^docker/
- -closed
actions:
label:
add: ["scope: infra"]
- name: "label scope: distributed"
conditions:
- files~=^fastvideo/distributed/
- -closed
actions:
label:
add: ["scope: distributed"]
- name: "label scope: docs"
conditions:
- files~=^docs/
- -closed
actions:
label:
add: ["scope: docs"]
- name: "label scope: ui"
conditions:
- files~=^ui/
- -closed
actions:
label:
add: ["scope: ui"]
- name: "label scope: model"
conditions:
- or:
- files~=^fastvideo/models/
- files~=^fastvideo/layers/
- files~=^fastvideo/configs/models/
- -closed
actions:
label:
add: ["scope: model"]
# ============================================================
# Pre-commit failure help comment
# ============================================================
- name: comment on pre-commit failure
conditions:
- check-failure~=pre-commit
- -closed
actions:
comment:
message: |
## Pre-commit checks failed
Hi @{{author}}, the pre-commit checks have failed. To fix them locally:
```bash
# Install pre-commit if you haven't already
uv pip install pre-commit
pre-commit install
# Run all checks and auto-fix what's possible
pre-commit run --all-files
```
Common fixes:
- **yapf**: `yapf -i <file>` (formatting)
- **ruff**: `ruff check --fix <file>` (linting)
- **codespell**: `codespell --write-changes <file>` (spelling)
After fixing, commit and push the changes. The checks will re-run automatically.
For future commits, `pre-commit` will run automatically on changed files before each commit.
# ============================================================
# Merge conflict detection
# ============================================================
- name: label conflicting PRs
conditions:
- conflict
- -closed
- label!=stale
actions:
label:
add: [needs-rebase]
comment:
message: |
This PR has merge conflicts with the base branch. Please rebase:
```bash
git fetch origin main
git rebase origin/main
# Resolve any conflicts, then:
git push --force-with-lease
```
- name: remove conflict label when resolved
conditions:
- -conflict
- -closed
- label=needs-rebase
actions:
label:
remove: [needs-rebase]
# ============================================================
# Auto-merge and auto-rebase
# ============================================================
- name: auto-merge when ready and all checks pass
conditions:
- label=ready
- "title~=(?i)^\\[(feat|feature|bugfix|fix|refactor|perf|ci|doc|docs|misc|chore|kernel|new.?model)\\]"
- "#approved-reviews-by>=1"
- check-success~=pre-commit
- check-success=fastcheck-passed
- check-success=full-suite-passed
- -conflict
- -closed
- -draft
actions:
merge:
method: squash
- name: auto-update when ready
conditions:
- label=ready
- "#approved-reviews-by>=1"
- -conflict
- -closed
- -draft
actions:
update: {}
# ============================================================
# PR title format help
# ============================================================
- name: comment on invalid PR title format
conditions:
- -closed
- -draft
- "-title~=(?i)^\\[(feat|feature|bugfix|fix|refactor|perf|ci|doc|docs|misc|chore|kernel|new.?model)\\]"
actions:
comment:
message: |
## ⚠️ PR title format required
Your PR title must start with a type tag in brackets. Examples:
- `[feat] Add new model support`
- `[bugfix] Fix VAE tiling corruption`
- `[refactor] Restructure training pipeline`
- `[perf] Optimize attention kernel`
- `[ci] Update test infrastructure`
- `[docs] Add inference guide`
- `[misc] Clean up configs`
- `[new-model] Port Flux2 to FastVideo`
Valid tags: `feat`, `feature`, `bugfix`, `fix`, `refactor`, `perf`, `ci`, `doc`, `docs`, `misc`, `chore`, `kernel`, `new-model`
Please update your PR title and the merge protection check will pass automatically.
merge_protections_settings:
reporting_method: check-runs
+106
View File
@@ -0,0 +1,106 @@
name: Build Image Template
on:
workflow_call:
inputs:
python_version:
required: true
type: string
dockerfile_path:
required: true
type: string
tag_suffix:
required: true
type: string
jobs:
build-and-push:
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Free up disk space
run: |
# Display initial space
echo "Initial disk space:"
df -h
# Remove large directories directly
sudo rm -rf /usr/share/dotnet
sudo rm -rf /usr/local/lib/android
sudo rm -rf /opt/ghc
sudo rm -rf /usr/local/share/boost
sudo rm -rf /usr/share/swift
sudo rm -rf /usr/local/lib/node_modules
sudo rm -rf /usr/local/share/powershell
sudo rm -rf /usr/share/rust
sudo rm -rf /usr/local/.ghcup
# Remove cached files
sudo rm -rf /var/lib/apt/lists/*
sudo rm -rf /var/cache/apt/archives/*
# Clean Docker
docker system prune -af --volumes
# Display available space after cleanup
echo "Disk space after cleanup:"
df -h
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Login to GitHub Container Registry
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.repository_owner }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Prepare tags
id: prepare-tags
run: |
SHORT_SHA=$(echo ${{ github.sha }} | cut -c1-7)
TAGS="type=raw,value=${{ inputs.tag_suffix }}-latest"
TAGS="${TAGS}\ntype=raw,value=${{ inputs.tag_suffix }}-sha-${SHORT_SHA}"
# Set Python 3.10 as the default image
if [[ "${{ inputs.python_version }}" == "3.10" ]]; then
TAGS="${TAGS}\ntype=raw,value=latest"
fi
{
echo "tags<<EOF"
echo -e "$TAGS"
echo "EOF"
} >> $GITHUB_OUTPUT
- name: Extract metadata for Docker
id: meta
uses: docker/metadata-action@v5
with:
images: ghcr.io/${{ github.repository }}/fastvideo-dev
tags: ${{ steps.prepare-tags.outputs.tags }}
- name: Build and push Docker image
id: build-push
uses: docker/build-push-action@v6
with:
context: .
file: ${{ inputs.dockerfile_path }}
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=gha
cache-to: type=gha,mode=max
- name: Success message
run: |
echo "✅ Python ${{ inputs.python_version }} image successfully built and pushed to ghcr.io/${{ github.repository }}/fastvideo-dev:${{ inputs.tag_suffix }}-latest"
echo "To run tests with this image, manually trigger the 'Run Tests' workflow."
+80
View File
@@ -0,0 +1,80 @@
name: Aggregate Test Status
on:
status:
permissions:
statuses: write
jobs:
aggregate:
if: >-
github.event.context == 'direct-test-completed'
&& github.event.state == 'success'
runs-on: ubuntu-latest
steps:
- name: Check and update aggregate status
uses: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea # v7.0.1
with:
script: |
const sha = context.payload.sha;
const { data } = await github.rest.repos.getCombinedStatusForRef({
owner: context.repo.owner,
repo: context.repo.repo,
ref: sha,
per_page: 100,
});
const bkStatuses = data.statuses.filter(
s => s.context.startsWith('buildkite/ci/')
);
const FASTCHECK_PREFIX = 'buildkite/ci/microscope-';
const FULL_SUITE_PREFIXES = [
'buildkite/ci/test-tube-',
'buildkite/ci/bar-chart-',
];
const fastcheck = bkStatuses.filter(
s => s.context.startsWith(FASTCHECK_PREFIX)
);
const fullSuite = bkStatuses.filter(
s => FULL_SUITE_PREFIXES.some(p => s.context.startsWith(p))
);
if (
fastcheck.length > 0
&& fastcheck.every(s => s.state === 'success')
) {
core.info(
`All ${fastcheck.length} fastcheck tests passed — updating fastcheck-passed`
);
await github.rest.repos.createCommitStatus({
owner: context.repo.owner,
repo: context.repo.repo,
sha,
state: 'success',
context: 'fastcheck-passed',
description:
`All ${fastcheck.length} fastcheck tests passed`,
});
}
if (
fullSuite.length > 0
&& fullSuite.every(s => s.state === 'success')
) {
core.info(
`All ${fullSuite.length} full suite tests passed — updating full-suite-passed`
);
await github.rest.repos.createCommitStatus({
owner: context.repo.owner,
repo: context.repo.repo,
sha,
state: 'success',
context: 'full-suite-passed',
description:
`All ${fullSuite.length} full suite tests passed`,
});
}
+32
View File
@@ -0,0 +1,32 @@
name: pre-commit
on:
pull_request:
branches: [main]
workflow_call:
inputs:
ref:
description: 'Git ref to checkout (defaults to github.ref)'
required: false
type: string
permissions:
contents: read
jobs:
pre-commit:
if: github.event_name == 'workflow_call' || github.event.pull_request.draft != true
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
ref: ${{ inputs.ref || '' }}
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- run: echo "::add-matcher::.github/workflows/matchers/actionlint.json"
- run: echo "::add-matcher::.github/workflows/matchers/mypy.json"
- run: echo "::add-matcher::.github/workflows/matchers/ruff.json"
- uses: pre-commit/action@v3.0.1
with:
extra_args: --all-files --hook-stage manual
+271
View File
@@ -0,0 +1,271 @@
name: Slash Commands
on:
issue_comment:
types: [created]
permissions:
contents: read
pull-requests: write
statuses: write
jobs:
handle-merge:
if: >-
github.event.issue.pull_request != null
&& startsWith(github.event.comment.body, '/merge')
runs-on: ubuntu-latest
steps:
- name: Check write permission
id: perm
uses: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea # v7.0.1
with:
script: |
const { data: perm } = await github.rest.repos.getCollaboratorPermissionLevel({
owner: context.repo.owner,
repo: context.repo.repo,
username: context.payload.comment.user.login,
});
const hasWrite = ['admin', 'write'].includes(perm.permission);
if (!hasWrite) {
core.setFailed(`User ${context.payload.comment.user.login} lacks write permission (has: ${perm.permission}).`);
}
core.setOutput('has_write', String(hasWrite));
- name: Add ready label and react
id: label
if: steps.perm.outputs.has_write == 'true'
uses: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea # v7.0.1
with:
script: |
const owner = context.repo.owner;
const repo = context.repo.repo;
const prNumber = context.payload.issue.number;
try { await github.rest.issues.removeLabel({ owner, repo, issue_number: prNumber, name: 'ready' }); } catch {}
await github.rest.issues.addLabels({ owner, repo, issue_number: prNumber, labels: ['ready'] });
await github.rest.reactions.createForIssueComment({
owner, repo,
comment_id: context.payload.comment.id,
content: 'rocket',
});
const { data: pr } = await github.rest.pulls.get({ owner, repo, pull_number: prNumber });
core.setOutput('pr_sha', pr.head.sha);
core.setOutput('pr_branch', pr.head.ref);
core.setOutput('pr_number', String(prNumber));
- name: Trigger Full Suite
if: steps.perm.outputs.has_write == 'true'
env:
BUILDKITE_API_TOKEN: ${{ secrets.BUILDKITE_API_TOKEN }}
PR_SHA: ${{ steps.label.outputs.pr_sha }}
PR_BRANCH: ${{ steps.label.outputs.pr_branch }}
PR_NUMBER: ${{ steps.label.outputs.pr_number }}
BK_ORG: ${{ vars.BUILDKITE_ORG_SLUG }}
BK_PIPELINE: ${{ vars.BUILDKITE_PIPELINE_SLUG }}
run: |
curl -sS --fail-with-body -X POST \
"https://api.buildkite.com/v2/organizations/${BK_ORG}/pipelines/${BK_PIPELINE}/builds" \
-H "Authorization: Bearer $BUILDKITE_API_TOKEN" \
-H "Content-Type: application/json" \
--data-raw "$(jq -n \
--arg commit "$PR_SHA" \
--arg branch "$PR_BRANCH" \
--arg message "Full Suite for PR #${PR_NUMBER} (via /merge)" \
--argjson pr_id "$PR_NUMBER" \
'{
commit: $commit,
branch: $branch,
message: $message,
ignore_pipeline_branch_filters: true,
pull_request_id: $pr_id,
pull_request_base_branch: "main",
env: {
TEST_SCOPE: "full",
FULL_SUITE: "true",
PR_NUMBER: ($pr_id | tostring)
}
}')"
parse-command:
if: >-
github.event.issue.pull_request != null
&& startsWith(github.event.comment.body, '/test')
runs-on: ubuntu-latest
outputs:
test_type: ${{ steps.parse.outputs.test_type }}
test_scope: ${{ steps.parse.outputs.test_scope }}
full_suite: ${{ steps.parse.outputs.full_suite }}
pr_sha: ${{ steps.pr.outputs.sha }}
pr_branch: ${{ steps.pr.outputs.branch }}
has_write: ${{ steps.perm.outputs.has_write }}
steps:
- name: Check write permission
id: perm
uses: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea # v7.0.1
with:
script: |
const { data: perm } = await github.rest.repos.getCollaboratorPermissionLevel({
owner: context.repo.owner,
repo: context.repo.repo,
username: context.payload.comment.user.login,
});
const hasWrite = ['admin', 'write'].includes(perm.permission);
core.setOutput('has_write', String(hasWrite));
if (!hasWrite) {
core.info(`User ${context.payload.comment.user.login} lacks write permission — ignoring.`);
}
- name: Parse /test command
id: parse
if: steps.perm.outputs.has_write == 'true'
shell: bash
env:
COMMENT: ${{ github.event.comment.body }}
run: |
set -euo pipefail
TEST_NAME=$(echo "$COMMENT" | grep -oP '(?<=/test\s)\S+' | head -1 || true)
VALID="encoder vae transformer kernel unit ssim training lora-inference lora-training distillation self-forcing vsa vmoba performance api full fastcheck pre-commit"
if [ -z "$TEST_NAME" ] || ! echo "$VALID" | grep -qw "$TEST_NAME"; then
echo "Unknown test: '$TEST_NAME'. Valid: $VALID"
exit 1
fi
declare -A MAP=(
[encoder]=encoder [vae]=vae [transformer]=transformer
[kernel]=kernel_tests [unit]=unit_test
[ssim]=ssim [training]=training
[lora-inference]=inference_lora [lora-training]=training_lora
[distillation]=distillation_dmd [self-forcing]=self_forcing
[vsa]=training_vsa [vmoba]=inference_vmoba
[performance]=performance [api]=api_server
)
if [ "$TEST_NAME" = "full" ]; then
{
echo "test_type=all"
echo "test_scope=full"
echo "full_suite=true"
} >> "$GITHUB_OUTPUT"
elif [ "$TEST_NAME" = "fastcheck" ]; then
{
echo "test_type=fastcheck"
echo "test_scope=fastcheck"
echo "full_suite=false"
} >> "$GITHUB_OUTPUT"
elif [ "$TEST_NAME" = "pre-commit" ]; then
{
echo "test_type="
echo "test_scope=precommit"
echo "full_suite=false"
} >> "$GITHUB_OUTPUT"
else
{
echo "test_type=${MAP[$TEST_NAME]}"
echo "test_scope=direct"
echo "full_suite=false"
} >> "$GITHUB_OUTPUT"
fi
- name: Get PR details
id: pr
if: steps.perm.outputs.has_write == 'true'
uses: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea # v7.0.1
with:
script: |
const { data: pr } = await github.rest.pulls.get({
owner: context.repo.owner,
repo: context.repo.repo,
pull_number: context.payload.issue.number,
});
core.setOutput('sha', pr.head.sha);
core.setOutput('branch', pr.head.ref);
- name: React to comment
if: steps.perm.outputs.has_write == 'true'
uses: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea # v7.0.1
with:
script: |
await github.rest.reactions.createForIssueComment({
owner: context.repo.owner,
repo: context.repo.repo,
comment_id: context.payload.comment.id,
content: 'rocket',
});
pre-commit:
needs: parse-command
if: >-
needs.parse-command.outputs.has_write == 'true'
&& needs.parse-command.outputs.test_scope == 'precommit'
uses: ./.github/workflows/ci-precommit.yml
with:
ref: refs/pull/${{ github.event.issue.number }}/merge
post-precommit-status:
needs: [parse-command, pre-commit]
if: always() && needs.parse-command.outputs.test_scope == 'precommit'
runs-on: ubuntu-latest
steps:
- uses: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea # v7.0.1
env:
PR_SHA: ${{ needs.parse-command.outputs.pr_sha }}
RESULT: ${{ needs.pre-commit.result }}
with:
script: |
const state = process.env.RESULT === 'success' ? 'success' : 'failure';
await github.rest.repos.createCommitStatus({
owner: context.repo.owner,
repo: context.repo.repo,
sha: process.env.PR_SHA,
state,
context: 'pre-commit',
description: `Triggered via /test pre-commit (${state})`,
});
trigger-buildkite:
needs: parse-command
if: >-
needs.parse-command.outputs.has_write == 'true'
&& needs.parse-command.outputs.test_type != ''
runs-on: ubuntu-latest
steps:
- name: Trigger Buildkite
env:
BUILDKITE_API_TOKEN: ${{ secrets.BUILDKITE_API_TOKEN }}
PR_SHA: ${{ needs.parse-command.outputs.pr_sha }}
PR_BRANCH: ${{ needs.parse-command.outputs.pr_branch }}
PR_NUMBER: ${{ github.event.issue.number }}
TEST_SCOPE: ${{ needs.parse-command.outputs.test_scope }}
FULL_SUITE: ${{ needs.parse-command.outputs.full_suite }}
TEST_TYPE: ${{ needs.parse-command.outputs.test_type }}
BK_ORG: ${{ vars.BUILDKITE_ORG_SLUG }}
BK_PIPELINE: ${{ vars.BUILDKITE_PIPELINE_SLUG }}
run: |
curl -sS --fail-with-body -X POST \
"https://api.buildkite.com/v2/organizations/${BK_ORG}/pipelines/${BK_PIPELINE}/builds" \
-H "Authorization: Bearer $BUILDKITE_API_TOKEN" \
-H "Content-Type: application/json" \
--data-raw "$(jq -n \
--arg commit "$PR_SHA" \
--arg branch "$PR_BRANCH" \
--arg message "/test ${TEST_TYPE} on PR #${PR_NUMBER}" \
--argjson pr_id "$PR_NUMBER" \
--arg test_scope "$TEST_SCOPE" \
--arg full_suite "$FULL_SUITE" \
--arg test_type "$TEST_TYPE" \
--arg pr_number "$PR_NUMBER" \
'{
commit: $commit,
branch: $branch,
message: $message,
ignore_pipeline_branch_filters: true,
pull_request_id: $pr_id,
pull_request_base_branch: "main",
env: {
TEST_SCOPE: $test_scope,
FULL_SUITE: $full_suite,
TEST_TYPE: $test_type,
PR_NUMBER: $pr_number
}
}')"
@@ -0,0 +1,83 @@
name: Trigger Full Suite
on:
pull_request_target:
types: [labeled, synchronize]
permissions:
contents: read
pull-requests: read
concurrency:
group: full-suite-${{ github.event.pull_request.number }}
cancel-in-progress: false
jobs:
trigger:
if: >-
(github.event.action == 'labeled' && github.event.label.name == 'ready')
|| github.event.action == 'synchronize'
runs-on: ubuntu-latest
steps:
- name: Check ready label
id: check
uses: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea # v7.0.1
with:
script: |
const { data: pr } = await github.rest.pulls.get({
owner: context.repo.owner,
repo: context.repo.repo,
pull_number: context.payload.pull_request.number,
});
const hasReady = pr.labels.some(l => l.name === 'ready');
core.setOutput('has_ready', String(hasReady));
if (!hasReady) core.info('No ready label — skipping Full Suite trigger.');
- name: Cancel previous Buildkite builds
if: steps.check.outputs.has_ready == 'true'
env:
BUILDKITE_API_TOKEN: ${{ secrets.BUILDKITE_API_TOKEN }}
PR_BRANCH: ${{ github.event.pull_request.head.ref }}
run: |
# Find running builds for this branch with TEST_SCOPE=full and cancel them
builds=$(curl -sS -H "Authorization: Bearer $BUILDKITE_API_TOKEN" \
"https://api.buildkite.com/v2/organizations/${{ vars.BUILDKITE_ORG_SLUG }}/pipelines/${{ vars.BUILDKITE_PIPELINE_SLUG }}/builds?branch=${PR_BRANCH}&state=running,scheduled" \
| jq -r '.[] | select(try (.env.TEST_SCOPE == "full") catch false) | .number')
for build_num in $builds; do
echo "Cancelling Buildkite build #$build_num"
curl -sS -X PUT -H "Authorization: Bearer $BUILDKITE_API_TOKEN" \
"https://api.buildkite.com/v2/organizations/${{ vars.BUILDKITE_ORG_SLUG }}/pipelines/${{ vars.BUILDKITE_PIPELINE_SLUG }}/builds/${build_num}/cancel"
done
- name: Trigger Buildkite Full Suite
if: steps.check.outputs.has_ready == 'true'
env:
BUILDKITE_API_TOKEN: ${{ secrets.BUILDKITE_API_TOKEN }}
PR_SHA: ${{ github.event.pull_request.head.sha }}
PR_BRANCH: ${{ github.event.pull_request.head.ref }}
PR_NUMBER: ${{ github.event.pull_request.number }}
BK_ORG: ${{ vars.BUILDKITE_ORG_SLUG }}
BK_PIPELINE: ${{ vars.BUILDKITE_PIPELINE_SLUG }}
run: |
curl -sS --fail-with-body -X POST \
"https://api.buildkite.com/v2/organizations/${BK_ORG}/pipelines/${BK_PIPELINE}/builds" \
-H "Authorization: Bearer $BUILDKITE_API_TOKEN" \
-H "Content-Type: application/json" \
--data-raw "$(jq -n \
--arg commit "$PR_SHA" \
--arg branch "$PR_BRANCH" \
--arg message "Full Suite for PR #${PR_NUMBER}" \
--argjson pr_id "$PR_NUMBER" \
'{
commit: $commit,
branch: $branch,
message: $message,
ignore_pipeline_branch_filters: true,
pull_request_id: $pr_id,
pull_request_base_branch: "main",
env: {
TEST_SCOPE: "full",
FULL_SUITE: "true",
PR_NUMBER: ($pr_id | tostring)
}
}')"
@@ -0,0 +1,65 @@
name: Auto-Label Issues
on:
issues:
types: [opened, edited]
permissions:
issues: write
jobs:
label-issues:
if: github.repository == 'hao-ai-lab/FastVideo'
runs-on: ubuntu-latest
steps:
- name: Label by keywords
uses: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea # v7.0.1
with:
script: |
const title = context.payload.issue.title.toLowerCase();
const body = (context.payload.issue.body || '').toLowerCase();
const text = title + ' ' + body;
const labels = [];
const rules = [
// scope labels (shared with PR labeling via Mergify)
// Mapping: label → repo directories
// scope: training → fastvideo/train/, fastvideo/training/, fastvideo/distillation/
// scope: inference → fastvideo/pipelines/, fastvideo/entrypoints/, fastvideo/worker/
// scope: attention → fastvideo/attention/
// scope: kernel → fastvideo-kernel/, csrc/
// scope: model → fastvideo/models/, fastvideo/layers/, fastvideo/configs/models/
// scope: data → fastvideo/dataset/, fastvideo/pipelines/preprocess/
// scope: distributed → fastvideo/distributed/
// scope: docs → docs/
{ keywords: ['training', 'finetune', 'fine-tune', 'lora', 'fsdp', 'distill'], label: 'scope: training' },
{ keywords: ['inference', 'generate', 'pipeline', 'slow', 'latency'], label: 'scope: inference' },
{ keywords: ['attention', 'vsa', 'flash', 'sta', 'vmoba', 'sparse attn'], label: 'scope: attention' },
{ keywords: ['kernel', 'csrc', 'cuda kernel', 'thunderkittens'], label: 'scope: kernel' },
{ keywords: ['wan', 'hunyuan', 'mochi', 'ltx', 'cogvideo', 'flux', 'sd3', 'cosmos'], label: 'scope: model' },
{ keywords: ['dataset', 'dataloader', 'preprocessing', 'preprocess'], label: 'scope: data' },
{ keywords: ['distributed', 'sequence parallel', 'fsdp', 'tensor parallel', 'multi-node', 'multi-gpu'], label: 'scope: distributed' },
{ keywords: ['docs', 'documentation', 'tutorial', 'example'], label: 'scope: docs' },
// issue-only labels (cross-module, no single repo directory)
{ keywords: ['install', 'setup', 'pip', 'cuda', 'uv ', 'import error', 'modulenotfound'], label: 'installation' },
{ keywords: ['memory', 'oom', 'out of memory', 'gpu memory', 'vram'], label: 'performance' },
{ keywords: ['windows', 'macos', 'mac os', 'apple', 'mps', 'rocm', 'amd', 'npu'], label: 'platform' },
];
for (const rule of rules) {
if (rule.keywords.some(kw => text.includes(kw))) {
labels.push(rule.label);
}
}
if (labels.length > 0) {
await github.rest.issues.addLabels({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: context.payload.issue.number,
labels: labels,
});
console.log(`Added labels: ${labels.join(', ')}`);
} else {
console.log('No keyword matches found');
}
+51
View File
@@ -0,0 +1,51 @@
name: Close Stale Issues and PRs
on:
schedule:
# Daily at 1:30 AM UTC
- cron: '30 1 * * *'
jobs:
stale:
if: github.repository == 'hao-ai-lab/FastVideo'
permissions:
issues: write
pull-requests: write
actions: write
runs-on: ubuntu-latest
steps:
- uses: actions/stale@997185467fa4f803885201cee163a9f38240193d # v10.1.1
with:
operations-per-run: 500
exempt-draft-pr: true
exempt-issue-labels: 'keep-open,pinned,security,Bug,RFC'
exempt-pr-labels: 'keep-open,pinned'
labels-to-add-when-unstale: 'unstale'
labels-to-remove-when-stale: 'unstale'
days-before-issue-stale: 90
days-before-issue-close: 30
stale-issue-label: 'stale'
stale-issue-message: >
This issue has been automatically marked as stale because it has not
had any activity within 90 days. It will be automatically closed if
no further activity occurs within 30 days. Leave a comment if you
feel this issue should remain open. Thank you!
close-issue-message: >
This issue has been automatically closed due to inactivity. Please
feel free to reopen if you feel it is still relevant. Thank you!
days-before-pr-stale: 60
days-before-pr-close: 14
stale-pr-label: 'stale'
stale-pr-message: >
This pull request has been automatically marked as stale because it
has not had any activity within 60 days. It will be automatically
closed if no further activity occurs within 14 days. Leave a comment
if you feel this pull request should remain open. Thank you!
close-pr-message: >
This pull request has been automatically closed due to inactivity.
Please feel free to reopen if you intend to continue working on it.
Thank you!
+56
View File
@@ -0,0 +1,56 @@
name: Welcome First-Time Contributors
on:
issues:
types: [opened]
pull_request_target:
types: [opened]
permissions:
issues: write
pull-requests: write
jobs:
welcome:
if: github.repository == 'hao-ai-lab/FastVideo'
runs-on: ubuntu-latest
steps:
- uses: actions/first-interaction@34f15e814fe48ac9312ccf29db4e74fa767cbab7 # v1.3.0
with:
repo-token: ${{ secrets.GITHUB_TOKEN }}
issue-message: |
Welcome to FastVideo! Thanks for opening your first issue.
To help us investigate, please include:
- **FastVideo version**: `pip show fastvideo`
- **GPU**: `nvidia-smi` output (GPU model, driver, CUDA version)
- **Python version**: `python --version`
- **OS**: e.g., Ubuntu 22.04
If this is a bug, a minimal reproduction script helps us fix it faster.
Useful links:
- [Documentation](https://hao-ai-lab.github.io/FastVideo)
- [Contributing Guide](https://hao-ai-lab.github.io/FastVideo/contributing/overview/)
- [Slack](https://join.slack.com/t/fastvideo/shared_invite/zt-3f4lao1uq-u~Ipx6Lt4J27AlD2y~IdLQ)
pr-message: |
Welcome to FastVideo! Thanks for your first pull request.
**How our CI works:**
PRs run a two-tier CI system:
1. **Pre-commit** — formatting (yapf), linting (ruff), type checking (mypy). Runs immediately on every PR.
2. **Fastcheck** — core GPU tests (encoders, VAEs, transformers, kernels, unit tests). Runs automatically via Buildkite on relevant file changes (~10-15 min).
3. **Full Suite** — integration tests, training pipelines, SSIM regression. Runs only when a reviewer adds the `ready` label.
**Before your PR is reviewed:**
- [ ] `pre-commit run --all-files` passes locally
- [ ] You've added or updated tests for your changes
- [ ] The PR description explains what and why
If pre-commit fails, a bot comment will explain how to fix it. Fastcheck and Full Suite results appear in the Checks section below.
**Useful links:**
- [Contributing Guide](https://hao-ai-lab.github.io/FastVideo/contributing/overview/)
- [Development Roadmap](https://github.com/hao-ai-lab/FastVideo/issues/899)
- [Slack](https://join.slack.com/t/fastvideo/shared_invite/zt-3f4lao1uq-u~Ipx6Lt4J27AlD2y~IdLQ)
+67
View File
@@ -0,0 +1,67 @@
name: Build and Push Docker Images
on:
workflow_dispatch:
inputs:
python_3_10:
description: 'Build Python 3.10 image'
required: false
default: false
type: boolean
python_3_11:
description: 'Build Python 3.11 image'
required: false
default: false
type: boolean
python_3_12:
description: 'Build Python 3.12 image'
required: false
default: false
type: boolean
python_3_12_cuda_12_9:
description: 'Build Python 3.12 image Cuda 12.9'
required: false
default: false
type: boolean
permissions:
contents: read
packages: write
jobs:
build-python-3-10:
if: ${{ github.event.inputs.python_3_10 == 'true' }}
uses: ./.github/workflows/_template-build-image.yml
with:
python_version: '3.10'
dockerfile_path: docker/Dockerfile.python3.10
tag_suffix: py3.10
secrets: inherit
build-python-3-11:
if: ${{ github.event.inputs.python_3_11 == 'true' }}
uses: ./.github/workflows/_template-build-image.yml
with:
python_version: '3.11'
dockerfile_path: docker/Dockerfile.python3.11
tag_suffix: py3.11
secrets: inherit
build-python-3-12:
if: ${{ github.event.inputs.python_3_12 == 'true' }}
uses: ./.github/workflows/_template-build-image.yml
with:
python_version: '3.12'
dockerfile_path: docker/Dockerfile.python3.12
tag_suffix: py3.12
secrets: inherit
build-python-3-12-cuda-12-9:
if: ${{ github.event.inputs.python_3_12_cuda_12_9 == 'true' }}
uses: ./.github/workflows/_template-build-image.yml
with:
python_version: '3.12'
dockerfile_path: docker/Dockerfile.python3.12.cuda12.9.1
tag_suffix: py3.12-cuda12.9.1
secrets: inherit
+72
View File
@@ -0,0 +1,72 @@
name: Deploy Documentation
on:
push:
branches: [ main ]
paths:
- 'docs/**'
- 'mkdocs.yml'
- 'requirements-mkdocs.txt'
- '.github/workflows/infra-docs.yml'
pull_request:
branches: [ main ]
paths:
- 'docs/**'
- 'mkdocs.yml'
- 'requirements-mkdocs.txt'
- '.github/workflows/infra-docs.yml'
permissions:
contents: read
pages: write
id-token: write
concurrency:
group: "pages"
cancel-in-progress: false
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: '3.12'
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements-mkdocs.txt
- name: Setup Pages
uses: actions/configure-pages@v4
- name: Generate docs examples
run: python docs/generate_examples.py
- name: Check docs links
run: python scripts/check_docs_links.py
- name: Build documentation
run: mkdocs build
- name: Upload artifact
uses: actions/upload-pages-artifact@v3
with:
path: ./site
deploy:
environment:
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}
runs-on: ubuntu-latest
needs: build
if: github.ref == 'refs/heads/main'
steps:
- name: Deploy to GitHub Pages
id: deployment
uses: actions/deploy-pages@v4
@@ -0,0 +1,17 @@
{
"problemMatcher": [
{
"owner": "actionlint",
"pattern": [
{
"regexp": "^(?:\\x1b\\[\\d+m)?(.+?)(?:\\x1b\\[\\d+m)*:(?:\\x1b\\[\\d+m)*(\\d+)(?:\\x1b\\[\\d+m)*:(?:\\x1b\\[\\d+m)*(\\d+)(?:\\x1b\\[\\d+m)*: (?:\\x1b\\[\\d+m)*(.+?)(?:\\x1b\\[\\d+m)* \\[(.+?)\\]$",
"file": 1,
"line": 2,
"column": 3,
"message": 4,
"code": 5
}
]
}
]
}
+16
View File
@@ -0,0 +1,16 @@
{
"problemMatcher": [
{
"owner": "mypy",
"pattern": [
{
"regexp": "^(.+):(\\d+):\\s(error|warning):\\s(.+)$",
"file": 1,
"line": 2,
"severity": 3,
"message": 4
}
]
}
]
}
+17
View File
@@ -0,0 +1,17 @@
{
"problemMatcher": [
{
"owner": "ruff",
"pattern": [
{
"regexp": "^(.+):(\\d+):(\\d+): (\\w+) (.+)$",
"file": 1,
"line": 2,
"column": 3,
"code": 4,
"message": 5
}
]
}
]
}
+28
View File
@@ -0,0 +1,28 @@
name: Publish to Comfy registry
on:
workflow_dispatch:
push:
branches:
- main
- master
paths:
- "pyproject.toml"
permissions:
issues: write
jobs:
publish-node:
name: Publish Custom Node to registry
runs-on: ubuntu-latest
if: ${{ github.repository_owner == 'hao-ai-lab' }}
steps:
- name: Check out code
uses: actions/checkout@v4
with:
submodules: true
- name: Publish Custom Node
uses: Comfy-Org/publish-node-action@v1
with:
## Add your own personal access token to your Github Repository secrets and reference it here.
personal_access_token: ${{ secrets.REGISTRY_ACCESS_TOKEN }}
+71
View File
@@ -0,0 +1,71 @@
name: Publish FastVideo to PyPI on Version Change
on:
push:
branches:
- main
paths:
- 'pyproject.toml' # Trigger when pyproject.toml changes
workflow_dispatch:
jobs:
check-version-change:
runs-on: ubuntu-latest
outputs:
version-changed: ${{ steps.check-version.outputs.changed }}
new-version: ${{ steps.check-version.outputs.new-version }}
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
fetch-depth: 2
- name: Check if version changed
id: check-version
run: |
# Get current commit's version
NEW_VERSION=$(grep -oP "version\\s*=\\s*\"\\K[^\"]+\"" pyproject.toml)
echo "New version: $NEW_VERSION"
# Get previous version from git history
OLD_VERSION=$(git show HEAD~1:./pyproject.toml | grep -oP "version\\s*=\\s*\"\\K[^\"]+\"" || echo "0.0.0")
echo "Old version: $OLD_VERSION"
if [ "$NEW_VERSION" != "$OLD_VERSION" ]; then
echo "Version changed from $OLD_VERSION to $NEW_VERSION"
echo "changed=true" >> $GITHUB_OUTPUT
echo "new-version=$NEW_VERSION" >> $GITHUB_OUTPUT
else
echo "Version did not change"
echo "changed=false" >> $GITHUB_OUTPUT
fi
build-publish-main:
needs: check-version-change
if: ${{ needs.check-version-change.outputs.version-changed == 'true' || github.event_name == 'workflow_dispatch' }}
runs-on: ubuntu-latest
permissions:
id-token: write # Needed for OIDC Trusted Publishing
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.10'
- name: Install build dependencies
run: |
python -m pip install --upgrade pip
pip install build twine wheel
- name: Build package
run: |
python -m build
- name: Publish release distributions to PyPI
uses: pypa/gh-action-pypi-publish@release/v1
with:
packages-dir: dist/
+225
View File
@@ -0,0 +1,225 @@
name: Publish FastVideo Kernel to PyPI on Version Change
on:
push:
branches:
- main
paths:
- "fastvideo-kernel/pyproject.toml"
workflow_dispatch:
jobs:
check-version-change:
runs-on: ubuntu-latest
outputs:
version-changed: ${{ steps.check-version.outputs.changed }}
new-version: ${{ steps.check-version.outputs.new-version }}
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
fetch-depth: 2
- name: Check if version changed
id: check-version
run: |
cd fastvideo-kernel
# Get current commit's version from pyproject.toml
# Use ^ to match start of line to avoid matching minimum-version
NEW_VERSION=$(grep -oP '^version\s*=\s*"\K[^"]+' pyproject.toml)
echo "New version: $NEW_VERSION"
# Get previous version from git history
# Note: git show expects path relative to repo root
OLD_VERSION=$(git show HEAD~1:fastvideo-kernel/pyproject.toml | grep -oP '^version\s*=\s*"\K[^"]+' || echo "0.0.0")
echo "Old version: $OLD_VERSION"
if [ "$NEW_VERSION" != "$OLD_VERSION" ]; then
echo "Version changed from $OLD_VERSION to $NEW_VERSION"
echo "changed=true" >> "$GITHUB_OUTPUT"
echo "new-version=$NEW_VERSION" >> "$GITHUB_OUTPUT"
else
echo "Version did not change"
echo "changed=false" >> "$GITHUB_OUTPUT"
fi
build_wheels:
name: Build Wheel
needs: check-version-change
if: ${{ needs.check-version-change.outputs.version-changed == 'true' || github.event_name == 'workflow_dispatch' }}
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
os: [ubuntu-22.04]
python-version: ['3.10', '3.11', '3.12']
torch-cuda:
# - torch-version: '2.5.1'
# cuda-version: '12.4.1'
# torch-cuda-short: 'cu124'
# - torch-version: '2.6.0'
# cuda-version: '12.6.3'
# torch-cuda-short: 'cu126'
# - torch-version: '2.7.1'
# cuda-version: '12.8.0'
# torch-cuda-short: 'cu128'
# - torch-version: '2.9.1'
# cuda-version: '12.8.0'
# torch-cuda-short: 'cu128'
- torch-version: '2.10.0'
cuda-version: '12.8.0'
torch-cuda-short: 'cu128'
steps:
- name: Free up disk space
run: |
echo "Initial disk space:"
df -h
# Remove large directories
sudo rm -rf /usr/share/dotnet
sudo rm -rf /usr/local/lib/android
sudo rm -rf /opt/ghc
sudo rm -rf /usr/local/share/boost
sudo rm -rf /usr/share/swift
sudo rm -rf /usr/local/lib/node_modules
sudo rm -rf /usr/local/share/powershell
sudo rm -rf /usr/share/rust
sudo rm -rf /usr/local/.ghcup
# Remove cached files
sudo rm -rf /var/lib/apt/lists/*
sudo rm -rf /var/cache/apt/archives/*
echo "Disk space after cleanup:"
df -h
- name: Checkout
uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Install CUDA ${{ matrix.torch-cuda.cuda-version }}
uses: Jimver/cuda-toolkit@v0.2.21
id: cuda-toolkit
with:
cuda: ${{ matrix.torch-cuda.cuda-version }}
linux-local-args: '["--toolkit"]'
method: 'network'
- name: Install dependencies (GCC, Clang, CUDA Paths, Git)
run: |
sudo apt update
sudo apt install -y git patchelf gcc-11 g++-11 clang-11
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-11 100 --slave /usr/bin/g++ g++ /usr/bin/g++-11
# Allow Git to Access Safe Directory
git config --global --add safe.directory /__w/FastVideo/FastVideo
# Set CUDA environment variables
export CUDA_HOME=/usr/local/cuda-${{ matrix.torch-cuda.cuda-version }}
export PATH=${CUDA_HOME}/bin:${PATH}
export LD_LIBRARY_PATH=${CUDA_HOME}/lib64:$LD_LIBRARY_PATH
# Verify installation
gcc --version
g++ --version
clang-11 --version
nvcc --version
- name: Install PyTorch ${{ matrix.torch-cuda.torch-version }}+cu${{ matrix.torch-cuda.cuda-version }}
run: |
pip install --upgrade pip
pip install typing-extensions==4.12.2
pip install --no-cache-dir torch==${{ matrix.torch-cuda.torch-version }} --index-url https://download.pytorch.org/whl/${{matrix.torch-cuda.torch-cuda-short}}
nvcc --version
python --version
python -c "import torch; print('PyTorch:', torch.__version__)"
python -c "import torch; print('CUDA:', torch.version.cuda)"
python -c "from torch.utils import cpp_extension; print (cpp_extension.CUDA_HOME)"
- name: Build wheel
run: |
export PYTHONPATH=$GITHUB_WORKSPACE:$PYTHONPATH
pip install setuptools ninja packaging wheel triton scikit-build-core cmake build
cd fastvideo-kernel
git submodule update --init --recursive # Ensure ThunderKittens submodule is initialized
# Release builds are produced on GPU-less runners, so force-enable TK and target Hopper.
export TORCH_CUDA_ARCH_LIST="9.0a"
export CMAKE_ARGS="${CMAKE_ARGS:-} -DFASTVIDEO_KERNEL_BUILD_TK=ON -DCMAKE_CUDA_ARCHITECTURES=90a"
# Build standard wheel (no local version suffix) for PyPI
python -m build --wheel --outdir dist
# Fix the wheel to be manylinux compliant
pip install auditwheel
# Point auditwheel at torch libs, but do not vendor them into the wheel.
TORCH_LIB_DIR=$(python - <<'PY'
import os
import torch
print(os.path.join(os.path.dirname(torch.__file__), "lib"))
PY
)
export LD_LIBRARY_PATH="${TORCH_LIB_DIR}:${LD_LIBRARY_PATH}"
# Target manylinux_2_35 (Ubuntu 22.04 native)
auditwheel repair dist/*.whl --plat manylinux_2_35_x86_64 -w fixed_dist \
--exclude libtorch_cuda.so \
--exclude libtorch_cpu.so \
--exclude libtorch.so \
--exclude libc10.so \
--exclude libc10_cuda.so \
--exclude libtorch_python.so
# Move fixed wheels back to dist for upload consistency
rm dist/*.whl
mv fixed_dist/*.whl dist/
- name: Upload wheel artifact
# Only upload if it's the "main" CUDA version we want on PyPI
# We upload all to artifacts for inspection/GH releases, but give them distinct artifact names
uses: actions/upload-artifact@v4
with:
name: fastvideo_kernel-py${{ matrix.python-version }}-${{ matrix.torch-cuda.torch-cuda-short }}-torch${{ matrix.torch-cuda.torch-version }}
path: fastvideo-kernel/dist/*.whl
retention-days: 90
publish_package:
name: Publish package
needs: [build_wheels, check-version-change]
if: ${{ needs.check-version-change.outputs.version-changed == 'true' || github.event_name == 'workflow_dispatch' }}
runs-on: ubuntu-22.04
permissions:
id-token: write # Needed for OIDC Trusted Publishing
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.10'
- name: Download PyPI wheels
uses: actions/download-artifact@v4
with:
path: fastvideo-kernel/dist/
pattern: 'fastvideo_kernel-py*'
merge-multiple: true
- name: Build source distribution
run: |
pip install build scikit-build-core cmake ninja
cd fastvideo-kernel
# We don't need full CUDA/Torch to just package the source (sdist)
python -m build --sdist --outdir dist
- name: Publish release distributions to PyPI
uses: pypa/gh-action-pypi-publish@release/v1
with:
packages-dir: fastvideo-kernel/dist/
+69 -29
View File
@@ -1,17 +1,11 @@
ucf101_stride4x4x4
__pycache__
*.mp4
.ipynb_checkpoints
*.pth
UCF-101/
results/
build/
fastvideo.egg-info/
wandb/
.idea
*.ipynb
*.jpg
*.mp3
*.safetensors
*.mp4
*.png
@@ -20,35 +14,81 @@ wandb/
*.pt
cache_dir/
wandb/
test*
sample_video*
sample_image*
512*
720*
1024*
debug*
private*
caption*
*deepspeed*
revised*
129f*
all*
read*
YSH*
*pick*
*ysh*
hw*
257f*
513f*
taming*
221hw*
65x512x512
venv/
.venv/
runs/
samples/
Miniconda3-latest-Linux-x86_64.sh
*validation/
data/
outputs/
outputs_video
checkpoints/
sbatch.sh
*.out
env
*.o
**/build/
**.pyc
**.txt
*.log
weights/
logs/
# SSIM test outputs
fastvideo/tests/ssim/generated_videos/
**/.cache/**
# Distribution / packaging
build/
dist/
*.egg-info/
*.egg
eggs/
.eggs/
# MkDocs documentation
site/
docs/getting_started/examples/
docs/inference/examples/
docs/training/examples/
docs/distillation/examples/
!requirements-mkdocs.txt
# VSCode
.vscode/
# DS Store
.DS_Store
# vim swap files
*.swo
*.swp
# Python pickle files
*.pkl
# Reference videos
!fastvideo/tests/ssim/reference_videos/**/*.mp4
# Static images
!docs/assets/images/**/*.png
!comfyui/assets/**/*.png
!comfyui/assets/**/*.gif
!assets/images/**/*.png
!assets/images/**/*.jpg
!assets/images/**/*.jpeg
!assets/images/**/*.gif
!assets/videos/**/*.mp4
dmd_t2v_output/
preprocess_output_text/
# Next.js / Node artifacts under ui/: see ui/.gitignore
.claude/
.codex/
.sisyphus/
openspec/
fastvideo/tests/ssim/reference_videos/**
+6
View File
@@ -0,0 +1,6 @@
[submodule "fastvideo-kernel/include/tk"]
path = fastvideo-kernel/include/tk
url = https://github.com/HazyResearch/ThunderKittens.git
[submodule "fastvideo-kernel/include/cutlass"]
path = fastvideo-kernel/include/cutlass
url = https://github.com/NVIDIA/cutlass.git
+1
View File
@@ -0,0 +1 @@
WRN 2026-03-26T13:46:33.469 ?.19646 server_start:193: Failed to start server: operation not permitted: /var/folders/z_/h_6myyk14d1b7z87z3vy4mjh0000gn/T/nvim.dsynkd/iSe0el/nvim.19646.0
+81
View File
@@ -0,0 +1,81 @@
default_stages:
- pre-commit # Run locally
- manual # Run in CI
exclude: |
(?x)(
fastvideo/third_party/.*|
fastvideo-kernel/.*|
assets/.*|
tests/.*|
demo/.*|
predict\.py|
scripts/.*|
assets/prompts/.*|
fastvideo/data_preprocess/.*|
fastvideo/dataset/.*|
fastvideo/models/.*|
fastvideo/sample/.*|
fastvideo/train\.py|
fastvideo/utils/.*|
examples/.*|
\.agents/.*|
.github/workflows/publish-fastvideo.yml|
.github/workflows/_template-build-image.yml|
docs/source/inference/support_matrix.md
)
repos:
- repo: https://github.com/google/yapf
rev: v0.43.0
hooks:
- id: yapf
args: [--in-place, --verbose]
additional_dependencies: [toml] # TODO: Remove when yapf is upgraded
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.11.12
hooks:
- id: ruff
args: [--output-format, github, --fix]
- repo: https://github.com/codespell-project/codespell
rev: v2.4.1
hooks:
- id: codespell
additional_dependencies: ['tomli']
args: ['--toml', 'pyproject.toml']
# - repo: https://github.com/PyCQA/isort
# rev: 6.0.1
# hooks:
# - id: isort
- repo: https://github.com/jackdewinter/pymarkdown
rev: v0.9.30
hooks:
- id: pymarkdown
args: [fix]
- repo: https://github.com/rhysd/actionlint
rev: v1.7.7
hooks:
- id: actionlint
- repo: https://github.com/pre-commit/mirrors-mypy
rev: v1.15.0
hooks:
- id: mypy
args: [--python-version, '3.10', --follow-imports, "skip", "--disable-error-code", "union-attr", "--disable-error-code", "override" ]
additional_dependencies: [types-aiofiles, types-cachetools, types-setuptools, types-PyYAML, types-requests]
- repo: local
hooks:
- id: check-filenames
name: Check for spaces in all filenames
entry: bash
args:
- -c
- 'git ls-files | grep -v "^\"*fastvideo/tests/ssim/" | grep -v "^\"*fastvideo/tests/inference/lora/L40S_reference_videos/" | grep " " && echo "Filenames should not contain spaces!" && exit 1 || exit 0'
language: system
always_run: true
pass_filenames: false
# Keep `suggestion` last
- id: suggestion
name: Suggestion
entry: bash -c 'echo "To bypass pre-commit hooks, add --no-verify to git commit."'
language: system
verbose: true
pass_filenames: false
# Insert new entries above the `suggestion` entry
+1
View File
@@ -0,0 +1 @@
3.12
+56
View File
@@ -0,0 +1,56 @@
# Repository Guidelines
## Project Structure & Module Organization
- Core Python package: `fastvideo/` (models, pipelines, training, distributed runtime, CLI entrypoints).
- CUDA/custom kernels: `fastvideo-kernel/` (separate build/test flow).
- Tests:
- `fastvideo/tests/` for package-level tests (dataset, encoders, inference, training, SSIM, workflow).
- `tests/local_tests/` for additional local/component checks.
- Docs and guides: `docs/` (MkDocs source), with contributor docs in `docs/contributing/`.
- Runnable examples and scripts: `examples/` and `scripts/`.
- Static assets: `assets/` (including `assets/images/`, `assets/videos/`, and `assets/prompts/`) and `comfyui/assets/`.
## Build, Test, and Development Commands
- `uv pip install -e .[dev]`: editable install with lint/test extras.
- `pre-commit install --hook-type pre-commit --hook-type commit-msg`: enable local hooks.
- `pre-commit run --all-files`: run formatter/lint/type/spelling checks.
- `pytest tests/`: run top-level test suite.
- `pytest fastvideo/tests/ -v`: run package tests.
- `pytest fastvideo/tests/ssim/ -vs`: run SSIM regression tests (GPU-heavy).
- `cd fastvideo-kernel && ./build.sh`: build kernel extensions.
## Coding Style & Naming Conventions
- Python 3.10+; 4-space indentation; keep code and imports readable and explicit.
- Style tools are configured in `pyproject.toml` and `.pre-commit-config.yaml`:
- `yapf` (format), `ruff` (lint, auto-fix), `mypy` (typing), `codespell`.
- Target line length is 80.
- Naming: `snake_case` for functions/files, `PascalCase` for classes, `UPPER_SNAKE_CASE` for constants.
## Testing Guidelines
- Use `pytest` and place tests near relevant domains (e.g., `fastvideo/tests/encoders/`).
- Prefer descriptive names like `test_<feature>_<expected_behavior>.py`.
- For new pipelines/backends, include at least one regression-oriented test; add SSIM coverage when output quality must be preserved.
- Document GPU assumptions in tests that require specific hardware.
## Commit & Pull Request Guidelines
- Follow existing commit style: short subject with optional tag prefix, e.g. `[bugfix]: ...`, `[feat]: ...`, `[misc]: ...`, and include PR reference like `(#1234)` when applicable.
- Keep commits focused by concern (feature, refactor, fix).
- PRs should include:
- clear problem/solution summary,
- test evidence (`pytest`/SSIM outputs or rationale if skipped),
- linked issue/PR context,
- screenshots or sample outputs for UI/demo/docs changes.
## Agent Infrastructure
This repository is agent-friendly. Before doing any work, read:
1. `.agents/onboarding/README.md` — full onboarding guide with step-by-step instructions.
2. `.agents/memory/codebase-map/README.md` — structural index of the entire repository.
3. `.agents/skills/` — available agent skills (check if one exists before writing code).
4. `.agents/workflows/` — SOPs for common procedures (experiment lifecycle, evaluation, etc.).
5. `.agents/lessons/` — known pitfalls and their documented fixes.
If you are exploring a new procedure that has no existing SOP, document your
progress in `.agents/exploration/` and flag it for review at the end of your
session.
+1
View File
@@ -0,0 +1 @@
@AGENTS.md
+183 -17
View File
@@ -1,21 +1,187 @@
MIT License
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
Copyright (c) 2024 PKU-YUAN's Group (袁粒课题组-北大信工) and Rabbitpre AI
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
1. Definitions.
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
+137 -88
View File
@@ -1,108 +1,157 @@
# Fast Video
This is currently based on Open-Sora-1.2.0: https://github.com/PKU-YuanGroup/Open-Sora-Plan/tree/294993ca78bf65dec1c3b6fb25541432c545eda9
<div align="center">
<img src=assets/logos/logo.svg width="30%"/>
</div>
## Envrironment
Change the index-url cuda version according to your system.
```
conda create -n fastvideo python=3.10.12
conda activate fastvideo
pip3 install torch==2.5.0 torchvision --index-url https://download.pytorch.org/whl/cu121
pip install git+https://github.com/huggingface/diffusers.git@76b7d86a9a5c0c2186efa09c4a67b5f5666ac9e3
pip install packaging ninja && pip install flash-attn==2.7.0.post2 --no-build-isolation
<p align="center">
| <a href="https://hao-ai-lab.github.io/FastVideo"><b>Documentation</b></a> | <a href="https://hao-ai-lab.github.io/FastVideo/inference/inference_quick_start/"><b> Quick Start</b></a> | <a href="https://github.com/hao-ai-lab/FastVideo/discussions/982" target="_blank"><b>Weekly Dev Meeting</b></a> | 🟣💬 <a href="https://join.slack.com/t/fastvideo/shared_invite/zt-3f4lao1uq-u~Ipx6Lt4J27AlD2y~IdLQ" target="_blank"> <b>Slack</b> </a> | 🟣💬 <a href="https://github.com/hao-ai-lab/FastVideo/discussions/1097" target="_blank"> <b> WeChat </b> </a> |
</p>
**FastVideo is a unified post-training and real-time inference framework for accelerated video generation.**
## NEWS
- `2026/03/17`: Release Live demo: [Into the Dreamverse: Vibe Directing in FastVideo](https://dreamverse.fastvideo.org/), check out the [Blog](https://haoailab.com/blogs/dreamverse/).
- `2026/03/13`: Release Live demo: [Create a 5s 1080p Video in 4.5s with FastVideo on a Single GPU](https://1080p.fastvideo.org/), check out the [Blog](https://haoailab.com/blogs/fastvideo_realtime_1080p/).
- `2025/11/19`: Release [CausalWan2.2 I2V A14B Preview](https://huggingface.co/FastVideo/CausalWan2.2-I2V-A14B-Preview-Diffusers) models, [Blog](https://hao-ai-lab.github.io/blogs/fastvideo_causalwan_preview/) and [Inference Code!](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_self_forcing_causal_wan2_2_i2v.py).
- `2025/08/04`: Release [FastWan](https://hao-ai-lab.github.io/FastVideo/distillation/dmd) models and [Sparse-Distillation](https://hao-ai-lab.github.io/blogs/fastvideo_post_training/).
### More News
- `2025/06/14`: Release finetuning and inference code for [VSA](https://arxiv.org/pdf/2505.13389).
- `2025/04/24`: [FastVideo V1](https://hao-ai-lab.github.io/blogs/fastvideo/) is released!
- `2025/02/18`: Release the inference code for [Sliding Tile Attention](https://hao-ai-lab.github.io/blogs/sta/).
## Key Features
FastVideo has the following features:
- End-to-end post-training support for bidirectional and autoregressive models:
- Support full finetuning and LoRA finetuning for state-of-the-art open video DiTs
- Data preprocessing pipeline for video, image, and text data
- Distribution Matching Distillation (DMD2) stepwise distillation.
- Sparse attention with [Video Sparse Attention](https://arxiv.org/pdf/2505.13389)
- [Sparse distillation](https://hao-ai-lab.github.io/blogs/fastvideo_post_training/) to achieve >50x denoising speedup
- Scalable training with FSDP2, sequence parallelism, and selective activation checkpointing.
- Causal distillation through Self-Forcing
- See this [page](https://hao-ai-lab.github.io/FastVideo/training/overview/) for full list of supported models and recipes.
- State-of-the-art performance optimizations for inference
- Sequence Parallelism for distributed inference
- Multiple state-of-the-art attention backends
- User-friendly CLI and Python API
- See this [page](https://hao-ai-lab.github.io/FastVideo/inference/optimizations/) for full list of supported optimizations.
- Diverse hardware and OS support
- Support H100, A100, 4090
- Support Linux, Windows, MacOS
- See this [page](https://hao-ai-lab.github.io/FastVideo/inference/support_matrix/) for full list of supported models, hardware assumptions, and optimization compatibility.
## Getting Started
We recommend using [uv](https://docs.astral.sh/uv/) to create a clean environment. If you previously used Conda, switching to uv generally gives faster and more stable installs.
```bash
# Create and activate a new uv environment
uv venv --python 3.12 --seed
source .venv/bin/activate
# Install FastVideo
uv pip install fastvideo
```
```
pip install -e . && pip install -e ".[train]"
sudo apt-get update && apt install screen && pip install watch gpustat
Please see our [docs](https://hao-ai-lab.github.io/FastVideo/getting_started/installation/) for more detailed installation instructions.
## Sparse Distillation
For our sparse distillation techniques, please see our [distillation docs](https://hao-ai-lab.github.io/FastVideo/distillation/dmd/) and check out our [blog](https://hao-ai-lab.github.io/blogs/fastvideo_post_training/).
See below for recipes and datasets:
| Model | Sparse Distillation | Dataset |
| ------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- |
| [FastWan2.1-T2V-1.3B](https://huggingface.co/FastVideo/FastWan2.1-T2V-1.3B-Diffusers) | [Recipe](https://github.com/hao-ai-lab/FastVideo/tree/main/examples/distill/Wan2.1-T2V/Wan-Syn-Data-480P) | [FastVideo Synthetic Wan2.1 480P](https://huggingface.co/datasets/FastVideo/Wan-Syn_77x448x832_600k) |
| [FastWan2.2-TI2V-5B](https://huggingface.co/FastVideo/FastWan2.2-TI2V-5B-Diffusers) | [Recipe](https://github.com/hao-ai-lab/FastVideo/tree/main/examples/distill/Wan2.2-TI2V-5B-Diffusers/Data-free) | [FastVideo Synthetic Wan2.2 720P](https://huggingface.co/datasets/FastVideo/Wan2.2-Syn-121x704x1280_32k) |
## Inference
### Generating Your First Video
Here's a minimal example to generate a video using the default settings. Make sure VSA kernels are [installed](https://hao-ai-lab.github.io/FastVideo/attention/vsa/#installation). Create a file called `example.py` with the following code:
```python
import os
from fastvideo import VideoGenerator
def main():
os.environ["FASTVIDEO_ATTENTION_BACKEND"] = "VIDEO_SPARSE_ATTN"
# Create a video generator with a pre-trained model
generator = VideoGenerator.from_pretrained(
"FastVideo/FastWan2.1-T2V-1.3B-Diffusers",
num_gpus=1, # Adjust based on your hardware
)
# Define a prompt for your video
prompt = "A curious raccoon peers through a vibrant field of yellow sunflowers, its eyes wide with interest."
# Generate the video
video = generator.generate_video(
prompt,
output_path="my_videos/", # Controls where videos are saved
save_video=True
)
if __name__ == '__main__':
main()
```
## Prepare Data & Models
We've prepared some debug data to facilitate development. To make sure the training pipeline is correct, train on the debug data and make sure the model overfit on it (feed it the same text prompt and see if the output video is the same as the training data)
Run the script with:
```
mkdir data && mkdir data/outputs/
python scripts/download_hf.py --repo_id=Stealths-Video/mochi_diffuser --local_dir=data/mochi --repo_type=model
python scripts/download_hf.py --repo_id=Stealths-Video/Merge-30k-Data --local_dir=data/Merge-30k-Data --repo_type=dataset
python scripts/download_hf.py --repo_id=Stealths-Video/validation_embeddings --local_dir=data/validation_embeddings --repo_type=dataset
cd data/Merge-30k-Data
cat Merged30K.tar.gz.part.* > Merged30K.tar.gz
rm Merged30K.tar.gz.part.*
tar --use-compress-program="pigz --processes 64" -xvf Merged30K.tar.gz
mv ephemeral/hao.zhang/codefolder/FastVideo-OSP/data/Merged-30K-Data/* .
rm -r ephemeral
rm Merged30K.tar.gz
cd ../..
```bash
python example.py
```
## Things Learned
1. shift8 clear but got structural artifacts
2. lq, 0.025 vague
3. adv not really helpful
4. shift8 euler steps 50 v.s. 100 very similar
5. 为啥image不会越distill越炸
6. EMA, 大batchsize, 1.5,2.5,3.5,4.5
7. Must have schedule
8. phase 1, 2 learning rate 5e-6不行
For a more detailed guide, please see our [inference quick start](https://hao-ai-lab.github.io/FastVideo/inference/inference_quick_start/).
## Experiments
Scripts are located at scripts/experiment_N.sh
## More Guides
1. pcm_linear_quadratic, euler_steps 50, 0.025
2. pcm_linear_quadratic, euler_steps 50, 0.05
3. shift 8, euler_steps 100
4. shift 8, euler_steps 50
5. shift 8, euler_steps 100, adv
6. pcm_linear_quadratic, euler_steps 50, 0.025, adv
7. pcm_linear_quadratic, euler_steps 50, 0.05, multiphase 125
8. pcm_linear_quadratic, euler_steps 50, 0.1, linear_range 0.75
9. pcm_linear_quadratic, euler_steps 50, 0.05, range 0.75
10. pcm_linear_quadratic, euler_steps 50, 0.05, batchsize 32
11. pcm_linear_quadratic, euler_steps 50, learning rate,1e-7
12. shift1, euler_steps 50
- [Design Overview](https://hao-ai-lab.github.io/FastVideo/design/overview/)
- [Distillation Guide](https://hao-ai-lab.github.io/FastVideo/distillation/dmd/)
- [Contribution Guide](https://hao-ai-lab.github.io/FastVideo/contributing/overview/)
13. pcm_linear_quadratic, euler_steps 50, 0.1, linear_range 0.75, phase 1
14. 4.5 cfg, validation no cfg, pcm_linear_quadratic, euler_steps 50, 0.1, linear_range 0.75
15. pcm_linear_quadratic, euler_steps 50, 0.15, linear_range 0.75
16. pcm_linear_quadratic, euler_steps 50, 0.1, linear_range 0.75 ema 0.95, decay 0.0
## Awesome work using FastVideo or our research projects
- [SGLang](https://github.com/sgl-project/sglang/tree/main/python/sglang/multimodal_gen): SGLang's diffusion inference functionality is based on a fork of FastVideo on Sept. 24, 2025.
- [DanceGRPO](https://github.com/XueZeyue/DanceGRPO): A unified framework to adapt Group Relative Policy Optimization (GRPO) to visual generation paradigms. Code based on FastVideo.
- [SRPO](https://github.com/Tencent-Hunyuan/SRPO): A method to directly align the full diffusion trajectory with fine-grained human preference. Code based on FastVideo.
- [DCM](https://github.com/Vchitect/DCM): Dual-expert consistency model for efficient and high-quality video generation. Code based on FastVideo.
- [HY-WorldPlay](https://github.com/Tencent-Hunyuan/HY-WorldPlay): An action-conditioned world model model trained using FastVideo framework.
- [Hunyuan Video 1.5](https://github.com/Tencent-Hunyuan/HunyuanVideo-1.5): A leading lightweight video generation model, where they proposed SSTA based on Sliding Tile Attention.
- [Kandinsky-5.0](https://github.com/kandinskylab/kandinsky-5): A family of diffusion models for video & image generation, where their NABLA attention includes a Sliding Tile Attention branch.
- [LongCat Video](https://github.com/meituan-longcat/LongCat-Video): A foundational video generation model with 13.6B parameters with block-sparse attention similar to Video Sparse Attention.
17. no cfg, validation no cfg, pcm_linear_quadratic, euler_steps 50, 0.1, linear_range 0.75
18. shift16, euler_steps 50
## 🤝 Contributing
19. 4step_infer_shift16_euler_50
20. 4step_infer_shift12_euler_50
21. 4step_infer_lq_euler_50_thresh0.1_lrg_0.75
22. pcm_linear_quadratic, euler_steps 50, 0.1, linear_range 0.75, phase 1, lr 1e-7
23. lq_euler_50_thres0.1_lrg_0.75_bs_64
24. lq_euler_50_thres0.1_lrg_0.75_lr5e-7
We welcome all contributions. Please check out our guide [here](https://hao-ai-lab.github.io/FastVideo/contributing/overview/).
See details in [development roadmap](https://github.com/hao-ai-lab/FastVideo/issues/899).
## Acknowledgement
We learned the design and reused code from the following projects: [Wan-Video](https://github.com/Wan-Video), [ThunderKittens](https://github.com/HazyResearch/ThunderKittens), [DMD2](https://github.com/tianweiy/DMD2), [diffusers](https://github.com/huggingface/diffusers), [xDiT](https://github.com/xdit-project/xDiT), [vLLM](https://github.com/vllm-project/vllm), [SGLang](https://github.com/sgl-project/sglang). We thank [MBZUAI](https://ifm.mbzuai.ac.ae/), [Anyscale](https://www.anyscale.com/), and [GMI Cloud](https://www.gmicloud.ai/) for their support throughout this project.
25. shift1_euler_50_0.75_phase1
26. kill
27. pcm_linear_quadratic, euler_steps 50, 0.1, linear_range 0.75, phase 1, ema 0.95, cfg 4.5
## Citation
28. lq_euler_50_thresh0.1_lrg_0.75_phase1_ema0.95
29. lq_euler_50_thres0.1_lrg_0.75_phase_ema0.95_cfg7
30. lq_euler_50_thresh0.1_lrg_0.75_phase1_ema0.98_cfg4.5
31. lq_euler_50_thresh0.1_lrg_0.75_phase1_lr_3e-7
32. lq_euler_50_thresh0.15_lrg_0.75_phase1_ema0.95_cfg4.5
33. lq_euler_50_thres0.1_linear_range_0.75_repro
34. lq_euler_50_thres0.1_lrg_0.75_reproduc
If you find FastVideo useful, please consider citing our research work:
35. pcm_linear_quadratic, euler_steps 50, 0.1, linear_range 0.75, phase 1, learning rate 5e-6
36. pcm_linear_quadratic, euler_steps 50, 0.1, linear_range 0.75, phase 2, learning rate 1e-6
37. pcm_linear_quadratic, euler_steps 50, 0.1, linear_range 0.75, phase 2, learning rate 5e-6
38. lq_euler_50_thres0.1_linear_range_0.75, learning rate 5e-6
39. lq_euler_50_thres0.1_linear_range_0.75, learning rate 1e-5
40. lq_euler_50_thres0.1_lrg_0.75_phase1_lr1e-6_repro
```bibtex
@article{zhang2025vsa,
title={Vsa: Faster video diffusion with trainable sparse attention},
author={Zhang, Peiyuan and Chen, Yongqi and Huang, Haofeng and Lin, Will and Liu, Zhengzhong and Stoica, Ion and Xing, Eric and Zhang, Hao},
journal={arXiv preprint arXiv:2505.13389},
year={2025}
}
41. lq_euler_50_thres0.1_lrg_0.75_reproduce
42. pcm_linear_quadratic, euler_steps 50, 0.1, linear_range 0.75, phase 4, learning rate 1e-6
43. pcm_linear_quadratic, euler_steps 50, 0.1, linear_range 0.75, phase 1, learning rate 1e-6, cfg 6.0
44. lq_euler_50_thres0.1_lrg_0.75_phase1_lr_5e-6_test_norm
45. lq_euler_50_thres0.1_lrg_0.75_phase1_lr_5e-6_pred_decay_0.1_latent14
46-48. lq_euler_50_thres0.1_lrg_0.75_phase1_lr1e-6, l2 or l1, decay weight 0.1 to 0.001
49.
@article{zhang2025fast,
title={Fast video generation with sliding tile attention},
author={Zhang, Peiyuan and Chen, Yongqi and Su, Runlong and Ding, Hangliang and Stoica, Ion and Liu, Zhengzhong and Zhang, Hao},
journal={arXiv preprint arXiv:2502.04507},
year={2025}
}
```
+10
View File
@@ -0,0 +1,10 @@
try:
from .comfyui.video_generator.nodes import (NODE_CLASS_MAPPINGS, NODE_DISPLAY_NAME_MAPPINGS)
WEB_DIRECTORY = "./web"
__all__ = ['NODE_CLASS_MAPPINGS', 'NODE_DISPLAY_NAME_MAPPINGS', 'WEB_DIRECTORY']
except ImportError:
# ComfyUI environment not available, skip comfyui imports
NODE_CLASS_MAPPINGS = {}
NODE_DISPLAY_NAME_MAPPINGS = {}
WEB_DIRECTORY = "./web"
__all__ = ['NODE_CLASS_MAPPINGS', 'NODE_DISPLAY_NAME_MAPPINGS', 'WEB_DIRECTORY']
Binary file not shown.

After

Width:  |  Height:  |  Size: 1.3 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.3 MiB

Binary file not shown.
Binary file not shown.
Binary file not shown.

After

Width:  |  Height:  |  Size: 22 MiB

Binary file not shown.
Binary file not shown.

After

Width:  |  Height:  |  Size: 194 KiB

+18
View File
@@ -0,0 +1,18 @@
<svg width="252" height="105" viewBox="0 0 252 105" fill="none" xmlns="http://www.w3.org/2000/svg">
<path d="M89.4843 55.5457H101.361L87.7028 101H74.638L89.4843 55.5457Z" fill="#356CFF"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M96.0167 1.00057H112.645L118.583 48.273H104.924L103.737 39.7882H79.9827L67.5117 55.5457H85.3273L43.1638 101H28.3174L22.3789 55.5457H33.6621L38.4129 91.3031L58.604 68.2729H44.3515L96.0167 1.00057ZM100.768 13.1217L87.7028 29.4852H103.143L100.768 13.1217Z" fill="#356CFF"/>
<path d="M37.2252 1.00057L22.3789 48.273H36.0375L40.7884 30.6974L62.6727 30.6974L69.6727 21.0005L43.7576 21.0004L46.7269 11.9096L77.6727 11.9096L86 1.00057L37.2252 1.00057Z" fill="#356CFF"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M108.488 55.5457L94.2351 101C94.2351 101 105.518 101 120.959 101C136.399 101 144.078 93.0133 148.276 79.788C152.432 68.0157 153.027 55.5457 136.399 55.5457C119.771 55.5457 108.488 55.5457 108.488 55.5457ZM109.081 90.697L116.802 65.8487C116.802 65.8487 120.959 65.8487 132.242 65.8487C143.525 65.8487 137.586 78.5759 135.211 84.0304C133.307 88.4021 127.491 90.697 122.74 90.697C117.989 90.697 109.081 90.697 109.081 90.697Z" fill="#356CFF"/>
<path d="M173.188 1.00056L168.625 11.9096C168.625 11.9096 149.386 11.9092 142.525 11.9095C135.664 11.9098 136.586 20.3944 141.337 20.3944H159.747C168.654 20.3944 166.961 33.6899 163.904 38.5761C160.467 44.0675 157.371 48.273 148.463 48.273L125.188 48.273L124 37.97L147.87 37.97C153.808 37.97 156.184 29.4852 151.433 29.4852H131.836C120.142 29.4852 125.897 1.00043 141.337 1.00043L173.188 1.00056Z" fill="#356CFF"/>
<path d="M179.938 1.00056L175.688 11.9096L191.221 11.9096L179.938 48.273H192.409L203.692 11.9096L219.132 11.9095L223.289 1.00043L179.938 1.00056Z" fill="#356CFF"/>
<path d="M161.341 55.5457H202.845L198.5 65.8487H169.654L167.279 73.7268H188.5L184.749 82.8177H164.31L161.934 90.697H190.251L186.624 101H146.494L161.341 55.5457Z" fill="#356CFF"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M230.821 54.9391C255.169 54.9391 251.776 67.0602 249.231 77.9692C246.686 88.8783 240.917 101 217.757 101C194.596 101 195.606 88.8783 199.347 77.9692C203.089 67.0602 206.473 54.9391 230.821 54.9391ZM237.948 77.9692C239.984 70.6965 240.917 65.242 228.446 65.242C215.975 65.242 211.818 71.9087 210.037 77.9692C208.255 84.0298 208.255 91.3025 219.538 91.3025C230.821 91.3025 235.911 85.2419 237.948 77.9692Z" fill="#356CFF"/>
<path d="M173.188 1.00056L168.625 11.9096C168.625 11.9096 149.386 11.9092 142.525 11.9095C135.664 11.9098 136.586 20.3944 141.337 20.3944M173.188 1.00056C173.188 1.00056 156.777 1.00043 141.337 1.00043M173.188 1.00056L141.337 1.00043M141.337 20.3944C146.088 20.3944 150.839 20.3944 159.747 20.3944M141.337 20.3944H159.747M159.747 20.3944C168.654 20.3944 166.961 33.6899 163.904 38.5761C160.467 44.0675 157.371 48.273 148.463 48.273M148.463 48.273C139.556 48.273 125.188 48.273 125.188 48.273M148.463 48.273L125.188 48.273M125.188 48.273L124 37.97M124 37.97C124 37.97 141.931 37.97 147.87 37.97M124 37.97L147.87 37.97M147.87 37.97C153.808 37.97 156.184 29.4852 151.433 29.4852M151.433 29.4852C146.682 29.4852 138.962 29.4852 131.836 29.4852M151.433 29.4852H131.836M131.836 29.4852C120.142 29.4852 125.897 1.00043 141.337 1.00043M37.2252 1.00057L22.3789 48.273H36.0375L40.7884 30.6974L62.6727 30.6974L69.6727 21.0005L43.7576 21.0004L46.7269 11.9096L77.6727 11.9096L86 1.00057L37.2252 1.00057ZM96.0167 1.00057H112.645L118.583 48.273H104.924L103.737 39.7882H79.9827L67.5117 55.5457H85.3273L43.1638 101H28.3174L22.3789 55.5457H33.6621L38.4129 91.3031L58.604 68.2729H44.3515L96.0167 1.00057ZM87.7028 29.4852L100.768 13.1217L103.143 29.4852H87.7028ZM89.4843 55.5457H101.361L87.7028 101H74.638L89.4843 55.5457ZM108.488 55.5457L94.2351 101C94.2351 101 105.518 101 120.959 101C136.399 101 144.078 93.0133 148.276 79.788C152.432 68.0157 153.027 55.5457 136.399 55.5457C119.771 55.5457 108.488 55.5457 108.488 55.5457ZM116.802 65.8487L109.081 90.697C109.081 90.697 117.989 90.697 122.74 90.697C127.491 90.697 133.307 88.4021 135.211 84.0304C137.586 78.5759 143.525 65.8487 132.242 65.8487C120.959 65.8487 116.802 65.8487 116.802 65.8487ZM179.938 1.00056L175.688 11.9096L191.221 11.9096L179.938 48.273H192.409L203.692 11.9096L219.132 11.9095L223.289 1.00043L179.938 1.00056ZM161.341 55.5457H202.845L198.5 65.8487H169.654L167.279 73.7268H188.5L184.749 82.8177H164.31L161.934 90.697H190.251L186.624 101H146.494L161.341 55.5457ZM230.821 54.9391C255.169 54.9391 251.776 67.0602 249.231 77.9692C246.686 88.8783 240.917 101 217.757 101C194.596 101 195.606 88.8783 199.347 77.9692C203.089 67.0602 206.473 54.9391 230.821 54.9391ZM228.446 65.242C240.917 65.242 239.984 70.6965 237.948 77.9692C235.911 85.2419 230.821 91.3025 219.538 91.3025C208.255 91.3025 208.255 84.0298 210.037 77.9692C211.818 71.9087 215.975 65.242 228.446 65.242Z" stroke="#356CFF" stroke-width="1.18771"/>
<path d="M15.2524 55.5451L21.191 100.999L24.7541 100.999L18.8156 55.5451L15.2524 55.5451Z" fill="#356CFF" stroke="#356CFF" stroke-width="1.18771"/>
<path d="M8.12646 55.5451L14.065 100.999L15.2527 100.999L9.31417 55.5451L8.12646 55.5451Z" fill="#356CFF" stroke="#356CFF" stroke-width="1.18771"/>
<path d="M1 55.5451L6.93853 100.999L7.53239 100.999L1.59385 55.5451L1 55.5451Z" fill="#356CFF" stroke="#356CFF" stroke-width="0.593853"/>
<path d="M15.2524 48.2724L30.0988 1H33.6619L18.8156 48.2724H15.2524Z" fill="#356CFF" stroke="#356CFF" stroke-width="1.18771"/>
<path d="M8.12646 48.2724L22.9728 1H24.1605L9.31417 48.2724H8.12646Z" fill="#356CFF" stroke="#356CFF" stroke-width="1.18771"/>
<path d="M1 48.2724L15.8463 1H16.4402L1.59385 48.2724H1Z" fill="#356CFF" stroke="#356CFF" stroke-width="0.593853"/>
<path d="M85.3271 55.5457H67.5116L87 12.7363L44.3513 68.2729H58.6038L43.1636 101L85.3271 55.5457Z" fill="#FDC717" stroke="#FDC717" stroke-width="1.18771" stroke-miterlimit="16"/>
</svg>

After

Width:  |  Height:  |  Size: 5.7 KiB

BIN
View File
Binary file not shown.

After

Width:  |  Height:  |  Size: 490 KiB

+6
View File
@@ -0,0 +1,6 @@
<svg width="160" height="93" viewBox="0 0 160 93" fill="none" xmlns="http://www.w3.org/2000/svg">
<path d="M28.8511 91.66L57.6319 1.86368H64.5394L35.7585 91.66H28.8511Z" fill="#356CFF" stroke="#356CFF" stroke-width="2.30244"/>
<path d="M15.0376 91.66L43.8185 1.86368H46.1209L17.3401 91.66H15.0376Z" fill="#356CFF" stroke="#356CFF" stroke-width="2.30244"/>
<path d="M1.22217 91.66L30.003 1.86366H31.1543L2.3734 91.66H1.22217Z" fill="#356CFF" stroke="#356CFF" stroke-width="1.15122"/>
<path d="M71.4465 1.86483L42.666 91.6599H69.144L78.3538 58.2746H123.251L129.007 39.855H84.1099L89.866 22.5868H152.032L157.788 1.86483H71.4465Z" fill="#356CFF" stroke="#356CFF" stroke-width="2.30244"/>
</svg>

After

Width:  |  Height:  |  Size: 691 B

Binary file not shown.

After

Width:  |  Height:  |  Size: 113 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.2 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 229 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 168 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 103 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 148 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 155 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 723 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 723 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 875 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 664 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 62 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 686 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 957 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 585 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 558 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 942 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 890 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 433 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 595 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 781 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 783 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 762 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 68 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 147 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 89 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 133 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 213 KiB

Some files were not shown because too many files have changed in this diff Show More