add readme

Co-authored-by: RandNMR73 <notomatthew31@gmail.com>
Co-authored-by: JerryZhou54 <zhouw.jerry2017@outlook.com>
Co-authored-by: Davids048 <jundasu@ucsd.edu>
Co-authored-by: Peiyuan Zhang <a1286225768@slurm-h200-204-239.slurm-compute.tenant-slurm.svc.cluster.local>

.github/workflows/docs.yml

@@ -45,6 +45,12 @@ jobs:
       - name: Setup Pages
         uses: actions/configure-pages@v4
 
+      - name: Generate docs examples
+        run: python docs/generate_examples.py
+
+      - name: Check docs links
+        run: python scripts/check_docs_links.py
+
       - name: Build documentation
         run: mkdocs build
 
@@ -63,4 +69,4 @@ jobs:
     steps:
       - name: Deploy to GitHub Pages
         id: deployment
-        uses: actions/deploy-pages@v4
+        uses: actions/deploy-pages@v4

.github/workflows/fastvideo-kernel-publish.yml

@@ -64,7 +64,10 @@ jobs:
             # - torch-version: '2.7.1'
             #   cuda-version: '12.8.0'
             #   torch-cuda-short: 'cu128'
-            - torch-version: '2.9.1'
+            # - torch-version: '2.9.1'
+            #   cuda-version: '12.8.0'
+            #   torch-cuda-short: 'cu128'
+            - torch-version: '2.10.0'
               cuda-version: '12.8.0'
               torch-cuda-short: 'cu128'
 

.gitignore

@@ -18,6 +18,7 @@ venv/
 .venv/
 runs/
 samples/
+Miniconda3-latest-Linux-x86_64.sh
 *validation/
 data/
 outputs/
@@ -33,6 +34,11 @@ env
 *.log
 weights/
 
+# SSIM test outputs
+fastvideo/tests/ssim/generated_videos/
+**/.cache/**
+
+
 # Distribution / packaging
 build/
 dist/
@@ -69,6 +75,11 @@ docs/distillation/examples/
 !docs/assets/images/**/*.png
 !comfyui/assets/**/*.png
 !comfyui/assets/**/*.gif
+!assets/images/**/*.png
+!assets/images/**/*.jpg
+!assets/images/**/*.jpeg
+!assets/images/**/*.gif
+!assets/videos/**/*.mp4
 
 dmd_t2v_output/
 preprocess_output_text/

.pre-commit-config.yaml

@@ -10,7 +10,7 @@ exclude: |
     demo/.*|
     predict\.py|
     scripts/.*|
-    prompts/.*|
+    assets/prompts/.*|
     fastvideo/data_preprocess/.*|
     fastvideo/dataset/.*|
     fastvideo/models/.*|

AGENTS.md

@@ -0,0 +1,42 @@
+# Repository Guidelines
+
+## Project Structure & Module Organization
+- Core Python package: `fastvideo/` (models, pipelines, training, distributed runtime, CLI entrypoints).
+- CUDA/custom kernels: `fastvideo-kernel/` (separate build/test flow).
+- Tests:
+  - `fastvideo/tests/` for package-level tests (dataset, encoders, inference, training, SSIM, workflow).
+  - `tests/local_tests/` for additional local/component checks.
+- Docs and guides: `docs/` (MkDocs source), with contributor docs in `docs/contributing/`.
+- Runnable examples and scripts: `examples/` and `scripts/`.
+- Static assets: `assets/` (including `assets/images/`, `assets/videos/`, and `assets/prompts/`) and `comfyui/assets/`.
+
+## Build, Test, and Development Commands
+- `uv pip install -e .[dev]`: editable install with lint/test extras.
+- `pre-commit install --hook-type pre-commit --hook-type commit-msg`: enable local hooks.
+- `pre-commit run --all-files`: run formatter/lint/type/spelling checks.
+- `pytest tests/`: run top-level test suite.
+- `pytest fastvideo/tests/ -v`: run package tests.
+- `pytest fastvideo/tests/ssim/ -vs`: run SSIM regression tests (GPU-heavy).
+- `cd fastvideo-kernel && ./build.sh`: build kernel extensions.
+
+## Coding Style & Naming Conventions
+- Python 3.10+; 4-space indentation; keep code and imports readable and explicit.
+- Style tools are configured in `pyproject.toml` and `.pre-commit-config.yaml`:
+  - `yapf` (format), `ruff` (lint, auto-fix), `mypy` (typing), `codespell`.
+- Target line length is 80.
+- Naming: `snake_case` for functions/files, `PascalCase` for classes, `UPPER_SNAKE_CASE` for constants.
+
+## Testing Guidelines
+- Use `pytest` and place tests near relevant domains (e.g., `fastvideo/tests/encoders/`).
+- Prefer descriptive names like `test_<feature>_<expected_behavior>.py`.
+- For new pipelines/backends, include at least one regression-oriented test; add SSIM coverage when output quality must be preserved.
+- Document GPU assumptions in tests that require specific hardware.
+
+## Commit & Pull Request Guidelines
+- Follow existing commit style: short subject with optional tag prefix, e.g. `[bugfix]: ...`, `[feat]: ...`, `[misc]: ...`, and include PR reference like `(#1234)` when applicable.
+- Keep commits focused by concern (feature, refactor, fix).
+- PRs should include:
+  - clear problem/solution summary,
+  - test evidence (`pytest`/SSIM outputs or rationale if skipped),
+  - linked issue/PR context,
+  - screenshots or sample outputs for UI/demo/docs changes.

CLAUDE.md

@@ -0,0 +1 @@
+@AGENTS.md

README.md

@@ -1,169 +1,101 @@
-<div align="center">
-<img src=assets/logos/logo.svg width="30%"/>
-</div>
+# Sliding Tile Attention (STA) Branch
 
-<p align="center">
-     | <a href="https://hao-ai-lab.github.io/FastVideo"><b>Documentation</b></a> | <a href="https://hao-ai-lab.github.io/FastVideo/inference/inference_quick_start/"><b> Quick Start</b></a> | <a href="https://github.com/hao-ai-lab/FastVideo/discussions/982"  target="_blank"><b>Weekly Dev Meeting</b></a>  | 🟣💬 <a href="https://join.slack.com/t/fastvideo/shared_invite/zt-3f4lao1uq-u~Ipx6Lt4J27AlD2y~IdLQ" target="_blank"> <b>Slack</b> </a> |  🟣💬 <a href="https://ibb.co/sv3MMKyv" target="_blank"> <b> WeChat </b> </a> |
-</p>
+This branch is a stash/testing branch for people who want to try
+Sliding Tile Attention (STA). The top-level README is intentionally
+STA-only.
 
-**FastVideo is a unified post-training and inference framework for accelerated video generation.**
+## What is STA
 
-## NEWS
-- ```2025/11/19```: Release [CausalWan2.2 I2V A14B Preview](https://huggingface.co/FastVideo/CausalWan2.2-I2V-A14B-Preview-Diffusers) models, [Blog](https://hao-ai-lab.github.io/blogs/fastvideo_causalwan_preview/) and [Inference Code!](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_self_forcing_causal_wan2_2_i2v.py)
-- ```2025/08/04```: Release [FastWan](https://hao-ai-lab.github.io/FastVideo/distillation/dmd) models and [Sparse-Distillation](https://hao-ai-lab.github.io/blogs/fastvideo_post_training/).
+Sliding Tile Attention is an optimized attention backend for
+window-based video generation.
 
-<details>
-<summary>More</summary>
+- Blog: https://hao-ai-lab.github.io/blogs/sta/
+- Paper: https://arxiv.org/abs/2502.04507
+- In-repo STA docs: `docs/attention/sta/index.md`
 
-- ```2025/06/14```: Release finetuning and inference code for [VSA](https://arxiv.org/pdf/2505.13389)
-- ```2025/04/24```: [FastVideo V1](https://hao-ai-lab.github.io/blogs/fastvideo/) is released!
-- ```2025/02/18```: Release the inference code for [Sliding Tile Attention](https://hao-ai-lab.github.io/blogs/sta/).
+## Setup
 
-</details>
-
-## Key Features
-
-FastVideo has the following features:
-- End-to-end post-training support for bidirectional and autoregressive models:
-  - Support full finetuning and LoRA finetuning for state-of-the-art open video DiTs
-  - Data preprocessing pipeline for video, image, and text data
-  - Distribution Matching Distillation (DMD2) stepwise distillation.
-  - Sparse attention with [Video Sparse Attention](https://arxiv.org/pdf/2505.13389)
-  - [Sparse distillation](https://hao-ai-lab.github.io/blogs/fastvideo_post_training/) to achineve >50x denoising speedup
-  - Scalable training with FSDP2, sequence parallelism, and selective activation checkpointing.
-  - Causal distillation through Self-Forcing
-  - See this [page](https://hao-ai-lab.github.io/FastVideo/training/overview/) for full list of supported models and recipes.
-- State-of-the-art performance optimizations for inference
-  - Sequence Parallelism for distributed inference
-  - Multiple state-of-the-art attention backends
-  - User-friendly CLI and Python API
-  - See this [page](https://hao-ai-lab.github.io/FastVideo/inference/optimizations/) for full list of supported optimizations.
-- Diverse hardware and OS support
-  - Support H100, A100, 4090
-  - Support Linux, Windows, MacOS
-  - See this [page](https://hao-ai-lab.github.io/FastVideo/inference/hardware_support/) for full list of supported hardware and OS.
-
-## Getting Started
-We recommend using an environment manager such as `Conda` to create a clean environment:
+Install FastVideo from source:
 
 ```bash
-# Create and activate a new conda environment
-conda create -n fastvideo python=3.12
-conda activate fastvideo
-
-# Install FastVideo
-pip install fastvideo
+uv pip install -e .
 ```
 
-Please see our [docs](https://hao-ai-lab.github.io/FastVideo/getting_started/installation/) for more detailed installation instructions.
-
-## Sparse Distillation
-For our sparse distillation techniques, please see our [distillation docs](https://hao-ai-lab.github.io/FastVideo/distillation/dmd/) and check out our [blog](https://hao-ai-lab.github.io/blogs/fastvideo_post_training/).
-
-See below for recipes and datasets:
-
-|                                            Model                                              |                                               Sparse Distillation                                                 |                                                  Dataset                                                  |
-|:-------------------------------------------------------------------------------------------:  |:---------------------------------------------------------------------------------------------------------------:  |:--------------------------------------------------------------------------------------------------------: |
-| [FastWan2.1-T2V-1.3B](https://huggingface.co/FastVideo/FastWan2.1-T2V-1.3B-Diffusers)         |    [Recipe](https://github.com/hao-ai-lab/FastVideo/tree/main/examples/distill/Wan2.1-T2V/Wan-Syn-Data-480P)      | [FastVideo Synthetic Wan2.1 480P](https://huggingface.co/datasets/FastVideo/Wan-Syn_77x448x832_600k)      |
-| [FastWan2.1-T2V-14B-Preview](https://huggingface.co/FastVideo/FastWan2.1-T2V-14B-Diffusers)   |                                                   Coming soon!                                                    |   [FastVideo Synthetic Wan2.1 720P](https://huggingface.co/datasets/FastVideo/Wan-Syn_77x768x1280_250k)   |
-| [FastWan2.2-TI2V-5B](https://huggingface.co/FastVideo/FastWan2.2-TI2V-5B-Diffusers)           | [Recipe](https://github.com/hao-ai-lab/FastVideo/tree/main/examples/distill/Wan2.2-TI2V-5B-Diffusers/Data-free)   | [FastVideo Synthetic Wan2.2 720P](https://huggingface.co/datasets/FastVideo/Wan2.2-Syn-121x704x1280_32k)  |
-
-## Inference
-### Generating Your First Video
-Here's a minimal example to generate a video using the default settings. Make sure VSA kernels are [installed](https://hao-ai-lab.github.io/FastVideo/video_sparse_attention/installation/). Create a file called `example.py` with the following code:
-
-```python
-import os
-from fastvideo import VideoGenerator
-
-def main():
-    os.environ["FASTVIDEO_ATTENTION_BACKEND"] = "VIDEO_SPARSE_ATTN"
-
-    # Create a video generator with a pre-trained model
-    generator = VideoGenerator.from_pretrained(
-        "FastVideo/FastWan2.1-T2V-1.3B-Diffusers",
-        num_gpus=1,  # Adjust based on your hardware
-    )
-
-    # Define a prompt for your video
-    prompt = "A curious raccoon peers through a vibrant field of yellow sunflowers, its eyes wide with interest."
-
-    # Generate the video
-    video = generator.generate_video(
-        prompt,
-        return_frames=True,  # Also return frames from this call (defaults to False)
-        output_path="my_videos/",  # Controls where videos are saved
-        save_video=True
-    )
-
-if __name__ == '__main__':
-    main()
-```
-
-Run the script with:
+Build the STA kernel package:
 
 ```bash
-python example.py
+cd fastvideo-kernel
+./build.sh
+cd ..
 ```
 
-For a more detailed guide, please see our [inference quick start](https://hao-ai-lab.github.io/FastVideo/inference/inference_quick_start/).
+## Run STA Inference
 
-### Other docs:
+STA backend:
 
-- [Design Overview](https://hao-ai-lab.github.io/FastVideo/design/overview/)
-- [Contribution Guide](https://hao-ai-lab.github.io/FastVideo/getting_started/installation/)
+```bash
+export FASTVIDEO_ATTENTION_BACKEND=SLIDING_TILE_ATTN
+```
 
-## Distillation and Finetuning
-- [Distillation Guide](https://hao-ai-lab.github.io/FastVideo/distillation/dmd/)
-<!-- - [Finetuning Guide](https://hao-ai-lab.github.io/FastVideo/training/finetune.html) -->
+Ready-to-run examples:
 
-## Awesome work using FastVideo or our research projects
+- HunyuanVideo: `scripts/inference/v1_inference_hunyuan_STA.sh`
+- Wan2.1-T2V-14B: `scripts/inference/v1_inference_wan_STA.sh`
 
-- [SGLang](https://github.com/sgl-project/sglang/tree/main/python/sglang/multimodal_gen): SGLang's diffusion inference functionality is based  on a fork of FastVideo on Sept. 24, 2025. [![Star](https://img.shields.io/github/stars/sgl-project/sglang.svg?style=social&label=Star)](https://github.com/sgl-project/sglang)
+Run:
 
-- [DanceGRPO](https://github.com/XueZeyue/DanceGRPO): A  unified framework to adapt Group Relative Policy Optimization (GRPO) to visual generation paradigms. Code based on FastVideo. [![Star](https://img.shields.io/github/stars/XueZeyue/DanceGRPO.svg?style=social&label=Star)](https://github.com/XueZeyue/DanceGRPO)
-- [SRPO](https://github.com/Tencent-Hunyuan/SRPO): A method to directly align the full diffusion trajectory with fine-grained human preference. Code based on FastVideo. [![Star](https://img.shields.io/github/stars/Tencent-Hunyuan/SRPO.svg?style=social&label=Star)](https://github.com/Tencent-Hunyuan/SRPO)
-- [DCM](https://github.com/Vchitect/DCM): Dual-expert consistency model for efficient and high-quality video generation. Code based on FastVideo. [![Star](https://img.shields.io/github/stars/Vchitect/DCM.svg?style=social&label=Star)](https://github.com/Vchitect/DCM)
-- [Hunyuan Video 1.5](https://github.com/Tencent-Hunyuan/HunyuanVideo-1.5): A leading lightweight video generation model, where they proposed SSTA based on Sliding Tile Attention. [![Star](https://img.shields.io/github/stars/Tencent-Hunyuan/HunyuanVideo-1.5.svg?style=social&label=Star)](https://github.com/Tencent-Hunyuan/HunyuanVideo-1.5)
-- [Kandinsky-5.0](https://github.com/kandinskylab/kandinsky-5): A family of diffusion models for video & image generation, where their NABLA attention includes a Sliding Tile Attention branch. [![Star](https://img.shields.io/github/stars/kandinskylab/kandinsky-5.svg?style=social&label=Star)](https://github.com/kandinskylab/kandinsky-5)
-- [LongCat Video](https://github.com/meituan-longcat/LongCat-Video): A foundational video generation model with 13.6B parameters with block-sparse attention similar to Video Sparse Attention. [![Star](https://img.shields.io/github/stars/meituan-longcat/LongCat-Video.svg?style=social&label=Star)](https://github.com/meituan-longcat/LongCat-Video)
+```bash
+bash scripts/inference/v1_inference_hunyuan_STA.sh
+# or
+bash scripts/inference/v1_inference_wan_STA.sh
+```
 
-## 🤝 Contributing
+Both scripts already set STA-related env vars:
 
-We welcome all contributions. Please check out our guide [here](https://hao-ai-lab.github.io/FastVideo/contributing/overview/).
-See details in [development roadmap](https://github.com/hao-ai-lab/FastVideo/issues/899).
-## Acknowledgement
-We learned and reused code from the following projects:
-- [Wan-Video](https://github.com/Wan-Video)
-- [ThunderKittens](https://github.com/HazyResearch/ThunderKittens)
-- [Triton](https://github.com/triton-lang/triton)
-- [DMD2](https://github.com/tianweiy/DMD2)
-- [diffusers](https://github.com/huggingface/diffusers)
-- [xDiT](https://github.com/xdit-project/xDiT)
-- [vLLM](https://github.com/vllm-project/vllm)
-- [SGLang](https://github.com/sgl-project/sglang)
+- `FASTVIDEO_ATTENTION_BACKEND=SLIDING_TILE_ATTN`
+- `FASTVIDEO_ATTENTION_CONFIG` to an STA mask strategy JSON
 
-We thank [MBZUAI](https://ifm.mbzuai.ac.ae/), [Anyscale](https://www.anyscale.com/), and [GMI Cloud](https://www.gmicloud.ai/) for their support throughout this project.
+## STA Mask Strategy Files
+
+- HunyuanVideo config: `assets/mask_strategy_hunyuan.json`
+- Wan config: `assets/mask_strategy_wan.json`
+
+## STA Mask Search (Wan2.1-T2V-14B)
+
+Run mask search + tuning from repo root:
+
+```bash
+bash examples/inference/sta_mask_search/inference_wan_sta.sh
+```
+
+What this script does:
+
+- Runs `STA_searching` first.
+- Runs `STA_tuning` next (`skip_time_steps=12` by default).
+- Uses prompt shards from `assets/prompt_0.txt` to `assets/prompt_7.txt`.
+
+Important notes:
+
+- Default script is set to 8 GPUs (`num_gpu=8`). If needed, edit
+  `examples/inference/sta_mask_search/inference_wan_sta.sh`.
+- STA searching/tuning currently supports `69x768x1280` (Wan setting).
+
+Generated files:
+
+- Search results: `output/mask_search_result_pos_1280x768/`
+- Tuned strategy: `output/mask_search_strategy_1280x768/mask_strategy_s12.json`
+
+Use the tuned mask for STA inference:
+
+```bash
+export FASTVIDEO_ATTENTION_BACKEND=SLIDING_TILE_ATTN
+export FASTVIDEO_ATTENTION_CONFIG=output/mask_search_strategy_1280x768/mask_strategy_s12.json
+python examples/inference/sta_mask_search/wan_example.py --STA_mode STA_inference --num_gpus 1
+```
 
 ## Citation
-If you find FastVideo useful, please considering citing our work:
 
 ```bibtex
-@software{fastvideo2024,
-  title        = {FastVideo: A Unified Framework for Accelerated Video Generation},
-  author       = {The FastVideo Team},
-  url          = {https://github.com/hao-ai-lab/FastVideo},
-  month        = apr,
-  year         = {2024},
-}
-
-@article{zhang2025vsa,
-  title={Vsa: Faster video diffusion with trainable sparse attention},
-  author={Zhang, Peiyuan and Chen, Yongqi and Huang, Haofeng and Lin, Will and Liu, Zhengzhong and Stoica, Ion and Xing, Eric and Zhang, Hao},
-  journal={arXiv preprint arXiv:2505.13389},
-  year={2025}
-}
-
 @article{zhang2025fast,
   title={Fast video generation with sliding tile attention},
   author={Zhang, Peiyuan and Chen, Yongqi and Su, Runlong and Ding, Hangliang and Stoica, Ion and Liu, Zhengzhong and Zhang, Hao},

assets/prompt_0.txt

@@ -0,0 +1 @@
+Will Smith casually eats noodles, his relaxed demeanor contrasting with the energetic background of a bustling street food market. The scene captures a mix of humor and authenticity. Mid-shot framing, vibrant lighting.

assets/prompt_1.txt

@@ -0,0 +1 @@
+A lone hiker stands atop a towering cliff, silhouetted against the vast horizon. The rugged landscape stretches endlessly beneath, its earthy tones blending into the soft blues of the sky. The scene captures the spirit of exploration and human resilience. High angle, dynamic framing, with soft natural lighting emphasizing the grandeur of nature.

assets/prompt_2.txt

@@ -0,0 +1 @@
+A hand with delicate fingers picks up a bright yellow lemon from a wooden bowl filled with lemons and sprigs of mint against a peach-colored background. The hand gently tosses the lemon up and catches it, showcasing its smooth texture. A beige string bag sits beside the bowl, adding a rustic touch to the scene. Additional lemons, one halved, are scattered around the base of the bowl. The even lighting enhances the vibrant colors and creates a fresh, inviting atmosphere.

assets/prompt_3.txt

@@ -0,0 +1 @@
+A curious raccoon peers through a vibrant field of yellow sunflowers, its eyes wide with interest. The playful yet serene atmosphere is complemented by soft natural light filtering through the petals. Mid-shot, warm and cheerful tones.

assets/prompt_4.txt

@@ -0,0 +1 @@
+A superintelligent humanoid robot waking up. The robot has a sleek metallic body with futuristic design features. Its glowing red eyes are the focal point, emanating a sharp, intense light as it powers on. The scene is set in a dimly lit, high-tech laboratory filled with glowing control panels, robotic arms, and holographic screens. The setting emphasizes advanced technology and an atmosphere of mystery. The ambiance is eerie and dramatic, highlighting the moment of awakening and the robots immense intelligence. Photorealistic style with a cinematic, dark sci-fi aesthetic. Aspect ratio: 16:9 --v 6.1

assets/prompt_5.txt

@@ -0,0 +1 @@
+fox in the forest close-up quickly turned its head to the left

assets/prompt_6.txt

@@ -0,0 +1 @@
+Man walking his dog in the woods on a hot sunny day

assets/prompt_7.txt

@@ -0,0 +1 @@
+A majestic lion strides across the golden savanna, its powerful frame glistening under the warm afternoon sun. The tall grass ripples gently in the breeze, enhancing the lion's commanding presence. The tone is vibrant, embodying the raw energy of the wild. Low angle, steady tracking shot, cinematic.

assets/prompts/mixkit_i2v.jsonl

@@ -1,110 +1,110 @@
 [
     {
         "prompt": "Young man skating with a skateboard on the ramps with graffiti of a park with trees, on a sunny day.", 
-        "image_path": "images/mixkit-boy-skating-with-a-skateboard-in-a-park-with-ramps-34389.png"
+        "image_path": "assets/images/mixkit-boy-skating-with-a-skateboard-in-a-park-with-ramps-34389.png"
     },
     {
         "prompt": "In the midst of the joyous New Year's Eve celebration, the cheerful group of friends, their spirits lifted by the festivities, decides to immortalize the moment with a vibrant snapshot", 
-        "image_path": "images/mixkit-a-cheerful-group-of-friends-celebrate-new-years-eve-and-51525.png"
+        "image_path": "assets/images/mixkit-a-cheerful-group-of-friends-celebrate-new-years-eve-and-51525.png"
     },
     {
         "prompt": "A man and a woman playing in a field with grass, during a bright afternoon, while cars pass by in the distance.", 
-        "image_path": "images/mixkit-a-cute-couple-playing-on-the-grass-4688.png"
+        "image_path": "assets/images/mixkit-a-cute-couple-playing-on-the-grass-4688.png"
     },
     {
         "prompt": "Aerial view of a rocky mountain in the forest at a sunny day drone flight footage", 
-        "image_path": "images/mixkit-aerial-view-of-a-rocky-mountain-in-the-forest-50589.png"
+        "image_path": "assets/images/mixkit-aerial-view-of-a-rocky-mountain-in-the-forest-50589.png"
     },
     {
         "prompt": "A little girl wearing a pink security helmet and denim overall discovers the art of cycling amidst the serene park, as the camera captures her graceful progress.", 
-        "image_path": "images/mixkit-a-little-girl-cruises-through-the-forest-path-on-her-50088.png"
+        "image_path": "assets/images/mixkit-a-little-girl-cruises-through-the-forest-path-on-her-50088.png"
     },
     {
         "prompt": "Aerial shot of a beach shore with sea waves. Big rocks on the sand at an alone beach.", 
-        "image_path": "images/mixkit-aerial-shot-of-a-beach-with-sea-waves-1087.png"
+        "image_path": "assets/images/mixkit-aerial-shot-of-a-beach-with-sea-waves-1087.png"
     },
     {
         "prompt": "Young woman cleaning her house decorated with plants and decorations, while dancing happily to music in her headphones.", 
-        "image_path": "images/mixkit-woman-cleaning-her-house-dancing-happy-43379.png"
+        "image_path": "assets/images/mixkit-woman-cleaning-her-house-dancing-happy-43379.png"
     },
     {
         "prompt": "Aerial tour in a meadow surrounded by hills on the horizon, while some birds fly low over a lake.", 
-        "image_path": "images/mixkit-birds-flying-low-over-a-lake-in-a-meadow-41417.png"
+        "image_path": "assets/images/mixkit-birds-flying-low-over-a-lake-in-a-meadow-41417.png"
     },
     {
         "prompt": "In the video, a lone rider guides a majestic horse across an expansive, open field as the sun sets in the background. The rider, dressed in a classic blue shirt and wide-brimmed hat, sits confidently in the saddle, silhouetted against the warm glow of the evening sky. The horse moves gracefully, its mane and tail flowing with each step, creating a sense of harmony between horse and rider. Surrounding the pair, towering trees form a natural border, their leaves gently rustling in the breeze. The shadows lengthen on the ground, accentuating the serene and timeless feel of the scene. The distant hills and wooden fences frame the horizon, adding depth to the tranquil landscape. A few horses graze peacefully in the background, blending into the pastoral setting. The overall ambiance evokes a sense of calmness and quietude, capturing a perfect moment in the golden light of dusk.", 
-        "image_path": "images/mixkit-a-rancher-riding-a-horse-at-sunset-1143.png"
+        "image_path": "assets/images/mixkit-a-rancher-riding-a-horse-at-sunset-1143.png"
     },
     {
         "prompt": "In the video, a martial artist dressed in a traditional white uniform with a black belt demonstrates a series of precise movements against a stark black background. The individual gracefully transitions between stances, embodying a sense of focused discipline and control. Each motion is executed with a deliberate pace, showcasing the fluidity of martial arts techniques. The soft lighting creates subtle highlights on the uniform, adding depth to the figure as it moves. The practitioner begins with an open-hand pose, feet firmly grounded, gradually shifting to a powerful forward punch. The fluidity of the sequence displays a mastery of balance and poise. Every trajectory of the limbs is precise and deliberate, capturing the elegance and strength of martial arts. The serene, isolated setting enhances the intensity and concentration of the practitioner. This visual presentation is an elegant interplay of motion and stillness, displaying the art form's discipline and grace.", 
-        "image_path": "images/mixkit-a-young-man-practicing-his-karate-moves-49635.png"
+        "image_path": "assets/images/mixkit-a-young-man-practicing-his-karate-moves-49635.png"
     },
     {
         "prompt": "In a serene and softly lit yoga studio, three individuals engage in a yoga session, each performing an upward-facing stretch. The central figure is a woman with shoulder-length brown hair, dressed in a light cropped top and green leggings, her posture reflecting grace and concentration. To her right, another participant, a woman in a purple outfit, mirrors the pose with equal poise. On her left, a person with a bun focuses intently, supported slightly by yoga blocks beneath their hands. The warm-colored wooden floor contrasts soothingly with the soft pastel mural on the back wall, featuring an abstract design and partial visage of a serene face. Natural light floods the space from a large window on the right, where lush greens peek through, adding an element of tranquility. In the corner of the room, a collection of meditation instruments, including a gong and a Buddha statue, subtly frame the peaceful setting. The mood is calm yet focused, as all three participants are deeply engaged in their practice. The scene combines elements of balance, harmony, and a shared journey towards mindfulness. This depiction captures the essence of a yoga session that blends personal growth with collective experience.", 
-        "image_path": "images/mixkit-small-group-of-people-doing-yoga-together-43730.png"
+        "image_path": "assets/images/mixkit-small-group-of-people-doing-yoga-together-43730.png"
     },
     {
         "prompt": "In the deep blue expanse of the ocean, two dolphins glide effortlessly, their sleek bodies reflecting the sunlight filtering through the water. The prominent shadows and caustics create a shimmering effect on their skin, capturing the beauty of their natural habitat. Each dolphin moves with a fluid grace, occasionally interacting with gentle nudges, showcasing their playful and social nature. The scene is vibrant and dynamic, with the clear blue background accentuating the dolphins' movements, making it an ideal subject for AI recreation.", 
-        "image_path": "images/mixkit-dolphins-underwater-4133.png"
+        "image_path": "assets/images/mixkit-dolphins-underwater-4133.png"
     },
     {
         "prompt": "A bustling ski slope comes alive with skiers descending a pristine, snow-covered hill, surrounded by towering, snow-draped evergreens. Several figures stand atop the slope, silhouetted against a clear blue sky, preparing to embark on their ski run. The chair lift on the right continuously drops off eager adventurers, adding to the excitement at the hilltop. Each skier, clad in colorful winter gear, carves distinct paths into the textured snow as they weave their way down. The interplay of sunlight and shadows accentuates the myriad tracks etched into the slope, creating a dynamic visual rhythm. The scene captures a vibrant winter wonderland, full of action and the thrill of a perfect ski day.", 
-        "image_path": "images/mixkit-skiers-on-a-snowy-slope-3327.png"
+        "image_path": "assets/images/mixkit-skiers-on-a-snowy-slope-3327.png"
     },
     {
         "prompt": "A determined climber is scaling a massive rock face, showcasing exceptional strength and skill. The person, clad in a teal shirt and dark pants, climbs with precision, their movements measured and deliberate. They are secured by climbing gear, which includes ropes and a harness, emphasizing their commitment to safety. The rugged texture of the sandy-colored rock provides an imposing backdrop, adding drama and scale to the climb. In the distance, other large rock formations and sparse vegetation can be seen under a bright, overcast sky, contributing to the natural and adventurous atmosphere. The scene captures a moment of focus and challenge, highlighting the climber's tenacity and the breathtaking environment.",
-        "image_path": "images/mixkit-alpinist-climbing-a-huge-rock-in-a-desert-43306.png"
+        "image_path": "assets/images/mixkit-alpinist-climbing-a-huge-rock-in-a-desert-43306.png"
     },
     {
         "prompt": "A silver SUV drives along a winding, snow-covered mountain road, with dense pine trees blanketed in snow lining both sides. The scene is serene, with the vehicle moving smoothly, possibly on a winter journey or vacation. As the SUV disappears around the bend, another, darker SUV follows, creating a sense of motion and perspective on the snow-dusted asphalt. The towering, snow-laden rock formation to the right contrasts with the dark green of the pines, highlighting the peacefulness of the wintry landscape.",
-        "image_path": "images/mixkit-curve-on-a-snowy-forest-road-3317.png"
+        "image_path": "assets/images/mixkit-curve-on-a-snowy-forest-road-3317.png"
     },
     {
         "prompt": "A solitary boat glides across the expansive, tranquil expanse of a serene lake. The vessel leaves a gentle wake behind, creating delicate ripples across the mirror-like surface. The water appears a rich shade of teal, seamlessly blending with the sky at the horizon. Silhouettes of distant trees are faintly visible, creating a picturesque backdrop that enhances the solitary journey of the boat. The sky is a calm gradient, shifting from soft oranges near the shore to the pale blues above. In the distance, a few slender poles emerge from the water, remnants of an old structure or natural formation. The mood of the scene is one of peace and solitude, with the boat journeying steadily through the quiet landscape. There is a sense of endless possibilities as the boat moves toward the unseen beyond the frame. The simplicity and stillness of the scene invite contemplation and reflection, encapsulating a perfect moment of quietude on the water.",
-        "image_path": "images/mixkit-motorboat-on-a-large-lake-with-turquoise-blue-waters-4996.png"
+        "image_path": "assets/images/mixkit-motorboat-on-a-large-lake-with-turquoise-blue-waters-4996.png"
     },
     {
         "prompt": "A man wearing grey shorts jumps rope in a gym, weights and gym equipment in the background.",
-        "image_path": "images/gray_short_man.jpg"
+        "image_path": "assets/images/gray_short_man.jpg"
     },
     {
         "prompt": "Flying over a peninsula covered in bushy trees, while discovering the sea around it, painted a beautiful turquoise blue, on a sunny day.",
-        "image_path": "images/peninsula.jpg"
+        "image_path": "assets/images/peninsula.jpg"
     },
     {
         "prompt": "Skillful cyclist doing a wheelie on a bike while riding through a forest, on a dirt road, surrounded by many trees, in the morning.",
-        "image_path": "images/cyclist.jpg"
+        "image_path": "assets/images/cyclist.jpg"
     },
     {
         "prompt": "Some friends dancing and having fun together in circles, at a party surrounded by colored lights at a party, in a fancy old place, in a view from below them.",
-        "image_path": "images/friends.jpg"
+        "image_path": "assets/images/friends.jpg"
     },
     {
         "prompt": "A saxophonist wearing a blazer dances while playing a song in a park.",
-        "image_path": "images/saxophonist.jpg"
+        "image_path": "assets/images/saxophonist.jpg"
     },
     {
         "prompt": "Romantic couple embracing and looking at each other in the middle of a forest, during a break on a road trip through nature.",
-        "image_path": "images/romance.jpg"
+        "image_path": "assets/images/romance.jpg"
     },
     {
         "prompt": "Man dressed in 80's style dances very happily in his kitchen while listening to music on his radio and drinking wine.",
-        "image_path": "images/80s_dance.jpg"
+        "image_path": "assets/images/80s_dance.jpg"
     },
     {
         "prompt": "Pair of jazz musicians performing a song with their saxophone and trombone on an abandoned train.",
-        "image_path": "images/jazz.jpg"
+        "image_path": "assets/images/jazz.jpg"
     },
     {
         "prompt": "A young woman with short hair wearing pink sunglasses chews gum and makes a bubble gum with the city in the background.",
-        "image_path": "images/pink.jpg"
+        "image_path": "assets/images/pink.jpg"
     },
     {
         "prompt": "Natural aerial landscape with a relief covered with abundant trees and vegetation and a thick layer of mist.",
-        "image_path": "images/natural.jpg"
+        "image_path": "assets/images/natural.jpg"
     },
     {
         "prompt": "Loving couple sitting on a log on the shore of a lake outside, sharing an affectionate hug.",
-        "image_path": "images/couple.jpg"
+        "image_path": "assets/images/couple.jpg"
     }
 ]

assets/videos/README.md

@@ -0,0 +1,7 @@
+# FastVideo/assets/videos
+
+This folder is used to store **video assets for examples**, primarily **input videos** consumed by scripts under `FastVideo/examples/`.
+
+- **Typical contents**: short input clips for demos (e.g., video2world / image2video examples).
+- **Non-critical**: these assets are for convenience and are not required to use the FastVideo library.
+- **Large files**: avoid committing large videos to git; prefer shared storage or download-on-demand.

demo/gradio_web_demo.py

@@ -1,195 +0,0 @@
-import argparse
-import os
-import tempfile
-
-import gradio as gr
-import torch
-from diffusers import FlowMatchEulerDiscreteScheduler
-from diffusers.utils import export_to_video
-
-from fastvideo.distill.solver import PCMFMScheduler
-from fastvideo.models.mochi_hf.modeling_mochi import MochiTransformer3DModel
-from fastvideo.models.mochi_hf.pipeline_mochi import MochiPipeline
-
-
-def init_args():
-    parser = argparse.ArgumentParser()
-    parser.add_argument("--prompts", nargs="+", default=[])
-    parser.add_argument("--num_frames", type=int, default=25)
-    parser.add_argument("--height", type=int, default=480)
-    parser.add_argument("--width", type=int, default=848)
-    parser.add_argument("--num_inference_steps", type=int, default=8)
-    parser.add_argument("--guidance_scale", type=float, default=4.5)
-    parser.add_argument("--model_path", type=str, default="data/mochi")
-    parser.add_argument("--seed", type=int, default=12345)
-    parser.add_argument("--transformer_path", type=str, default=None)
-    parser.add_argument("--scheduler_type", type=str, default="pcm_linear_quadratic")
-    parser.add_argument("--lora_checkpoint_dir", type=str, default=None)
-    parser.add_argument("--shift", type=float, default=8.0)
-    parser.add_argument("--num_euler_timesteps", type=int, default=50)
-    parser.add_argument("--linear_threshold", type=float, default=0.1)
-    parser.add_argument("--linear_range", type=float, default=0.75)
-    parser.add_argument("--cpu_offload", action="store_true")
-    return parser.parse_args()
-
-
-def load_model(args):
-    if args.scheduler_type == "euler":
-        scheduler = FlowMatchEulerDiscreteScheduler()
-    else:
-        linear_quadratic = True if "linear_quadratic" in args.scheduler_type else False
-        scheduler = PCMFMScheduler(
-            1000,
-            args.shift,
-            args.num_euler_timesteps,
-            linear_quadratic,
-            args.linear_threshold,
-            args.linear_range,
-        )
-
-    if args.transformer_path:
-        transformer = MochiTransformer3DModel.from_pretrained(args.transformer_path)
-    else:
-        transformer = MochiTransformer3DModel.from_pretrained(args.model_path, subfolder="transformer/")
-
-    pipe = MochiPipeline.from_pretrained(args.model_path, transformer=transformer, scheduler=scheduler)
-    pipe.enable_vae_tiling()
-    # pipe.to(device)
-    # if args.cpu_offload:
-    pipe.enable_sequential_cpu_offload()
-    return pipe
-
-
-def generate_video(
-    prompt,
-    negative_prompt,
-    use_negative_prompt,
-    seed,
-    guidance_scale,
-    num_frames,
-    height,
-    width,
-    num_inference_steps,
-    randomize_seed=False,
-):
-    if randomize_seed:
-        seed = torch.randint(0, 1000000, (1, )).item()
-
-    generator = torch.Generator(device="cuda").manual_seed(seed)
-
-    if not use_negative_prompt:
-        negative_prompt = None
-
-    with torch.autocast("cuda", dtype=torch.bfloat16):
-        output = pipe(
-            prompt=[prompt],
-            negative_prompt=negative_prompt,
-            height=height,
-            width=width,
-            num_frames=num_frames,
-            num_inference_steps=num_inference_steps,
-            guidance_scale=guidance_scale,
-            generator=generator,
-        ).frames[0]
-
-    output_path = os.path.join(tempfile.mkdtemp(), "output.mp4")
-    export_to_video(output, output_path, fps=30)
-    return output_path, seed
-
-
-examples = [
-    "A hand enters the frame, pulling a sheet of plastic wrap over three balls of dough placed on a wooden surface. The plastic wrap is stretched to cover the dough more securely. The hand adjusts the wrap, ensuring that it is tight and smooth over the dough. The scene focuses on the hand’s movements as it secures the edges of the plastic wrap. No new objects appear, and the camera remains stationary, focusing on the action of covering the dough.",
-    "A vintage train snakes through the mountains, its plume of white steam rising dramatically against the jagged peaks. The cars glint in the late afternoon sun, their deep crimson and gold accents lending a touch of elegance. The tracks carve a precarious path along the cliffside, revealing glimpses of a roaring river far below. Inside, passengers peer out the large windows, their faces lit with awe as the landscape unfolds.",
-    "A crowded rooftop bar buzzes with energy, the city skyline twinkling like a field of stars in the background. Strings of fairy lights hang above, casting a warm, golden glow over the scene. Groups of people gather around high tables, their laughter blending with the soft rhythm of live jazz. The aroma of freshly mixed cocktails and charred appetizers wafts through the air, mingling with the cool night breeze.",
-]
-
-args = init_args()
-pipe = load_model(args)
-print("load model successfully")
-with gr.Blocks() as demo:
-    gr.Markdown("# Fastvideo Mochi Video Generation Demo")
-
-    with gr.Group():
-        with gr.Row():
-            prompt = gr.Text(
-                label="Prompt",
-                show_label=False,
-                max_lines=1,
-                placeholder="Enter your prompt",
-                container=False,
-            )
-            run_button = gr.Button("Run", scale=0)
-        result = gr.Video(label="Result", show_label=False)
-
-    with gr.Accordion("Advanced options", open=False):
-        with gr.Group():
-            with gr.Row():
-                height = gr.Slider(
-                    label="Height",
-                    minimum=256,
-                    maximum=1024,
-                    step=32,
-                    value=args.height,
-                )
-                width = gr.Slider(label="Width", minimum=256, maximum=1024, step=32, value=args.width)
-
-            with gr.Row():
-                num_frames = gr.Slider(
-                    label="Number of Frames",
-                    minimum=21,
-                    maximum=163,
-                    value=args.num_frames,
-                )
-                guidance_scale = gr.Slider(
-                    label="Guidance Scale",
-                    minimum=1,
-                    maximum=12,
-                    value=args.guidance_scale,
-                )
-                num_inference_steps = gr.Slider(
-                    label="Inference Steps",
-                    minimum=4,
-                    maximum=100,
-                    value=args.num_inference_steps,
-                )
-
-            with gr.Row():
-                use_negative_prompt = gr.Checkbox(label="Use negative prompt", value=False)
-            negative_prompt = gr.Text(
-                label="Negative prompt",
-                max_lines=1,
-                placeholder="Enter a negative prompt",
-                visible=False,
-            )
-
-            seed = gr.Slider(label="Seed", minimum=0, maximum=1000000, step=1, value=args.seed)
-            randomize_seed = gr.Checkbox(label="Randomize seed", value=True)
-            seed_output = gr.Number(label="Used Seed")
-
-    gr.Examples(examples=examples, inputs=prompt)
-
-    use_negative_prompt.change(
-        fn=lambda x: gr.update(visible=x),
-        inputs=use_negative_prompt,
-        outputs=negative_prompt,
-    )
-
-    run_button.click(
-        fn=generate_video,
-        inputs=[
-            prompt,
-            negative_prompt,
-            use_negative_prompt,
-            seed,
-            guidance_scale,
-            num_frames,
-            height,
-            width,
-            num_inference_steps,
-            randomize_seed,
-        ],
-        outputs=[result, seed_output],
-    )
-
-if __name__ == "__main__":
-    demo.queue(max_size=20).launch(server_name="0.0.0.0", server_port=7860)

demo/qualitative_results.md

@@ -1,15 +0,0 @@
-Fast-Hunyuan comparison with original Hunyuan, achieving an 8X diffusion speed boost with the FastVideo framework.
-
-https://github.com/user-attachments/assets/064ac1d2-11ed-4a0c-955b-4d412a96ef30
-
-Fast-Mochi comparison with original Mochi, achieving an 8X diffusion speed boost with the FastVideo framework.
-
-https://github.com/user-attachments/assets/5fbc4596-56d6-43aa-98e0-da472cf8e26c
-
-Comparison between OpenAI Sora, original Hunyuan and FastHunyuan
-
-https://github.com/user-attachments/assets/d323b712-3f68-42b2-952b-94f6a49c4836
-
-Comparison between original FastHunyuan, LLM-INT8 quantized FastHunyuan and NF4 quantized FastHunyuan
-
-https://github.com/user-attachments/assets/cf89efb5-5f68-4949-a085-f41c1ef26c94

docker/Dockerfile.python3.10

@@ -43,7 +43,7 @@ RUN source $HOME/.local/bin/env && \
     source /opt/venv/bin/activate && \
     uv pip install --no-cache-dir --upgrade pip && \
     uv pip install --no-cache-dir .[dev] && \
-    uv pip install --no-cache-dir https://github.com/mjun0812/flash-attention-prebuild-wheels/releases/download/v0.5.4/flash_attn-2.8.3%2Bcu128torch2.9-cp310-cp310-linux_x86_64.whl
+    uv pip install --no-cache-dir https://github.com/mjun0812/flash-attention-prebuild-wheels/releases/download/v0.7.16/flash_attn-2.8.3+cu128torch2.10-cp310-cp310-linux_x86_64.whl
 
 COPY . .
 

docker/Dockerfile.python3.11

@@ -43,7 +43,7 @@ RUN source $HOME/.local/bin/env && \
     source /opt/venv/bin/activate && \
     uv pip install --no-cache-dir --upgrade pip && \
     uv pip install --no-cache-dir .[dev] && \
-    uv pip install --no-cache-dir https://github.com/mjun0812/flash-attention-prebuild-wheels/releases/download/v0.5.4/flash_attn-2.8.3%2Bcu128torch2.9-cp311-cp311-linux_x86_64.whl
+    uv pip install --no-cache-dir https://github.com/mjun0812/flash-attention-prebuild-wheels/releases/download/v0.7.16/flash_attn-2.8.3+cu128torch2.10-cp311-cp311-linux_x86_64.whl
 
 COPY . .
 

docker/Dockerfile.python3.12

@@ -43,7 +43,7 @@ RUN source $HOME/.local/bin/env && \
     source /opt/venv/bin/activate && \
     uv pip install --no-cache-dir --upgrade pip && \
     uv pip install --no-cache-dir .[dev] && \
-    uv pip install --no-cache-dir https://github.com/mjun0812/flash-attention-prebuild-wheels/releases/download/v0.5.4/flash_attn-2.8.3%2Bcu128torch2.9-cp312-cp312-linux_x86_64.whl
+    uv pip install --no-cache-dir https://github.com/mjun0812/flash-attention-prebuild-wheels/releases/download/v0.7.16/flash_attn-2.8.3+cu128torch2.10-cp312-cp312-linux_x86_64.whl
 
 COPY . .
 

docs/api/fastvideo.md

@@ -121,6 +121,15 @@ This page contains the complete API reference for the FastVideo library.
       show_root_toc_entry: true
       heading_level: 4
 
+## fastvideo.registry
+
+::: fastvideo.registry
+    options:
+      show_source: true
+      show_root_heading: true
+      show_root_toc_entry: true
+      heading_level: 3
+
 ## fastvideo.pipelines
 
 ::: fastvideo.pipelines

docs/contributing/coding_agents.md

@@ -324,8 +324,8 @@ Purpose:
 Action:
 
 - Add a new pipeline config + sampling params.
-- Register them in `fastvideo/configs/pipelines/registry.py` and
-  `fastvideo/configs/sample/registry.py`.
+- Register them in `fastvideo/registry.py` using explicit
+  `register_configs(...)` blocks (this file is the single source of truth now).
 
 ### 5) Wire pipeline stages
 

docs/contributing/developer_env/runpod.md

@@ -17,7 +17,7 @@ You can easily use the FastVideo Pod Template on [RunPod](https://www.runpod.io)
 - Select the "FastVideo" or "fastvideo-dev" Pod Template.
 ![RunPod Pod Template Selection](../../assets/images/runpod_create.png)
 
-- Set the Pod name to "<name>-<FastVideo>-<date>".
+- Set the Pod name to "`<name>-<FastVideo>-<date>`".
 
 - Finally, once the pod is deployed (will take a few minutes as the image is being pulled), you can SSH into it using "SSH exposed over TCP". You'll need to use the matching private ssh key you provided.
 ![RunPod SSH](../../assets/images/runpod_ssh.png)

docs/design/overview.md

@@ -51,8 +51,9 @@ runtime parameters consistent:
 - `fastvideo/configs/pipelines/`: pipeline wiring and required components.
 - `fastvideo/configs/sample/`: default sampling parameters (steps, frames,
   guidance scale, resolution, fps).
-- `fastvideo/configs/registry.py`: pipeline registry.
-- `fastvideo/configs/sample/registry.py`: sampling param registry.
+- `fastvideo/registry.py`: unified registry for pipeline config + sampling
+  defaults and model metadata resolution, defined via explicit
+  `register_configs(...)` blocks (no separate dict registries).
 
 `FastVideoArgs` (in `fastvideo/fastvideo_args.py`) provides runtime settings and
 is passed into pipeline construction and stages.

docs/examples/examples_index.md

@@ -6,7 +6,7 @@ All documented examples are autogenerated using [generate_examples.py](https://g
 
 ## Examples
 
-- [Examples Distillation Index](distillation/examples/examples_distillation_index.md)
-- [Examples Training Index](training/examples/examples_training_index.md)
-- [Examples Inference Index](inference/examples/examples_inference_index.md)
+- [Examples Distillation Index](../distillation/examples/examples_distillation_index.md)
+- [Examples Training Index](../training/examples/examples_training_index.md)
+- [Examples Inference Index](../inference/examples/examples_inference_index.md)
 

docs/generate_examples.py

@@ -2,6 +2,7 @@
 # adapted from vllm: https://github.com/vllm-project/vllm/blob/v0.7.3/docs/source/generate_examples.py
 
 import itertools
+import os
 import re
 from dataclasses import dataclass, field
 from pathlib import Path
@@ -126,8 +127,20 @@ class Example:
         Raises:
             IndexError: If no Markdown files are found in the directory.
         """ # noqa: E501
-        return self.path if self.path.is_file() else list(
-            self.path.glob("*.md")).pop()
+        if self.path.is_file():
+            return self.path
+
+        markdown_files = sorted(self.path.glob("*.md"))
+        if not markdown_files:
+            raise IndexError(f"No Markdown files found in {self.path}")
+
+        readme_files = [
+            f for f in markdown_files if f.name.lower() == "readme.md"
+        ]
+        if readme_files:
+            return readme_files[0]
+
+        return markdown_files[0]
 
     def determine_other_files(self) -> list[Path]:
         """
@@ -521,11 +534,11 @@ def generate_examples(generate_main_index: bool = False) -> None:
             # Add to main index if it exists
             if generate_main_index and examples_index:
                 main_index_dir = examples_index.path.parent
-                rel_path = category_index.path.relative_to(
-                    main_index_dir.parent)
+                rel_path = os.path.relpath(category_index.path,
+                                           start=main_index_dir)
                 examples_index.documents.insert(
                     0,
-                    str(rel_path).replace(".md", ""))
+                    str(rel_path).replace("\\", "/").replace(".md", ""))
 
             # Write the category index file
             with open(category_index.path, "w+") as f:

docs/getting_started/installation.md

@@ -18,12 +18,25 @@ conda activate fastvideo
 pip install fastvideo
 ```
 
+### Using uv
+
+```bash
+# Create and activate a new uv environment
+uv venv --python 3.12 --seed
+source .venv/bin/activate
+
+uv pip install fastvideo
+```
+
 ### From source
 
 ```bash
 git clone https://github.com/hao-ai-lab/FastVideo.git
 cd FastVideo
 pip install -e .
+
+# or if you are using uv
+uv pip install -e .
 ```
 
 Also optionally install flash-attn:

docs/getting_started/quick_start.md

@@ -79,5 +79,6 @@ if __name__ == '__main__':
 
 - [Installation Guide](installation.md) - Detailed installation instructions
 - [Configuration](../inference/configuration.md) - Learn about configuration options
-- [Examples](../inference/examples/) - Explore more examples
+- [Examples](../inference/examples/examples_inference_index.md) - Explore more
+  examples
 - [Optimizations](../inference/optimizations.md) - Performance optimization tips

docs/inference/add_pipeline.md

@@ -101,14 +101,17 @@ Replace standard attention with FastVideo's optimized attention:
 ```python
 # Local attention patterns
 from fastvideo.attention import LocalAttention
-from fastvideo.attention.backends.abstract import _Backend
+from fastvideo.platforms.interface import AttentionBackendEnum
 self.attn = LocalAttention(
     num_heads=num_heads,
     head_size=head_dim,
     dropout_rate=0.0,
     softmax_scale=None,
     causal=False,
-    supported_attention_backends=(_Backend.FLASH_ATTN, _Backend.TORCH_SDPA)
+    supported_attention_backends=(
+        AttentionBackendEnum.FLASH_ATTN,
+        AttentionBackendEnum.TORCH_SDPA,
+    )
 )
 
 # Distributed attention for long sequences
@@ -119,29 +122,60 @@ self.attn = DistributedAttention(
     dropout_rate=0.0,
     softmax_scale=None,
     causal=False,
-    supported_attention_backends=(_Backend.SLIDING_TILE_ATTN, _Backend.FLASH_ATTN, _Backend.TORCH_SDPA)
+    supported_attention_backends=(
+        AttentionBackendEnum.SLIDING_TILE_ATTN,
+        AttentionBackendEnum.FLASH_ATTN,
+        AttentionBackendEnum.TORCH_SDPA,
+    )
 )
 ```
 
 #### Define supported backend selection
 
 ```python
-   _supported_attention_backends = (_Backend.FLASH_ATTN, _Backend.TORCH_SDPA)
+_supported_attention_backends = (
+    AttentionBackendEnum.FLASH_ATTN,
+    AttentionBackendEnum.TORCH_SDPA,
+)
 ```
 
 ### Registering Models
 
-Register implemented modules in the model registry:
+Register implemented modules for auto‑discovery by adding `EntryClass` in each
+model module (the registry scans for it):
 
 ```python
-# In fastvideo/models/registry.py
-_TEXT_TO_VIDEO_DIT_MODELS = {
-    "YourTransformerModel": ("dits", "yourmodule", "YourTransformerClass"),
-}
+# In fastvideo/models/dits/your_module.py
+class YourTransformerModel(...):
+    ...
 
-_VAE_MODELS = {
-    "YourVAEModel": ("vaes", "yourvae", "YourVAEClass"),
-}
+# Entry point for model registry
+EntryClass = YourTransformerModel
+```
+
+```python
+# In fastvideo/models/vaes/your_vae.py
+class YourVAEModel(...):
+    ...
+
+# Entry point for model registry
+EntryClass = YourVAEModel
+```
+
+Register pipeline config + sampling defaults in the unified registry:
+
+```python
+# In fastvideo/registry.py
+register_configs(
+    sampling_param_cls=YourSamplingParam,
+    pipeline_config_cls=YourPipelineConfig,
+    hf_model_paths=[
+        "org/your-model-id",
+    ],
+    model_detectors=[
+        lambda path: "your-model" in path.lower(),
+    ],
+)
 ```
 
 ## Step 2: Directory Structure

docs/inference/cli.md

@@ -1,104 +1,85 @@
 # FastVideo CLI Inference
 
-The FastVideo CLI provides a quick way to access the FastVideo inference pipeline for video generation. For more advanced usage,
-see the Python interface [here](examples/basic.md).
+The FastVideo CLI exposes the same core inference controls as the Python API.
 
 ## Basic Usage
 
-The basic command to generate a video is:
+Use either:
+
+1. `--model-path` + `--prompt`
+2. `--model-path` + `--prompt-txt` (batch prompts, one line per prompt)
+3. `--config` (JSON/YAML)
 
 ```bash
-fastvideo generate --model-path {MODEL_PATH} --prompt {PROMPT}
+fastvideo generate --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \
+  --prompt "A cat playing with a ball of yarn"
 ```
 
-### Required Parameters
+```bash
+fastvideo generate --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \
+  --prompt-txt prompts.txt
+```
 
-- `--model-path {MODEL_PATH}`: Path to the model or model ID
-- `--prompt {PROMPT}`: Text description for the video you want to generate
+You cannot provide both `--prompt` and `--prompt-txt` in the same run.
 
-## Common Arguments
-
-To see all the options, you can use the `--help` flag:
+## View All Arguments
 
 ```bash
 fastvideo generate --help
 ```
 
-### Hardware Configuration
+Arguments come from:
 
-- `--num-gpus {NUM_GPUS}`: Number of GPUs to use
-- `--tp-size {TP_SIZE}`: Tensor parallelism size (only for the encoder, should not be larger than 1 if text encoder offload is enabled, as layerwise offload + prefetch is faster)
-- `--sp-size {SP_SIZE}`: Sequence parallelism size (Typically should match the number of GPUs)
+- FastVideo runtime args (`FastVideoArgs`)
+- Sampling args (`SamplingParam`)
+- Pipeline config args (`PipelineConfig`)
 
-#### Video Configuration
+## Common Arguments
 
-- `--height {HEIGHT}`: Height of the generated video
-- `--width {WIDTH}`: Width of the generated video
-- `--num-frames {NUM_FRAMES}`: Number of frames to generate
-- `--fps {FPS}`: Frames per second for the saved video
+### Parallelism
 
-#### Generation Parameters
+- `--num-gpus`
+- `--sp-size`
+- `--tp-size`
 
-- `--num-inference-steps {STEPS}`: Number of denoising steps
-- `--negative-prompt {PROMPT}`: Negative prompt to guide generation away from certain concepts
-- `--seed {SEED}`: Random seed for reproducible generation
+### Sampling
 
-#### Output Options
+- `--num-frames`
+- `--height` / `--width`
+- `--num-inference-steps`
+- `--guidance-scale`
+- `--seed`
+- `--negative-prompt`
 
-- `--output-path {PATH}`: Directory to save the generated video
-- `--save-video`: Whether to save the video to disk
-- `--return-frames`: Whether to return the raw frames
+### Output
 
-## Using Configuration Files
+- `--output-path`
+- `--save-video` / `--no-save-video`
+- `--return-frames`
 
-Instead of specifying all parameters on the command line, you can use a configuration file:
+### Offloading and Performance
+
+- `--dit-layerwise-offload`
+- `--use-fsdp-inference`
+- `--text-encoder-cpu-offload`
+- `--image-encoder-cpu-offload`
+- `--vae-cpu-offload`
+- `--enable-torch-compile`
+- `--torch-compile-kwargs`
+
+## Using Config Files
 
 ```bash
-fastvideo generate --config {CONFIG_FILE_PATH}
+fastvideo generate --config config.yaml
 ```
 
-The config file should be in JSON or YAML format with the same parameter names as the CLI options. Command-line arguments will take precedence over settings in the configuration file, allowing you to override specific values while keeping the rest from the config file.
+Config files can be JSON or YAML. CLI flags override config-file values.
 
-Example configuration file (config.json):
-
-```json
-{
-    "model_path": "FastVideo/FastHunyuan-diffusers",
-    "prompt": "A beautiful woman in a red dress walking down a street",
-    "output_path": "outputs/",
-    "num_gpus": 2,
-    "sp_size": 2,
-    "tp_size": 1,
-    "num_frames": 45,
-    "height": 720,
-    "width": 1280,
-    "num_inference_steps": 6,
-    "seed": 1024,
-    "fps": 24,
-    "precision": "bf16",
-    "vae_precision": "fp16",
-    "vae_tiling": true,
-    "vae_sp": true,
-    "vae_config": {
-        "load_encoder": false,
-        "load_decoder": true,
-        "tile_sample_min_height": 256,
-        "tile_sample_min_width": 256
-    },
-    "text_encoder_precisions": [
-        "fp16",
-        "fp16"
-    ],
-    "mask_strategy_file_path": null,
-    "enable_torch_compile": false
-}
-```
-
-Or using YAML format (config.yaml):
+Example `config.yaml`:
 
 ```yaml
 model_path: "FastVideo/FastHunyuan-diffusers"
-prompt: "A beautiful woman in a red dress walking down a street"
+prompt: "A capybara lounging in a hammock"
 output_path: "outputs/"
 num_gpus: 2
 sp_size: 2
@@ -108,44 +89,34 @@ height: 720
 width: 1280
 num_inference_steps: 6
 seed: 1024
-fps: 24
-precision: "bf16"
+dit_precision: "bf16"
 vae_precision: "fp16"
 vae_tiling: true
 vae_sp: true
-vae_config:
-  load_encoder: false
-  load_decoder: true
-  tile_sample_min_height: 256
-  tile_sample_min_width: 256
-text_encoder_precisions:
-  - "fp16"
-  - "fp16"
-mask_strategy_file_path: null
 enable_torch_compile: false
 ```
 
+Notes:
+
+- Use `dit_precision` / `vae_precision` (not `precision`).
+- Nested config objects are supported, for example `vae_config` and
+  `dit_config`.
+
 ## Examples
 
-Generating a simple video:
+Simple generation:
 
 ```bash
-fastvideo generate --model-path FastVideo/FastHunyuan-diffusers --prompt "A cat playing with a ball of yarn" --num-frames 45 --height 720 --width 1280 --num-inference-steps 6 --seed 1024 --output-path outputs/
+fastvideo generate \
+  --model-path FastVideo/FastHunyuan-diffusers \
+  --prompt "A cat playing with a ball of yarn" \
+  --num-frames 45 --height 720 --width 1280 \
+  --num-inference-steps 6 --seed 1024 \
+  --output-path outputs/
 ```
 
-Using a negative prompt to avoid certain elements:
+Config + CLI override:
 
 ```bash
-fastvideo generate --model-path FastVideo/FastHunyuan-diffusers --prompt "A beautiful forest landscape" --negative-prompt "people, buildings, roads"
+fastvideo generate --config config.yaml --prompt "A panda skiing at sunset"
 ```
-
-Combining command line arguments and a configuration file:
-
-```bash
-fastvideo generate --config config.json --prompt "A capybara lounging in a hammock"
-```
-
-## Troubleshooting
-
-- If you encounter CUDA out-of-memory errors, try reducing the video dimensions or number of frames, or the number of inference steps.
-- For reproducible results, set the same seed value between runs.

docs/inference/configuration.md

@@ -1,4 +1,3 @@
-
 # Configuration
 
 ## Multi-GPU Setup
@@ -18,7 +17,8 @@ generator = VideoGenerator.from_pretrained(
 - `PipelineConfig`: Initialization time parameters
 - `SamplingParam`: Generation time parameters
 
-You can customize various parameters when generating videos using the `PipelineConfig` and `SamplingParam` class:
+You can customize generation behavior using `PipelineConfig` and
+`SamplingParam`:
 
 ```python
 from fastvideo import VideoGenerator, SamplingParam, PipelineConfig
@@ -27,12 +27,12 @@ def main():
     model_name = "Wan-AI/Wan2.1-T2V-1.3B-Diffusers"
     config = PipelineConfig.from_pretrained(model_name)
     config.vae_precision = "fp16"
-    config.dit_cpu_offload = True
 
     # Create the generator
     generator = VideoGenerator.from_pretrained(
         model_name,
         num_gpus=1,
+        dit_layerwise_offload=True,  # FastVideoArgs option
         pipeline_config=config
     )
 
@@ -72,6 +72,34 @@ if __name__ == '__main__':
     main()
 ```
 
+## JSON/YAML Config Files (CLI)
+
+The CLI supports `--config` with JSON or YAML. Command-line arguments override
+config file values.
+
+```bash
+fastvideo generate --config config.yaml
+```
+
+Use CLI argument names as keys (underscore or hyphen is accepted). Example:
+
+```yaml
+model_path: "FastVideo/FastHunyuan-diffusers"
+prompt: "A capybara relaxing in a hammock"
+num_gpus: 2
+sp_size: 2
+num_frames: 45
+height: 720
+width: 1280
+num_inference_steps: 6
+seed: 1024
+dit_precision: "bf16"
+vae_precision: "fp16"
+vae_tiling: true
+vae_sp: true
+enable_torch_compile: false
+```
+
 ## Performance Optimization
 
 For configuring optimizations, please see our [optimizations guide](optimizations.md)

docs/inference/inference_quick_start.md

@@ -61,7 +61,8 @@ python example.py
 
 The generated video will be saved in the current directory under `my_videos/`  
 
-More inference example scripts can be found in `scripts/inference/`
+More inference scripts and recipes can be found in `examples/inference/` and
+`scripts/inference/`.
 
 ## Available Models
 
@@ -103,11 +104,10 @@ Common issues and their solutions:
 If you encounter CUDA out of memory errors:
 
 - Reduce `num_frames` or video resolution
-- Enable memory optimization with `enable_model_cpu_offload`
+- Enable FastVideo offloading options such as `dit_layerwise_offload=True`
+  (single GPU) or `use_fsdp_inference=True` (multi-GPU)
 - Try a smaller model or use distilled versions
 - Use `num_gpus` > 1 if multiple GPUs are available
-- Try enabling FSDP inference with `use_fsdp_inference=True` (may slow down generation)
-- Try enabling DiT layerwise offload with `dit_layerwise_offload=True` (now only a few models support this, but may introduce less overhead than FSDP)
 
 ### Slow Generation
 

docs/inference/optimizations.md

@@ -25,6 +25,9 @@ This page describes the various options for speeding up generation times in Fast
 - Video Sparse Attention: `FASTVIDEO_ATTENTION_BACKEND=VIDEO_SPARSE_ATTN`
 - Sage Attention: `FASTVIDEO_ATTENTION_BACKEND=SAGE_ATTN`
 - Sage Attention 3: `FASTVIDEO_ATTENTION_BACKEND=SAGE_ATTN_THREE`
+- Video MoBA Attention: `FASTVIDEO_ATTENTION_BACKEND=VMOBA_ATTN`
+- Sparse Linear Attention: `FASTVIDEO_ATTENTION_BACKEND=SLA_ATTN`
+- SageSLA Attention: `FASTVIDEO_ATTENTION_BACKEND=SAGE_SLA_ATTN`
 
 ### Configuring Backends
 
@@ -70,22 +73,15 @@ python setup.py install
 
 **`SLIDING_TILE_ATTN`**
 
-```bash
-pip install st_attn==0.0.4
-```
-
-Please see [this page](../attention/sta/index.md) for more installation instructions.
+Sliding Tile Attention is provided by `fastvideo-kernel`.
+See [STA docs](../attention/sta/index.md) for installation details.
 
 ### Video Sparse Attention
 
 **`VIDEO_SPARSE_ATTN`**
 
-```bash
-git submodule update --init --recursive
-python setup_vsa.py install
-```
-
-Please see [this page](../attention/vsa/index.md) for more installation instructions.
+Video Sparse Attention is provided by `fastvideo-kernel`.
+See [VSA docs](../attention/vsa/index.md) for installation details.
 
 ### Sage Attention
 
@@ -103,7 +99,7 @@ python setup.py install  # or pip install -e .
 
 **`SAGE_ATTN_THREE`**
 
-[SageAttention 3](https://huggingface.co/jt-zhang/SageAttention3) is an advanced attention mechanism that leverages FP4 quantization and Blackwell GPU Tensor Cores for significant performance improvements.
+[SageAttention 3](https://github.com/thu-ml/SageAttention/tree/main/sageattention3_blackwell) is an advanced attention mechanism that leverages FP4 quantization and Blackwell GPU Tensor Cores for significant performance improvements.
 
 #### Hardware Requirements
 
@@ -113,11 +109,13 @@ python setup.py install  # or pip install -e .
 
 Note that Sage Attention 3 requires `python>=3.13`, `torch>=2.8.0`, `CUDA >=12.8`. If you are using `uv` and using `torch==2.8.0` make sure that `sentencepiece==0.2.1` in the pyproject.toml file.
 
-To use Sage Attention 3 in FastVideo, first get access to the SageAttention3 code, then move `sageattn/` and `setup.py` to the directory `fastvideo/attention/backends`, then install from using:
+To use Sage Attention 3 in FastVideo, follow the `README.md` in the linked repository to install the package from source.
 
-```bash
-python setup.py install
-```
+### V-MoBA / SLA / SageSLA
+
+These backends are model-specific and require the corresponding kernels and
+dependencies. Use the support matrix and model examples to confirm compatibility
+before enabling them.
 
 ## Teacache
 

docs/inference/support_matrix.md

@@ -1,6 +1,10 @@
 # Compatibility Matrix
 
-The table below shows every supported model and optimizations supported for them.
+This page summarizes common model + optimization combinations.
+
+For the canonical, code-level list of model IDs recognized by
+`VideoGenerator.from_pretrained(...)`, see the registrations in
+`fastvideo/registry.py` (`register_configs(...)` entries).
 
 The symbols used have the following meanings:
 
@@ -10,7 +14,9 @@ The symbols used have the following meanings:
 
 ## Models x Optimization
 
-The `HuggingFace Model ID` can be directly pass to `from_pretrained()` methods and FastVideo will use the optimal default parameters when initializing and generating videos.
+The `HuggingFace Model ID` can be passed directly to
+`from_pretrained()`. FastVideo then uses model-specific default settings for
+pipeline initialization and sampling.
 
 <style>
   /* Target tables in this section */
@@ -53,9 +59,9 @@ The `HuggingFace Model ID` can be directly pass to `from_pretrained()` methods a
 | Wan2.1 T2V 14B | `Wan-AI/Wan2.1-T2V-14B-Diffusers` | 480P, 720P | ✅ | ✅* | ✅ | ⭕ | ⭕ |
 | Wan2.1 I2V 480P | `Wan-AI/Wan2.1-I2V-14B-480P-Diffusers` | 480P | ✅ | ✅* | ✅ | ⭕ | ⭕ |
 | Wan2.1 I2V 720P | `Wan-AI/Wan2.1-I2V-14B-720P-Diffusers` | 720P | ✅ | ✅ | ✅ | ⭕ | ⭕ |
-| StepVideo T2V | `FastVideo/stepvideo-t2v-diffusers` | 768px768px204f<br>544px992px204f<br>544px992px136f | ❌ | ❌ | ✅ | ⭕ | ⭕ |
 | TurboWan2.1 T2V 1.3B | `loayrashid/TurboWan2.1-T2V-1.3B-Diffusers` | 480P | ⭕ | ⭕ | ⭕ | ⭕ | ⭕ |
 | TurboWan2.1 T2V 14B | `loayrashid/TurboWan2.1-T2V-14B-Diffusers` | 480P, 720P | ⭕ | ⭕ | ⭕ | ⭕ | ⭕ |
+| TurboWan2.2 I2V A14B | `loayrashid/TurboWan2.2-I2V-A14B-Diffusers` | 480P<br>720P | ⭕ | ⭕ | ⭕ | ⭕ | ⭕ |
 | LongCat T2V 13.6B | See note** | 480P<br>720P | ❌ | ❌ | ❌ | ⭕ | ✅ |
 | Matrix Game 2.0 Base | `FastVideo/Matrix-Game-2.0-Base-Diffusers` | 352x640 | ⭕ | ⭕ | ⭕ | ⭕ | ⭕ |
 | Matrix Game 2.0 GTA | `FastVideo/Matrix-Game-2.0-GTA-Diffusers` | 352x640 | ⭕ | ⭕ | ⭕ | ⭕ | ⭕ |
@@ -63,10 +69,13 @@ The `HuggingFace Model ID` can be directly pass to `from_pretrained()` methods a
 
 **Note**: Wan2.2 TI2V 5B has some quality issues when performing I2V generation. We are working on fixing this issue.
 
-## Special requirements
+## Canonical Supported IDs
 
-### StepVideo T2V
-- The self-attention in text-encoder (step_llm) only supports CUDA capabilities sm_80 sm_86 and sm_90
+The authoritative source for model-ID recognition is
+`fastvideo/registry.py`. If a model ID is registered there, FastVideo can
+resolve default pipeline and sampling configuration for it.
+
+## Special requirements
 
 ### Sliding Tile Attention
 - Currently only Hopper GPUs (H100s) are supported.

docs/utilities/debugging.md

@@ -0,0 +1,75 @@
+# Debugging
+
+This page collects practical debugging steps for FastVideo inference issues.
+
+## Collect Environment Info
+
+From the repository root, run:
+
+```bash
+python collect_env.py
+```
+
+Attach the output when filing a GitHub issue.
+
+## Increase Logging
+
+FastVideo logging level is controlled by environment variables:
+
+```bash
+FASTVIDEO_LOGGING_LEVEL=DEBUG \
+FASTVIDEO_STAGE_LOGGING=1 \
+python your_script.py
+```
+
+Useful variables:
+
+- `FASTVIDEO_LOGGING_LEVEL`: `DEBUG`, `INFO`, `WARNING`, `ERROR`
+- `FASTVIDEO_STAGE_LOGGING`: print per-stage timings during pipeline execution
+- `FASTVIDEO_ATTENTION_BACKEND`: force an attention backend (for example
+  `TORCH_SDPA` or `FLASH_ATTN`)
+
+## Common Failure Modes
+
+### Out-of-memory
+
+Try, in order:
+
+1. Reduce `height`, `width`, `num_frames`, or `num_inference_steps`.
+2. Enable offloading flags such as `dit_layerwise_offload` (single GPU) or
+   `use_fsdp_inference` (multi-GPU).
+3. Enable `vae_cpu_offload`, `image_encoder_cpu_offload`, and
+   `text_encoder_cpu_offload`.
+
+See [Inference Offloading](../inference/offloading.md) for recommended
+combinations.
+
+### Attention backend import errors
+
+If forcing a backend fails, verify optional dependencies are installed:
+
+- `FLASH_ATTN`: `flash-attn`
+- `SLIDING_TILE_ATTN` and `VIDEO_SPARSE_ATTN`: `fastvideo-kernel`
+- `SAGE_ATTN` / `SAGE_ATTN_THREE`: SageAttention packages
+
+As a fallback, use:
+
+```bash
+export FASTVIDEO_ATTENTION_BACKEND=TORCH_SDPA
+```
+
+### Configuration parsing errors
+
+When using `--config`, keep keys aligned with CLI argument names (underscores or
+hyphens are both accepted). For nested config values, use nested objects
+(`vae_config`, `dit_config`) rather than dotted keys.
+
+## Issue Template
+
+When opening an issue, include:
+
+- exact command or Python snippet,
+- model ID/path,
+- full traceback,
+- `collect_env.py` output,
+- whether the problem reproduces with `FASTVIDEO_ATTENTION_BACKEND=TORCH_SDPA`.

examples/distill/SFHunyuan1.5-T2V/distill_dmd.slurm

@@ -1,172 +0,0 @@
-#!/bin/bash
-#SBATCH --job-name=dmd_3333
-#SBATCH --partition=main
-#SBATCH --nodes=4
-#SBATCH --ntasks=4
-#SBATCH --ntasks-per-node=1
-#SBATCH --gres=gpu:8
-#SBATCH --cpus-per-task=128
-#SBATCH --mem=1440G
-#SBATCH --output=dmd_3333_output/dmd_3333_%j.out
-#SBATCH --error=dmd_3333_output/dmd_3333_%j.err
-#SBATCH --exclusive
-
-# Basic Info
-export NCCL_P2P_DISABLE=1
-export TORCH_NCCL_ENABLE_MONITORING=0
-export NCCL_DEBUG_SUBSYS=INIT,NET
-# different cache dir for different processes
-export TRITON_CACHE_DIR=/tmp/triton_cache_${SLURM_PROCID}
-export MASTER_PORT=29500
-export NODE_RANK=$SLURM_PROCID
-nodes=( $(scontrol show hostnames $SLURM_JOB_NODELIST) )
-export MASTER_ADDR=${nodes[0]}
-export TOKENIZERS_PARALLELISM=false
-# export WANDB_API_KEY="8d9f4b39abd68eb4e29f6fc010b7ee71a2207cde"
-export WANDB_API_KEY="50632ebd88ffd970521cec9ab4a1a2d7e85bfc45"
-# export WANDB_API_KEY='your_wandb_api_key_here'
-export WANDB_BASE_URL="https://api.wandb.ai"
-export WANDB_MODE=online
-export FASTVIDEO_ATTENTION_BACKEND=FLASH_ATTN
-
-source ~/conda/miniconda/bin/activate
-conda activate wei-fv-distill
-export HOME="/mnt/weka/home/hao.zhang/wei"
-
-# Configs
-NUM_GPUS=8
-NUM_TOTAL_GPUS=32
-
-# Model paths for Self-Forcing DMD distillation with Hunyuan1.5:
-GENERATOR_MODEL_PATH="weizhou03/SFHunyuanVideo-1.5-Diffusers-480p_t2v"  # Updated to Wan2.2
-REAL_SCORE_MODEL_PATH="hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-480p_t2v" # Teacher model
-FAKE_SCORE_MODEL_PATH="hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-480p_t2v"
-# REAL_SCORE_MODEL_PATH="hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-480p_i2v_step_distilled"
-# FAKE_SCORE_MODEL_PATH="hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-480p_i2v_step_distilled" # Critic model
-
-# DATA_DIR="data/test-text-preprocessing/Node_0_GPU_1_File_1/combined_parquet_dataset/"
-# DATA_DIR="/mnt/weka/home/hao.zhang/matthew/FastVideo/data/test-text-preprocessing"
-DATA_DIR="/mnt/weka/home/hao.zhang/wl/sharefs/wl/release/vidprom_16k_text_embed"
-DATA_DIR_2="/mnt/weka/home/hao.zhang/wl/sharefs/wl/release/fv-ode-preprocessing-16k-hy15-121"
-# DATA_DIR=data/crush-smol_processed_t2v/combined_parquet_dataset
-# DATA_DIR="/mnt/sharefs/users/hao.zhang/Vchitect-2M/Wan-Syn-upload/latents_i2v/train/"
-# VALIDATION_DATASET_FILE="data/crush-smol-single_processed_t2v/validation.json"
-VALIDATION_DATASET_FILE="/mnt/weka/home/hao.zhang/wl/FastVideo/examples/distill/Wan2.2-TI2V-5B-Diffusers/Data-free/validation_64.json"
-
-# export CUDA_VISIBLE_DEVICES=4,5
-# IP=[MASTER NODE IP]
-
-training_args=(
-  --tracker_project_name SFhy1.5_t2v_distill_self_forcing_dmd  # Updated for Wan2.2
-  --output_dir "/mnt/weka/home/hao.zhang/wei/SFhy1.5_distill_3333_1e-5_1e-5_cfg6_corrected_scheduler"
-  --wandb_run_name "self_forcing_3333_context_forcing"
-  --max_train_steps 4000
-  --train_batch_size 1
-  --train_sp_batch_size 1
-  --gradient_accumulation_steps 1
-  --num_latent_t 21
-  --num_height 480
-  --num_width 848
-  --enable_gradient_checkpointing_type "full"
-  --simulate_generator_forward
-  --log_visualization
-  --num_frames 81
-  --num_frame_per_block 3  # Frame generation block size for self-forcing
-  # --resume-from-checkpoint "/mnt/weka/home/hao.zhang/wei/SFhy1.5_distill_1333/checkpoint-300"
-  --init_weights_from_safetensors /mnt/weka/home/hao.zhang/wei/hy15_worldplay_tf_init_3333/checkpoint-2800/transformer/diffusion_pytorch_model.safetensors
-  # --init_weights_from_safetensors /mnt/weka/home/hao.zhang/wei/hy15_worldplay_df_init_1333/checkpoint-11600/transformer/diffusion_pytorch_model.safetensors
-  # --init_weights_from_safetensors /mnt/weka/home/hao.zhang/wei/hy15_distilled_ode_init_3333/checkpoint-2400/transformer/diffusion_pytorch_model.safetensors
-)
-
-parallel_args=(
-  --num_gpus $NUM_TOTAL_GPUS # 64
-  --sp_size 1
-  --tp_size 1
-  --hsdp_replicate_dim 1 # 64
-  --hsdp_shard_dim $NUM_TOTAL_GPUS
-)
-
-model_args=(
-  --model_path $GENERATOR_MODEL_PATH
-  --pretrained_model_name_or_path $GENERATOR_MODEL_PATH
-  --real_score_model_path $REAL_SCORE_MODEL_PATH
-  --fake_score_model_path $FAKE_SCORE_MODEL_PATH
-)
-
-dataset_args=(
-  --data_path "$DATA_DIR"
-  # --data_path_2 "$DATA_DIR_2"
-  --dataloader_num_workers 4
-)
-
-validation_args=(
-  --log_validation
-  --validation_dataset_file "$VALIDATION_DATASET_FILE"
-  --validation_steps 50
-  --validation_sampling_steps "8"
-  --validation_guidance_scale "1.0" # not used for dmd inference
-  --text-encoder-cpu-offload
-  # --vae_cpu_offload True
-)
-
-optimizer_args=(
-  --optimizer-type "muon"
-  --learning_rate 1e-5
-  --mixed_precision "bf16"
-  --training_state_checkpointing_steps 100
-  --weight_only_checkpointing_steps 500
-  --weight_decay 0.01
-  --betas '0.0,0.999'
-  --max_grad_norm 1.0
-)
-
-miscellaneous_args=(
-  --inference_mode False
-  --checkpoints_total_limit 3
-  --training_cfg_rate 0.0
-  --dit_precision "fp32"
-  --flow_shift 5
-  --seed 1000
-  --use_ema True
-  --ema_decay 0.99
-  --ema_start_step 200
-)
-
-dmd_args=(
-  --warp_denoising_step
-  # --dmd_denoising_steps '1000,875,750,625,500,375,250,125'
-  # --dmd_denoising_steps '1000,760,520,280'
-  # --dmd_denoising_steps '1000,750,500,250'
-  --min_timestep_ratio 0.02
-  --max_timestep_ratio 0.98
-  --dfake_gen_update_ratio 5
-  --real_score_guidance_scale 3.5
-  --fake_score_learning_rate 5e-6
-  --fake_score_betas '0.0,0.999'
-)
-
-self_forcing_args=(
-  --independent_first_frame False  # Whether to treat first frame independently
-  --same_step_across_blocks True  # Whether to use same denoising step across all blocks
-  --last_step_only False  # Whether to only use the last denoising step
-  --context_noise 0  # Amount of noise to add during context caching (0 = no noise)
-  # --use-ode-init True
-  # --use-context-forcing True
-)
-
-srun torchrun \
---nnodes $SLURM_JOB_NUM_NODES \
---nproc_per_node $NUM_GPUS \
---node_rank $SLURM_PROCID \
---rdzv_backend=c10d \
---rdzv_endpoint="$MASTER_ADDR:$MASTER_PORT" \
-    fastvideo/training/hy15_self_forcing_distillation_pipeline.py \
-    "${parallel_args[@]}" \
-    "${model_args[@]}" \
-    "${dataset_args[@]}" \
-    "${training_args[@]}" \
-    "${optimizer_args[@]}" \
-    "${validation_args[@]}" \
-    "${miscellaneous_args[@]}" \
-    "${dmd_args[@]}" \
-    "${self_forcing_args[@]}"

examples/distill/SFWan2.1-T2V/distill_dmd.slurm

@@ -1,172 +0,0 @@
-#!/bin/bash
-#SBATCH --job-name=distill_dmd_t2v_1.3b
-#SBATCH --nodes=8
-#SBATCH --ntasks=8
-#SBATCH --ntasks-per-node=1
-#SBATCH --gres=gpu:8
-#SBATCH --cpus-per-task=128
-#SBATCH --mem=1440G
-#SBATCH --output=distill_dmd_t2v_1.3b_output/distill_dmd_t2v_1.3b_%j.out
-#SBATCH --error=distill_dmd_t2v_1.3b_output/distill_dmd_t2v_1.3b_%j.err
-#SBATCH --exclusive
-
-# Basic Info
-export NCCL_P2P_DISABLE=1
-export TORCH_NCCL_ENABLE_MONITORING=0
-export NCCL_DEBUG_SUBSYS=INIT,NET
-# different cache dir for different processes
-# export TRITON_CACHE_DIR=/tmp/triton_cache_${SLURM_PROCID}
-export MASTER_PORT=29500
-export NODE_RANK=$SLURM_PROCID
-nodes=( $(scontrol show hostnames $SLURM_JOB_NODELIST) )
-export MASTER_ADDR=${nodes[0]}
-export TOKENIZERS_PARALLELISM=false
-# export WANDB_API_KEY="8d9f4b39abd68eb4e29f6fc010b7ee71a2207cde"
-export WANDB_API_KEY="50632ebd88ffd970521cec9ab4a1a2d7e85bfc45"
-# export WANDB_API_KEY='your_wandb_api_key_here'
-export WANDB_BASE_URL="https://api.wandb.ai"
-export WANDB_MODE=online
-export FASTVIDEO_ATTENTION_BACKEND=FLASH_ATTN
-
-export HF_HOME=/mnt/data/huggingface
-export HF_DATASETS_CACHE="/tmp/hf_datasets_${SLURM_JOB_ID}_${SLURM_PROCID}"
-mkdir -p "${HF_DATASETS_CACHE}"
-export CUDA_HOME=/usr/local/cuda
-export PATH="$CUDA_HOME/bin:$PATH"
-export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/home/shared-bin/local/usr/lib/aarch64-linux-gnu:$CUDA_HOME/lib64"
-export CUDACXX="$CUDA_HOME/bin/nvcc"
-
-source ~/miniconda3/bin/activate
-conda activate fastvideo
-
-# Configs
-NUM_GPUS=8
-NUM_TOTAL_GPUS=64
-
-# Model paths for Self-Forcing DMD distillation:
-GENERATOR_MODEL_PATH="wlsaidhi/SFWan2.1-T2V-1.3B-Diffusers"
-REAL_SCORE_MODEL_PATH="Wan-AI/Wan2.1-T2V-14B-Diffusers"  # Teacher model
-FAKE_SCORE_MODEL_PATH="Wan-AI/Wan2.1-T2V-1.3B-Diffusers"  # Critic model
-
-DATA_DIR="/mnt/data/vidprom_filtered_extended_umt5_text_embed"
-# DATA_DIR="/mnt/data/mixkit_wan_1.3b_processed_t2v_distributed"
-DATA_DIR_2="/mnt/data/mixkit_wan_1.3b_processed_t2v_distributed"
-VALIDATION_DATASET_FILE="/mnt/home/zhouw.jerry2017/FastVideo/examples/distill/SFWan2.1-T2V/validation_64.json"
-# export CUDA_VISIBLE_DEVICES=4,5
-# IP=[MASTER NODE IP]
-
-training_args=(
-  --tracker_project_name SFwan_t2v_distill_self_forcing_dmd
-  --wandb_run_name causal_forcing_vidprom_extended
-  --output_dir /mnt/data/wan_cf_df_distill_1.3b
-  --max_train_steps 4000
-  --train_batch_size 1
-  --train_sp_batch_size 1
-  --gradient_accumulation_steps 1
-  --num_latent_t 21
-  --num_height 480
-  --num_width 832
-  --enable_gradient_checkpointing_type "full"
-  --log_visualization
-  --simulate_generator_forward
-  --num_frames 81
-  --num_frame_per_block 3  # Frame generation block size for self-forcing
-  --enable_gradient_masking
-  --gradient_mask_last_n_frames 21
-)
-
-parallel_args=(
-  --num_gpus $NUM_TOTAL_GPUS # 64
-  --sp_size 1
-  --tp_size 1
-  --hsdp_replicate_dim 1 # 64
-  --hsdp_shard_dim $NUM_TOTAL_GPUS
-)
-
-model_args=(
-  --model_path $GENERATOR_MODEL_PATH  # TODO: check if you can remove this in this script
-  --pretrained_model_name_or_path $GENERATOR_MODEL_PATH
-  --real_score_model_path $REAL_SCORE_MODEL_PATH
-  --fake_score_model_path $FAKE_SCORE_MODEL_PATH
-)
-
-dataset_args=(
-  --data_path "$DATA_DIR"
-  # --data_path_2 "$DATA_DIR_2"
-  --dataloader_num_workers 4
-)
-
-validation_args=(
-  --log_validation
-  --validation_dataset_file "$VALIDATION_DATASET_FILE"
-  --validation_steps 100
-  --validation_sampling_steps "4"
-  --validation_guidance_scale "1.0"
-)
-
-optimizer_args=(
-  --optimizer-type "adamw"
-  --learning_rate 1e-5
-  --mixed_precision "bf16"
-  --training_state_checkpointing_steps 500
-  --weight_only_checkpointing_steps 500
-  --weight_decay 0.01
-  --betas '0.0,0.999'
-  --max_grad_norm 1.0
-)
-
-miscellaneous_args=(
-  --inference_mode False
-  --checkpoints_total_limit 3
-  --training_cfg_rate 0.0
-  --dit_precision "fp32"
-  --flow_shift 5
-  --seed 1000
-  --use_ema True
-  --ema_decay 0.99
-  --ema_start_step 200
-#   --init_weights_from_safetensors "/mnt/data/wan_tf_ode_init_3333/checkpoint-1600/transformer/diffusion_pytorch_model.safetensors"
-#   --init_weights_from_safetensors "/mnt/data/wan_tf_init_3333/checkpoint-800/transformer/diffusion_pytorch_model.safetensors"
-  # --init_weights_from_safetensors "/mnt/data/Causal-Forcing-wan1.3b/causal_ode.safetensors"
-  # --init_weights_from_safetensors "/mnt/data/wan_ar_diffusion_3333_mixkit_shift5/checkpoint-7200/transformer/diffusion_pytorch_model.safetensors"
-  --init_weights_from_safetensors "/mnt/data/wan_ar_diffusion_3333_mixkit_shift5/checkpoint-4800/transformer/diffusion_pytorch_model.safetensors"
-  # --init_weights_from_safetensors "/mnt/data/wan_tf_ode_init_3333_mixkit_shift5/checkpoint-2000/transformer/diffusion_pytorch_model.safetensors"
-  # --init_weights_from_safetensors "/mnt/data/wan_ar_df_diffusion_3333_mixkit_shift5/checkpoint-4800/transformer/diffusion_pytorch_model.safetensors"
-)
-
-dmd_args=(
-  --dmd_denoising_steps '1000,750,500,250'
-  --min_timestep_ratio 0.02
-  --max_timestep_ratio 0.98
-  --dfake_gen_update_ratio 5
-  --real_score_guidance_scale 3.0
-  --fake_score_learning_rate 2e-6
-  --fake_score_betas '0.0,0.999'
-  --warp_denoising_step
-  # --use-decoupled-dmd True
-  # --use-gan-loss True
-)
-
-self_forcing_args=(
-  --independent_first_frame False  # Whether to treat first frame independently
-  --same_step_across_blocks True  # Whether to use same denoising step across all blocks
-  --last_step_only False  # Whether to only use the last denoising step
-  --context_noise 0  # Amount of noise to add during context caching (0 = no noise)
-)
-
-srun torchrun \
---nnodes $SLURM_JOB_NUM_NODES \
---nproc_per_node $NUM_GPUS \
---node_rank $SLURM_PROCID \
---rdzv_backend=c10d \
---rdzv_endpoint="$MASTER_ADDR:$MASTER_PORT" \
-    fastvideo/training/wan_self_forcing_distillation_pipeline.py \
-    "${parallel_args[@]}" \
-    "${model_args[@]}" \
-    "${dataset_args[@]}" \
-    "${training_args[@]}" \
-    "${optimizer_args[@]}" \
-    "${validation_args[@]}" \
-    "${miscellaneous_args[@]}" \
-    "${dmd_args[@]}" \
-    "${self_forcing_args[@]}"

examples/inference/basic/basic.py

@@ -15,7 +15,7 @@ def main():
         use_fsdp_inference=False, # set to True if GPU is out of memory
         dit_cpu_offload=False,
         vae_cpu_offload=False,
-        text_encoder_cpu_offload=False,
+        text_encoder_cpu_offload=True,
         pin_cpu_memory=True, # set to false if low CPU RAM or hit obscure "CUDA error: Invalid argument"
         # image_encoder_cpu_offload=False,
     )

examples/inference/basic/basic_cosmos2_5_i2w.py

@@ -0,0 +1,50 @@
+# SPDX-License-Identifier: Apache-2.0
+from fastvideo import VideoGenerator
+from fastvideo.configs.sample import SamplingParam
+
+
+def main():
+    # Point this to your local diffusers model dir (or replace with a HF model ID).
+    model_path = "KyleShao/Cosmos-Predict2.5-2B-Diffusers"
+
+    generator = VideoGenerator.from_pretrained(
+        model_path,
+        num_gpus=1,
+        use_fsdp_inference=False,  # set True if GPU is out of memory
+        dit_cpu_offload=False,
+        vae_cpu_offload=False,
+        text_encoder_cpu_offload=True,
+        pin_cpu_memory=True,
+    )
+
+    sampling_param = SamplingParam.from_pretrained(model_path)
+
+    # image2world example from official repo
+    image_path = "assets/images/bus_terminal.jpg"
+
+    prompt = (
+        "A nighttime city bus terminal gradually shifts from stillness to subtle movement. "
+        "At first, multiple double-decker buses are parked under the glow of overhead lights, "
+        "with a central bus labeled '87D' facing forward and stationary. "
+        "As the video progresses, the bus in the middle moves ahead slowly, its headlights brightening the surrounding area "
+        "and casting reflections onto adjacent vehicles. "
+        "The motion creates space in the lineup, signaling activity within the otherwise quiet station. "
+        "It then comes to a smooth stop, resuming its position in line. "
+        "Overhead signage in Chinese characters remains illuminated, enhancing the vibrant, urban night scene."
+    )
+
+    generator.generate_video(
+        prompt,
+        sampling_param=sampling_param,
+        image_path=str(image_path),
+        num_cond_frames=1,
+        output_path="outputs_video/cosmos2_5_i2w.mp4",
+        save_video=True,
+    )
+
+    generator.shutdown()
+
+
+if __name__ == "__main__":
+    main()
+

examples/inference/basic/basic_cosmos2_5_t2w.py

@@ -1,4 +1,5 @@
 from fastvideo import VideoGenerator
+from fastvideo.configs.sample import SamplingParam
 
 
 def main():
@@ -15,19 +16,27 @@ def main():
         pin_cpu_memory=True,
     )
 
+    # Load default sampling parameters (negative_prompt, resolution, steps, etc.)
+    sampling_param = SamplingParam.from_pretrained(model_path)
+
     prompt = (
-       "A high-definition video captures the precision of robotic welding in an industrial setting. The first frame showcases a robotic arm, equipped with a welding torch, positioned over a large metal structure. The welding process is in full swing, with bright sparks and intense light illuminating the scene, creating a vivid display of blue and white hues. A significant amount of smoke billows around the welding area, partially obscuring the view but emphasizing the heat and activity. The background reveals parts of the workshop environment, including a ventilation system and various pieces of machinery, indicating a busy and functional industrial workspace. As the video progresses, the robotic arm maintains its steady position, continuing the welding process and moving to its left. The welding torch consistently emits sparks and light, and the smoke continues to rise, diffusing slightly as it moves upward. The metal surface beneath the torch shows ongoing signs of heating and melting. The scene retains its industrial ambiance, with the welding sparks and smoke dominating the visual field, underscoring the ongoing nature of the welding operation."
+        "A high-definition video captures the precision of robotic welding in an industrial setting. "
+        "The first frame showcases a robotic arm, equipped with a welding torch, positioned over a large metal structure. "
+        "The welding process is in full swing, with bright sparks and intense light illuminating the scene, "
+        "creating a vivid display of blue and white hues. "
+        "A significant amount of smoke billows around the welding area, partially obscuring the view but emphasizing the heat and activity. "
+        "The background reveals parts of the workshop environment, including a ventilation system and various pieces of machinery, "
+        "indicating a busy and functional industrial workspace. "
+        "As the video progresses, the robotic arm maintains its steady position, continuing the welding process and moving to its left. "
+        "The welding torch consistently emits sparks and light, and the smoke continues to rise, diffusing slightly as it moves upward. "
+        "The metal surface beneath the torch shows ongoing signs of heating and melting. "
+        "The scene retains its industrial ambiance, with the welding sparks and smoke dominating the visual field, "
+        "underscoring the ongoing nature of the welding operation."
     )
 
-    video = generator.generate_video(
+    generator.generate_video(
         prompt,
-        negative_prompt="",
-        height=704,
-        width=1280,
-        num_frames=77,
-        num_inference_steps=35,
-        guidance_scale=7.0,
-        fps=24,
+        sampling_param=sampling_param,
         output_path="outputs_video/cosmos2_5_t2w.mp4",
         save_video=True,
     )

examples/inference/basic/basic_cosmos2_5_v2w.py

@@ -0,0 +1,53 @@
+# SPDX-License-Identifier: Apache-2.0
+from fastvideo import VideoGenerator
+from fastvideo.configs.sample import SamplingParam
+
+
+def main():
+    # Point this to your local diffusers model dir (or replace with a HF model ID).
+    model_path = "KyleShao/Cosmos-Predict2.5-2B-Diffusers"
+
+    generator = VideoGenerator.from_pretrained(
+        model_path,
+        num_gpus=1,
+        use_fsdp_inference=False,  # set True if GPU is out of memory
+        dit_cpu_offload=False,
+        vae_cpu_offload=False,
+        text_encoder_cpu_offload=True,
+        pin_cpu_memory=True,
+    )
+
+    sampling_param = SamplingParam.from_pretrained(model_path)
+
+    # video2world example from official repo
+    video_path = "assets/videos/robot_pouring.mp4"
+
+    prompt = (
+        "A robotic arm, primarily white with black joints and cables, is shown in a clean, modern indoor setting with a white tabletop. "
+        "The arm, equipped with a gripper holding a small, light green pitcher, is positioned above a clear glass containing a reddish-brown liquid and a spoon. "
+        "The robotic arm is in the process of pouring a transparent liquid into the glass. "
+        "To the left of the pitcher, there is an opened jar with a similar reddish-brown substance visible through its transparent body. "
+        "In the background, a vase with white flowers and a brown couch are partially visible, adding to the contemporary ambiance. "
+        "The lighting is bright, casting soft shadows on the table. "
+        "The robotic arm's movements are smooth and controlled, demonstrating precision in its task. "
+        "As the video progresses, the robotic arm completes the pour, leaving the glass half-filled with the reddish-brown liquid. "
+        "The jar remains untouched throughout the sequence, and the spoon inside the glass remains stationary. "
+        "The other robotic arm on the right side also stays stationary throughout the video. "
+        "The final frame captures the robotic arm with the pitcher finishing the pour, with the glass now filled to a higher level, while the pitcher is slightly tilted but still held securely by the gripper."
+    )
+
+    generator.generate_video(
+        prompt,
+        sampling_param=sampling_param,
+        video_path=str(video_path),
+        num_cond_frames=1,
+        output_path="outputs_video/cosmos2_5_v2w.mp4",
+        save_video=True,
+    )
+
+    generator.shutdown()
+
+
+if __name__ == "__main__":
+    main()
+

examples/inference/basic/basic_gamecraft.py

@@ -0,0 +1,119 @@
+# SPDX-License-Identifier: Apache-2.0
+"""
+Basic inference script for HunyuanGameCraft video generation.
+
+HunyuanGameCraft generates game-like videos with camera/action control.
+It takes an optional image input and generates video with camera motion
+based on simple action commands (forward, left, right, backward, rotations).
+
+Available actions:
+    - forward (w): Move camera forward
+    - backward (s): Move camera backward  
+    - left (a): Move camera left (strafe)
+    - right (d): Move camera right (strafe)
+    - left_rot: Rotate camera left (pan)
+    - right_rot: Rotate camera right (pan)
+    - up_rot: Rotate camera up (tilt)
+    - down_rot: Rotate camera down (tilt)
+
+T2V vs I2V:
+    - Default: I2V (uses a default reference image). Set GAMECRAFT_I2V_IMAGE to a
+      URL or path to use a different image.
+    - T2V only (no reference image): run with GAMECRAFT_I2V_IMAGE= (empty).
+"""
+import os
+
+import torch
+
+from fastvideo import VideoGenerator
+from fastvideo.models.camera import create_camera_trajectory
+
+# Model configuration (use GAMECRAFT_MODEL_PATH for local weights)
+MODEL_PATH = os.environ.get("GAMECRAFT_MODEL_PATH", "FastVideo/HunyuanGameCraft-Diffusers")
+
+# Default prompts for demo
+DEFAULT_PROMPTS = {
+    "village": "A charming medieval village with cobblestone streets, thatched-roof houses, and vibrant flower gardens under a bright blue sky.",
+    "temple": "A majestic ancient temple stands under a clear blue sky, its grandeur highlighted by towering Doric columns and intricate architectural details.",
+    "forest": "A lush green forest with tall trees, dappled sunlight filtering through the leaves, and a winding dirt path.",
+    "beach": "A tropical beach with crystal clear turquoise water, white sand, and palm trees swaying in the breeze.",
+}
+
+# I2V: default reference image (URL). Can override with a local path.
+DEFAULT_I2V_IMAGE_URL = (
+    "https://huggingface.co/datasets/huggingface/documentation-images/"
+    "resolve/main/diffusers/astronaut.jpg"
+)
+DEFAULT_I2V_PROMPT = (
+    "An astronaut hatching from an egg, on the surface of the moon, "
+    "the darkness and depth of space realised in the background."
+)
+
+OUTPUT_PATH = "video_samples_gamecraft"
+
+
+def main():
+    # Initialize generator
+    # FastVideo will automatically download weights from HuggingFace
+    generator = VideoGenerator.from_pretrained(
+        MODEL_PATH,
+        num_gpus=1,
+        use_fsdp_inference=True,
+        dit_cpu_offload=True,
+        vae_cpu_offload=True,
+        text_encoder_cpu_offload=True,
+        pin_cpu_memory=True,
+    )
+
+    # Video parameters
+    height = 704
+    width = 1280
+    num_frames = 33
+    action = "forward"
+    action_speed = 0.2
+
+    # Create camera trajectory (Plücker coordinates)
+    camera_states = create_camera_trajectory(
+        action=action,
+        height=height,
+        width=width,
+        num_frames=num_frames,
+        action_speed=action_speed,
+        dtype=torch.bfloat16,
+    )
+    print(f"Camera states shape: {camera_states.shape}")
+
+    # I2V vs T2V: unset GAMECRAFT_I2V_IMAGE -> I2V (default image). Set to "" -> T2V.
+    env_image = os.environ.get("GAMECRAFT_I2V_IMAGE")
+    if env_image is None:
+        image_path = DEFAULT_I2V_IMAGE_URL  # default: I2V
+    elif env_image.strip() == "":
+        image_path = None  # T2V
+    else:
+        image_path = env_image.strip()  # I2V with given URL/path
+
+    is_i2v = image_path is not None
+    prompt = DEFAULT_I2V_PROMPT if is_i2v else DEFAULT_PROMPTS["temple"]
+    print(f"Mode: {'I2V' if is_i2v else 'T2V'}, prompt: {prompt[:60]}...")
+
+    gen_kw = dict(
+        prompt=prompt,
+        negative_prompt="",
+        camera_states=camera_states,
+        height=height,
+        width=width,
+        num_frames=num_frames,
+        num_inference_steps=50,
+        guidance_scale=6.0,
+        seed=42,
+        fps=24,
+        output_path=OUTPUT_PATH,
+        save_video=True,
+    )
+    if is_i2v:
+        gen_kw["image_path"] = image_path
+    generator.generate_video(**gen_kw)
+
+
+if __name__ == "__main__":
+    main()

examples/inference/basic/basic_hy15.py

@@ -39,4 +39,4 @@ def main():
 
 
 if __name__ == "__main__":
-    main()
+    main()

examples/inference/basic/basic_hy15_1080p.py

@@ -0,0 +1,43 @@
+from fastvideo import VideoGenerator
+import json
+# from fastvideo.configs.sample import SamplingParam
+
+OUTPUT_PATH = "video_samples_hy15_1080p"
+def main():
+    # FastVideo will automatically use the optimal default arguments for the
+    # model.
+    # If a local path is provided, FastVideo will make a best effort
+    # attempt to identify the optimal arguments.
+    generator = VideoGenerator.from_pretrained(
+        "weizhou03/HunyuanVideo-1.5-Diffusers-1080p-2SR", # 480p -> 720p -> 1080p
+        # or "weizhou03/HunyuanVideo-1.5-Diffusers-1080p" # 720p -> 1080p
+        # FastVideo will automatically handle distributed setup
+        num_gpus=1,
+        use_fsdp_inference=False, # set to True if GPU is out of memory
+        dit_cpu_offload=True,
+        vae_cpu_offload=True,
+        text_encoder_cpu_offload=True,
+        pin_cpu_memory=True, # set to false if low CPU RAM or hit obscure "CUDA error: Invalid argument"
+        # image_encoder_cpu_offload=False,
+    )
+
+    prompt = (
+        "A curious raccoon peers through a vibrant field of yellow sunflowers, its eyes "
+        "wide with interest. The playful yet serene atmosphere is complemented by soft "
+        "natural light filtering through the petals. Mid-shot, warm and cheerful tones."
+    )
+
+    video = generator.generate_video(prompt, output_path=OUTPUT_PATH, save_video=True, negative_prompt="")
+
+    prompt2 = (
+        "A majestic lion strides across the golden savanna, its powerful frame "
+        "glistening under the warm afternoon sun. The tall grass ripples gently in "
+        "the breeze, enhancing the lion's commanding presence. The tone is vibrant, "
+        "embodying the raw energy of the wild. Low angle, steady tracking shot, "
+        "cinematic.")
+
+    video2 = generator.generate_video(prompt2, output_path=OUTPUT_PATH, save_video=True, negative_prompt="")
+
+
+if __name__ == "__main__":
+    main()

examples/inference/basic/basic_hyworld.py

@@ -1,165 +1,11 @@
-# SPDX-License-Identifier: Apache-2.0
-"""
-Basic example for HYWorld (HY-WorldPlay) video generation using FastVideo.
-
-This example replicates the same functionality as HY-WorldPlay/run.sh,
-demonstrating image-to-video generation with camera trajectory control.
-"""
-
-import time
-import math
-import numpy as np
-import imageio
-import torchvision
-from einops import rearrange
-
 from fastvideo import VideoGenerator
-from fastvideo.pipelines import ForwardBatch
-from fastvideo.utils import shallow_asdict, align_to
-from fastvideo.logger import init_logger
-
-from fastvideo.models.dits.hyworld.pose import pose_to_input, compute_latent_num
 from fastvideo.models.dits.hyworld.resolution_utils import get_resolution_from_image
 
-logger = init_logger(__name__)
-
-class HYWorldVideoGenerator(VideoGenerator):
-    """Extended VideoGenerator that adds HYWorld-specific parameters to batch.extra."""
-
-    def _generate_single_video(self, prompt: str, sampling_param=None, **kwargs):
-        """Override to add viewmats, Ks, and action to batch.extra."""
-        fastvideo_args = self.fastvideo_args
-        pipeline_config = fastvideo_args.pipeline_config
-
-        if sampling_param is None:
-            from fastvideo.configs.sample import SamplingParam
-            sampling_param = SamplingParam.from_pretrained(fastvideo_args.model_path)
-
-        # Update sampling param with kwargs
-        if kwargs:
-            for key, value in kwargs.items():
-                if hasattr(sampling_param, key):
-                    setattr(sampling_param, key, value)
-
-        # Get pose string from sampling_param or kwargs
-        pose = kwargs.get('pose', getattr(sampling_param, 'POSE', 'w-31'))
-        num_frames = kwargs.get('num_frames', getattr(sampling_param, 'num_frames', 125))
-
-        # Calculate number of latents
-        latent_num = compute_latent_num(num_frames)
-
-        # Convert pose to viewmats, Ks, and action
-        viewmats, Ks, action = pose_to_input(pose, latent_num)
-
-        # Convert to tensors and add batch dimension
-        viewmats = viewmats.unsqueeze(0)  # (1, T, 4, 4)
-        Ks = Ks.unsqueeze(0)  # (1, T, 3, 3)
-        action = action.unsqueeze(0)  # (1, T)
-
-        # Validate inputs
-        prompt = prompt.strip()
-        sampling_param = sampling_param.__class__(**shallow_asdict(sampling_param))
-        output_path = kwargs.get("output_path", sampling_param.output_path)
-        sampling_param.prompt = prompt
-
-        if sampling_param.negative_prompt is not None:
-            sampling_param.negative_prompt = sampling_param.negative_prompt.strip()
-
-        # Validate dimensions
-        if (sampling_param.height <= 0 or sampling_param.width <= 0 or
-            sampling_param.num_frames <= 0):
-            raise ValueError(
-                f"Height, width, and num_frames must be positive integers")
-
-        temporal_scale_factor = pipeline_config.vae_config.arch_config.temporal_compression_ratio
-        num_frames = sampling_param.num_frames
-        num_gpus = fastvideo_args.num_gpus
-        use_temporal_scaling_frames = pipeline_config.vae_config.use_temporal_scaling_frames
-
-        # Adjust number of frames based on number of GPUs
-        if use_temporal_scaling_frames:
-            orig_latent_num_frames = (num_frames - 1) // temporal_scale_factor + 1
-        else:
-            orig_latent_num_frames = sampling_param.num_frames // 17 * 3
-
-        if orig_latent_num_frames % fastvideo_args.num_gpus != 0:
-            if use_temporal_scaling_frames:
-                new_num_frames = (orig_latent_num_frames - 1) * temporal_scale_factor + 1
-            else:
-                divisor = math.lcm(3, num_gpus)
-                orig_latent_num_frames = (
-                    (orig_latent_num_frames + divisor - 1) // divisor) * divisor
-                new_num_frames = orig_latent_num_frames // 3 * 17
-
-            logger.info(
-                "Adjusting number of frames from %s to %s based on number of GPUs (%s)",
-                sampling_param.num_frames, new_num_frames, fastvideo_args.num_gpus)
-            sampling_param.num_frames = new_num_frames
-
-        # Calculate sizes
-        target_height = align_to(sampling_param.height, 16)
-        target_width = align_to(sampling_param.width, 16)
-
-        # Calculate latent sizes
-        latents_size = [(sampling_param.num_frames - 1) // 4 + 1,
-                        sampling_param.height // 8, sampling_param.width // 8]
-        n_tokens = latents_size[0] * latents_size[1] * latents_size[2]
-
-        # Prepare batch
-        batch = ForwardBatch(
-            **shallow_asdict(sampling_param),
-            eta=0.0,
-            n_tokens=n_tokens,
-            VSA_sparsity=fastvideo_args.VSA_sparsity,
-        )
-
-        # Add HYWorld-specific parameters to batch.extra
-        batch.extra['viewmats'] = viewmats
-        batch.extra['Ks'] = Ks
-        batch.extra['action'] = action
-        batch.extra['chunk_latent_frames'] = 16  # For bidirectional model
-
-        # Run inference
-        start_time = time.perf_counter()
-        output_batch = self.executor.execute_forward(batch, fastvideo_args)
-        samples = output_batch.output
-        logging_info = output_batch.logging_info
-
-        gen_time = time.perf_counter() - start_time
-        logger.info("Generated successfully in %.2f seconds", gen_time)
-
-        # Process outputs
-        videos = rearrange(samples, "b c t h w -> t b c h w")
-        frames = []
-        for x in videos:
-            x = torchvision.utils.make_grid(x, nrow=6)
-            x = x.transpose(0, 1).transpose(1, 2).squeeze(-1)
-            frames.append((x * 255).numpy().astype(np.uint8))
-
-        # Save video if requested
-        if batch.save_video:
-            imageio.mimsave(output_path, frames, fps=batch.fps, format="mp4")
-            logger.info("Saved video to %s", output_path)
-
-        if batch.return_frames:
-            return frames
-        else:
-            return {
-                "samples": samples,
-                "frames": frames,
-                "prompts": prompt,
-                "size": (target_height, target_width, batch.num_frames),
-                "generation_time": gen_time,
-                "logging_info": logging_info,
-                "trajectory": output_batch.trajectory_latents,
-                "trajectory_timesteps": output_batch.trajectory_timesteps,
-                "trajectory_decoded": output_batch.trajectory_decoded,
-            }
-
 # Default prompt from HY-WorldPlay run.sh
 DEFAULT_PROMPT = 'A paved pathway leads towards a stone arch bridge spanning a calm body of water.  Lush green trees and foliage line the path and the far bank of the water. A traditional-style pavilion with a tiered, reddish-brown roof sits on the far shore. The water reflects the surrounding greenery and the sky.  The scene is bathed in soft, natural light, creating a tranquil and serene atmosphere. The pathway is composed of large, rectangular stones, and the bridge is constructed of light gray stone.  The overall composition emphasizes the peaceful and harmonious nature of the landscape.'
 DEFAULT_IMAGE = 'https://raw.githubusercontent.com/Tencent-Hunyuan/HY-WorldPlay/main/assets/img/test.png'
 
+OUTPUT_PATH = "video_samples_hyworld"
 def main():
     import argparse
 
@@ -169,7 +15,7 @@ def main():
     parser.add_argument("--prompt", type=str, default=DEFAULT_PROMPT, help="Text prompt for video generation")
     parser.add_argument("--image", type=str, default=DEFAULT_IMAGE, help="Path or URL to input image")
     parser.add_argument("--pose", type=str, default='w-31', help="Pose string (e.g., 'a-31', 'w-31', 's-31', 'd-31')")
-    parser.add_argument("--output_path", type=str, default='video_samples_hyworld', help="Output video path")
+    parser.add_argument("--output_path", type=str, default=OUTPUT_PATH, help="Output video path")
     parser.add_argument("--num-frames", type=int, default=125, help="Number of frames")
     parser.add_argument("--seed", type=int, default=1, help="Random seed")
     parser.add_argument("--resolution", type=str, default="480p", help="Only support 480p for now")
@@ -185,8 +31,7 @@ def main():
 
     # Initialize generator
     print("\nInitializing VideoGenerator for HYWorld...")
-    
-    generator = HYWorldVideoGenerator.from_pretrained(
+    generator = VideoGenerator.from_pretrained(
         "FastVideo/HY-WorldPlay-Bidirectional-Diffusers",
         num_gpus=1,
         use_fsdp_inference=True,
@@ -198,11 +43,12 @@ def main():
     )
 
     # Generate video
+    # The pose string is automatically converted to camera matrices by the pipeline
     print("\nGenerating video...")
-    start_time = time.time()
-    video = generator.generate_video(
+    generator.generate_video(
         prompt=args.prompt,
         image_path=args.image,
+        pose=args.pose,  # Camera trajectory control
         output_path=args.output_path,
         save_video=True,
         negative_prompt="",
@@ -211,13 +57,9 @@ def main():
         height=HEIGHT,
         width=WIDTH,
         seed=args.seed,
-        pose=args.pose,
     )
-    elapsed = time.time() - start_time
 
-    print(f"\nVideo generated successfully!")
-    print(f"Saved to: {args.output_path}")
-    print(f"Time: {elapsed:.2f}s")
+    print(f"\nVideo saved to: {args.output_path}")
 
 
 if __name__ == "__main__":

examples/inference/basic/basic_lingbotworld_base_cam.py

@@ -0,0 +1,49 @@
+from fastvideo import VideoGenerator
+from fastvideo.models.dits.lingbotworld.cam_utils import prepare_camera_embedding
+
+# from fastvideo.configs.sample import SamplingParam
+OUTPUT_PATH = "video_samples_lingbotworld"
+def main():
+    # FastVideo will automatically use the optimal default arguments for the
+    # model.
+    # If a local path is provided, FastVideo will make a best effort
+    # attempt to identify the optimal arguments.
+    generator = VideoGenerator.from_pretrained(
+         "FastVideo/LingBot-World-Base-Cam-Diffusers",
+        # FastVideo will automatically handle distributed setup
+        num_gpus=1,
+        use_fsdp_inference=False, # set to True if GPU is out of memory
+        dit_cpu_offload=True, # DiT need to be offloaded for MoE
+        vae_cpu_offload=False,
+        text_encoder_cpu_offload=True,
+        # Set pin_cpu_memory to false if CPU RAM is limited and there're no frequent CPU-GPU transfer
+        pin_cpu_memory=True,
+        # image_encoder_cpu_offload=False,
+    )
+
+    num_frames = 81
+    prompt = "The video presents a soaring journey through a fantasy jungle. The wind whips past the rider's blue hands gripping the reins, causing the leather straps to vibrate. The ancient gothic castle approaches steadily, its stone details becoming clearer against the backdrop of floating islands and distant waterfalls."
+    image_path = "https://raw.githubusercontent.com/Robbyant/lingbot-world/main/examples/00/image.jpg"
+    action_path = "examples/inference/basic/lingbotworld_examples/00"
+    c2ws_plucker_emb, num_frames = prepare_camera_embedding(
+        action_path=action_path,
+        num_frames=num_frames,
+        height=480,
+        width=832,
+        spatial_scale=8,
+    )
+
+    generator.generate_video(
+        prompt,
+        image_path=image_path,
+        output_path=OUTPUT_PATH,
+        save_video=True,
+        num_frames=num_frames,
+        height=480,
+        width=832,
+        c2ws_plucker_emb=c2ws_plucker_emb,
+    )
+
+
+if __name__ == "__main__":
+    main()

examples/inference/basic/basic_ltx2.py

@@ -1,5 +1,6 @@
 from fastvideo import VideoGenerator
 
+
 PROMPT = (
     "A warm sunny backyard. The camera starts in a tight cinematic close-up "
     "of a woman and a man in their 30s, facing each other with serious "
@@ -16,19 +17,23 @@ PROMPT = (
 
 
 def main() -> None:
+    # Uses FastVideo default sampling settings for LTX2 base.
     generator = VideoGenerator.from_pretrained(
-        "FastVideo/LTX2-Distilled-Diffusers",
+        "Davids048/LTX2-Base-Diffusers",
         num_gpus=1,
     )
 
-    output_path = "outputs_video/ltx2_basic/output_ltx2_distilled_t2v.mp4"
+    output_path = "outputs_video/ltx2_basic/output_ltx2_base_t2v_1088_1920_1.1.mp4"
     generator.generate_video(
         prompt=PROMPT,
         output_path=output_path,
         save_video=True,
+        num_frames=121,
+        height=1088,
+        width=1920,
     )
     generator.shutdown()
 
 
 if __name__ == "__main__":
-    main()
+    main()

examples/inference/basic/basic_ltx2_distilled.py

@@ -0,0 +1,35 @@
+from fastvideo import VideoGenerator
+
+PROMPT = (
+    "A warm sunny backyard. The camera starts in a tight cinematic close-up "
+    "of a woman and a man in their 30s, facing each other with serious "
+    "expressions. The woman, emotional and dramatic, says softly, \"That's "
+    "it... Dad's lost it. And we've lost Dad.\" The man exhales, slightly "
+    "annoyed: \"Stop being so dramatic, Jess.\" A beat. He glances aside, "
+    "then mutters defensively, \"He's just having fun.\" The camera slowly "
+    "pans right, revealing the grandfather in the garden wearing enormous "
+    "butterfly wings, waving his arms in the air like he's trying to take "
+    "off. He shouts, \"Wheeeew!\" as he flaps his wings with full commitment. "
+    "The woman covers her face, on the verge of tears. The tone is deadpan, "
+    "absurd, and quietly tragic."
+)
+import os
+os.environ["FASTVIDEO_ATTENTION_BACKEND"] = "FLASH_ATTN"
+
+def main() -> None:
+    generator = VideoGenerator.from_pretrained(
+        "FastVideo/LTX2-Distilled-Diffusers",
+        num_gpus=4,
+    )
+
+    output_path = "outputs_video/ltx2_basic/output_ltx2_distilled_t2v.mp4"
+    generator.generate_video(
+        prompt=PROMPT,
+        output_path=output_path,
+        save_video=True,
+    )
+    generator.shutdown()
+
+
+if __name__ == "__main__":
+    main()

examples/inference/basic/basic_sd35_t2i.py

@@ -0,0 +1,137 @@
+# SPDX-License-Identifier: Apache-2.0
+
+from __future__ import annotations
+
+import argparse
+import os
+import re
+from typing import List
+
+
+DEFAULT_PROMPTS = [
+    "a photo of a cat",
+    "a cinematic photo of a red panda wearing a tiny backpack, standing on a rainy neon-lit street at night, shallow depth of field, sharp focus, 35mm, bokeh",
+]
+
+
+def _safe_filename(text: str, max_len: int = 100) -> str:
+    """
+    Make a stable, filesystem-friendly filename base.
+    VideoGenerator uses prompt[:100].strip() internally, so we mirror that,
+    but also remove path separators and other problematic characters.
+    """
+    s = text[:max_len].strip()
+    s = s.replace(os.sep, "_")
+    if os.altsep:
+        s = s.replace(os.altsep, "_")
+    s = re.sub(r"\s+", " ", s)
+    s = re.sub(r"[^A-Za-z0-9 .,_-]", "_", s)
+    s = s.strip(" .")
+    return s or "prompt"
+
+
+def _remove_existing_outputs(out_dir: str, filename_base: str) -> None:
+    """
+    Ensure deterministic naming by deleting any existing outputs that would
+    cause VideoGenerator to append suffixes like _1, _2, etc.
+    """
+    if not os.path.isdir(out_dir):
+        return
+
+    pattern = re.compile(rf"^{re.escape(filename_base)}(_\d+)?\.(mp4|png)$")
+    for fn in os.listdir(out_dir):
+        if pattern.match(fn):
+            try:
+                os.remove(os.path.join(out_dir, fn))
+            except FileNotFoundError:
+                pass
+
+
+def parse_args() -> argparse.Namespace:
+    p = argparse.ArgumentParser(description="Run SD3.5 Medium text-to-image with FastVideo VideoGenerator.")
+    p.add_argument("--model-path", default="stabilityai/stable-diffusion-3.5-medium", help="Path to local diffusers-format SD3.5 weights directory.")
+    p.add_argument(
+        "--out-dir",
+        "--outdir",
+        default="outputs/sd35/samples",
+        help="Output directory for generated mp4 files.",
+    )
+    p.add_argument(
+        "--prompt",
+        action="append",
+        default=None,
+        help="Prompt text. Repeat --prompt multiple times to generate multiple samples.",
+    )
+    p.add_argument("--negative", default="lowres, blurry, jpeg artifacts, watermark, text", help="Negative prompt.")
+    p.add_argument(
+        "--backend",
+        default=None,
+        help="Set FASTVIDEO_ATTENTION_BACKEND (e.g. TORCH_SDPA). If omitted, respects the existing env var.",
+    )
+    p.add_argument("--seed", type=int, default=42, help="Base seed. Each prompt uses seed + prompt_idx.")
+    p.add_argument("--height", type=int, default=768, help="Output height.")
+    p.add_argument("--width", type=int, default=768, help="Output width.")
+    p.add_argument("--steps", type=int, default=28, help="Number of inference steps.")
+    p.add_argument("--guidance", type=float, default=6.0, help="Guidance scale.")
+    p.add_argument("--num-gpus", type=int, default=1, help="Number of GPUs to use.")
+    return p.parse_args()
+
+
+def main() -> None:
+    args = parse_args()
+    prompts: List[str] = args.prompt if args.prompt else DEFAULT_PROMPTS
+
+    if args.backend:
+        os.environ["FASTVIDEO_ATTENTION_BACKEND"] = args.backend
+
+    from fastvideo import VideoGenerator
+
+    os.makedirs(args.out_dir, exist_ok=True)
+
+    init_kwargs = {
+        "num_gpus": args.num_gpus,
+        "workload_type": "t2i",
+        "sp_size": 1,
+        "tp_size": 1,
+        "dit_cpu_offload": False,
+        "dit_layerwise_offload": False,
+        "text_encoder_cpu_offload": False,
+        "vae_cpu_offload": False,
+        "image_encoder_cpu_offload": False,
+        "pin_cpu_memory": False,
+        "use_fsdp_inference": False,
+    }
+
+    generator = VideoGenerator.from_pretrained(model_path=args.model_path, **init_kwargs)
+    try:
+        for i, prompt in enumerate(prompts):
+            seed = args.seed + i
+
+            filename_base = f"sd35_{i:02d}_seed{seed}_{_safe_filename(prompt, max_len=80)}"
+            _remove_existing_outputs(args.out_dir, filename_base)
+
+            output_path = os.path.join(args.out_dir, f"{filename_base}.png")
+            print(f"[sd35] prompt_idx={i} seed={seed} output_path={output_path}")
+
+            generation_kwargs = {
+                "output_path": output_path,
+                "height": args.height,
+                "width": args.width,
+                "num_frames": 1,
+                "fps": 1,
+                "num_inference_steps": args.steps,
+                "guidance_scale": args.guidance,
+                "seed": seed,
+                "negative_prompt": args.negative,
+                "save_video": True,
+            }
+
+            generator.generate_video(prompt, **generation_kwargs)
+
+        print(f"[sd35] done. outputs written to: {args.out_dir}")
+    finally:
+        generator.shutdown()
+
+
+if __name__ == "__main__":
+    main()

examples/inference/basic/basic_self_forcing_causal_wan2_2_i2v.py

@@ -31,7 +31,7 @@ def main():
     sampling_param.height = 480
     sampling_param.seed = 1000
 
-    with open("prompts/mixkit_i2v.jsonl", "r") as f:
+    with open("assets/prompts/mixkit_i2v.jsonl", "r") as f:
         prompt_image_pairs = json.load(f)
 
     for prompt_image_pair in prompt_image_pairs:

examples/inference/basic/basic_wan2_2_Fun.py

@@ -10,13 +10,12 @@ def main():
     # If a local path is provided, FastVideo will make a best effort
     # attempt to identify the optimal arguments.
     generator = VideoGenerator.from_pretrained(
-        "weizhou03/Wan2.1-Fun-1.3B-InP-Diffusers",
-        # "IRMChen/Wan2.1-Fun-1.3B-Control-Diffusers",
+        "IRMChen/Wan2.1-Fun-1.3B-Control-Diffusers",
         # "alibaba-pai/Wan2.2-Fun-A14B-Control",
         # FastVideo will automatically handle distributed setup
         num_gpus=1,
         use_fsdp_inference=False, # set to True if GPU is out of memory
-        dit_cpu_offload=False, # DiT need to be offloaded for MoE
+        dit_cpu_offload=True, # DiT need to be offloaded for MoE
         vae_cpu_offload=False,
         text_encoder_cpu_offload=True,
         # Set pin_cpu_memory to false if CPU RAM is limited and there're no frequent CPU-GPU transfer
@@ -24,19 +23,14 @@ def main():
         # image_encoder_cpu_offload=False,
     )
 
-    # prompt = "一位年轻女性穿着一件粉色的连衣裙，裙子上有白色的装饰和粉色的纽扣。她的头发是紫色的，头上戴着一个红色的大蝴蝶结，显得非常可爱和精致。她还戴着一个红色的领结，整体造型充满了少女感和活力。她的表情温柔，双手轻轻交叉放在身前，姿态优雅。背景是简单的灰色，没有任何多余的装饰，使得人物更加突出。她的妆容清淡自然，突显了她的清新气质。整体画面给人一种甜美、梦幻的感觉，仿佛置身于童话世界中。"
-    # negative_prompt = "色调艳丽，过曝，静态，细节模糊不清，字幕，风格，作品，画作，画面，静止，整体发灰，最差质量，低质量，JPEG压缩残留，丑陋的，残缺的，多余的手指，画得不好的手部，画得不好的脸部，畸形的，毁容的，形态畸形的肢体，手指融合，静止不动的画面，杂乱的背景，三条腿，背景人很多，倒着走"
+    prompt = "一位年轻女性穿着一件粉色的连衣裙，裙子上有白色的装饰和粉色的纽扣。她的头发是紫色的，头上戴着一个红色的大蝴蝶结，显得非常可爱和精致。她还戴着一个红色的领结，整体造型充满了少女感和活力。她的表情温柔，双手轻轻交叉放在身前，姿态优雅。背景是简单的灰色，没有任何多余的装饰，使得人物更加突出。她的妆容清淡自然，突显了她的清新气质。整体画面给人一种甜美、梦幻的感觉，仿佛置身于童话世界中。"
+    negative_prompt = "色调艳丽，过曝，静态，细节模糊不清，字幕，风格，作品，画作，画面，静止，整体发灰，最差质量，低质量，JPEG压缩残留，丑陋的，残缺的，多余的手指，画得不好的手部，画得不好的脸部，畸形的，毁容的，形态畸形的肢体，手指融合，静止不动的画面，杂乱的背景，三条腿，背景人很多，倒着走"
     # prompt                  = "A young woman with beautiful, clear eyes and blonde hair stands in the forest, wearing a white dress and a crown. Her expression is serene, reminiscent of a movie star, with fair and youthful skin. Her brown long hair flows in the wind. The video quality is very high, with a clear view. High quality, masterpiece, best quality, high resolution, ultra-fine, fantastical."
     # negative_prompt         = "Twisted body, limb deformities, text captions, comic, static, ugly, error, messy code."
-    # image_path = "https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset_Wan2_2/v1.0/8.png"
-    # control_video_path = "https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset_Wan2_2/v1.0/pose.mp4"
-    import json, os
-    with open("/mnt/weka/home/hao.zhang/wei/FastVideo/data/mixkit_i2v.jsonl", "r") as f:
-        prompt_image_pairs = json.load(f)
-    for d in prompt_image_pairs:
-        prompt = d["prompt"]
-        image_path = os.path.join("/mnt/weka/home/hao.zhang/wei/FastVideo", d["image_path"])
-        video = generator.generate_video(prompt, image_path=image_path, output_path=OUTPUT_PATH, output_video_name=OUTPUT_NAME, save_video=True)
+    image_path = "https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset_Wan2_2/v1.0/8.png"
+    control_video_path = "https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset_Wan2_2/v1.0/pose.mp4"
+
+    video = generator.generate_video(prompt, negative_prompt=negative_prompt, image_path=image_path, video_path=control_video_path, output_path=OUTPUT_PATH, output_video_name=OUTPUT_NAME, save_video=True)
 
 if __name__ == "__main__":
     main()

examples/inference/basic/basic_wan2_2_ti2v.py

@@ -1,6 +1,6 @@
 from fastvideo import VideoGenerator
 
-OUTPUT_PATH = "video_samples_wan2_2_5B_ti2v_720p_77"
+OUTPUT_PATH = "video_samples_wan2_2_5B_ti2v"
 def main():
     # FastVideo will automatically use the optimal default arguments for the
     # model.
@@ -12,29 +12,17 @@ def main():
         # FastVideo will automatically handle distributed setup
         num_gpus=1,
         use_fsdp_inference=False, # set to True if GPU is out of memory
-        dit_cpu_offload=False,
+        dit_cpu_offload=True,
         vae_cpu_offload=False,
         text_encoder_cpu_offload=True,
         pin_cpu_memory=True, # set to false if low CPU RAM or hit obscure "CUDA error: Invalid argument"
         # image_encoder_cpu_offload=False,
     )
 
-    import json, os
-    with open("/mnt/weka/home/hao.zhang/wei/FastVideo/data/mixkit_i2v_720p.jsonl", "r") as f:
-        prompt_image_pairs = json.load(f)
-    for d in prompt_image_pairs:
-        prompt = d["prompt"]
-        image_path = os.path.join("/mnt/weka/home/hao.zhang/wei/FastVideo", d["image_path"])
-        video = generator.generate_video(prompt, image_path=image_path, output_path=OUTPUT_PATH, output_video_name=OUTPUT_PATH, save_video=True, num_frames=77)
-    return
-
     # I2V is triggered just by passing in an image_path argument
     prompt = "Summer beach vacation style, a white cat wearing sunglasses sits on a surfboard. The fluffy-furred feline gazes directly at the camera with a relaxed expression. Blurred beach scenery forms the background featuring crystal-clear waters, distant green hills, and a blue sky dotted with white clouds. The cat assumes a naturally relaxed posture, as if savoring the sea breeze and warm sunlight. A close-up shot highlights the feline's intricate details and the refreshing atmosphere of the seaside."
     image_path = "https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/wan_i2v_input.JPG"
-    # prompt = "In the midst of the joyous New Year's Eve celebration, the cheerful group of friends, their spirits lifted by the festivities, decides to immortalize the moment with a vibrant snapshot"
-    # image_path = "images/mixkit-a-cheerful-group-of-friends-celebrate-new-years-eve-and-51525.png"
     video = generator.generate_video(prompt, output_path=OUTPUT_PATH, save_video=True, image_path=image_path)
-    return
 
     # Generate another video with a different prompt, without reloading the
     # model!

examples/inference/gradio/serving/gradio_frontend.py

@@ -188,8 +188,9 @@ def load_example_prompts():
     prompt_to_image = {}
     # Try to find the JSON file relative to project root
     possible_json_paths = [
-        Path("prompts/mixkit_i2v.jsonl"),
-        Path(__file__).parent.parent.parent.parent / "prompts" / "mixkit_i2v.jsonl",
+        Path("assets/prompts/mixkit_i2v.jsonl"),
+        Path(__file__).resolve().parents[4] / "assets" / "prompts" /
+        "mixkit_i2v.jsonl",
     ]
     json_path = None
     for path in possible_json_paths:
@@ -201,8 +202,8 @@ def load_example_prompts():
         try:
             with open(json_path, "r", encoding='utf-8') as f:
                 data = json.load(f)
-                # Get the project root directory (parent of prompts directory)
-                project_root = json_path.parent.parent
+                # Resolve paths relative to repository root.
+                project_root = Path(__file__).resolve().parents[4]
                 for item in data:
                     prompt_text = item.get("prompt", "").strip()
                     image_path = item.get("image_path", "")
@@ -736,8 +737,8 @@ def main():
         allowed_paths=[
             os.path.abspath("outputs"), 
             os.path.abspath("fastvideo-logos"),
-            os.path.abspath("prompts"),
-            os.path.abspath("images"),
+            os.path.abspath("assets/prompts"),
+            os.path.abspath("assets/images"),
             os.path.abspath(tempfile.gettempdir()),
             os.path.abspath(os.path.join(tempfile.gettempdir(), "gradio")),
         ]
@@ -747,4 +748,4 @@ def main():
 
 
 if __name__ == "__main__":
-    main() 
+    main() 

examples/inference/sta_mask_search/inference_wan_sta.sh

@@ -5,7 +5,7 @@ export FASTVIDEO_ATTENTION_BACKEND=SLIDING_TILE_ATTN
 export MODEL_BASE=Wan-AI/Wan2.1-T2V-14B-Diffusers
 
 base_port=29503
-num_gpu=1
+num_gpu=8
 gpu_ids=$(seq 0 $((num_gpu-1)))
 skip_time_steps=12
 

examples/training/consistency_finetune/causal_ode_init/finetune_ode_init.sh

@@ -4,7 +4,7 @@ export WANDB_BASE_URL="https://api.wandb.ai"
 export WANDB_MODE=online
 export TOKENIZERS_PARALLELISM=false
 
-MODEL_PATH="Wan-AI/Wan2.1-T2V-1.3B-Diffusers"
+MODEL_PATH="wlsaidhi/SFWan2.1-T2V-1.3B-Diffusers"
 DATA_DIR="data/crush-smol_processed_t2v_1_3b_ode_init/"
 VALIDATION_DATASET_FILE="$(dirname "$0")/validation.json"
 NUM_GPUS=1
@@ -14,7 +14,6 @@ NUM_GPUS=1
 training_args=(
   --tracker_project_name "wan_ode_init"
   --output_dir "wan_ode_init_crush_smol"
-  --override_transformer_cls_name "CausalWanTransformer3DModel"
   --wandb_run_name "wan_ode_init_crush_smol"
   --max_train_steps 6000
   --train_batch_size 1
@@ -34,7 +33,7 @@ parallel_args=(
   --sp_size 1
   --tp_size 1
   --hsdp_replicate_dim 1
-  --hsdp_shard_dim 1
+  --hsdp_shard_dim $NUM_GPUS
 )
 
 # Model arguments
@@ -51,20 +50,17 @@ dataset_args=(
 
 # Validation arguments
 validation_args=(
-  --log_validation
-  --validation_dataset_file "$VALIDATION_DATASET_FILE"
-  --validation_steps 50
-  --validation_sampling_steps "50"
-  --validation_guidance_scale "6.0"
+  --log-visualization
+  --visualization-steps 100
 )
 
 # Optimizer arguments
 optimizer_args=(
-  --learning_rate 6e-6
+  --learning_rate 1e-5
   --mixed_precision "bf16"
   --weight_only_checkpointing_steps 1000
   --training_state_checkpointing_steps 1000
-  --weight_decay 1e-4
+  --weight_decay 0.01
   --max_grad_norm 1.0
 )
 

examples/training/consistency_finetune/causal_ode_init/hy15_finetune_ode_init.slurm

@@ -1,124 +0,0 @@
-#!/bin/bash
-#SBATCH --job-name=hy15_ode_3333
-#SBATCH --partition=main
-#SBATCH --nodes=8
-#SBATCH --ntasks=8
-#SBATCH --ntasks-per-node=1
-#SBATCH --gres=gpu:8
-#SBATCH --cpus-per-task=128
-#SBATCH --mem=1440G
-#SBATCH --output=hy15_ode_3333_output/hy15_ode_3333_%j.out
-#SBATCH --error=hy15_ode_3333_output/hy15_ode_3333_%j.err
-#SBATCH --exclusive
-
-# Basic Info
-export NCCL_P2P_DISABLE=1
-export TORCH_NCCL_ENABLE_MONITORING=0
-export NCCL_DEBUG_SUBSYS=INIT,NET
-# different cache dir for different processes
-export TRITON_CACHE_DIR=/tmp/triton_cache_${SLURM_PROCID}
-export MASTER_PORT=29500
-export NODE_RANK=$SLURM_PROCID
-nodes=( $(scontrol show hostnames $SLURM_JOB_NODELIST) )
-export MASTER_ADDR=${nodes[0]}
-export TOKENIZERS_PARALLELISM=false
-export WANDB_API_KEY='your_wandb_api_key_here'
-export WANDB_BASE_URL="https://api.wandb.ai"
-export WANDB_MODE=online
-export FASTVIDEO_ATTENTION_BACKEND=FLASH_ATTN
-
-# Configs
-NUM_GPUS=8
-NUM_TOTAL_GPUS=64
-
-MODEL_PATH="weizhou03/SFHunyuanVideo-1.5-Diffusers-480p_t2v"
-DATA_DIR="your_data_dir"
-VALIDATION_DATASET_FILE="your_validation_data_dir"
-# IP=[MASTER NODE IP]
-
-# Training arguments
-training_args=(
-  --tracker_project_name "hy15_ode_init"
-  --output_dir "your_output_dir"
-  --wandb_run_name "your_wandb_run_name"
-  --warp_denoising_step
-  --dmd_denoising_steps '1000,760,520,280,0'
-  --log_visualization
-  --visualization_steps 100
-  --max_train_steps 4000
-  --train_batch_size 1
-  --train_sp_batch_size 1
-  --gradient_accumulation_steps 2
-  --num_latent_t 21
-  --num_height 480
-  --num_width 848
-  --num_frames 81
-  --enable_gradient_checkpointing_type "full"
-)
-
-# Parallel arguments
-parallel_args=(
-  --num_gpus $NUM_TOTAL_GPUS # 64
-  --sp_size 1
-  --tp_size 1
-  --hsdp_replicate_dim 1 # 64
-  --hsdp_shard_dim $NUM_TOTAL_GPUS
-)
-
-# Model arguments
-model_args=(
-  --model_path $MODEL_PATH
-  --pretrained_model_name_or_path $MODEL_PATH
-  --text-encoder-cpu-offload
-)
-
-# Dataset arguments
-dataset_args=(
-  --data_path "$DATA_DIR" 
-  --dataloader_num_workers 4
-)
-
-# Optimizer arguments
-optimizer_args=(
-  --learning_rate 1e-5
-  --mixed_precision "bf16"
-  --training_state_checkpointing_steps 400
-  --weight_decay 0.01
-  --max_grad_norm 1.0
-)
-
-validation_args=(
-  # --log_validation
-  --validation_dataset_file "$VALIDATION_DATASET_FILE"
-  --validation_steps 10
-  --validation_sampling_steps "4"
-  --validation_guidance_scale "1.0" # not used for dmd inference
-  # --vae_cpu_offload True
-)
-
-# Miscellaneous arguments
-miscellaneous_args=(
-  --inference_mode False
-  --checkpoints_total_limit 6
-  --training_cfg_rate 0.1
-  --multi_phased_distill_schedule "4000-1"
-  --not_apply_cfg_solver
-  --dit_precision "fp32"
-  --num_euler_timesteps 50
-)
-
-# If you do not have 32 GPUs and to fit in memory, you can: 1. increase sp_size. 2. reduce num_latent_t
-srun torchrun \
---nnodes $SLURM_JOB_NUM_NODES \
---nproc_per_node $NUM_GPUS \
---node_rank $SLURM_PROCID \
---rdzv_backend=c10d \
---rdzv_endpoint="$MASTER_ADDR:$MASTER_PORT" \
-    fastvideo/training/ode_causal_pipeline.py \
-    "${parallel_args[@]}" \
-    "${model_args[@]}" \
-    "${dataset_args[@]}" \
-    "${training_args[@]}" \
-    "${optimizer_args[@]}" \
-    "${validation_args[@]}" \
-    "${miscellaneous_args[@]}"

examples/training/consistency_finetune/causal_ode_init/hy15_finetune_tf_init.slurm

@@ -1,133 +0,0 @@
-#!/bin/bash
-#SBATCH --job-name=hy15_tf_3333
-#SBATCH --partition=main
-#SBATCH --nodes=8
-#SBATCH --ntasks=8
-#SBATCH --ntasks-per-node=1
-#SBATCH --gres=gpu:8
-#SBATCH --cpus-per-task=128
-#SBATCH --mem=1440G
-#SBATCH --output=hy15_tf_3333_output/hy15_tf_3333_%j.out
-#SBATCH --error=hy15_tf_3333_output/hy15_tf_3333_%j.err
-#SBATCH --exclusive
-
-# Basic Info
-export NCCL_P2P_DISABLE=1
-export TORCH_NCCL_ENABLE_MONITORING=0
-export NCCL_DEBUG_SUBSYS=INIT,NET
-# different cache dir for different processes
-export TRITON_CACHE_DIR=/tmp/triton_cache_${SLURM_PROCID}
-export MASTER_PORT=29500
-export NODE_RANK=$SLURM_PROCID
-nodes=( $(scontrol show hostnames $SLURM_JOB_NODELIST) )
-export MASTER_ADDR=${nodes[0]}
-export TOKENIZERS_PARALLELISM=false
-# export WANDB_API_KEY="8d9f4b39abd68eb4e29f6fc010b7ee71a2207cde"
-export WANDB_API_KEY="50632ebd88ffd970521cec9ab4a1a2d7e85bfc45"
-# export WANDB_API_KEY='your_wandb_api_key_here'
-export WANDB_BASE_URL="https://api.wandb.ai"
-export WANDB_MODE=online
-export FASTVIDEO_ATTENTION_BACKEND=FLASH_ATTN
-
-source ~/conda/miniconda/bin/activate
-conda activate wei-fv-distill
-export HOME="/mnt/weka/home/hao.zhang/wei"
-
-# Configs
-NUM_GPUS=8
-NUM_TOTAL_GPUS=64
-
-MODEL_PATH="weizhou03/SFHunyuanVideo-1.5-Diffusers-480p_t2v"
-DATA_DIR="/mnt/weka/home/hao.zhang/wl/sharefs/wl/release/fv-ode-preprocessing-16k-hy15-121"
-VALIDATION_DATASET_FILE="/mnt/weka/home/hao.zhang/wl/FastVideo/examples/distill/Wan2.2-TI2V-5B-Diffusers/Data-free/validation_64.json"
-# IP=[MASTER NODE IP]
-
-# Training arguments
-training_args=(
-  --tracker_project_name "hy15_ode_init"
-  --output_dir "/mnt/weka/home/hao.zhang/wei/hy15_worldplay_tf_init_3333"
-  --wandb_run_name "hy15_worldplay_tf_init_3333"
-  --warp_denoising_step
-  --dmd_denoising_steps '1000,760,520,280,0'
-  --log_visualization
-  --visualization_steps 100
-  --max_train_steps 30000
-  --train_batch_size 1
-  --train_sp_batch_size 1
-  --gradient_accumulation_steps 1
-  --num_latent_t 21
-  --num_height 480
-  --num_width 848
-  --num_frames 81
-  --enable_gradient_checkpointing_type "full"
-)
-
-# Parallel arguments
-parallel_args=(
-  --num_gpus $NUM_TOTAL_GPUS # 64
-  --sp_size 1
-  --tp_size 1
-  --hsdp_replicate_dim 1 # 64
-  --hsdp_shard_dim $NUM_TOTAL_GPUS
-)
-
-# Model arguments
-model_args=(
-  --model_path $MODEL_PATH
-  --pretrained_model_name_or_path $MODEL_PATH
-  --text-encoder-cpu-offload
-)
-
-# Dataset arguments
-dataset_args=(
-  --data_path "$DATA_DIR" 
-  --dataloader_num_workers 4
-)
-
-# Optimizer arguments
-optimizer_args=(
-  --optimizer-type "muon"
-  --learning_rate 1e-5
-  --mixed_precision "bf16"
-  --training_state_checkpointing_steps 400
-  --weight_decay 0.01
-  --max_grad_norm 1.0
-)
-
-validation_args=(
-  # --log_validation
-  --validation_dataset_file "$VALIDATION_DATASET_FILE"
-  --validation_steps 10
-  --validation_sampling_steps "4"
-  --validation_guidance_scale "1.0" # not used for dmd inference
-  # --vae_cpu_offload True
-)
-
-# Miscellaneous arguments
-miscellaneous_args=(
-  --inference_mode False
-  --checkpoints_total_limit 6
-  --training_cfg_rate 0.1
-  --multi_phased_distill_schedule "4000-1"
-  --not_apply_cfg_solver
-  --dit_precision "fp32"
-  --num_euler_timesteps 50
-  --ema_start_step 0
-  # --enable_gradient_checkpointing_type "full"
-)
-
-# If you do not have 32 GPUs and to fit in memory, you can: 1. increase sp_size. 2. reduce num_latent_t
-srun torchrun \
---nnodes $SLURM_JOB_NUM_NODES \
---nproc_per_node $NUM_GPUS \
---node_rank $SLURM_PROCID \
---rdzv_backend=c10d \
---rdzv_endpoint="$MASTER_ADDR:$MASTER_PORT" \
-    fastvideo/training/hy15_ode_causal_pipeline.py \
-    "${parallel_args[@]}" \
-    "${model_args[@]}" \
-    "${dataset_args[@]}" \
-    "${training_args[@]}" \
-    "${optimizer_args[@]}" \
-    "${validation_args[@]}" \
-    "${miscellaneous_args[@]}"

examples/training/consistency_finetune/causal_ode_init/wan_df_init.slurm

@@ -1,137 +0,0 @@
-#!/bin/bash
-#SBATCH --job-name=wan_tf_3333
-#SBATCH --nodes=8
-#SBATCH --ntasks=8
-#SBATCH --ntasks-per-node=1
-#SBATCH --gres=gpu:8
-#SBATCH --cpus-per-task=128
-#SBATCH --mem=1440G
-#SBATCH --output=wan_tf_3333_output/wan_tf_3333_%j.out
-#SBATCH --error=wan_tf_3333_output/wan_tf_3333_%j.err
-#SBATCH --exclusive
-
-# Basic Info
-export NCCL_P2P_DISABLE=1
-export TORCH_NCCL_ENABLE_MONITORING=0
-export NCCL_DEBUG_SUBSYS=INIT,NET
-# different cache dir for different processes
-# export TRITON_CACHE_DIR=/tmp/triton_cache_${SLURM_PROCID}
-export MASTER_PORT=29500
-export NODE_RANK=$SLURM_PROCID
-nodes=( $(scontrol show hostnames $SLURM_JOB_NODELIST) )
-export MASTER_ADDR=${nodes[0]}
-export TOKENIZERS_PARALLELISM=false
-# export WANDB_API_KEY="8d9f4b39abd68eb4e29f6fc010b7ee71a2207cde"
-export WANDB_API_KEY="50632ebd88ffd970521cec9ab4a1a2d7e85bfc45"
-# export WANDB_API_KEY='your_wandb_api_key_here'
-export WANDB_BASE_URL="https://api.wandb.ai"
-export WANDB_MODE=online
-export FASTVIDEO_ATTENTION_BACKEND=FLASH_ATTN
-
-export HF_HOME=/mnt/data/huggingface
-export CUDA_HOME=/usr/local/cuda
-export PATH="$CUDA_HOME/bin:$PATH"
-export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/home/shared-bin/local/usr/lib/aarch64-linux-gnu:$CUDA_HOME/lib64"
-export CUDACXX="$CUDA_HOME/bin/nvcc"
-
-source ~/miniconda3/bin/activate
-conda activate fastvideo
-
-# Configs
-NUM_GPUS=8
-NUM_TOTAL_GPUS=64
-
-MODEL_PATH="Wan-AI/Wan2.1-T2V-1.3B-Diffusers"
-DATA_DIR="/mnt/data/mixkit_wan_1.3b_processed_t2v_distributed"
-VALIDATION_DATASET_FILE="/mnt/weka/home/hao.zhang/wl/FastVideo/examples/distill/Wan2.2-TI2V-5B-Diffusers/Data-free/validation_64.json"
-# IP=[MASTER NODE IP]
-
-# Training arguments
-training_args=(
-  --tracker_project_name "wan_tf_init"
-  --output_dir "/mnt/data/wan_ar_df_diffusion_3333_mixkit_shift5"
-  --wandb_run_name "wan_df_init_3333"
-  --warp_denoising_step
-  --dmd_denoising_steps '1000,750,500,250,0'
-  --log_visualization
-  --visualization_steps 100
-  --max_train_steps 30000
-  --train_batch_size 1
-  --train_sp_batch_size 1
-  --gradient_accumulation_steps 1
-  --num_latent_t 21
-  --num_height 480
-  --num_width 832
-  --num_frames 81
-  --override-transformer-cls-name "CausalWanTransformer3DModel"
-)
-
-# Parallel arguments
-parallel_args=(
-  --num_gpus $NUM_TOTAL_GPUS # 64
-  --sp_size 1
-  --tp_size 1
-  --hsdp_replicate_dim 1 # 64
-  --hsdp_shard_dim $NUM_TOTAL_GPUS
-)
-
-# Model arguments
-model_args=(
-  --model_path $MODEL_PATH
-  --pretrained_model_name_or_path $MODEL_PATH
-  --text-encoder-cpu-offload
-)
-
-# Dataset arguments
-dataset_args=(
-  --data_path "$DATA_DIR" 
-  --dataloader_num_workers 4
-)
-
-# Optimizer arguments
-optimizer_args=(
-  --optimizer-type "adamw"
-  --learning_rate 1e-5
-  --mixed_precision "bf16"
-  --training_state_checkpointing_steps 800
-  --weight_decay 0.01
-  --max_grad_norm 1.0
-)
-
-validation_args=(
-  # --log_validation
-  --validation_dataset_file "$VALIDATION_DATASET_FILE"
-  --validation_steps 10
-  --validation_sampling_steps "4"
-  --validation_guidance_scale "1.0" # not used for dmd inference
-  # --vae_cpu_offload True
-)
-
-# Miscellaneous arguments
-miscellaneous_args=(
-  --inference_mode False
-  --checkpoints_total_limit 6
-  --training_cfg_rate 0.1
-  --multi_phased_distill_schedule "4000-1"
-  --not_apply_cfg_solver
-  --dit_precision "fp32"
-  --num_euler_timesteps 50
-  --ema_start_step 0
-  --enable_gradient_checkpointing_type "full"
-)
-
-# If you do not have 32 GPUs and to fit in memory, you can: 1. increase sp_size. 2. reduce num_latent_t
-srun torchrun \
---nnodes $SLURM_JOB_NUM_NODES \
---nproc_per_node $NUM_GPUS \
---node_rank $SLURM_PROCID \
---rdzv_backend=c10d \
---rdzv_endpoint="$MASTER_ADDR:$MASTER_PORT" \
-    fastvideo/training/ar_diffusion_pipeline.py \
-    "${parallel_args[@]}" \
-    "${model_args[@]}" \
-    "${dataset_args[@]}" \
-    "${training_args[@]}" \
-    "${optimizer_args[@]}" \
-    "${validation_args[@]}" \
-    "${miscellaneous_args[@]}"

examples/training/consistency_finetune/causal_ode_init/wan_tf_init.slurm

@@ -1,138 +0,0 @@
-#!/bin/bash
-#SBATCH --job-name=wan_tf_3333
-#SBATCH --nodes=8
-#SBATCH --ntasks=8
-#SBATCH --ntasks-per-node=1
-#SBATCH --gres=gpu:8
-#SBATCH --cpus-per-task=128
-#SBATCH --mem=1440G
-#SBATCH --output=wan_tf_3333_output/wan_tf_3333_%j.out
-#SBATCH --error=wan_tf_3333_output/wan_tf_3333_%j.err
-#SBATCH --exclusive
-
-# Basic Info
-export NCCL_P2P_DISABLE=1
-export TORCH_NCCL_ENABLE_MONITORING=0
-export NCCL_DEBUG_SUBSYS=INIT,NET
-# different cache dir for different processes
-# export TRITON_CACHE_DIR=/tmp/triton_cache_${SLURM_PROCID}
-export MASTER_PORT=29500
-export NODE_RANK=$SLURM_PROCID
-nodes=( $(scontrol show hostnames $SLURM_JOB_NODELIST) )
-export MASTER_ADDR=${nodes[0]}
-export TOKENIZERS_PARALLELISM=false
-# export WANDB_API_KEY="8d9f4b39abd68eb4e29f6fc010b7ee71a2207cde"
-export WANDB_API_KEY="50632ebd88ffd970521cec9ab4a1a2d7e85bfc45"
-# export WANDB_API_KEY='your_wandb_api_key_here'
-export WANDB_BASE_URL="https://api.wandb.ai"
-export WANDB_MODE=online
-export FASTVIDEO_ATTENTION_BACKEND=FLASH_ATTN
-
-export HF_HOME=/mnt/data/huggingface
-export CUDA_HOME=/usr/local/cuda
-export PATH="$CUDA_HOME/bin:$PATH"
-export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/home/shared-bin/local/usr/lib/aarch64-linux-gnu:$CUDA_HOME/lib64"
-export CUDACXX="$CUDA_HOME/bin/nvcc"
-
-source ~/miniconda3/bin/activate
-conda activate fastvideo
-
-# Configs
-NUM_GPUS=8
-NUM_TOTAL_GPUS=64
-
-MODEL_PATH="Wan-AI/Wan2.1-T2V-1.3B-Diffusers"
-DATA_DIR="/mnt/data/mixkit_wan_1.3b_processed_t2v_distributed"
-VALIDATION_DATASET_FILE="/mnt/weka/home/hao.zhang/wl/FastVideo/examples/distill/Wan2.2-TI2V-5B-Diffusers/Data-free/validation_64.json"
-# IP=[MASTER NODE IP]
-
-# Training arguments
-training_args=(
-  --tracker_project_name "wan_tf_init"
-  --output_dir "/mnt/data/wan_ar_diffusion_3333_mixkit_shift5"
-  --wandb_run_name "wan_tf_init_3333"
-  --warp_denoising_step
-  --dmd_denoising_steps '1000,750,500,250,0'
-  --log_visualization
-  --visualization_steps 100
-  --max_train_steps 30000
-  --train_batch_size 1
-  --train_sp_batch_size 1
-  --gradient_accumulation_steps 1
-  --num_latent_t 21
-  --num_height 480
-  --num_width 832
-  --num_frames 81
-  --override-transformer-cls-name "CausalWanTransformer3DModel"
-)
-
-# Parallel arguments
-parallel_args=(
-  --num_gpus $NUM_TOTAL_GPUS # 64
-  --sp_size 1
-  --tp_size 1
-  --hsdp_replicate_dim 1 # 64
-  --hsdp_shard_dim $NUM_TOTAL_GPUS
-)
-
-# Model arguments
-model_args=(
-  --model_path $MODEL_PATH
-  --pretrained_model_name_or_path $MODEL_PATH
-  --text-encoder-cpu-offload
-)
-
-# Dataset arguments
-dataset_args=(
-  --data_path "$DATA_DIR" 
-  --dataloader_num_workers 4
-)
-
-# Optimizer arguments
-optimizer_args=(
-  --optimizer-type "adamw"
-  --learning_rate 1e-5
-  --mixed_precision "bf16"
-  --training_state_checkpointing_steps 800
-  --weight_decay 0.01
-  --max_grad_norm 1.0
-)
-
-validation_args=(
-  # --log_validation
-  --validation_dataset_file "$VALIDATION_DATASET_FILE"
-  --validation_steps 10
-  --validation_sampling_steps "4"
-  --validation_guidance_scale "1.0" # not used for dmd inference
-  # --vae_cpu_offload True
-)
-
-# Miscellaneous arguments
-miscellaneous_args=(
-  --inference_mode False
-  --checkpoints_total_limit 6
-  --training_cfg_rate 0.1
-  --multi_phased_distill_schedule "4000-1"
-  --not_apply_cfg_solver
-  --dit_precision "fp32"
-  --num_euler_timesteps 50
-  --ema_start_step 0
-  --enable_gradient_checkpointing_type "full"
-  --use-tf True
-)
-
-# If you do not have 32 GPUs and to fit in memory, you can: 1. increase sp_size. 2. reduce num_latent_t
-srun torchrun \
---nnodes $SLURM_JOB_NUM_NODES \
---nproc_per_node $NUM_GPUS \
---node_rank $SLURM_PROCID \
---rdzv_backend=c10d \
---rdzv_endpoint="$MASTER_ADDR:$MASTER_PORT" \
-    fastvideo/training/ar_diffusion_pipeline.py \
-    "${parallel_args[@]}" \
-    "${model_args[@]}" \
-    "${dataset_args[@]}" \
-    "${training_args[@]}" \
-    "${optimizer_args[@]}" \
-    "${validation_args[@]}" \
-    "${miscellaneous_args[@]}"

examples/training/consistency_finetune/causal_ode_init/wan_tf_ode_init.slurm

@@ -1,138 +0,0 @@
-#!/bin/bash
-#SBATCH --job-name=wan_tf_ode_init_3333
-#SBATCH --nodes=8
-#SBATCH --ntasks=8
-#SBATCH --ntasks-per-node=1
-#SBATCH --gres=gpu:8
-#SBATCH --cpus-per-task=128
-#SBATCH --mem=1440G
-#SBATCH --output=wan_tf_ode_init_3333_output/wan_tf_ode_init_3333_%j.out
-#SBATCH --error=wan_tf_ode_init_3333_output/wan_tf_ode_init_3333_%j.err
-#SBATCH --exclusive
-
-# Basic Info
-export NCCL_P2P_DISABLE=1
-export TORCH_NCCL_ENABLE_MONITORING=0
-export NCCL_DEBUG_SUBSYS=INIT,NET
-# different cache dir for different processes
-# export TRITON_CACHE_DIR=/tmp/triton_cache_${SLURM_PROCID}
-export MASTER_PORT=29500
-export NODE_RANK=$SLURM_PROCID
-nodes=( $(scontrol show hostnames $SLURM_JOB_NODELIST) )
-export MASTER_ADDR=${nodes[0]}
-export TOKENIZERS_PARALLELISM=false
-# export WANDB_API_KEY="8d9f4b39abd68eb4e29f6fc010b7ee71a2207cde"
-export WANDB_API_KEY="50632ebd88ffd970521cec9ab4a1a2d7e85bfc45"
-# export WANDB_API_KEY='your_wandb_api_key_here'
-export WANDB_BASE_URL="https://api.wandb.ai"
-export WANDB_MODE=online
-export FASTVIDEO_ATTENTION_BACKEND=FLASH_ATTN
-
-export HF_HOME=/mnt/data/huggingface
-export CUDA_HOME=/usr/local/cuda
-export PATH="$CUDA_HOME/bin:$PATH"
-export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/home/shared-bin/local/usr/lib/aarch64-linux-gnu:$CUDA_HOME/lib64"
-export CUDACXX="$CUDA_HOME/bin/nvcc"
-
-source ~/miniconda3/bin/activate
-conda activate fastvideo
-
-# Configs
-NUM_GPUS=8
-NUM_TOTAL_GPUS=64
-
-MODEL_PATH="wlsaidhi/SFWan2.1-T2V-1.3B-Diffusers"
-DATA_DIR="/mnt/data/fv-tf-ode-preprocessing-6k-mixkit-wan-1.3b/"
-VALIDATION_DATASET_FILE="/mnt/weka/home/hao.zhang/wl/FastVideo/examples/distill/Wan2.2-TI2V-5B-Diffusers/Data-free/validation_64.json"
-# IP=[MASTER NODE IP]
-
-# Training arguments
-training_args=(
-  --tracker_project_name "wan_tf_ode_init"
-  --output_dir "/mnt/data/wan_tf_ode_init_3333_mixkit_shift5"
-  --wandb_run_name "wan_tf_ode_init_3333_mixkit_shift5"
-  --warp_denoising_step
-  --dmd_denoising_steps '1000,750,500,250'
-  --log_visualization
-  --visualization_steps 100
-  --max_train_steps 30000
-  --train_batch_size 1
-  --train_sp_batch_size 1
-  --gradient_accumulation_steps 1
-  --num_latent_t 21
-  --num_height 480
-  --num_width 832
-  --num_frames 81
-  --enable_gradient_checkpointing_type "full"
-  --init_weights_from_safetensors "/mnt/data/wan_ar_diffusion_3333_mixkit_shift5/checkpoint-2400/transformer/diffusion_pytorch_model.safetensors"
-)
-
-# Parallel arguments
-parallel_args=(
-  --num_gpus $NUM_TOTAL_GPUS # 64
-  --sp_size 1
-  --tp_size 1
-  --hsdp_replicate_dim 1 # 64
-  --hsdp_shard_dim $NUM_TOTAL_GPUS
-)
-
-# Model arguments
-model_args=(
-  --model_path $MODEL_PATH
-  --pretrained_model_name_or_path $MODEL_PATH
-  --text-encoder-cpu-offload
-)
-
-# Dataset arguments
-dataset_args=(
-  --data_path "$DATA_DIR" 
-  --dataloader_num_workers 4
-)
-
-# Optimizer arguments
-optimizer_args=(
-  --optimizer-type "adamw"
-  --learning_rate 1e-5
-  --mixed_precision "bf16"
-  --training_state_checkpointing_steps 500
-  --weight_decay 0.01
-  --max_grad_norm 1.0
-)
-
-validation_args=(
-  # --log_validation
-  --validation_dataset_file "$VALIDATION_DATASET_FILE"
-  --validation_steps 10
-  --validation_sampling_steps "4"
-  --validation_guidance_scale "1.0" # not used for dmd inference
-  # --vae_cpu_offload True
-)
-
-# Miscellaneous arguments
-miscellaneous_args=(
-  --inference_mode False
-  --checkpoints_total_limit 6
-  --training_cfg_rate 0.1
-  --multi_phased_distill_schedule "4000-1"
-  --not_apply_cfg_solver
-  --dit_precision "fp32"
-  --num_euler_timesteps 50
-  --ema_start_step 0
-  # --enable_gradient_checkpointing_type "full"
-)
-
-# If you do not have 32 GPUs and to fit in memory, you can: 1. increase sp_size. 2. reduce num_latent_t
-srun torchrun \
---nnodes $SLURM_JOB_NUM_NODES \
---nproc_per_node $NUM_GPUS \
---node_rank $SLURM_PROCID \
---rdzv_backend=c10d \
---rdzv_endpoint="$MASTER_ADDR:$MASTER_PORT" \
-    fastvideo/training/ode_causal_pipeline.py \
-    "${parallel_args[@]}" \
-    "${model_args[@]}" \
-    "${dataset_args[@]}" \
-    "${training_args[@]}" \
-    "${optimizer_args[@]}" \
-    "${validation_args[@]}" \
-    "${miscellaneous_args[@]}"

examples/training/finetune/Wan2.1-VSA/Wan-Syn-Data/I2V-14B-VSA.slurm

@@ -44,8 +44,8 @@ VALIDATION_DATASET_FILE=your_validation_dataset_file
 # export CUDA_VISIBLE_DEVICES=4,5
 # IP=[MASTER NODE IP]
 
-# Training arguments
-training_args=(
+# Core training arguments
+core_training_args=(
   --tracker_project_name wan_i2v_VSA
   --output_dir "checkpoints/wan_i2v_finetune_VSA"
   --max_train_steps 4000
@@ -53,10 +53,14 @@ training_args=(
   --train_sp_batch_size 1
   --gradient_accumulation_steps 1
   --num_latent_t 21
+  --enable_gradient_checkpointing_type "full"
+)
+
+# Validation generation shape arguments (used during training validation)
+validation_generation_shape_args=(
   --num_height 720
   --num_width 1280
   --num_frames 81
-  --enable_gradient_checkpointing_type "full"
 )
 
 # Parallel arguments
@@ -127,8 +131,9 @@ srun torchrun \
     "${parallel_args[@]}" \
     "${model_args[@]}" \
     "${dataset_args[@]}" \
-    "${training_args[@]}" \
+    "${core_training_args[@]}" \
+    "${validation_generation_shape_args[@]}" \
     "${optimizer_args[@]}" \
     "${validation_args[@]}" \
     "${miscellaneous_args[@]}" \
-    "${vsa_args[@]}"  
+    "${vsa_args[@]}"