diff --git a/docs/inference/open_h3_regenerate_2k.md b/docs/inference/open_h3_regenerate_2k.md new file mode 100644 index 000000000..d741e1a86 --- /dev/null +++ b/docs/inference/open_h3_regenerate_2k.md @@ -0,0 +1,133 @@ +# Open-H3-Regenerate-2K + +`OpenH3Regenerate2KPipeline` regenerates a 768P [MiniMax-H3](https://huggingface.co/MiniMaxAI/MiniMax-H3) draft video +(1344x768, 121 frames, 24 fps) at **2560x1440** with three refine steps. It is video to video: the draft is the input, +and the output is a silent 2K MP4. FastVideo ports NVlabs' +[Open-H3-Regenerate-2K](https://huggingface.co/Efficient-Large-Model/Open-H3-Regenerate-2K) +([project page](https://nvlabs.github.io/Sana/Open-H3-Regenerate-2K/), +[reference code](https://github.com/NVlabs/Sana/tree/sol-engine/models/Open-H3-Regenerate-2K)); the dense BF16 +`standard_fwd` path of that repo is the parity reference. The accelerated `sol_engine_fwd` path (Sol-Attn, MXFP8, CUDA +graphs) and the fast 24-layer VAE decoder are not ported. + +```text +draft video -> H3 VAE encode -> latent upscaler (2x, 3D conv) -> 3 Euler refine steps -> H3 VAE decode -> 2560x1440 + H3 DiT + merged LoRA + Sharp curve AdaLN branches +``` + +## What runs + +| Step | Details | +|---|---| +| Draft encode | `ffmpeg` resamples to 24 fps, cover-scales (bilinear) and centre-crops to 1344x768 (`draft_size: 768p`, or 1024x576 for `576p`), holds the last frame to 121 frames plus 3 copies. The FP32 H3 VAE posterior is sampled under seed 42. | +| Latent upscale | The author's 3D-conv upscaler (`fastvideo/models/upsamplers/minimax_h3_latent_upscaler.py`), BF16, to the 160x90 latent grid. | +| Refine | Euler on sigmas `[0.909375, 0.857143, 0.666667, 0]`, CFG 1, model time `1 - sigma`, initial state `(1 - 0.909375) * upscaled + 0.909375 * noise`. `num_inference_steps: 2` uses `[0.7, 0.494118, 0]` (softer, about 30% faster). | +| Transformer | The base MiniMax-H3 DiT (the same module as the other H3 pipelines) with the released merged LoRA (FP32, rank 576 on 312 linears, scale 1) and 50 LMS Sharp curve AdaLN branches. | +| Decode | The released H3 VAE; the output is trimmed to 121 frames. | + +The merged LoRA loads through the standard H3 community-LoRA path as a constructor-time adapter: the pipeline sets +`lora_path` to `/lora/open_h3_regenerate_2k_lora.safetensors` when you do not pass one. The Sharp branches take an +8-D timestep curve (`lora/adaln_t_table.bin`, 1025x8) instead of the layer input, so they cannot be folded into weights; +`sharp_curve.py` adds them to the AdaLN projection outputs with forward hooks. + +**Audio.** The reference feeds an encoded-silence audio stream to the DiT as conditioning and writes the video with no +audio track (`-an`); it neither regenerates nor passes through the draft's audio. FastVideo does the same: the draft's +soundtrack is ignored and the output MP4 has no audio. + +## Weights + +The release repo keeps the unmodified MiniMax-H3 components (a copy of `MiniMaxAI/MiniMax-H3` at `6818f6c`) under +`base_model/` next to `lora/` and `latent_upsampler/`. Point `model_path` at the Hub id or at a local copy of that layout. +`adaln_t_table.bin` is base-derived and is not redistributed in the release repo: put the 32,800-byte file (SHA-256 +`c02a6c11888297688c1e6278185ea1f947023acfc69f9003bbcdcec9a229a8e7`) at `lora/adaln_t_table.bin`. The reference +`download.py` fetches it by byte range from `Comfy-Org/MiniMax-H3` at revision +`e5eb578a89295337b8ff433a035929ce0279e0b6` (tensor `adaln_t_table` of +`diffusion_models/minimax_h3_ref2va_pruned_int8_convrot.safetensors`) and verifies the hash. + +A GB200/B200-class GPU is needed (about 150 GB with the DiT, the 2K activations and the VAE). + +## Usage + +```bash +python examples/inference/basic/basic_open_h3_regenerate_2k.py \ + --model-path Efficient-Large-Model/Open-H3-Regenerate-2K \ + --video draft_768p.mp4 --prompt "A red fox trots through fresh snow" --output outputs/regen2k +``` + +The prompt conditions the refine steps (use the draft's own prompt). Drafts from other video models are better encoded at +576P (`--draft-size 576p`). `--steps 2` runs the two-step variant. The output is always 2560x1440; the first 121 frames of +the draft are used. + +## Serving + +```bash +fastvideo serve --config examples/serving/openai_open_h3_regenerate_2k.yaml +``` + +`examples/serving/openai_open_h3_regenerate_2k.yaml` loads the release on one GPU, sets `media_roots`, and pins the 2K +defaults. The draft arrives as the typed `video_reference` of `POST /v1/videos/sync` (the response body is the MP4) or of +`POST /v1/videos` (an async job; poll `GET /v1/videos/{id}` and download `GET /v1/videos/{id}/content`): + +```bash +curl -sS -X POST http://localhost:8000/v1/videos/sync \ + -H 'Content-Type: application/json' \ + -d '{ + "prompt": "A red fox trots through fresh snow", + "video_reference": {"video_url": "https://example.com/draft_768p.mp4"}, + "seed": 1766580932 + }' \ + -o regen_2k.mp4 -D - +``` + +The response headers carry `X-Request-Id`, `X-Model`, `X-Inference-Time-S` and `X-Stage-Durations`. Rules: + +- Exactly one `video_reference` (or the legacy `video_path` / `video_url`) and no image or audio references. +- `http(s)` URLs must resolve to public addresses unless `server.allow_private_media_urls` is set. A local path is accepted + only inside a directory listed in `server.media_roots`. `data:` URLs are refused for video references; this policy is + unchanged. +- `size`, `width` and `height` may be omitted or `2560x1440`; `num_frames` may be omitted or 124 (the frame count of the + VAE grid; the video has 121 frames); `fps` must be 24. `seconds` is not accepted. +- `num_inference_steps` is 3 (default) or 2; `seed` sets the refine noise. + +## Parity with the reference + +Measured on one GB200 (tray hpc-rack-1-5, BF16, dense attention) against the reference `standard_fwd` with the released +VAE, same draft, seed (1766580932), prompt and VAE posterior seed (42). The reference's noise is a CUDA generator draw +that FastVideo reproduces (the same `torch.Generator(device="cuda")` seed), so no injected noise was needed. + +| Stage / quantity | Fox draft (FastVideo H3 t2v, 768P) | Dance draft (576P, project page) | +|---|---|---| +| Draft latent, upscaled latent, DiT input | bit-identical | bit-identical | +| One DiT forward (LoRA + Sharp), relative L2 | 1.1e-2 | 2.6e-2 | +| Refined latent, relative L2 | 1.3e-2 | 3.7e-2 | +| Decoded video PSNR / SSIM (21 frames) | 45.2 dB / 0.993 | 42.1 dB / 0.992 | + +The differences come from text-encoder rounding (relative L2 1e-3 on the prompt embeddings) and BF16 kernel choices; +the reference reports the same order (1.4% relative L2 on the refined latent between its own runs). + +Warm timings, one GB200 (seconds): + +| | Upscale | Refine (3 steps) | VAE decode | Total | +|---|---:|---:|---:|---:| +| Reference `standard_fwd` | 0.87 | 71.6 | 15.0 | 87.5 | +| FastVideo (end to end, incl. draft encode 5-10 s and decode to host) | 5.9-10.7 (encode + upscale + silence) | 69.5 | 25.0 | 107.5 | + +The first request builds nothing extra but loads weights (about 70 s once). The decode stage is slower than the +reference because it also copies 121 frames to host memory and the MP4 encode is outside both columns. + +## Licences + +The release is not covered by one licence; its `README.md` says so, and the runtime code's licence does not replace the +model licences. FastVideo only loads these weights and does not redistribute them. + +- The refiner LoRA, the fast H3 VAE decoder and `base_model/` (a copy of `MiniMaxAI/MiniMax-H3` at `6818f6c`: DiT, VAEs, + text encoder) follow the + [MiniMax H3 Community License](https://huggingface.co/MiniMaxAI/MiniMax-H3/blob/6818f6c32d12b210915e44ad56a4228c2608f160/LICENSE), + including its territory, use and redistribution terms (`base_model/LICENSE`, `NOTICE`). The Qwen3-VL text encoder's + Apache-2.0 text ships as `LICENSE-Qwen.txt`. +- The latent upscaler follows its own licence: + [LBH-123-AI/Comfyui_Minimax_h3_latent_Upscaler](https://github.com/LBH-123-AI/Comfyui_Minimax_h3_latent_Upscaler) + (MIT, per the reference code) and its [weights](https://huggingface.co/LBH-123-AI/Minimax_h3_latent_Upscaler) + (`latent_upsampler/LICENSE`, Apache-2.0). `minimax_h3_latent_upscaler.py` is a port of that node's network. +- The merged LoRA includes [LightX2V FL2VA Turbo](https://huggingface.co/lightx2v/Minimax-h3-Turbo) and + [LMS Sharp](https://huggingface.co/Alissonerdx/Minimax-H3-ComfyUI), which keep their own licences; the Sharp + `adaln_t_table` is read from [Comfy-Org/MiniMax-H3](https://huggingface.co/Comfy-Org/MiniMax-H3). diff --git a/docs/inference/support_matrix.md b/docs/inference/support_matrix.md index 722e5cf28..7e1df1cdd 100644 --- a/docs/inference/support_matrix.md +++ b/docs/inference/support_matrix.md @@ -68,6 +68,7 @@ column links a runnable script in `examples/inference/basic/` where one exists. | matrixgame | `FastVideo/Matrix-Game-2.0-Base-Distilled-Diffusers`
`FastVideo/Matrix-Game-2.0-GTA-Distilled-Diffusers`
`FastVideo/Matrix-Game-2.0-TempleRun-Distilled-Diffusers`
`FastVideo/Matrix-Game-2.0-Base-Diffusers`
`FastVideo/Matrix-Game-2.0-GTA-Diffusers`
`FastVideo/Matrix-Game-2.0-TempleRun-Diffusers`
`mignonjia/mg_longtuning_distilled_zelda`
`mignonjia/mg_sf_distilled_zelda_1k_steps`
`mignonjia/mg_sf_distilled_zelda`
`mignonjia/mg_causal_zelda`
`mignonjia/mg_bidirectional_zelda` | I2V | [basic_matrixgame2.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_matrixgame2.py) | | matrixgame | `FastVideo/Matrix-Game-3.0-Base-Distilled-Diffusers` | I2V | [basic_matrixgame3.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_matrixgame3.py) | | minimax_h3 | `MiniMaxAI/MiniMax-H3` | T2V, I2V | [T2VA](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_minimax_h3_t2v.py)
[FL2VA](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_minimax_h3_fl2va.py)
[Ref2VA](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_minimax_h3_ref2va.py) | +| open_h3_regenerate_2k | `Efficient-Large-Model/Open-H3-Regenerate-2K` | — | [basic_open_h3_regenerate_2k.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_open_h3_regenerate_2k.py) | | sd35 | `stabilityai/stable-diffusion-3.5-medium` | T2I | [basic_sd35_t2i.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_sd35_t2i.py) | | qwen_image21 | `Qwen/Qwen-Image-2.1` | T2I, I2I | [basic_qwen_image21.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_qwen_image21.py) | | stable_audio | `FastVideo/stable-audio-open-1.0-Diffusers` | T2V | [basic_stable_audio.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_stable_audio.py) | @@ -121,6 +122,9 @@ and distilled) generate video with audio from text, optionally plus an image (se `kandinsky6_sr` IDs upscale an existing video (see the [Video SR guide](kandinsky6_sr.md)). +**Note (Open-H3-Regenerate-2K)**: `open_h3_regenerate_2k` regenerates an existing 768P MiniMax-H3 draft video at +2560x1440 (video to video, silent output); see the [Open-H3-Regenerate-2K guide](open_h3_regenerate_2k.md). + **Note (LTX-2.5)**: T2V and I2V produce synchronized video and audio. The official `Lightricks/LTX-2.5` repository uses gated, split component files, so it must first be converted to FastVideo's component layout. See the