nodes/hao-ai-lab-FastVideo
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

[docs]: Open-H3-Regenerate-2K guide and support matrix entry

Browse Source
This commit is contained in:
Aryan Kumar
2026-10-29 00:27:33 -07:00
parent 4519f8c488
commit 63ef3435d4
2 changed files with 137 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/inference/open_h3_regenerate_2k.md
+133
View File
@@ -0,0 +1,133 @@
# Open-H3-Regenerate-2K
`OpenH3Regenerate2KPipeline` regenerates a 768P [MiniMax-H3](https://huggingface.co/MiniMaxAI/MiniMax-H3) draft video
(1344x768, 121 frames, 24 fps) at **2560x1440** with three refine steps. It is video to video: the draft is the input,
and the output is a silent 2K MP4. FastVideo ports NVlabs'
[Open-H3-Regenerate-2K](https://huggingface.co/Efficient-Large-Model/Open-H3-Regenerate-2K)
([project page](https://nvlabs.github.io/Sana/Open-H3-Regenerate-2K/),
[reference code](https://github.com/NVlabs/Sana/tree/sol-engine/models/Open-H3-Regenerate-2K)); the dense BF16
`standard_fwd` path of that repo is the parity reference. The accelerated `sol_engine_fwd` path (Sol-Attn, MXFP8, CUDA
graphs) and the fast 24-layer VAE decoder are not ported.
```text
draft video -> H3 VAE encode -> latent upscaler (2x, 3D conv) -> 3 Euler refine steps -> H3 VAE decode -> 2560x1440
H3 DiT + merged LoRA + Sharp curve AdaLN branches
```
## What runs
| Step | Details |
|---|---|
| Draft encode | `ffmpeg` resamples to 24 fps, cover-scales (bilinear) and centre-crops to 1344x768 (`draft_size: 768p`, or 1024x576 for `576p`), holds the last frame to 121 frames plus 3 copies. The FP32 H3 VAE posterior is sampled under seed 42. |
| Latent upscale | The author's 3D-conv upscaler (`fastvideo/models/upsamplers/minimax_h3_latent_upscaler.py`), BF16, to the 160x90 latent grid. |
| Refine | Euler on sigmas `[0.909375, 0.857143, 0.666667, 0]`, CFG 1, model time `1 - sigma`, initial state `(1 - 0.909375) * upscaled + 0.909375 * noise`. `num_inference_steps: 2` uses `[0.7, 0.494118, 0]` (softer, about 30% faster). |
| Transformer | The base MiniMax-H3 DiT (the same module as the other H3 pipelines) with the released merged LoRA (FP32, rank 576 on 312 linears, scale 1) and 50 LMS Sharp curve AdaLN branches. |
| Decode | The released H3 VAE; the output is trimmed to 121 frames. |
The merged LoRA loads through the standard H3 community-LoRA path as a constructor-time adapter: the pipeline sets
`lora_path` to `<model>/lora/open_h3_regenerate_2k_lora.safetensors` when you do not pass one. The Sharp branches take an
8-D timestep curve (`lora/adaln_t_table.bin`, 1025x8) instead of the layer input, so they cannot be folded into weights;
`sharp_curve.py` adds them to the AdaLN projection outputs with forward hooks.
**Audio.** The reference feeds an encoded-silence audio stream to the DiT as conditioning and writes the video with no
audio track (`-an`); it neither regenerates nor passes through the draft's audio. FastVideo does the same: the draft's
soundtrack is ignored and the output MP4 has no audio.
## Weights
The release repo keeps the unmodified MiniMax-H3 components (a copy of `MiniMaxAI/MiniMax-H3` at `6818f6c`) under
`base_model/` next to `lora/` and `latent_upsampler/`. Point `model_path` at the Hub id or at a local copy of that layout.
`adaln_t_table.bin` is base-derived and is not redistributed in the release repo: put the 32,800-byte file (SHA-256
`c02a6c11888297688c1e6278185ea1f947023acfc69f9003bbcdcec9a229a8e7`) at `lora/adaln_t_table.bin`. The reference
`download.py` fetches it by byte range from `Comfy-Org/MiniMax-H3` at revision
`e5eb578a89295337b8ff433a035929ce0279e0b6` (tensor `adaln_t_table` of
`diffusion_models/minimax_h3_ref2va_pruned_int8_convrot.safetensors`) and verifies the hash.
A GB200/B200-class GPU is needed (about 150 GB with the DiT, the 2K activations and the VAE).
## Usage
```bash
python examples/inference/basic/basic_open_h3_regenerate_2k.py \
--model-path Efficient-Large-Model/Open-H3-Regenerate-2K \
--video draft_768p.mp4 --prompt "A red fox trots through fresh snow" --output outputs/regen2k
```
The prompt conditions the refine steps (use the draft's own prompt). Drafts from other video models are better encoded at
576P (`--draft-size 576p`). `--steps 2` runs the two-step variant. The output is always 2560x1440; the first 121 frames of
the draft are used.
## Serving
```bash
fastvideo serve --config examples/serving/openai_open_h3_regenerate_2k.yaml
```
`examples/serving/openai_open_h3_regenerate_2k.yaml` loads the release on one GPU, sets `media_roots`, and pins the 2K
defaults. The draft arrives as the typed `video_reference` of `POST /v1/videos/sync` (the response body is the MP4) or of
`POST /v1/videos` (an async job; poll `GET /v1/videos/{id}` and download `GET /v1/videos/{id}/content`):
```bash
curl -sS -X POST http://localhost:8000/v1/videos/sync \
-H 'Content-Type: application/json' \
-d '{
"prompt": "A red fox trots through fresh snow",
"video_reference": {"video_url": "https://example.com/draft_768p.mp4"},
"seed": 1766580932
}' \
-o regen_2k.mp4 -D -
```
The response headers carry `X-Request-Id`, `X-Model`, `X-Inference-Time-S` and `X-Stage-Durations`. Rules:
- Exactly one `video_reference` (or the legacy `video_path` / `video_url`) and no image or audio references.
- `http(s)` URLs must resolve to public addresses unless `server.allow_private_media_urls` is set. A local path is accepted
only inside a directory listed in `server.media_roots`. `data:` URLs are refused for video references; this policy is
unchanged.
- `size`, `width` and `height` may be omitted or `2560x1440`; `num_frames` may be omitted or 124 (the frame count of the
VAE grid; the video has 121 frames); `fps` must be 24. `seconds` is not accepted.
- `num_inference_steps` is 3 (default) or 2; `seed` sets the refine noise.
## Parity with the reference
Measured on one GB200 (tray hpc-rack-1-5, BF16, dense attention) against the reference `standard_fwd` with the released
VAE, same draft, seed (1766580932), prompt and VAE posterior seed (42). The reference's noise is a CUDA generator draw
that FastVideo reproduces (the same `torch.Generator(device="cuda")` seed), so no injected noise was needed.
| Stage / quantity | Fox draft (FastVideo H3 t2v, 768P) | Dance draft (576P, project page) |
|---|---|---|
| Draft latent, upscaled latent, DiT input | bit-identical | bit-identical |
| One DiT forward (LoRA + Sharp), relative L2 | 1.1e-2 | 2.6e-2 |
| Refined latent, relative L2 | 1.3e-2 | 3.7e-2 |
| Decoded video PSNR / SSIM (21 frames) | 45.2 dB / 0.993 | 42.1 dB / 0.992 |
The differences come from text-encoder rounding (relative L2 1e-3 on the prompt embeddings) and BF16 kernel choices;
the reference reports the same order (1.4% relative L2 on the refined latent between its own runs).
Warm timings, one GB200 (seconds):
| | Upscale | Refine (3 steps) | VAE decode | Total |
|---|---:|---:|---:|---:|
| Reference `standard_fwd` | 0.87 | 71.6 | 15.0 | 87.5 |
| FastVideo (end to end, incl. draft encode 5-10 s and decode to host) | 5.9-10.7 (encode + upscale + silence) | 69.5 | 25.0 | 107.5 |
The first request builds nothing extra but loads weights (about 70 s once). The decode stage is slower than the
reference because it also copies 121 frames to host memory and the MP4 encode is outside both columns.
## Licences
The release is not covered by one licence; its `README.md` says so, and the runtime code's licence does not replace the
model licences. FastVideo only loads these weights and does not redistribute them.
- The refiner LoRA, the fast H3 VAE decoder and `base_model/` (a copy of `MiniMaxAI/MiniMax-H3` at `6818f6c`: DiT, VAEs,
text encoder) follow the
[MiniMax H3 Community License](https://huggingface.co/MiniMaxAI/MiniMax-H3/blob/6818f6c32d12b210915e44ad56a4228c2608f160/LICENSE),
including its territory, use and redistribution terms (`base_model/LICENSE`, `NOTICE`). The Qwen3-VL text encoder's
Apache-2.0 text ships as `LICENSE-Qwen.txt`.
- The latent upscaler follows its own licence:
[LBH-123-AI/Comfyui_Minimax_h3_latent_Upscaler](https://github.com/LBH-123-AI/Comfyui_Minimax_h3_latent_Upscaler)
(MIT, per the reference code) and its [weights](https://huggingface.co/LBH-123-AI/Minimax_h3_latent_Upscaler)
(`latent_upsampler/LICENSE`, Apache-2.0). `minimax_h3_latent_upscaler.py` is a port of that node's network.
- The merged LoRA includes [LightX2V FL2VA Turbo](https://huggingface.co/lightx2v/Minimax-h3-Turbo) and
[LMS Sharp](https://huggingface.co/Alissonerdx/Minimax-H3-ComfyUI), which keep their own licences; the Sharp
`adaln_t_table` is read from [Comfy-Org/MiniMax-H3](https://huggingface.co/Comfy-Org/MiniMax-H3).
docs/inference/support_matrix.md
+4
View File
@@ -68,6 +68,7 @@ column links a runnable script in `examples/inference/basic/` where one exists.
| matrixgame | `FastVideo/Matrix-Game-2.0-Base-Distilled-Diffusers`<br />`FastVideo/Matrix-Game-2.0-GTA-Distilled-Diffusers`<br />`FastVideo/Matrix-Game-2.0-TempleRun-Distilled-Diffusers`<br />`FastVideo/Matrix-Game-2.0-Base-Diffusers`<br />`FastVideo/Matrix-Game-2.0-GTA-Diffusers`<br />`FastVideo/Matrix-Game-2.0-TempleRun-Diffusers`<br />`mignonjia/mg_longtuning_distilled_zelda`<br />`mignonjia/mg_sf_distilled_zelda_1k_steps`<br />`mignonjia/mg_sf_distilled_zelda`<br />`mignonjia/mg_causal_zelda`<br />`mignonjia/mg_bidirectional_zelda` | I2V | [basic_matrixgame2.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_matrixgame2.py) |
| matrixgame | `FastVideo/Matrix-Game-3.0-Base-Distilled-Diffusers` | I2V | [basic_matrixgame3.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_matrixgame3.py) |
| minimax_h3 | `MiniMaxAI/MiniMax-H3` | T2V, I2V | [T2VA](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_minimax_h3_t2v.py)<br />[FL2VA](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_minimax_h3_fl2va.py)<br />[Ref2VA](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_minimax_h3_ref2va.py) |
| open_h3_regenerate_2k | `Efficient-Large-Model/Open-H3-Regenerate-2K` | — | [basic_open_h3_regenerate_2k.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_open_h3_regenerate_2k.py) |
| sd35 | `stabilityai/stable-diffusion-3.5-medium` | T2I | [basic_sd35_t2i.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_sd35_t2i.py) |
| qwen_image21 | `Qwen/Qwen-Image-2.1` | T2I, I2I | [basic_qwen_image21.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_qwen_image21.py) |
| stable_audio | `FastVideo/stable-audio-open-1.0-Diffusers` | T2V | [basic_stable_audio.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_stable_audio.py) |
@@ -121,6 +122,9 @@ and distilled) generate video with audio from text, optionally plus an image (se
`kandinsky6_sr` IDs upscale an existing video (see the
[Video SR guide](kandinsky6_sr.md)).
**Note (Open-H3-Regenerate-2K)**: `open_h3_regenerate_2k` regenerates an existing 768P MiniMax-H3 draft video at
2560x1440 (video to video, silent output); see the [Open-H3-Regenerate-2K guide](open_h3_regenerate_2k.md).
**Note (LTX-2.5)**: T2V and I2V produce synchronized video and audio. The
official `Lightricks/LTX-2.5` repository uses gated, split component files, so
it must first be converted to FastVideo's component layout. See the