Merge hao-ai-lab/FastVideo main into alex/openai-data-url-refs

Conflict in request_adapter.py with #2001, which adds continuation_video and
materializes it through _materialize_video. This branch replaced that helper
with materialize_clip, so continuation_video now goes through materialize_clip
as well, with the same "continuation" upload name. As a result it gets the same
policy as the other video fields: media_roots, the private-URL check, the size
cap, and base64 data:video/... URLs.

- docs/design/server_contracts/openai.md: continuation_video is listed in the
  video row of the media-source table.
- docs/inference/fasth3_omniref.md: continuation_video may be a data:video URL.
- test_openai_serving_engine.py: an inline continuation video is decoded,
  tracked and removed, and a data:audio one is refused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
alexzms
2026-11-05 01:33:06 +00:00
co-authored by Claude Opus 5.5
29 changed files with 1488 additions and 86 deletions
@@ -628,6 +628,7 @@ surfaces:
audio_reference: request.inputs.references
video_path: request.inputs.video_path
video_url: request.inputs.video_path
continuation_video: request.inputs.references
video_params: request.sampling.width,height,num_frames,fps
size:
target: request.sampling.width,height
+1 -1
View File
@@ -94,7 +94,7 @@ Reference objects carry one string each through `image_url`, `video_url`, and
| Field | `http(s)` URL | `data:` URL | Local path |
| --- | --- | --- | --- |
| `image_reference[].image_url`, `input_reference`, `reference_url` | yes | `data:image/...;base64,` | under `server.media_roots` |
| `video_reference[].video_url`, `video_path`, `video_url` | yes | `data:video/...;base64,` | under `server.media_roots` |
| `video_reference[].video_url`, `video_path`, `video_url`, `continuation_video` | yes | `data:video/...;base64,` | under `server.media_roots` |
| `audio_reference[].audio_url` | yes | `data:audio/...;base64,` | under `server.media_roots` |
`input_reference` also accepts a multipart upload. `file_id` references are
+36
View File
@@ -26,3 +26,39 @@ to receive the decoded CPU tensor in `GenerationResult.samples` without building
preview grids or writing media. Pixel samples retain the existing float `[0, 1]`
contract. `return_frames=True` continues to return both samples and preview frames.
The legacy sampling/keyword interface accepts the same `return_samples` option.
### Choosing an output filename
Set `request.output.output_video_name` to choose a filename when
`request.output.output_path` is a directory:
```python
result = generator.generate({
"prompt": "A waterfall in the mountains",
"output": {"output_path": "outputs/", "output_video_name": "my_clip"},
})
# Saves outputs/my_clip.mp4 when that file does not already exist.
```
An explicit file path such as `outputs/custom.mp4` takes precedence over
`output_video_name`. Without a name, the filename is derived from the prompt
(or the input video for prompt-less super-resolution pipelines). Names are
sanitized, and existing files are preserved by adding numeric suffixes such as
`my_clip_1.mp4`, including when processing multiple prompts. A matching extension
in the name is not duplicated. Image workloads accept PNG, JPEG, or WebP names
and default to PNG; audio workloads use WAV.
For requests that save media, the sanitized filename is checked against the
destination filesystem's filename byte limit, including its extension and any
collision suffix. If the limit cannot be determined, 255 bytes is used.
Overlong names are rejected before model execution; use a shorter name rather
than relying on automatic truncation. Multibyte characters may use more than
one byte each. Prompt-file batches retain their existing per-prompt error
handling.
Requests with `save_video=False` or `output_type="latent"` do not prepare an
output file: filename validation, directory creation, and collision checks are
skipped. This also applies to latent output when `save_video=True`.
The legacy `SamplingParam` and `generate_video` keyword interfaces accept the
same `output_video_name` option.
+32
View File
@@ -148,6 +148,38 @@ Select the NVFP4 decoder per configuration with
holds base MiniMax-H3's full 36-block VAE, so both the 36-block quality tier and
the 8-block fast tier apply.
## Continuing a video
A Ref2VA request can continue an earlier video. It takes the earlier video's last
5, 22 or 39 frames (and their sound, if any) as a continuation clip, and the
generated video starts by regenerating those frames at the same positions: the
clip's latent rows take the coordinates of the target's first latent frames and
audio latents, the reference context of
[ComfyUI-H3-Continuum](https://github.com/ukr8b3g-cmyk/ComfyUI-H3-Continuum).
Drop the clip's frames, and the matching audio, from the new video, then append
the rest to the earlier one.
- Over HTTP, set `continuation_video` to the clip's URL, a base64
`data:video/...` URL or a local path (under `server.media_roots`) beside the
usual references. In Python, append
`MiniMaxH3Reference(source=clip, media_type="continuation")` to `references`.
- The clip is resampled to 24 fps and must then have 5, 22 or 39 frames on the
target canvas. A request takes at most one clip and still needs at least one
reference; the clip does not count toward the reference limits and is not
shown to the text encoder.
- `num_frames` counts the clip: 22 frames plus five new seconds is 142,
aligned up to 158.
- Write the prompt from the clip's actual end state. When the prompt's opening
contradicts the clip, the model tends to cut to a new shot right after it.
- Each continuation drifts a little toward higher saturation and sharpness, so
fewer, longer segments (two 15-second ones rather than six 5-second ones) hold
up better.
Measured on a FastH3 OmniRef PDD student at 1344×768, 22-frame clips re-encoded
from the earlier output: the jump at a join was about half that of conditioning
the next segment on the earlier last frame as an image reference, and the sound
continued instead of restarting from near silence.
## Environment
The profiles set these; set them yourself when driving the YAML recipes.
@@ -12,6 +12,17 @@ format, which stores already augmented `ref_visual_anchor` tensors. Keep using
`preprocess_minimax_h3_ref2va` for that format. Do not rename or directly combine
shards from these two schemas.
The SFT preprocessor keeps each target clip's own geometry: the canvas follows
the clip's aspect ratio at the inference short edge (`resolve_canvas_size`), and
the frame count is the longest `17 * n + 5` that fits the clip, capped at 362
(about 15 seconds; a longer clip trains on its start). Clips under 124 frames at
24 fps fail. Target audio is cut to the samples covered by that frame count's
whole audio latents, so the record matches what inference generates. To train
on those lengths, leave `training.data.num_latent_t` and `num_frames` unset in
the training config. When they are set, every row is cropped to that length;
for example, the 124-frame overfit values crop a 10-second clip to its first
5.17 seconds while its caption still describes all 10.
## Input
Provide your own local MiniMax-H3 checkpoint with `vae`, `audio_vae`, `tokenizer`,