Merge hao-ai-lab/FastVideo main into alex/openai-data-url-refs
Conflict in request_adapter.py with #2001, which adds continuation_video and materializes it through _materialize_video. This branch replaced that helper with materialize_clip, so continuation_video now goes through materialize_clip as well, with the same "continuation" upload name. As a result it gets the same policy as the other video fields: media_roots, the private-URL check, the size cap, and base64 data:video/... URLs. - docs/design/server_contracts/openai.md: continuation_video is listed in the video row of the media-source table. - docs/inference/fasth3_omniref.md: continuation_video may be a data:video URL. - test_openai_serving_engine.py: an inline continuation video is decoded, tracked and removed, and a data:audio one is refused. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -628,6 +628,7 @@ surfaces:
|
||||
audio_reference: request.inputs.references
|
||||
video_path: request.inputs.video_path
|
||||
video_url: request.inputs.video_path
|
||||
continuation_video: request.inputs.references
|
||||
video_params: request.sampling.width,height,num_frames,fps
|
||||
size:
|
||||
target: request.sampling.width,height
|
||||
|
||||
@@ -94,7 +94,7 @@ Reference objects carry one string each through `image_url`, `video_url`, and
|
||||
| Field | `http(s)` URL | `data:` URL | Local path |
|
||||
| --- | --- | --- | --- |
|
||||
| `image_reference[].image_url`, `input_reference`, `reference_url` | yes | `data:image/...;base64,` | under `server.media_roots` |
|
||||
| `video_reference[].video_url`, `video_path`, `video_url` | yes | `data:video/...;base64,` | under `server.media_roots` |
|
||||
| `video_reference[].video_url`, `video_path`, `video_url`, `continuation_video` | yes | `data:video/...;base64,` | under `server.media_roots` |
|
||||
| `audio_reference[].audio_url` | yes | `data:audio/...;base64,` | under `server.media_roots` |
|
||||
|
||||
`input_reference` also accepts a multipart upload. `file_id` references are
|
||||
|
||||
@@ -26,3 +26,39 @@ to receive the decoded CPU tensor in `GenerationResult.samples` without building
|
||||
preview grids or writing media. Pixel samples retain the existing float `[0, 1]`
|
||||
contract. `return_frames=True` continues to return both samples and preview frames.
|
||||
The legacy sampling/keyword interface accepts the same `return_samples` option.
|
||||
|
||||
### Choosing an output filename
|
||||
|
||||
Set `request.output.output_video_name` to choose a filename when
|
||||
`request.output.output_path` is a directory:
|
||||
|
||||
```python
|
||||
result = generator.generate({
|
||||
"prompt": "A waterfall in the mountains",
|
||||
"output": {"output_path": "outputs/", "output_video_name": "my_clip"},
|
||||
})
|
||||
# Saves outputs/my_clip.mp4 when that file does not already exist.
|
||||
```
|
||||
|
||||
An explicit file path such as `outputs/custom.mp4` takes precedence over
|
||||
`output_video_name`. Without a name, the filename is derived from the prompt
|
||||
(or the input video for prompt-less super-resolution pipelines). Names are
|
||||
sanitized, and existing files are preserved by adding numeric suffixes such as
|
||||
`my_clip_1.mp4`, including when processing multiple prompts. A matching extension
|
||||
in the name is not duplicated. Image workloads accept PNG, JPEG, or WebP names
|
||||
and default to PNG; audio workloads use WAV.
|
||||
|
||||
For requests that save media, the sanitized filename is checked against the
|
||||
destination filesystem's filename byte limit, including its extension and any
|
||||
collision suffix. If the limit cannot be determined, 255 bytes is used.
|
||||
Overlong names are rejected before model execution; use a shorter name rather
|
||||
than relying on automatic truncation. Multibyte characters may use more than
|
||||
one byte each. Prompt-file batches retain their existing per-prompt error
|
||||
handling.
|
||||
|
||||
Requests with `save_video=False` or `output_type="latent"` do not prepare an
|
||||
output file: filename validation, directory creation, and collision checks are
|
||||
skipped. This also applies to latent output when `save_video=True`.
|
||||
|
||||
The legacy `SamplingParam` and `generate_video` keyword interfaces accept the
|
||||
same `output_video_name` option.
|
||||
|
||||
@@ -148,6 +148,38 @@ Select the NVFP4 decoder per configuration with
|
||||
holds base MiniMax-H3's full 36-block VAE, so both the 36-block quality tier and
|
||||
the 8-block fast tier apply.
|
||||
|
||||
## Continuing a video
|
||||
|
||||
A Ref2VA request can continue an earlier video. It takes the earlier video's last
|
||||
5, 22 or 39 frames (and their sound, if any) as a continuation clip, and the
|
||||
generated video starts by regenerating those frames at the same positions: the
|
||||
clip's latent rows take the coordinates of the target's first latent frames and
|
||||
audio latents, the reference context of
|
||||
[ComfyUI-H3-Continuum](https://github.com/ukr8b3g-cmyk/ComfyUI-H3-Continuum).
|
||||
Drop the clip's frames, and the matching audio, from the new video, then append
|
||||
the rest to the earlier one.
|
||||
|
||||
- Over HTTP, set `continuation_video` to the clip's URL, a base64
|
||||
`data:video/...` URL or a local path (under `server.media_roots`) beside the
|
||||
usual references. In Python, append
|
||||
`MiniMaxH3Reference(source=clip, media_type="continuation")` to `references`.
|
||||
- The clip is resampled to 24 fps and must then have 5, 22 or 39 frames on the
|
||||
target canvas. A request takes at most one clip and still needs at least one
|
||||
reference; the clip does not count toward the reference limits and is not
|
||||
shown to the text encoder.
|
||||
- `num_frames` counts the clip: 22 frames plus five new seconds is 142,
|
||||
aligned up to 158.
|
||||
- Write the prompt from the clip's actual end state. When the prompt's opening
|
||||
contradicts the clip, the model tends to cut to a new shot right after it.
|
||||
- Each continuation drifts a little toward higher saturation and sharpness, so
|
||||
fewer, longer segments (two 15-second ones rather than six 5-second ones) hold
|
||||
up better.
|
||||
|
||||
Measured on a FastH3 OmniRef PDD student at 1344×768, 22-frame clips re-encoded
|
||||
from the earlier output: the jump at a join was about half that of conditioning
|
||||
the next segment on the earlier last frame as an image reference, and the sound
|
||||
continued instead of restarting from near silence.
|
||||
|
||||
## Environment
|
||||
|
||||
The profiles set these; set them yourself when driving the YAML recipes.
|
||||
|
||||
@@ -12,6 +12,17 @@ format, which stores already augmented `ref_visual_anchor` tensors. Keep using
|
||||
`preprocess_minimax_h3_ref2va` for that format. Do not rename or directly combine
|
||||
shards from these two schemas.
|
||||
|
||||
The SFT preprocessor keeps each target clip's own geometry: the canvas follows
|
||||
the clip's aspect ratio at the inference short edge (`resolve_canvas_size`), and
|
||||
the frame count is the longest `17 * n + 5` that fits the clip, capped at 362
|
||||
(about 15 seconds; a longer clip trains on its start). Clips under 124 frames at
|
||||
24 fps fail. Target audio is cut to the samples covered by that frame count's
|
||||
whole audio latents, so the record matches what inference generates. To train
|
||||
on those lengths, leave `training.data.num_latent_t` and `num_frames` unset in
|
||||
the training config. When they are set, every row is cropped to that length;
|
||||
for example, the 124-frame overfit values crop a 10-second clip to its first
|
||||
5.17 seconds while its caption still describes all 10.
|
||||
|
||||
## Input
|
||||
|
||||
Provide your own local MiniMax-H3 checkpoint with `vae`, `audio_vae`, `tokenizer`,
|
||||
|
||||
Reference in New Issue
Block a user