Review follow-ups for the inline video/audio reference decoder:
- decode_data_url now undoes percent escapes and strips whitespace in
64 KiB chunks and stops as soon as the payload must decode to more than
server.media_reference_max_bytes. An oversized request costs memory in
proportion to the limit, not to its own size. Escapes are never split
across chunks.
- Data URL checks lower-case only the scheme instead of copying the whole
URL.
- Tests: the wrapped/percent-escaped payload decodes the same at chunk
sizes that split escapes and line breaks; an oversized payload is refused
after a few chunks; and a request cancelled before its decoding thread
writes the file still leaves no upload once the thread finishes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`POST /v1/videos` decoded `data:image/...` references but refused
`data:` video references and passed every `audio_url` to the pipeline
unchanged, so clients without a public URL could not send a reference
video or audio file (SGLang Diffusion accepts all three).
- `video_reference` / `video_path` / `video_url` accept
`data:video/...;base64,` and `audio_reference` accepts
`data:audio/...;base64,`. Each decoded file is capped by
`server.media_reference_max_bytes` (checked from the payload length
before decoding), written under `output_dir/uploads`, kept in field
order, and deleted when the job ends.
- Invalid base64, a non-base64 data URL, or a media type that does not
match the field returns HTTP 400. Percent escapes and line-wrapped
payloads decode, as in SGLang.
- Audio references are now materialized like video references: http(s)
URLs go through the media policy download (public addresses, byte
cap) and local paths must sit under `server.media_roots`. Before, the
raw string reached PyAV, which opened any local file or URL.
- `docs/design/server_contracts/openai.md` lists the accepted forms per
field.
Fixes#2018
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The unit lane fails on main in two contract tests. All three failures are
bookkeeping that drifted from the code, not runtime bugs:
- `test_env_policy.py::test_env_doc_table_matches_registry`: the generated
table in docs/contributing/env_vars.md is stale for
FASTVIDEO_MLX_QUANT_GEMM. Regenerated with the documented command
(`python fastvideo/tests/contract/test_env_policy.py`).
- `test_env_policy.py::test_env_access_follows_policy`: two new violations.
- fastvideo/tests/mlx/test_mlx_affine_dq_gemm.py wrote
FASTVIDEO_MLX_QUANT_GEMM through monkeypatch.setenv; now uses the
registered field's override() through the env_overrides fixture.
- fastvideo/tests/train/models/test_load_kandinsky5.py read
FASTVIDEO_ATTENTION_BACKEND with `not in os.environ`; now uses
envs.FASTVIDEO_ATTENTION_BACKEND.is_set() and override(). Its entry in
KNOWN_VIOLATIONS is deleted, as the shrinking-list rule requires.
- `test_schema_parity_inventory.py::..._extension_fields_are_classified`:
MiniMaxH3PipelineConfig declares video_scheduler_shift and
audio_scheduler_shift (training-time noise clocks rejected by
check_pipeline_config at inference); both are added to
pipeline_config_extensions.internal_only.
Test plan (CPU, no GPU required):
python -m pytest fastvideo/tests/contract/test_env_policy.py \
fastvideo/tests/api/test_schema_parity_inventory.py \
fastvideo/tests/mlx/test_mlx_affine_dq_gemm.py -q
# 45 passed
python -m pytest fastvideo/tests/contract/ fastvideo/tests/api/ -q -m "not gpu"
# 501 passed
VideoGenerator.from_config calls from_fastvideo_args with a log_queue keyword
argument. The StreamingVideoGenerator override did not accept it, so
StreamingVideoGenerator.from_pretrained raised TypeError. The override now
accepts log_queue and passes it to the executor through VideoGenerator.__init__.
A new test covers the call.
The HunyuanVideo batch test wrote MASTER_PORT, MASTER_ADDR and
FASTVIDEO_ATTENTION_BACKEND with monkeypatch.setenv, so the environment-variable
contract test failed. The fixture now uses envs.override_external and the
registry override. The values and the restore behavior do not change.
These speed-ups for the MiniMax-H3 pipelines keep the output bit-identical. Each
one has a switch that is off by default:
- FASTVIDEO_MINIMAX_H3_EXACT_KERNELS: Triton RoPE, AdaLN modulation, gated
residual and SwiGLU kernels that round to bf16 at the same points as the
eager ops.
- FASTVIDEO_H3_VSA_HEADS_FIRST_TILE: VSA-H3 tiles are written directly in the
heads-first layout of the kernel.
- FASTVIDEO_H3_VAE_TILE_PARALLEL: VAE spatial tiles are split across the
sequence-parallel ranks.
- FASTVIDEO_H3_REF2VA_MEMO_ENTRIES: repeat reference encodes are reused.
The example basic_fasth3_omniref_pdd.py --lossless-accel turns them all on.
On GB200 with 4 GPUs, MiniMax-H3 text to video with audio (124 frames,
1344x768) gives bit-identical frames and audio with the first three switches
on and off.
MultiprocExecutor created two streaming multiprocessing queues for every
generator, but only StreamingVideoGenerator uses them. Each queue uses POSIX
named semaphores in /dev/shm. When a job epilog deleted these files, spawned
workers failed to start. The executor now creates the queues only when
FastVideoArgs.enable_streaming_ipc_queues is set. StreamingVideoGenerator sets
it on a copy of its arguments in queue mode. Standard generation creates no
streaming semaphores.
The distributed_setup test fixture now sets MASTER_ADDR and MASTER_PORT when
they are not set, with a new port for each test, and restores them after the
test. pyproject.toml now declares orjson and python-multipart, which the
OpenAI-compatible server needs. Two documents about the SSIM reference
bootstrap are corrected.
For Wan VSA inference with 64-token tiles, each attention layer scattered Q, K,
V and the gate into a tile buffer and then made four transposed copies. On
SM100, one Triton kernel (tile_to_bhsd) now does the scatter and the transposes
in one pass. Training, 256-token tiles, other GPUs and builds without the kernel
use the previous path. FASTVIDEO_DISABLE_VSA64_FUSED_LAYOUT=1 turns the fused
path off.
On GB200 the layout and attention step goes from 15.7 ms to 8.6 ms. FastWan 1.3B
frames are bit-exact with the previous path at sequence parallel 1 and 2.
The HunyuanVideo and HunyuanVideo 1.5 transformers failed on the first forward
pass when the batch had more than one sample. The modulation inputs had the
shape [B, C], but the fused ops expect [B, 1, C], so they could not broadcast
over the sequence. The models now add the sequence axis, as the refiner and the
final layer already do. Results for batch size 1 do not change.
On SM100, the Attn-QAT training kernel now uses a faster route for non-causal
BF16 attention with equal query and key lengths of 16384 tokens or more. The
forward pass processes full KV tiles without bounds masks, and a separate masked
step handles the partial tail tile. The backward pass uses 64x128 dQ and dK/dV
tiles. Outputs, softmax statistics and all gradients are bitwise identical to the
previous route. FASTVIDEO_ATTN_QAT_SM100_OPTIMIZED=0 turns off the SM100 route,
and FASTVIDEO_ATTN_QAT_SM100_WIDE_BWD=0 keeps the 64x64 backward tiles.
On GB200 with 6 heads and 31200 tokens, the forward kernel time goes from 9.14 ms
to 6.96 ms and the backward kernel time from 17.41 ms to 15.85 ms.
For n > 1, the image endpoints returned one preview grid instead of n images.
The endpoints now run one model batch, save each decoded sample as its own file,
and return one URL or base64 item for each image. This applies to generations
and edits.
A keyword-only return_samples field (default False) is added to OutputConfig,
SamplingParam and ForwardBatch. Batched image requests set it and do not build
preview frames. The worker keeps the sample tensor for these requests, and
non-output ranks clear the field. Defaults and positional arguments do not
change.
Ray sets CUDA_VISIBLE_DEVICES of each GPU actor to the GPU of that actor. The
import of fastvideo initializes CUDA, so the executor cannot change the device
list later. On a node with more than one GPU, each worker after the first worker
failed with "invalid device ordinal".
On NVIDIA GPUs, each worker actor now keeps the device list of its raylet
(RAY_EXPERIMENTAL_NOSET_CUDA_VISIBLE_DEVICES=1). Each worker uses the position of
its GPU in that list as its local rank. The safetensors loader now reads each
file on node-group rank 0, which is the rank that sends the broadcast.
ParquetDatasetWriter.flush() named each shard from the chunk index plus the number
of files in the worker directory. When two flushes split the chunks in different
ways, a new shard could get the name of an existing shard and replace it. The
data in the old shard was lost without an error.
New shards now get numbers after the highest existing shard number in all worker
directories. One helper writes each shard to a temporary file and renames it. It
raises FileExistsError instead of an overwrite. flush(write_remainder=True) now
also counts the remainder rows in its return value.
The legacy LTX-2 trainer did not give the clip fps to the DiT. The DiT then
kept temporal RoPE positions in frames during training, but inference uses
seconds (fps 24). The trainer now passes the fps of the stored latents (24 when
the value is missing).
The video transform stage resampled frames to train_fps but kept the source fps
in the batch. The saved clips then had the source fps, and the audio was cut to
the wrong length. The stage now sets the batch fps to train_fps.
The new regression test runs in the CI unit lane.
