With enable_torch_compile_vae (#1734, opt-in) inductor autotunes kernels per
process, so chunk decodes on other ranks differ from the serial rank's decode
the way two serial processes differ (GB200 @124f: max 63/255 on <0.5% of
pixels, mean ~1e-2/255; audio and chunk 0 bit-identical). Eager decoder (the
default) stays bitwise-equal to serial decode_to_pixels - measured, both
strategies, x3, 124f+345f.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two schedule fixes sized from the first GB200 tray measurements (job 2659,
serial 7.7s/21.2s at 124f/345f):
1. Every rank now decodes its round-0 chunk BEFORE the metadata broadcast.
Non-leader ranks previously blocked on the broadcast until the leader
finished chunk 0, serializing a full extra chunk-decode into round 0
(visible as 1.9x instead of ~2.6x at 7 chunks / 4 ranks). Same reorder
on the encode path.
2. The leader's per-chunk joining work (blend, denormalize, clamp, output
copies) moves to a dedicated CUDA side stream. It depends only on
already-gathered segments, but on the main stream it delayed the
leader's next-round decode and therefore every rank's next collective
(~0.1s/chunk on the critical path). Gathered storage is pinned to the
assembly stream via record_stream; the driver drains the stream in a
finally so an exception cannot leave an in-flight DMA into the output
buffer. Stream placement does not change op order or values, so the
bitwise-parity contract is untouched.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
vae_parallel_decode / vae_parallel_encode / vae_parallel_decode_strategy are
model-specific optimization knobs (compatibility_only, like VSA_sparsity).
VSA_tile_size came in with the merged tile-64 route without an inventory
entry and failed test_fastvideo_args_fields_are_classified on the whole
stack; classify it the same way. The remaining pipeline_config inventory
gaps (image_encoder_precisions, ...) predate this branch and are left for
the owning PRs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Under SP>1 the H3 video VAE decoded all temporal chunks serially on the
output rank while the other ranks idled (#1703's gate), and every rank
encoded the full reference video redundantly. Chunk decodes and clip
encodes have no cross-chunk data dependency - only the joining (overlap
blend, trim, denormalize, moment concat) is sequential - so both are
round-robined across the sequence-parallel ranks:
- fastvideo/models/vaes/minimax_h3_parallel.py: decode_to_pixels_parallel
gathers each round's decoded segments (body+halo tail, one contiguous
slice per chunk) to the SP group's first rank via NCCL gather (or
all_gather, FASTVIDEO_VAE_PARALLEL_DECODE_STRATEGY), which replays the
serial blend/trim/denormalize/copy semantics with the same VAE methods -
bitwise-equal to serial decode_to_pixels by construction. Placeholder
rounds keep collective participation uniform; the leader decodes chunk 0
first and broadcasts dtype/shape metadata so placeholders never guess the
autocast dtype. encode_pixels_parallel all-gathers per-clip moments
(latent-sized) so every rank keeps the identical full posterior,
preserving the all-ranks-hold-latents contract.
- decoding stage: output gate moves from world rank 0 to the SP group's
first rank (identical in the single-group e2e case; correct for the
trainer validation callback, which consumes each group leader's batch);
with vae_parallel_decode every rank enters the decode body so no
rank-dependent branch guards the collectives.
- latent preparation: opt-in clip-parallel reference encode on the same
seam (vae_parallel_encode).
- knobs: FastVideoArgs.vae_parallel_decode/encode (+ --vae-parallel-decode,
--vae-parallel-encode, FASTVIDEO_VAE_PARALLEL_DECODE/ENCODE env
parse-once adapters), default OFF.
- _copy_chunk_pixels factored out of _decode_to_pixels so serial and
parallel share one output-copy path (behavior unchanged).
Tests: threaded fake-group CPU suite drives the real SPMD functions
end-to-end (world sizes 2-5, both strategies, pad/blend/trim geometries,
token_drop=0, batched slicing, placeholder rounds) bit-exact vs the serial
APIs; GPU regression (torchrun world>1 gated) asserts bitwise parity under
fp16 autocast with real NCCL plus repeat-determinism x3.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
_print_info_once passes stacklevel=2 into logger.info, and init_logger's
patched _info passed its own stacklevel=2 positionally into logger.log on
top of the caller's kwarg -> TypeError on every info_once call. Honor an
explicit stacklevel instead of passing the keyword twice.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Extend the training-side regional-compile port (internal 863e87342, GO
verdict vsa_gate/compile_ab/VERDICT.md) to inference so every H3 run can
compile the 52 transformer blocks. Opt-in and off by default:
- New FastVideoArgs.inference_torch_compile (CLI --inference-torch-compile),
env FASTVIDEO_INFERENCE_TORCH_COMPILE=1 folded in __post_init__ (the
attention_backend parse-once pattern), reachable through
PipelineSelection.experimental {"inference_torch_compile": true} exactly
like the VSA_sparsity / VSA_tile_size knobs.
- maybe_load_fsdp_model applies the compile right after the transformer
loads: per-_compile_conditions block, fullgraph=True + inductor
options.emulate_precision_casts injected, no user kwargs needed (the
compile-A/B verdict recipe). prepare_for_compile still runs first, so the
H3 fusion-inertness warning of #1735 is preserved.
- _regional_compile_unsupported_reason ported from the training loader:
VSA / VSA-H3 backends and the FASTVIDEO_DISABLE_ATTENTION_COMPILE=1
escape hatch degrade the transformer to eager with one warning; FLASH_ATTN
on flash-attn 3 is rejected with an actionable message. FA4/FA2/SDPA
compile through their existing custom-op boundaries (forward-only is
enough at inference; the training port's backward custom ops are not
needed under no_grad).
- FASTVIDEO_DISABLE_ATTENTION_COMPILE default flipped to 0 (attention is
traced by default), matching the training port and upstream #1718 —
eager runs are unaffected (torch.compiler.disable is inert outside
dynamo).
- enable_torch_compile + inference_torch_compile together skip the
pipeline-level DiT compile (the loader already owns those forwards).
- examples: --inference-torch-compile on basic_minimax_h3_t2v.py (dense
target) and basic_fasth3.py (exercises the VSA degrade path).
- CPU-safe contract tests for the guard and the kwargs injection.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Will Lin