Resolve the FP4 attention kernel and backend conflicts with main's #1980 versions
(re-adding install_fp4_vsa_attention), keep main's FP32 FSDP grouping, and keep both
save_inference and save_tagged in the checkpointer.
Three source recipes (T2VA, keyframes, Ref2VA) on one model page under
MiniMax H3, with knobs for task, resolution tier, aspect bucket,
duration, NVFP4 decoder and text encoder, offload mode, and GPU count.
The page uses the official Pika-Labs Hugging Face avatar.
The runbook now gives the FASTVIDEO_ATTENTION_BACKEND=FLASH_ATTN
FASTVIDEO_FA4=1 launch prefix and the SDPA fallback, the tier override,
and measured GB200 timings and memory. The example selects the bucket
with --resolution/--aspect, the frame count with --duration, and
--offload resident|lazy.
An adapter that trains a subset of the wrapped linears (Pika skips the
AdaLN projections) logged one warning per layer, about 200 per run.
Log one summary with the count and a few names.
The sampling param and presets fill 768x1344 before input preparation, so
an exact p768 bucket always won and the configured tier never applied.
Resolve the omitted canvas in apply_request_constraints, where explicit
request fields are still distinguishable; explicit sizes keep their tier
and one-sided sizes are rejected.
Applying the automatic Pika adapter inside the transformer's lazy load ran
_drop_pinned_host_copies, which read __dict__ through every LazyModule
proxy and reloaded the already-released text encoder and both VAEs.
Skip unmaterialized proxies and clear the real module of resident ones.
Expose the exported H3 NVFP4 VAE decoder through ComponentConfig and map it to FastVideoArgs, alongside the existing text-encoder weight override. Add example flags and docs for the 36L quality and 8L fast decoder tiers, including the 8L depth-cut and no-NVFP4-VAE-encoder caveats. Add compat translation tests.
The constructor LoRA adapter (Pika H3 Flex's released adapter) reads
param_names_mapping through the LazyModule proxy while the materialize
transform is still running. Nothing was published yet, so the access
re-entered materialize and reloaded the 66 GB transformer repeatedly,
ending in CUDA OOM with lazy_module_load=True (the example default).
Publish the loaded instance before running the transform.
Remove the unsupported pipeline.preset selection from the example; the compatibility adapter rejects it before model loading. Load the last image as a PIL image because the request adapter does not do it for the stage. Document that the local Ref2VA path enforces reference counts and the 22-frame video floor, but not Pika's hosted per-clip/per-modality duration caps.
Add Pika H3 Flex (MiniMax-H3 plus the released fl2va/ref2va rank-128 adapters) as a FastVideo model family: registry/config/presets, bucket geometry with tiered reference sizing, automatic adapter selection and manifest validation, the fixed six-forward re-noise sampler, T2VA/I2VA/L2VA/FL2VA/Ref2VA pipeline support, an example, docs, and CPU unit tests.
GPU end-to-end validation is still pending. This change does not add PDD, VSA, or new kernel paths.
BaseLayerWithLoRA.forward was decorated with torch.compile, so wrapping a linear also compiled its base layer.
Compiling a packed NVFP4/FP8 linear on its own changes its numerics, so a wrapped layer at strength 0 could differ
from the unwrapped layer. forward now adds no compiled region: the base layer runs exactly as it would unwrapped
(eagerly, or traced with a caller's compiled block), and only the adapter delta is compiled.
The unit lane fails on main in two contract tests. All three failures are
bookkeeping that drifted from the code, not runtime bugs:
- `test_env_policy.py::test_env_doc_table_matches_registry`: the generated
table in docs/contributing/env_vars.md is stale for
FASTVIDEO_MLX_QUANT_GEMM. Regenerated with the documented command
(`python fastvideo/tests/contract/test_env_policy.py`).
- `test_env_policy.py::test_env_access_follows_policy`: two new violations.
- fastvideo/tests/mlx/test_mlx_affine_dq_gemm.py wrote
FASTVIDEO_MLX_QUANT_GEMM through monkeypatch.setenv; now uses the
registered field's override() through the env_overrides fixture.
- fastvideo/tests/train/models/test_load_kandinsky5.py read
FASTVIDEO_ATTENTION_BACKEND with `not in os.environ`; now uses
envs.FASTVIDEO_ATTENTION_BACKEND.is_set() and override(). Its entry in
KNOWN_VIOLATIONS is deleted, as the shrinking-list rule requires.
- `test_schema_parity_inventory.py::..._extension_fields_are_classified`:
MiniMaxH3PipelineConfig declares video_scheduler_shift and
audio_scheduler_shift (training-time noise clocks rejected by
check_pipeline_config at inference); both are added to
pipeline_config_extensions.internal_only.
Test plan (CPU, no GPU required):
python -m pytest fastvideo/tests/contract/test_env_policy.py \
fastvideo/tests/api/test_schema_parity_inventory.py \
fastvideo/tests/mlx/test_mlx_affine_dq_gemm.py -q
# 45 passed
python -m pytest fastvideo/tests/contract/ fastvideo/tests/api/ -q -m "not gpu"
# 501 passed
- Allow the frozen FP32 boundary modules through the FSDP mixed-dtype
guard when a model declares replicate_fp32_modules.
- Make the tile-64 FP4 kernel and its reference handle a listed block
with an empty query half instead of propagating NaN; add a CPU
regression test.
- Compare generated row ids in the row-plan test.
- Consume method checkpoint/stop requests after the pre-loop eval, so
best/ always refers to the evaluated weights.
- Remove the unused audio-stats module and its claim.
- Refresh the V2 model description and drop the dead compression script
reference.
- V2 config now matches the canonical 16-GPU stage 1: warmup 10,
min LR ratio 0.3, HSDP 16.
- State that the published V2.1 checkpoint is linears-only (BF16 VSA
attention); the FP4 adaptation is a separate, non-public stage.
- Name the released checkpoints and starting models.
- Vendor and pin both calibrated amax tables.
- Add the V2 prompt/embedding and OmniRef eval data-prep scripts.
- Include V2 width/height/num_frames and the train manifest in the step-0
fingerprint so a dimension change cannot reuse a stale baseline.
- Honor CompressionSpec.first_steps in the V2 rung scope.
- Mark tagged checkpoints weights-only in metadata and teach the exporter
to resolve and load them, so best/ can be exported.
- Add a tile-64 backward comparison against the STE reference.
Add the OmniRef (Ref2VA, VSA tile 128) and V2/V2.1 Consumer (T2VA,
VSA tile 64) profiles of one quantization-aware distillation recipe:
NVFP4 W4A4 q/k/v/out and FFN linears in all 50 blocks, the FP4
sparse-attention emulator, frozen VSA gates and norms, output plus
layer-reconstruction regression against a frozen bf16 teacher, and
best-by-score checkpoints with audio eligibility guards.
Training code: the PDD-QAD method, the two H3 model plugins, the
NVFP4 emulated linear, and the tile-64/128 Triton emulator with its
PyTorch numeric reference. The VSA attention impl gains a fine
attention override hook and lets tile 128 run grad-capable forwards
through it. The trainer gains opt-in tagged checkpoints and early
stop hooks for the QAD guards.
Also add two runnable configs, the training guide, and CPU/GPU tests.
Aryan Kumar