nodes/hao-ai-lab-FastVideo
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

[fix]: align the QAD configs and docs with the released provenance

Browse Source 
- V2 config now matches the canonical 16-GPU stage 1: warmup 10,
  min LR ratio 0.3, HSDP 16.
- State that the published V2.1 checkpoint is linears-only (BF16 VSA
  attention); the FP4 adaptation is a separate, non-public stage.
- Name the released checkpoints and starting models.
- Vendor and pin both calibrated amax tables.
- Add the V2 prompt/embedding and OmniRef eval data-prep scripts.
This commit is contained in:
Aryan Kumar
2026-10-19 12:38:43 -07:00
parent 5685ab40ca
commit fed3ce18b6
9 changed files with 1042 additions and 50 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/training/minimax_h3_fp4_qad.md
+116 -29
View File
@@ -8,20 +8,45 @@ the FastH3 (MiniMax-H3) family. One recipe covers two model profiles:
| OmniRef | `omniref_pdd.MiniMaxH3OmniRefPDDModel` | Ref2VA | 8-forward PDD ladder | 128 |
| V2 / V2.1 Consumer | `v2_t2v.MiniMaxH3V2T2VModel` | T2VA | 8-step DMD | 64 |
Both profiles ship the same deployed numerics: NVFP4 W4A4 linears in all 50
blocks plus FP4 sparse attention. QAD trains the student against a frozen bf16
teacher so that quantization error is absorbed by the weights instead of
appearing as drift, ghosting or audio loss at inference.
Both profiles use the same shared recipe: NVFP4 W4A4 linears in all 50 blocks,
the FP4 sparse-attention emulator, and the same objective and guards. QAD
trains the student against a frozen bf16 teacher so that quantization error is
absorbed by the weights instead of appearing as drift, ghosting or audio loss
at inference. OmniRef's published v2 checkpoint includes FP4 attention; the
published V2.1 checkpoint predates it and keeps BF16 VSA attention (see
[Released checkpoints](#released-checkpoints)).
The runnable configs are:
- `examples/train/configs/knowledge_distillation/minimax_h3/omniref_nvfp4_qad.yaml`
- `examples/train/configs/knowledge_distillation/minimax_h3/v2_nvfp4_qad.yaml`
## Released checkpoints
| Role | Checkpoint |
|---|---|
| OmniRef QAD v2 result | `FastVideo/FastH3-Omni-NVFP4-QAD-v2` |
| V2.1 result | `FastVideo/FastVideo-FastH3-8-Step-V2.1-NVFP4-Consumer` |
| V2 bf16 teacher / starting weights | `FastVideo/FastVideo-FastH3-8-Step-V2` |
| NVFP4 consumer baseline | `FastVideo/FastVideo-FastH3-8-Step-V2-NVFP4-Consumer` |
The OmniRef config starts from the **calibrated step-3500 OmniRef checkpoint
used as QAD step 0**, not from the released `-v2` result; point `init_from` at
that checkpoint. The V2 config starts from the bf16 V2 snapshot.
The published V2.1 checkpoint is the **linears-only variant** of this
schedule: NVFP4 W4A4 linears with BF16 VSA attention, 300 steps. The shared
FP4 recipe on this page additionally runs the FP4 attention emulator from step
0; it is the same training code and objective, but it does not reproduce the
published V2.1 checkpoint bit for bit. Set
`models.student.qad.fp4_attention.enabled: false` to get the published
linears-only variant.
## Target numerics
The student runs exactly the deployed inference numerics from step 0, so the
starting point is the released NVFP4 model rather than a bf16 approximation.
The student starts from the bf16 masters, and the deployed numerics are active
from the first forward, so every update sees the quantization error it is
trained to absorb.
- **Linears:** NVFP4 W4A4 for `attn.to_q`, `attn.to_k`, `attn.to_v`,
`attn.to_out`, `ff.fc_in` and `ff.fc_out` in every block.
@@ -40,9 +65,16 @@ starting point is the released NVFP4 model rather than a bf16 approximation.
`proj_out` and the audio projections stay fp32 and frozen, matching
inference.
The calibrated `amax` table is produced once per released checkpoint and
shipped with it. `qad.linears.amax_sha256` pins the table; the run copies it
into the output directory and refuses to start if the bytes differ.
The calibrated `amax` tables for both profiles are vendored under
`examples/train/calibration/fasth3/`:
| Table | sha256 |
|---|---|
| `amax-omniref-mse-bf16-top8.json` | `14b93ec3…df083d` |
| `amax-consumer.json` | `0800516c…d39e52` |
`qad.linears.amax_sha256` pins the table; the run copies it into the output
directory and refuses to start if the bytes differ.
## Training objective
@@ -80,12 +112,49 @@ in the sampled output.
- **Resume:** checkpoints carry the step-0 baseline and guard state; do not
delete `method.qad_baseline` if you edit a checkpoint by hand.
## Data preparation
V2/V2.1 is data-free: it needs prompts and their Qwen3-VL text embeddings, not
VAE latents.
1. Build the split manifests from prompt sources (parquet shards with a
`prompt` column, JSONL with a `prompt` field):
```bash
python scripts/distill/minimax_h3_v2_qad/build_v2_prompts.py \
--parquet 'data/prompts/*.parquet' --out-dir data/fasth3-v2 --eval-count 30
```
This writes `train.jsonl` and `eval.jsonl` plus an `embeddings/` directory.
2. Fill the embeddings (safe to rerun and shard across GPUs):
```bash
python scripts/distill/minimax_h3_v2_qad/precompute_v2_text.py \
--config examples/inference/basic/basic_fasth3_spark_v2_nvfp4.yaml \
--model-path data/models/FastH3-V2 \
--manifests data/fasth3-v2/train.jsonl data/fasth3-v2/eval.jsonl \
--shard 0 --num-shards 8
```
The V2 config's `train_manifest` and `eval_manifest` point at these files.
OmniRef training rows are preprocessed Ref2VA parquet rows that carry the
reference latents, text embeddings and geometry; the encoded dataset itself is
an input to this recipe and is not produced by the training code. Provide a
`MANIFEST.jsonl` with `case`, `id`, `parquet` and `resolution` fields and point
`models.student.data.manifests` at it.
`scripts/distill/minimax_h3_omniref_qad/generate_omniref_latents.py` builds the
held-out eval plan and runs the PDD ladder once per row: use `--eval-dir` to
write the held-out `manifest.json` consumed by `models.student.data.eval_manifest`
(and the raw latents for decoding).
## Running
Fill in the placeholder paths in the config (starting checkpoint, `amax`
table, latent manifests, output directory), then launch with `torchrun`.
Fill in the placeholder paths in the config (starting checkpoint, data
manifests, output directory), then launch with `torchrun`.
OmniRef, 4 nodes x 4 GPUs (SP 4):
OmniRef, 4 nodes x 4 GPUs (sequence parallel 4):
```bash
torchrun --nnodes=4 --nproc_per_node=4 --node_rank=$NODE_RANK \
@@ -94,30 +163,34 @@ torchrun --nnodes=4 --nproc_per_node=4 --node_rank=$NODE_RANK \
--config examples/train/configs/knowledge_distillation/minimax_h3/omniref_nvfp4_qad.yaml
```
V2/V2.1, 4 GPUs:
```bash
torchrun --nproc_per_node=4 \
-m fastvideo.train.entrypoint.train \
--config examples/train/configs/knowledge_distillation/minimax_h3/v2_nvfp4_qad.yaml
```
The V2/V2.1 recipe has an optional lower-learning-rate continuation stage. It
keeps the same numerics and objective and resumes from the first stage's `best`
DCP. The knobs are listed at the bottom of the config:
V2/V2.1, 4 nodes x 4 GPUs (16-way HSDP, lr 5e-6, 300 steps):
```bash
torchrun --nnodes=4 --nproc_per_node=4 --node_rank=$NODE_RANK \
--master_addr=$MASTER_ADDR --master_port=$MASTER_PORT \
-m fastvideo.train.entrypoint.train \
--config examples/train/configs/knowledge_distillation/minimax_h3/v2_nvfp4_qad.yaml \
--method.init_student_dcp outputs/v2_1_nvfp4_qad/best \
--training.optimizer.learning_rate 1.5e-6 \
--training.distributed.num_gpus 16 \
--training.distributed.hsdp_shard_dim 16 \
--training.checkpoint.output_dir outputs/v2_1_nvfp4_qad_continuation
--config examples/train/configs/knowledge_distillation/minimax_h3/v2_nvfp4_qad.yaml
```
To reproduce the published V2.1 linears-only checkpoint, disable FP4 attention
for the same schedule:
```bash
--models.student.qad.fp4_attention.enabled false
```
## FP4-attention adaptation
The internal follow-up that this recipe tracks started from a linears-only
300-step DCP, enabled FP4 attention, and trained 300 more steps at lr 1.5e-6
with no warmup and a constant LR on 16 GPUs. It is an **adaptation of the
linears-only checkpoint to FP4 attention**, not a second epoch of the FP4
stage 1 above: the first stage's attention numerics were BF16, and the
continuation needs a DCP of that linears-only model (the packed HF release is
not one). It is therefore not part of the public reproduction path, and the
published V2.1 checkpoint remains the linears-only result. The exact overrides
are listed at the bottom of the V2 config.
## Checkpoints
Runs write regular `checkpoint-<step>/` DCP checkpoints plus a `best/`
@@ -155,6 +228,10 @@ python -m fastvideo.train.entrypoint.dcp_to_diffusers \
| `fastvideo/attention/backends/fp4_vsa_qat.py` | VSA fine-attention override |
| `fastvideo-kernel/python/fastvideo_kernel/triton_kernels/attn_qat_vsa_train.py` | Tile 64/128 FP4 block-sparse Triton kernel |
| `fastvideo-kernel/python/fastvideo_kernel/triton_kernels/attn_qat_vsa_reference.py` | PyTorch numeric reference |
| `scripts/distill/minimax_h3_v2_qad/build_v2_prompts.py` | V2 prompt split manifests |
| `scripts/distill/minimax_h3_v2_qad/precompute_v2_text.py` | V2 Qwen3-VL text embeddings |
| `scripts/distill/minimax_h3_omniref_qad/generate_omniref_latents.py` | OmniRef held-out eval plan and latents |
| `examples/train/calibration/fasth3/` | Pinned FFN activation `amax` tables |
## Expected behavior
@@ -165,3 +242,13 @@ teacher-forced audio error clearly drops. Gains slow after roughly 60-100
steps; the remaining error is dominated by FP4 attention. Evaluate a checkpoint
with the full-eval samples, not the teacher-forced score alone, before
promoting it.
Released reference points (lower is better):
| Run | Metric | Baseline | Selected |
|---|---|---|---|
| V2.1 (linears-only) | mean per-forward x0 rel-L2 | 0.431 | 0.351 (video 0.382, audio 0.319) |
| OmniRef QAD v2 | teacher-forced score | 0.362 | 0.303 at step 120 (video 0.198, audio 0.409) |
The V2.1 baseline is the NVFP4 Consumer model before QAD; the OmniRef baseline
is the calibrated checkpoint at QAD step 0.