[train] MiniMax-H3 config suite: SFT sparsity sweep, data-free DMD2, 32-GPU Slurm launch, README

- fine_tuning/minimax_h3: FA4 dense control, VSA sparsity 0/0.8/0.9/0.95/0.97
  overfits, effective-batch-2 via gradient accumulation over two videos
  (validated: both losses decline, ~10 s/step on 4x GB200), and the 2-GPU
  configs kept as negative results (sp=2 backward working set OOMs 184 GiB).
- distribution_matching/minimax_h3: data-free DMD2 (rollout_mode: simulate)
  overfit and its 32-GPU scale-up (8 trays, sp=4, HSDP 8x4, global batch 64
  = 8 DP groups x accum 8).
- examples/train/slurm/dmd2_32xgb200.sbatch: login-node launch deriving
  world size/rendezvous from the allocation, per-node /dev/shm weight cache,
  per-node log dirs, CLUSTER-marked partition/account/container hooks.
- README.md in both config dirs: launch, topology/batch semantics, measured
  step times, memory findings, shard cache, preprocessing.
- pyproject: explicit setuptools include list.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

examples/train/configs/distribution_matching/minimax_h3/README.md

@@ -0,0 +1,87 @@
+# MiniMax-H3 training configs (DMD2 distillation + SFT overfit)
+
+Configs for training the 33B dual-modality (video+audio) MiniMax-H3 DiT with
+the modular trainer (`fastvideo/train/`), validated on 4x GB200 (192 GB,
+aarch64). SFT overfit configs live in
+[`../../fine_tuning/minimax_h3/`](../../fine_tuning/minimax_h3/); the 8-tray
+Slurm launch is [`../../../slurm/dmd2_32xgb200.sbatch`](../../../slurm/dmd2_32xgb200.sbatch).
+
+## Configs
+
+| Config | What it is |
+|---|---|
+| `dmd2_vsa0_overfit.yaml` | Single-sample DMD2, student on VIDEO_SPARSE_ATTN_H3 at sparsity 0 (parity vs dense), teacher/critic on FLASH_ATTN/FA4, `rollout_mode: data_latent` |
+| `dmd2_simulate_fa4_overfit.yaml` | **Data-free** DMD2 (`rollout_mode: simulate`): student rolls out from pure noise through `dmd_denoising_steps`; parquet VAE latents ignored, only text conditioning used. All roles dense FA4 |
+| `dmd2_simulate_fa4_32gpu.yaml` | The simulate recipe scaled to 8 trays x 4 GPUs, global batch 64 |
+| `../../fine_tuning/minimax_h3/sft_*.yaml` | SFT overfit family: FA4 dense control, VSA at sparsity 0 / 0.8 / 0.9 / 0.95 / 0.97, and `sft_fa4_bs2_overfit.yaml` (effective batch 2 over two videos) |
+
+## Launch
+
+```bash
+# single node (4x GB200)
+FASTVIDEO_FA4=1 \
+FASTVIDEO_WEIGHT_SHARD_CACHE=/dev/shm/fastvideo-wcache \
+PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
+bash examples/train/run.sh examples/train/configs/distribution_matching/minimax_h3/dmd2_simulate_fa4_overfit.yaml
+
+# 8 trays / 32 GPUs from a Slurm login node (global batch 64)
+sbatch examples/train/slurm/dmd2_32xgb200.sbatch
+```
+
+`FASTVIDEO_FA4=1` selects the FA4 path inside the FLASH_ATTN backend and is
+required for the dense roles. Per-role backends come from
+`models.<role>.attention_backend` — do not set `FASTVIDEO_ATTENTION_BACKEND`
+globally.
+
+## Topology and batch semantics
+
+- `sp_size: 4` — one packed `[text|cond|audio|video]` document is
+  sequence-parallel across the 4 GPUs of a node; HSDP mesh
+  `(replicate=nodes, shard=4)` keeps parameters sharded only inside a node
+  (~145 GiB/rank for DMD2's three 33B roles with full gradient checkpointing).
+- **The H3 packed pipeline is batch-1 per forward by design** (documents with
+  different caption lengths cannot stack). Batching composes through
+  `training.loop.gradient_accumulation_steps`, which the trainer loops
+  natively — global batch = DP groups x accumulation rounds. The 32-GPU
+  config's global 64 = 8 DP groups x 8 rounds; validated locally as
+  effective-batch-2 (`sft_fa4_bs2_overfit.yaml`, two videos, both losses
+  declining, ~10 s/step = 2 micro fwd+bwd).
+- DMD2 TTUR: critic every step, generator every
+  `generator_update_interval`-th. `rollout_mode: simulate` costs ~4 dense
+  forwards per critic step and ~7 + 2 backwards per generator step
+  (~10/20 s at 1344x768x124f on GB200).
+
+## Measured reference points (1344x768, 124 frames, sp=4)
+
+- SFT step time: FA4 dense 4.9 s/it; VSA 90% 4.8; 95% 4.49; 97% 4.33;
+  VSA-at-density-1 11.2 (VSA machinery overhead — attention is only ~1/3 of
+  step FLOPs, so sparsity is mostly an inference win: 3-step validation
+  sample ~120 s dense vs ~59 s at 95%).
+- 2x GB200 does NOT fit this model/sequence without CPU offload: the first
+  backward alone allocates ~178 GiB at sp=2 (params 31 + grads 31 + ~116
+  working set) before optimizer states exist.
+- `betas: [0.0, 0.999]` auto-selects a buffer-free AdamW
+  (`fastvideo/train/utils/optimizer.py`), saving one full set of optimizer
+  state (~15.4 GiB/rank/model) — required for DMD2's two trainable roles.
+
+## Weight shard cache
+
+`FASTVIDEO_WEIGHT_SHARD_CACHE=<tmpfs dir>` caches each rank's post-shard
+DTensor chunks after the first full load
+(`fastvideo/models/loader/shard_cache.py`); relaunches rebuild each 33B role
+in ~2-4 s instead of ~10 min of cold-NFS reads (DMD2's three roles share one
+entry). On multi-node runs with node-local cache dirs set
+`FASTVIDEO_WEIGHT_SHARD_CACHE_PER_NODE=1` (the sbatch does) so every node
+writes its own copy. Any validation failure degrades to the normal full load.
+
+## Data
+
+Single/two-sample overfit parquets are produced by
+`fastvideo/pipelines/preprocess/preprocess_minimax_h3_overfit.py` (one mp4
+with soundtrack + prompt -> one t2va row; >=124 frames at 24 fps and an audio
+track are required). The preprocessor writes exactly one
+`data_00000.parquet` per output dir — build multi-sample sets by copying rows
+into one dir as `data_00000.parquet`, `data_00001.parquet`, ... The
+validation callback samples every prompt in `callbacks.validation.dataset_file`
+at the method's exact training sigmas (`dmd_denoising_steps` is injected into
+the validation pipeline).

examples/train/configs/distribution_matching/minimax_h3/dmd2_simulate_fa4_32gpu.yaml

@@ -0,0 +1,123 @@
+# 32x GB200 (8 trays x 4 GPUs) data-free DMD2 on MiniMax H3, all-dense FA4.
+#
+# Topology: sp_size=4 within a tray (one packed document per SP group), HSDP
+# mesh (replicate=8, shard=4) — params sharded only inside a tray (keeps the
+# proven ~145 GiB/rank profile from the 4-GPU runs), gradients replicated
+# across trays. 32/4 = 8 data-parallel groups.
+#
+# Global batch 64 = 8 DP groups x gradient_accumulation_steps 8 x micro-batch
+# 1. H3's packed-sequence pipeline is batch-1 per forward by design (one
+# interleaved [text|cond|audio|video] document); batching composes through
+# accumulation, which the trainer loops natively.
+#
+# Launch: sbatch examples/train/slurm/dmd2_32xgb200.sbatch
+#
+# NOTE: data_path below still points at the 2-row overfit set — good for a
+# 32-GPU smoke test only. Swap in the real preprocessed t2va dataset before a
+# production run.
+
+models:
+  student:
+    _target_: fastvideo.train.models.minimax_h3.MiniMaxH3DMDModel
+    init_from: /mnt/models/MiniMax-H3
+    trainable: true
+    enable_gradient_checkpointing_type: full
+    attention_backend: FLASH_ATTN
+  teacher:
+    _target_: fastvideo.train.models.minimax_h3.MiniMaxH3DMDModel
+    init_from: /mnt/models/MiniMax-H3
+    trainable: false
+    disable_custom_init_weights: true
+    attention_backend: FLASH_ATTN
+  critic:
+    _target_: fastvideo.train.models.minimax_h3.MiniMaxH3DMDModel
+    init_from: /mnt/models/MiniMax-H3
+    trainable: true
+    disable_custom_init_weights: true
+    enable_gradient_checkpointing_type: full
+    attention_backend: FLASH_ATTN
+
+method:
+  _target_: fastvideo.train.methods.distribution_matching.dmd2.DMD2Method
+  rollout_mode: simulate
+  generator_update_interval: 5
+  real_score_guidance_scale: 3.5
+  dmd_denoising_steps: [1000, 757, 522]
+  min_timestep_ratio: 0.02
+  max_timestep_ratio: 0.98
+  cfg_uncond:
+    text: zero
+
+  fake_score_learning_rate: 8.0e-6
+  fake_score_betas: [0.0, 0.999]
+  fake_score_lr_scheduler: constant
+
+training:
+  distributed:
+    num_gpus: 32
+    sp_size: 4
+    tp_size: 1
+    hsdp_replicate_dim: 8
+    hsdp_shard_dim: 4
+
+  data:
+    data_path: /mnt/h3-dmd2-overfit/data_bs2  # smoke-test set; swap for real data
+    preprocessed_data_type: t2va
+    dataloader_num_workers: 0
+    train_batch_size: 1
+    training_cfg_rate: 0.0
+    seed: 42
+    num_latent_t: 37
+    num_height: 768
+    num_width: 1344
+    num_frames: 124
+
+  optimizer:
+    learning_rate: 2.0e-6
+    betas: [0.0, 0.999]
+    weight_decay: 0.01
+    lr_scheduler: constant
+    lr_warmup_steps: 0
+
+  loop:
+    max_train_steps: 4000
+    # 8 accumulation rounds x 8 DP groups = global batch 64 per optimizer step.
+    gradient_accumulation_steps: 8
+
+  checkpoint:
+    output_dir: outputs/minimax_h3_dmd2_simulate_fa4_32gpu
+    # Multi-node runs should checkpoint; DCP saves are large — point
+    # output_dir at a volume with headroom or set back to 0 to disable.
+    training_state_checkpointing_steps: 500
+    checkpoints_total_limit: 3
+
+  tracker:
+    project_name: h3-dmd2-vsa
+    run_name: dmd2_simulate_fa4_32gpu
+
+  model:
+    precondition_outputs: false
+    enable_gradient_checkpointing_type: full
+
+  dit_precision: bf16
+
+callbacks:
+  grad_clip:
+    _target_: fastvideo.train.callbacks.grad_clip.GradNormClipCallback
+    max_grad_norm: 1.0
+  validation:
+    _target_: fastvideo.train.callbacks.validation.ValidationCallback
+    pipeline_target: fastvideo.pipelines.basic.minimax_h3.minimax_h3_pipeline.MiniMaxH3Pipeline
+    dataset_file: /mnt/h3-dmd2-overfit/c17_bs2_validation.json
+    every_steps: 250
+    run_at_start: true
+    sampling_steps: [3]
+    guidance_scale: 1.0
+    num_frames: 124
+    num_videos_per_prompt: 1
+    use_validation_media_conditioning: false
+    offload_training_state: true
+    text_encoder_cpu_offload: true
+    vae_cpu_offload: true
+
+pipeline: {}

examples/train/configs/distribution_matching/minimax_h3/dmd2_simulate_fa4_overfit.yaml

@@ -0,0 +1,130 @@
+# Single-sample DMD2 DATA-FREE overfit: rollout_mode=simulate on MiniMax H3
+# with FLASH_ATTN (FA4) for every role — no VSA in the loop.
+#
+# - simulate rollout: the student builds its own trajectory from pure noise
+#   through dmd_denoising_steps (no-grad prefix steps, one with-grad step at a
+#   sampled target index). VAE latents in the parquet are ignored (zeros
+#   latents sized from training.data dims); only the text conditioning is
+#   consumed.
+# - Student/teacher/critic: FLASH_ATTN (run with FASTVIDEO_FA4=1). DMD2 still
+#   passes attn_kind="vsa" for the student rollout, which degrades to dense
+#   because attn_metadata_vsa stays None for non-VSA backends.
+# - Data: same ONE preprocessed t2va parquet row (768x1344, 124 frames @ 24
+#   fps) as dmd2_vsa0_overfit.yaml; see examples/distill/MiniMax-H3/.
+
+models:
+  student:
+    _target_: fastvideo.train.models.minimax_h3.MiniMaxH3DMDModel
+    init_from: /mnt/models/MiniMax-H3
+    trainable: true
+    enable_gradient_checkpointing_type: full
+    attention_backend: FLASH_ATTN
+  teacher:
+    _target_: fastvideo.train.models.minimax_h3.MiniMaxH3DMDModel
+    init_from: /mnt/models/MiniMax-H3
+    trainable: false
+    disable_custom_init_weights: true
+    attention_backend: FLASH_ATTN
+  critic:
+    _target_: fastvideo.train.models.minimax_h3.MiniMaxH3DMDModel
+    init_from: /mnt/models/MiniMax-H3
+    trainable: true
+    disable_custom_init_weights: true
+    enable_gradient_checkpointing_type: full
+    attention_backend: FLASH_ATTN
+
+method:
+  _target_: fastvideo.train.methods.distribution_matching.dmd2.DMD2Method
+  rollout_mode: simulate
+  generator_update_interval: 5
+  real_score_guidance_scale: 3.5
+  dmd_denoising_steps: [1000, 757, 522]
+  min_timestep_ratio: 0.02
+  max_timestep_ratio: 0.98
+  # H3 has no negative-prompt encoder at training time; unconditional teacher
+  # forwards zero the text embeddings instead.
+  cfg_uncond:
+    text: zero
+
+  # Critic optimizer (required — no fallback to training.optimizer)
+  fake_score_learning_rate: 8.0e-6
+  fake_score_betas: [0.0, 0.999]
+  fake_score_lr_scheduler: constant
+
+training:
+  distributed:
+    num_gpus: 4
+    sp_size: 4
+    tp_size: 1
+    hsdp_replicate_dim: 1
+    hsdp_shard_dim: 4
+
+  data:
+    # Preprocessed-latents parquet dir; simulate mode only reads the text
+    # conditioning from it (latents are zeros sized from the dims below).
+    data_path: /mnt/h3-dmd2-overfit/data
+    preprocessed_data_type: t2va
+    dataloader_num_workers: 0
+    train_batch_size: 1
+    training_cfg_rate: 0.0
+    seed: 42
+    num_latent_t: 37
+    num_height: 768
+    num_width: 1344
+    num_frames: 124
+
+  optimizer:
+    learning_rate: 2.0e-6
+    betas: [0.0, 0.999]
+    weight_decay: 0.01
+    lr_scheduler: constant
+    lr_warmup_steps: 0
+
+  loop:
+    max_train_steps: 2000
+    gradient_accumulation_steps: 1
+
+  checkpoint:
+    output_dir: outputs/minimax_h3_dmd2_simulate_fa4_overfit
+    # 0 disables training-state checkpointing (no resume; avoids ~400+ GiB
+    # DCP saves on the 87%-full volume).
+    training_state_checkpointing_steps: 0
+    checkpoints_total_limit: 3
+
+  tracker:
+    project_name: h3-dmd2-vsa
+    run_name: dmd2_simulate_fa4_overfit
+
+  model:
+    precondition_outputs: false
+    enable_gradient_checkpointing_type: full
+
+  dit_precision: bf16
+
+callbacks:
+  grad_clip:
+    _target_: fastvideo.train.callbacks.grad_clip.GradNormClipCallback
+    max_grad_norm: 1.0
+  validation:
+    _target_: fastvideo.train.callbacks.validation.ValidationCallback
+    pipeline_target: fastvideo.pipelines.basic.minimax_h3.minimax_h3_pipeline.MiniMaxH3Pipeline
+    dataset_file: /mnt/h3-dmd2-overfit/c17_validation.json
+    every_steps: 50
+    # Sample the untrained baseline at step 0 (costs the validation pipeline
+    # build upfront: text-encoder + VAE loads before the first train step).
+    run_at_start: true
+    sampling_steps: [3]
+    guidance_scale: 1.0
+    num_frames: 124
+    num_videos_per_prompt: 1
+    use_validation_media_conditioning: false
+    offload_training_state: true
+    text_encoder_cpu_offload: true
+    vae_cpu_offload: true
+  # EMA disabled: its GPU shadow costs ~15.4 GiB/rank (OOM on 184 GiB cards).
+  # ema:
+  #   _target_: fastvideo.train.callbacks.ema.EMACallback
+  #   decay: 0.98
+  #   start_iter: 0
+
+pipeline: {}

examples/train/configs/distribution_matching/minimax_h3/dmd2_vsa0_overfit.yaml

@@ -13,19 +13,19 @@
 models:
   student:
     _target_: fastvideo.train.models.minimax_h3.MiniMaxH3DMDModel
-    init_from: MiniMaxAI/MiniMax-H3
+    init_from: /mnt/models/MiniMax-H3
     trainable: true
     enable_gradient_checkpointing_type: full
     attention_backend: VIDEO_SPARSE_ATTN_H3
   teacher:
     _target_: fastvideo.train.models.minimax_h3.MiniMaxH3DMDModel
-    init_from: MiniMaxAI/MiniMax-H3
+    init_from: /mnt/models/MiniMax-H3
     trainable: false
     disable_custom_init_weights: true
     attention_backend: FLASH_ATTN
   critic:
     _target_: fastvideo.train.models.minimax_h3.MiniMaxH3DMDModel
-    init_from: MiniMaxAI/MiniMax-H3
+    init_from: /mnt/models/MiniMax-H3
     trainable: true
     disable_custom_init_weights: true
     enable_gradient_checkpointing_type: full
@@ -87,7 +87,9 @@ training:
 
   checkpoint:
     output_dir: outputs/minimax_h3_dmd2_vsa0_overfit
-    training_state_checkpointing_steps: 250
+    # 0 disables training-state checkpointing (no resume; avoids ~400+ GiB
+    # DCP saves on the 87%-full volume).
+    training_state_checkpointing_steps: 0
     checkpoints_total_limit: 3
 
   tracker:
@@ -107,8 +109,11 @@ callbacks:
   validation:
     _target_: fastvideo.train.callbacks.validation.ValidationCallback
     pipeline_target: fastvideo.pipelines.basic.minimax_h3.minimax_h3_pipeline.MiniMaxH3Pipeline
-    dataset_file: examples/training/finetune/Wan2.1-Fun-1.3B-InP/crush_smol/validation.json
-    every_steps: 250
+    dataset_file: /mnt/h3-dmd2-overfit/c17_validation.json
+    every_steps: 50
+    # Sample the untrained baseline at step 0 (costs the validation pipeline
+    # build upfront: text-encoder + VAE loads before the first train step).
+    run_at_start: true
     sampling_steps: [3]
     guidance_scale: 1.0
     num_frames: 124
@@ -117,9 +122,12 @@ callbacks:
     offload_training_state: true
     text_encoder_cpu_offload: true
     vae_cpu_offload: true
-  ema:
-    _target_: fastvideo.train.callbacks.ema.EMACallback
-    decay: 0.98
-    start_iter: 0
+  # EMA disabled: its GPU shadow costs ~15.4 GiB/rank, which put the student
+  # update step ~1 GiB over the 184 GiB cards (OOM at step 5/10). Re-enable
+  # once the shadow can live on CPU.
+  # ema:
+  #   _target_: fastvideo.train.callbacks.ema.EMACallback
+  #   decay: 0.98
+  #   start_iter: 0
 
 pipeline: {}

examples/train/configs/fine_tuning/minimax_h3/README.md

@@ -0,0 +1,13 @@
+# MiniMax-H3 SFT overfit configs
+
+Single- and two-sample SFT overfit runs used to validate the VSA-H3 backend,
+sparsity sweep (0 / 0.8 / 0.9 / 0.95 / 0.97), the FA4 dense control, and
+effective-batch-2 via gradient accumulation (`sft_fa4_bs2_overfit.yaml`).
+
+Launch, topology, batch semantics, measured step times, and the weight shard
+cache are documented in
+[`../../distribution_matching/minimax_h3/README.md`](../../distribution_matching/minimax_h3/README.md).
+
+Note: `sft_vsa95_overfit_2gpu.yaml` / `sft_vsa97_overfit_2gpu.yaml` are kept
+as negative results — 2x GB200 cannot fit this model/sequence without CPU
+offload (backward-pass working set alone OOMs 184 GiB cards at sp=2).

examples/train/configs/fine_tuning/minimax_h3/sft_fa4_bs2_overfit.yaml

@@ -0,0 +1,93 @@
+# Single-sample SFT overfit: MiniMax H3 with the VSA-H3 backend at sparsity
+# FA4 dense control (student on FLASH_ATTN, run with FASTVIDEO_FA4=1) for
+# distribution_matching/minimax_h3/dmd2_vsa0_overfit.yaml (same c17_v2
+# preprocessed record, same validation prompt). Optimizer recipe follows the
+# crush-smol H3 T2VA overfit (overfit_minimax_h3_t2va.yaml), scaled to one
+# 4-GPU node with pure-FSDP sharding.
+
+models:
+  student:
+    _target_: fastvideo.train.models.minimax_h3.MiniMaxH3Model
+    init_from: /mnt/models/MiniMax-H3
+    trainable: true
+    enable_gradient_checkpointing_type: full
+    attention_backend: FLASH_ATTN
+
+method:
+  _target_: fastvideo.train.methods.fine_tuning.finetune.FineTuneMethod
+
+training:
+  distributed:
+    num_gpus: 4
+    sp_size: 4
+    tp_size: 1
+    hsdp_replicate_dim: 1
+    hsdp_shard_dim: 4
+
+  data:
+    data_path: /mnt/h3-dmd2-overfit/data_bs2
+    preprocessed_data_type: t2va
+    dataloader_num_workers: 0
+    train_batch_size: 1  # per-micro-batch; effective batch 2 via grad accumulation
+    training_cfg_rate: 0.0
+    seed: 42
+    num_latent_t: 37
+    num_height: 768
+    num_width: 1344
+    num_frames: 124
+
+  # Student VSA sparsity (0.0 = every tile selected, dense parity).
+  vsa:
+    sparsity: 0.0
+
+  optimizer:
+    learning_rate: 5.0e-5
+    betas: [0.9, 0.999]
+    weight_decay: 0.0
+    lr_scheduler: constant
+    lr_warmup_steps: 0
+
+  loop:
+    max_train_steps: 2000
+    gradient_accumulation_steps: 2
+
+  checkpoint:
+    output_dir: outputs/minimax_h3_sft_fa4_bs2_overfit
+    # 0 disables training-state checkpointing (no resume; avoids large DCP
+    # saves on the shared volume).
+    training_state_checkpointing_steps: 0
+    checkpoints_total_limit: 3
+
+  tracker:
+    project_name: h3-dmd2-vsa
+    run_name: sft_fa4_bs2_overfit
+
+  model:
+    precondition_outputs: false
+    enable_gradient_checkpointing_type: full
+
+  dit_precision: bf16
+
+callbacks:
+  grad_clip:
+    _target_: fastvideo.train.callbacks.grad_clip.GradNormClipCallback
+    max_grad_norm: 1.0
+  validation:
+    _target_: fastvideo.train.callbacks.validation.ValidationCallback
+    pipeline_target: fastvideo.pipelines.basic.minimax_h3.minimax_h3_pipeline.MiniMaxH3Pipeline
+    dataset_file: /mnt/h3-dmd2-overfit/c17_bs2_validation.json
+    every_steps: 50
+    # Sample the untrained baseline at step 0.
+    run_at_start: true
+    # SFT is not a few-step generator; sample on the full 50-step schedule
+    # (mirrors the crush-smol overfit validation).
+    sampling_steps: [50]
+    guidance_scale: 1.0
+    num_frames: 124
+    num_videos_per_prompt: 1
+    use_validation_media_conditioning: false
+    offload_training_state: true
+    text_encoder_cpu_offload: true
+    vae_cpu_offload: true
+
+pipeline: {}

examples/train/configs/fine_tuning/minimax_h3/sft_fa4_overfit.yaml

@@ -0,0 +1,93 @@
+# Single-sample SFT overfit: MiniMax H3 with the VSA-H3 backend at sparsity
+# FA4 dense control (student on FLASH_ATTN, run with FASTVIDEO_FA4=1) for
+# distribution_matching/minimax_h3/dmd2_vsa0_overfit.yaml (same c17_v2
+# preprocessed record, same validation prompt). Optimizer recipe follows the
+# crush-smol H3 T2VA overfit (overfit_minimax_h3_t2va.yaml), scaled to one
+# 4-GPU node with pure-FSDP sharding.
+
+models:
+  student:
+    _target_: fastvideo.train.models.minimax_h3.MiniMaxH3Model
+    init_from: /mnt/models/MiniMax-H3
+    trainable: true
+    enable_gradient_checkpointing_type: full
+    attention_backend: FLASH_ATTN
+
+method:
+  _target_: fastvideo.train.methods.fine_tuning.finetune.FineTuneMethod
+
+training:
+  distributed:
+    num_gpus: 4
+    sp_size: 4
+    tp_size: 1
+    hsdp_replicate_dim: 1
+    hsdp_shard_dim: 4
+
+  data:
+    data_path: /mnt/h3-dmd2-overfit/data
+    preprocessed_data_type: t2va
+    dataloader_num_workers: 0
+    train_batch_size: 1
+    training_cfg_rate: 0.0
+    seed: 42
+    num_latent_t: 37
+    num_height: 768
+    num_width: 1344
+    num_frames: 124
+
+  # Student VSA sparsity (0.0 = every tile selected, dense parity).
+  vsa:
+    sparsity: 0.0
+
+  optimizer:
+    learning_rate: 5.0e-5
+    betas: [0.9, 0.999]
+    weight_decay: 0.0
+    lr_scheduler: constant
+    lr_warmup_steps: 0
+
+  loop:
+    max_train_steps: 2000
+    gradient_accumulation_steps: 1
+
+  checkpoint:
+    output_dir: outputs/minimax_h3_sft_fa4_overfit
+    # 0 disables training-state checkpointing (no resume; avoids large DCP
+    # saves on the shared volume).
+    training_state_checkpointing_steps: 0
+    checkpoints_total_limit: 3
+
+  tracker:
+    project_name: h3-dmd2-vsa
+    run_name: sft_fa4_overfit
+
+  model:
+    precondition_outputs: false
+    enable_gradient_checkpointing_type: full
+
+  dit_precision: bf16
+
+callbacks:
+  grad_clip:
+    _target_: fastvideo.train.callbacks.grad_clip.GradNormClipCallback
+    max_grad_norm: 1.0
+  validation:
+    _target_: fastvideo.train.callbacks.validation.ValidationCallback
+    pipeline_target: fastvideo.pipelines.basic.minimax_h3.minimax_h3_pipeline.MiniMaxH3Pipeline
+    dataset_file: /mnt/h3-dmd2-overfit/c17_validation.json
+    every_steps: 50
+    # Sample the untrained baseline at step 0.
+    run_at_start: true
+    # SFT is not a few-step generator; sample on the full 50-step schedule
+    # (mirrors the crush-smol overfit validation).
+    sampling_steps: [50]
+    guidance_scale: 1.0
+    num_frames: 124
+    num_videos_per_prompt: 1
+    use_validation_media_conditioning: false
+    offload_training_state: true
+    text_encoder_cpu_offload: true
+    vae_cpu_offload: true
+
+pipeline: {}

examples/train/configs/fine_tuning/minimax_h3/sft_vsa0_overfit.yaml

@@ -0,0 +1,93 @@
+# Single-sample SFT overfit: MiniMax H3 with the VSA-H3 backend at sparsity
+# 0.0 — the fine-tuning counterpart of
+# distribution_matching/minimax_h3/dmd2_vsa0_overfit.yaml (same c17_v2
+# preprocessed record, same validation prompt). Optimizer recipe follows the
+# crush-smol H3 T2VA overfit (overfit_minimax_h3_t2va.yaml), scaled to one
+# 4-GPU node with pure-FSDP sharding.
+
+models:
+  student:
+    _target_: fastvideo.train.models.minimax_h3.MiniMaxH3Model
+    init_from: /mnt/models/MiniMax-H3
+    trainable: true
+    enable_gradient_checkpointing_type: full
+    attention_backend: VIDEO_SPARSE_ATTN_H3
+
+method:
+  _target_: fastvideo.train.methods.fine_tuning.finetune.FineTuneMethod
+
+training:
+  distributed:
+    num_gpus: 4
+    sp_size: 4
+    tp_size: 1
+    hsdp_replicate_dim: 1
+    hsdp_shard_dim: 4
+
+  data:
+    data_path: /mnt/h3-dmd2-overfit/data
+    preprocessed_data_type: t2va
+    dataloader_num_workers: 0
+    train_batch_size: 1
+    training_cfg_rate: 0.0
+    seed: 42
+    num_latent_t: 37
+    num_height: 768
+    num_width: 1344
+    num_frames: 124
+
+  # Student VSA sparsity (0.0 = every tile selected, dense parity).
+  vsa:
+    sparsity: 0.0
+
+  optimizer:
+    learning_rate: 5.0e-5
+    betas: [0.9, 0.999]
+    weight_decay: 0.0
+    lr_scheduler: constant
+    lr_warmup_steps: 0
+
+  loop:
+    max_train_steps: 2000
+    gradient_accumulation_steps: 1
+
+  checkpoint:
+    output_dir: outputs/minimax_h3_sft_vsa0_overfit
+    # 0 disables training-state checkpointing (no resume; avoids large DCP
+    # saves on the shared volume).
+    training_state_checkpointing_steps: 0
+    checkpoints_total_limit: 3
+
+  tracker:
+    project_name: h3-dmd2-vsa
+    run_name: sft_vsa0_overfit
+
+  model:
+    precondition_outputs: false
+    enable_gradient_checkpointing_type: full
+
+  dit_precision: bf16
+
+callbacks:
+  grad_clip:
+    _target_: fastvideo.train.callbacks.grad_clip.GradNormClipCallback
+    max_grad_norm: 1.0
+  validation:
+    _target_: fastvideo.train.callbacks.validation.ValidationCallback
+    pipeline_target: fastvideo.pipelines.basic.minimax_h3.minimax_h3_pipeline.MiniMaxH3Pipeline
+    dataset_file: /mnt/h3-dmd2-overfit/c17_validation.json
+    every_steps: 50
+    # Sample the untrained baseline at step 0.
+    run_at_start: true
+    # SFT is not a few-step generator; sample on the full 50-step schedule
+    # (mirrors the crush-smol overfit validation).
+    sampling_steps: [50]
+    guidance_scale: 1.0
+    num_frames: 124
+    num_videos_per_prompt: 1
+    use_validation_media_conditioning: false
+    offload_training_state: true
+    text_encoder_cpu_offload: true
+    vae_cpu_offload: true
+
+pipeline: {}

examples/train/configs/fine_tuning/minimax_h3/sft_vsa80_overfit.yaml

@@ -0,0 +1,93 @@
+# Single-sample SFT overfit: MiniMax H3 with the VSA-H3 backend at sparsity
+# 0.8 (80% of video tiles pruned) — sparse variant of sft_vsa0_overfit.yaml;
+# distribution_matching/minimax_h3/dmd2_vsa0_overfit.yaml (same c17_v2
+# preprocessed record, same validation prompt). Optimizer recipe follows the
+# crush-smol H3 T2VA overfit (overfit_minimax_h3_t2va.yaml), scaled to one
+# 4-GPU node with pure-FSDP sharding.
+
+models:
+  student:
+    _target_: fastvideo.train.models.minimax_h3.MiniMaxH3Model
+    init_from: /mnt/models/MiniMax-H3
+    trainable: true
+    enable_gradient_checkpointing_type: full
+    attention_backend: VIDEO_SPARSE_ATTN_H3
+
+method:
+  _target_: fastvideo.train.methods.fine_tuning.finetune.FineTuneMethod
+
+training:
+  distributed:
+    num_gpus: 4
+    sp_size: 4
+    tp_size: 1
+    hsdp_replicate_dim: 1
+    hsdp_shard_dim: 4
+
+  data:
+    data_path: /mnt/h3-dmd2-overfit/data
+    preprocessed_data_type: t2va
+    dataloader_num_workers: 0
+    train_batch_size: 1
+    training_cfg_rate: 0.0
+    seed: 42
+    num_latent_t: 37
+    num_height: 768
+    num_width: 1344
+    num_frames: 124
+
+  # Student VSA sparsity (0.8 = top-20% of tiles attended).
+  vsa:
+    sparsity: 0.8
+
+  optimizer:
+    learning_rate: 5.0e-5
+    betas: [0.9, 0.999]
+    weight_decay: 0.0
+    lr_scheduler: constant
+    lr_warmup_steps: 0
+
+  loop:
+    max_train_steps: 2000
+    gradient_accumulation_steps: 1
+
+  checkpoint:
+    output_dir: outputs/minimax_h3_sft_vsa80_overfit
+    # 0 disables training-state checkpointing (no resume; avoids large DCP
+    # saves on the shared volume).
+    training_state_checkpointing_steps: 0
+    checkpoints_total_limit: 3
+
+  tracker:
+    project_name: h3-dmd2-vsa
+    run_name: sft_vsa80_overfit
+
+  model:
+    precondition_outputs: false
+    enable_gradient_checkpointing_type: full
+
+  dit_precision: bf16
+
+callbacks:
+  grad_clip:
+    _target_: fastvideo.train.callbacks.grad_clip.GradNormClipCallback
+    max_grad_norm: 1.0
+  validation:
+    _target_: fastvideo.train.callbacks.validation.ValidationCallback
+    pipeline_target: fastvideo.pipelines.basic.minimax_h3.minimax_h3_pipeline.MiniMaxH3Pipeline
+    dataset_file: /mnt/h3-dmd2-overfit/c17_validation.json
+    every_steps: 50
+    # Sample the untrained baseline at step 0.
+    run_at_start: true
+    # SFT is not a few-step generator; sample on the full 50-step schedule
+    # (mirrors the crush-smol overfit validation).
+    sampling_steps: [50]
+    guidance_scale: 1.0
+    num_frames: 124
+    num_videos_per_prompt: 1
+    use_validation_media_conditioning: false
+    offload_training_state: true
+    text_encoder_cpu_offload: true
+    vae_cpu_offload: true
+
+pipeline: {}

examples/train/configs/fine_tuning/minimax_h3/sft_vsa90_overfit.yaml

@@ -0,0 +1,93 @@
+# Single-sample SFT overfit: MiniMax H3 with the VSA-H3 backend at sparsity
+# 0.9 (90% of video tiles pruned) — sparse variant of sft_vsa0_overfit.yaml;
+# distribution_matching/minimax_h3/dmd2_vsa0_overfit.yaml (same c17_v2
+# preprocessed record, same validation prompt). Optimizer recipe follows the
+# crush-smol H3 T2VA overfit (overfit_minimax_h3_t2va.yaml), scaled to one
+# 4-GPU node with pure-FSDP sharding.
+
+models:
+  student:
+    _target_: fastvideo.train.models.minimax_h3.MiniMaxH3Model
+    init_from: /mnt/models/MiniMax-H3
+    trainable: true
+    enable_gradient_checkpointing_type: full
+    attention_backend: VIDEO_SPARSE_ATTN_H3
+
+method:
+  _target_: fastvideo.train.methods.fine_tuning.finetune.FineTuneMethod
+
+training:
+  distributed:
+    num_gpus: 4
+    sp_size: 4
+    tp_size: 1
+    hsdp_replicate_dim: 1
+    hsdp_shard_dim: 4
+
+  data:
+    data_path: /mnt/h3-dmd2-overfit/data
+    preprocessed_data_type: t2va
+    dataloader_num_workers: 0
+    train_batch_size: 1
+    training_cfg_rate: 0.0
+    seed: 42
+    num_latent_t: 37
+    num_height: 768
+    num_width: 1344
+    num_frames: 124
+
+  # Student VSA sparsity (0.9 = top-10% of tiles attended).
+  vsa:
+    sparsity: 0.9
+
+  optimizer:
+    learning_rate: 5.0e-5
+    betas: [0.9, 0.999]
+    weight_decay: 0.0
+    lr_scheduler: constant
+    lr_warmup_steps: 0
+
+  loop:
+    max_train_steps: 2000
+    gradient_accumulation_steps: 1
+
+  checkpoint:
+    output_dir: outputs/minimax_h3_sft_vsa90_overfit
+    # 0 disables training-state checkpointing (no resume; avoids large DCP
+    # saves on the shared volume).
+    training_state_checkpointing_steps: 0
+    checkpoints_total_limit: 3
+
+  tracker:
+    project_name: h3-dmd2-vsa
+    run_name: sft_vsa90_overfit
+
+  model:
+    precondition_outputs: false
+    enable_gradient_checkpointing_type: full
+
+  dit_precision: bf16
+
+callbacks:
+  grad_clip:
+    _target_: fastvideo.train.callbacks.grad_clip.GradNormClipCallback
+    max_grad_norm: 1.0
+  validation:
+    _target_: fastvideo.train.callbacks.validation.ValidationCallback
+    pipeline_target: fastvideo.pipelines.basic.minimax_h3.minimax_h3_pipeline.MiniMaxH3Pipeline
+    dataset_file: /mnt/h3-dmd2-overfit/c17_validation.json
+    every_steps: 50
+    # Sample the untrained baseline at step 0.
+    run_at_start: true
+    # SFT is not a few-step generator; sample on the full 50-step schedule
+    # (mirrors the crush-smol overfit validation).
+    sampling_steps: [50]
+    guidance_scale: 1.0
+    num_frames: 124
+    num_videos_per_prompt: 1
+    use_validation_media_conditioning: false
+    offload_training_state: true
+    text_encoder_cpu_offload: true
+    vae_cpu_offload: true
+
+pipeline: {}

examples/train/configs/fine_tuning/minimax_h3/sft_vsa95_overfit.yaml

@@ -0,0 +1,93 @@
+# Single-sample SFT overfit: MiniMax H3 with the VSA-H3 backend at sparsity
+# 0.95 (95% of video tiles pruned) — sparse variant of sft_vsa0_overfit.yaml;
+# distribution_matching/minimax_h3/dmd2_vsa0_overfit.yaml (same c17_v2
+# preprocessed record, same validation prompt). Optimizer recipe follows the
+# crush-smol H3 T2VA overfit (overfit_minimax_h3_t2va.yaml), scaled to one
+# 4-GPU node with pure-FSDP sharding.
+
+models:
+  student:
+    _target_: fastvideo.train.models.minimax_h3.MiniMaxH3Model
+    init_from: /mnt/models/MiniMax-H3
+    trainable: true
+    enable_gradient_checkpointing_type: full
+    attention_backend: VIDEO_SPARSE_ATTN_H3
+
+method:
+  _target_: fastvideo.train.methods.fine_tuning.finetune.FineTuneMethod
+
+training:
+  distributed:
+    num_gpus: 4
+    sp_size: 4
+    tp_size: 1
+    hsdp_replicate_dim: 1
+    hsdp_shard_dim: 4
+
+  data:
+    data_path: /mnt/h3-dmd2-overfit/data
+    preprocessed_data_type: t2va
+    dataloader_num_workers: 0
+    train_batch_size: 1
+    training_cfg_rate: 0.0
+    seed: 42
+    num_latent_t: 37
+    num_height: 768
+    num_width: 1344
+    num_frames: 124
+
+  # Student VSA sparsity (0.9 = top-10% of tiles attended).
+  vsa:
+    sparsity: 0.95
+
+  optimizer:
+    learning_rate: 5.0e-5
+    betas: [0.9, 0.999]
+    weight_decay: 0.0
+    lr_scheduler: constant
+    lr_warmup_steps: 0
+
+  loop:
+    max_train_steps: 2000
+    gradient_accumulation_steps: 1
+
+  checkpoint:
+    output_dir: outputs/minimax_h3_sft_vsa95_overfit
+    # 0 disables training-state checkpointing (no resume; avoids large DCP
+    # saves on the shared volume).
+    training_state_checkpointing_steps: 0
+    checkpoints_total_limit: 3
+
+  tracker:
+    project_name: h3-dmd2-vsa
+    run_name: sft_vsa95_overfit
+
+  model:
+    precondition_outputs: false
+    enable_gradient_checkpointing_type: full
+
+  dit_precision: bf16
+
+callbacks:
+  grad_clip:
+    _target_: fastvideo.train.callbacks.grad_clip.GradNormClipCallback
+    max_grad_norm: 1.0
+  validation:
+    _target_: fastvideo.train.callbacks.validation.ValidationCallback
+    pipeline_target: fastvideo.pipelines.basic.minimax_h3.minimax_h3_pipeline.MiniMaxH3Pipeline
+    dataset_file: /mnt/h3-dmd2-overfit/c17_validation.json
+    every_steps: 50
+    # Sample the untrained baseline at step 0.
+    run_at_start: true
+    # SFT is not a few-step generator; sample on the full 50-step schedule
+    # (mirrors the crush-smol overfit validation).
+    sampling_steps: [50]
+    guidance_scale: 1.0
+    num_frames: 124
+    num_videos_per_prompt: 1
+    use_validation_media_conditioning: false
+    offload_training_state: true
+    text_encoder_cpu_offload: true
+    vae_cpu_offload: true
+
+pipeline: {}

examples/train/configs/fine_tuning/minimax_h3/sft_vsa95_overfit_2gpu.yaml

@@ -0,0 +1,93 @@
+# Single-sample SFT overfit: MiniMax H3 with the VSA-H3 backend at sparsity
+# 0.95 (95% of video tiles pruned) — sparse variant of sft_vsa0_overfit.yaml;
+# distribution_matching/minimax_h3/dmd2_vsa0_overfit.yaml (same c17_v2
+# preprocessed record, same validation prompt). Optimizer recipe follows the
+# crush-smol H3 T2VA overfit (overfit_minimax_h3_t2va.yaml), scaled to one
+# 4-GPU node with pure-FSDP sharding.
+
+models:
+  student:
+    _target_: fastvideo.train.models.minimax_h3.MiniMaxH3Model
+    init_from: /mnt/models/MiniMax-H3
+    trainable: true
+    enable_gradient_checkpointing_type: full
+    attention_backend: VIDEO_SPARSE_ATTN_H3
+
+method:
+  _target_: fastvideo.train.methods.fine_tuning.finetune.FineTuneMethod
+
+training:
+  distributed:
+    num_gpus: 2
+    sp_size: 2
+    tp_size: 1
+    hsdp_replicate_dim: 1
+    hsdp_shard_dim: 2
+
+  data:
+    data_path: /mnt/h3-dmd2-overfit/data
+    preprocessed_data_type: t2va
+    dataloader_num_workers: 0
+    train_batch_size: 1
+    training_cfg_rate: 0.0
+    seed: 42
+    num_latent_t: 37
+    num_height: 768
+    num_width: 1344
+    num_frames: 124
+
+  # Student VSA sparsity (0.95 = top-5% of tiles attended).
+  vsa:
+    sparsity: 0.95
+
+  optimizer:
+    learning_rate: 5.0e-5
+    betas: [0.0, 0.999]  # beta1=0 -> buffer-free AdamW; std betas OOM at sp=2
+    weight_decay: 0.0
+    lr_scheduler: constant
+    lr_warmup_steps: 0
+
+  loop:
+    max_train_steps: 2000
+    gradient_accumulation_steps: 1
+
+  checkpoint:
+    output_dir: outputs/minimax_h3_sft_vsa95_overfit
+    # 0 disables training-state checkpointing (no resume; avoids large DCP
+    # saves on the shared volume).
+    training_state_checkpointing_steps: 0
+    checkpoints_total_limit: 3
+
+  tracker:
+    project_name: h3-dmd2-vsa
+    run_name: sft_vsa95_overfit
+
+  model:
+    precondition_outputs: false
+    enable_gradient_checkpointing_type: full
+
+  dit_precision: bf16
+
+callbacks:
+  grad_clip:
+    _target_: fastvideo.train.callbacks.grad_clip.GradNormClipCallback
+    max_grad_norm: 1.0
+  validation:
+    _target_: fastvideo.train.callbacks.validation.ValidationCallback
+    pipeline_target: fastvideo.pipelines.basic.minimax_h3.minimax_h3_pipeline.MiniMaxH3Pipeline
+    dataset_file: /mnt/h3-dmd2-overfit/c17_validation.json
+    every_steps: 50
+    # Sample the untrained baseline at step 0.
+    run_at_start: true
+    # SFT is not a few-step generator; sample on the full 50-step schedule
+    # (mirrors the crush-smol overfit validation).
+    sampling_steps: [50]
+    guidance_scale: 1.0
+    num_frames: 124
+    num_videos_per_prompt: 1
+    use_validation_media_conditioning: false
+    offload_training_state: true
+    text_encoder_cpu_offload: true
+    vae_cpu_offload: true
+
+pipeline: {}

examples/train/configs/fine_tuning/minimax_h3/sft_vsa97_overfit.yaml

@@ -0,0 +1,93 @@
+# Single-sample SFT overfit: MiniMax H3 with the VSA-H3 backend at sparsity
+# 0.97 (97% of video tiles pruned) — sparse variant of sft_vsa0_overfit.yaml;
+# distribution_matching/minimax_h3/dmd2_vsa0_overfit.yaml (same c17_v2
+# preprocessed record, same validation prompt). Optimizer recipe follows the
+# crush-smol H3 T2VA overfit (overfit_minimax_h3_t2va.yaml), scaled to one
+# 4-GPU node with pure-FSDP sharding.
+
+models:
+  student:
+    _target_: fastvideo.train.models.minimax_h3.MiniMaxH3Model
+    init_from: /mnt/models/MiniMax-H3
+    trainable: true
+    enable_gradient_checkpointing_type: full
+    attention_backend: VIDEO_SPARSE_ATTN_H3
+
+method:
+  _target_: fastvideo.train.methods.fine_tuning.finetune.FineTuneMethod
+
+training:
+  distributed:
+    num_gpus: 4
+    sp_size: 4
+    tp_size: 1
+    hsdp_replicate_dim: 1
+    hsdp_shard_dim: 4
+
+  data:
+    data_path: /mnt/h3-dmd2-overfit/data
+    preprocessed_data_type: t2va
+    dataloader_num_workers: 0
+    train_batch_size: 1
+    training_cfg_rate: 0.0
+    seed: 42
+    num_latent_t: 37
+    num_height: 768
+    num_width: 1344
+    num_frames: 124
+
+  # Student VSA sparsity (0.9 = top-10% of tiles attended).
+  vsa:
+    sparsity: 0.97
+
+  optimizer:
+    learning_rate: 5.0e-5
+    betas: [0.9, 0.999]
+    weight_decay: 0.0
+    lr_scheduler: constant
+    lr_warmup_steps: 0
+
+  loop:
+    max_train_steps: 2000
+    gradient_accumulation_steps: 1
+
+  checkpoint:
+    output_dir: outputs/minimax_h3_sft_vsa97_overfit
+    # 0 disables training-state checkpointing (no resume; avoids large DCP
+    # saves on the shared volume).
+    training_state_checkpointing_steps: 0
+    checkpoints_total_limit: 3
+
+  tracker:
+    project_name: h3-dmd2-vsa
+    run_name: sft_vsa97_overfit
+
+  model:
+    precondition_outputs: false
+    enable_gradient_checkpointing_type: full
+
+  dit_precision: bf16
+
+callbacks:
+  grad_clip:
+    _target_: fastvideo.train.callbacks.grad_clip.GradNormClipCallback
+    max_grad_norm: 1.0
+  validation:
+    _target_: fastvideo.train.callbacks.validation.ValidationCallback
+    pipeline_target: fastvideo.pipelines.basic.minimax_h3.minimax_h3_pipeline.MiniMaxH3Pipeline
+    dataset_file: /mnt/h3-dmd2-overfit/c17_validation.json
+    every_steps: 50
+    # Sample the untrained baseline at step 0.
+    run_at_start: true
+    # SFT is not a few-step generator; sample on the full 50-step schedule
+    # (mirrors the crush-smol overfit validation).
+    sampling_steps: [50]
+    guidance_scale: 1.0
+    num_frames: 124
+    num_videos_per_prompt: 1
+    use_validation_media_conditioning: false
+    offload_training_state: true
+    text_encoder_cpu_offload: true
+    vae_cpu_offload: true
+
+pipeline: {}

examples/train/configs/fine_tuning/minimax_h3/sft_vsa97_overfit_2gpu.yaml

@@ -0,0 +1,93 @@
+# Single-sample SFT overfit: MiniMax H3 with the VSA-H3 backend at sparsity
+# 0.97 (97% of video tiles pruned) — sparse variant of sft_vsa0_overfit.yaml;
+# distribution_matching/minimax_h3/dmd2_vsa0_overfit.yaml (same c17_v2
+# preprocessed record, same validation prompt). Optimizer recipe follows the
+# crush-smol H3 T2VA overfit (overfit_minimax_h3_t2va.yaml), scaled to one
+# 4-GPU node with pure-FSDP sharding.
+
+models:
+  student:
+    _target_: fastvideo.train.models.minimax_h3.MiniMaxH3Model
+    init_from: /mnt/models/MiniMax-H3
+    trainable: true
+    enable_gradient_checkpointing_type: full
+    attention_backend: VIDEO_SPARSE_ATTN_H3
+
+method:
+  _target_: fastvideo.train.methods.fine_tuning.finetune.FineTuneMethod
+
+training:
+  distributed:
+    num_gpus: 2
+    sp_size: 2
+    tp_size: 1
+    hsdp_replicate_dim: 1
+    hsdp_shard_dim: 2
+
+  data:
+    data_path: /mnt/h3-dmd2-overfit/data
+    preprocessed_data_type: t2va
+    dataloader_num_workers: 0
+    train_batch_size: 1
+    training_cfg_rate: 0.0
+    seed: 42
+    num_latent_t: 37
+    num_height: 768
+    num_width: 1344
+    num_frames: 124
+
+  # Student VSA sparsity (0.97 = top-3% of tiles attended).
+  vsa:
+    sparsity: 0.97
+
+  optimizer:
+    learning_rate: 5.0e-5
+    betas: [0.0, 0.999]  # beta1=0 -> buffer-free AdamW; std betas OOM at sp=2
+    weight_decay: 0.0
+    lr_scheduler: constant
+    lr_warmup_steps: 0
+
+  loop:
+    max_train_steps: 2000
+    gradient_accumulation_steps: 1
+
+  checkpoint:
+    output_dir: outputs/minimax_h3_sft_vsa97_overfit
+    # 0 disables training-state checkpointing (no resume; avoids large DCP
+    # saves on the shared volume).
+    training_state_checkpointing_steps: 0
+    checkpoints_total_limit: 3
+
+  tracker:
+    project_name: h3-dmd2-vsa
+    run_name: sft_vsa97_overfit
+
+  model:
+    precondition_outputs: false
+    enable_gradient_checkpointing_type: full
+
+  dit_precision: bf16
+
+callbacks:
+  grad_clip:
+    _target_: fastvideo.train.callbacks.grad_clip.GradNormClipCallback
+    max_grad_norm: 1.0
+  validation:
+    _target_: fastvideo.train.callbacks.validation.ValidationCallback
+    pipeline_target: fastvideo.pipelines.basic.minimax_h3.minimax_h3_pipeline.MiniMaxH3Pipeline
+    dataset_file: /mnt/h3-dmd2-overfit/c17_validation.json
+    every_steps: 50
+    # Sample the untrained baseline at step 0.
+    run_at_start: true
+    # SFT is not a few-step generator; sample on the full 50-step schedule
+    # (mirrors the crush-smol overfit validation).
+    sampling_steps: [50]
+    guidance_scale: 1.0
+    num_frames: 124
+    num_videos_per_prompt: 1
+    use_validation_media_conditioning: false
+    offload_training_state: true
+    text_encoder_cpu_offload: true
+    vae_cpu_offload: true
+
+pipeline: {}

examples/train/slurm/dmd2_32xgb200.sbatch

@@ -0,0 +1,79 @@
+#!/bin/bash
+# 8-tray (8 x 4x GB200 = 32 GPU) MiniMax-H3 DMD2 launch from a Slurm login node.
+#
+#   sbatch examples/train/slurm/dmd2_32xgb200.sbatch
+#   sbatch --nodes=2 examples/train/slurm/dmd2_32xgb200.sbatch          # smaller sweep
+#   CONFIG=path/to/other.yaml sbatch examples/train/slurm/dmd2_32xgb200.sbatch
+#
+# The paired config (dmd2_simulate_fa4_32gpu.yaml) assumes 8 nodes:
+# hsdp_replicate_dim must equal the node count and num_gpus the world size —
+# this script forwards overrides derived from the actual allocation, so
+# shrinking --nodes "just works" (global batch shrinks with it).
+#
+# Cluster-specific knobs (partition/account/container) are marked CLUSTER.
+
+#SBATCH --job-name=h3-dmd2-32xgb200
+#SBATCH --nodes=8
+#SBATCH --ntasks-per-node=1
+#SBATCH --gpus-per-node=4
+#SBATCH --exclusive
+#SBATCH --time=48:00:00
+#SBATCH --output=slurm-%x-%j.out
+##SBATCH --partition=<CLUSTER: partition>
+##SBATCH --account=<CLUSTER: account>
+
+set -euo pipefail
+
+# ── paths (shared FS visible on every tray) ──────────────────────────────
+REPO="${REPO:-/mnt/FastVideo}"
+VENV="${VENV:-${REPO}/.venv}"
+CONFIG="${CONFIG:-${REPO}/examples/train/configs/distribution_matching/minimax_h3/dmd2_simulate_fa4_32gpu.yaml}"
+
+# ── topology derived from the allocation ─────────────────────────────────
+NNODES="${SLURM_JOB_NUM_NODES}"
+GPUS_PER_NODE="${SLURM_GPUS_PER_NODE:-4}"
+GPUS_PER_NODE="${GPUS_PER_NODE##*:}"   # "4" or "gb200:4" -> 4
+WORLD_SIZE=$((NNODES * GPUS_PER_NODE))
+MASTER_ADDR="$(scontrol show hostnames "${SLURM_JOB_NODELIST}" | head -n 1)"
+MASTER_PORT="${MASTER_PORT:-29501}"
+
+echo "=== H3 DMD2 multi-node launch ==="
+echo "Nodes:        ${NNODES} (${SLURM_JOB_NODELIST})"
+echo "World size:   ${WORLD_SIZE}"
+echo "Rendezvous:   ${MASTER_ADDR}:${MASTER_PORT}"
+echo "Config:       ${CONFIG}"
+
+# ── per-rank environment (srun propagates exported vars) ─────────────────
+export FASTVIDEO_FA4=1
+export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
+export TOKENIZERS_PARALLELISM=false
+# Node-local weight cache: relaunches rebuild all three 33B roles from tmpfs
+# in seconds. PER_NODE=1 makes every tray write its own /dev/shm copy
+# (single-writer semantics would leave 7 of 8 trays permanently cold).
+export FASTVIDEO_WEIGHT_SHARD_CACHE=/dev/shm/fastvideo-wcache
+export FASTVIDEO_WEIGHT_SHARD_CACHE_PER_NODE=1
+# W&B: credentials come from ~/.netrc on the shared home; flip to offline if
+# compute nodes have no egress.
+export WANDB_MODE="${WANDB_MODE:-online}"
+# CLUSTER: NCCL fabric tuning if needed (NVLink partitioning, IB HCA choice):
+# export NCCL_IB_HCA=...  NCCL_SOCKET_IFNAME=...
+
+export NNODES MASTER_ADDR MASTER_PORT
+export NUM_GPUS="${GPUS_PER_NODE}"
+
+# CLUSTER: if trays require a container, wrap the srun payload accordingly,
+# e.g. add `--container-image=... --container-mounts=/mnt:/mnt` (pyxis) so
+# /mnt (repo, models, data) and /dev/shm are available inside.
+srun --kill-on-bad-exit=1 --ntasks-per-node=1 bash -c "
+  set -euo pipefail
+  export NODE_RANK=\${SLURM_NODEID}
+  # One log dir per node — parallel tee writers on one NFS file garble it.
+  export LOG_DIR=examples/train/logs/\${SLURM_JOB_ID}-node\${SLURM_NODEID}
+  source '${VENV}/bin/activate'
+  cd '${REPO}'
+  bash examples/train/run.sh '${CONFIG}' \
+    --training.distributed.num_gpus ${WORLD_SIZE} \
+    --training.distributed.hsdp_replicate_dim ${NNODES} \
+    --training.distributed.hsdp_shard_dim ${GPUS_PER_NODE} \
+    --training.distributed.sp_size ${GPUS_PER_NODE}
+"

pyproject.toml

@@ -208,7 +208,8 @@ dreamverse-mock-server = "dreamverse.mock_server:cli"
 
 [tool.setuptools.packages.find]
 where = [".", "apps/dreamverse"]
-exclude = ["assets*", "docker*", "docs", "scripts*", "apps*", "dreamverse.tests*"]
+include = ["fastvideo", "fastvideo.*", "dreamverse", "dreamverse.*"]
+exclude = ["dreamverse.tests*"]
 
 [tool.setuptools.package-data]
 dreamverse = ["prompts/*.md"]