diff --git a/examples/train/configs/distribution_matching/minimax_h3/README.md b/examples/train/configs/distribution_matching/minimax_h3/README.md new file mode 100644 index 000000000..1dc1d8d6c --- /dev/null +++ b/examples/train/configs/distribution_matching/minimax_h3/README.md @@ -0,0 +1,87 @@ +# MiniMax-H3 training configs (DMD2 distillation + SFT overfit) + +Configs for training the 33B dual-modality (video+audio) MiniMax-H3 DiT with +the modular trainer (`fastvideo/train/`), validated on 4x GB200 (192 GB, +aarch64). SFT overfit configs live in +[`../../fine_tuning/minimax_h3/`](../../fine_tuning/minimax_h3/); the 8-tray +Slurm launch is [`../../../slurm/dmd2_32xgb200.sbatch`](../../../slurm/dmd2_32xgb200.sbatch). + +## Configs + +| Config | What it is | +|---|---| +| `dmd2_vsa0_overfit.yaml` | Single-sample DMD2, student on VIDEO_SPARSE_ATTN_H3 at sparsity 0 (parity vs dense), teacher/critic on FLASH_ATTN/FA4, `rollout_mode: data_latent` | +| `dmd2_simulate_fa4_overfit.yaml` | **Data-free** DMD2 (`rollout_mode: simulate`): student rolls out from pure noise through `dmd_denoising_steps`; parquet VAE latents ignored, only text conditioning used. All roles dense FA4 | +| `dmd2_simulate_fa4_32gpu.yaml` | The simulate recipe scaled to 8 trays x 4 GPUs, global batch 64 | +| `../../fine_tuning/minimax_h3/sft_*.yaml` | SFT overfit family: FA4 dense control, VSA at sparsity 0 / 0.8 / 0.9 / 0.95 / 0.97, and `sft_fa4_bs2_overfit.yaml` (effective batch 2 over two videos) | + +## Launch + +```bash +# single node (4x GB200) +FASTVIDEO_FA4=1 \ +FASTVIDEO_WEIGHT_SHARD_CACHE=/dev/shm/fastvideo-wcache \ +PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \ +bash examples/train/run.sh examples/train/configs/distribution_matching/minimax_h3/dmd2_simulate_fa4_overfit.yaml + +# 8 trays / 32 GPUs from a Slurm login node (global batch 64) +sbatch examples/train/slurm/dmd2_32xgb200.sbatch +``` + +`FASTVIDEO_FA4=1` selects the FA4 path inside the FLASH_ATTN backend and is +required for the dense roles. Per-role backends come from +`models..attention_backend` — do not set `FASTVIDEO_ATTENTION_BACKEND` +globally. + +## Topology and batch semantics + +- `sp_size: 4` — one packed `[text|cond|audio|video]` document is + sequence-parallel across the 4 GPUs of a node; HSDP mesh + `(replicate=nodes, shard=4)` keeps parameters sharded only inside a node + (~145 GiB/rank for DMD2's three 33B roles with full gradient checkpointing). +- **The H3 packed pipeline is batch-1 per forward by design** (documents with + different caption lengths cannot stack). Batching composes through + `training.loop.gradient_accumulation_steps`, which the trainer loops + natively — global batch = DP groups x accumulation rounds. The 32-GPU + config's global 64 = 8 DP groups x 8 rounds; validated locally as + effective-batch-2 (`sft_fa4_bs2_overfit.yaml`, two videos, both losses + declining, ~10 s/step = 2 micro fwd+bwd). +- DMD2 TTUR: critic every step, generator every + `generator_update_interval`-th. `rollout_mode: simulate` costs ~4 dense + forwards per critic step and ~7 + 2 backwards per generator step + (~10/20 s at 1344x768x124f on GB200). + +## Measured reference points (1344x768, 124 frames, sp=4) + +- SFT step time: FA4 dense 4.9 s/it; VSA 90% 4.8; 95% 4.49; 97% 4.33; + VSA-at-density-1 11.2 (VSA machinery overhead — attention is only ~1/3 of + step FLOPs, so sparsity is mostly an inference win: 3-step validation + sample ~120 s dense vs ~59 s at 95%). +- 2x GB200 does NOT fit this model/sequence without CPU offload: the first + backward alone allocates ~178 GiB at sp=2 (params 31 + grads 31 + ~116 + working set) before optimizer states exist. +- `betas: [0.0, 0.999]` auto-selects a buffer-free AdamW + (`fastvideo/train/utils/optimizer.py`), saving one full set of optimizer + state (~15.4 GiB/rank/model) — required for DMD2's two trainable roles. + +## Weight shard cache + +`FASTVIDEO_WEIGHT_SHARD_CACHE=` caches each rank's post-shard +DTensor chunks after the first full load +(`fastvideo/models/loader/shard_cache.py`); relaunches rebuild each 33B role +in ~2-4 s instead of ~10 min of cold-NFS reads (DMD2's three roles share one +entry). On multi-node runs with node-local cache dirs set +`FASTVIDEO_WEIGHT_SHARD_CACHE_PER_NODE=1` (the sbatch does) so every node +writes its own copy. Any validation failure degrades to the normal full load. + +## Data + +Single/two-sample overfit parquets are produced by +`fastvideo/pipelines/preprocess/preprocess_minimax_h3_overfit.py` (one mp4 +with soundtrack + prompt -> one t2va row; >=124 frames at 24 fps and an audio +track are required). The preprocessor writes exactly one +`data_00000.parquet` per output dir — build multi-sample sets by copying rows +into one dir as `data_00000.parquet`, `data_00001.parquet`, ... The +validation callback samples every prompt in `callbacks.validation.dataset_file` +at the method's exact training sigmas (`dmd_denoising_steps` is injected into +the validation pipeline). diff --git a/examples/train/configs/distribution_matching/minimax_h3/dmd2_simulate_fa4_32gpu.yaml b/examples/train/configs/distribution_matching/minimax_h3/dmd2_simulate_fa4_32gpu.yaml new file mode 100644 index 000000000..4e1054ff7 --- /dev/null +++ b/examples/train/configs/distribution_matching/minimax_h3/dmd2_simulate_fa4_32gpu.yaml @@ -0,0 +1,123 @@ +# 32x GB200 (8 trays x 4 GPUs) data-free DMD2 on MiniMax H3, all-dense FA4. +# +# Topology: sp_size=4 within a tray (one packed document per SP group), HSDP +# mesh (replicate=8, shard=4) — params sharded only inside a tray (keeps the +# proven ~145 GiB/rank profile from the 4-GPU runs), gradients replicated +# across trays. 32/4 = 8 data-parallel groups. +# +# Global batch 64 = 8 DP groups x gradient_accumulation_steps 8 x micro-batch +# 1. H3's packed-sequence pipeline is batch-1 per forward by design (one +# interleaved [text|cond|audio|video] document); batching composes through +# accumulation, which the trainer loops natively. +# +# Launch: sbatch examples/train/slurm/dmd2_32xgb200.sbatch +# +# NOTE: data_path below still points at the 2-row overfit set — good for a +# 32-GPU smoke test only. Swap in the real preprocessed t2va dataset before a +# production run. + +models: + student: + _target_: fastvideo.train.models.minimax_h3.MiniMaxH3DMDModel + init_from: /mnt/models/MiniMax-H3 + trainable: true + enable_gradient_checkpointing_type: full + attention_backend: FLASH_ATTN + teacher: + _target_: fastvideo.train.models.minimax_h3.MiniMaxH3DMDModel + init_from: /mnt/models/MiniMax-H3 + trainable: false + disable_custom_init_weights: true + attention_backend: FLASH_ATTN + critic: + _target_: fastvideo.train.models.minimax_h3.MiniMaxH3DMDModel + init_from: /mnt/models/MiniMax-H3 + trainable: true + disable_custom_init_weights: true + enable_gradient_checkpointing_type: full + attention_backend: FLASH_ATTN + +method: + _target_: fastvideo.train.methods.distribution_matching.dmd2.DMD2Method + rollout_mode: simulate + generator_update_interval: 5 + real_score_guidance_scale: 3.5 + dmd_denoising_steps: [1000, 757, 522] + min_timestep_ratio: 0.02 + max_timestep_ratio: 0.98 + cfg_uncond: + text: zero + + fake_score_learning_rate: 8.0e-6 + fake_score_betas: [0.0, 0.999] + fake_score_lr_scheduler: constant + +training: + distributed: + num_gpus: 32 + sp_size: 4 + tp_size: 1 + hsdp_replicate_dim: 8 + hsdp_shard_dim: 4 + + data: + data_path: /mnt/h3-dmd2-overfit/data_bs2 # smoke-test set; swap for real data + preprocessed_data_type: t2va + dataloader_num_workers: 0 + train_batch_size: 1 + training_cfg_rate: 0.0 + seed: 42 + num_latent_t: 37 + num_height: 768 + num_width: 1344 + num_frames: 124 + + optimizer: + learning_rate: 2.0e-6 + betas: [0.0, 0.999] + weight_decay: 0.01 + lr_scheduler: constant + lr_warmup_steps: 0 + + loop: + max_train_steps: 4000 + # 8 accumulation rounds x 8 DP groups = global batch 64 per optimizer step. + gradient_accumulation_steps: 8 + + checkpoint: + output_dir: outputs/minimax_h3_dmd2_simulate_fa4_32gpu + # Multi-node runs should checkpoint; DCP saves are large — point + # output_dir at a volume with headroom or set back to 0 to disable. + training_state_checkpointing_steps: 500 + checkpoints_total_limit: 3 + + tracker: + project_name: h3-dmd2-vsa + run_name: dmd2_simulate_fa4_32gpu + + model: + precondition_outputs: false + enable_gradient_checkpointing_type: full + + dit_precision: bf16 + +callbacks: + grad_clip: + _target_: fastvideo.train.callbacks.grad_clip.GradNormClipCallback + max_grad_norm: 1.0 + validation: + _target_: fastvideo.train.callbacks.validation.ValidationCallback + pipeline_target: fastvideo.pipelines.basic.minimax_h3.minimax_h3_pipeline.MiniMaxH3Pipeline + dataset_file: /mnt/h3-dmd2-overfit/c17_bs2_validation.json + every_steps: 250 + run_at_start: true + sampling_steps: [3] + guidance_scale: 1.0 + num_frames: 124 + num_videos_per_prompt: 1 + use_validation_media_conditioning: false + offload_training_state: true + text_encoder_cpu_offload: true + vae_cpu_offload: true + +pipeline: {} diff --git a/examples/train/configs/distribution_matching/minimax_h3/dmd2_simulate_fa4_overfit.yaml b/examples/train/configs/distribution_matching/minimax_h3/dmd2_simulate_fa4_overfit.yaml new file mode 100644 index 000000000..1bad6c6f1 --- /dev/null +++ b/examples/train/configs/distribution_matching/minimax_h3/dmd2_simulate_fa4_overfit.yaml @@ -0,0 +1,130 @@ +# Single-sample DMD2 DATA-FREE overfit: rollout_mode=simulate on MiniMax H3 +# with FLASH_ATTN (FA4) for every role — no VSA in the loop. +# +# - simulate rollout: the student builds its own trajectory from pure noise +# through dmd_denoising_steps (no-grad prefix steps, one with-grad step at a +# sampled target index). VAE latents in the parquet are ignored (zeros +# latents sized from training.data dims); only the text conditioning is +# consumed. +# - Student/teacher/critic: FLASH_ATTN (run with FASTVIDEO_FA4=1). DMD2 still +# passes attn_kind="vsa" for the student rollout, which degrades to dense +# because attn_metadata_vsa stays None for non-VSA backends. +# - Data: same ONE preprocessed t2va parquet row (768x1344, 124 frames @ 24 +# fps) as dmd2_vsa0_overfit.yaml; see examples/distill/MiniMax-H3/. + +models: + student: + _target_: fastvideo.train.models.minimax_h3.MiniMaxH3DMDModel + init_from: /mnt/models/MiniMax-H3 + trainable: true + enable_gradient_checkpointing_type: full + attention_backend: FLASH_ATTN + teacher: + _target_: fastvideo.train.models.minimax_h3.MiniMaxH3DMDModel + init_from: /mnt/models/MiniMax-H3 + trainable: false + disable_custom_init_weights: true + attention_backend: FLASH_ATTN + critic: + _target_: fastvideo.train.models.minimax_h3.MiniMaxH3DMDModel + init_from: /mnt/models/MiniMax-H3 + trainable: true + disable_custom_init_weights: true + enable_gradient_checkpointing_type: full + attention_backend: FLASH_ATTN + +method: + _target_: fastvideo.train.methods.distribution_matching.dmd2.DMD2Method + rollout_mode: simulate + generator_update_interval: 5 + real_score_guidance_scale: 3.5 + dmd_denoising_steps: [1000, 757, 522] + min_timestep_ratio: 0.02 + max_timestep_ratio: 0.98 + # H3 has no negative-prompt encoder at training time; unconditional teacher + # forwards zero the text embeddings instead. + cfg_uncond: + text: zero + + # Critic optimizer (required — no fallback to training.optimizer) + fake_score_learning_rate: 8.0e-6 + fake_score_betas: [0.0, 0.999] + fake_score_lr_scheduler: constant + +training: + distributed: + num_gpus: 4 + sp_size: 4 + tp_size: 1 + hsdp_replicate_dim: 1 + hsdp_shard_dim: 4 + + data: + # Preprocessed-latents parquet dir; simulate mode only reads the text + # conditioning from it (latents are zeros sized from the dims below). + data_path: /mnt/h3-dmd2-overfit/data + preprocessed_data_type: t2va + dataloader_num_workers: 0 + train_batch_size: 1 + training_cfg_rate: 0.0 + seed: 42 + num_latent_t: 37 + num_height: 768 + num_width: 1344 + num_frames: 124 + + optimizer: + learning_rate: 2.0e-6 + betas: [0.0, 0.999] + weight_decay: 0.01 + lr_scheduler: constant + lr_warmup_steps: 0 + + loop: + max_train_steps: 2000 + gradient_accumulation_steps: 1 + + checkpoint: + output_dir: outputs/minimax_h3_dmd2_simulate_fa4_overfit + # 0 disables training-state checkpointing (no resume; avoids ~400+ GiB + # DCP saves on the 87%-full volume). + training_state_checkpointing_steps: 0 + checkpoints_total_limit: 3 + + tracker: + project_name: h3-dmd2-vsa + run_name: dmd2_simulate_fa4_overfit + + model: + precondition_outputs: false + enable_gradient_checkpointing_type: full + + dit_precision: bf16 + +callbacks: + grad_clip: + _target_: fastvideo.train.callbacks.grad_clip.GradNormClipCallback + max_grad_norm: 1.0 + validation: + _target_: fastvideo.train.callbacks.validation.ValidationCallback + pipeline_target: fastvideo.pipelines.basic.minimax_h3.minimax_h3_pipeline.MiniMaxH3Pipeline + dataset_file: /mnt/h3-dmd2-overfit/c17_validation.json + every_steps: 50 + # Sample the untrained baseline at step 0 (costs the validation pipeline + # build upfront: text-encoder + VAE loads before the first train step). + run_at_start: true + sampling_steps: [3] + guidance_scale: 1.0 + num_frames: 124 + num_videos_per_prompt: 1 + use_validation_media_conditioning: false + offload_training_state: true + text_encoder_cpu_offload: true + vae_cpu_offload: true + # EMA disabled: its GPU shadow costs ~15.4 GiB/rank (OOM on 184 GiB cards). + # ema: + # _target_: fastvideo.train.callbacks.ema.EMACallback + # decay: 0.98 + # start_iter: 0 + +pipeline: {} diff --git a/examples/train/configs/distribution_matching/minimax_h3/dmd2_vsa0_overfit.yaml b/examples/train/configs/distribution_matching/minimax_h3/dmd2_vsa0_overfit.yaml index fb907d8a7..d5bccb7e2 100644 --- a/examples/train/configs/distribution_matching/minimax_h3/dmd2_vsa0_overfit.yaml +++ b/examples/train/configs/distribution_matching/minimax_h3/dmd2_vsa0_overfit.yaml @@ -13,19 +13,19 @@ models: student: _target_: fastvideo.train.models.minimax_h3.MiniMaxH3DMDModel - init_from: MiniMaxAI/MiniMax-H3 + init_from: /mnt/models/MiniMax-H3 trainable: true enable_gradient_checkpointing_type: full attention_backend: VIDEO_SPARSE_ATTN_H3 teacher: _target_: fastvideo.train.models.minimax_h3.MiniMaxH3DMDModel - init_from: MiniMaxAI/MiniMax-H3 + init_from: /mnt/models/MiniMax-H3 trainable: false disable_custom_init_weights: true attention_backend: FLASH_ATTN critic: _target_: fastvideo.train.models.minimax_h3.MiniMaxH3DMDModel - init_from: MiniMaxAI/MiniMax-H3 + init_from: /mnt/models/MiniMax-H3 trainable: true disable_custom_init_weights: true enable_gradient_checkpointing_type: full @@ -87,7 +87,9 @@ training: checkpoint: output_dir: outputs/minimax_h3_dmd2_vsa0_overfit - training_state_checkpointing_steps: 250 + # 0 disables training-state checkpointing (no resume; avoids ~400+ GiB + # DCP saves on the 87%-full volume). + training_state_checkpointing_steps: 0 checkpoints_total_limit: 3 tracker: @@ -107,8 +109,11 @@ callbacks: validation: _target_: fastvideo.train.callbacks.validation.ValidationCallback pipeline_target: fastvideo.pipelines.basic.minimax_h3.minimax_h3_pipeline.MiniMaxH3Pipeline - dataset_file: examples/training/finetune/Wan2.1-Fun-1.3B-InP/crush_smol/validation.json - every_steps: 250 + dataset_file: /mnt/h3-dmd2-overfit/c17_validation.json + every_steps: 50 + # Sample the untrained baseline at step 0 (costs the validation pipeline + # build upfront: text-encoder + VAE loads before the first train step). + run_at_start: true sampling_steps: [3] guidance_scale: 1.0 num_frames: 124 @@ -117,9 +122,12 @@ callbacks: offload_training_state: true text_encoder_cpu_offload: true vae_cpu_offload: true - ema: - _target_: fastvideo.train.callbacks.ema.EMACallback - decay: 0.98 - start_iter: 0 + # EMA disabled: its GPU shadow costs ~15.4 GiB/rank, which put the student + # update step ~1 GiB over the 184 GiB cards (OOM at step 5/10). Re-enable + # once the shadow can live on CPU. + # ema: + # _target_: fastvideo.train.callbacks.ema.EMACallback + # decay: 0.98 + # start_iter: 0 pipeline: {} diff --git a/examples/train/configs/fine_tuning/minimax_h3/README.md b/examples/train/configs/fine_tuning/minimax_h3/README.md new file mode 100644 index 000000000..8a27caf2d --- /dev/null +++ b/examples/train/configs/fine_tuning/minimax_h3/README.md @@ -0,0 +1,13 @@ +# MiniMax-H3 SFT overfit configs + +Single- and two-sample SFT overfit runs used to validate the VSA-H3 backend, +sparsity sweep (0 / 0.8 / 0.9 / 0.95 / 0.97), the FA4 dense control, and +effective-batch-2 via gradient accumulation (`sft_fa4_bs2_overfit.yaml`). + +Launch, topology, batch semantics, measured step times, and the weight shard +cache are documented in +[`../../distribution_matching/minimax_h3/README.md`](../../distribution_matching/minimax_h3/README.md). + +Note: `sft_vsa95_overfit_2gpu.yaml` / `sft_vsa97_overfit_2gpu.yaml` are kept +as negative results — 2x GB200 cannot fit this model/sequence without CPU +offload (backward-pass working set alone OOMs 184 GiB cards at sp=2). diff --git a/examples/train/configs/fine_tuning/minimax_h3/sft_fa4_bs2_overfit.yaml b/examples/train/configs/fine_tuning/minimax_h3/sft_fa4_bs2_overfit.yaml new file mode 100644 index 000000000..4081b059d --- /dev/null +++ b/examples/train/configs/fine_tuning/minimax_h3/sft_fa4_bs2_overfit.yaml @@ -0,0 +1,93 @@ +# Single-sample SFT overfit: MiniMax H3 with the VSA-H3 backend at sparsity +# FA4 dense control (student on FLASH_ATTN, run with FASTVIDEO_FA4=1) for +# distribution_matching/minimax_h3/dmd2_vsa0_overfit.yaml (same c17_v2 +# preprocessed record, same validation prompt). Optimizer recipe follows the +# crush-smol H3 T2VA overfit (overfit_minimax_h3_t2va.yaml), scaled to one +# 4-GPU node with pure-FSDP sharding. + +models: + student: + _target_: fastvideo.train.models.minimax_h3.MiniMaxH3Model + init_from: /mnt/models/MiniMax-H3 + trainable: true + enable_gradient_checkpointing_type: full + attention_backend: FLASH_ATTN + +method: + _target_: fastvideo.train.methods.fine_tuning.finetune.FineTuneMethod + +training: + distributed: + num_gpus: 4 + sp_size: 4 + tp_size: 1 + hsdp_replicate_dim: 1 + hsdp_shard_dim: 4 + + data: + data_path: /mnt/h3-dmd2-overfit/data_bs2 + preprocessed_data_type: t2va + dataloader_num_workers: 0 + train_batch_size: 1 # per-micro-batch; effective batch 2 via grad accumulation + training_cfg_rate: 0.0 + seed: 42 + num_latent_t: 37 + num_height: 768 + num_width: 1344 + num_frames: 124 + + # Student VSA sparsity (0.0 = every tile selected, dense parity). + vsa: + sparsity: 0.0 + + optimizer: + learning_rate: 5.0e-5 + betas: [0.9, 0.999] + weight_decay: 0.0 + lr_scheduler: constant + lr_warmup_steps: 0 + + loop: + max_train_steps: 2000 + gradient_accumulation_steps: 2 + + checkpoint: + output_dir: outputs/minimax_h3_sft_fa4_bs2_overfit + # 0 disables training-state checkpointing (no resume; avoids large DCP + # saves on the shared volume). + training_state_checkpointing_steps: 0 + checkpoints_total_limit: 3 + + tracker: + project_name: h3-dmd2-vsa + run_name: sft_fa4_bs2_overfit + + model: + precondition_outputs: false + enable_gradient_checkpointing_type: full + + dit_precision: bf16 + +callbacks: + grad_clip: + _target_: fastvideo.train.callbacks.grad_clip.GradNormClipCallback + max_grad_norm: 1.0 + validation: + _target_: fastvideo.train.callbacks.validation.ValidationCallback + pipeline_target: fastvideo.pipelines.basic.minimax_h3.minimax_h3_pipeline.MiniMaxH3Pipeline + dataset_file: /mnt/h3-dmd2-overfit/c17_bs2_validation.json + every_steps: 50 + # Sample the untrained baseline at step 0. + run_at_start: true + # SFT is not a few-step generator; sample on the full 50-step schedule + # (mirrors the crush-smol overfit validation). + sampling_steps: [50] + guidance_scale: 1.0 + num_frames: 124 + num_videos_per_prompt: 1 + use_validation_media_conditioning: false + offload_training_state: true + text_encoder_cpu_offload: true + vae_cpu_offload: true + +pipeline: {} diff --git a/examples/train/configs/fine_tuning/minimax_h3/sft_fa4_overfit.yaml b/examples/train/configs/fine_tuning/minimax_h3/sft_fa4_overfit.yaml new file mode 100644 index 000000000..5ee0d30ea --- /dev/null +++ b/examples/train/configs/fine_tuning/minimax_h3/sft_fa4_overfit.yaml @@ -0,0 +1,93 @@ +# Single-sample SFT overfit: MiniMax H3 with the VSA-H3 backend at sparsity +# FA4 dense control (student on FLASH_ATTN, run with FASTVIDEO_FA4=1) for +# distribution_matching/minimax_h3/dmd2_vsa0_overfit.yaml (same c17_v2 +# preprocessed record, same validation prompt). Optimizer recipe follows the +# crush-smol H3 T2VA overfit (overfit_minimax_h3_t2va.yaml), scaled to one +# 4-GPU node with pure-FSDP sharding. + +models: + student: + _target_: fastvideo.train.models.minimax_h3.MiniMaxH3Model + init_from: /mnt/models/MiniMax-H3 + trainable: true + enable_gradient_checkpointing_type: full + attention_backend: FLASH_ATTN + +method: + _target_: fastvideo.train.methods.fine_tuning.finetune.FineTuneMethod + +training: + distributed: + num_gpus: 4 + sp_size: 4 + tp_size: 1 + hsdp_replicate_dim: 1 + hsdp_shard_dim: 4 + + data: + data_path: /mnt/h3-dmd2-overfit/data + preprocessed_data_type: t2va + dataloader_num_workers: 0 + train_batch_size: 1 + training_cfg_rate: 0.0 + seed: 42 + num_latent_t: 37 + num_height: 768 + num_width: 1344 + num_frames: 124 + + # Student VSA sparsity (0.0 = every tile selected, dense parity). + vsa: + sparsity: 0.0 + + optimizer: + learning_rate: 5.0e-5 + betas: [0.9, 0.999] + weight_decay: 0.0 + lr_scheduler: constant + lr_warmup_steps: 0 + + loop: + max_train_steps: 2000 + gradient_accumulation_steps: 1 + + checkpoint: + output_dir: outputs/minimax_h3_sft_fa4_overfit + # 0 disables training-state checkpointing (no resume; avoids large DCP + # saves on the shared volume). + training_state_checkpointing_steps: 0 + checkpoints_total_limit: 3 + + tracker: + project_name: h3-dmd2-vsa + run_name: sft_fa4_overfit + + model: + precondition_outputs: false + enable_gradient_checkpointing_type: full + + dit_precision: bf16 + +callbacks: + grad_clip: + _target_: fastvideo.train.callbacks.grad_clip.GradNormClipCallback + max_grad_norm: 1.0 + validation: + _target_: fastvideo.train.callbacks.validation.ValidationCallback + pipeline_target: fastvideo.pipelines.basic.minimax_h3.minimax_h3_pipeline.MiniMaxH3Pipeline + dataset_file: /mnt/h3-dmd2-overfit/c17_validation.json + every_steps: 50 + # Sample the untrained baseline at step 0. + run_at_start: true + # SFT is not a few-step generator; sample on the full 50-step schedule + # (mirrors the crush-smol overfit validation). + sampling_steps: [50] + guidance_scale: 1.0 + num_frames: 124 + num_videos_per_prompt: 1 + use_validation_media_conditioning: false + offload_training_state: true + text_encoder_cpu_offload: true + vae_cpu_offload: true + +pipeline: {} diff --git a/examples/train/configs/fine_tuning/minimax_h3/sft_vsa0_overfit.yaml b/examples/train/configs/fine_tuning/minimax_h3/sft_vsa0_overfit.yaml new file mode 100644 index 000000000..5bff4f466 --- /dev/null +++ b/examples/train/configs/fine_tuning/minimax_h3/sft_vsa0_overfit.yaml @@ -0,0 +1,93 @@ +# Single-sample SFT overfit: MiniMax H3 with the VSA-H3 backend at sparsity +# 0.0 — the fine-tuning counterpart of +# distribution_matching/minimax_h3/dmd2_vsa0_overfit.yaml (same c17_v2 +# preprocessed record, same validation prompt). Optimizer recipe follows the +# crush-smol H3 T2VA overfit (overfit_minimax_h3_t2va.yaml), scaled to one +# 4-GPU node with pure-FSDP sharding. + +models: + student: + _target_: fastvideo.train.models.minimax_h3.MiniMaxH3Model + init_from: /mnt/models/MiniMax-H3 + trainable: true + enable_gradient_checkpointing_type: full + attention_backend: VIDEO_SPARSE_ATTN_H3 + +method: + _target_: fastvideo.train.methods.fine_tuning.finetune.FineTuneMethod + +training: + distributed: + num_gpus: 4 + sp_size: 4 + tp_size: 1 + hsdp_replicate_dim: 1 + hsdp_shard_dim: 4 + + data: + data_path: /mnt/h3-dmd2-overfit/data + preprocessed_data_type: t2va + dataloader_num_workers: 0 + train_batch_size: 1 + training_cfg_rate: 0.0 + seed: 42 + num_latent_t: 37 + num_height: 768 + num_width: 1344 + num_frames: 124 + + # Student VSA sparsity (0.0 = every tile selected, dense parity). + vsa: + sparsity: 0.0 + + optimizer: + learning_rate: 5.0e-5 + betas: [0.9, 0.999] + weight_decay: 0.0 + lr_scheduler: constant + lr_warmup_steps: 0 + + loop: + max_train_steps: 2000 + gradient_accumulation_steps: 1 + + checkpoint: + output_dir: outputs/minimax_h3_sft_vsa0_overfit + # 0 disables training-state checkpointing (no resume; avoids large DCP + # saves on the shared volume). + training_state_checkpointing_steps: 0 + checkpoints_total_limit: 3 + + tracker: + project_name: h3-dmd2-vsa + run_name: sft_vsa0_overfit + + model: + precondition_outputs: false + enable_gradient_checkpointing_type: full + + dit_precision: bf16 + +callbacks: + grad_clip: + _target_: fastvideo.train.callbacks.grad_clip.GradNormClipCallback + max_grad_norm: 1.0 + validation: + _target_: fastvideo.train.callbacks.validation.ValidationCallback + pipeline_target: fastvideo.pipelines.basic.minimax_h3.minimax_h3_pipeline.MiniMaxH3Pipeline + dataset_file: /mnt/h3-dmd2-overfit/c17_validation.json + every_steps: 50 + # Sample the untrained baseline at step 0. + run_at_start: true + # SFT is not a few-step generator; sample on the full 50-step schedule + # (mirrors the crush-smol overfit validation). + sampling_steps: [50] + guidance_scale: 1.0 + num_frames: 124 + num_videos_per_prompt: 1 + use_validation_media_conditioning: false + offload_training_state: true + text_encoder_cpu_offload: true + vae_cpu_offload: true + +pipeline: {} diff --git a/examples/train/configs/fine_tuning/minimax_h3/sft_vsa80_overfit.yaml b/examples/train/configs/fine_tuning/minimax_h3/sft_vsa80_overfit.yaml new file mode 100644 index 000000000..97e4c995e --- /dev/null +++ b/examples/train/configs/fine_tuning/minimax_h3/sft_vsa80_overfit.yaml @@ -0,0 +1,93 @@ +# Single-sample SFT overfit: MiniMax H3 with the VSA-H3 backend at sparsity +# 0.8 (80% of video tiles pruned) — sparse variant of sft_vsa0_overfit.yaml; +# distribution_matching/minimax_h3/dmd2_vsa0_overfit.yaml (same c17_v2 +# preprocessed record, same validation prompt). Optimizer recipe follows the +# crush-smol H3 T2VA overfit (overfit_minimax_h3_t2va.yaml), scaled to one +# 4-GPU node with pure-FSDP sharding. + +models: + student: + _target_: fastvideo.train.models.minimax_h3.MiniMaxH3Model + init_from: /mnt/models/MiniMax-H3 + trainable: true + enable_gradient_checkpointing_type: full + attention_backend: VIDEO_SPARSE_ATTN_H3 + +method: + _target_: fastvideo.train.methods.fine_tuning.finetune.FineTuneMethod + +training: + distributed: + num_gpus: 4 + sp_size: 4 + tp_size: 1 + hsdp_replicate_dim: 1 + hsdp_shard_dim: 4 + + data: + data_path: /mnt/h3-dmd2-overfit/data + preprocessed_data_type: t2va + dataloader_num_workers: 0 + train_batch_size: 1 + training_cfg_rate: 0.0 + seed: 42 + num_latent_t: 37 + num_height: 768 + num_width: 1344 + num_frames: 124 + + # Student VSA sparsity (0.8 = top-20% of tiles attended). + vsa: + sparsity: 0.8 + + optimizer: + learning_rate: 5.0e-5 + betas: [0.9, 0.999] + weight_decay: 0.0 + lr_scheduler: constant + lr_warmup_steps: 0 + + loop: + max_train_steps: 2000 + gradient_accumulation_steps: 1 + + checkpoint: + output_dir: outputs/minimax_h3_sft_vsa80_overfit + # 0 disables training-state checkpointing (no resume; avoids large DCP + # saves on the shared volume). + training_state_checkpointing_steps: 0 + checkpoints_total_limit: 3 + + tracker: + project_name: h3-dmd2-vsa + run_name: sft_vsa80_overfit + + model: + precondition_outputs: false + enable_gradient_checkpointing_type: full + + dit_precision: bf16 + +callbacks: + grad_clip: + _target_: fastvideo.train.callbacks.grad_clip.GradNormClipCallback + max_grad_norm: 1.0 + validation: + _target_: fastvideo.train.callbacks.validation.ValidationCallback + pipeline_target: fastvideo.pipelines.basic.minimax_h3.minimax_h3_pipeline.MiniMaxH3Pipeline + dataset_file: /mnt/h3-dmd2-overfit/c17_validation.json + every_steps: 50 + # Sample the untrained baseline at step 0. + run_at_start: true + # SFT is not a few-step generator; sample on the full 50-step schedule + # (mirrors the crush-smol overfit validation). + sampling_steps: [50] + guidance_scale: 1.0 + num_frames: 124 + num_videos_per_prompt: 1 + use_validation_media_conditioning: false + offload_training_state: true + text_encoder_cpu_offload: true + vae_cpu_offload: true + +pipeline: {} diff --git a/examples/train/configs/fine_tuning/minimax_h3/sft_vsa90_overfit.yaml b/examples/train/configs/fine_tuning/minimax_h3/sft_vsa90_overfit.yaml new file mode 100644 index 000000000..a7a243e2a --- /dev/null +++ b/examples/train/configs/fine_tuning/minimax_h3/sft_vsa90_overfit.yaml @@ -0,0 +1,93 @@ +# Single-sample SFT overfit: MiniMax H3 with the VSA-H3 backend at sparsity +# 0.9 (90% of video tiles pruned) — sparse variant of sft_vsa0_overfit.yaml; +# distribution_matching/minimax_h3/dmd2_vsa0_overfit.yaml (same c17_v2 +# preprocessed record, same validation prompt). Optimizer recipe follows the +# crush-smol H3 T2VA overfit (overfit_minimax_h3_t2va.yaml), scaled to one +# 4-GPU node with pure-FSDP sharding. + +models: + student: + _target_: fastvideo.train.models.minimax_h3.MiniMaxH3Model + init_from: /mnt/models/MiniMax-H3 + trainable: true + enable_gradient_checkpointing_type: full + attention_backend: VIDEO_SPARSE_ATTN_H3 + +method: + _target_: fastvideo.train.methods.fine_tuning.finetune.FineTuneMethod + +training: + distributed: + num_gpus: 4 + sp_size: 4 + tp_size: 1 + hsdp_replicate_dim: 1 + hsdp_shard_dim: 4 + + data: + data_path: /mnt/h3-dmd2-overfit/data + preprocessed_data_type: t2va + dataloader_num_workers: 0 + train_batch_size: 1 + training_cfg_rate: 0.0 + seed: 42 + num_latent_t: 37 + num_height: 768 + num_width: 1344 + num_frames: 124 + + # Student VSA sparsity (0.9 = top-10% of tiles attended). + vsa: + sparsity: 0.9 + + optimizer: + learning_rate: 5.0e-5 + betas: [0.9, 0.999] + weight_decay: 0.0 + lr_scheduler: constant + lr_warmup_steps: 0 + + loop: + max_train_steps: 2000 + gradient_accumulation_steps: 1 + + checkpoint: + output_dir: outputs/minimax_h3_sft_vsa90_overfit + # 0 disables training-state checkpointing (no resume; avoids large DCP + # saves on the shared volume). + training_state_checkpointing_steps: 0 + checkpoints_total_limit: 3 + + tracker: + project_name: h3-dmd2-vsa + run_name: sft_vsa90_overfit + + model: + precondition_outputs: false + enable_gradient_checkpointing_type: full + + dit_precision: bf16 + +callbacks: + grad_clip: + _target_: fastvideo.train.callbacks.grad_clip.GradNormClipCallback + max_grad_norm: 1.0 + validation: + _target_: fastvideo.train.callbacks.validation.ValidationCallback + pipeline_target: fastvideo.pipelines.basic.minimax_h3.minimax_h3_pipeline.MiniMaxH3Pipeline + dataset_file: /mnt/h3-dmd2-overfit/c17_validation.json + every_steps: 50 + # Sample the untrained baseline at step 0. + run_at_start: true + # SFT is not a few-step generator; sample on the full 50-step schedule + # (mirrors the crush-smol overfit validation). + sampling_steps: [50] + guidance_scale: 1.0 + num_frames: 124 + num_videos_per_prompt: 1 + use_validation_media_conditioning: false + offload_training_state: true + text_encoder_cpu_offload: true + vae_cpu_offload: true + +pipeline: {} diff --git a/examples/train/configs/fine_tuning/minimax_h3/sft_vsa95_overfit.yaml b/examples/train/configs/fine_tuning/minimax_h3/sft_vsa95_overfit.yaml new file mode 100644 index 000000000..025129a2f --- /dev/null +++ b/examples/train/configs/fine_tuning/minimax_h3/sft_vsa95_overfit.yaml @@ -0,0 +1,93 @@ +# Single-sample SFT overfit: MiniMax H3 with the VSA-H3 backend at sparsity +# 0.95 (95% of video tiles pruned) — sparse variant of sft_vsa0_overfit.yaml; +# distribution_matching/minimax_h3/dmd2_vsa0_overfit.yaml (same c17_v2 +# preprocessed record, same validation prompt). Optimizer recipe follows the +# crush-smol H3 T2VA overfit (overfit_minimax_h3_t2va.yaml), scaled to one +# 4-GPU node with pure-FSDP sharding. + +models: + student: + _target_: fastvideo.train.models.minimax_h3.MiniMaxH3Model + init_from: /mnt/models/MiniMax-H3 + trainable: true + enable_gradient_checkpointing_type: full + attention_backend: VIDEO_SPARSE_ATTN_H3 + +method: + _target_: fastvideo.train.methods.fine_tuning.finetune.FineTuneMethod + +training: + distributed: + num_gpus: 4 + sp_size: 4 + tp_size: 1 + hsdp_replicate_dim: 1 + hsdp_shard_dim: 4 + + data: + data_path: /mnt/h3-dmd2-overfit/data + preprocessed_data_type: t2va + dataloader_num_workers: 0 + train_batch_size: 1 + training_cfg_rate: 0.0 + seed: 42 + num_latent_t: 37 + num_height: 768 + num_width: 1344 + num_frames: 124 + + # Student VSA sparsity (0.9 = top-10% of tiles attended). + vsa: + sparsity: 0.95 + + optimizer: + learning_rate: 5.0e-5 + betas: [0.9, 0.999] + weight_decay: 0.0 + lr_scheduler: constant + lr_warmup_steps: 0 + + loop: + max_train_steps: 2000 + gradient_accumulation_steps: 1 + + checkpoint: + output_dir: outputs/minimax_h3_sft_vsa95_overfit + # 0 disables training-state checkpointing (no resume; avoids large DCP + # saves on the shared volume). + training_state_checkpointing_steps: 0 + checkpoints_total_limit: 3 + + tracker: + project_name: h3-dmd2-vsa + run_name: sft_vsa95_overfit + + model: + precondition_outputs: false + enable_gradient_checkpointing_type: full + + dit_precision: bf16 + +callbacks: + grad_clip: + _target_: fastvideo.train.callbacks.grad_clip.GradNormClipCallback + max_grad_norm: 1.0 + validation: + _target_: fastvideo.train.callbacks.validation.ValidationCallback + pipeline_target: fastvideo.pipelines.basic.minimax_h3.minimax_h3_pipeline.MiniMaxH3Pipeline + dataset_file: /mnt/h3-dmd2-overfit/c17_validation.json + every_steps: 50 + # Sample the untrained baseline at step 0. + run_at_start: true + # SFT is not a few-step generator; sample on the full 50-step schedule + # (mirrors the crush-smol overfit validation). + sampling_steps: [50] + guidance_scale: 1.0 + num_frames: 124 + num_videos_per_prompt: 1 + use_validation_media_conditioning: false + offload_training_state: true + text_encoder_cpu_offload: true + vae_cpu_offload: true + +pipeline: {} diff --git a/examples/train/configs/fine_tuning/minimax_h3/sft_vsa95_overfit_2gpu.yaml b/examples/train/configs/fine_tuning/minimax_h3/sft_vsa95_overfit_2gpu.yaml new file mode 100644 index 000000000..53f0595e2 --- /dev/null +++ b/examples/train/configs/fine_tuning/minimax_h3/sft_vsa95_overfit_2gpu.yaml @@ -0,0 +1,93 @@ +# Single-sample SFT overfit: MiniMax H3 with the VSA-H3 backend at sparsity +# 0.95 (95% of video tiles pruned) — sparse variant of sft_vsa0_overfit.yaml; +# distribution_matching/minimax_h3/dmd2_vsa0_overfit.yaml (same c17_v2 +# preprocessed record, same validation prompt). Optimizer recipe follows the +# crush-smol H3 T2VA overfit (overfit_minimax_h3_t2va.yaml), scaled to one +# 4-GPU node with pure-FSDP sharding. + +models: + student: + _target_: fastvideo.train.models.minimax_h3.MiniMaxH3Model + init_from: /mnt/models/MiniMax-H3 + trainable: true + enable_gradient_checkpointing_type: full + attention_backend: VIDEO_SPARSE_ATTN_H3 + +method: + _target_: fastvideo.train.methods.fine_tuning.finetune.FineTuneMethod + +training: + distributed: + num_gpus: 2 + sp_size: 2 + tp_size: 1 + hsdp_replicate_dim: 1 + hsdp_shard_dim: 2 + + data: + data_path: /mnt/h3-dmd2-overfit/data + preprocessed_data_type: t2va + dataloader_num_workers: 0 + train_batch_size: 1 + training_cfg_rate: 0.0 + seed: 42 + num_latent_t: 37 + num_height: 768 + num_width: 1344 + num_frames: 124 + + # Student VSA sparsity (0.95 = top-5% of tiles attended). + vsa: + sparsity: 0.95 + + optimizer: + learning_rate: 5.0e-5 + betas: [0.0, 0.999] # beta1=0 -> buffer-free AdamW; std betas OOM at sp=2 + weight_decay: 0.0 + lr_scheduler: constant + lr_warmup_steps: 0 + + loop: + max_train_steps: 2000 + gradient_accumulation_steps: 1 + + checkpoint: + output_dir: outputs/minimax_h3_sft_vsa95_overfit + # 0 disables training-state checkpointing (no resume; avoids large DCP + # saves on the shared volume). + training_state_checkpointing_steps: 0 + checkpoints_total_limit: 3 + + tracker: + project_name: h3-dmd2-vsa + run_name: sft_vsa95_overfit + + model: + precondition_outputs: false + enable_gradient_checkpointing_type: full + + dit_precision: bf16 + +callbacks: + grad_clip: + _target_: fastvideo.train.callbacks.grad_clip.GradNormClipCallback + max_grad_norm: 1.0 + validation: + _target_: fastvideo.train.callbacks.validation.ValidationCallback + pipeline_target: fastvideo.pipelines.basic.minimax_h3.minimax_h3_pipeline.MiniMaxH3Pipeline + dataset_file: /mnt/h3-dmd2-overfit/c17_validation.json + every_steps: 50 + # Sample the untrained baseline at step 0. + run_at_start: true + # SFT is not a few-step generator; sample on the full 50-step schedule + # (mirrors the crush-smol overfit validation). + sampling_steps: [50] + guidance_scale: 1.0 + num_frames: 124 + num_videos_per_prompt: 1 + use_validation_media_conditioning: false + offload_training_state: true + text_encoder_cpu_offload: true + vae_cpu_offload: true + +pipeline: {} diff --git a/examples/train/configs/fine_tuning/minimax_h3/sft_vsa97_overfit.yaml b/examples/train/configs/fine_tuning/minimax_h3/sft_vsa97_overfit.yaml new file mode 100644 index 000000000..41bc9719d --- /dev/null +++ b/examples/train/configs/fine_tuning/minimax_h3/sft_vsa97_overfit.yaml @@ -0,0 +1,93 @@ +# Single-sample SFT overfit: MiniMax H3 with the VSA-H3 backend at sparsity +# 0.97 (97% of video tiles pruned) — sparse variant of sft_vsa0_overfit.yaml; +# distribution_matching/minimax_h3/dmd2_vsa0_overfit.yaml (same c17_v2 +# preprocessed record, same validation prompt). Optimizer recipe follows the +# crush-smol H3 T2VA overfit (overfit_minimax_h3_t2va.yaml), scaled to one +# 4-GPU node with pure-FSDP sharding. + +models: + student: + _target_: fastvideo.train.models.minimax_h3.MiniMaxH3Model + init_from: /mnt/models/MiniMax-H3 + trainable: true + enable_gradient_checkpointing_type: full + attention_backend: VIDEO_SPARSE_ATTN_H3 + +method: + _target_: fastvideo.train.methods.fine_tuning.finetune.FineTuneMethod + +training: + distributed: + num_gpus: 4 + sp_size: 4 + tp_size: 1 + hsdp_replicate_dim: 1 + hsdp_shard_dim: 4 + + data: + data_path: /mnt/h3-dmd2-overfit/data + preprocessed_data_type: t2va + dataloader_num_workers: 0 + train_batch_size: 1 + training_cfg_rate: 0.0 + seed: 42 + num_latent_t: 37 + num_height: 768 + num_width: 1344 + num_frames: 124 + + # Student VSA sparsity (0.9 = top-10% of tiles attended). + vsa: + sparsity: 0.97 + + optimizer: + learning_rate: 5.0e-5 + betas: [0.9, 0.999] + weight_decay: 0.0 + lr_scheduler: constant + lr_warmup_steps: 0 + + loop: + max_train_steps: 2000 + gradient_accumulation_steps: 1 + + checkpoint: + output_dir: outputs/minimax_h3_sft_vsa97_overfit + # 0 disables training-state checkpointing (no resume; avoids large DCP + # saves on the shared volume). + training_state_checkpointing_steps: 0 + checkpoints_total_limit: 3 + + tracker: + project_name: h3-dmd2-vsa + run_name: sft_vsa97_overfit + + model: + precondition_outputs: false + enable_gradient_checkpointing_type: full + + dit_precision: bf16 + +callbacks: + grad_clip: + _target_: fastvideo.train.callbacks.grad_clip.GradNormClipCallback + max_grad_norm: 1.0 + validation: + _target_: fastvideo.train.callbacks.validation.ValidationCallback + pipeline_target: fastvideo.pipelines.basic.minimax_h3.minimax_h3_pipeline.MiniMaxH3Pipeline + dataset_file: /mnt/h3-dmd2-overfit/c17_validation.json + every_steps: 50 + # Sample the untrained baseline at step 0. + run_at_start: true + # SFT is not a few-step generator; sample on the full 50-step schedule + # (mirrors the crush-smol overfit validation). + sampling_steps: [50] + guidance_scale: 1.0 + num_frames: 124 + num_videos_per_prompt: 1 + use_validation_media_conditioning: false + offload_training_state: true + text_encoder_cpu_offload: true + vae_cpu_offload: true + +pipeline: {} diff --git a/examples/train/configs/fine_tuning/minimax_h3/sft_vsa97_overfit_2gpu.yaml b/examples/train/configs/fine_tuning/minimax_h3/sft_vsa97_overfit_2gpu.yaml new file mode 100644 index 000000000..543e54952 --- /dev/null +++ b/examples/train/configs/fine_tuning/minimax_h3/sft_vsa97_overfit_2gpu.yaml @@ -0,0 +1,93 @@ +# Single-sample SFT overfit: MiniMax H3 with the VSA-H3 backend at sparsity +# 0.97 (97% of video tiles pruned) — sparse variant of sft_vsa0_overfit.yaml; +# distribution_matching/minimax_h3/dmd2_vsa0_overfit.yaml (same c17_v2 +# preprocessed record, same validation prompt). Optimizer recipe follows the +# crush-smol H3 T2VA overfit (overfit_minimax_h3_t2va.yaml), scaled to one +# 4-GPU node with pure-FSDP sharding. + +models: + student: + _target_: fastvideo.train.models.minimax_h3.MiniMaxH3Model + init_from: /mnt/models/MiniMax-H3 + trainable: true + enable_gradient_checkpointing_type: full + attention_backend: VIDEO_SPARSE_ATTN_H3 + +method: + _target_: fastvideo.train.methods.fine_tuning.finetune.FineTuneMethod + +training: + distributed: + num_gpus: 2 + sp_size: 2 + tp_size: 1 + hsdp_replicate_dim: 1 + hsdp_shard_dim: 2 + + data: + data_path: /mnt/h3-dmd2-overfit/data + preprocessed_data_type: t2va + dataloader_num_workers: 0 + train_batch_size: 1 + training_cfg_rate: 0.0 + seed: 42 + num_latent_t: 37 + num_height: 768 + num_width: 1344 + num_frames: 124 + + # Student VSA sparsity (0.97 = top-3% of tiles attended). + vsa: + sparsity: 0.97 + + optimizer: + learning_rate: 5.0e-5 + betas: [0.0, 0.999] # beta1=0 -> buffer-free AdamW; std betas OOM at sp=2 + weight_decay: 0.0 + lr_scheduler: constant + lr_warmup_steps: 0 + + loop: + max_train_steps: 2000 + gradient_accumulation_steps: 1 + + checkpoint: + output_dir: outputs/minimax_h3_sft_vsa97_overfit + # 0 disables training-state checkpointing (no resume; avoids large DCP + # saves on the shared volume). + training_state_checkpointing_steps: 0 + checkpoints_total_limit: 3 + + tracker: + project_name: h3-dmd2-vsa + run_name: sft_vsa97_overfit + + model: + precondition_outputs: false + enable_gradient_checkpointing_type: full + + dit_precision: bf16 + +callbacks: + grad_clip: + _target_: fastvideo.train.callbacks.grad_clip.GradNormClipCallback + max_grad_norm: 1.0 + validation: + _target_: fastvideo.train.callbacks.validation.ValidationCallback + pipeline_target: fastvideo.pipelines.basic.minimax_h3.minimax_h3_pipeline.MiniMaxH3Pipeline + dataset_file: /mnt/h3-dmd2-overfit/c17_validation.json + every_steps: 50 + # Sample the untrained baseline at step 0. + run_at_start: true + # SFT is not a few-step generator; sample on the full 50-step schedule + # (mirrors the crush-smol overfit validation). + sampling_steps: [50] + guidance_scale: 1.0 + num_frames: 124 + num_videos_per_prompt: 1 + use_validation_media_conditioning: false + offload_training_state: true + text_encoder_cpu_offload: true + vae_cpu_offload: true + +pipeline: {} diff --git a/examples/train/slurm/dmd2_32xgb200.sbatch b/examples/train/slurm/dmd2_32xgb200.sbatch new file mode 100644 index 000000000..9f76ea54b --- /dev/null +++ b/examples/train/slurm/dmd2_32xgb200.sbatch @@ -0,0 +1,79 @@ +#!/bin/bash +# 8-tray (8 x 4x GB200 = 32 GPU) MiniMax-H3 DMD2 launch from a Slurm login node. +# +# sbatch examples/train/slurm/dmd2_32xgb200.sbatch +# sbatch --nodes=2 examples/train/slurm/dmd2_32xgb200.sbatch # smaller sweep +# CONFIG=path/to/other.yaml sbatch examples/train/slurm/dmd2_32xgb200.sbatch +# +# The paired config (dmd2_simulate_fa4_32gpu.yaml) assumes 8 nodes: +# hsdp_replicate_dim must equal the node count and num_gpus the world size — +# this script forwards overrides derived from the actual allocation, so +# shrinking --nodes "just works" (global batch shrinks with it). +# +# Cluster-specific knobs (partition/account/container) are marked CLUSTER. + +#SBATCH --job-name=h3-dmd2-32xgb200 +#SBATCH --nodes=8 +#SBATCH --ntasks-per-node=1 +#SBATCH --gpus-per-node=4 +#SBATCH --exclusive +#SBATCH --time=48:00:00 +#SBATCH --output=slurm-%x-%j.out +##SBATCH --partition= +##SBATCH --account= + +set -euo pipefail + +# ── paths (shared FS visible on every tray) ────────────────────────────── +REPO="${REPO:-/mnt/FastVideo}" +VENV="${VENV:-${REPO}/.venv}" +CONFIG="${CONFIG:-${REPO}/examples/train/configs/distribution_matching/minimax_h3/dmd2_simulate_fa4_32gpu.yaml}" + +# ── topology derived from the allocation ───────────────────────────────── +NNODES="${SLURM_JOB_NUM_NODES}" +GPUS_PER_NODE="${SLURM_GPUS_PER_NODE:-4}" +GPUS_PER_NODE="${GPUS_PER_NODE##*:}" # "4" or "gb200:4" -> 4 +WORLD_SIZE=$((NNODES * GPUS_PER_NODE)) +MASTER_ADDR="$(scontrol show hostnames "${SLURM_JOB_NODELIST}" | head -n 1)" +MASTER_PORT="${MASTER_PORT:-29501}" + +echo "=== H3 DMD2 multi-node launch ===" +echo "Nodes: ${NNODES} (${SLURM_JOB_NODELIST})" +echo "World size: ${WORLD_SIZE}" +echo "Rendezvous: ${MASTER_ADDR}:${MASTER_PORT}" +echo "Config: ${CONFIG}" + +# ── per-rank environment (srun propagates exported vars) ───────────────── +export FASTVIDEO_FA4=1 +export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True +export TOKENIZERS_PARALLELISM=false +# Node-local weight cache: relaunches rebuild all three 33B roles from tmpfs +# in seconds. PER_NODE=1 makes every tray write its own /dev/shm copy +# (single-writer semantics would leave 7 of 8 trays permanently cold). +export FASTVIDEO_WEIGHT_SHARD_CACHE=/dev/shm/fastvideo-wcache +export FASTVIDEO_WEIGHT_SHARD_CACHE_PER_NODE=1 +# W&B: credentials come from ~/.netrc on the shared home; flip to offline if +# compute nodes have no egress. +export WANDB_MODE="${WANDB_MODE:-online}" +# CLUSTER: NCCL fabric tuning if needed (NVLink partitioning, IB HCA choice): +# export NCCL_IB_HCA=... NCCL_SOCKET_IFNAME=... + +export NNODES MASTER_ADDR MASTER_PORT +export NUM_GPUS="${GPUS_PER_NODE}" + +# CLUSTER: if trays require a container, wrap the srun payload accordingly, +# e.g. add `--container-image=... --container-mounts=/mnt:/mnt` (pyxis) so +# /mnt (repo, models, data) and /dev/shm are available inside. +srun --kill-on-bad-exit=1 --ntasks-per-node=1 bash -c " + set -euo pipefail + export NODE_RANK=\${SLURM_NODEID} + # One log dir per node — parallel tee writers on one NFS file garble it. + export LOG_DIR=examples/train/logs/\${SLURM_JOB_ID}-node\${SLURM_NODEID} + source '${VENV}/bin/activate' + cd '${REPO}' + bash examples/train/run.sh '${CONFIG}' \ + --training.distributed.num_gpus ${WORLD_SIZE} \ + --training.distributed.hsdp_replicate_dim ${NNODES} \ + --training.distributed.hsdp_shard_dim ${GPUS_PER_NODE} \ + --training.distributed.sp_size ${GPUS_PER_NODE} +" diff --git a/pyproject.toml b/pyproject.toml index d7630ba34..d113e3e32 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -208,7 +208,8 @@ dreamverse-mock-server = "dreamverse.mock_server:cli" [tool.setuptools.packages.find] where = [".", "apps/dreamverse"] -exclude = ["assets*", "docker*", "docs", "scripts*", "apps*", "dreamverse.tests*"] +include = ["fastvideo", "fastvideo.*", "dreamverse", "dreamverse.*"] +exclude = ["dreamverse.tests*"] [tool.setuptools.package-data] dreamverse = ["prompts/*.md"]