120 lines
4.8 KiB
Markdown
120 lines
4.8 KiB
Markdown
# World-Model: Matrix-Game 2.0 I2V
|
|
|
|
Training scenarios for the Matrix-Game 2.0 I2V world model on Solaris (Minecraft)
|
|
data and Zelda data, using the new YAML-driven trainer
|
|
(`fastvideo/train/entrypoint/train.py`).
|
|
|
|
## Solaris Configs
|
|
|
|
| Config | Method | Student | Notes |
|
|
|---|---|---|---|
|
|
| `solaris/finetune_i2v.yaml` | `FineTuneMethod` | `MatrixGame2Model` (bidirectional) | Multi-step SFT from `mg_bidirectional_Solaris`. |
|
|
| `solaris/dfsft_causal_i2v.yaml` | `DiffusionForcingSFTMethod` | `MatrixGame2CausalModel` | Diffusion-Forcing SFT with chunkwise timesteps. |
|
|
| `solaris/self_forcing_causal_i2v.yaml` | `SelfForcingMethod` | `MatrixGame2CausalModel` | Matrix-Game 2.0 DMD/Self-Forcing distillation; teacher = bidirectional, critic = bidirectional. |
|
|
|
|
## Zelda Configs
|
|
|
|
| Config | Method | Student | Notes |
|
|
|---|---|---|---|
|
|
| `zelda/finetune_i2v.yaml` | `FineTuneMethod` | `MatrixGame2Model` (bidirectional) | Zelda bidirectional I2V finetuning from `FastVideo/Matrix-Game-2.0-Base-Diffusers`. Uses 33-frame clips and Zelda validation with action overlays. |
|
|
| `zelda/dfsft_causal_i2v.yaml` | `DiffusionForcingSFTMethod` | `MatrixGame2CausalModel` | Zelda causal Diffusion-Forcing SFT from `mignonjia/mg_bidirectional_zelda`. Uses the same Zelda data, resolution, optimizer, and validation defaults as the Zelda finetune config. |
|
|
| `zelda/self_forcing_causal_i2v.yaml` | `SelfForcingMethod` | `MatrixGame2CausalModel` | Zelda DMD/Self-Forcing distillation; student init = `mignonjia/mg_causal_zelda`, teacher = bidirectional (`mignonjia/mg_bidirectional_zelda`), critic = bidirectional. |
|
|
| `zelda/streaming_long_tuning_causal_i2v.yaml` | `StreamingLongTuningMethod` | `MatrixGame2CausalModel` | LongLive-style streaming long tuning from the 1k-step Zelda self-forcing checkpoint. |
|
|
|
|
Zelda world-model distillation is a two-run workflow: first run
|
|
`zelda/self_forcing_causal_i2v.yaml` to train or load the 1k-step
|
|
self-forcing checkpoint (`mignonjia/mg_sf_distilled_zelda_1k_steps`), then run
|
|
`zelda/streaming_long_tuning_causal_i2v.yaml` for the 3k-step streaming
|
|
long-tuning stage. The long-tuning YAML starts from that 1k-step checkpoint; it
|
|
does not run the short self-forcing stage inside the same config.
|
|
|
|
## Zelda Training Data
|
|
|
|
The Zelda training configs use `data/zeldam2-clean` as a suggested local path.
|
|
Download the dataset from Hugging Face before running those configs:
|
|
|
|
```bash
|
|
python scripts/huggingface/download_hf.py \
|
|
--repo_id mignonjia/zeldam2-clean \
|
|
--local_dir data/zeldam2-clean \
|
|
--repo_type dataset
|
|
```
|
|
|
|
You can store the dataset elsewhere; update `training.data.data_path` in the
|
|
YAML to point at that location.
|
|
|
|
## Multi3D Training Data
|
|
|
|
`zelda/finetune_i2v.yaml` includes an optional, commented-out Multi3D entry.
|
|
Enable it only when you want to mix Zelda with multi-game data from
|
|
`data/multi3d_games`. You can store this dataset anywhere; before enabling it,
|
|
update the matching commented `training.data.data_path` key in the YAML to the
|
|
correct location.
|
|
|
|
To mix datasets in a training YAML, set `training.data.data_path` to a
|
|
path-to-repeat-count mapping. For example, `zelda/finetune_i2v.yaml` can use
|
|
`data/zeldam2-clean: 1` and `# data/multi3d_games: 10`; uncommenting the
|
|
Multi3D entry repeats the multi-game parquet list ten times before training
|
|
samples are shuffled.
|
|
|
|
## World Model Validation Data
|
|
|
|
The Zelda validation configs expect a small public validation bundle under
|
|
`data/zelda_validation_data`.
|
|
|
|
Download it from Hugging Face before running the Zelda scenarios:
|
|
|
|
```bash
|
|
python scripts/huggingface/download_hf.py \
|
|
--repo_id mignonjia/zelda_validation_data \
|
|
--local_dir data/zelda_validation_data \
|
|
--repo_type dataset
|
|
```
|
|
|
|
The bundle contains `validation_zelda.json`, `images/`, and `actions/`.
|
|
The Zelda configs point
|
|
`callbacks.validation.dataset_file` at
|
|
`data/zelda_validation_data/validation_zelda.json`.
|
|
|
|
## Usage
|
|
|
|
### Solaris
|
|
|
|
```bash
|
|
bash examples/train/run.sh \
|
|
examples/train/scenario/worldmodel/solaris/finetune_i2v.yaml
|
|
|
|
bash examples/train/run.sh \
|
|
examples/train/scenario/worldmodel/solaris/dfsft_causal_i2v.yaml
|
|
|
|
bash examples/train/run.sh \
|
|
examples/train/scenario/worldmodel/solaris/self_forcing_causal_i2v.yaml
|
|
```
|
|
|
|
### Zelda
|
|
|
|
```bash
|
|
# Finetuning / DFSFT
|
|
bash examples/train/run.sh \
|
|
examples/train/scenario/worldmodel/zelda/finetune_i2v.yaml
|
|
|
|
bash examples/train/run.sh \
|
|
examples/train/scenario/worldmodel/zelda/dfsft_causal_i2v.yaml
|
|
|
|
# Distillation / long tuning
|
|
bash examples/train/run.sh \
|
|
examples/train/scenario/worldmodel/zelda/self_forcing_causal_i2v.yaml
|
|
|
|
bash examples/train/run.sh \
|
|
examples/train/scenario/worldmodel/zelda/streaming_long_tuning_causal_i2v.yaml
|
|
```
|
|
|
|
Override any field on the command line:
|
|
|
|
```bash
|
|
bash examples/train/run.sh \
|
|
examples/train/scenario/worldmodel/solaris/dfsft_causal_i2v.yaml \
|
|
--training.distributed.num_gpus 8 \
|
|
--training.optimizer.learning_rate 1e-5
|
|
```
|