Kaiqin Kongandmergify[bot]
773d44b875
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
56 lines
2.7 KiB
Markdown
56 lines
2.7 KiB
Markdown
# `fastvideo/models/` — Model Implementations
|
|
|
|
**Generated:** 2026-05-02
|
|
|
|
DiT / VAE / encoder / scheduler / upsampler / audio model classes. **Pre-commit excludes this directory** — yapf/ruff/mypy do not run on commits here. Match neighboring file style manually.
|
|
|
|
## Layout
|
|
|
|
```
|
|
models/
|
|
├── dits/
|
|
│ ├── <model>.py # Single-file DiT (wanvideo, ltx2, hunyuanvideo, cosmos, ...)
|
|
│ ├── hyworld/ # Multi-file DiT family
|
|
│ ├── lingbotworld/ # ditto
|
|
│ └── matrixgame2/ # ditto
|
|
├── vaes/ # AutoencoderKL variants per model family
|
|
├── encoders/ # T5, CLIP, Llama, Qwen2.5, Gemma, SigLIP, Reason1, audio conditioner
|
|
├── schedulers/ # FlowMatch / EulerDiscrete / DPM custom schedulers
|
|
├── upsamplers/ # Hunyuan15 super-resolution
|
|
├── audio/ # Audio-VAE/decoder modules (LTX-2 audio, Stable Audio)
|
|
├── camera/ # Camera-conditioning modules (Gen3C)
|
|
└── loader/ # component_loader.py, fsdp_load.py, weight_utils.py
|
|
```
|
|
|
|
`loader/component_loader.py` is the central entry point that the pipeline uses
|
|
to instantiate model components from a HF directory. New components plug in
|
|
through `register_*` calls or by extending the `ComponentLoader` mappings.
|
|
|
|
## Adding a Model Component (DiT / VAE / Encoder)
|
|
|
|
1. Read `fastvideo/layers/AGENTS.md` first — it defines which tensor-parallel
|
|
linear / attention layer to use. Do not freelance.
|
|
2. Define the arch in `models/<role>/<model>.py`. Mirror the official reference's
|
|
constructor args; do not "improve" the layer choices.
|
|
3. Add the matching arch config in `configs/models/<role>/<model>.py`.
|
|
4. Expose `param_names_mapping` on the config — it is the **source of truth** for
|
|
the converter under `scripts/checkpoint_conversion/`.
|
|
5. Use `init_logger(__name__)`, not stdlib logging.
|
|
|
|
## State-Dict Discipline
|
|
|
|
- The native component's `state_dict()` defines target keys + shapes.
|
|
- Conversion scripts (`scripts/checkpoint_conversion/`) bend to the model, not
|
|
the other way around.
|
|
- Fused QKV / packed MLP layouts must be documented in the config or the model
|
|
module — converters need to split/fuse accordingly.
|
|
|
|
## Anti-Patterns
|
|
|
|
- Importing `transformers` / `diffusers` model classes at runtime inside the
|
|
forward path — these belong in the loader, not the architecture file.
|
|
- Adding training-only state (EMA buffers, optimizer state) to the inference
|
|
state-dict surface.
|
|
- Calling `torch.distributed` directly. Go through `fastvideo.distributed`.
|
|
- Treating this directory as lint-clean. It isn't (see pre-commit excludes).
|