Davids048andClaude Fable 5.1
a5aa64ab6b
VideoGenerator.from_pretrained(model_path, config) takes the same nested
settings as from_config; the 24 flat keywords, from_pretrained_kwargs_to_config,
FROM_PRETRAINED_KWARGS, and every flat_field in the schema are deleted.
nvfp4_fa4 is the typed field engine.attention.nvfp4_fa4, applied by a
resolution step. 65 call sites and the docs use the nested form; the three
kwargs golden cases are config cases with byte-identical results.
Every value is decided during resolution; no runtime code calls
with_override. The device offload policy (unified memory, MPS, layerwise
conflicts, lazy module load) runs as resolution steps in the main process
(fastvideo/api/device_policy.py), and the worker runs on the config it
receives. The LTX-2 refine defaults from model_index.json and the MiniMax-H3
schedule from fastvideo_inference.json are resolution steps
(fastvideo/api/checkpoint_defaults.py) that read local checkpoints or download
only those files; the H3 pipeline validates the schedule against the loaded
schedulers but no longer writes it. The preprocessing entry point passes the
downloaded local path as run state instead of a config override. Teacher and
critic models load with override_transformer_cls_name as a load argument.
The test isolation turns the device policy off, as it blocks downloads, so
that the goldens do not depend on the machine.
Against the 3f6893a0 goldens, every field still matches except boundary_ratio
and ltx2_vae_tiling (DESIGN section 9) and the fields that no longer exist.
The launcher and trainer YAML verifications report 0 unexplained differences.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
GLM-Image Local Tests
Local-only parity and smoke tests for the glm_image port: FastVideo vs the
official HF transformers + diffusers GLM-Image implementation. These are
not run in CI — they need the full weights, CUDA, and (for the reference
side) the diffusers GLM-Image classes; each test skips with an actionable
message when a prerequisite is missing.
What you need
| Weights | zai-org/GLM-Image → official_weights/glm_image (~30 GB: AR encoder 4 shards, transformer 3 shards, vae + text_encoder) |
| Runtime deps | transformers>=5.15.0 (repo floor; the AR encoder GlmImageForConditionalGeneration first ships in 5.0.0) and diffusers>=0.38.0 — both committed in pyproject.toml |
| Test-only deps | the diffusers GlmImageTransformer2DModel / GlmImagePipeline reference classes need diffusers>=0.37.0.dev0 (used only by the parity tests) |
Download weights (from the repo root; cache via HF_HOME if / is small):
python ".agents/skills/add-model-01-prep/scripts/download_hf_weights.py" \
"zai-org/GLM-Image" "official_weights/glm_image"
Optionally install the diffusers reference classes to run the parity side:
uv pip install -U "git+https://github.com/huggingface/diffusers.git@ff3b86b4"
Parity was checked against transformers a65bf6c9 + diffusers ff3b86b4.
Parity results (GB200)
| Component | Test | Result |
|---|---|---|
transformer (DiT) |
test_transformer_parity.py |
fp32 cosine 1.000000 (100% within 5e-3); bf16 cosine 0.999969 (element-wise rounding over 30 layers vs a different SDPA kernel — not a bug; RoPE proven exactly equivalent). fp32 + bf16 + strict-load |
vae (AutoencoderKL) |
test_vae_parity.py |
decode + encode + config, 3/3 |
text_encoder (T5) + ByT5 tokenizer |
test_t5_parity.py |
tokenizer + glyph extract + embeds, 3/3 |
vision_language_encoder (GLM-4-9B AR) |
test_ar_parity.py |
surface + to/eval + greedy determinism, 3/3 |
pipeline (T2I) |
test_pipeline_parity.py |
injected-input determinism: latent cosine 0.999997, decoded-image MAE 0.159/255. validity + deterministic parity, 2/2 |
pipeline (I2I / edit) |
test_edit_pipeline_parity.py |
validity / wiring gate (condition image enters via a KV-cache write pass, not noise-the-latent). diffusers distribution parity deferred (stochastic AR) |
processor (GlmImageProcessor) and scheduler (FlowMatchEulerDiscreteScheduler,
shared flow-matching) are exercised via the AR and pipeline tests.
Run:
pytest tests/local_tests/glm_image/test_transformer_parity.py -v -s
pytest tests/local_tests/glm_image/test_vae_parity.py -v -s
pytest tests/local_tests/glm_image/test_t5_parity.py -v -s
pytest tests/local_tests/glm_image/test_ar_parity.py -v -s
pytest tests/local_tests/glm_image/test_pipeline_parity.py -v -s
pytest tests/local_tests/glm_image/test_edit_pipeline_parity.py -v -s
Design notes (load-bearing — don't regress)
- Pipeline parity is by injection. Full-pipeline pixel parity is ill-posed
(stochastic AR + independent latent RNG), so the pipeline tests inject matched
(prompt_embeds, prior_token_ids, latents)into both the diffusers pipeline and the realGlmImageDenoisingStage, then compare denoised latents + decoded image — not raw end-to-end pixels. - HF imports are confined to the lazy-wrapper loader
fastvideo/models/encoders/glm_image_ar_loader.py, never the production loaders (production-boundary rule). - Tokenizer is ByT5, matching the diffusers reference (not plain T5);
compute_glyph_embedsmirrors diffusers_get_glyph_embeds. - Model-specific stages live under
fastvideo/pipelines/basic/glm_image/stages/per theadd-modelFiles Map. - Param mapping is handled in-place at load via
param_names_mapping(fastvideo/configs/models/dits/glm_image.py) + the native VAE/encoder configs — no conversion script; the loader raises on any unmatched param andtest_transformer_strict_load.pyasserts completeness withstrict=True.