leffffandClaude Fable 5
5aae1c093a
- Never touch the user's documented dataset: preprocess roots are now env-overridable (KANDINSKY5_OVERFIT_DATA_DIR/_OUTPUT_DIR) and the nightly keeps every artifact under a single test-owned data/kandinsky5_e2e/ root, cleaning only that - Override method.generator_update_interval to 1 in the E2E stage-2 run: with the recipe's 5, a 3-step run performed zero student updates and the exported "student" was stage-1 weights - Harden the media oracle: exact frame count/geometry (closes the SSIM shorter-clip truncation hole), per-frame luma spatial std (solid clip at reference mean scored 15.4 on the old global RGB std), consecutive-frame temporal diff (frozen clips), and MS-SSIM floor recalibrated to 0.95 against measured known-bads (solid 0.9189, frozen 0.8597, good 1.0000); known-bad clips are permanent self-calibrating regressions in fastvideo/tests/workflow/test_kandinsky5_e2e_media_oracle.py - Capability-gate ATTN_QAT_INFER: is_attn_qat_infer_available() now requires an sm_120/sm_121 device in addition to the extension import (CUDA 13 wheels can bundle the extension on H100/GB200, where the first kernel call failed instead of falling back); the real platform resolver is covered in fastvideo/tests/api/test_attn_qat_infer_capability_gate.py - Remove the now-stale golden reference: the generator-update fix changes what stage 2 trains, so the reference must be re-recorded from the corrected run (KANDINSKY5_E2E_WRITE_REFERENCE=1 bootstrap + review) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
109 lines
3.4 KiB
Markdown
109 lines
3.4 KiB
Markdown
# Debugging
|
|
|
|
This page collects practical debugging steps for FastVideo inference issues.
|
|
|
|
## Collect Environment Info
|
|
|
|
From the repository root, run:
|
|
|
|
```bash
|
|
python collect_env.py
|
|
```
|
|
|
|
Attach the output when filing a GitHub issue.
|
|
|
|
## Increase Logging
|
|
|
|
FastVideo logging level is controlled by environment variables:
|
|
|
|
```bash
|
|
FASTVIDEO_LOGGING_LEVEL=DEBUG \
|
|
FASTVIDEO_STAGE_LOGGING=1 \
|
|
python your_script.py
|
|
```
|
|
|
|
Useful variables:
|
|
|
|
- `FASTVIDEO_LOGGING_LEVEL`: `DEBUG`, `INFO`, `WARNING`, `ERROR`
|
|
- `FASTVIDEO_STAGE_LOGGING`: print per-stage timings during pipeline execution
|
|
- `FASTVIDEO_ATTENTION_BACKEND`: force an attention backend (for example
|
|
`TORCH_SDPA`, `FLASH_ATTN`, `SAGE_ATTN_THREE`, or
|
|
`ATTN_QAT_INFER`, or `ATTN_QAT_TRAIN`)
|
|
|
|
## Layer-by-Layer Activation Tracing
|
|
|
|
For numerical-divergence debugging — typically when porting a new model and
|
|
needing to find the first layer where FastVideo and an upstream reference
|
|
produce different outputs — use the env-gated activation trace mode:
|
|
|
|
```bash
|
|
FASTVIDEO_TRACE_ACTIVATIONS=1 \
|
|
FASTVIDEO_TRACE_LAYERS="^transformer\.blocks\.\d+$" \
|
|
FASTVIDEO_TRACE_OUTPUT=/tmp/fv_trace.jsonl \
|
|
python your_script.py
|
|
```
|
|
|
|
The trace dumps per-tensor stats (`abs_mean`, `sum`, `shape`, etc.) to a
|
|
JSONL file. Run the same workload with tracing on the upstream side, then
|
|
`diff` the two files to localize the first divergent layer.
|
|
|
|
See [Activation Trace Mode](../contributing/activation_trace.md) for the
|
|
full guide (env var reference, JSONL output schema, parity-debug workflow,
|
|
performance impact, and troubleshooting).
|
|
|
|
## Common Failure Modes
|
|
|
|
### Out-of-memory
|
|
|
|
Try, in order:
|
|
|
|
1. Reduce `height`, `width`, `num_frames`, or `num_inference_steps`.
|
|
2. Enable offloading flags such as `dit_layerwise_offload` (single GPU) or
|
|
`use_fsdp_inference` (multi-GPU).
|
|
3. Enable `vae_cpu_offload`, `image_encoder_cpu_offload`, and
|
|
`text_encoder_cpu_offload`.
|
|
|
|
See [Inference Offloading](../inference/offloading.md) for recommended
|
|
combinations.
|
|
|
|
### Attention backend import errors
|
|
|
|
If forcing a backend fails, verify optional dependencies are installed:
|
|
|
|
- `FLASH_ATTN`: `flash-attn`
|
|
- `VIDEO_SPARSE_ATTN`: `fastvideo-kernel`
|
|
- `SLIDING_TILE_ATTN`: STA legacy workflow in
|
|
`sta_do_not_delete` + `fastvideo-kernel`
|
|
- `SAGE_ATTN`: SageAttention package
|
|
- `SAGE_ATTN_THREE`: upstream `sageattn3` package
|
|
- `ATTN_QAT_INFER`: `fastvideo-kernel` checkout/source install that exposes
|
|
`attn_qat_infer`, AND a consumer-Blackwell sm_120 GPU -- on any
|
|
other device the backend reports unavailable (even if a CUDA 13 wheel
|
|
bundles the extension) and selection falls back to FlashAttention
|
|
- `ATTN_QAT_TRAIN`: `fastvideo-kernel`; its runtime-JIT Triton implementation
|
|
selects an optimized route on SM100, joins the quantized and STE P@V paths on
|
|
SM120, and retains the previous route for unsupported configurations. See
|
|
[Attn-QAT Training](../training/attn_qat.md) for architecture controls.
|
|
|
|
As a fallback, use:
|
|
|
|
```bash
|
|
export FASTVIDEO_ATTENTION_BACKEND=TORCH_SDPA
|
|
```
|
|
|
|
### Configuration parsing errors
|
|
|
|
When using `--config`, keep keys aligned with CLI argument names (underscores or
|
|
hyphens are both accepted). For nested config values, use nested objects
|
|
(`vae_config`, `dit_config`) rather than dotted keys.
|
|
|
|
## Issue Template
|
|
|
|
When opening an issue, include:
|
|
|
|
- exact command or Python snippet,
|
|
- model ID/path,
|
|
- full traceback,
|
|
- `collect_env.py` output,
|
|
- whether the problem reproduces with `FASTVIDEO_ATTENTION_BACKEND=TORCH_SDPA`.
|