Files

T

leffffandClaude Fable 5 5aae1c093a [bugfix]: address third-round Kandinsky5 QAD review findings

- Never touch the user's documented dataset: preprocess roots are now
  env-overridable (KANDINSKY5_OVERFIT_DATA_DIR/_OUTPUT_DIR) and the
  nightly keeps every artifact under a single test-owned
  data/kandinsky5_e2e/ root, cleaning only that
- Override method.generator_update_interval to 1 in the E2E stage-2 run:
  with the recipe's 5, a 3-step run performed zero student updates and the
  exported "student" was stage-1 weights
- Harden the media oracle: exact frame count/geometry (closes the SSIM
  shorter-clip truncation hole), per-frame luma spatial std (solid clip at
  reference mean scored 15.4 on the old global RGB std), consecutive-frame
  temporal diff (frozen clips), and MS-SSIM floor recalibrated to 0.95
  against measured known-bads (solid 0.9189, frozen 0.8597, good 1.0000);
  known-bad clips are permanent self-calibrating regressions in
  fastvideo/tests/workflow/test_kandinsky5_e2e_media_oracle.py
- Capability-gate ATTN_QAT_INFER: is_attn_qat_infer_available() now
  requires an sm_120/sm_121 device in addition to the extension import
  (CUDA 13 wheels can bundle the extension on H100/GB200, where the first
  kernel call failed instead of falling back); the real platform resolver
  is covered in fastvideo/tests/api/test_attn_qat_infer_capability_gate.py
- Remove the now-stale golden reference: the generator-update fix changes
  what stage 2 trains, so the reference must be re-recorded from the
  corrected run (KANDINSKY5_E2E_WRITE_REFERENCE=1 bootstrap + review)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

2026-07-26 02:31:15 -07:00

3.4 KiB

Raw Blame History

Debugging

This page collects practical debugging steps for FastVideo inference issues.

Collect Environment Info

From the repository root, run:

python collect_env.py

Attach the output when filing a GitHub issue.

Increase Logging

FastVideo logging level is controlled by environment variables:

FASTVIDEO_LOGGING_LEVEL=DEBUG \
FASTVIDEO_STAGE_LOGGING=1 \
python your_script.py

Useful variables:

FASTVIDEO_LOGGING_LEVEL: DEBUG, INFO, WARNING, ERROR
FASTVIDEO_STAGE_LOGGING: print per-stage timings during pipeline execution
FASTVIDEO_ATTENTION_BACKEND: force an attention backend (for example TORCH_SDPA, FLASH_ATTN, SAGE_ATTN_THREE, or ATTN_QAT_INFER, or ATTN_QAT_TRAIN)

Layer-by-Layer Activation Tracing

For numerical-divergence debugging — typically when porting a new model and needing to find the first layer where FastVideo and an upstream reference produce different outputs — use the env-gated activation trace mode:

FASTVIDEO_TRACE_ACTIVATIONS=1 \
FASTVIDEO_TRACE_LAYERS="^transformer\.blocks\.\d+$" \
FASTVIDEO_TRACE_OUTPUT=/tmp/fv_trace.jsonl \
python your_script.py

The trace dumps per-tensor stats (abs_mean, sum, shape, etc.) to a JSONL file. Run the same workload with tracing on the upstream side, then diff the two files to localize the first divergent layer.

See Activation Trace Mode for the full guide (env var reference, JSONL output schema, parity-debug workflow, performance impact, and troubleshooting).

Common Failure Modes

Out-of-memory

Try, in order:

Reduce height, width, num_frames, or num_inference_steps.
Enable offloading flags such as dit_layerwise_offload (single GPU) or use_fsdp_inference (multi-GPU).
Enable vae_cpu_offload, image_encoder_cpu_offload, and text_encoder_cpu_offload.

See Inference Offloading for recommended combinations.

Attention backend import errors

If forcing a backend fails, verify optional dependencies are installed:

FLASH_ATTN: flash-attn
VIDEO_SPARSE_ATTN: fastvideo-kernel
SLIDING_TILE_ATTN: STA legacy workflow in sta_do_not_delete + fastvideo-kernel
SAGE_ATTN: SageAttention package
SAGE_ATTN_THREE: upstream sageattn3 package
ATTN_QAT_INFER: fastvideo-kernel checkout/source install that exposes attn_qat_infer, AND a consumer-Blackwell sm_120 GPU -- on any other device the backend reports unavailable (even if a CUDA 13 wheel bundles the extension) and selection falls back to FlashAttention
ATTN_QAT_TRAIN: fastvideo-kernel; its runtime-JIT Triton implementation selects an optimized route on SM100, joins the quantized and STE P@V paths on SM120, and retains the previous route for unsupported configurations. See Attn-QAT Training for architecture controls.

As a fallback, use:

export FASTVIDEO_ATTENTION_BACKEND=TORCH_SDPA

Configuration parsing errors

When using --config, keep keys aligned with CLI argument names (underscores or hyphens are both accepted). For nested config values, use nested objects (vae_config, dit_config) rather than dotted keys.

Issue Template

When opening an issue, include:

exact command or Python snippet,
model ID/path,
full traceback,
collect_env.py output,
whether the problem reproduces with FASTVIDEO_ATTENTION_BACKEND=TORCH_SDPA.

3.4 KiB Raw Blame History