hao-ai-lab-FastVideo/docs/contributing/profiling.md

Profiling FastVideo

For GB200 sequence-parallel transport tuning and its long-training memory policy, see Ulysses performance.

!!! warning Profiling is only intended for FastVideo developers and maintainers to understand the proportion of time spent in different parts of the codebase. FastVideo end-users should never turn on profiling as it will significantly slow down inference.

Profiling with PyTorch

FastVideo exposes a process-wide torch profiler that you can enable via environment variables. Set FASTVIDEO_TORCH_PROFILER_DIR to an absolute directory path to start collecting traces, and specify the regions you want recorded with FASTVIDEO_TORCH_PROFILE_REGIONS:

FASTVIDEO_TORCH_PROFILER_DIR=/mnt/traces/fastvideo \
FASTVIDEO_TORCH_PROFILE_REGIONS="profiler_region_model_loading,profiler_region_training_step"

All profiled regions must be registered in fastvideo.profiler; the current list includes:

• profiler_region_model_loading — pipeline/module loading
• profiler_region_inference_pre_denoising
• profiler_region_inference_denoising
• profiler_region_inference_post_denoising
• profiler_region_training_checkpoint_saving
• profiler_region_training_dit
• profiler_region_training_validation
• profiler_region_training_epoch
• profiler_region_training_step
• profiler_region_training_backward
• profiler_region_training_optimizer
• profiler_region_distillation_teacher_forward
• profiler_region_distillation_student_forward
• profiler_region_distillation_loss
• profiler_region_distillation_update

While profiling is enabled, FastVideo records additional annotations:

• fastvideo.region::<name> spans are emitted when entering a region.
• fastvideo.profiler.enable_collection / fastvideo.profiler.disable_collection events mark when torch profiler collection is toggled on or off.

Only one profiler instance is created per process; subsequent pipelines reuse the same controller. If you set FASTVIDEO_TORCH_PROFILE_REGIONS incorrectly (e.g. misspelled name), FastVideo logs a warning and ignores that entry.

Additional knobs:

• FASTVIDEO_TORCH_PROFILER_RECORD_SHAPES
• FASTVIDEO_TORCH_PROFILER_WITH_PROFILE_MEMORY
• FASTVIDEO_TORCH_PROFILER_WITH_STACK
• FASTVIDEO_TORCH_PROFILER_WITH_FLOPS

Traces can be visualized using https://ui.perfetto.dev/.

Best Practices

• Keep the profiled step count small; traces can be large and slow down job shutdown while the profiler flushes data.
• After profiling, clean up trace directories to avoid filling disk storage.
• When adding new regions, register them in fastvideo.profiler and wrap the corresponding code block with with self.profiler_controller.region("your_region"): or the @profile_region decorator.

Related: Activation Trace Mode

For per-layer numerical-divergence debugging (parity bring-up against an upstream reference), use the env-gated Activation Trace Mode instead of the torch profiler. Activation trace dumps per-tensor stats to JSONL for offline diff-ing; the torch profiler captures kernel timing. They solve different problems and can run together if needed.