66 lines
3.1 KiB
Markdown
66 lines
3.1 KiB
Markdown
# Profiling FastVideo
|
|
|
|
For GB200 sequence-parallel transport tuning and its long-training memory
|
|
policy, see [Ulysses performance](ulysses_performance.md).
|
|
|
|
:::warning
|
|
Profiling is only intended for FastVideo developers and maintainers to understand the proportion of time spent in different parts of the codebase. **FastVideo end-users should never turn on profiling** as it will significantly slow down inference.
|
|
:::
|
|
|
|
## Profiling with PyTorch
|
|
|
|
FastVideo exposes a process-wide torch profiler that you can enable via environment variables. Set `FASTVIDEO_TORCH_PROFILER_DIR` to an absolute directory path to start collecting traces, and specify the regions you want recorded with `FASTVIDEO_TORCH_PROFILE_REGIONS`:
|
|
|
|
```bash
|
|
FASTVIDEO_TORCH_PROFILER_DIR=/mnt/traces/fastvideo \
|
|
FASTVIDEO_TORCH_PROFILE_REGIONS="profiler_region_model_loading,profiler_region_training_step"
|
|
```
|
|
|
|
All profiled regions must be registered in `fastvideo.profiler`; the current list includes:
|
|
|
|
- `profiler_region_model_loading` — pipeline/module loading
|
|
- `profiler_region_inference_pre_denoising`
|
|
- `profiler_region_inference_denoising`
|
|
- `profiler_region_inference_post_denoising`
|
|
- `profiler_region_training_checkpoint_saving`
|
|
- `profiler_region_training_dit`
|
|
- `profiler_region_training_validation`
|
|
- `profiler_region_training_epoch`
|
|
- `profiler_region_training_step`
|
|
- `profiler_region_training_backward`
|
|
- `profiler_region_training_optimizer`
|
|
- `profiler_region_distillation_teacher_forward`
|
|
- `profiler_region_distillation_student_forward`
|
|
- `profiler_region_distillation_loss`
|
|
- `profiler_region_distillation_update`
|
|
|
|
While profiling is enabled, FastVideo records additional annotations:
|
|
|
|
- `fastvideo.region::<name>` spans are emitted when entering a region.
|
|
- `fastvideo.profiler.enable_collection` / `fastvideo.profiler.disable_collection` events mark when torch profiler collection is toggled on or off.
|
|
|
|
Only one profiler instance is created per process; subsequent pipelines reuse the same controller. If you set `FASTVIDEO_TORCH_PROFILE_REGIONS` incorrectly (e.g. misspelled name), FastVideo logs a warning and ignores that entry.
|
|
|
|
Additional knobs:
|
|
|
|
- `FASTVIDEO_TORCH_PROFILER_RECORD_SHAPES`
|
|
- `FASTVIDEO_TORCH_PROFILER_WITH_PROFILE_MEMORY`
|
|
- `FASTVIDEO_TORCH_PROFILER_WITH_STACK`
|
|
- `FASTVIDEO_TORCH_PROFILER_WITH_FLOPS`
|
|
|
|
Traces can be visualized using [https://ui.perfetto.dev/](https://ui.perfetto.dev/).
|
|
|
|
### Best Practices
|
|
|
|
- Keep the profiled step count small; traces can be large and slow down job shutdown while the profiler flushes data.
|
|
- After profiling, clean up trace directories to avoid filling disk storage.
|
|
- When adding new regions, register them in `fastvideo.profiler` and wrap the corresponding code block with `with self.profiler_controller.region("your_region"):` or the `@profile_region` decorator.
|
|
|
|
## Related: Activation Trace Mode
|
|
|
|
For per-layer **numerical-divergence** debugging (parity bring-up against an
|
|
upstream reference), use the env-gated [Activation Trace Mode](activation_trace.md)
|
|
instead of the torch profiler. Activation trace dumps per-tensor stats to
|
|
JSONL for offline `diff`-ing; the torch profiler captures kernel timing.
|
|
They solve different problems and can run together if needed.
|