hao-ai-lab-FastVideo/docs/contributing/startup_profiling.md

Profiling model startup

Measure startup separately from inference latency before choosing what to retain between development runs. The initial instrumentation covers component totals and the native VAE safetensors loader, including Wan. It does not implement a weight donor or CUDA IPC.

Run the Wan VAE profile

Use a prepared CUDA environment and a local Wan 2.1 1.3B Diffusers VAE component directory:

python examples/benchmark/vae_startup.py \
  --model-path /path/to/checkpoint/vae \
  --output /tmp/vae-startup.json \
  --tensor-output /tmp/vae-output.pt \
  --synchronize

The script measures torch and FastVideo/helper imports separately, device and process-group setup, VAE loading, and the first decode of a fixed seeded latent input. It preserves launcher-provided rendezvous settings; otherwise it chooses a local rendezvous port. It uses one GPU, FP32, and the default cached Wan VAE path. Imports and configuration can themselves initialize CUDA. The cuda_initialized_before_device_setup field records whether the explicit device-setup interval begins with an existing context; import time includes any initialization side effects. It verifies output shape and finiteness; those checks alone do not establish parity. Compare the saved tensors against an independent unchanged-source run.

Run the script in independent processes to measure repeated startup. Record the source revision, checkpoint revision, hardware, runtime versions, cache state, and process wall time alongside the JSON. The script excludes checkpoint downloads. Cached local files do not establish a cold filesystem-cache run.

Collect component timings in a worker

from fastvideo.models.loader.startup import collect_startup_timings

with collect_startup_timings() as timings:
    pipeline = build_pipeline(...)  # Your existing construction call.
report = timings.to_dict()

Collection is opt-in and local to the current context/process. Install a collector inside each spawned worker to measure its component loads; wrapping only a multiprocess driver does not collect worker timings. Lazy components are recorded when they actually load, if collection is still active.

Each phase has an ID, parent ID, start offset, duration, and success/error status. Parent durations include child durations: do not sum both. Recording duration also includes gaps between phases. A failed phase propagates its exception and records an error; the collector restores the previous context on exit.

Default durations measure host wall time, including asynchronous submission. For GPU completion timing, supply a synchronization callback. The example uses a callback that synchronizes only after CUDA is initialized, so the first callback does not move context initialization out of the device-setup interval. Synchronization changes overlap and adds overhead. Compare end-to-end latency with an uninstrumented run; do not interpret instrumented timings as free.

Native VAE boundaries

VAELoader.load() retains the existing disk-loading behavior. Internally it now separates two operations:

• build_model() constructs the native module and moves its initial tensors to the target device. The caller supplies the resolved class, architecture config, and dtype context.
• load_weights() reads safetensors, preserves checkpoint remapping and loading strictness, assigns weights, applies existing overlays/memory-format changes, and switches the module to evaluation mode.

Phases are vae.construct, vae.device_transfer, vae.checkpoint_read, vae.weight_assignment, and vae.finalize, nested inside component.vae when called through PipelineComponentLoader. Checkpoint reads can map files lazily; page faults and transfers can occur during assignment. These labels describe loader operations, not independent physical disk-I/O and PCIe measurements.

Checkpoint-backed special constructors retain their existing paths and only receive the component-total interval. The transformer and shared text-encoder loaders also expose the boundaries described below. First-use compilation and kernel warm-up still need separate measurement. Reusing GPU tensor storage across processes will require a separate compatibility manifest, ownership/lifetime protocol, and loader implementation. The extracted load_weights() method still copies from disk; a future IPC loader must bind storage instead of calling load_state_dict() and claiming zero-copy attachment.

Profile a complete Wan pipeline

The whole-pipeline example loads the DiT, text encoder, tokenizer, and decoder VAE, then verifies a fixed tiny request:

python examples/benchmark/wan_pipeline_startup.py \
  --model-path /path/to/wan-diffusers-checkpoint \
  --output /tmp/wan-startup.json \
  --tensor-output /tmp/wan-output.pt \
  --synchronize

It selects one GPU and the in-process executor so the collector sees all loads. Add --fsdp to exercise single-rank FSDP inference; the default is unsharded. The JSON records resolved offload/FSDP settings and the live transformer FSDP-module status; assertions require both to match the request. Both paths disable CPU offload and compilation and use the checkpoint's native Wan 2.1 1.3B pipeline defaults. The request is 9 frames at 128x128, four steps, seed 1024, with a fixed prompt. It checks tensor shape and finiteness and can save the output for an independent original/refactored comparison. It is not a video-quality or production-throughput benchmark. No video file is written.

pipeline.create includes configuration, worker/device setup and component loading. Its children include the component.* totals. first_request measures the complete tiny request, including any first-use warm-up and output handling; it does not separate JIT, autotuning, text encoding, denoising, and decoding. The reported CUDA-initialized flag identifies earlier import side effects.

Native transformer phases are transformer.construct (meta construction), transformer.shard_setup, transformer.hydrate, transformer.finalize, and transformer.compile_setup. The last two cover post-load conversion/buffers and compile registration respectively, not the first execution of compiled code. The sharding and compile-setup intervals may be near zero when disabled.

Native shared encoder-loader phases are text_encoder.construct, text_encoder.hydrate, text_encoder.device_transfer, and text_encoder.offload_setup. The shared path also serves image encoders; use the parent component.* name to identify the role. HF passthrough encoders report text_encoder.pretrained as one combined construction/loading operation.

Hydration intervals contain consumption of the lazy checkpoint iterators, conversion/fusion, and weight assignment. They deliberately do not call iterator creation a disk-read measurement. In particular, a CUDA IPC loader would not necessarily save the entire hydration interval. Keep donor startup, attachment, compatibility validation, and an independent disk-loaded confirmation in any future comparison.