137 lines
6.9 KiB
Markdown
137 lines
6.9 KiB
Markdown
# Profiling model startup
|
|
|
|
Measure startup separately from inference latency before choosing what to retain
|
|
between development runs. The initial instrumentation covers component totals
|
|
and the native VAE safetensors loader, including Wan. It does not implement a
|
|
weight donor or CUDA IPC.
|
|
|
|
## Run the Wan VAE profile
|
|
|
|
Use a prepared CUDA environment and a local Wan 2.1 1.3B Diffusers VAE component directory:
|
|
|
|
```bash
|
|
python examples/benchmark/vae_startup.py \
|
|
--model-path /path/to/checkpoint/vae \
|
|
--output /tmp/vae-startup.json \
|
|
--tensor-output /tmp/vae-output.pt \
|
|
--synchronize
|
|
```
|
|
|
|
The script measures torch and FastVideo/helper imports separately, device and
|
|
process-group setup, VAE loading, and the first decode of a fixed seeded latent
|
|
input. It preserves launcher-provided rendezvous settings; otherwise it chooses a
|
|
local rendezvous port. It uses one GPU, FP32, and the default cached Wan VAE path.
|
|
Imports and configuration can themselves initialize CUDA. The
|
|
`cuda_initialized_before_device_setup` field records whether the explicit
|
|
device-setup interval begins with an existing context; import time includes
|
|
any initialization side effects.
|
|
It verifies output shape and finiteness; those checks alone do not establish
|
|
parity. Compare the saved tensors against an independent unchanged-source run.
|
|
|
|
Run the script in independent processes to measure repeated startup. Record the
|
|
source revision, checkpoint revision, hardware, runtime versions, cache state,
|
|
and process wall time alongside the JSON. The script excludes checkpoint
|
|
downloads. Cached local files do not establish a cold filesystem-cache run.
|
|
|
|
## Collect component timings in a worker
|
|
|
|
```python
|
|
from fastvideo.models.loader.startup import collect_startup_timings
|
|
|
|
with collect_startup_timings() as timings:
|
|
pipeline = build_pipeline(...) # Your existing construction call.
|
|
report = timings.to_dict()
|
|
```
|
|
|
|
Collection is opt-in and local to the current context/process. Install a
|
|
collector inside each spawned worker to measure its component loads; wrapping
|
|
only a multiprocess driver does not collect worker timings. Lazy components are
|
|
recorded when they actually load, if collection is still active.
|
|
|
|
Each phase has an ID, parent ID, start offset, duration, and success/error status.
|
|
Parent durations include child durations: do not sum both. Recording duration
|
|
also includes gaps between phases. A failed phase propagates its exception and
|
|
records an error; the collector restores the previous context on exit.
|
|
|
|
Default durations measure host wall time, including asynchronous submission.
|
|
For GPU completion timing, supply a synchronization callback. The example uses a
|
|
callback that synchronizes only after CUDA is initialized, so the first callback
|
|
does not move context initialization out of the device-setup interval.
|
|
Synchronization changes overlap and adds overhead. Compare end-to-end latency
|
|
with an uninstrumented run; do not interpret instrumented timings as free.
|
|
|
|
## Native VAE boundaries
|
|
|
|
`VAELoader.load()` retains the existing disk-loading behavior. Internally it now
|
|
separates two operations:
|
|
|
|
- `build_model()` constructs the native module and moves its initial tensors to
|
|
the target device. The caller supplies the resolved class, architecture config,
|
|
and dtype context.
|
|
- `load_weights()` reads safetensors, preserves checkpoint remapping and loading
|
|
strictness, assigns weights, applies existing overlays/memory-format changes,
|
|
and switches the module to evaluation mode.
|
|
|
|
Phases are `vae.construct`, `vae.device_transfer`, `vae.checkpoint_read`,
|
|
`vae.weight_assignment`, and `vae.finalize`, nested inside `component.vae` when
|
|
called through `PipelineComponentLoader`. Checkpoint reads can map files lazily;
|
|
page faults and transfers can occur during assignment. These labels describe
|
|
loader operations, not independent physical disk-I/O and PCIe measurements.
|
|
|
|
Checkpoint-backed special constructors retain their existing paths and only
|
|
receive the component-total interval. The transformer and shared text-encoder loaders also expose the boundaries
|
|
described below. First-use compilation and kernel warm-up still need separate
|
|
measurement. Reusing GPU tensor
|
|
storage across processes will require a separate compatibility manifest,
|
|
ownership/lifetime protocol, and loader implementation. The extracted `load_weights()` method still copies from disk; a future IPC
|
|
loader must bind storage instead of calling `load_state_dict()` and claiming
|
|
zero-copy attachment.
|
|
|
|
## Profile a complete Wan pipeline
|
|
|
|
The whole-pipeline example loads the DiT, text encoder, tokenizer, and decoder
|
|
VAE, then verifies a fixed tiny request:
|
|
|
|
```bash
|
|
python examples/benchmark/wan_pipeline_startup.py \
|
|
--model-path /path/to/wan-diffusers-checkpoint \
|
|
--output /tmp/wan-startup.json \
|
|
--tensor-output /tmp/wan-output.pt \
|
|
--synchronize
|
|
```
|
|
|
|
It selects one GPU and the in-process executor so the collector sees all loads.
|
|
Add `--fsdp` to exercise single-rank FSDP inference; the default is unsharded.
|
|
The JSON records resolved offload/FSDP settings and the live transformer
|
|
FSDP-module status; assertions require both to match the request.
|
|
Both paths disable CPU offload and compilation and use the checkpoint's native
|
|
Wan 2.1 1.3B pipeline defaults. The request is 9 frames at 128x128, four steps,
|
|
seed 1024, with a fixed prompt. It checks tensor shape and finiteness and can
|
|
save the output for an independent original/refactored comparison. It is not a
|
|
video-quality or production-throughput benchmark. No video file is written.
|
|
|
|
`pipeline.create` includes configuration, worker/device setup and component
|
|
loading. Its children include the `component.*` totals. `first_request` measures
|
|
the complete tiny request, including any first-use warm-up and output handling;
|
|
it does not separate JIT, autotuning, text encoding, denoising, and decoding.
|
|
The reported CUDA-initialized flag identifies earlier import side effects.
|
|
|
|
Native transformer phases are `transformer.construct` (meta construction),
|
|
`transformer.shard_setup`, `transformer.hydrate`, `transformer.finalize`, and
|
|
`transformer.compile_setup`. The last two cover post-load conversion/buffers and
|
|
compile registration respectively, not the first execution of compiled code.
|
|
The sharding and compile-setup intervals may be near zero when disabled.
|
|
|
|
Native shared encoder-loader phases are `text_encoder.construct`,
|
|
`text_encoder.hydrate`, `text_encoder.device_transfer`, and
|
|
`text_encoder.offload_setup`. The shared path also serves image encoders; use the
|
|
parent `component.*` name to identify the role. HF passthrough encoders report
|
|
`text_encoder.pretrained` as one combined construction/loading operation.
|
|
|
|
Hydration intervals contain consumption of the lazy checkpoint iterators,
|
|
conversion/fusion, and weight assignment. They deliberately do not call iterator
|
|
creation a disk-read measurement. In particular, a CUDA IPC loader would not
|
|
necessarily save the entire hydration interval. Keep donor startup, attachment,
|
|
compatibility validation, and an independent disk-loaded confirmation in any
|
|
future comparison.
|