hao-ai-lab-FastVideo/v2/HANDOFF.md

Handoff — GPU bring-up of the v2 torch backend

For: an agent on a GPU box, branched from will/mini-fastvideo. Your job: take the written-not-run cuda backend to runs-and-generates, then commit + push.

Everything below is committed on will/mini-fastvideo and CPU-tested (204 tests pass). The torch path was authored on a machine with no GPU and no torch, so it is grounded in the real fastvideo APIs and cross-checked against the source, but never executed. That's what you finish.

---

0. Orientation (read these first, in order)

1. v2/README.md — what the whole v2 mini is (the (recipe, runtime) runtime; "architecture is real, kernels are toys"). The "Honest scope" paragraph says exactly what's wired.
2. v2/platform/backends/GPU_BRINGUP.md — your checklist: the ordered 10-step bring-up + the risk table (A–G), each tied to a # BRINGUP marker in the source. This handoff is orientation + process; GPU_BRINGUP.md is the work.
3. This file — the meta-instructions (verify bar, commit/push, gotchas).

1. What's already done (commits on this branch)

d6d0580a [fix]  correct GPU adapters against real fastvideo API (cross-check findings)
b8d78f40 [feat] real torch/CUDA backend (written-not-run) behind the cuda cells
27791b51 [feat] static-buffer capture form for the cudagraph step body (Path A)
ae6a170d [feat] piecewise CUDA-graph capture/replay at the step boundary (Path A)
9308d87e [feat] route all diffusion loops + RL recompute through the kernel table
7490c590 [feat] multi-backend dispatch substrate (device/arch/kernel registries)

The dispatch substrate (two tuple-keyed registries COMPONENTS(kind,device,variant) + KERNELS(op,device,arch,variant), a detected Platform, numpy terminal + parity oracle), the universal kernel seam (all diffusion loops + RL recompute go through model.platform.kernels), the piecewise cudagraph lifecycle, and the torch backend cells are all in place. On a GPU box, Platform.detect() returns a cuda platform and resolves the torch cells instead of the numpy toys — the loops/policies/scheduler/training are unchanged; only the resolved implementations differ.

2. The files you'll touch

File	What it is
v2/platform/backends/torch_adapters.py	TorchWanDiT / TorchWanVAE / TorchT5Encoder — wrap the real fastvideo.models.* (named by each card's load_id) to the mini's duck-typed surface. Built via the real FastVideo loaders.
v2/platform/backends/torch_kernels.py	torch flow_match_step / flow_sde_step (plain elementwise — there is no fused solver kernel in fastvideo-kernel; don't look for one).
v2/platform/backends/torch_cuda.py	registers the cuda cells as lazy trampolines (torch imported only inside builder bodies).
v2/card/specs.py	ComponentSpec.checkpoint — the per-component weights source (empty on toys; you fill it in).

The surface the adapters must honor (what the loops call): dit(latent, text_embed, sigma) -> velocity · vae.decode(latent) / vae.encode(video) · text_encoder.encode(text). The CPU toys in v2/models/backend.py are the reference behavior.

3. Your task (the gating items — full detail in GPU_BRINGUP.md)

1. Env: install torch + the parent fastvideo package + weights. (fastvideo source lives at /Users/willlin/src/FastVideo.)
2. Risk A — the one blocking gap: the builders call _load_via_fastvideo(...) → the real loaders need a FastVideoArgs, which _fastvideo_args(spec) builds minimally from spec.checkpoint. Confirm/extend its fields (model config, precision, parallelism). And stamp ComponentSpec.checkpoint onto the wan21 card — a tiny helper that maps a model root onto the three components is the cleanest way (the toy cards leave it "").
3. Work the risk list (A–G in GPU_BRINGUP.md). The interface contracts were cross-checked as matching (DiT returns bare velocity; timestep=sigma*1000; encode().mode(); .last_hidden_state; no fused solver kernel) — confirm them numerically. The construction layer was fixed (real loaders, set_forward_context, latent normalization, UMT5-from-config). What's left is box-dependent: FastVideoArgs fields, shift_factor placement/sign, exact tokenizer kwargs, FSDP sharding.
4. Bring up in order: build each component in isolation → one DiT step → one solver step → VAE decode → full t2v → SDE rollout → cudagraph capture (last).

4. The verification bar (how you know it's right)

• CPU suite must stay green: python3 -m pytest v2/ -q → still 204 passed. The torch path is gated available=False off-GPU; importing the backends must never import torch. If you break either, you broke the substrate. (v2/tests/test_torch_backend.py pins these.)
• Parity oracle is the spec: the substrate's whole point is that a real backend matches the numpy reference on the consistency ladder. On GPU, compare a full generation against a known-good fastvideo output — use the parent repo's SSIM regression harness (fastvideo/tests/ssim/). Target C4 (SSIM / artifact quality); component/trajectory parity (C0/C1) is bit-level vs the reference pipeline.
• Don't trust "it ran" — trust "it matched." A wrong timestep scale or shift_factor produces plausible-but-wrong video, not a crash (risks B/D). Diff against a reference, don't eyeball.

5. Commit + push

• You are on a GPU branch (branched from will/mini-fastvideo). Commit your bring-up fixes there, focused by concern (e.g. one commit per confirmed risk), in the existing style ([fix]/[feat] …).
• NEVER add Claude as a co-author (repo policy, /Users/willlin/src/.claude/CLAUDE.md).
• Do not rewrite or force-push the six commits above — build on top.
• When the CPU suite is green and a GPU generation matches the reference, push your branch.
• If you launch training/inference with wandb, log in with the token in the project CLAUDE.md (/Users/willlin/src/.claude/CLAUDE.md) — do not paste it into any committed file.

6. Gotchas (don't relearn these the hard way)

• No fused solver kernel exists. fastvideo-kernel ships only attention/norm/quant primitives; the cuda flow_match_step/flow_sde_step are plain torch by design. Don't hunt for a .cu solver.
• from_pretrained is not the loader. WanTransformer3DModel/AutoencoderKLWan have none — the real path is the *Loader().load(model_path, fastvideo_args) classes in fastvideo/models/loader/component_loader.py. The loader resolves the class from the checkpoint config (this is what makes UMT5-vs-T5 correct without hardcoding).
• T5 needs set_forward_context. A bare encoder forward reads stale/None global context.
• The loop surface stays numpy for bring-up; adapters marshal numpy↔torch at the boundary. A torch-native surface (latent on-device through forward→combine→solver) is the perf follow-up (Risk G), not bring-up — don't rewrite cfg.combine/precision.cast/the samplers yet.
• mse_grad_step raises on the cuda DiT (Risk F). RL/distill training on GPU is a separate workstream; inference + SDE rollout bring-up don't need it.
• cudagraph capture is last. The wan21 loop declares breakable_cudagraph; the v2 capturer models the lifecycle with a numpy StaticWorkspace. Capturing a real torch.cuda.CUDAGraph is GPU-only work and the riskiest step — leave it until inference is verified.