[docs] PAN2: review follow-ups

- Guardrail install points to ifm-ai/pan2-guardrail, as in the vLLM-Omni and Diffusers ports.
- The basic example runs the release's own text-to-video structured caption (10 s, 241 frames) instead of a plain prompt.
- Docs and the flow-shift comment name the release checkpoint IFM/PAN2; the docs and cookbook no longer describe its release status or the parity tests' checkpoint.
- FASTVIDEO_FA4 is set only while the PAN2 pipeline loads, instead of for the rest of the process.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
William Lin
2026-11-02 22:45:33 +00:00
co-authored by Claude Opus 5.5
parent f67c5766f6
commit 8abf874081
10 changed files with 169 additions and 42 deletions
+7 -9
View File
@@ -18,12 +18,13 @@ from the `pan2_t2v` and `pan2_i2v` presets:
|---|---|
| Resolution | 704x1248 (height and width must be multiples of 32) |
| Frames | 313, 13 s at 24 fps (`num_frames - 1` must be a multiple of 4) |
| Steps | 50, uniform sigmas from 1 to 0 with the checkpoint's flow shift (9 for `IFM/PAN2-TEST`, the reference sampler's value) |
| Steps | 50, uniform sigmas from 1 to 0 with the checkpoint's flow shift (9 for `IFM/PAN2`, the reference sampler's value) |
| Guidance | 2.5, classifier-free guidance, the reference launcher's default (3 before 2026-10-23). Image-to-video keeps the first latent frame out of guidance (`i2v_skip_first_latent_cfg`, on by default, as in the reference sampler) |
| Negative prompt | The release's structured negative caption (`negative_prompt.json` of the PAN2 repository) with its three `time_range_sec` values set to the clip length, `0 to 20` for 481 frames at 24 fps, as in the reference sampler. It is used whenever the request's negative prompt is empty; a single space gives the empty unconditional prompt PAN 2 trained with |
PAN 2 is trained on the structured captions its prompt rewriter produces, serialized as one-line JSON. Short raw
prompts work, with weaker results.
PAN 2 is trained on the structured captions its prompt rewriter produces, serialized as one-line JSON: the style,
lighting and camera, each subject, the first frame, and timed events. `basic_pan2.py` runs the release's own
text-to-video caption, a 10 s clip. Short raw prompts, like the one in the snippet below, work with weaker results.
## Usage
@@ -55,12 +56,12 @@ stored pixels: an EXIF orientation tag does not rotate it.
## Guardrails
PAN 2 runs with guardrails by default, as in its vLLM-Omni and Diffusers integrations. The
[`pan2-guardrail`](https://github.com/MBZUAI-IFM/PAN2-guardrail) package checks every prompt before any compute and every
[`pan2-guardrail`](https://github.com/ifm-ai/pan2-guardrail) package checks every prompt before any compute and every
generated video after decoding. A blocked prompt or video raises `PAN2GuardrailViolationError`. Install the package
before loading the pipeline:
```bash
pip install git+https://github.com/MBZUAI-IFM/PAN2-guardrail
pip install git+https://github.com/ifm-ai/pan2-guardrail
```
| Setting | Default | What it does |
@@ -99,7 +100,7 @@ PAN2 sampling turns these on by default (`PAN2PipelineConfig`); pass any of them
| `vae_decode_kernel_parallel` | With 2, 4 or 8 GPUs, splits that decode over them instead of decoding the whole video on each: a 20 s video takes 3.0 s on one B300, 1.55 s on two, 0.79 s on four and 0.47 s on eight. |
| `dit_fused_kernels` | Runs the DiT's fused Triton kernels for Q/K RMSNorm + RoPE and LayerNorm + modulation. Their rounding differs slightly from the unfused path. |
On Blackwell GPUs (sm_100 and newer) with FlashAttention-4 (`flash_attn.cute`) installed, PAN2 also attends with FA4 unless `FASTVIDEO_FA4` is set: 288 s instead of 670 s with the Torch SDPA fallback for 241 frames on one B300. Set `FASTVIDEO_FA4=0` to keep the backend FastVideo picks otherwise.
On Blackwell GPUs (sm_100 and newer) with FlashAttention-4 (`flash_attn.cute`) installed, PAN2 also attends with FA4 unless `FASTVIDEO_FA4` is set: 288 s instead of 670 s with the Torch SDPA fallback for 241 frames on one B300. Set `FASTVIDEO_FA4=0` to keep the backend FastVideo picks otherwise. PAN2 sets the variable only while it loads, but the attention modules keep FA4 for the rest of the process, so a later pipeline in the same process also attends with it.
On GPUs with room for every component (B200, B300), keep the text encoder resident as well:
`text_encoder_cpu_offload=False`.
@@ -112,9 +113,6 @@ For devices that cannot hold the DiT, see [CPU offloading](offloading.md) for `d
`dit_layerwise_offload`. Hardware recipes and timings reported by vLLM-Omni use its own offload implementation;
they have not been validated for FastVideo.
The full `IFM/PAN2` checkpoint is not public yet. Local parity tests use the public random-weight
`wuqing157/tiny-pan2-modular-pipe` checkpoint; it verifies the integration and produces noise rather than useful video.
## Differences from the vLLM-Omni integration
- The Cache-DiT `quality` policy and CFG-parallel execution are vLLM-Omni serving features and are not ported.