[docs] PAN2: review follow-ups

- Guardrail install points to ifm-ai/pan2-guardrail, as in the vLLM-Omni and Diffusers ports.
- The basic example runs the release's own text-to-video structured caption (10 s, 241 frames) instead of a plain prompt.
- Docs and the flow-shift comment name the release checkpoint IFM/PAN2; the docs and cookbook no longer describe its release status or the parity tests' checkpoint.
- FASTVIDEO_FA4 is set only while the PAN2 pipeline loads, instead of for the rest of the process.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
William Lin
2026-11-02 22:45:33 +00:00
co-authored by Claude Opus 5.5
parent f67c5766f6
commit 8abf874081
10 changed files with 169 additions and 42 deletions
-1
View File
@@ -404,7 +404,6 @@
"evidence": "Source-backed",
"expected_artifact": "MP4 video under video_samples_pan2/",
"limitations": [
"The full IFM/PAN2 checkpoint is not public yet; numerical parity is checked with a public random-weight checkpoint.",
"The example keeps the 29B DiT resident in bf16 and offloads only the text encoder. GPU memory requirements have not been established for this recipe; set PAN2_NUM_GPUS to shard video tokens with sequence parallelism, which does not shard the DiT weights."
]
},
+5 -8
View File
@@ -11,13 +11,12 @@ for the model, sampler and parallelism details.
## Run on a GPU
Follow the [CUDA installation guide](../getting_started/installation/gpu.md), then run
from the repository root. The full `IFM/PAN2` checkpoint is not public yet; pass a
local directory or private repo you can access with `PAN2_MODEL_PATH`. PAN 2 runs
with its prompt and video guardrails by default, so install them first (see
[Guardrails](../inference/pan2.md#guardrails)):
from the repository root. Set `PAN2_MODEL_PATH` to run from a local copy of the
checkpoint. PAN 2 runs with its prompt and video guardrails by default, so install
them first (see [Guardrails](../inference/pan2.md#guardrails)):
```bash
pip install git+https://github.com/MBZUAI-IFM/PAN2-guardrail
pip install git+https://github.com/ifm-ai/pan2-guardrail
python examples/inference/basic/basic_pan2.py
```
@@ -74,6 +73,4 @@ Both recipes are **Source-backed**: the model ID, example and serving YAML are
checked against each other, and the serving defaults are tested against the
registered `pan2_t2v` preset. No GPU generation, memory measurement or HTTP
client run is recorded, so hardware fit and throughput are unknown. Numerical
parity with the reference implementation is checked on CPU with a public
random-weight checkpoint, which verifies the port and produces noise rather than
useful video.
parity with the reference implementation is checked by the PAN 2 parity tests.
+1 -1
View File
@@ -49,7 +49,7 @@ never deleted. See `examples/serving/README.md`.
| `openai_hunyuan15_i2v_480p.yaml` | HunyuanVideo 1.5 I2V 480p | `i2v` | CUDA REST | `hunyuan15-i2v-480p` | Source-backed; preset renders 720x1280 |
| `openai_kandinsky6_ti2va.yaml` | Kandinsky 6 Pro TI2VA | `t2v` | CUDA REST | `kandinsky6-ti2va` | Source-backed |
| `openai_kandinsky6_ti2va_distilled.yaml` | Kandinsky 6 Pro-distill TI2VA | `t2v` | CUDA REST | `kandinsky6-ti2va-distilled` | Source-backed |
| `openai_pan2.yaml` | PAN 2 | `t2v` (a reference switches to image-to-video) | CUDA REST | `pan2/cuda-rest` | Source-backed; checkpoint not public, no recorded run |
| `openai_pan2.yaml` | PAN 2 | `t2v` (a reference switches to image-to-video) | CUDA REST | `pan2/cuda-rest` | Source-backed; no recorded run |
| `openai_ltx2_distilled.yaml` | LTX2 Distilled | `t2v` | CUDA REST | `ltx2-distilled/cuda-rest` | Source-backed |
| `streaming_demo.yaml` | LTX2 Distilled | `t2v` | CUDA streaming | `ltx2-distilled/cuda-streaming` | Source-backed |
| `mlx_fasth3.yaml` | FastH3 V1 converted | `t2v` | MLX REST | none (legacy catalog only) | Source runtime; T2VA only |
+7 -9
View File
@@ -18,12 +18,13 @@ from the `pan2_t2v` and `pan2_i2v` presets:
|---|---|
| Resolution | 704x1248 (height and width must be multiples of 32) |
| Frames | 313, 13 s at 24 fps (`num_frames - 1` must be a multiple of 4) |
| Steps | 50, uniform sigmas from 1 to 0 with the checkpoint's flow shift (9 for `IFM/PAN2-TEST`, the reference sampler's value) |
| Steps | 50, uniform sigmas from 1 to 0 with the checkpoint's flow shift (9 for `IFM/PAN2`, the reference sampler's value) |
| Guidance | 2.5, classifier-free guidance, the reference launcher's default (3 before 2026-10-23). Image-to-video keeps the first latent frame out of guidance (`i2v_skip_first_latent_cfg`, on by default, as in the reference sampler) |
| Negative prompt | The release's structured negative caption (`negative_prompt.json` of the PAN2 repository) with its three `time_range_sec` values set to the clip length, `0 to 20` for 481 frames at 24 fps, as in the reference sampler. It is used whenever the request's negative prompt is empty; a single space gives the empty unconditional prompt PAN 2 trained with |
PAN 2 is trained on the structured captions its prompt rewriter produces, serialized as one-line JSON. Short raw
prompts work, with weaker results.
PAN 2 is trained on the structured captions its prompt rewriter produces, serialized as one-line JSON: the style,
lighting and camera, each subject, the first frame, and timed events. `basic_pan2.py` runs the release's own
text-to-video caption, a 10 s clip. Short raw prompts, like the one in the snippet below, work with weaker results.
## Usage
@@ -55,12 +56,12 @@ stored pixels: an EXIF orientation tag does not rotate it.
## Guardrails
PAN 2 runs with guardrails by default, as in its vLLM-Omni and Diffusers integrations. The
[`pan2-guardrail`](https://github.com/MBZUAI-IFM/PAN2-guardrail) package checks every prompt before any compute and every
[`pan2-guardrail`](https://github.com/ifm-ai/pan2-guardrail) package checks every prompt before any compute and every
generated video after decoding. A blocked prompt or video raises `PAN2GuardrailViolationError`. Install the package
before loading the pipeline:
```bash
pip install git+https://github.com/MBZUAI-IFM/PAN2-guardrail
pip install git+https://github.com/ifm-ai/pan2-guardrail
```
| Setting | Default | What it does |
@@ -99,7 +100,7 @@ PAN2 sampling turns these on by default (`PAN2PipelineConfig`); pass any of them
| `vae_decode_kernel_parallel` | With 2, 4 or 8 GPUs, splits that decode over them instead of decoding the whole video on each: a 20 s video takes 3.0 s on one B300, 1.55 s on two, 0.79 s on four and 0.47 s on eight. |
| `dit_fused_kernels` | Runs the DiT's fused Triton kernels for Q/K RMSNorm + RoPE and LayerNorm + modulation. Their rounding differs slightly from the unfused path. |
On Blackwell GPUs (sm_100 and newer) with FlashAttention-4 (`flash_attn.cute`) installed, PAN2 also attends with FA4 unless `FASTVIDEO_FA4` is set: 288 s instead of 670 s with the Torch SDPA fallback for 241 frames on one B300. Set `FASTVIDEO_FA4=0` to keep the backend FastVideo picks otherwise.
On Blackwell GPUs (sm_100 and newer) with FlashAttention-4 (`flash_attn.cute`) installed, PAN2 also attends with FA4 unless `FASTVIDEO_FA4` is set: 288 s instead of 670 s with the Torch SDPA fallback for 241 frames on one B300. Set `FASTVIDEO_FA4=0` to keep the backend FastVideo picks otherwise. PAN2 sets the variable only while it loads, but the attention modules keep FA4 for the rest of the process, so a later pipeline in the same process also attends with it.
On GPUs with room for every component (B200, B300), keep the text encoder resident as well:
`text_encoder_cpu_offload=False`.
@@ -112,9 +113,6 @@ For devices that cannot hold the DiT, see [CPU offloading](offloading.md) for `d
`dit_layerwise_offload`. Hardware recipes and timings reported by vLLM-Omni use its own offload implementation;
they have not been validated for FastVideo.
The full `IFM/PAN2` checkpoint is not public yet. Local parity tests use the public random-weight
`wuqing157/tiny-pan2-modular-pipe` checkpoint; it verifies the integration and produces noise rather than useful video.
## Differences from the vLLM-Omni integration
- The Cache-DiT `quality` policy and CFG-parallel execution are vLLM-Omni serving features and are not ported.