nodes/hao-ai-lab-FastVideo
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

[new-model] Re-add PAN2 text-to-video and image-to-video (reverts #2025)

Browse Source 
This reverts commit 6743f4b4b, restoring #1961.
This commit is contained in:
William Lin
2026-11-03 20:31:55 +00:00
parent 661d18ba55
commit fa4e3659c7
46 changed files with 3990 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/assets/cookbook-recipes.json
+27
View File
@@ -381,6 +381,33 @@
"expected_artifact": "Upscaled MP4 under outputs_video/kandinsky6_sr/",
"related": ["kandinsky6-vsr", "kandinsky6-ti2va-piflow"]
},
{
"id": "pan2-ti2v",
"family": "pan2",
"stage": "inference",
"task": "Text or image to video",
"label": "PAN 2",
"summary": "Generate a 13-second, 704 × 1248 video at 24 fps from text, or set IMAGE_PATH in the maintained example to condition on a first frame.",
"model": "IFM/PAN2",
"source": "examples/inference/basic/basic_pan2.py",
"serving": {
"source": "examples/serving/openai_pan2.yaml",
"install": "UV_TORCH_BACKEND=cu130 uv pip install -e .",
"task": "Text to video",
"limitations": [
"This server walkthrough sends a text prompt only. The server API also accepts an image reference (image_reference or input_reference) and then runs image-to-video from that first frame, but the checked-in clients do not upload one."
]
},
"command": "python examples/inference/basic/basic_pan2.py",
"gpu_types": ["NVIDIA"],
"hardware": {"platform": "cuda", "gpu_count": 1, "evidence": "source-configured"},
"evidence": "Source-backed",
"expected_artifact": "MP4 video under video_samples_pan2/",
"limitations": [
"The full IFM/PAN2 checkpoint is not public yet; numerical parity is checked with a public random-weight checkpoint.",
"The example keeps the 29B DiT resident in bf16 and offloads only the text encoder. GPU memory requirements have not been established for this recipe; set PAN2_NUM_GPUS to shard video tokens with sequence parallelism, which does not shard the DiT weights."
]
},
{
"id": "flux2-klein-t2i",
"family": "flux",
docs/assets/logos/SOURCES.md
+1
View File
@@ -13,6 +13,7 @@ plain typographic tile instead — that tile is a UI placeholder, not a logo.
| `nvidia.webp` | [Official NVIDIA Hugging Face organization avatar](https://huggingface.co/nvidia) | Cosmos and GEN3C cards, Cosmos page header |
| `kandinsky.webp` | [Official Kandinsky Lab Hugging Face organization avatar](https://huggingface.co/kandinskylab) | Kandinsky 5 and Kandinsky 6 cards and page headers |
| `black-forest-labs.webp` | [Official Black Forest Labs Hugging Face organization avatar](https://huggingface.co/black-forest-labs) | FLUX family card and page header |
| `ifm.webp` | [Official Institute of Foundation Models (IFM) Hugging Face organization avatar](https://huggingface.co/IFM) | PAN 2 family card and page header |
| `qwen.webp` | [Official Qwen Hugging Face organization avatar](https://huggingface.co/Qwen) | Qwen-Image-2.1 family card and page header |
| `minimax.webp` | [Official MiniMax Hugging Face organization avatar](https://huggingface.co/MiniMaxAI) | MiniMax H3 family card and page header |
| `pika.webp` | [Official Pika-Labs Hugging Face organization avatar](https://huggingface.co/Pika-Labs) | Pika H3 Flex card and page header |
docs/assets/logos/ifm.webp
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 4.2 KiB

docs/cookbook/index.md
+1
View File
@@ -6,6 +6,7 @@ The interactive catalog groups models by family and generates YAML for supported
- [FastH3](minimax-h3.md)
- [Wan and FastMetal](wan.md)
- [LTX](ltx.md)
- [PAN 2](pan2.md)
- [FLUX](flux.md)
- [Qwen-Image-2.1](qwen-image21.md)
- [Stable Diffusion](stable-diffusion.md)
docs/cookbook/pan2.md
+76
View File
@@ -0,0 +1,76 @@
# PAN 2 recipes
<RecipeFamily family="pan2" />
Generate a video from text, or from a first frame, with `IFM/PAN2`. One pipeline
serves both modes: leave out the first frame for text-to-video and supply one
for image-to-video. The defaults are 704×1248, 313 frames (13 s at 24 fps),
50 steps and guidance 3.0. See [PAN 2 Text/Image to Video](../inference/pan2.md)
for the model, sampler and parallelism details.
## Run on a GPU
Follow the [CUDA installation guide](../getting_started/installation/gpu.md), then run
from the repository root. The full `IFM/PAN2` checkpoint is not public yet; pass a
local directory or private repo you can access with `PAN2_MODEL_PATH`.
```bash
python examples/inference/basic/basic_pan2.py
```
Set `IMAGE_PATH` in the script to a first frame for image-to-video. With more
GPUs, `PAN2_NUM_GPUS=4` shards the video tokens with sequence parallelism; the
count must divide PAN 2's 32 attention heads. Each rank still holds the full
29B DiT, so extra GPUs speed up generation but do not reduce per-device weight
memory. See [CPU offloading](../inference/offloading.md) for devices that cannot
hold the DiT.
## Serve with the OpenAI-compatible API
`examples/serving/openai_pan2.yaml` serves PAN 2 through the shared
`/v1/videos` routes. It mirrors the Python example: one GPU, the text encoder
offloaded between requests, and the pipeline's own sampling defaults.
```bash
fastvideo serve --config examples/serving/openai_pan2.yaml --server.host 127.0.0.1
```
Text-to-video uses the same request as any other served video model, with the
alias `pan2`:
```bash
curl --fail-with-body http://127.0.0.1:8000/v1/videos \
-H 'Content-Type: application/json' \
-d '{"model": "pan2", "prompt": "A red fox trots through fresh snow in a pine forest at dawn."}'
```
For image-to-video, add a first frame in `image_reference` (or the legacy
`input_reference`). The server then runs image-to-video; no second deployment or
workload is needed.
```bash
curl --fail-with-body http://127.0.0.1:8000/v1/videos \
-H 'Content-Type: application/json' \
-d '{"model": "pan2", "prompt": "The fox turns toward the camera.",
"image_reference": {"image_url": "https://example.com/first-frame.png"}}'
```
Local image paths must sit under `server.media_roots`, and private or loopback
URLs need `server.allow_private_media_urls`. Poll `/v1/videos/{id}` and download
`/v1/videos/{id}/content` as in the [serving guide](openai-api.md). Generation
time for the default 13-second clip is not measured, so raise the client's
polling deadline for long jobs instead of treating a slow job as failed.
The cookbook's generated serving clients send a text prompt only. The image
request above is the supported way to supply a first frame over HTTP, and no
recorded generation backs it yet.
## Evidence
Both recipes are **Source-backed**: the model ID, example and serving YAML are
checked against each other, and the serving defaults are tested against the
registered `pan2_t2v` preset. No GPU generation, memory measurement or HTTP
client run is recorded, so hardware fit and throughput are unknown. Numerical
parity with the reference implementation is checked on CPU with a public
random-weight checkpoint, which verifies the port and produces noise rather than
useful video.
docs/cookbook/recipes/pan2.yaml
+38
View File
@@ -0,0 +1,38 @@
title: PAN 2
summary: Text-to-video and first-frame image-to-video from the 29B PAN 2 mixture-of-transformers model.
deployments:
cuda-rest:
runtime: fastvideo-cuda-rest
workload: t2v
defaults: examples/serving/openai_pan2.yaml
requirements:
- >-
Requests without a reference run text-to-video. Add image_reference or input_reference
to condition on a first frame; the same server and pipeline handle both.
- >-
To use more than one GPU, set sp_size equal to num_gpus (Ulysses sequence parallelism).
It must divide PAN 2's 32 attention heads, and each rank still holds the full DiT weights.
options:
- server.*
- generator.engine.num_gpus
- generator.engine.parallelism.*
- "!generator.engine.parallelism.hsdp_*"
- generator.engine.offload.text_encoder
- generator.engine.offload.vae
- default_request.sampling.num_frames
- default_request.sampling.height
- default_request.sampling.width
- default_request.sampling.fps
overrides:
default_request.sampling.num_frames:
default: 313
description: num_frames - 1 must be a multiple of 4.
default_request.sampling.height:
default: 704
minimum: 32
multipleOf: 32
default_request.sampling.width:
default: 1248
minimum: 32
multipleOf: 32
docs/cookbook/serving-coverage.md
+1
View File
@@ -49,6 +49,7 @@ never deleted. See `examples/serving/README.md`.
| `openai_hunyuan15_i2v_480p.yaml` | HunyuanVideo 1.5 I2V 480p | `i2v` | CUDA REST | `hunyuan15-i2v-480p` | Source-backed; preset renders 720x1280 |
| `openai_kandinsky6_ti2va.yaml` | Kandinsky 6 Pro TI2VA | `t2v` | CUDA REST | `kandinsky6-ti2va` | Source-backed |
| `openai_kandinsky6_ti2va_distilled.yaml` | Kandinsky 6 Pro-distill TI2VA | `t2v` | CUDA REST | `kandinsky6-ti2va-distilled` | Source-backed |
| `openai_pan2.yaml` | PAN 2 | `t2v` (a reference switches to image-to-video) | CUDA REST | `pan2/cuda-rest` | Source-backed; checkpoint not public, no recorded run |
| `openai_ltx2_distilled.yaml` | LTX2 Distilled | `t2v` | CUDA REST | `ltx2-distilled/cuda-rest` | Source-backed |
| `streaming_demo.yaml` | LTX2 Distilled | `t2v` | CUDA streaming | `ltx2-distilled/cuda-streaming` | Source-backed |
| `mlx_fasth3.yaml` | FastH3 V1 converted | `t2v` | MLX REST | none (legacy catalog only) | Source runtime; T2VA only |
docs/design/inference_schema_parity_inventory.yaml
+12
View File
@@ -295,6 +295,18 @@ surfaces:
- fastvideo.configs.pipelines.hunyuan15.Hunyuan15T2V720PConfig
- fastvideo.configs.pipelines.hyworld.HYWorldConfig
- fastvideo.configs.pipelines.hyworld.Hunyuan15T2V480PConfig
text_system_prompt:
sources:
- fastvideo.configs.pipelines.pan2.PAN2PipelineConfig
text_encoder_skip_layers:
sources:
- fastvideo.configs.pipelines.pan2.PAN2PipelineConfig
text_max_sequence_length:
sources:
- fastvideo.configs.pipelines.pan2.PAN2PipelineConfig
i2v_skip_first_latent_cfg:
sources:
- fastvideo.configs.pipelines.pan2.PAN2PipelineConfig
text_encoder_max_lengths:
sources:
- fastvideo.configs.pipelines.hunyuan15.Hunyuan15I2V480PStepDistilledConfig
docs/generate_examples.py
+1
View File
@@ -35,6 +35,7 @@ COOKBOOK_FAMILIES = {
"cosmos",
"kandinsky5",
"kandinsky6",
"pan2",
"flux",
"qwen_image21",
"sd35",
docs/inference/pan2.md
+86
View File
@@ -0,0 +1,86 @@
# PAN 2 Text/Image to Video
PAN 2 is a 29B mixture-of-transformers video DiT with separate video and text streams, a Qwen3.5 text encoder and the
Wan2.2 VAE. One pipeline serves text-to-video and image-to-video: pass `image_path` to condition on a first frame and
leave it out for text only.
## Model
| Hub repo | Pipeline | Workloads | Example |
|---|---|---|---|
| `IFM/PAN2` | `PAN2ModularPipeline` | T2V, I2V | [`basic_pan2.py`](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_pan2.py) |
PAN2 checkpoints use the Modular Diffusers layout (`modular_model_index.json`). Any repo id or local directory whose
name contains `pan2` (or whose manifest declares `PAN2ModularPipeline`) resolves to this pipeline. The defaults come
from the `pan2_t2v` and `pan2_i2v` presets:
| Setting | Default |
|---|---|
| Resolution | 704x1248 (height and width must be multiples of 32) |
| Frames | 313, 13 s at 24 fps (`num_frames - 1` must be a multiple of 4) |
| Steps | 50, uniform sigmas from 1 to 0 with the checkpoint's flow shift (9 for `IFM/PAN2-TEST`, the reference sampler's value) |
| Guidance | 3.0, classifier-free guidance. Image-to-video keeps the first latent frame out of guidance (`i2v_skip_first_latent_cfg`, on by default, as in the reference sampler) |
| Negative prompt | None unless the request carries one. The reference sampler always sends the release's structured negative caption (`negative_prompt.json` of the PAN2 repository) with its three `time_range_sec` values set to the clip length, `0 to 20` for 481 frames at 24 fps |
PAN 2 is trained on the structured captions its prompt rewriter produces, serialized as one-line JSON. Short raw
prompts work, with weaker results.
## Usage
```bash
python examples/inference/basic/basic_pan2.py
```
```python
from fastvideo import VideoGenerator
generator = VideoGenerator.from_pretrained(
"IFM/PAN2",
num_gpus=1,
dit_cpu_offload=False,
dit_layerwise_offload=False,
text_encoder_cpu_offload=True,
)
generator.generate_video(
"A red fox trots through fresh snow in a pine forest at dawn.",
image_path=None, # or the path of a first frame for image-to-video
output_path="video_samples_pan2",
)
```
For image-to-video, the first frame is scaled to cover the requested resolution and center cropped, then encoded by the
VAE into the first latent frame of the conditioning input.
## Parallelism
With `num_gpus > 1`, FastVideo runs Ulysses sequence parallelism: the video tokens are sharded across ranks and the
text tokens are replicated. The number of GPUs must divide the 32 attention heads. Sequence-parallel padding of the
video tokens is trimmed before attention, so no attention mask is needed.
Padding is always excluded, without an opt-in setting. At 704x1248 with 313 frames, the 67,782 video tokens need
two padding tokens at SP=4. They are removed before joining the text tokens in every attention block. This provides
the same exclusion as vLLM-Omni's unconditional padding mask in
[revision `e0c4c731`](https://github.com/vllm-project/vllm-omni/commit/e0c4c73158a5e4f44eda8eaa8879f7fd9cedbd74).
The trimmed sequence can use either FlashAttention or Torch SDPA.
```bash
PAN2_NUM_GPUS=4 python examples/inference/basic/basic_pan2.py
```
## Memory and offloading
The example keeps the DiT on the GPUs and offloads the text encoder between requests. Sequence parallelism shards
video tokens; each rank still holds the DiT weights. Increasing `num_gpus` alone does not shard those weights.
For devices that cannot hold the DiT, see [CPU offloading](offloading.md) for `dit_cpu_offload` and
`dit_layerwise_offload`. Hardware recipes and timings reported by vLLM-Omni use its own offload implementation;
they have not been validated for FastVideo.
The full `IFM/PAN2` checkpoint is not public yet. Local parity tests use the public random-weight
`wuqing157/tiny-pan2-modular-pipe` checkpoint; it verifies the integration and produces noise rather than useful video.
## Differences from the vLLM-Omni integration
- The optional prompt and video guardrails (`pan2-guardrail`) are not included.
- The Cache-DiT `quality` policy and CFG-parallel execution are vLLM-Omni serving features and are not ported.
- The text encoder is a FastVideo-native Qwen3.5 port. Its Gated DeltaNet layers use a PyTorch chunked delta rule
instead of the `fla` kernels.
docs/inference/support_matrix.md
+1
View File
@@ -70,6 +70,7 @@ column links a runnable script in `examples/inference/basic/` where one exists.
| minimax_h3 | `MiniMaxAI/MiniMax-H3` | T2V, I2V | [T2VA](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_minimax_h3_t2v.py)<br />[FL2VA](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_minimax_h3_fl2va.py)<br />[Ref2VA](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_minimax_h3_ref2va.py) |
| pika_h3_flex | `Pika-Labs/Pika-H3-Flex` | T2V, I2V | [T2VA/I2VA/L2VA/FL2VA/Ref2VA](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_pika_h3_flex.py) |
| open_h3_regenerate_2k | `Efficient-Large-Model/Open-H3-Regenerate-2K` | — | [basic_open_h3_regenerate_2k.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_open_h3_regenerate_2k.py) |
| pan2 | `IFM/PAN2` | T2V, I2V | [basic_pan2.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_pan2.py) |
| sd35 | `stabilityai/stable-diffusion-3.5-medium` | T2I | [basic_sd35_t2i.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_sd35_t2i.py) |
| qwen_image21 | `Qwen/Qwen-Image-2.1` | T2I, I2I | [basic_qwen_image21.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_qwen_image21.py) |
| stable_audio | `FastVideo/stable-audio-open-1.0-Diffusers` | T2V | [basic_stable_audio.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_stable_audio.py) |