SolitaryThinker
26719dcfe7
Port of vllm-project/vllm-omni#8602. PAN2 is a 29B mixture-of-transformers video DiT (separate video and text streams joined in one attention per block) with a Qwen3.5 text encoder and the Wan2.2 VAE. One pipeline serves text-to-video and first-frame image-to-video. - Native PAN2 DiT; per-stream Q/K/V fused at load time, Ulysses SP with replicated text tokens through DistributedAttention. - Native Qwen3.5 text encoder (Gated DeltaNet + gated attention) with a PyTorch chunked delta rule. - Reuses the Wan2.2 VAE and the flow-match Euler scheduler. - PAN2 stages: chat-template prompt crop at hidden_states[-3], first-frame conditioning, uniform sigmas with shift 7, fp32 latents. Adds a DenoisingStage.prepare_scheduler_model_output hook for the last one. - Registry entry for the Modular Diffusers PAN2ModularPipeline manifest, pan2_t2v / pan2_i2v presets, example, docs, and local parity tests against a diffusers/transformers reference on the public tiny checkpoint.
41 lines
1.4 KiB
Python
41 lines
1.4 KiB
Python
import os
|
|
|
|
from fastvideo import VideoGenerator
|
|
|
|
# PAN2 is trained on the structured captions its prompt rewriter produces (serialized as one-line JSON). Plain
|
|
# prompts like this one work, with weaker results.
|
|
PROMPT = ("A red fox trots through fresh snow in a pine forest at dawn, its breath visible in the cold air. "
|
|
"The camera tracks it from the side at ground level as low golden light filters through the trees.")
|
|
|
|
OUTPUT_PATH = "video_samples_pan2"
|
|
MODEL_PATH = os.environ.get("PAN2_MODEL_PATH", "IFM/PAN2")
|
|
|
|
# Set to a first frame (e.g. "assets/images/girl.png") for image-to-video; the same pipeline runs both modes.
|
|
IMAGE_PATH = None
|
|
|
|
|
|
def main():
|
|
# The 29B DiT runs in bf16; with more GPUs, num_gpus enables Ulysses sequence parallelism (the 32 attention
|
|
# heads must be divisible by it).
|
|
generator = VideoGenerator.from_pretrained(
|
|
MODEL_PATH,
|
|
num_gpus=int(os.environ.get("PAN2_NUM_GPUS", "1")),
|
|
use_fsdp_inference=False,
|
|
dit_cpu_offload=False,
|
|
vae_cpu_offload=False,
|
|
text_encoder_cpu_offload=True,
|
|
pin_cpu_memory=True,
|
|
)
|
|
|
|
# Defaults come from the pan2 preset: 704x1248, 313 frames (13 s at 24 fps), 50 steps, guidance 3.0.
|
|
generator.generate_video(
|
|
PROMPT,
|
|
image_path=IMAGE_PATH,
|
|
output_path=OUTPUT_PATH,
|
|
save_video=True,
|
|
)
|
|
|
|
|
|
if __name__ == "__main__":
|
|
main()
|