nodes/hao-ai-lab-FastVideo
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

[misc]: remove DreamX World, HunyuanGameCraft, GLM-Image, LongCat, TurboDiffusion, and Z-Image

Browse Source 
These ports were marked pending removal in #1820. Each has a newer,
maintained counterpart in the repo, and their model-specific hooks
leaked into shared stages, request fields, CLI args, and loaders.

Remove each model's code, configs, presets, examples, converters,
tests, cookbook pages, CI planner rules, and registry entries, along
with the shared-code hooks only they used:

- camera kwargs (GameCraft camera_states, DreamX y_camera) in the
  shared denoising loop
- LongCat refine branches in latent preparation, the decoding crop,
  the --bsa-* CLI args, its train model, and third_party/longcat_video
- Z-Image cfg_normalization/cfg_truncation and GameCraft/DreamX
  camera/action fields in SamplingParam, ForwardBatch, and the schema
- the GLM-Image vision-language loader and the TurboDiffusion rCM
  scheduler and SLA auto-select in the train entrypoint
- the Wan definition groups that only preserved DreamX detector order

Removed env vars and their legacy aliases move to DEPRECATED_VARIABLES
per the env-var policy. Matrix Game 2.0 stays: the new trainer's world
model scenarios depend on it. The SLA and BSA attention backends stay.
The TurboDiffusion int8 kernels in fastvideo-kernel are left for a
separate kernel PR.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
SolitaryThinker
2026-10-06 22:40:54 +00:00
co-authored by Claude Opus 5.5
parent 02a027c49f
commit d692d1cc3b
236 changed files with 243 additions and 29701 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/checkpoint_conversion/AGENTS.md
-8
View File
@@ -9,19 +9,11 @@
```
checkpoint_conversion/
├── convert_gamecraft_full.py # Combined DiT + VAE
├── convert_gamecraft_vae.py # VAE only
├── convert_gamecraft_weights.py # DiT only
├── convert_gen3c_to_fastvideo.py
├── convert_ltx2_weights.py
├── convert_minimax_h3_adaln_rank.py # Rank-reduces AdaLN in place; same family
├── convert_turbodiffusion_to_diffusers.py
├── convert_turbodiffusion_i2v_to_diffusers.py
├── extract_llava_text_encoder.py # Encoder extraction from a multimodal repo
├── longcat_to_fastvideo.py
├── stable_audio_to_diffusers.py
├── wan_to_diffusers.py
├── validate_longcat_weights.py # Post-conversion validation
├── pt_to_safetensors.py # Generic format flip
└── create_hf_repo.py # Push to HF after conversion
```
scripts/checkpoint_conversion/convert_gamecraft_full.py
-239
View File
@@ -1,239 +0,0 @@
# SPDX-License-Identifier: Apache-2.0
"""
Convert the full Hunyuan-GameCraft-1.0 model to FastVideo diffusers format.
This script converts:
1. Transformer (DiT) weights from DeepSpeed format
2. VAE weights
3. Text encoders (LLaVA-LLaMA-3-8B and CLIP)
4. Creates model_index.json and scheduler config
Usage:
python scripts/checkpoint_conversion/convert_gamecraft_full.py \
--input-dir Hunyuan-GameCraft-1.0/weights \
--output-dir official_weights/hunyuan-gamecraft-diffusers
# Or specify individual paths:
python scripts/checkpoint_conversion/convert_gamecraft_full.py \
--transformer Hunyuan-GameCraft-1.0/weights/gamecraft_models/mp_rank_00_model_states.pt \
--vae Hunyuan-GameCraft-1.0/weights/stdmodels/vae_3d/hyvae/checkpoint-step-270000.ckpt \
--text-encoder Hunyuan-GameCraft-1.0/weights/stdmodels/llava-llama-3-8b-v1_1-transformers \
--text-encoder-2 Hunyuan-GameCraft-1.0/weights/stdmodels/openai_clip-vit-large-patch14 \
--output-dir official_weights/hunyuan-gamecraft-diffusers
"""
from __future__ import annotations
import argparse
import json
import shutil
from pathlib import Path
import torch
from safetensors.torch import save_file
# Import conversion functions
from convert_gamecraft_weights import convert_weights as convert_transformer
from convert_gamecraft_vae import convert_gamecraft_vae
def create_model_index(output_dir: Path) -> None:
"""Create model_index.json for the pipeline."""
model_index = {
"_class_name": "HunyuanGameCraftPipeline",
"_diffusers_version": "0.30.0",
"scheduler": ["diffusers", "FlowMatchEulerDiscreteScheduler"],
"text_encoder": ["transformers", "LlamaModel"],
"text_encoder_2": ["transformers", "CLIPTextModel"],
"tokenizer": ["transformers", "AutoTokenizer"],
"tokenizer_2": ["transformers", "CLIPTokenizer"],
"transformer": ["fastvideo", "HunyuanGameCraftTransformer3DModel"],
"vae": ["fastvideo", "AutoencoderKLCausal3D"]
}
with open(output_dir / "model_index.json", "w") as f:
json.dump(model_index, f, indent=2)
print(f"Created model_index.json")
def create_scheduler_config(output_dir: Path) -> None:
"""Create scheduler config for FlowMatchEulerDiscreteScheduler."""
scheduler_dir = output_dir / "scheduler"
scheduler_dir.mkdir(parents=True, exist_ok=True)
scheduler_config = {
"_class_name": "FlowMatchEulerDiscreteScheduler",
"_diffusers_version": "0.30.0",
"base_image_seq_len": 256,
"base_shift": 0.5,
"invert_sigmas": False,
"max_shift": 1.15,
"num_train_timesteps": 1000,
"shift": 7.0,
"use_dynamic_shifting": True
}
with open(scheduler_dir / "scheduler_config.json", "w") as f:
json.dump(scheduler_config, f, indent=2)
print(f"Created scheduler config")
def copy_text_encoder(src_dir: Path, dst_dir: Path, name: str) -> None:
"""Copy text encoder files to output directory."""
if not src_dir.exists():
print(f"Warning: {name} not found at {src_dir}")
return
dst_dir.mkdir(parents=True, exist_ok=True)
# Copy all files
for src_file in src_dir.iterdir():
if src_file.is_file():
shutil.copy2(src_file, dst_dir / src_file.name)
print(f"Copied {name} to {dst_dir}")
def copy_tokenizer(src_dir: Path, dst_dir: Path, name: str) -> None:
"""Copy tokenizer files to output directory."""
if not src_dir.exists():
print(f"Warning: {name} not found at {src_dir}")
return
dst_dir.mkdir(parents=True, exist_ok=True)
# Copy tokenizer-related files
tokenizer_files = [
"tokenizer.json",
"tokenizer_config.json",
"special_tokens_map.json",
"vocab.json",
"merges.txt",
"added_tokens.json",
]
for filename in tokenizer_files:
src_file = src_dir / filename
if src_file.exists():
shutil.copy2(src_file, dst_dir / filename)
print(f"Copied {name} to {dst_dir}")
def convert_full_model(
transformer_path: Path | None = None,
vae_path: Path | None = None,
text_encoder_path: Path | None = None,
text_encoder_2_path: Path | None = None,
input_dir: Path | None = None,
output_dir: Path = Path("official_weights/hunyuan-gamecraft-diffusers"),
verbose: bool = False,
) -> None:
"""Convert all components of GameCraft to diffusers format."""
# Resolve paths from input_dir if individual paths not provided
if input_dir is not None:
if transformer_path is None:
transformer_path = input_dir / "gamecraft_models" / "mp_rank_00_model_states.pt"
if vae_path is None:
vae_path = input_dir / "stdmodels" / "vae_3d" / "hyvae" / "checkpoint-step-270000.ckpt"
if not vae_path.exists():
vae_path = input_dir / "stdmodels" / "vae_3d" / "hyvae" / "pytorch_model.pt"
if text_encoder_path is None:
text_encoder_path = input_dir / "stdmodels" / "llava-llama-3-8b-v1_1-transformers"
if text_encoder_2_path is None:
text_encoder_2_path = input_dir / "stdmodels" / "openai_clip-vit-large-patch14"
output_dir.mkdir(parents=True, exist_ok=True)
print(f"Converting GameCraft to diffusers format at {output_dir}")
print("=" * 60)
# 1. Convert transformer
if transformer_path and transformer_path.exists():
print(f"\n[1/5] Converting transformer from {transformer_path}")
convert_transformer(
input_path=transformer_path,
output_dir=output_dir,
save_config=True,
verbose=verbose,
)
else:
print(f"\n[1/5] Skipping transformer (not found: {transformer_path})")
# 2. Convert VAE
if vae_path and vae_path.exists():
print(f"\n[2/5] Converting VAE from {vae_path}")
convert_gamecraft_vae(
input_path=vae_path,
output_dir=output_dir / "vae",
copy_config=True,
)
else:
print(f"\n[2/5] Skipping VAE (not found: {vae_path})")
# 3. Copy text encoder (LLaMA)
if text_encoder_path and text_encoder_path.exists():
print(f"\n[3/5] Copying text encoder from {text_encoder_path}")
copy_text_encoder(text_encoder_path, output_dir / "text_encoder", "text_encoder")
copy_tokenizer(text_encoder_path, output_dir / "tokenizer", "tokenizer")
else:
print(f"\n[3/5] Skipping text_encoder (not found: {text_encoder_path})")
# 4. Copy text encoder 2 (CLIP)
if text_encoder_2_path and text_encoder_2_path.exists():
print(f"\n[4/5] Copying text encoder 2 from {text_encoder_2_path}")
copy_text_encoder(text_encoder_2_path, output_dir / "text_encoder_2", "text_encoder_2")
copy_tokenizer(text_encoder_2_path, output_dir / "tokenizer_2", "tokenizer_2")
else:
print(f"\n[4/5] Skipping text_encoder_2 (not found: {text_encoder_2_path})")
# 5. Create model_index.json and scheduler
print(f"\n[5/5] Creating model_index.json and scheduler config")
create_model_index(output_dir)
create_scheduler_config(output_dir)
print("\n" + "=" * 60)
print(f"Conversion complete! Output: {output_dir}")
print("\nDirectory structure:")
for item in sorted(output_dir.rglob("*")):
if item.is_file():
rel_path = item.relative_to(output_dir)
size_mb = item.stat().st_size / (1024 * 1024)
print(f" {rel_path} ({size_mb:.1f} MB)")
def main():
parser = argparse.ArgumentParser(
description="Convert full Hunyuan-GameCraft-1.0 model to FastVideo diffusers format.")
parser.add_argument("--input-dir", type=str, default=None, help="Path to Hunyuan-GameCraft-1.0/weights directory")
parser.add_argument("--transformer",
type=str,
default=None,
help="Path to transformer checkpoint (mp_rank_00_model_states.pt)")
parser.add_argument("--vae", type=str, default=None, help="Path to VAE checkpoint")
parser.add_argument("--text-encoder", type=str, default=None, help="Path to text encoder (LLaVA-LLaMA) directory")
parser.add_argument("--text-encoder-2", type=str, default=None, help="Path to text encoder 2 (CLIP) directory")
parser.add_argument("--output-dir",
type=str,
default="official_weights/hunyuan-gamecraft-diffusers",
help="Output directory for converted model")
parser.add_argument("--verbose", "-v", action="store_true", help="Print detailed conversion info")
args = parser.parse_args()
# Require either input-dir or individual paths
if args.input_dir is None and args.transformer is None:
parser.error("Either --input-dir or individual component paths must be provided")
convert_full_model(
transformer_path=Path(args.transformer) if args.transformer else None,
vae_path=Path(args.vae) if args.vae else None,
text_encoder_path=Path(args.text_encoder) if args.text_encoder else None,
text_encoder_2_path=Path(args.text_encoder_2) if args.text_encoder_2 else None,
input_dir=Path(args.input_dir) if args.input_dir else None,
output_dir=Path(args.output_dir),
verbose=args.verbose,
)
if __name__ == "__main__":
main()
scripts/checkpoint_conversion/convert_gamecraft_vae.py
-93
View File
@@ -1,93 +0,0 @@
# SPDX-License-Identifier: Apache-2.0
"""
Convert GameCraft VAE weights to FastVideo safetensors format.
The official GameCraft VAE (checkpoint-step-270000.ckpt or pytorch_model.pt) uses
a "vae." prefix. This script extracts and converts for FastVideo GameCraftVAE.
Usage:
python scripts/checkpoint_conversion/convert_gamecraft_vae.py \
--input Hunyuan-GameCraft-1.0/weights/stdmodels/vae_3d/hyvae/checkpoint-step-270000.ckpt \
--output official_weights/hunyuan-gamecraft/vae
"""
from __future__ import annotations
import argparse
import json
import shutil
from pathlib import Path
import torch
from safetensors.torch import save_file
def convert_gamecraft_vae(
input_path: Path,
output_dir: Path,
copy_config: bool = True,
) -> dict:
"""Convert official GameCraft VAE checkpoint to FastVideo format."""
ckpt = torch.load(input_path, map_location="cpu", weights_only=True)
if "state_dict" in ckpt:
state_dict = ckpt["state_dict"]
else:
state_dict = ckpt
# Extract VAE weights and strip "vae." prefix
vae_sd = {k.replace("vae.", ""): v for k, v in state_dict.items() if k.startswith("vae.")}
print(f"Extracted {len(vae_sd)} VAE parameters")
output_dir.mkdir(parents=True, exist_ok=True)
save_file(vae_sd, output_dir / "diffusion_pytorch_model.safetensors")
print(f"Saved to {output_dir / 'diffusion_pytorch_model.safetensors'}")
if copy_config:
config_src = input_path.parent / "config.json"
if config_src.exists():
config_dst = output_dir / "config.json"
config = json.loads(config_src.read_text())
config["_class_name"] = "AutoencoderKLCausal3D"
with open(config_dst, "w") as f:
json.dump(config, f, indent=2)
print(f"Saved config to {config_dst}")
else:
print(f"Warning: config.json not found at {config_src}")
return {"total": len(vae_sd)}
def main():
parser = argparse.ArgumentParser(description="Convert GameCraft VAE weights to FastVideo safetensors format.")
parser.add_argument(
"--input",
type=str,
required=True,
help="Path to official VAE checkpoint (.ckpt or .pt)",
)
parser.add_argument(
"--output",
type=str,
default="official_weights/hunyuan-gamecraft/vae",
help="Output directory for converted weights",
)
parser.add_argument(
"--no-config",
action="store_true",
help="Don't copy config.json",
)
args = parser.parse_args()
input_path = Path(args.input)
if not input_path.exists():
raise FileNotFoundError(f"Input not found: {input_path}")
stats = convert_gamecraft_vae(
input_path=input_path,
output_dir=Path(args.output),
copy_config=not args.no_config,
)
print(f"\nConversion complete! Total parameters: {stats['total']}")
if __name__ == "__main__":
main()
scripts/checkpoint_conversion/convert_gamecraft_weights.py
-242
View File
@@ -1,242 +0,0 @@
# SPDX-License-Identifier: Apache-2.0
"""
Convert Hunyuan-GameCraft-1.0 weights from DeepSpeed format to FastVideo format.
The official GameCraft checkpoint (mp_rank_00_model_states.pt) uses slightly different
naming conventions than FastVideo. This script converts the weights to be compatible
with FastVideo's HunyuanGameCraft model.
Usage:
python scripts/checkpoint_conversion/convert_gamecraft_weights.py \
--input Hunyuan-GameCraft-1.0/weights/gamecraft_models/mp_rank_00_model_states.pt \
--output official_weights/hunyuan-gamecraft/transformer/diffusion_pytorch_model.safetensors
"""
from __future__ import annotations
import argparse
import json
import re
from collections import OrderedDict
from pathlib import Path
import torch
from safetensors.torch import save_file
# Mapping from official GameCraft naming to FastVideo naming
# GameCraft weights are already close to FastVideo format, but need minor adjustments
PARAM_NAME_MAP: dict[str, str] = {
# MLP naming: fc1 -> fc_in, fc2 -> fc_out
r"^(.*)\.img_mlp\.fc1\.(.*)$": r"\1.img_mlp.fc_in.\2",
r"^(.*)\.img_mlp\.fc2\.(.*)$": r"\1.img_mlp.fc_out.\2",
r"^(.*)\.txt_mlp\.fc1\.(.*)$": r"\1.txt_mlp.fc_in.\2",
r"^(.*)\.txt_mlp\.fc2\.(.*)$": r"\1.txt_mlp.fc_out.\2",
# Single block MLP naming
r"^single_blocks\.(\d+)\.mlp\.fc1\.(.*)$": r"single_blocks.\1.mlp.fc_in.\2",
r"^single_blocks\.(\d+)\.mlp\.fc2\.(.*)$": r"single_blocks.\1.mlp.fc_out.\2",
# Token refiner naming - rename individual_token_refiner.blocks to refiner_blocks
# This is applied first, then subsequent mappings handle the sub-components
r"^txt_in\.individual_token_refiner\.blocks\.(\d+)\.(.*)$": r"txt_in.refiner_blocks.\1.\2",
# Vector in naming
r"^vector_in\.in_layer\.(.*)$": r"vector_in.fc_in.\1",
r"^vector_in\.out_layer\.(.*)$": r"vector_in.fc_out.\1",
# Time embedder naming
r"^time_in\.mlp\.0\.(.*)$": r"time_in.mlp.fc_in.\1",
r"^time_in\.mlp\.2\.(.*)$": r"time_in.mlp.fc_out.\1",
# Guidance embedder naming (if present)
r"^guidance_in\.mlp\.0\.(.*)$": r"guidance_in.mlp.fc_in.\1",
r"^guidance_in\.mlp\.2\.(.*)$": r"guidance_in.mlp.fc_out.\1",
# Final layer adaLN modulation
r"^final_layer\.adaLN_modulation\.1\.(.*)$": r"final_layer.adaLN_modulation.linear.\1",
# Text in c_embedder (context embedder) naming
r"^txt_in\.c_embedder\.linear_1\.(.*)$": r"txt_in.c_embedder.fc_in.\1",
r"^txt_in\.c_embedder\.linear_2\.(.*)$": r"txt_in.c_embedder.fc_out.\1",
# Text in t_embedder (time embedder) naming
r"^txt_in\.t_embedder\.mlp\.0\.(.*)$": r"txt_in.t_embedder.mlp.fc_in.\1",
r"^txt_in\.t_embedder\.mlp\.2\.(.*)$": r"txt_in.t_embedder.mlp.fc_out.\1",
# Refiner block MLP naming - need to match the actual checkpoint names
r"^txt_in\.refiner_blocks\.(\d+)\.mlp\.fc1\.(.*)$": r"txt_in.refiner_blocks.\1.mlp.fc_in.\2",
r"^txt_in\.refiner_blocks\.(\d+)\.mlp\.fc2\.(.*)$": r"txt_in.refiner_blocks.\1.mlp.fc_out.\2",
# Refiner block adaLN modulation naming - matches .adaLN_modulation.1.weight/bias
r"^txt_in\.refiner_blocks\.(\d+)\.adaLN_modulation\.1\.(.*)$":
r"txt_in.refiner_blocks.\1.adaLN_modulation.linear.\2",
# Camera net - keep as-is (already correct)
# r"^camera_net\.(.*)$": r"camera_net.\1",
}
def apply_mapping(key: str) -> str:
"""Apply name mapping to convert official key to FastVideo key.
Mappings are applied repeatedly until no more changes occur.
This allows for chained transformations.
"""
prev_key = None
while prev_key != key:
prev_key = key
for pattern, replacement in PARAM_NAME_MAP.items():
if re.match(pattern, key):
key = re.sub(pattern, replacement, key)
break # Apply only one mapping per iteration
return key
def load_deepspeed_checkpoint(path: Path) -> dict[str, torch.Tensor]:
"""Load weights from a DeepSpeed checkpoint file."""
checkpoint = torch.load(path, map_location="cpu")
if "module" in checkpoint:
state_dict = checkpoint["module"]
elif "state_dict" in checkpoint:
state_dict = checkpoint["state_dict"]
elif "model" in checkpoint:
state_dict = checkpoint["model"]
else:
state_dict = checkpoint
if not isinstance(state_dict, dict):
raise TypeError(f"Expected dict state_dict, got {type(state_dict)}")
return state_dict
def convert_weights(
input_path: Path,
output_dir: Path,
save_config: bool = True,
verbose: bool = False,
) -> dict[str, int]:
"""
Convert GameCraft weights to FastVideo format.
Returns:
dict with conversion statistics
"""
print(f"Loading checkpoint from {input_path}")
state_dict = load_deepspeed_checkpoint(input_path)
print(f"Found {len(state_dict)} parameters")
# Convert weights
converted_state_dict: OrderedDict[str, torch.Tensor] = OrderedDict()
renamed_count = 0
unchanged_count = 0
for key, value in state_dict.items():
new_key = apply_mapping(key)
if new_key != key:
if verbose:
print(f" {key} -> {new_key}")
renamed_count += 1
else:
unchanged_count += 1
converted_state_dict[new_key] = value
print(f"Renamed {renamed_count} parameters, {unchanged_count} unchanged")
# Create output directory structure
transformer_dir = output_dir / "transformer"
transformer_dir.mkdir(parents=True, exist_ok=True)
# Save as safetensors
output_path = transformer_dir / "diffusion_pytorch_model.safetensors"
print(f"Saving converted weights to {output_path}")
save_file(converted_state_dict, str(output_path))
# Save config if requested
if save_config:
config = {
"_class_name": "HunyuanGameCraftTransformer3DModel",
"_fastvideo_version": "0.1.0",
"in_channels": 33, # 16 + 16 + 1 for concat mode
"out_channels": 16,
"hidden_size": 3072,
"num_attention_heads": 24,
"attention_head_dim": 128,
"num_layers": 20,
"num_single_layers": 40,
"num_refiner_layers": 2,
"patch_size": [1, 2, 2],
"rope_axes_dim": [16, 56, 56],
"text_embed_dim": 4096,
"pooled_projection_dim": 768,
"mlp_ratio": 4.0,
"guidance_embeds": False,
"camera_net": True,
}
config_path = transformer_dir / "config.json"
with open(config_path, "w") as f:
json.dump(config, f, indent=2)
print(f"Saved config to {config_path}")
# Verify camera_net weights
camera_keys = [k for k in converted_state_dict.keys() if "camera_net" in k]
print(f"\nCamera net weights ({len(camera_keys)}):")
for k in camera_keys:
print(f" {k}: {converted_state_dict[k].shape}")
# Verify img_in weights (should be 33 channels)
if "img_in.proj.weight" in converted_state_dict:
img_in_shape = converted_state_dict["img_in.proj.weight"].shape
print(f"\nimg_in.proj.weight shape: {img_in_shape}")
expected_in_channels = 33
actual_in_channels = img_in_shape[1]
if actual_in_channels != expected_in_channels:
print(f" WARNING: Expected {expected_in_channels} input channels, got {actual_in_channels}")
else:
print(f" OK: 33 input channels (16 latent + 16 gt_latent + 1 mask)")
return {
"total": len(converted_state_dict),
"renamed": renamed_count,
"unchanged": unchanged_count,
}
def main():
parser = argparse.ArgumentParser(description="Convert Hunyuan-GameCraft-1.0 weights to FastVideo format.")
parser.add_argument("--input",
type=str,
required=True,
help="Path to official GameCraft checkpoint (mp_rank_00_model_states.pt)")
parser.add_argument("--output",
type=str,
default="official_weights/hunyuan-gamecraft",
help="Output directory for converted weights")
parser.add_argument("--no-config", action="store_true", help="Don't save config.json")
parser.add_argument("--verbose", "-v", action="store_true", help="Print all renamed parameters")
args = parser.parse_args()
input_path = Path(args.input)
output_dir = Path(args.output)
if not input_path.exists():
raise FileNotFoundError(f"Input file not found: {input_path}")
stats = convert_weights(
input_path=input_path,
output_dir=output_dir,
save_config=not args.no_config,
verbose=args.verbose,
)
print(f"\nConversion complete!")
print(f" Total parameters: {stats['total']}")
print(f" Renamed: {stats['renamed']}")
print(f" Unchanged: {stats['unchanged']}")
if __name__ == "__main__":
main()
scripts/checkpoint_conversion/convert_turbodiffusion_i2v_to_diffusers.py
-197
View File
@@ -1,197 +0,0 @@
#!/usr/bin/env python3
"""
Convert TurboDiffusion I2V .pth checkpoint to Diffusers safetensors format.
TurboDiffusion I2V uses two models: high-noise and low-noise.
This script converts both checkpoints to Diffusers format.
Usage:
python convert_turbodiffusion_i2v_to_diffusers.py \
--high_noise_path /path/to/TurboWan2.2-I2V-A14B-high-720P.pth \
--low_noise_path /path/to/TurboWan2.2-I2V-A14B-low-720P.pth \
--output_dir /path/to/output \
--reference_repo Wan-AI/Wan2.1-I2V-14B-720P-Diffusers
"""
import argparse
import os
import re
import json
import torch
import shutil
import glob
from safetensors import safe_open
from safetensors.torch import save_file
from huggingface_hub import snapshot_download
# Weight mapping from TurboDiffusion -> Diffusers/FastVideo format
# Same as T2V but may need additional I2V-specific mappings
TURBODIFFUSION_WEIGHT_MAPPING = {
# Self attention
r"^blocks\.(\d+)\.self_attn\.q\.(.*)$": r"blocks.\1.to_q.\2",
r"^blocks\.(\d+)\.self_attn\.k\.(.*)$": r"blocks.\1.to_k.\2",
r"^blocks\.(\d+)\.self_attn\.v\.(.*)$": r"blocks.\1.to_v.\2",
r"^blocks\.(\d+)\.self_attn\.o\.(.*)$": r"blocks.\1.to_out.\2",
r"^blocks\.(\d+)\.self_attn\.norm_q\.(.*)$": r"blocks.\1.norm_q.\2",
r"^blocks\.(\d+)\.self_attn\.norm_k\.(.*)$": r"blocks.\1.norm_k.\2",
# Cross attention
r"^blocks\.(\d+)\.cross_attn\.q\.(.*)$": r"blocks.\1.attn2.to_q.\2",
r"^blocks\.(\d+)\.cross_attn\.k\.(.*)$": r"blocks.\1.attn2.to_k.\2",
r"^blocks\.(\d+)\.cross_attn\.v\.(.*)$": r"blocks.\1.attn2.to_v.\2",
r"^blocks\.(\d+)\.cross_attn\.o\.(.*)$": r"blocks.\1.attn2.to_out.\2",
r"^blocks\.(\d+)\.cross_attn\.norm_q\.(.*)$": r"blocks.\1.attn2.norm_q.\2",
r"^blocks\.(\d+)\.cross_attn\.norm_k\.(.*)$": r"blocks.\1.attn2.norm_k.\2",
# I2V-specific cross attention (add_k/v_proj for image context)
r"^blocks\.(\d+)\.cross_attn\.add_k\.(.*)$": r"blocks.\1.attn2.add_k_proj.\2",
r"^blocks\.(\d+)\.cross_attn\.add_v\.(.*)$": r"blocks.\1.attn2.add_v_proj.\2",
r"^blocks\.(\d+)\.cross_attn\.norm_add_k\.(.*)$": r"blocks.\1.attn2.norm_added_k.\2",
r"^blocks\.(\d+)\.cross_attn\.norm_add_q\.(.*)$": r"blocks.\1.attn2.norm_added_q.\2",
# Norms and FFN
r"^blocks\.(\d+)\.norm1\.(.*)$": r"blocks.\1.norm1.\2",
r"^blocks\.(\d+)\.norm3\.(.*)$": r"blocks.\1.self_attn_residual_norm.norm.\2",
r"^blocks\.(\d+)\.norm2\.(.*)$": r"blocks.\1.norm3.\2",
r"^blocks\.(\d+)\.ffn\.0\.(.*)$": r"blocks.\1.ffn.fc_in.\2",
r"^blocks\.(\d+)\.ffn\.2\.(.*)$": r"blocks.\1.ffn.fc_out.\2",
r"^blocks\.(\d+)\.modulation$": r"blocks.\1.scale_shift_table",
# Embeddings
r"^text_embedding\.0\.(.*)$": r"condition_embedder.text_embedder.fc_in.\1",
r"^text_embedding\.2\.(.*)$": r"condition_embedder.text_embedder.fc_out.\1",
r"^time_embedding\.0\.(.*)$": r"condition_embedder.time_embedder.mlp.fc_in.\1",
r"^time_embedding\.2\.(.*)$": r"condition_embedder.time_embedder.mlp.fc_out.\1",
r"^time_projection\.1\.(.*)$": r"condition_embedder.time_modulation.linear.\1",
# Head
r"^head\.head\.(.*)$": r"proj_out.\1",
r"^head\.norm\.(.*)$": r"norm_out.\1",
r"^head\.modulation$": r"scale_shift_table",
# SLA proj_l weights
r"^blocks\.(\d+)\.self_attn\.attn_op\.local_attn\.proj_l\.(.*)$": r"blocks.\1.attn1.attn_impl.proj_l.\2",
}
SKIP_PATTERNS = []
def should_skip_key(key: str) -> bool:
for pattern in SKIP_PATTERNS:
if re.match(pattern, key):
return True
return False
def convert_key(turbo_key: str) -> str:
for pattern, replacement in TURBODIFFUSION_WEIGHT_MAPPING.items():
if re.match(pattern, turbo_key):
return re.sub(pattern, replacement, turbo_key)
return turbo_key
def reshape_patch_embedding(tensor: torch.Tensor, target_shape: tuple) -> torch.Tensor:
if len(tensor.shape) == 2 and len(target_shape) == 5:
return tensor.view(target_shape)
return tensor
def get_reference_shapes(reference_repo: str) -> dict:
print(f"Downloading reference model shapes from {reference_repo}...")
local_dir = snapshot_download(
repo_id=reference_repo,
allow_patterns=["transformer/config.json", "transformer/diffusion_pytorch_model*.safetensors"],
local_dir_use_symlinks=False)
weight_files = glob.glob(os.path.join(local_dir, "transformer", "*.safetensors"))
shapes = {}
for wf in weight_files:
with safe_open(wf, framework="pt") as f:
for key in f.keys():
shapes[key] = f.get_tensor(key).shape
return shapes, local_dir
def convert_checkpoint(input_path: str, output_dir: str, ref_shapes: dict, model_name: str) -> None:
"""Convert a single TurboDiffusion checkpoint to Diffusers format."""
print(f"\n{'='*60}")
print(f"Converting {model_name}: {input_path}")
print(f"{'='*60}")
turbo_state_dict = torch.load(input_path, map_location="cpu", weights_only=True)
print(f"Loaded {len(turbo_state_dict)} keys")
converted_state_dict = {}
skipped_keys = []
for turbo_key, tensor in turbo_state_dict.items():
if should_skip_key(turbo_key):
skipped_keys.append(turbo_key)
continue
new_key = convert_key(turbo_key)
if "patch_embedding" in new_key and new_key in ref_shapes:
target_shape = ref_shapes[new_key]
if tensor.shape != target_shape:
print(f"Reshaping {new_key}: {tensor.shape} -> {target_shape}")
tensor = reshape_patch_embedding(tensor, target_shape)
if new_key in ref_shapes:
if tensor.shape != ref_shapes[new_key]:
print(f"WARNING: Shape mismatch for {new_key}: got {tensor.shape}, expected {ref_shapes[new_key]}")
converted_state_dict[new_key] = tensor
print(f"Converted: {len(converted_state_dict)} keys, Skipped: {len(skipped_keys)} keys")
os.makedirs(output_dir, exist_ok=True)
output_path = os.path.join(output_dir, "diffusion_pytorch_model.safetensors")
print(f"Saving to {output_path}...")
save_file(converted_state_dict, output_path)
return converted_state_dict
def main():
parser = argparse.ArgumentParser(description="Convert TurboDiffusion I2V checkpoints to Diffusers format")
parser.add_argument("--high_noise_path",
type=str,
required=True,
help="Path to high-noise TurboDiffusion .pth checkpoint")
parser.add_argument("--low_noise_path",
type=str,
required=True,
help="Path to low-noise TurboDiffusion .pth checkpoint")
parser.add_argument("--output_dir", type=str, required=True, help="Output directory for converted safetensors")
parser.add_argument("--reference_repo",
type=str,
default="Wan-AI/Wan2.2-I2V-A14B-Diffusers",
help="Reference HF repo to get expected tensor shapes")
args = parser.parse_args()
# Get reference shapes
ref_shapes, ref_local_dir = get_reference_shapes(args.reference_repo)
print(f"Got {len(ref_shapes)} reference shapes")
# Convert high-noise model
high_noise_output = os.path.join(args.output_dir, "transformer_high")
convert_checkpoint(args.high_noise_path, high_noise_output, ref_shapes, "high-noise")
# Convert low-noise model
low_noise_output = os.path.join(args.output_dir, "transformer_low")
convert_checkpoint(args.low_noise_path, low_noise_output, ref_shapes, "low-noise")
# Copy config.json to both
src_config = os.path.join(ref_local_dir, "transformer", "config.json")
shutil.copy(src_config, os.path.join(high_noise_output, "config.json"))
shutil.copy(src_config, os.path.join(low_noise_output, "config.json"))
print("Copied config.json to both transformer directories")
print(f"\n{'='*60}")
print(f"Conversion complete!")
print(f"High-noise model: {high_noise_output}")
print(f"Low-noise model: {low_noise_output}")
print(f"{'='*60}")
if __name__ == "__main__":
main()
scripts/checkpoint_conversion/convert_turbodiffusion_to_diffusers.py
-204
View File
@@ -1,204 +0,0 @@
#!/usr/bin/env python3
"""
Convert TurboDiffusion .pth checkpoint to Diffusers safetensors format.
This script:
1. Loads the TurboDiffusion .pth checkpoint
2. Applies weight key renaming to match FastVideo/Diffusers format
3. Reshapes patch_embedding from [D, C*P] to [D, C, P_t, P_h, P_w]
4. Saves as sharded safetensors files compatible with diffusers
Usage:
python convert_turbodiffusion_to_diffusers.py \
--input_path /path/to/TurboWan2.1-T2V-1.3B-480P.pth \
--output_dir /path/to/output/transformer \
--reference_repo Wan-AI/Wan2.1-T2V-1.3B-Diffusers
"""
import argparse
import os
import re
import json
import torch
import shutil
import glob
from safetensors import safe_open
from safetensors.torch import save_file
from huggingface_hub import snapshot_download
# Weight mapping from TurboDiffusion -> Diffusers/FastVideo format
TURBODIFFUSION_WEIGHT_MAPPING = {
# Self attention
r"^blocks\.(\d+)\.self_attn\.q\.(.*)$": r"blocks.\1.to_q.\2",
r"^blocks\.(\d+)\.self_attn\.k\.(.*)$": r"blocks.\1.to_k.\2",
r"^blocks\.(\d+)\.self_attn\.v\.(.*)$": r"blocks.\1.to_v.\2",
r"^blocks\.(\d+)\.self_attn\.o\.(.*)$": r"blocks.\1.to_out.\2",
r"^blocks\.(\d+)\.self_attn\.norm_q\.(.*)$": r"blocks.\1.norm_q.\2",
r"^blocks\.(\d+)\.self_attn\.norm_k\.(.*)$": r"blocks.\1.norm_k.\2",
# Cross attention
r"^blocks\.(\d+)\.cross_attn\.q\.(.*)$": r"blocks.\1.attn2.to_q.\2",
r"^blocks\.(\d+)\.cross_attn\.k\.(.*)$": r"blocks.\1.attn2.to_k.\2",
r"^blocks\.(\d+)\.cross_attn\.v\.(.*)$": r"blocks.\1.attn2.to_v.\2",
r"^blocks\.(\d+)\.cross_attn\.o\.(.*)$": r"blocks.\1.attn2.to_out.\2",
r"^blocks\.(\d+)\.cross_attn\.norm_q\.(.*)$": r"blocks.\1.attn2.norm_q.\2",
r"^blocks\.(\d+)\.cross_attn\.norm_k\.(.*)$": r"blocks.\1.attn2.norm_k.\2",
# Norms and FFN
r"^blocks\.(\d+)\.norm1\.(.*)$": r"blocks.\1.norm1.\2",
r"^blocks\.(\d+)\.norm3\.(.*)$": r"blocks.\1.self_attn_residual_norm.norm.\2",
r"^blocks\.(\d+)\.norm2\.(.*)$": r"blocks.\1.norm3.\2",
r"^blocks\.(\d+)\.ffn\.0\.(.*)$": r"blocks.\1.ffn.fc_in.\2",
r"^blocks\.(\d+)\.ffn\.2\.(.*)$": r"blocks.\1.ffn.fc_out.\2",
r"^blocks\.(\d+)\.modulation$": r"blocks.\1.scale_shift_table",
# Embeddings - DON'T add .proj here! WanVideoArchConfig's param_names_mapping will add it
# patch_embedding.weight stays as patch_embedding.weight (HF format needs this)
r"^text_embedding\.0\.(.*)$": r"condition_embedder.text_embedder.fc_in.\1",
r"^text_embedding\.2\.(.*)$": r"condition_embedder.text_embedder.fc_out.\1",
r"^time_embedding\.0\.(.*)$": r"condition_embedder.time_embedder.mlp.fc_in.\1",
r"^time_embedding\.2\.(.*)$": r"condition_embedder.time_embedder.mlp.fc_out.\1",
r"^time_projection\.1\.(.*)$": r"condition_embedder.time_modulation.linear.\1",
# Head
r"^head\.head\.(.*)$": r"proj_out.\1",
r"^head\.norm\.(.*)$": r"norm_out.\1",
r"^head\.modulation$": r"scale_shift_table",
# SLA proj_l weights - include them! They're the distilled attention weights
r"^blocks\.(\d+)\.self_attn\.attn_op\.local_attn\.proj_l\.(.*)$": r"blocks.\1.attn1.attn_impl.proj_l.\2",
}
# No keys to skip - we want all weights including proj_l
SKIP_PATTERNS = []
def should_skip_key(key: str) -> bool:
"""Check if a key should be skipped (SLA-specific weights)."""
for pattern in SKIP_PATTERNS:
if re.match(pattern, key):
return True
return False
def convert_key(turbo_key: str) -> str:
"""Convert TurboDiffusion key to Diffusers format."""
for pattern, replacement in TURBODIFFUSION_WEIGHT_MAPPING.items():
if re.match(pattern, turbo_key):
return re.sub(pattern, replacement, turbo_key)
return turbo_key # Return unchanged if no pattern matches
def reshape_patch_embedding(tensor: torch.Tensor, target_shape: tuple) -> torch.Tensor:
"""Reshape patch_embedding from [D, C*P_t*P_h*P_w] to [D, C, P_t, P_h, P_w]."""
if len(tensor.shape) == 2 and len(target_shape) == 5:
return tensor.view(target_shape)
return tensor
def get_reference_shapes(reference_repo: str) -> dict:
"""Download reference model and get expected shapes for patch_embedding."""
print(f"Downloading reference model shapes from {reference_repo}...")
# Download just the transformer config and a weight file to get shapes
local_dir = snapshot_download(
repo_id=reference_repo,
allow_patterns=["transformer/config.json", "transformer/diffusion_pytorch_model*.safetensors"],
local_dir_use_symlinks=False)
# Load the first safetensors file to get shapes
weight_files = glob.glob(os.path.join(local_dir, "transformer", "*.safetensors"))
shapes = {}
for wf in weight_files:
with safe_open(wf, framework="pt") as f:
for key in f.keys():
shapes[key] = f.get_tensor(key).shape
return shapes
def main():
parser = argparse.ArgumentParser(description="Convert TurboDiffusion checkpoint to Diffusers format")
parser.add_argument("--input_path", type=str, required=True, help="Path to TurboDiffusion .pth checkpoint")
parser.add_argument("--output_dir", type=str, required=True, help="Output directory for converted safetensors")
parser.add_argument("--reference_repo",
type=str,
default="Wan-AI/Wan2.1-T2V-1.3B-Diffusers",
help="Reference HF repo to get expected tensor shapes")
parser.add_argument("--skip_sla_weights",
action="store_true",
default=False,
help="Skip SLA-specific weights (proj_l) that aren't in base model")
args = parser.parse_args()
# Load TurboDiffusion checkpoint
print(f"Loading TurboDiffusion checkpoint from {args.input_path}...")
turbo_state_dict = torch.load(args.input_path, map_location="cpu", weights_only=True)
print(f"Loaded {len(turbo_state_dict)} keys")
# Get reference shapes for reshaping
ref_shapes = get_reference_shapes(args.reference_repo)
print(f"Got {len(ref_shapes)} reference shapes")
# Convert keys and reshape tensors
converted_state_dict = {}
skipped_keys = []
for turbo_key, tensor in turbo_state_dict.items():
# Skip SLA-specific weights if requested
if args.skip_sla_weights and should_skip_key(turbo_key):
skipped_keys.append(turbo_key)
continue
# Convert key name
new_key = convert_key(turbo_key)
# Reshape patch_embedding if needed
if "patch_embedding" in new_key and new_key in ref_shapes:
target_shape = ref_shapes[new_key]
if tensor.shape != target_shape:
print(f"Reshaping {new_key}: {tensor.shape} -> {target_shape}")
tensor = reshape_patch_embedding(tensor, target_shape)
# Verify shape matches reference if available
if new_key in ref_shapes:
if tensor.shape != ref_shapes[new_key]:
print(f"WARNING: Shape mismatch for {new_key}: got {tensor.shape}, expected {ref_shapes[new_key]}")
converted_state_dict[new_key] = tensor
print(f"\nConversion summary:")
print(f" Converted: {len(converted_state_dict)} keys")
print(f" Skipped (SLA): {len(skipped_keys)} keys")
if skipped_keys:
print(f"\nSkipped SLA keys (first 5):")
for k in skipped_keys[:5]:
print(f" - {k}")
# Create output directory
os.makedirs(args.output_dir, exist_ok=True)
# Save as safetensors
output_path = os.path.join(args.output_dir, "diffusion_pytorch_model.safetensors")
print(f"\nSaving to {output_path}...")
save_file(converted_state_dict, output_path)
# Copy config.json from reference
ref_local = snapshot_download(repo_id=args.reference_repo,
allow_patterns=["transformer/config.json"],
local_dir_use_symlinks=False)
src_config = os.path.join(ref_local, "transformer", "config.json")
dst_config = os.path.join(args.output_dir, "config.json")
shutil.copy(src_config, dst_config)
print(f"Copied config.json")
print(f"\nDone! Converted weights saved to: {args.output_dir}")
print(f"\nNext steps:")
print(f" 1. Use create_hf_repo.py to create a complete diffusers repo:")
print(f" python scripts/checkpoint_conversion/create_hf_repo.py \\")
print(f" --repo_id Wan-AI/Wan2.1-T2V-1.3B-Diffusers \\")
print(f" --local_dir /tmp/turbodiffusion-wan \\")
print(f" --checkpoint_dir {args.output_dir} \\")
print(f" --push_to_hub --upload_repo_id YOUR_USERNAME/TurboWan-Diffusers")
if __name__ == "__main__":
main()
scripts/checkpoint_conversion/dreamx_world_ar_to_diffusers.py
-124
View File
@@ -1,124 +0,0 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: Apache-2.0
"""Convert DreamX-World-5B autoregressive weights to FastVideo layout.
The HF repository stores one raw official ``model.safetensors`` whose keys match
FastVideo's native ``DreamXWorldARTransformer3DModel``. The converter writes a
Diffusers-like root with ``transformer/config.json`` and reusable Wan2.2
components. Use ``--symlink-transformer`` locally to avoid duplicating the 21GB
AR tensor file.
"""
from __future__ import annotations
import argparse
import json
import shutil
from pathlib import Path
TRANSFORMER_CONFIG: dict[str, object] = {
"_class_name": "DreamXWorldARTransformer3DModel",
"model_type": "ti2v",
"patch_size": [1, 2, 2],
"text_len": 512,
"num_attention_heads": 24,
"attention_head_dim": 128,
"in_channels": 48,
"out_channels": 48,
"text_dim": 4096,
"freq_dim": 256,
"ffn_dim": 14336,
"num_layers": 30,
"local_attn_size": 12,
"sink_size": 3,
"cross_attn_norm": True,
"qk_norm": True,
"eps": 1e-6,
"add_control_adapter": True,
"cam_method": "prope",
"attn_compress": 4,
"cam_self_attn_layers": list(range(30)),
"num_frames_per_block": 3,
}
MODEL_INDEX: dict[str, object] = {
"_class_name": "DreamXWorldARPipeline",
"_diffusers_version": "0.31.0",
"scheduler": ["diffusers", "FlowMatchEulerDiscreteScheduler"],
"text_encoder": ["transformers", "UMT5EncoderModel"],
"tokenizer": ["transformers", "AutoTokenizer"],
"transformer": ["diffusers", "DreamXWorldARTransformer3DModel"],
"vae": ["diffusers", "AutoencoderKLWan"],
}
REUSED_COMPONENTS = ("scheduler", "text_encoder", "tokenizer", "vae")
def _source_safetensors(source: Path) -> Path:
if source.is_file():
return source
path = source / "model.safetensors"
if not path.exists():
raise FileNotFoundError(f"Missing AR model.safetensors under {source}")
return path
def convert_transformer(source: Path, output: Path, symlink_transformer: bool) -> None:
src = _source_safetensors(source)
transformer_dir = output / "transformer"
transformer_dir.mkdir(parents=True, exist_ok=True)
(transformer_dir / "config.json").write_text(json.dumps(TRANSFORMER_CONFIG, indent=2) + "\n")
dst = transformer_dir / "model.safetensors"
if dst.is_symlink() and not dst.exists():
dst.unlink() # dangling symlink: replace so re-conversion self-heals
if dst.exists():
return
if symlink_transformer:
dst.symlink_to(src.resolve())
else:
shutil.copy2(src, dst)
def _copy_or_link_component(component: str, component_source: Path, output: Path, symlink: bool) -> None:
src = component_source / component
dst = output / component
if not src.exists():
raise FileNotFoundError(f"Missing reused component source: {src}")
if dst.is_symlink() and not dst.exists():
dst.unlink() # dangling symlink: replace so re-conversion self-heals
if dst.exists():
return
if symlink:
dst.symlink_to(src.resolve(), target_is_directory=src.is_dir())
elif src.is_dir():
shutil.copytree(src, dst)
else:
shutil.copy2(src, dst)
def write_model_index(output: Path, component_source: Path | None, symlink_components: bool) -> None:
if component_source is not None:
for component in REUSED_COMPONENTS:
_copy_or_link_component(component, component_source, output, symlink_components)
missing = [component for component in REUSED_COMPONENTS if not (output / component).exists()]
if missing:
print("skipping model_index.json; missing reused components: " + ", ".join(missing))
return
(output / "model_index.json").write_text(json.dumps(MODEL_INDEX, indent=2) + "\n")
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--source", type=Path, required=True)
parser.add_argument("--output", type=Path, required=True)
parser.add_argument("--component-source", type=Path)
parser.add_argument("--symlink-components", action="store_true")
parser.add_argument("--symlink-transformer", action="store_true")
args = parser.parse_args()
convert_transformer(args.source, args.output, args.symlink_transformer)
write_model_index(args.output, args.component_source, args.symlink_components)
if __name__ == "__main__":
main()
scripts/checkpoint_conversion/dreamx_world_to_diffusers.py
-226
View File
@@ -1,226 +0,0 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: Apache-2.0
"""Convert DreamX-World-5B-Cam raw transformer weights to FastVideo-loadable format.
The GD-ML/DreamX-World-5B-Cam repository stores the transformer as raw
DreamX/Wan official shards. FastVideo's TransformerLoader expects a Diffusers-like
transformer folder with a config.json and safetensors whose keys can be mapped by
WanVideoConfig.param_names_mapping. This script performs the raw official ->
Diffusers-like key rename and writes the DreamX 5B-Cam transformer config.
Example:
python scripts/checkpoint_conversion/dreamx_world_to_diffusers.py \
--source official_weights/dreamx_world \
--output converted_weights/dreamx_world
"""
from __future__ import annotations
import argparse
import json
import re
import shutil
from collections import OrderedDict
from pathlib import Path
import torch
from huggingface_hub import save_torch_state_dict
from safetensors import safe_open
from safetensors.torch import load_file
OFFICIAL_TO_DIFFUSERS_MAPPING: dict[str, str] = {
r"^text_embedding\.0\.(.*)$": r"condition_embedder.text_embedder.linear_1.\1",
r"^text_embedding\.2\.(.*)$": r"condition_embedder.text_embedder.linear_2.\1",
r"^time_embedding\.0\.(.*)$": r"condition_embedder.time_embedder.linear_1.\1",
r"^time_embedding\.2\.(.*)$": r"condition_embedder.time_embedder.linear_2.\1",
r"^time_projection\.1\.(.*)$": r"condition_embedder.time_proj.\1",
r"^img_emb\.proj\.0\.(.*)$": r"condition_embedder.image_embedder.norm1.\1",
r"^img_emb\.proj\.1\.(.*)$": r"condition_embedder.image_embedder.ff.net.0.proj.\1",
r"^img_emb\.proj\.3\.(.*)$": r"condition_embedder.image_embedder.ff.net.2.\1",
r"^img_emb\.proj\.4\.(.*)$": r"condition_embedder.image_embedder.norm2.\1",
r"^head\.modulation": r"scale_shift_table",
r"^head\.head\.(.*)$": r"proj_out.\1",
r"^blocks\.(\d+)\.self_attn\.q\.(.*)$": r"blocks.\1.attn1.to_q.\2",
r"^blocks\.(\d+)\.self_attn\.k\.(.*)$": r"blocks.\1.attn1.to_k.\2",
r"^blocks\.(\d+)\.self_attn\.v\.(.*)$": r"blocks.\1.attn1.to_v.\2",
r"^blocks\.(\d+)\.self_attn\.o\.(.*)$": r"blocks.\1.attn1.to_out.0.\2",
r"^blocks\.(\d+)\.self_attn\.norm_q\.(.*)$": r"blocks.\1.attn1.norm_q.\2",
r"^blocks\.(\d+)\.self_attn\.norm_k\.(.*)$": r"blocks.\1.attn1.norm_k.\2",
r"^blocks\.(\d+)\.cross_attn\.q\.(.*)$": r"blocks.\1.attn2.to_q.\2",
r"^blocks\.(\d+)\.cross_attn\.k\.(.*)$": r"blocks.\1.attn2.to_k.\2",
r"^blocks\.(\d+)\.cross_attn\.k_img\.(.*)$": r"blocks.\1.attn2.add_k_proj.\2",
r"^blocks\.(\d+)\.cross_attn\.v\.(.*)$": r"blocks.\1.attn2.to_v.\2",
r"^blocks\.(\d+)\.cross_attn\.v_img\.(.*)$": r"blocks.\1.attn2.add_v_proj.\2",
r"^blocks\.(\d+)\.cross_attn\.o\.(.*)$": r"blocks.\1.attn2.to_out.0.\2",
r"^blocks\.(\d+)\.cross_attn\.norm_q\.(.*)$": r"blocks.\1.attn2.norm_q.\2",
r"^blocks\.(\d+)\.cross_attn\.norm_k\.(.*)$": r"blocks.\1.attn2.norm_k.\2",
r"^blocks\.(\d+)\.cross_attn\.norm_k_img\.(.*)$": r"blocks.\1.attn2.norm_added_k.\2",
r"^blocks\.(\d+)\.ffn\.0\.(.*)$": r"blocks.\1.ffn.net.0.proj.\2",
r"^blocks\.(\d+)\.ffn\.2\.(.*)$": r"blocks.\1.ffn.net.2.\2",
r"^blocks\.(\d+)\.modulation": r"blocks.\1.scale_shift_table",
r"^blocks\.(\d+)\.norm3\.(.*)$": r"blocks.\1.norm2.\2",
}
TRANSFORMER_CONFIG: dict[str, object] = {
"_class_name": "DreamXWorldTransformer3DModel",
"patch_size": [1, 2, 2],
"text_len": 512,
"num_attention_heads": 24,
"attention_head_dim": 128,
"in_channels": 48,
"out_channels": 48,
"text_dim": 4096,
"freq_dim": 256,
"ffn_dim": 14336,
"num_layers": 30,
"cross_attn_norm": True,
"qk_norm": "rms_norm_across_heads",
"eps": 1e-6,
"image_dim": None,
"added_kv_proj_dim": None,
"rope_max_seq_len": 1024,
"add_control_adapter": True,
"cam_method": "prope",
"attn_compress": 1,
"cam_self_attn_layers": None,
}
MODEL_INDEX: dict[str, object] = {
"_class_name": "DreamXWorldPipeline",
"_diffusers_version": "0.31.0",
"scheduler": ["diffusers", "FlowMatchEulerDiscreteScheduler"],
"text_encoder": ["transformers", "UMT5EncoderModel"],
"tokenizer": ["transformers", "AutoTokenizer"],
"transformer": ["diffusers", "DreamXWorldTransformer3DModel"],
"vae": ["diffusers", "AutoencoderKLWan"],
}
REUSED_COMPONENTS = ("scheduler", "text_encoder", "tokenizer", "vae")
def map_transformer_key(key: str) -> str:
for pattern, replacement in OFFICIAL_TO_DIFFUSERS_MAPPING.items():
if re.match(pattern, key):
return re.sub(pattern, replacement, key)
return key
def _safetensor_files(source: Path) -> list[Path]:
if source.is_file():
if source.suffix != ".safetensors":
raise ValueError(f"Only .safetensors files are supported, got {source}")
return [source]
index_path = source / "diffusion_pytorch_model.safetensors.index.json"
if index_path.exists():
index = json.loads(index_path.read_text())
return sorted({source / shard for shard in index["weight_map"].values()})
files = sorted(source.glob("*.safetensors"))
if not files:
raise FileNotFoundError(f"No safetensors files found under {source}")
return files
def convert_transformer(source: Path, output: Path, max_shard_size: str) -> None:
transformer_dir = output / "transformer"
transformer_dir.mkdir(parents=True, exist_ok=True)
(transformer_dir / "config.json").write_text(json.dumps(TRANSFORMER_CONFIG, indent=2) + "\n")
converted: OrderedDict[str, torch.Tensor] = OrderedDict()
for shard in _safetensor_files(source):
print(f"loading {shard}")
for key, tensor in load_file(shard, device="cpu").items():
new_key = map_transformer_key(key)
if new_key in converted:
raise ValueError(f"Duplicate converted key: {new_key}")
converted[new_key] = tensor
print(f"saving {len(converted)} tensors to {transformer_dir}")
save_torch_state_dict(converted, transformer_dir, max_shard_size=max_shard_size)
def _copy_or_link_component(component: str, component_source: Path, output: Path, symlink: bool) -> None:
src = component_source / component
dst = output / component
if not src.exists():
raise FileNotFoundError(f"Missing reused component source: {src}")
if dst.is_symlink() and not dst.exists():
dst.unlink() # dangling symlink: replace so re-conversion self-heals
if dst.exists():
print(f"keeping existing {dst}")
return
if symlink:
dst.symlink_to(src.resolve(), target_is_directory=src.is_dir())
print(f"linked {dst} -> {src}")
elif src.is_dir():
shutil.copytree(src, dst)
print(f"copied {src} -> {dst}")
else:
shutil.copy2(src, dst)
print(f"copied {src} -> {dst}")
def write_model_index(output: Path, component_source: Path | None, symlink_components: bool) -> None:
if component_source is not None:
for component in REUSED_COMPONENTS:
_copy_or_link_component(component, component_source, output, symlink_components)
missing = [component for component in REUSED_COMPONENTS if not (output / component).exists()]
if missing:
print("skipping model_index.json; missing reused components: " + ", ".join(missing))
print("pass --component-source <Wan2.2 Diffusers root> to copy or link reused components")
return
(output / "model_index.json").write_text(json.dumps(MODEL_INDEX, indent=2) + "\n")
print(f"wrote {output / 'model_index.json'}")
def analyze(source: Path) -> None:
total = 0
unchanged = 0
examples: list[tuple[str, str]] = []
for shard in _safetensor_files(source):
with safe_open(shard, framework="pt", device="cpu") as tensors:
for key in tensors:
total += 1
new_key = map_transformer_key(key)
unchanged += int(new_key == key)
if len(examples) < 20 and new_key != key:
examples.append((key, new_key))
print(f"total_keys={total} unchanged_keys={unchanged}")
for old, new in examples:
print(f"{old} -> {new}")
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--source",
type=Path,
required=True,
help="DreamX raw transformer directory or safetensors file")
parser.add_argument("--output",
type=Path,
required=True,
help="Output model root; transformer/ is created inside it")
parser.add_argument("--max-shard-size", default="10GB")
parser.add_argument(
"--component-source",
type=Path,
help="Optional Wan2.2 Diffusers root whose scheduler/text_encoder/tokenizer/vae components are reused.",
)
parser.add_argument(
"--symlink-components",
action="store_true",
help="Symlink reused components from --component-source instead of copying them.",
)
parser.add_argument("--analyze", action="store_true", help="Only print key mapping summary")
args = parser.parse_args()
if args.analyze:
analyze(args.source)
else:
convert_transformer(args.source, args.output, args.max_shard_size)
write_model_index(args.output, args.component_source, args.symlink_components)
if __name__ == "__main__":
main()
scripts/checkpoint_conversion/extract_llava_text_encoder.py
-232
View File
@@ -1,232 +0,0 @@
# SPDX-License-Identifier: Apache-2.0
"""
Extract LlamaModel text encoder from LLaVA weights for GameCraft/HunyuanVideo.
LLaVA stores the text encoder as `language_model.model.XXX`, but FastVideo/HunyuanVideo
expects just `XXX` (e.g., `layers.0.self_attn.q_proj.weight` instead of
`language_model.model.layers.0.self_attn.q_proj.weight`).
This script extracts just the LlamaModel weights and renames them.
Usage:
python scripts/checkpoint_conversion/extract_llava_text_encoder.py \
--input Hunyuan-GameCraft-1.0/weights/stdmodels/llava-llama-3-8b-v1_1-transformers \
--output official_weights/hunyuan-gamecraft-diffusers/text_encoder
"""
from __future__ import annotations
import argparse
import json
from pathlib import Path
from collections import OrderedDict
import torch
from safetensors import safe_open
from safetensors.torch import save_file
def extract_text_encoder(
input_dir: Path,
output_dir: Path,
) -> dict:
"""Extract LlamaModel from LLaVA weights."""
# Find all safetensor shards
shard_files = sorted(input_dir.glob("model-*.safetensors"))
if not shard_files:
raise FileNotFoundError(f"No safetensor files found in {input_dir}")
print(f"Found {len(shard_files)} weight shards")
# Collect all weights, filtering for language_model.model prefix
extracted_weights: OrderedDict[str, torch.Tensor] = OrderedDict()
for shard_file in shard_files:
print(f"Processing {shard_file.name}...")
with safe_open(str(shard_file), framework="pt") as f:
for key in f.keys():
# Extract only language_model.model weights
if key.startswith("language_model.model."):
# Strip the prefix
new_key = key.replace("language_model.model.", "")
extracted_weights[new_key] = f.get_tensor(key)
print(f"Extracted {len(extracted_weights)} weights")
# Save to output directory
output_dir.mkdir(parents=True, exist_ok=True)
# Save as single file (it should be <10GB which is fine for safetensors)
output_file = output_dir / "model.safetensors"
print(f"Saving to {output_file}...")
save_file(extracted_weights, str(output_file))
# Create config.json for LlamaModel
config = {
"_name_or_path": "llava-llama-3-8b-v1_1-text_encoder",
"architectures": ["LlamaModel"],
"attention_bias": False,
"attention_dropout": 0.0,
"bos_token_id": 128000,
"eos_token_id": 128001,
"head_dim": 128,
"hidden_act": "silu",
"hidden_size": 4096,
"initializer_range": 0.02,
"intermediate_size": 14336,
"max_position_embeddings": 8192,
"mlp_bias": False,
"model_type": "llama",
"num_attention_heads": 32,
"num_hidden_layers": 32,
"num_key_value_heads": 8,
"pretraining_tp": 1,
"rms_norm_eps": 1e-05,
"rope_scaling": None,
"rope_theta": 500000.0,
"tie_word_embeddings": False,
"torch_dtype": "float16",
"transformers_version": "4.48.0",
"use_cache": True,
"vocab_size": 128320
}
with open(output_dir / "config.json", "w") as f:
json.dump(config, f, indent=2)
print(f"Saved config.json")
# Copy tokenizer files if they exist
tokenizer_files = [
"tokenizer.json",
"tokenizer_config.json",
"special_tokens_map.json",
]
for fname in tokenizer_files:
src = input_dir / fname
if src.exists():
import shutil
shutil.copy2(src, output_dir / fname)
print(f"Copied {fname}")
return {"total": len(extracted_weights)}
def extract_clip_text_encoder(
input_dir: Path,
output_dir: Path,
) -> dict:
"""Extract CLIPTextModel from full CLIP model weights."""
# Find weight file
weight_files = list(input_dir.glob("model.safetensors")) + \
list(input_dir.glob("pytorch_model.bin"))
if not weight_files:
raise FileNotFoundError(f"No weight files found in {input_dir}")
weight_file = weight_files[0]
print(f"Loading {weight_file}...")
if weight_file.suffix == ".safetensors":
from safetensors.torch import load_file
state_dict = load_file(str(weight_file))
else:
state_dict = torch.load(weight_file, map_location="cpu")
# Extract only text_model weights
extracted_weights: OrderedDict[str, torch.Tensor] = OrderedDict()
for key, value in state_dict.items():
# CLIP full model has text_model.XXX, we want just XXX
if key.startswith("text_model."):
new_key = key.replace("text_model.", "")
extracted_weights[new_key] = value
if not extracted_weights:
# Maybe it's already just the text model
extracted_weights = OrderedDict(state_dict)
print(f"Extracted {len(extracted_weights)} weights")
# Save to output directory
output_dir.mkdir(parents=True, exist_ok=True)
output_file = output_dir / "model.safetensors"
print(f"Saving to {output_file}...")
save_file(extracted_weights, str(output_file))
# Create config.json for CLIPTextModel
config = {
"_name_or_path": "openai/clip-vit-large-patch14",
"architectures": ["CLIPTextModel"],
"attention_dropout": 0.0,
"bos_token_id": 0,
"dropout": 0.0,
"eos_token_id": 2,
"hidden_act": "quick_gelu",
"hidden_size": 768,
"initializer_factor": 1.0,
"initializer_range": 0.02,
"intermediate_size": 3072,
"layer_norm_eps": 1e-05,
"max_position_embeddings": 77,
"model_type": "clip_text_model",
"num_attention_heads": 12,
"num_hidden_layers": 12,
"pad_token_id": 1,
"projection_dim": 768,
"torch_dtype": "float16",
"transformers_version": "4.48.0",
"vocab_size": 49408
}
with open(output_dir / "config.json", "w") as f:
json.dump(config, f, indent=2)
print(f"Saved config.json")
# Copy tokenizer files if they exist
tokenizer_files = [
"tokenizer.json",
"tokenizer_config.json",
"special_tokens_map.json",
"vocab.json",
"merges.txt",
]
for fname in tokenizer_files:
src = input_dir / fname
if src.exists():
import shutil
shutil.copy2(src, output_dir / fname)
print(f"Copied {fname}")
return {"total": len(extracted_weights)}
def main():
parser = argparse.ArgumentParser(description="Extract text encoder from LLaVA or CLIP model weights.")
parser.add_argument("--input", type=str, required=True, help="Input directory containing model weights")
parser.add_argument("--output", type=str, required=True, help="Output directory for extracted weights")
parser.add_argument("--type",
type=str,
choices=["llama", "clip"],
default="llama",
help="Type of text encoder to extract")
args = parser.parse_args()
input_dir = Path(args.input)
output_dir = Path(args.output)
if not input_dir.exists():
raise FileNotFoundError(f"Input directory not found: {input_dir}")
if args.type == "llama":
stats = extract_text_encoder(input_dir, output_dir)
else:
stats = extract_clip_text_encoder(input_dir, output_dir)
print(f"\nExtraction complete! Total weights: {stats['total']}")
if __name__ == "__main__":
main()
scripts/checkpoint_conversion/longcat_to_fastvideo.py
-706
View File
@@ -1,706 +0,0 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: Apache-2.0
"""
Convert LongCat weights to FastVideo native format.
This script performs a complete conversion from original LongCat weights
to FastVideo native implementation in a single step:
1. Converts transformer weights (with QKV/KV splitting)
2. Copies other components (VAE, text encoder, tokenizer, scheduler)
3. Converts LoRA weights (cfg_step_lora, refinement_lora)
4. Updates config files to point to native model
Usage:
python scripts/checkpoint_conversion/longcat_to_fastvideo.py \
--source /path/to/LongCat-Video/weights/LongCat-Video \
--output weights/longcat-native \
--validate
"""
import argparse
import glob
import json
import re
import shutil
from pathlib import Path
from collections import OrderedDict
import torch
from safetensors.torch import load_file, save_file
from tqdm import tqdm
def split_qkv(qkv_weight: torch.Tensor, qkv_bias: torch.Tensor | None = None):
"""Split fused QKV projection into separate Q, K, V."""
dim = qkv_weight.shape[0] // 3
q, k, v = torch.chunk(qkv_weight, 3, dim=0)
if qkv_bias is not None:
q_bias, k_bias, v_bias = torch.chunk(qkv_bias, 3, dim=0)
else:
q_bias = k_bias = v_bias = None
return (q, k, v), (q_bias, k_bias, v_bias)
def split_kv(kv_weight: torch.Tensor, kv_bias: torch.Tensor | None = None):
"""Split fused KV projection into separate K, V."""
dim = kv_weight.shape[0] // 2
k, v = torch.chunk(kv_weight, 2, dim=0)
if kv_bias is not None:
k_bias, v_bias = torch.chunk(kv_bias, 2, dim=0)
else:
k_bias = v_bias = None
return (k, v), (k_bias, v_bias)
def convert_transformer_weights(source_weights: dict[str, torch.Tensor]) -> dict[str, torch.Tensor]:
"""
Convert LongCat transformer weights to native FastVideo format.
Main transformations:
1. Split fused QKV projections (self-attention)
2. Split fused KV projections (cross-attention)
3. Rename parameters according to mapping
"""
converted = OrderedDict()
processed_keys = set()
print(" Converting transformer weights...")
for key, value in tqdm(source_weights.items(), desc=" Processing parameters"):
if key in processed_keys:
continue
# === Embedders ===
if key.startswith("x_embedder."):
new_key = key.replace("x_embedder.", "patch_embed.")
converted[new_key] = value
elif key.startswith("t_embedder.mlp.0."):
new_key = key.replace("t_embedder.mlp.0.", "time_embedder.linear_1.")
converted[new_key] = value
elif key.startswith("t_embedder.mlp.2."):
new_key = key.replace("t_embedder.mlp.2.", "time_embedder.linear_2.")
converted[new_key] = value
elif key.startswith("y_embedder.y_proj.0."):
new_key = key.replace("y_embedder.y_proj.0.", "caption_embedder.linear_1.")
converted[new_key] = value
elif key.startswith("y_embedder.y_proj.2."):
new_key = key.replace("y_embedder.y_proj.2.", "caption_embedder.linear_2.")
converted[new_key] = value
# === Self-Attention QKV Splitting ===
elif ".attn.qkv.weight" in key:
block_idx = key.split(".")[1]
qkv_weight = value
qkv_bias_key = key.replace(".weight", ".bias")
qkv_bias = source_weights.get(qkv_bias_key)
(q, k, v), (q_bias, k_bias, v_bias) = split_qkv(qkv_weight, qkv_bias)
converted[f"blocks.{block_idx}.self_attn.to_q.weight"] = q
converted[f"blocks.{block_idx}.self_attn.to_k.weight"] = k
converted[f"blocks.{block_idx}.self_attn.to_v.weight"] = v
if q_bias is not None:
converted[f"blocks.{block_idx}.self_attn.to_q.bias"] = q_bias
converted[f"blocks.{block_idx}.self_attn.to_k.bias"] = k_bias
converted[f"blocks.{block_idx}.self_attn.to_v.bias"] = v_bias
processed_keys.add(key)
if qkv_bias is not None:
processed_keys.add(qkv_bias_key)
elif ".attn.qkv.bias" in key:
continue
elif ".attn.proj." in key:
new_key = key.replace(".attn.proj.", ".self_attn.to_out.")
converted[new_key] = value
elif ".attn.q_norm." in key or ".attn.k_norm." in key:
new_key = key.replace(".attn.", ".self_attn.")
converted[new_key] = value
# === Cross-Attention ===
elif ".cross_attn.q_linear." in key:
new_key = key.replace(".cross_attn.q_linear.", ".cross_attn.to_q.")
converted[new_key] = value
elif ".cross_attn.kv_linear.weight" in key:
block_idx = key.split(".")[1]
kv_weight = value
kv_bias_key = key.replace(".weight", ".bias")
kv_bias = source_weights.get(kv_bias_key)
(k, v), (k_bias, v_bias) = split_kv(kv_weight, kv_bias)
converted[f"blocks.{block_idx}.cross_attn.to_k.weight"] = k
converted[f"blocks.{block_idx}.cross_attn.to_v.weight"] = v
if k_bias is not None:
converted[f"blocks.{block_idx}.cross_attn.to_k.bias"] = k_bias
converted[f"blocks.{block_idx}.cross_attn.to_v.bias"] = v_bias
processed_keys.add(key)
if kv_bias is not None:
processed_keys.add(kv_bias_key)
elif ".cross_attn.kv_linear.bias" in key:
continue
elif ".cross_attn.proj." in key:
new_key = key.replace(".cross_attn.proj.", ".cross_attn.to_out.")
converted[new_key] = value
elif ".cross_attn.q_norm." in key or ".cross_attn.k_norm." in key:
converted[key] = value
# === Final Layer (must come BEFORE general transformer block patterns) ===
elif key.startswith("final_layer.adaLN_modulation.1."):
new_key = key.replace("final_layer.adaLN_modulation.1.", "final_layer.adaln_linear.")
converted[new_key] = value
# === Transformer Block AdaLN ===
elif ".adaLN_modulation.1." in key:
new_key = key.replace(".adaLN_modulation.1.", ".adaln_linear_1.")
converted[new_key] = value
# === Transformer Block Normalization ===
elif ".mod_norm_attn." in key or ".mod_norm_ffn." in key:
continue
elif ".pre_crs_attn_norm.weight" in key:
new_key = key.replace(".pre_crs_attn_norm.", ".norm_cross.")
converted[new_key] = value
elif ".pre_crs_attn_norm.bias" in key:
new_key = key.replace(".pre_crs_attn_norm.", ".norm_cross.")
converted[new_key] = value
# === FFN (SwiGLU) ===
elif ".ffn.w1." in key or ".ffn.w2." in key or ".ffn.w3." in key:
converted[key] = value
elif key.startswith("final_layer.norm_final."):
continue
elif key.startswith("final_layer.linear."):
new_key = key.replace("final_layer.linear.", "final_layer.proj.")
converted[new_key] = value
else:
print(f" ⚠️ Unknown key: {key}")
converted[key] = value
return converted
def validate_conversion(original: dict, converted: dict) -> bool:
"""Validate that conversion preserved all parameters correctly."""
print("\n Validating conversion...")
orig_count = sum(p.numel() for p in original.values())
conv_count = sum(p.numel() for p in converted.values())
dropped_count = 0
for key, value in original.items():
if ".mod_norm_attn." in key or ".mod_norm_ffn." in key:
dropped_count += value.numel()
elif "final_layer.norm_final." in key:
dropped_count += value.numel()
expected_conv_count = orig_count - dropped_count
print(f" Original parameters: {orig_count:,}")
print(f" Converted parameters: {conv_count:,}")
print(f" Dropped parameters (norms without params): {dropped_count:,}")
if conv_count != expected_conv_count:
print(f" ⚠️ Parameter count mismatch!")
return False
print(f" ✓ Parameter count matches")
# Verify QKV/KV splits
print("\n Verifying QKV/KV splits...")
num_blocks = 48
for i in range(num_blocks):
orig_qkv_weight = original.get(f"blocks.{i}.attn.qkv.weight")
if orig_qkv_weight is not None:
conv_q = converted[f"blocks.{i}.self_attn.to_q.weight"]
conv_k = converted[f"blocks.{i}.self_attn.to_k.weight"]
conv_v = converted[f"blocks.{i}.self_attn.to_v.weight"]
reconstructed = torch.cat([conv_q, conv_k, conv_v], dim=0)
if not torch.allclose(orig_qkv_weight, reconstructed):
print(f" ❌ QKV weight mismatch in block {i}")
return False
orig_kv_weight = original.get(f"blocks.{i}.cross_attn.kv_linear.weight")
if orig_kv_weight is not None:
conv_k = converted[f"blocks.{i}.cross_attn.to_k.weight"]
conv_v = converted[f"blocks.{i}.cross_attn.to_v.weight"]
reconstructed = torch.cat([conv_k, conv_v], dim=0)
if not torch.allclose(orig_kv_weight, reconstructed):
print(f" ❌ KV weight mismatch in block {i}")
return False
print(f" ✓ All splits verified successfully")
return True
def copy_component(source_dir: Path, output_dir: Path, component: str, mapping: dict = None) -> bool:
"""Copy a component directory, optionally with name mapping."""
source_name = mapping.get(component, component) if mapping else component
source_path = source_dir / source_name
if source_path.exists():
output_path = output_dir / component
if output_path.exists():
shutil.rmtree(output_path)
shutil.copytree(source_path, output_path)
print(f" ✓ {component} copied")
return True
else:
print(f" ⚠️ {component} not found, skipping")
return False
def create_model_index():
"""Create model_index.json for FastVideo native model."""
return {
"_class_name": "LongCatPipeline",
"_diffusers_version": "0.32.0",
"workload_type": "video-generation",
"tokenizer": ["transformers", "AutoTokenizer"],
"text_encoder": ["transformers", "UMT5EncoderModel"],
"vae": ["diffusers", "AutoencoderKLWan"],
"scheduler": ["diffusers", "FlowMatchEulerDiscreteScheduler"],
"transformer": ["diffusers", "LongCatTransformer3DModel"] # Native model
}
def update_transformer_config(transformer_dir: Path):
"""Update transformer config.json to point to native model."""
config_path = transformer_dir / "config.json"
if not config_path.exists():
print(" ⚠️ Transformer config not found, skipping")
return
with open(config_path, 'r') as f:
config = json.load(f)
if '_class_name' in config:
old_class = config['_class_name']
config['_class_name'] = 'LongCatTransformer3DModel'
print(f" Updated _class_name: {old_class} → LongCatTransformer3DModel")
else:
config['_class_name'] = 'LongCatTransformer3DModel'
print(f" Added _class_name: LongCatTransformer3DModel")
# Fix num_heads -> num_attention_heads for FastVideo compatibility
if 'num_heads' in config and 'num_attention_heads' not in config:
config['num_attention_heads'] = config.pop('num_heads')
print(f" Updated num_heads → num_attention_heads")
with open(config_path, 'w') as f:
json.dump(config, f, indent=2)
print(" ✓ Transformer config updated")
# ============================================================================
# LoRA Conversion Functions
# ============================================================================
def parse_lora_key(key: str) -> tuple[str, str]:
"""Parse LongCat LoRA key into module path and weight type."""
if key.startswith("lora___lorahyphen___"):
key = key[len("lora___lorahyphen___"):]
key = key.replace("___lorahyphen___", ".")
if ".lora_down.weight" in key:
return key.replace(".lora_down.weight", ""), "lora_down.weight"
elif ".lora_up.weight" in key:
return key.replace(".lora_up.weight", ""), "lora_up.weight"
elif ".lora_up.blocks." in key:
match = re.match(r"(.+)\.lora_up\.blocks\.(\d+)\.weight", key)
if match:
return match.group(1), f"lora_up.blocks.{match.group(2)}.weight"
elif ".alpha_scale" in key:
return key.replace(".alpha_scale", ""), "alpha_scale"
elif ".lora_alpha" in key:
return key.replace(".lora_alpha", ""), "lora_alpha"
raise ValueError(f"Unknown LoRA key format: {key}")
def map_lora_module(module_path: str) -> list[tuple[str, str]]:
"""Map LongCat module path to FastVideo paths. Returns [(path, component)]."""
# Self-attention QKV → Q, K, V
match = re.match(r"blocks\.(\d+)\.attn\.qkv", module_path)
if match:
b = match.group(1)
return [(f"blocks.{b}.self_attn.to_q", "q"), (f"blocks.{b}.self_attn.to_k", "k"),
(f"blocks.{b}.self_attn.to_v", "v")]
# Self-attention output
match = re.match(r"blocks\.(\d+)\.attn\.proj", module_path)
if match:
return [(f"blocks.{match.group(1)}.self_attn.to_out", "single")]
# Cross-attention Q
match = re.match(r"blocks\.(\d+)\.cross_attn\.q_linear", module_path)
if match:
return [(f"blocks.{match.group(1)}.cross_attn.to_q", "single")]
# Cross-attention KV → K, V
match = re.match(r"blocks\.(\d+)\.cross_attn\.kv_linear", module_path)
if match:
b = match.group(1)
return [(f"blocks.{b}.cross_attn.to_k", "k"), (f"blocks.{b}.cross_attn.to_v", "v")]
# FFN
match = re.match(r"blocks\.(\d+)\.ffn\.(w[123])", module_path)
if match:
return [(f"blocks.{match.group(1)}.ffn.{match.group(2)}", "single")]
# AdaLN modulation
match = re.match(r"blocks\.(\d+)\.adaLN_modulation\.1", module_path)
if match:
return [(f"blocks.{match.group(1)}.adaln_linear_1", "single")]
# Final layer
if module_path == "final_layer.adaLN_modulation.1":
return [("final_layer.adaln_linear", "single")]
if module_path == "final_layer.linear":
return [("final_layer.proj", "single")]
raise ValueError(f"Unknown LoRA module: {module_path}")
def convert_lora_weights(source_weights: dict[str, torch.Tensor], lora_name: str) -> dict[str, torch.Tensor]:
"""Convert LongCat LoRA to FastVideo format."""
print(f" Converting {lora_name}...")
print(f" Source keys: {len(source_weights)}")
converted = OrderedDict()
# Group by module
modules = {}
for key in source_weights.keys():
try:
module_path, weight_type = parse_lora_key(key)
if module_path not in modules:
modules[module_path] = {}
modules[module_path][weight_type] = key
except ValueError:
continue
# Process each module
for module_path, weight_keys in modules.items():
try:
targets = map_lora_module(module_path)
except ValueError:
continue
# Get alpha_scale if present (defaults to 1.0 if missing)
alpha_scale = 1.0
if "alpha_scale" in weight_keys:
alpha_scale_tensor = source_weights[weight_keys["alpha_scale"]]
alpha_scale = alpha_scale_tensor.item() if alpha_scale_tensor.numel() == 1 else float(
alpha_scale_tensor.mean())
# Handle lora_down (lora_A)
if "lora_down.weight" in weight_keys:
lora_down = source_weights[weight_keys["lora_down.weight"]]
if len(targets) == 1:
converted[f"{targets[0][0]}.lora_A"] = lora_down
# Compute alpha from alpha_scale and rank
rank = lora_down.shape[0]
alpha = alpha_scale * rank
converted[f"{targets[0][0]}.lora_alpha"] = torch.tensor(alpha, dtype=torch.float32)
else:
# Split for fused projections
n = len(targets)
rank = lora_down.shape[0] // n
for i, (path, _) in enumerate(targets):
converted[f"{path}.lora_A"] = lora_down[i * rank:(i + 1) * rank, :]
# Compute alpha from alpha_scale and rank for each split
alpha = alpha_scale * rank
converted[f"{path}.lora_alpha"] = torch.tensor(alpha, dtype=torch.float32)
# Handle lora_up (lora_B) - may have multiple blocks
lora_up_blocks = []
i = 0
while f"lora_up.blocks.{i}.weight" in weight_keys:
lora_up_blocks.append(source_weights[weight_keys[f"lora_up.blocks.{i}.weight"]])
i += 1
if lora_up_blocks:
# Multi-block LoRA: construct block-diagonal lora_B
# This is equivalent to the multi-block computation without modifying fastvideo
n_blocks = len(lora_up_blocks)
out_per_block, rank_per_block = lora_up_blocks[0].shape # e.g., [4096, 128]
if len(targets) == 1:
# Single layer with multi-block: create block-diagonal matrix
total_out = out_per_block * n_blocks
total_rank = rank_per_block * n_blocks
lora_B_blockdiag = torch.zeros(total_out, total_rank, dtype=lora_up_blocks[0].dtype)
for i in range(n_blocks):
lora_B_blockdiag[i * out_per_block:(i + 1) * out_per_block,
i * rank_per_block:(i + 1) * rank_per_block] = lora_up_blocks[i]
converted[f"{targets[0][0]}.lora_B"] = lora_B_blockdiag
# Note: rank for alpha calculation should be total_rank (will be computed from lora_A.shape[0])
else:
# Multi-block with split targets (e.g., QKV split)
# Each target gets one block
for i, (path, _) in enumerate(targets):
if i < n_blocks:
converted[f"{path}.lora_B"] = lora_up_blocks[i]
elif "lora_up.weight" in weight_keys:
lora_up = source_weights[weight_keys["lora_up.weight"]]
# Split if needed
if len(targets) == 1:
converted[f"{targets[0][0]}.lora_B"] = lora_up
else:
n = len(targets)
out_dim = lora_up.shape[0] // n
for i, (path, _) in enumerate(targets):
converted[f"{path}.lora_B"] = lora_up[i * out_dim:(i + 1) * out_dim, :]
else:
continue
print(f" Output keys: {len(converted)} (including lora_alpha)")
# Count how many lora_alpha values were added
alpha_count = sum(1 for k in converted.keys() if "lora_alpha" in k)
print(f" Alpha values saved: {alpha_count}")
return converted
def convert_loras(source_dir: Path, output_dir: Path) -> bool:
"""Convert all LoRA files in source directory."""
lora_source = source_dir / "lora"
if not lora_source.exists():
print(" No LoRA directory found, skipping")
return False
lora_files = list(lora_source.glob("*.safetensors"))
if not lora_files:
print(" No LoRA files found, skipping")
return False
print(f" Found {len(lora_files)} LoRA file(s)")
# Map LoRA filenames to subdirectory names for FastVideo compatibility
# Each LoRA gets its own directory under lora/
lora_subdir_mapping = {
"cfg_step_lora.safetensors": "distilled",
"refinement_lora.safetensors": "refinement",
}
for lora_file in lora_files:
try:
# Determine output subdirectory - use mapping if available, otherwise generic name
if lora_file.name in lora_subdir_mapping:
lora_subdir_name = lora_subdir_mapping[lora_file.name]
else:
# For unknown LoRAs, create subdirectory based on filename
lora_subdir_name = lora_file.stem
lora_output = output_dir / "lora" / lora_subdir_name
lora_output.mkdir(parents=True, exist_ok=True)
# Load
source_weights = load_file(str(lora_file))
# Convert
converted = convert_lora_weights(source_weights, lora_file.stem)
# Save
output_file = lora_output / lora_file.name
save_file(converted, str(output_file))
size_mb = output_file.stat().st_size / (1024**2)
print(f" ✓ {lora_file.name} → lora/{lora_subdir_name}/ ({size_mb:.1f} MB)")
except Exception as e:
print(f" ❌ Failed to convert {lora_file.name}: {e}")
return False
return True
def main():
parser = argparse.ArgumentParser(description="Convert LongCat weights to FastVideo native format")
parser.add_argument(
"--source",
type=str,
required=True,
help="Path to original LongCat weights (LongCat-Video/weights/LongCat-Video/)",
)
parser.add_argument(
"--output",
type=str,
required=True,
help="Path to output directory for native weights",
)
parser.add_argument(
"--validate",
action="store_true",
help="Run validation after conversion",
)
args = parser.parse_args()
source_dir = Path(args.source)
output_dir = Path(args.output)
# Check source directory
if not source_dir.exists():
print(f"❌ Error: Source directory not found: {source_dir}")
return 1
# Check for dit/transformer directory (original uses 'dit', we output to 'transformer')
transformer_source = source_dir / "dit"
if not transformer_source.exists():
print(f"❌ Error: DiT directory not found in source")
return 1
print("=" * 60)
print("LongCat → FastVideo Native Conversion")
print("=" * 60)
print(f"Source: {source_dir}")
print(f"Output: {output_dir}")
print()
# Step 1: Convert transformer weights
print("[Step 1/4] Converting transformer weights...")
# Load source weights
shard_files = sorted(glob.glob(str(transformer_source / "*.safetensors")))
if not shard_files:
print(f"❌ Error: No safetensors files found in {transformer_source}")
return 1
print(f" Found {len(shard_files)} shard(s)")
source_weights = {}
for shard_file in shard_files:
print(f" Loading {Path(shard_file).name}...")
source_weights.update(load_file(shard_file))
print(f" Loaded {len(source_weights)} parameters")
# Convert
converted_weights = convert_transformer_weights(source_weights)
print(f" Converted to {len(converted_weights)} parameters")
# Validate if requested
if args.validate:
if not validate_conversion(source_weights, converted_weights):
print("\n❌ Validation failed!")
return 1
print("\n✓ Validation passed!")
# Save
transformer_output = output_dir / "transformer"
transformer_output.mkdir(parents=True, exist_ok=True)
output_file = transformer_output / "model.safetensors"
print(f"\n Saving to {output_file}...")
save_file(converted_weights, str(output_file))
size_gb = output_file.stat().st_size / (1024**3)
print(f" ✓ Saved ({size_gb:.2f} GB)")
print()
# Step 2: Copy other components
print("[Step 2/5] Copying other components...")
output_dir.mkdir(parents=True, exist_ok=True)
components = ["vae", "text_encoder", "tokenizer", "scheduler"]
for component in components:
copy_component(source_dir, output_dir, component)
print()
# Step 3: Convert LoRA weights
print("[Step 3/5] Converting LoRA weights...")
convert_loras(source_dir, output_dir)
print()
# Step 4: Update transformer config
print("[Step 4/5] Updating transformer config...")
# Copy config.json from source if exists
source_config = transformer_source / "config.json"
output_config = transformer_output / "config.json"
if source_config.exists():
shutil.copy(source_config, output_config)
print(f" Copied config.json")
update_transformer_config(transformer_output)
print()
# Step 5: Create model_index.json
print("[Step 5/5] Creating model_index.json...")
model_index_path = output_dir / "model_index.json"
with open(model_index_path, 'w') as f:
json.dump(create_model_index(), f, indent=2)
print(f" ✓ Created {model_index_path}")
print()
print("=" * 60)
print("✓ Conversion Complete!")
print("=" * 60)
print(f"Native weights ready at: {output_dir}")
print()
print("Converted components:")
print(" ✓ Transformer (native FastVideo implementation)")
print(" ✓ VAE, text encoder, tokenizer, scheduler")
if (output_dir / "lora").exists():
lora_dirs = [d for d in (output_dir / "lora").iterdir() if d.is_dir()]
if lora_dirs:
print(f" ✓ LoRA weights ({len(lora_dirs)} adapters)")
for lora_dir in sorted(lora_dirs):
print(f" - lora/{lora_dir.name}/")
print()
print("Next steps:")
print()
print(" 1. Test basic generation:")
print(" from fastvideo import VideoGenerator")
print(f" generator = VideoGenerator.from_pretrained('{output_dir}')")
print(" video = generator.generate_video(")
print(" prompt='A cat playing piano',")
print(" num_inference_steps=50")
print(" )")
print()
if (output_dir / "lora" / "distilled").exists():
print(" 2. Test distilled generation (16 steps with LoRA):")
print(f" generator = VideoGenerator.from_pretrained('{output_dir}',")
print(f" lora_path='{output_dir}/lora/distilled',")
print(" lora_nickname='distilled')")
print(" video = generator.generate_video(")
print(" prompt='A cat playing piano',")
print(" num_inference_steps=16,")
print(" guidance_scale=1.0)")
print()
print()
return 0
if __name__ == "__main__":
exit(main())
scripts/checkpoint_conversion/validate_longcat_weights.py
-237
View File
@@ -1,237 +0,0 @@
import os
import json
import argparse
from pathlib import Path
from safetensors import safe_open
def validate_components(model_path):
"""Validate all model components exist."""
print("=" * 60)
print("VALIDATING COMPONENTS")
print("=" * 60)
components = {
"tokenizer": ["special_tokens_map.json", "tokenizer_config.json"],
"text_encoder": ["config.json", "model.safetensors.index.json"],
"vae": ["config.json", "diffusion_pytorch_model.safetensors"],
"scheduler": ["scheduler_config.json"],
"transformer": ["config.json", "diffusion_pytorch_model.safetensors.index.json"]
}
all_valid = True
for component, required_files in components.items():
component_path = os.path.join(model_path, component)
print(f"\n{component}:")
if not os.path.exists(component_path):
print(f" ✗ Directory not found")
all_valid = False
continue
for req_file in required_files:
file_path = os.path.join(component_path, req_file)
exists = os.path.exists(file_path)
symbol = "✓" if exists else "✗"
print(f" {symbol} {req_file}")
if not exists:
all_valid = False
return all_valid
def validate_dit_weights(dit_path):
"""Validate DiT weights structure."""
print("\n" + "=" * 60)
print("VALIDATING DiT WEIGHTS")
print("=" * 60)
# Load config
config_path = os.path.join(dit_path, "config.json")
with open(config_path) as f:
config = json.load(f)
hidden_size = config["hidden_size"]
depth = config["depth"]
num_heads = config["num_heads"]
print(f"\nArchitecture:")
print(f" - hidden_size: {hidden_size}")
print(f" - depth: {depth}")
print(f" - num_heads: {num_heads}")
# Load weight index
index_path = os.path.join(dit_path, "diffusion_pytorch_model.safetensors.index.json")
with open(index_path) as f:
index = json.load(f)
weight_map = index["weight_map"]
all_keys = list(weight_map.keys())
print(f"\nWeight statistics:")
print(f" - Total keys: {len(all_keys)}")
print(f" - Total size: {index['metadata']['total_size'] / 1e9:.2f} GB")
# Check structure
embedder_keys = [k for k in all_keys if 'embedder' in k]
block_keys = [k for k in all_keys if k.startswith('blocks.')]
final_keys = [k for k in all_keys if k.startswith('final_layer.')]
print(f"\nKey distribution:")
print(f" - Embedder layers: {len(embedder_keys)}")
print(f" - Transformer blocks: {len(block_keys)}")
print(f" - Final layer: {len(final_keys)}")
# Verify all blocks present
block_nums = set()
for key in block_keys:
if key.startswith('blocks.'):
block_num = int(key.split('.')[1])
block_nums.add(block_num)
expected_blocks = set(range(depth))
missing_blocks = expected_blocks - block_nums
if missing_blocks:
print(f"\n✗ Missing blocks: {sorted(missing_blocks)}")
return False
else:
print(f"\n✓ All {depth} blocks present (0-{depth-1})")
# Sample weights
first_shard = os.path.join(dit_path, "diffusion_pytorch_model-00001-of-00006.safetensors")
print(f"\nSampling weights from first shard:")
with safe_open(first_shard, framework="pt", device="cpu") as f:
sample_keys = [k for k in f.keys() if k in all_keys][:5]
for key in sample_keys:
tensor = f.get_tensor(key)
print(f" - {key}")
print(f" Shape: {tuple(tensor.shape)}, Dtype: {tensor.dtype}")
return True
def validate_shapes(dit_path):
"""Validate weight shapes match expected architecture."""
print("\n" + "=" * 60)
print("VALIDATING WEIGHT SHAPES")
print("=" * 60)
# Load config
config_path = os.path.join(dit_path, "config.json")
with open(config_path) as f:
config = json.load(f)
hidden_size = config["hidden_size"]
num_heads = config["num_heads"]
head_dim = hidden_size // num_heads
adaln_dim = config.get("adaln_tembed_dim", 512)
mlp_ratio = config.get("mlp_ratio", 4)
# Calculate FFN hidden_dim using SwiGLU formula from blocks.py
# hidden_dim = int(2 * (hidden_size * mlp_ratio) / 3)
# rounded to multiple_of=256
multiple_of = 256
ffn_hidden = int(2 * hidden_size * mlp_ratio / 3)
ffn_hidden = multiple_of * ((ffn_hidden + multiple_of - 1) // multiple_of)
# Expected shapes
expected = {
"x_embedder.proj.weight": (hidden_size, 16, 1, 2, 2),
"x_embedder.proj.bias": (hidden_size, ),
"t_embedder.mlp.0.weight": (adaln_dim, 256),
"t_embedder.mlp.2.weight": (adaln_dim, adaln_dim),
"y_embedder.y_proj.0.weight": (hidden_size, 4096),
"blocks.0.attn.qkv.weight": (3 * hidden_size, hidden_size),
"blocks.0.attn.q_norm.weight": (head_dim, ),
"blocks.0.attn.proj.weight": (hidden_size, hidden_size),
"blocks.0.cross_attn.q_linear.weight": (hidden_size, hidden_size),
"blocks.0.cross_attn.kv_linear.weight": (2 * hidden_size, hidden_size),
"blocks.0.ffn.w1.weight": (ffn_hidden, hidden_size),
"blocks.0.adaLN_modulation.1.weight": (6 * hidden_size, adaln_dim),
"final_layer.linear.weight": (64, hidden_size),
}
# Load and check
first_shard = os.path.join(dit_path, "diffusion_pytorch_model-00001-of-00006.safetensors")
all_valid = True
with safe_open(first_shard, framework="pt", device="cpu") as f:
for key, expected_shape in expected.items():
if key in f.keys():
tensor = f.get_tensor(key)
actual_shape = tuple(tensor.shape)
if actual_shape == expected_shape:
print(f"✓ {key}: {actual_shape}")
else:
print(f"✗ {key}: expected {expected_shape}, got {actual_shape}")
all_valid = False
return all_valid
def validate_model_index(model_path):
"""Validate model_index.json exists and is correct."""
print("\n" + "=" * 60)
print("VALIDATING MODEL INDEX")
print("=" * 60)
model_index_path = os.path.join(model_path, "model_index.json")
if not os.path.exists(model_index_path):
print("✗ model_index.json not found")
return False
with open(model_index_path) as f:
index = json.load(f)
required_keys = ["_class_name", "workload_type", "tokenizer", "text_encoder", "vae", "scheduler", "transformer"]
all_valid = True
for key in required_keys:
if key in index:
print(f"✓ {key}: {index[key]}")
else:
print(f"✗ {key}: missing")
all_valid = False
return all_valid
if __name__ == "__main__":
parser = argparse.ArgumentParser(description="Validate LongCat weights for FastVideo")
parser.add_argument("--model-path", type=str, required=True, help="Path to LongCat model directory")
parser.add_argument("--check-shapes", action="store_true", help="Also validate weight shapes (slower)")
args = parser.parse_args()
print(f"\nValidating: {args.model_path}\n")
# Run validations
components_valid = validate_components(args.model_path)
model_index_valid = validate_model_index(args.model_path)
dit_valid = validate_dit_weights(os.path.join(args.model_path, "transformer"))
if args.check_shapes:
shapes_valid = validate_shapes(os.path.join(args.model_path, "transformer"))
else:
shapes_valid = True
print("\nSkipping shape validation (use --check-shapes to enable)")
# Summary
print("\n" + "=" * 60)
print("VALIDATION SUMMARY")
print("=" * 60)
all_valid = components_valid and model_index_valid and dit_valid and shapes_valid
if all_valid:
print("✓ All validations passed!")
print("✓ Model ready for FastVideo")
else:
print("✗ Some validations failed")
print("✗ Please check errors above")
print("=" * 60)
scripts/inference/inference_longcat.yaml
-43
View File
@@ -1,43 +0,0 @@
# Run with:
# fastvideo generate --config scripts/inference/inference_longcat.yaml
generator:
model_path: FastVideo/LongCat-Video-T2V-Diffusers
engine:
num_gpus: 1
parallelism:
tp_size: 1
sp_size: 1
offload:
dit: false
text_encoder: false
vae: false
pin_cpu_memory: false
pipeline:
experimental:
enable_bsa: false
request:
prompt: >-
In a realistic photography style, a white boy around seven or eight years
old sits on a park bench, wearing a light blue T-shirt, denim shorts, and
white sneakers. He holds an ice cream cone with vanilla and chocolate
flavors, and beside him is a medium-sized golden Labrador. Smiling, the
boy offers the ice cream to the dog, who eagerly licks it with its tongue.
The sun is shining brightly, and the background features a green lawn and
several tall trees, creating a warm and loving scene.
negative_prompt: >-
Bright tones, overexposed, static, blurred details, subtitles, style,
works, paintings, images, static, overall gray, worst quality, low
quality, JPEG compression residue, ugly, incomplete, extra fingers,
poorly drawn hands, poorly drawn faces, deformed, disfigured, misshapen
limbs, fused fingers, still picture, messy background, three legs, many
people in the background, walking backwards
sampling:
seed: 42
num_frames: 93
height: 480
width: 832
fps: 15
num_inference_steps: 50
guidance_scale: 4.0
output:
output_path: outputs_video/longcat_t2v
scripts/inference/inference_longcat_distill.yaml
-44
View File
@@ -1,44 +0,0 @@
# Run with:
# fastvideo generate --config scripts/inference/inference_longcat_distill.yaml
generator:
model_path: FastVideo/LongCat-Video-T2V-Diffusers
engine:
num_gpus: 1
parallelism:
tp_size: 1
sp_size: 1
offload:
dit: false
pin_cpu_memory: false
pipeline:
components:
lora_path: FastVideo/LongCat-Video-T2V-Distilled-LoRA
experimental:
enable_bsa: false
lora_nickname: distilled
request:
prompt: >-
In a realistic photography style, a white boy around seven or eight years
old sits on a park bench, wearing a light blue T-shirt, denim shorts, and
white sneakers. He holds an ice cream cone with vanilla and chocolate
flavors, and beside him is a medium-sized golden Labrador. Smiling, the
boy offers the ice cream to the dog, who eagerly licks it with its tongue.
The sun is shining brightly, and the background features a green lawn and
several tall trees, creating a warm and loving scene.
negative_prompt: >-
Bright tones, overexposed, static, blurred details, subtitles, style,
works, paintings, images, static, overall gray, worst quality, low
quality, JPEG compression residue, ugly, incomplete, extra fingers,
poorly drawn hands, poorly drawn faces, deformed, disfigured, misshapen
limbs, fused fingers, still picture, messy background, three legs, many
people in the background, walking backwards
sampling:
seed: 42
num_frames: 93
height: 480
width: 832
fps: 15
num_inference_steps: 16
guidance_scale: 1.0
output:
output_path: outputs_video/longcat_distill
scripts/inference/inference_longcat_i2v.yaml
-42
View File
@@ -1,42 +0,0 @@
# Run with:
# fastvideo generate --config scripts/inference/inference_longcat_i2v.yaml
generator:
model_path: FastVideo/LongCat-Video-I2V-Diffusers
engine:
num_gpus: 1
parallelism:
tp_size: 1
sp_size: 1
offload:
dit: false
pin_cpu_memory: false
pipeline:
workload_type: i2v
experimental:
enable_bsa: false
request:
prompt: >-
A woman sits at a wooden table by the window in a cozy café. She reaches
out with her right hand, picks up the white coffee cup from the saucer,
and gently brings it to her lips to take a sip. After drinking, she places
the cup back on the table and looks out the window, enjoying the peaceful
atmosphere.
negative_prompt: >-
Bright tones, overexposed, static, blurred details, subtitles, style,
works, paintings, images, static, overall gray, worst quality, low
quality, JPEG compression residue, ugly, incomplete, extra fingers,
poorly drawn hands, poorly drawn faces, deformed, disfigured, misshapen
limbs, fused fingers, still picture, messy background, three legs, many
people in the background, walking backwards
inputs:
image_path: assets/girl.png
sampling:
seed: 42
num_frames: 93
height: 480
width: 480
fps: 15
num_inference_steps: 50
guidance_scale: 4.0
output:
output_path: outputs_video/longcat_i2v
scripts/inference/inference_longcat_refine_fromvideo.yaml
-51
View File
@@ -1,51 +0,0 @@
# Run with:
# fastvideo generate --config scripts/inference/inference_longcat_refine_fromvideo.yaml
generator:
model_path: FastVideo/LongCat-Video-T2V-Diffusers
engine:
num_gpus: 1
parallelism:
tp_size: 1
sp_size: 1
offload:
pin_cpu_memory: false
pipeline:
components:
lora_path: FastVideo/LongCat-Video-T2V-Refinement-LoRA
experimental:
enable_bsa: true
bsa_sparsity: 0.875
bsa_chunk_q: [4, 4, 8]
bsa_chunk_k: [4, 4, 8]
lora_nickname: refinement
request:
prompt: >-
In a realistic photography style, a white boy around seven or eight years
old sits on a park bench, wearing a light blue T-shirt, denim shorts, and
white sneakers. He holds an ice cream cone with vanilla and chocolate
flavors, and beside him is a medium-sized golden Labrador. Smiling, the
boy offers the ice cream to the dog, who eagerly licks it with its tongue.
The sun is shining brightly, and the background features a green lawn and
several tall trees, creating a warm and loving scene.
negative_prompt: >-
Bright tones, overexposed, static, blurred details, subtitles, style,
works, paintings, images, static, overall gray, worst quality, low
quality, JPEG compression residue, ugly, incomplete, extra fingers,
poorly drawn hands, poorly drawn faces, deformed, disfigured, misshapen
limbs, fused fingers, still picture, messy background, three legs, many
people in the background, walking backwards
inputs:
refine_from: outputs_video/longcat_distill/In a realistic photography style, a white boy around seven or eight years old sits on a park bench,.mp4
sampling:
seed: 42
height: 720
width: 1280
fps: 30
num_inference_steps: 50
guidance_scale: 1.0
extensions:
t_thresh: 0.5
spatial_refine_only: false
num_cond_frames: 0
output:
output_path: outputs_video/longcat_refine_720p
scripts/inference/inference_longcat_vc.yaml
-43
View File
@@ -1,43 +0,0 @@
# Run with:
# fastvideo generate --config scripts/inference/inference_longcat_vc.yaml
generator:
model_path: FastVideo/LongCat-Video-VC-Diffusers
engine:
num_gpus: 1
parallelism:
tp_size: 1
sp_size: 1
offload:
dit: false
pin_cpu_memory: false
pipeline:
experimental:
enable_bsa: false
request:
prompt: >-
A person rides a motorcycle along a long, straight road that stretches
between a body of water and a forested hillside. The rider steadily
accelerates, keeping the motorcycle centered between the guardrails, while
the scenery passes by on both sides. The video captures the journey from
the rider's perspective, emphasizing the sense of motion and adventure.
negative_prompt: >-
Bright tones, overexposed, static, blurred details, subtitles, style,
works, paintings, images, static, overall gray, worst quality, low
quality, JPEG compression residue, ugly, incomplete, extra fingers,
poorly drawn hands, poorly drawn faces, deformed, disfigured, misshapen
limbs, fused fingers, still picture, messy background, three legs, many
people in the background, walking backwards
inputs:
video_path: assets/motorcycle.mp4
sampling:
seed: 42
num_frames: 93
height: 480
width: 832
fps: 15
num_inference_steps: 50
guidance_scale: 4.0
extensions:
num_cond_frames: 13
output:
output_path: outputs_video/longcat_vc