hao-ai-lab-FastVideo/docs/utilities/lora.md

LoRA Extraction and Merging

Generic runtime adapter extraction plus the existing legacy merge utilities for FastVideo models.

Extract LoRA Adapter

python scripts/lora_extraction/extract_lora.py \
  --base Wan-AI/Wan2.2-TI2V-5B-Diffusers \
  --ft FastVideo/FastWan2.2-TI2V-5B-FullAttn-Diffusers \
  --out adapter_r32.safetensors \
  --rank 32 \
  --exact-tensor-pattern '^condition_embedder\.' \
  --exact-tensor-pattern '^proj_out\.weight$'

The extractor is runtime-agnostic and cannot determine from checkpoint tensors whether the target runtime wraps a given matrix as a LoRA layer. Use --exact-tensor-pattern for changed matrices that the runtime does not wrap; the extractor preserves them as exact .diff tensors instead of emitting factors that the runtime cannot apply. The Wan patterns above cover its excluded condition embedders and its unwrapped output projection.

Exact CPU SVD remains the default. For a large transformer, stream its indexed safetensors and factorize on a GPU:

python scripts/lora_extraction/extract_lora.py \
  --base <base-model-or-path> \
  --ft <finetuned-model-or-path> \
  --out adapter_r64.safetensors \
  --rank 64 \
  --load-mode indexed \
  --device cuda:0 \
  --svd-method randomized \
  --randomized-q 320 \
  --niter 4 \
  --factor-dtype float16 \
  --dense-dtype float32 \
  --replacement-dtype source

--load-mode indexed downloads only transformer/* for a Hugging Face model and reads one base/fine-tuned tensor pair at a time. The default auto mode tries indexed loading first and falls back to the legacy FastVideo pipeline loader; pipeline selects the legacy loader directly.

Important options:

• --base, --ft: Hugging Face model IDs or local paths.
• --rank, --full-rank: truncated or full factorization rank.
• --min-delta: omit tensors whose maximum absolute FP32 delta is at or below this threshold (default: 1e-8).
• --device: factorization device, such as cpu or cuda:0.
• --svd-method: exact or randomized.
• --randomized-q, --niter, --seed: randomized SVD accuracy and reproducibility.
• --factor-dtype, --dense-dtype, --replacement-dtype: adapter storage precision. Exact dense deltas default to float32.
• --exact-tensor-pattern: repeatable regex for a matrix that should remain an exact dense delta.
• --base-revision, --ft-revision: pin Hugging Face inputs in indexed mode; revisions are rejected for local paths and pipeline loading.
• --work-dir, --resume: resume an interrupted streaming extraction. Scratch is written to an output-specific namespace under fastvideo-lora-extract/, and only that namespace is cleaned up. Resume requires indexed safetensors and validates both checkpoints' index/shard fingerprints before reusing partial results.

The adapter retains changes that do not fit a low-rank product: .diff and .diff_b hold exact additive weight/bias deltas, .diff_param handles standalone parameters such as scale_shift_table, and .set_weight/.set_param hold parameters absent from the base checkpoint. Bit-identical parameters are omitted. The extractor writes an adjacent *.report.json with tensor counts, settings, and reconstruction residuals (factorized_weighted_relative_residual reports SVD truncation before --factor-dtype casting; factorized_weighted_reconstruction_relative_residual reports the shipped factors' fidelity including storage-precision loss).

For the validated MiniMax-H3 rank-64 command, including its exact-boundary patterns, see scripts/lora_extraction/README.md.

Mixed low-rank/dense adapters produced by the generic extractor must be supplied when constructing FastVideo through ComponentConfig(lora_path=...); their dense payload cannot be swapped later with set_lora_adapter. The legacy offline merger below retains its existing scope and is not part of this extraction workflow.

Legacy Merge Adapter

The command below documents the pre-existing merger for adapters it already supports. Do not pass a mixed adapter from the generic extractor to it: the legacy merger does not apply the adapter's exact dense or replacement payloads.

python scripts/lora_extraction/merge_lora.py \
  --base Wan-AI/Wan2.2-TI2V-5B-Diffusers \
  --adapter legacy_factor_only_adapter.safetensors \
  --ft FastVideo/FastWan2.2-TI2V-5B-FullAttn-Diffusers \
  --output merged_model

Options:

• --base: Base model (Hugging Face ID or local path)
• --adapter: LoRA adapter file (.safetensors)
• --ft: Fine-tuned model (for configuration)
• --output: Output directory

Validate Quality (Optional)

python scripts/lora_extraction/lora_inference_comparison.py \
  --base merged_model \
  --ft FastVideo/FastWan2.2-TI2V-5B-FullAttn-Diffusers \
  --adapter NONE \
  --output-dir results \
  --prompt "A cat sitting on a windowsill" \
  --seed 42 \
  --height 480 \
  --width 480 \
  --num-frames 49 \
  --num-inference-steps 32 \
  --compute-ssim \
  --compute-lpips

Options:

• --base: Merged model or base model path
• --ft: Fine-tuned model (reference)
• --adapter: Path to adapter or NONE
• --output-dir: Output directory
• --prompt: Text prompt (default: "A cat sitting on a windowsill")
• --seed: Random seed (default: 42)
• --height: Video height (default: 480)
• --width: Video width (default: 832)
• --num-frames: Number of frames (default: 49)
• --num-inference-steps: Inference steps (default: 32)
• --compute-ssim: Compute SSIM metric
• --compute-lpips: Compute LPIPS metric