GEN3C: 3D-Informed Camera-Controlled Video Generation

GEN3C is NVIDIA's Cosmos-7B-based video model for camera-controlled generation from a single image. The FastVideo integration supports the GEN3C I2V workflow, including 3D cache conditioning and tokenizer-based conditioning latents.

Key Features

Camera trajectory control: left/right/up/down/zoom_in/zoom_out/clockwise/counterclockwise
3D cache conditioning: depth prediction -> point cloud cache -> forward warping -> latent conditioning
Single-image to video generation: 121-frame generation with camera motion
Official raw checkpoint conversion: model.pt -> Diffusers/FastVideo layout

Model Sources

Official raw checkpoint (not Diffusers): nvidia/GEN3C-Cosmos-7B
Diffusers-format checkpoint: FastVideo/GEN3C-Cosmos-7B-Diffusers

Prerequisites

Install MoGe:

uv pip install git+https://github.com/microsoft/MoGe.git

If you hit ImportError: libGL.so.1 (common on Ubuntu/headless nodes), you can try installing OpenCV runtime libs:

sudo apt-get update
sudo apt-get install -y libgl1 libglib2.0-0 libsm6 libxext6 libxrender1

Quick Start

Option A: Use Diffusers-format weights directly

python examples/inference/basic/basic_gen3c.py \
  --model_path FastVideo/GEN3C-Cosmos-7B-Diffusers \
  --image_path /path/to/input.png \
  --prompt "" \
  --trajectory left \
  --movement_distance 0.3 \
  --camera_rotation center_facing \
  --num_inference_steps 35 \
  --guidance_scale 1.0 \
  --output_path outputs_video/gen3c_output.mp4

Option B: Convert official raw checkpoint locally

Download:

huggingface-cli download nvidia/GEN3C-Cosmos-7B --local-dir official_weights/GEN3C-Cosmos-7B

Convert:

python scripts/checkpoint_conversion/convert_gen3c_to_fastvideo.py \
  --source official_weights/GEN3C-Cosmos-7B/model.pt \
  --output converted_weights/GEN3C-Cosmos-7B

Run:

python examples/inference/basic/basic_gen3c.py \
  --model_path converted_weights/GEN3C-Cosmos-7B \
  --image_path /path/to/input.png \
  --prompt "" \
  --trajectory left \
  --movement_distance 0.3 \
  --camera_rotation center_facing \
  --num_inference_steps 35 \
  --guidance_scale 1.0 \
  --output_path outputs_video/gen3c_output.mp4

FastVideo Defaults

GEN3C defaults in FastVideo:

height=704, width=1280
num_frames=121
num_inference_steps=35
guidance_scale=1.0
fps=24

These values are defined in:

fastvideo/pipelines/basic/gen3c/profiles.py
fastvideo/configs/pipelines/gen3c.py

and align with the official GEN3C inference defaults in:

tmp/GEN3C/cosmos_predict1/diffusion/inference/inference_utils.py

Scheduler Note

The converted GEN3C Diffusers layout may include a FlowMatch scheduler config, but GEN3C denoising uses EDM preconditioning behavior. FastVideo's GEN3C pipeline enforces an EDM scheduler at runtime for parity with official inference behavior.

Implementation path:

fastvideo/pipelines/basic/gen3c/gen3c_pipeline.py

3D Cache Conditioning Path

FastVideo GEN3C conditioning stage performs:

MoGe depth estimation from input image
3D cache initialization
Camera trajectory generation
Forward rendering of warped frames + masks
VAE/tokenizer encoding of conditioning buffers
Denoising with condition mask + condition pose channels

Main implementation:

fastvideo/pipelines/basic/gen3c/gen3c_pipeline.py
fastvideo/pipelines/basic/gen3c/cache_3d.py
fastvideo/pipelines/basic/gen3c/depth_estimation.py
fastvideo/models/vaes/gen3c_tokenizer_vae.py

3.8 KiB Raw Blame History