130 lines
3.8 KiB
Markdown
130 lines
3.8 KiB
Markdown
# GEN3C: 3D-Informed Camera-Controlled Video Generation
|
|
|
|
[GEN3C](https://arxiv.org/abs/2503.03751) is NVIDIA's Cosmos-7B-based video model for camera-controlled generation from a single image. The FastVideo integration supports the GEN3C I2V workflow, including 3D cache conditioning and tokenizer-based conditioning latents.
|
|
|
|
## Key Features
|
|
|
|
- **Camera trajectory control**: `left/right/up/down/zoom_in/zoom_out/clockwise/counterclockwise`
|
|
- **3D cache conditioning**: depth prediction -> point cloud cache -> forward warping -> latent conditioning
|
|
- **Single-image to video generation**: 121-frame generation with camera motion
|
|
- **Official raw checkpoint conversion**: `model.pt` -> Diffusers/FastVideo layout
|
|
|
|
## Model Sources
|
|
|
|
- Official raw checkpoint (not Diffusers): `nvidia/GEN3C-Cosmos-7B`
|
|
- Diffusers-format checkpoint: `FastVideo/GEN3C-Cosmos-7B-Diffusers`
|
|
|
|
## Prerequisites
|
|
|
|
- Install MoGe:
|
|
|
|
```bash
|
|
uv pip install git+https://github.com/microsoft/MoGe.git
|
|
```
|
|
|
|
- If you hit `ImportError: libGL.so.1` (common on Ubuntu/headless nodes), you can try installing OpenCV runtime libs:
|
|
|
|
```bash
|
|
sudo apt-get update
|
|
sudo apt-get install -y libgl1 libglib2.0-0 libsm6 libxext6 libxrender1
|
|
```
|
|
|
|
## Quick Start
|
|
|
|
### Option A: Use Diffusers-format weights directly
|
|
|
|
```bash
|
|
python examples/inference/basic/basic_gen3c.py \
|
|
--model_path FastVideo/GEN3C-Cosmos-7B-Diffusers \
|
|
--image_path /path/to/input.png \
|
|
--prompt "" \
|
|
--trajectory left \
|
|
--movement_distance 0.3 \
|
|
--camera_rotation center_facing \
|
|
--num_inference_steps 35 \
|
|
--guidance_scale 1.0 \
|
|
--output_path outputs_video/gen3c_output.mp4
|
|
```
|
|
|
|
### Option B: Convert official raw checkpoint locally
|
|
|
|
1. Download:
|
|
|
|
```bash
|
|
huggingface-cli download nvidia/GEN3C-Cosmos-7B --local-dir official_weights/GEN3C-Cosmos-7B
|
|
```
|
|
|
|
1. Convert:
|
|
|
|
```bash
|
|
python scripts/checkpoint_conversion/convert_gen3c_to_fastvideo.py \
|
|
--source official_weights/GEN3C-Cosmos-7B/model.pt \
|
|
--output converted_weights/GEN3C-Cosmos-7B
|
|
```
|
|
|
|
1. Run:
|
|
|
|
```bash
|
|
python examples/inference/basic/basic_gen3c.py \
|
|
--model_path converted_weights/GEN3C-Cosmos-7B \
|
|
--image_path /path/to/input.png \
|
|
--prompt "" \
|
|
--trajectory left \
|
|
--movement_distance 0.3 \
|
|
--camera_rotation center_facing \
|
|
--num_inference_steps 35 \
|
|
--guidance_scale 1.0 \
|
|
--output_path outputs_video/gen3c_output.mp4
|
|
```
|
|
|
|
## FastVideo Defaults
|
|
|
|
GEN3C defaults in FastVideo:
|
|
|
|
- `height=704`, `width=1280`
|
|
- `num_frames=121`
|
|
- `num_inference_steps=35`
|
|
- `guidance_scale=1.0`
|
|
- `fps=24`
|
|
|
|
These values are defined in:
|
|
|
|
- `fastvideo/pipelines/basic/gen3c/profiles.py`
|
|
- `fastvideo/configs/pipelines/gen3c.py`
|
|
|
|
and align with the official GEN3C inference defaults in:
|
|
|
|
- `tmp/GEN3C/cosmos_predict1/diffusion/inference/inference_utils.py`
|
|
|
|
## Scheduler Note
|
|
|
|
The converted GEN3C Diffusers layout may include a FlowMatch scheduler config, but GEN3C denoising uses EDM preconditioning behavior. FastVideo's GEN3C pipeline enforces an EDM scheduler at runtime for parity with official inference behavior.
|
|
|
|
Implementation path:
|
|
|
|
- `fastvideo/pipelines/basic/gen3c/gen3c_pipeline.py`
|
|
|
|
## 3D Cache Conditioning Path
|
|
|
|
FastVideo GEN3C conditioning stage performs:
|
|
|
|
1. MoGe depth estimation from input image
|
|
2. 3D cache initialization
|
|
3. Camera trajectory generation
|
|
4. Forward rendering of warped frames + masks
|
|
5. VAE/tokenizer encoding of conditioning buffers
|
|
6. Denoising with condition mask + condition pose channels
|
|
|
|
Main implementation:
|
|
|
|
- `fastvideo/pipelines/basic/gen3c/gen3c_pipeline.py`
|
|
- `fastvideo/pipelines/basic/gen3c/cache_3d.py`
|
|
- `fastvideo/pipelines/basic/gen3c/depth_estimation.py`
|
|
- `fastvideo/models/vaes/gen3c_tokenizer_vae.py`
|
|
|
|
## References
|
|
|
|
- [GEN3C Paper](https://arxiv.org/abs/2503.03751)
|
|
- [Official Repository](https://github.com/nv-tlabs/GEN3C)
|
|
- [Official Checkpoint (raw)](https://huggingface.co/nvidia/GEN3C-Cosmos-7B)
|