SolitaryThinker
a5be058cf1
Wire up and verify the Cosmos3 image-to-video path. The denoise stage already handled the I2V conditioning math (encode -> condition mask -> keep frame 0 clean -> velocity zeroed on condition frames, all matching the framework), but the conditioning pixel clip was built wrong. Bug: `_image_to_video_tensor` zero-filled every frame after frame 0. The framework (`cosmos_framework.inference.vision.build_conditioned_video_batch`) instead fills frame 0 with the image and **repeats the last conditioning frame** for the rest of the clip (a static video) before VAE-encoding. Because the Wan VAE is temporal (4x), latent frame 0 (the kept-clean condition frame) depends on several pixel frames, so zero-filling produces a wrong conditioning latent. Fix `_image_to_video_tensor` to be framework-faithful: - aspect-preserving resize + center crop + uint8 quantization, then `/127.5 - 1` (mirrors `load_conditioning_image` / `_resize_and_center_crop`); - static-repeat fill across all frames (mirrors `build_conditioned_video_batch`). Parity: add `test_cosmos3_i2v_conditioning_parity.py` comparing the FastVideo conditioning clip against the framework's `load_conditioning_image` + repeat-fill across aspect/size/frame-count cases — bit-exact (max abs diff 0.0). Example: `examples/inference/basic/basic_cosmos3_i2v_new_api.py` (new VideoGenerator API, image via `InputConfig(image_path=...)`, `assets/images/cyclist.jpg` default). Verified on B200 (1280x704, 29f, 35 steps, real weights): frame 0 reproduces the conditioning image and the clip generates coherent forward motion following the prompt. Full cosmos3 suite: 98 passed, 0 skipped.
Basic Video Generation Tutorial
The VideoGenerator class provides the primary Python interface for doing offline video generation, which is interacting with a diffusion pipeline without using a separate inference api server.
Requirements
- At least a single NVIDIA GPU with CUDA 12.4.
- Python 3.10-3.12
Installation
If you have not installed FastVideo, please following these instructions first.
Usage
The first script in this example shows the most basic usage of FastVideo. If you are new to Python and FastVideo, you should start here.
# if you have not cloned the directory:
git clone https://github.com/hao-ai-lab/FastVideo.git && cd FastVideo
python examples/inference/basic/basic.py
For an example on Apple silicon:
python examples/inference/basic/basic_mps.py
For an example running DMD+VSA inference:
python examples/inference/basic/basic_dmd.py
For the typed config/request path added during the inference API refactor:
python examples/inference/basic/basic_dmd_new_api.py
Basic Walkthrough
All you need to generate videos using multi-gpus from state-of-the-art diffusion pipelines is the following few lines!
from fastvideo import VideoGenerator
def main():
generator = VideoGenerator.from_pretrained(
"Wan-AI/Wan2.1-T2V-1.3B-Diffusers",
num_gpus=1,
)
prompt = ("A curious raccoon peers through a vibrant field of yellow sunflowers, its eyes "
"wide with interest. The playful yet serene atmosphere is complemented by soft "
"natural light filtering through the petals. Mid-shot, warm and cheerful tones.")
video = generator.generate_video(prompt)
if __name__ == "__main__":
main()