SolitaryThinker
1ec268cf80
Wire text-to-video+sound end-to-end, completing the audio PR. - CFG velocity (`cosmos3_get_cfg_velocity`) + `Cosmos3DenoiseEngine` take an optional `sound_specs`: the denoise latent becomes a combined [vision | sound] flat vector; each step packs vision+sound, forwards the DiT, zeros the prediction on conditioning frames for both, and returns the concatenated velocity. The UniPC scheduler steps the joint vector. `Cosmos3SoundSpec` + `_split_sound_latent` added. - `Cosmos3DenoisingStage` t2vs path (gated on `COSMOS3_T2VS`): size the sound latent from the video duration (framework `create_placeholder_audio` + `get_latent_num_samples`), append sound noise, joint-denoise, split, AVAE-decode the sound latent, and set `batch.extra["audio"]` / `["audio_sample_rate"]` so the generator muxes a stereo 48 kHz AAC track into the mp4. Sound AVAE is lazy-loaded from `<model_path>/sound_tokenizer` (`_get_sound_vae`). - Parity (`test_cosmos3_sound_parity::test_t2vs_cfg_velocity_matches_framework`): the combined [vision|sound] sequential-CFG velocity vs a framework-DiT oracle — max abs diff = 0.0, mean abs diff = 0.0 across t2vs/i2vs cases. - Example `basic_cosmos3_t2vs_new_api.py`. Verified on B200 (real weights, 1280x704, 29f, 35 steps): coherent ocean/sunset video WITH a real stereo 48 kHz audio track (mean -10.2 dB, peak -0.4 dB; not silence). Full cosmos3 suite: 130 passed, 0 skipped.
Basic Video Generation Tutorial
The VideoGenerator class provides the primary Python interface for doing offline video generation, which is interacting with a diffusion pipeline without using a separate inference api server.
Requirements
- At least a single NVIDIA GPU with CUDA 12.4.
- Python 3.10-3.12
Installation
If you have not installed FastVideo, please following these instructions first.
Usage
The first script in this example shows the most basic usage of FastVideo. If you are new to Python and FastVideo, you should start here.
# if you have not cloned the directory:
git clone https://github.com/hao-ai-lab/FastVideo.git && cd FastVideo
python examples/inference/basic/basic.py
For an example on Apple silicon:
python examples/inference/basic/basic_mps.py
For an example running DMD+VSA inference:
python examples/inference/basic/basic_dmd.py
For the typed config/request path added during the inference API refactor:
python examples/inference/basic/basic_dmd_new_api.py
Basic Walkthrough
All you need to generate videos using multi-gpus from state-of-the-art diffusion pipelines is the following few lines!
from fastvideo import VideoGenerator
def main():
generator = VideoGenerator.from_pretrained(
"Wan-AI/Wan2.1-T2V-1.3B-Diffusers",
num_gpus=1,
)
prompt = ("A curious raccoon peers through a vibrant field of yellow sunflowers, its eyes "
"wide with interest. The playful yet serene atmosphere is complemented by soft "
"natural light filtering through the petals. Mid-shot, warm and cheerful tones.")
video = generator.generate_video(prompt)
if __name__ == "__main__":
main()