diff --git a/apps/dreamverse/arch.md b/apps/dreamverse/arch.md index eb532f17d..424dbfa9c 100644 --- a/apps/dreamverse/arch.md +++ b/apps/dreamverse/arch.md @@ -42,7 +42,7 @@ Near-term OSS note: - `apps/dreamverse/dreamverse/main.py`: websocket endpoint, request handling, session state machine, rewrite orchestration, REST routes, and stream relay - `apps/dreamverse/dreamverse/gpu_pool.py`: GPU worker processes, warmup, model - loading, and `generate_video()` calls through FastVideo + loading, and `generate()` calls through FastVideo - `apps/dreamverse/dreamverse/prompt_enhancer.py`: prompt enhancement, rollout rewrite execution, provider selection, and timeout/fallback behavior - `apps/dreamverse/dreamverse/rewrite_prompt_payload.py`: canonical rewrite request payload diff --git a/apps/dreamverse/gpu-pool.drawio b/apps/dreamverse/gpu-pool.drawio index 9a9d4d385..403c41dc0 100644 --- a/apps/dreamverse/gpu-pool.drawio +++ b/apps/dreamverse/gpu-pool.drawio @@ -207,7 +207,7 @@ - + @@ -534,7 +534,7 @@ - + diff --git a/apps/dreamverse/gpu-pool.svg b/apps/dreamverse/gpu-pool.svg index 125967086..2d300dbbf 100644 --- a/apps/dreamverse/gpu-pool.svg +++ b/apps/dreamverse/gpu-pool.svg @@ -1 +1 @@ -
gpu_pool.py — runtime architecture
gpu_pool.py — runtime architecture
Client (WebSocket)
Client (WebSocket)
GPUPool
GPUPool
GPUSlot (one per GPU)
GPUSlot (one per GPU)
start()
start()
shutdown()
shutdown()
join_user()
join_user()
user_step()
user_step()
leave_user()
leave_user()
_send_command()

puts on command_queue,
blocks on response_queue.get

**for user agonistic commands
_send_command()...
_send_command_tagged()

creates Future; registers in
_pending_futures[user_id];
ensures response reader; awaits future

**for user specific commands
_send_command_tagged()...
_response_reader()

a while loop that reads response_queue;
dispatch WorkerEvent:
• media → _stream_queues
• acks/errors → _pending_futures
_response_reader()...
«dict» _pending_futures

{user_id → asyncio.Future}

created by _send_command_tagged;
resolved + popped by _response_reader;
cleared on model reload or leave
«dict» _pending_futures...
«dict» _stream_queues: 
each is a queue of metadata that tell the consumer which part of the shared 
stream buffer to read next.

{user_id → asyncio.Queue}

created by register_stream_queue()
written by _response_reader (MediaInit / Chunk / Complete);
consumed by  AV loop → ws.send_bytes / send_json;
popped by leave_user(); cleared on reload
«dict» _stream_queues:...
── process boundary (mp.Queue + mp.RawArray) ──
── process boundary (mp.Queue + mp.RawArray) ──
command_queue
command_queue
response_queue
response_queue
shared_stream_buffer
(mp.RawArray, 256 MiB)

zero-copy mp4 bytes
shared_stream_buffer...
worker_ipc.py — shared types (imported on both sides of the process boundary)

Command(type: CommandType, payload: CommandPayload | None, user_id: str | None)
CommandPayload = UserStepPayload | WarmupPayload | ReloadModelPayload
WorkerEvent = StepComplete | WorkerError | JoinAck | LeaveAck | ReloadAck | WarmupComplete | MediaInit | MediaChunk | MediaComplete | InitAck | ShutdownAck
worker_ipc.py — shared types (imported on both sides of the process boundary)...
Worker Subprocess - in each GPUSlot
Worker Subprocess - in each GPUSlot
command dispatcher

gpu_worker_process() branches on
CommandType; asserts payload type

INIT / WARMUP / RELOAD_MODEL
USER_JOIN / USER_STEP / USER_LEAVE
SHUTDOWN
command dispatcher...
VideoGenerationWorker.generate_step()
ltx2_generation.py:380

reads + updates ContinuationState,
calls generator
VideoGenerationWorker.generate_step()...
stream_fmp4()
av_streaming.py:121

trims overlap, pipes to ffmpeg,
publishes StreamInit / StreamChunk /
StreamComplete via callback
stream_fmp4()...
VideoGenerator (fastvideo)

LTX2 DiT + refine upsampler
FP4 quant, torch.compile

owned by VideoGenerationWorker
ltx2_generation.py:211
VideoGenerator (fastvideo)...
ffmpeg subprocess

libx264 / *_nvenc
fragmented mp4
ffmpeg subprocess...
ContinuationState
ltx2_generation.py:89

• video_images: list[PIL.Image]
• audio_latents: torch.Tensor (CPU)

carried across segments
ContinuationState...
acquire
acquire
assigns slot
assigns slot
register future
register future
set future result
set future result
put(MediaEvent)
put(MediaEvent)
put Command
put Command
put Command
put Command
WorkerEvent
WorkerEvent
blocking .get
blocking .get
USER_STEP
USER_STEP
StepComplete / ack types
StepComplete / ack types
generator.generate_video()
generator.generate_video()
read / write
read / write
frames + audio
frames + audio
rawvideo + wav
rawvideo + wav
write mp4 bytes
at shared_write_offset
write mp4 bytes...
bytes from the buffer are sent through websocket 
based on info from the stream queue
bytes from the buffer are sent through websocket...
session AV loop (main.py:2124):
ws.send_bytes(chunk) — binary mp4 frames
ws.send_json({type: media_init | media_segment_complete | ...})
session AV loop (main.py:2124):...
MediaChunk (offset, len)
MediaChunk (offset, len)
Legend

■ blue client / external
■ green main-process pool/slot
(methods — italic label)
■ yellow containers (routing state)
■ red IPC primitives (mp.Queue, mp.RawArray)

Worker subprocess modules:
■ orange gpu_pool.py (dispatcher)
■ lavender ltx2_generation.py
■ teal av_streaming.py
■ gray worker_ipc.py (shared types)

Flow:
client → pool → slot
→ _send_command(_tagged) → command_queue
→ dispatcher → generate_step()
→ stream_fmp4() → ffmpeg
→ shared_buf + response_queue
→ _response_reader → futures / stream_queues
→ client awaits (via main.py AV loop)
Legend...
FastVideo video_generator
FastVideo video_genera...
reload model
reload model
Queue: owned by GPUSlot, passed to worker process as argument.
Queue: owned by GPUSlot, passed to w...Text is not SVG - cannot display \ No newline at end of file +
gpu_pool.py — runtime architecture
gpu_pool.py — runtime architecture
Client (WebSocket)
Client (WebSocket)
GPUPool
GPUPool
GPUSlot (one per GPU)
GPUSlot (one per GPU)
start()
start()
shutdown()
shutdown()
join_user()
join_user()
user_step()
user_step()
leave_user()
leave_user()
_send_command()

puts on command_queue,
blocks on response_queue.get

**for user agonistic commands
_send_command()...
_send_command_tagged()

creates Future; registers in
_pending_futures[user_id];
ensures response reader; awaits future

**for user specific commands
_send_command_tagged()...
_response_reader()

a while loop that reads response_queue;
dispatch WorkerEvent:
• media → _stream_queues
• acks/errors → _pending_futures
_response_reader()...
«dict» _pending_futures

{user_id → asyncio.Future}

created by _send_command_tagged;
resolved + popped by _response_reader;
cleared on model reload or leave
«dict» _pending_futures...
«dict» _stream_queues: 
each is a queue of metadata that tell the consumer which part of the shared 
stream buffer to read next.

{user_id → asyncio.Queue}

created by register_stream_queue()
written by _response_reader (MediaInit / Chunk / Complete);
consumed by  AV loop → ws.send_bytes / send_json;
popped by leave_user(); cleared on reload
«dict» _stream_queues:...
── process boundary (mp.Queue + mp.RawArray) ──
── process boundary (mp.Queue + mp.RawArray) ──
command_queue
command_queue
response_queue
response_queue
shared_stream_buffer
(mp.RawArray, 256 MiB)

zero-copy mp4 bytes
shared_stream_buffer...
worker_ipc.py — shared types (imported on both sides of the process boundary)

Command(type: CommandType, payload: CommandPayload | None, user_id: str | None)
CommandPayload = UserStepPayload | WarmupPayload | ReloadModelPayload
WorkerEvent = StepComplete | WorkerError | JoinAck | LeaveAck | ReloadAck | WarmupComplete | MediaInit | MediaChunk | MediaComplete | InitAck | ShutdownAck
worker_ipc.py — shared types (imported on both sides of the process boundary)...
Worker Subprocess - in each GPUSlot
Worker Subprocess - in each GPUSlot
command dispatcher

gpu_worker_process() branches on
CommandType; asserts payload type

INIT / WARMUP / RELOAD_MODEL
USER_JOIN / USER_STEP / USER_LEAVE
SHUTDOWN
command dispatcher...
VideoGenerationWorker.generate_step()
ltx2_generation.py:380

reads + updates ContinuationState,
calls generator
VideoGenerationWorker.generate_step()...
stream_fmp4()
av_streaming.py:121

trims overlap, pipes to ffmpeg,
publishes StreamInit / StreamChunk /
StreamComplete via callback
stream_fmp4()...
VideoGenerator (fastvideo)

LTX2 DiT + refine upsampler
FP4 quant, torch.compile

owned by VideoGenerationWorker
ltx2_generation.py:211
VideoGenerator (fastvideo)...
ffmpeg subprocess

libx264 / *_nvenc
fragmented mp4
ffmpeg subprocess...
ContinuationState
ltx2_generation.py:89

• video_images: list[PIL.Image]
• audio_latents: torch.Tensor (CPU)

carried across segments
ContinuationState...
acquire
acquire
assigns slot
assigns slot
register future
register future
set future result
set future result
put(MediaEvent)
put(MediaEvent)
put Command
put Command
put Command
put Command
WorkerEvent
WorkerEvent
blocking .get
blocking .get
USER_STEP
USER_STEP
StepComplete / ack types
StepComplete / ack types
generator.generate()
generator.generate()
read / write
read / write
frames + audio
frames + audio
rawvideo + wav
rawvideo + wav
write mp4 bytes
at shared_write_offset
write mp4 bytes...
bytes from the buffer are sent through websocket 
based on info from the stream queue
bytes from the buffer are sent through websocket...
session AV loop (main.py:2124):
ws.send_bytes(chunk) — binary mp4 frames
ws.send_json({type: media_init | media_segment_complete | ...})
session AV loop (main.py:2124):...
MediaChunk (offset, len)
MediaChunk (offset, len)
Legend

■ blue client / external
■ green main-process pool/slot
(methods — italic label)
■ yellow containers (routing state)
■ red IPC primitives (mp.Queue, mp.RawArray)

Worker subprocess modules:
■ orange gpu_pool.py (dispatcher)
■ lavender ltx2_generation.py
■ teal av_streaming.py
■ gray worker_ipc.py (shared types)

Flow:
client → pool → slot
→ _send_command(_tagged) → command_queue
→ dispatcher → generate_step()
→ stream_fmp4() → ffmpeg
→ shared_buf + response_queue
→ _response_reader → futures / stream_queues
→ client awaits (via main.py AV loop)
Legend...
FastVideo video_generator
FastVideo video_genera...
reload model
reload model
Queue: owned by GPUSlot, passed to worker process as argument.
Queue: owned by GPUSlot, passed to w...Text is not SVG - cannot display \ No newline at end of file diff --git a/apps/dreamverse/web/e2e/preset-prompt-generation.spec.ts b/apps/dreamverse/web/e2e/preset-prompt-generation.spec.ts index ba2231633..37571b33f 100644 --- a/apps/dreamverse/web/e2e/preset-prompt-generation.spec.ts +++ b/apps/dreamverse/web/e2e/preset-prompt-generation.spec.ts @@ -67,7 +67,7 @@ test.describe('preset prompt generation', () => { // on a B200 plus encode/transfer time. The "Continuation flipped // to Generating + Leave button rendered" pair above is the proof // the integration works: FE → /readyz → /curated-presets → WS - // /ws → BE → GPU pool → VideoGenerator.generate_video, all green. + // /ws → BE → GPU pool → VideoGenerator.generate, all green. const video = page.locator('video').first(); await expect(video).toHaveCount(1); }); diff --git a/docs/contributing/performance_benchmarks.md b/docs/contributing/performance_benchmarks.md index b2405eb42..e1115e5f5 100644 --- a/docs/contributing/performance_benchmarks.md +++ b/docs/contributing/performance_benchmarks.md @@ -112,7 +112,7 @@ per-metric policy with direction, percent threshold, absolute threshold, and a `test_inference_performance.py` temporarily sets `FASTVIDEO_STAGE_LOGGING=1` while it runs so pipeline stage execution times are available in -`generate_video(...).logging_info`. Stage logs use pipeline-unique keys such as +`generate(...).logging_info`. Stage logs use pipeline-unique keys such as `prompt_encoding_stage` so duplicate stage classes do not collide. For `PipelineStage` entries, shared component stage bases emit a stable `component_metric`: text encoding stages map to `text_encoder_time_s`, diff --git a/examples/inference/optimizations/qad_fp4_ab.py b/examples/inference/optimizations/qad_fp4_ab.py index a28406242..318c88c84 100644 --- a/examples/inference/optimizations/qad_fp4_ab.py +++ b/examples/inference/optimizations/qad_fp4_ab.py @@ -192,9 +192,7 @@ def main() -> None: last = _generate() torch.cuda.synchronize() wall = time.perf_counter() - t0 - # generate_video returns a plain dict; attribute access would always - # fall back to wall time. - gen_t = last.get("generation_time") if isinstance(last, dict) else None + gen_t = last.generation_time denoise_times.append(gen_t if gen_t is not None else wall) print(f"[qad] {tag} run {i + 1}/{runs}: {wall:.2f}s wall " f"(gen {denoise_times[-1]:.2f}s)") diff --git a/examples/inference/optimizations/torch_compile_example.py b/examples/inference/optimizations/torch_compile_example.py index bcf47f205..f9c39ce03 100644 --- a/examples/inference/optimizations/torch_compile_example.py +++ b/examples/inference/optimizations/torch_compile_example.py @@ -47,8 +47,8 @@ def main() -> None: def _run(tag: str) -> float: save = tag == "measured" - # Modern typed-request API (generate_video is deprecated). Same - # prompt/seed/shapes both runs so the compiled graph is reused. + # Typed request API. Same prompt/seed/shapes both runs so the + # compiled graph is reused. request: dict = { "prompt": PROMPT, "sampling": { diff --git a/fastvideo/api/compat.py b/fastvideo/api/compat.py index 0cef35295..876334f03 100644 --- a/fastvideo/api/compat.py +++ b/fastvideo/api/compat.py @@ -23,10 +23,6 @@ from fastvideo.api.schema import ( ContinuationState, GenerationRequest, GeneratorConfig, - InputConfig, - OutputConfig, - RequestRuntimeConfig, - SamplingConfig, ) from fastvideo.api.sampling_param import SamplingParam from fastvideo.fastvideo_args import FastVideoArgs @@ -34,16 +30,8 @@ from fastvideo.pipelines.basic.ltx2.stage_overrides import ( refine_preset_override_fields, refine_stage_override_fields, ) -from fastvideo.utils import shallow_asdict -_INPUT_FIELD_NAMES = {field.name for field in fields(InputConfig)} -_SAMPLING_FIELD_NAMES = {field.name for field in fields(SamplingConfig)} -_RUNTIME_FIELD_NAMES = {field.name for field in fields(RequestRuntimeConfig)} -_OUTPUT_FIELD_NAMES = {field.name for field in fields(OutputConfig)} _MISSING = object() -_LEGACY_REQUEST_ALIASES = { - "neg_prompt": "negative_prompt", -} REQUEST_BATCH_EXTRA_PASSTHROUGH_FIELDS = ( "ltx2_audio_latents", "ltx2_audio_clean_latent", @@ -198,34 +186,6 @@ def normalize_generation_request(request: GenerationRequest | Mapping[str, Any], return normalized -def legacy_generate_call_to_request( - prompt: str | None, - sampling_param: SamplingParam | None, - *, - mouse_cond: Any | None = None, - keyboard_cond: Any | None = None, - grid_sizes: Any | None = None, - legacy_kwargs: Mapping[str, Any] | None = None, -) -> GenerationRequest: - raw = _sampling_param_to_request_raw(sampling_param) - if prompt is not None: - raw["prompt"] = prompt - - for key, value in (legacy_kwargs or {}).items(): - _apply_request_field(raw, key, value) - - if mouse_cond is not None: - raw.setdefault("inputs", {})["mouse_cond"] = mouse_cond - if keyboard_cond is not None: - raw.setdefault("inputs", {})["keyboard_cond"] = keyboard_cond - if grid_sizes is not None: - raw.setdefault("inputs", {})["grid_sizes"] = grid_sizes - - normalized = parse_config(GenerationRequest, raw) - bind_generation_request_raw(normalized, raw) - return normalized - - def request_to_sampling_param( request: GenerationRequest, *, @@ -354,42 +314,6 @@ def _compile_config_to_torch_kwargs(compile_config: CompileConfig, ) -> dict[str return out -def _sampling_param_to_request_raw(sampling_param: SamplingParam | None, ) -> dict[str, Any]: - if sampling_param is None: - return {} - - raw: dict[str, Any] = {} - for key, value in shallow_asdict(sampling_param).items(): - if key == "prompt": - continue - _apply_request_field(raw, key, deepcopy(value)) - return raw - - -def _apply_request_field( - raw: dict[str, Any], - key: str, - value: Any, -) -> None: - key = _LEGACY_REQUEST_ALIASES.get(key, key) - if key == "negative_prompt": - raw["negative_prompt"] = value - return - if key in _INPUT_FIELD_NAMES: - raw.setdefault("inputs", {})[key] = value - return - if key in _SAMPLING_FIELD_NAMES: - raw.setdefault("sampling", {})[key] = value - return - if key in _RUNTIME_FIELD_NAMES: - raw.setdefault("runtime", {})[key] = value - return - if key in _OUTPUT_FIELD_NAMES: - raw.setdefault("output", {})[key] = value - return - raw.setdefault("extensions", {})[key] = value - - def request_to_batch_extra(request: GenerationRequest) -> dict[str, Any]: """Extract typed-request extensions consumed through ``ForwardBatch.extra``.""" return { @@ -587,7 +511,6 @@ __all__ = [ "explicit_request_updates", "generator_config_to_fastvideo_args", "legacy_from_pretrained_to_config", - "legacy_generate_call_to_request", "load_generator_config_from_file", "normalize_generation_request", "normalize_generator_config", diff --git a/fastvideo/api/sampling_param.py b/fastvideo/api/sampling_param.py index e420a13c0..c96239a94 100644 --- a/fastvideo/api/sampling_param.py +++ b/fastvideo/api/sampling_param.py @@ -140,7 +140,7 @@ class SamplingParam: ltx2_stg_blocks_audio: list[int] = field(default_factory=lambda: [29]) # LTX-2 image / video / continuation conditioning. These flow from - # generate_video(...) kwargs through ``sampling_param.update(kwargs)`` + # request ``extensions`` through ``sampling_param.update(...)`` # onto the ForwardBatch fields of the same name. ``ltx2_image_crf`` # gates the conditioning-image H.264 re-encode; the streaming # session controller passes ``ltx2_image_crf=0.0`` because it diff --git a/fastvideo/attention/backends/video_sparse_attn_h3.py b/fastvideo/attention/backends/video_sparse_attn_h3.py index a6f195ece..ad78a2135 100644 --- a/fastvideo/attention/backends/video_sparse_attn_h3.py +++ b/fastvideo/attention/backends/video_sparse_attn_h3.py @@ -19,7 +19,7 @@ backend differs from the Wan-tuned ``video_sparse_attn``: - Non-video *queries* are always dense. Non-video *keys* are either always-selected for every query ("exempt", default) or compete in top-k under a FLOP-matched budget ("compete") — the ablation axis, - switched per request via ``generate_video(..., vsa_mode=...)`` + switched per request via the request extension ``vsa_mode`` (default: exempt). Per-request scheduling knobs (``vsa_dense_first_n_steps``, ``vsa_dense_layers``) let mixed schedules run the diffuse steps/layers dense while pushing the rest harder. diff --git a/fastvideo/entrypoints/video_generator.py b/fastvideo/entrypoints/video_generator.py index 86126cd45..3ff211331 100644 --- a/fastvideo/entrypoints/video_generator.py +++ b/fastvideo/entrypoints/video_generator.py @@ -32,7 +32,6 @@ from fastvideo.api.compat import ( expand_request_prompt_batch, generator_config_to_fastvideo_args, legacy_from_pretrained_to_config, - legacy_generate_call_to_request, load_generator_config_from_file, normalize_generation_request, normalize_generator_config, @@ -401,80 +400,6 @@ class VideoGenerator: output=OutputConfig(save_video=False, return_frames=False), ) - def generate_video( - self, - prompt: str | None = None, - sampling_param: SamplingParam | None = None, - # Action control inputs (Matrix-Game) - mouse_cond: torch.Tensor | None = None, - keyboard_cond: torch.Tensor | None = None, - grid_sizes: tuple[int, int, int] | list[int] | torch.Tensor - | None = None, - **kwargs, - ) -> dict[str, Any] | list[dict[str, Any]]: - """ - Generate a video based on the given prompt. - - Args: - prompt: The prompt to use for generation (optional if prompt_txt is provided) - negative_prompt: The negative prompt to use (overrides the one in fastvideo_args) - output_path: Path to save the video (overrides the one in fastvideo_args) - prompt_path: Path to prompt file - save_video: Whether to save the video to disk - return_frames: Whether to include raw frames in the result dict - num_inference_steps: Number of denoising steps (overrides fastvideo_args) - guidance_scale: Classifier-free guidance scale (overrides fastvideo_args) - num_frames: Number of frames to generate (overrides fastvideo_args) - height: Height of generated video (overrides fastvideo_args) - width: Width of generated video (overrides fastvideo_args) - fps: Frames per second for saved video (overrides fastvideo_args) - seed: Random seed for generation (overrides fastvideo_args) - callback: Callback function called after each step - callback_steps: Number of steps between each callback - - Returns: - A metadata dictionary for single-prompt generation, or a list of - metadata dictionaries for prompt-file batch generation. - """ - log_queue = kwargs.pop("log_queue", None) - warnings.warn( - "VideoGenerator.generate_video(...) is deprecated; use " - "VideoGenerator.generate(request=...) instead.", - DeprecationWarning, - stacklevel=2, - ) - if log_queue: - self.executor.set_log_queue(log_queue) - - try: - extra_overrides: dict[str, Any] = {} - for _ek in _BATCH_EXTRA_PASSTHROUGH_KEYS: - if _ek in kwargs: - extra_overrides[_ek] = kwargs.pop(_ek) - - request = legacy_generate_call_to_request( - prompt, - sampling_param, - mouse_cond=mouse_cond, - keyboard_cond=keyboard_cond, - grid_sizes=grid_sizes, - legacy_kwargs=kwargs, - ) - - resolved_sampling_param = request_to_sampling_param( - request, - model_path=self.fastvideo_args.model_path, - ) - return self._generate_video_impl( - prompt=request.prompt, - sampling_param=resolved_sampling_param, - fastvideo_args=self.fastvideo_args, - **extra_overrides, - ) - finally: - if log_queue: - self.executor.clear_log_queue() - def _generate_request_impl( self, request: GenerationRequest, @@ -530,7 +455,7 @@ class VideoGenerator: fastvideo_args: FastVideoArgs | None = None, **kwargs, ) -> dict[str, Any] | list[np.ndarray] | list[dict[str, Any]]: - """Internal implementation of generate_video.""" + """Run one prompt, or each prompt of a prompt file, through the pipeline with ``sampling_param``.""" if fastvideo_args is None: fastvideo_args = self.fastvideo_args diff --git a/fastvideo/eval/datasets/base.py b/fastvideo/eval/datasets/base.py index 43f6cf3f3..3bf6cacc9 100644 --- a/fastvideo/eval/datasets/base.py +++ b/fastvideo/eval/datasets/base.py @@ -3,7 +3,7 @@ A :class:`PromptDataset` is an iterable of *sample dicts* describing the prompts and conditions for a benchmark. Each sample is a plain dict — no dataclass, no schema enforcement — that flows directly into both -generation (``VideoGenerator.generate_video(**sample)``) and scoring +generation (a ``VideoGenerator.generate`` request) and scoring (``Evaluator.evaluate(**eval_kwargs)``). The runner picks well-known keys (``prompt``, ``n_samples``, ``dimensions``, ``auxiliary_info``, ...) and passes the rest through. @@ -11,7 +11,6 @@ keys (``prompt``, ``n_samples``, ``dimensions``, ``auxiliary_info``, This matches the surrounding FastVideo style: * :class:`fastvideo.dataset.validation_dataset.ValidationDataset` yields dicts. -* :meth:`fastvideo.VideoGenerator.generate_video` consumes ``**kwargs``. * :meth:`fastvideo.eval.Evaluator.evaluate` consumes ``**kwargs``. To add a new benchmark: diff --git a/fastvideo/pipelines/basic/minimax_h3/stages/minimax_h3_denoising.py b/fastvideo/pipelines/basic/minimax_h3/stages/minimax_h3_denoising.py index bce4ea978..0810638a8 100644 --- a/fastvideo/pipelines/basic/minimax_h3/stages/minimax_h3_denoising.py +++ b/fastvideo/pipelines/basic/minimax_h3/stages/minimax_h3_denoising.py @@ -154,7 +154,7 @@ class MiniMaxH3DenoisingStage(PipelineStage): if vsa_metadata_builder is not None: vsa_patch_size = fastvideo_args.pipeline_config.dit_config.patch_size vsa_prefix_segments = _h3_vsa_prefix_segments(layout, vsa_patch_size) - # Per-request knobs (sweeps flip these between generate_video calls + # Per-request knobs (sweeps flip these between generate calls # without respawning workers); mode None defers to the env default. vsa_mode = batch.extra.get("vsa_mode", "exempt") if vsa_mode not in ("exempt", "compete"): diff --git a/fastvideo/pipelines/basic/stable_audio/stable_audio_pipeline.py b/fastvideo/pipelines/basic/stable_audio/stable_audio_pipeline.py index 7cb319c5c..4842785c6 100644 --- a/fastvideo/pipelines/basic/stable_audio/stable_audio_pipeline.py +++ b/fastvideo/pipelines/basic/stable_audio/stable_audio_pipeline.py @@ -62,7 +62,7 @@ def _disable_tf32_for_stable_audio() -> None: class StableAudioPipeline(ComposedPipelineBase): """Stable Audio Open 1.0 pipeline. - Mode is kwargs-driven on `generate_video()`: + Mode is driven by the request `extensions` passed to `generate()`: * Text-to-audio (default) -- `prompt=...`, `audio_end_in_s=...` * Audio-to-audio variation -- add `init_audio=ref` (and optionally diff --git a/fastvideo/pipelines/basic/stable_audio/stages/decoding.py b/fastvideo/pipelines/basic/stable_audio/stages/decoding.py index a18b15a3a..adbefa070 100644 --- a/fastvideo/pipelines/basic/stable_audio/stages/decoding.py +++ b/fastvideo/pipelines/basic/stable_audio/stages/decoding.py @@ -63,7 +63,7 @@ class StableAudioDecodingStage(PipelineStage): # Raw tensor for parity tests. batch.extra["decoded_audio"] = decoded.detach().cpu() - # `VideoGenerator.generate_video` is video-shaped (asserts + # `VideoGenerator.generate` is video-shaped (asserts # `output_batch.output is not None`); fill with a placeholder of # the expected `[B, 3, num_frames, H, W]` shape — the real audio # is on `batch.extra` above. Keep this compatibility shim until diff --git a/fastvideo/tests/api/test_extra_overrides_routing.py b/fastvideo/tests/api/test_extra_overrides_routing.py index 82da48bff..5a9cf41d9 100644 --- a/fastvideo/tests/api/test_extra_overrides_routing.py +++ b/fastvideo/tests/api/test_extra_overrides_routing.py @@ -13,8 +13,8 @@ invocation closed stdin before the writer thread had pushed every frame, surfacing as ``BrokenPipeError`` in the streaming server. These tests pin two contracts: - 1. ``_BATCH_EXTRA_PASSTHROUGH_KEYS`` lists the exact set of kwargs - pulled out of ``generate_video(**kwargs)`` for ``batch.extra``. + 1. ``_BATCH_EXTRA_PASSTHROUGH_KEYS`` lists the exact set of request + extensions that go to ``batch.extra``. 2. ``SamplingParam.update()`` raises ``ValueError`` on unknown keys. """ from __future__ import annotations diff --git a/fastvideo/tests/contract/test_dreamverse_shape.py b/fastvideo/tests/contract/test_dreamverse_shape.py index 9e909d6c4..7df69f362 100644 --- a/fastvideo/tests/contract/test_dreamverse_shape.py +++ b/fastvideo/tests/contract/test_dreamverse_shape.py @@ -13,10 +13,7 @@ This test makes sure: 1. The public typed API can represent everything Dreamverse currently passes at init time (``legacy_from_pretrained_to_config``). -2. The request-path Dreamverse uses (``generator.generate_video(**kwargs)`` - with per-segment flags) round-trips through the typed ``GenerationRequest`` - without reintroducing private-only fields at the public boundary. -3. Private-only Dreamverse fields that don't belong on the public +2. Private-only Dreamverse fields that don't belong on the public surface either go to ``pipeline.experimental`` / ``request.extensions`` (the documented escape hatch) or raise explicitly, rather than silently becoming part of the public compatibility promise. @@ -36,7 +33,6 @@ from fastvideo.api import ( ) from fastvideo.api.compat import ( legacy_from_pretrained_to_config, - legacy_generate_call_to_request, normalize_generation_request, ) @@ -151,35 +147,6 @@ class TestDreamverseRequestShape: """The per-segment Dreamverse request path mirrors OpenAI's shape plus a few LTX-2 knobs. All of them must have a typed home.""" - def test_basic_request_fields_round_trip(self): - # The request path calls legacy_generate_call_to_request with a - # prompt + legacy kwargs; verify the typed shape carries them. - legacy_kwargs = { - "num_frames": 121, - "height": 1024, - "width": 1536, - "num_inference_steps": 8, - "guidance_scale": 1.0, - "seed": 42, - "fps": 24, - "negative_prompt": "blurry", - } - request = legacy_generate_call_to_request( - prompt="a fox running", - sampling_param=None, - legacy_kwargs=legacy_kwargs, - ) - request = normalize_generation_request(request) - assert request.prompt == "a fox running" - assert request.negative_prompt == "blurry" - assert request.sampling.num_frames == 121 - assert request.sampling.height == 1024 - assert request.sampling.width == 1536 - assert request.sampling.num_inference_steps == 8 - assert request.sampling.guidance_scale == 1.0 - assert request.sampling.seed == 42 - assert request.sampling.fps == 24 - def test_return_state_reaches_output_config(self): """PR 7 added ``output.return_state`` — must survive the legacy translation path so Dreamverse callers can opt in.""" diff --git a/fastvideo/tests/entrypoints/test_video_generator.py b/fastvideo/tests/entrypoints/test_video_generator.py index 35d2e5c28..a3aaf9f21 100644 --- a/fastvideo/tests/entrypoints/test_video_generator.py +++ b/fastvideo/tests/entrypoints/test_video_generator.py @@ -897,79 +897,6 @@ def test_generate_removes_deleted_loaded_stage_overrides(monkeypatch, tmp_path): assert captured["sampling_param"].t_thresh == 0.5 -def test_generate_video_legacy_call_uses_legacy_impl(monkeypatch): - generator = _new_runtime_video_generator() - captured = {} - - def fake_generate_video_impl( - prompt=None, - sampling_param=None, - mouse_cond=None, - keyboard_cond=None, - grid_sizes=None, - **kwargs, - ): - captured["prompt"] = prompt - captured["sampling_param"] = sampling_param - captured["mouse_cond"] = mouse_cond - captured["keyboard_cond"] = keyboard_cond - captured["grid_sizes"] = grid_sizes - captured["kwargs"] = kwargs - return {"prompts": prompt, "video_path": "outputs/test.mp4"} - - monkeypatch.setattr(generator, "_generate_video_impl", fake_generate_video_impl) - - with pytest.warns(DeprecationWarning): - result = generator.generate_video( - prompt="legacy prompt", - num_frames=49, - output_path="outputs/legacy", - save_video=False, - log_queue="queue-token", - ) - - assert captured["prompt"] == "legacy prompt" - assert captured["sampling_param"].num_frames == 49 - assert captured["sampling_param"].output_path == "outputs/legacy" - assert captured["sampling_param"].save_video is False - assert result["video_path"] == "outputs/test.mp4" - - -def test_generate_video_legacy_call_routes_compat_kwargs(monkeypatch): - generator = _new_runtime_video_generator() - generator.fastvideo_args.pipeline_config = SimpleNamespace(embedded_cfg_scale=1.0) - captured = {} - - def fake_generate_video_impl( - prompt=None, - sampling_param=None, - mouse_cond=None, - keyboard_cond=None, - grid_sizes=None, - **kwargs, - ): - captured["prompt"] = prompt - captured["sampling_param"] = sampling_param - captured["kwargs"] = kwargs - return {"prompts": prompt, "video_path": "outputs/test.mp4"} - - monkeypatch.setattr(generator, "_generate_video_impl", fake_generate_video_impl) - - with pytest.warns(DeprecationWarning): - result = generator.generate_video( - prompt="legacy prompt", - neg_prompt="custom negative", - embedded_cfg_scale=7.5, - ) - - assert captured["prompt"] == "legacy prompt" - assert captured["sampling_param"].negative_prompt == "custom negative" - assert captured["sampling_param"].embedded_cfg_scale == 7.5 - assert captured["kwargs"]["fastvideo_args"] is generator.fastvideo_args - assert generator.fastvideo_args.pipeline_config.embedded_cfg_scale == 1.0 - assert result["video_path"] == "outputs/test.mp4" - - def test_generate_batch_prompt_file_returns_typed_results(tmp_path, monkeypatch): generator = _new_runtime_video_generator() _patch_sampling_param_from_pretrained(monkeypatch) diff --git a/fastvideo/worker/executor.py b/fastvideo/worker/executor.py index a7cce7ff7..c037c5e03 100644 --- a/fastvideo/worker/executor.py +++ b/fastvideo/worker/executor.py @@ -110,12 +110,12 @@ class Executor(ABC): @abstractmethod def set_log_queue(self, log_queue: Queue | None) -> None: - """Forward worker logs to the given queue. Call before generate_video.""" + """Forward worker logs to the given queue. Call before generate.""" self.collective_rpc("set_log_queue", kwargs={"log_queue": log_queue}) @abstractmethod def clear_log_queue(self) -> None: - """Stop forwarding worker logs to the queue. Call after generate_video.""" + """Stop forwarding worker logs to the queue. Call after generate.""" self.collective_rpc("clear_log_queue") @abstractmethod diff --git a/fastvideo/worker/multiproc_executor.py b/fastvideo/worker/multiproc_executor.py index 5c4d31875..aa95ad6e6 100644 --- a/fastvideo/worker/multiproc_executor.py +++ b/fastvideo/worker/multiproc_executor.py @@ -283,11 +283,11 @@ class MultiprocExecutor(Executor): raise RuntimeError(f"Worker {i} failed to merge LoRA weights") def set_log_queue(self, log_queue: Queue | None) -> None: - """Forward worker logs to the given queue. Call before generate_video.""" + """Forward worker logs to the given queue. Call before generate.""" self.collective_rpc("set_log_queue", kwargs={"log_queue": log_queue}) def clear_log_queue(self) -> None: - """Stop forwarding worker logs to the queue. Call after generate_video.""" + """Stop forwarding worker logs to the queue. Call after generate.""" self.collective_rpc("clear_log_queue") def collective_rpc(self, diff --git a/scripts/checkpoint_conversion/longcat_to_fastvideo.py b/scripts/checkpoint_conversion/longcat_to_fastvideo.py index 19a80601c..e787dbe22 100644 --- a/scripts/checkpoint_conversion/longcat_to_fastvideo.py +++ b/scripts/checkpoint_conversion/longcat_to_fastvideo.py @@ -682,20 +682,23 @@ def main(): print(" 1. Test basic generation:") print(" from fastvideo import VideoGenerator") print(f" generator = VideoGenerator.from_pretrained('{output_dir}')") - print(" video = generator.generate_video(") - print(" prompt='A cat playing piano',") - print(" num_inference_steps=50") - print(" )") + print(" video = generator.generate({") + print(" 'prompt': 'A cat playing piano',") + print(" 'sampling': {'num_inference_steps': 50},") + print(" })") print() if (output_dir / "lora" / "distilled").exists(): print(" 2. Test distilled generation (16 steps with LoRA):") - print(f" generator = VideoGenerator.from_pretrained('{output_dir}',") - print(f" lora_path='{output_dir}/lora/distilled',") - print(" lora_nickname='distilled')") - print(" video = generator.generate_video(") - print(" prompt='A cat playing piano',") - print(" num_inference_steps=16,") - print(" guidance_scale=1.0)") + print(" generator = VideoGenerator.from_config({") + print(f" 'model_path': '{output_dir}',") + print(" 'pipeline': {'components': {") + print(f" 'lora_path': '{output_dir}/lora/distilled',") + print(" 'lora_nickname': 'distilled'}},") + print(" })") + print(" video = generator.generate({") + print(" 'prompt': 'A cat playing piano',") + print(" 'sampling': {'num_inference_steps': 16, 'guidance_scale': 1.0},") + print(" })") print() print()