Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
0c16ec91b0 | ||
|
|
e57543b79d |
@@ -80,36 +80,26 @@ def _run_fastvideo_pipeline(model_path: Path, params: dict[str, Any]) -> Any:
|
||||
|
||||
generator = VideoGenerator.from_pretrained(
|
||||
str(model_path),
|
||||
{
|
||||
"engine": {
|
||||
"num_gpus": 1,
|
||||
"use_fsdp_inference": False,
|
||||
"offload": {
|
||||
"dit": False,
|
||||
"vae": False,
|
||||
"text_encoder": False,
|
||||
},
|
||||
},
|
||||
},
|
||||
num_gpus=1,
|
||||
use_fsdp_inference=False,
|
||||
dit_cpu_offload=False,
|
||||
vae_cpu_offload=False,
|
||||
text_encoder_cpu_offload=False,
|
||||
)
|
||||
try:
|
||||
return generator.generate({
|
||||
"prompt": params["prompt"],
|
||||
"negative_prompt": params.get("negative_prompt"),
|
||||
"sampling": {
|
||||
"height": params.get("height"),
|
||||
"width": params.get("width"),
|
||||
"num_frames": params.get("num_frames"),
|
||||
"fps": params.get("fps"),
|
||||
"num_inference_steps": params["num_inference_steps"],
|
||||
"guidance_scale": params.get("guidance_scale"),
|
||||
"seed": params["seed"],
|
||||
},
|
||||
"output": {
|
||||
"output_path": f"outputs_{_MODEL_FAMILY}/pipeline_parity",
|
||||
"save_video": False,
|
||||
},
|
||||
})
|
||||
return generator.generate_video(
|
||||
prompt=params["prompt"],
|
||||
negative_prompt=params.get("negative_prompt"),
|
||||
output_path=f"outputs_{_MODEL_FAMILY}/pipeline_parity",
|
||||
save_video=False,
|
||||
height=params.get("height"),
|
||||
width=params.get("width"),
|
||||
num_frames=params.get("num_frames"),
|
||||
fps=params.get("fps"),
|
||||
num_inference_steps=params["num_inference_steps"],
|
||||
guidance_scale=params.get("guidance_scale"),
|
||||
seed=params["seed"],
|
||||
)
|
||||
finally:
|
||||
generator.shutdown()
|
||||
|
||||
|
||||
@@ -84,9 +84,7 @@ fastvideo/configs/models/dits/__init__.py
|
||||
fastvideo/configs/models/encoders/__init__.py
|
||||
fastvideo/configs/models/vaes/__init__.py
|
||||
fastvideo/envs.py
|
||||
fastvideo/api/schema.py
|
||||
fastvideo/api/resolution.py
|
||||
fastvideo/api/inference_resolution.py
|
||||
fastvideo/fastvideo_args.py
|
||||
fastvideo/distributed/**
|
||||
fastvideo/layers/**
|
||||
fastvideo/attention/**
|
||||
|
||||
@@ -1,80 +0,0 @@
|
||||
---
|
||||
name: env-var-conventions
|
||||
description: Add, read, rename, or remove an environment variable in FastVideo, or change the environment-variable policy. Use before touching fastvideo/envs.py, os.environ, os.getenv, or monkeypatch.setenv in fastvideo/, and when fastvideo/tests/contract/test_env_policy.py fails.
|
||||
---
|
||||
|
||||
# Environment Variable Conventions
|
||||
|
||||
## Purpose
|
||||
|
||||
FastVideo registers its environment variables as typed fields in
|
||||
`fastvideo/envs.py`. The policy that governs them is
|
||||
`docs/contributing/env_vars.md`, and the contract test
|
||||
`fastvideo/tests/contract/test_env_policy.py` enforces the policy in the unit
|
||||
CI lane. This skill routes an environment-variable change through that policy.
|
||||
The policy doc is the single source of the rules; read it instead of relying
|
||||
on a summary here.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Read `docs/contributing/env_vars.md` in full.
|
||||
- Decide whether the setting belongs in an environment variable or an argument
|
||||
(rule 5 in the policy doc). Settings that users change per deployment are
|
||||
arguments; add them as typed config fields in `fastvideo/api/schema.py`
|
||||
instead.
|
||||
|
||||
## Inputs
|
||||
|
||||
| Parameter | Required | Description |
|
||||
| ---------- | -------- | -------------------------------------------------------------- |
|
||||
| `change` | Yes | Add, read, rename, or remove a variable, or change the policy. |
|
||||
| `variable` | Yes | The variable name, with the `FASTVIDEO_` prefix. |
|
||||
|
||||
## Steps
|
||||
|
||||
1. **Declare or edit the variable in `fastvideo/envs.py`.**
|
||||
- Pick the field type and category that the policy doc lists.
|
||||
- Write a description that states what the variable does and its units.
|
||||
- To rename, keep the old name in `deprecated_names`. To remove, add the
|
||||
name to `DEPRECATED_VARIABLES`. Update the uses in `examples/`,
|
||||
`scripts/`, `docs/`, `apps/`, and the tests.
|
||||
2. **Read the variable with `envs.NAME.get()` inside a function.**
|
||||
- In tests, change the value with `envs.NAME.override(value)`, and a variable
|
||||
outside the registry with `envs.override_external(name, value)`; the
|
||||
`env_overrides` fixture keeps either until the end of the test.
|
||||
- Name a variable that only tests read `FASTVIDEO_TEST_*`.
|
||||
- Do not call `os.environ`, `os.getenv`, or `monkeypatch.setenv` for a
|
||||
FastVideo variable.
|
||||
- To set a variable that another tool reads, call `envs.set_external`,
|
||||
`envs.setdefault_external`, or `envs.unset_external`.
|
||||
3. **Regenerate the table in the policy doc.**
|
||||
- Run `python fastvideo/tests/contract/test_env_policy.py`.
|
||||
4. **Run the contract test.**
|
||||
- Run `pytest fastvideo/tests/contract/test_env_policy.py`.
|
||||
- When the test reports a fixed known violation, delete or lower its entry
|
||||
in `KNOWN_VIOLATIONS`. Never add an entry to `KNOWN_VIOLATIONS`.
|
||||
5. **When the policy itself changes, update the policy doc and the contract
|
||||
test in the same pull request.**
|
||||
- The rules in `docs/contributing/env_vars.md`, the checks and allowlist in
|
||||
`fastvideo/tests/contract/test_env_policy.py`, and this skill must agree.
|
||||
|
||||
## Outputs
|
||||
|
||||
- A registry entry in `fastvideo/envs.py` and call sites that use
|
||||
`envs.NAME.get()`.
|
||||
- A regenerated table in `docs/contributing/env_vars.md`.
|
||||
- A passing `fastvideo/tests/contract/test_env_policy.py`.
|
||||
|
||||
## Example Usage
|
||||
|
||||
```
|
||||
Add a FASTVIDEO_DEBUG_MY_STAGE switch that logs MyStage inputs.
|
||||
```
|
||||
|
||||
## References
|
||||
|
||||
- `docs/contributing/env_vars.md`: the policy, the field types, and the
|
||||
violation kinds that the contract test reports.
|
||||
- `fastvideo/envs.py`: the registry.
|
||||
- `fastvideo/tests/contract/test_env_policy.py`: the contract test,
|
||||
`EXTERNAL_ALLOWLIST`, and `KNOWN_VIOLATIONS`.
|
||||
@@ -57,7 +57,7 @@ Hardcoded:
|
||||
- Quality tier: **`default`**. `full_quality` is a separate, deliberate
|
||||
operation.
|
||||
- HF repo: `FastVideo/ssim-reference-videos` (override via
|
||||
`FASTVIDEO_TEST_SSIM_REFERENCE_HF_REPO`).
|
||||
`FASTVIDEO_SSIM_REFERENCE_HF_REPO`).
|
||||
- Device folder: `L40S_reference_videos`.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
@@ -16,7 +16,6 @@ exec pytest \
|
||||
./fastvideo/tests/worker/ \
|
||||
./fastvideo/tests/training/test_trackers.py \
|
||||
./fastvideo/tests/attention/test_sdpa_metadata_mask_contract.py \
|
||||
./fastvideo/tests/attention/test_vsa_h3_tile_grad_safety.py \
|
||||
./fastvideo/tests/modal/test_kernel_build_cache.py \
|
||||
./fastvideo/tests/modal/test_pr_test.py \
|
||||
./fastvideo/tests/modal/test_ssim_test.py \
|
||||
|
||||
@@ -489,6 +489,7 @@ def classify_paths(paths: list[str]) -> MergePlan:
|
||||
_select_output_coverage(plan, path)
|
||||
continue
|
||||
if path in {
|
||||
"fastvideo/fastvideo_args.py",
|
||||
"fastvideo/forward_context.py",
|
||||
"fastvideo/image_processor.py",
|
||||
"fastvideo/registry.py",
|
||||
|
||||
@@ -92,7 +92,6 @@ jobs:
|
||||
fastvideo/tests/mlx/test_frame_upsample.py \
|
||||
fastvideo/tests/mlx/test_mlx_fast_spatial.py \
|
||||
fastvideo/tests/mlx/test_mlx_refine.py \
|
||||
fastvideo/tests/mlx/test_mlx_prompt_enhance.py \
|
||||
fastvideo/tests/mlx/test_mlx_prompt_to_video_decode.py \
|
||||
fastvideo/tests/mlx/test_mlx_wan22_prompt_cache_fingerprint.py \
|
||||
fastvideo/tests/mlx/test_wan22_sample.py \
|
||||
@@ -157,7 +156,6 @@ jobs:
|
||||
fastvideo/tests/mlx/test_frame_upsample.py \
|
||||
fastvideo/tests/mlx/test_mlx_fast_spatial.py \
|
||||
fastvideo/tests/mlx/test_mlx_refine.py \
|
||||
fastvideo/tests/mlx/test_mlx_prompt_enhance.py \
|
||||
fastvideo/tests/mlx/test_mlx_prompt_to_video_decode.py \
|
||||
fastvideo/tests/mlx/test_mlx_wan22_prompt_cache_fingerprint.py \
|
||||
fastvideo/tests/mlx/test_wan22_sample.py \
|
||||
|
||||
@@ -12,7 +12,7 @@
|
||||
- `2026/09/15`: Release [FastH3 8-Step V2](https://huggingface.co/FastVideo/FastVideo-FastH3-8-Step-V2), an eight-forward data-free DMD2 checkpoint distilled from MiniMax-H3 with 80% Video Sparse Attention. Run it with `examples/inference/basic/basic_fasth3_8step.py` or the [FastH3 8-Step V2 recipe](https://haoailab.com/FastVideo/cookbook/minimax-h3/).
|
||||
- `2026/09/01`: FastH3 now runs locally on Apple Silicon through MLX and on NVIDIA DGX Spark through CUDA 13, including two-Spark inference. Follow the [FastH3 recipes](https://haoailab.com/FastVideo/cookbook/minimax-h3/) and read the [Blog](https://haoailab.com/blogs/fasth3-local/).
|
||||
- `2026/08/27`: [FastH3 Preview v1](https://haoailab.com/blogs/fasth3-preview/) is an open-weight 4-step sparse-distilled MiniMax-H3 model for synchronized video-and-audio generation, developed in collaboration with [Nuva Lab](https://nuvalab.ai/) and the [NVIDIA FastGen team](https://github.com/NVlabs/FastGen). Download the recommended [VSA / Data-Free weights](https://huggingface.co/FastVideo/FastVideo-FastH3-4-step-Preview-v1-VSA-DataFree), or see the [full FastH3 collection](https://huggingface.co/collections/FastVideo/fastvideo-fasth3).
|
||||
- `2026/08/19`: FastVideo now supports MLX on Apple Silicon with [FastMetal-QAD](https://huggingface.co/collections/FastVideo/fastmetal), a family of 1.3B, 5B, and 14B models optimized for Mac. Follow the [MLX install guide](https://hao-ai-lab.github.io/FastVideo/getting_started/installation/mlx/) and read the [Blog](https://haoailab.com/blogs/fastmetal/).
|
||||
- `2026/08/19`: FastVideo now supports MLX on Apple Silicon with [FastMetal-QAD](https://huggingface.co/collections/FastVideo/fastmetal), a family of 1.3B, 5B, and 14B models optimized for Mac—follow the [Apple Silicon guide](https://hao-ai-lab.github.io/FastVideo/getting_started/installation/mps/) and read the [Blog](https://haoailab.com/blogs/fastmetal/).
|
||||
- `2026/06/23`: Release FastWan-QAD: 5s of Video generated in 1.8s E2E. See the [FastWan-QAD models](https://huggingface.co/FastVideo/FastWan-QAD-FP8-1.3B), [Attn-QAT training guide](https://haoailab.com/FastVideo/training/attn_qat/), and [blog](https://haoailab.com/blogs/fastwan-qad/).
|
||||
- `2026/03/17`: Release demo: Into the Dreamverse: Vibe Directing in FastVideo, check out the [Blog](https://haoailab.com/blogs/dreamverse/).
|
||||
- `2026/03/13`: Release demo: Create a 5s 1080p Video in 4.5s with FastVideo on a Single GPU, check out the [Blog](https://haoailab.com/blogs/fastvideo_realtime_1080p/).
|
||||
@@ -64,12 +64,13 @@ UV_TORCH_BACKEND=cu126 uv pip install fastvideo
|
||||
```
|
||||
|
||||
Use `UV_TORCH_BACKEND=cu130` on CUDA 13. Apple silicon users should follow the
|
||||
[MLX install guide](https://hao-ai-lab.github.io/FastVideo/getting_started/installation/mlx/).
|
||||
[MPS installation guide](https://hao-ai-lab.github.io/FastVideo/getting_started/installation/mps/).
|
||||
|
||||
> **On an Apple Silicon Mac?** Install with `uv pip install -e '.[mlx]'` from
|
||||
> a clone, then pick a recipe in the
|
||||
> [cookbook](https://haoailab.com/FastVideo/cookbook/). See the
|
||||
> [MLX install guide](https://hao-ai-lab.github.io/FastVideo/getting_started/installation/mlx/).
|
||||
> **On an Apple Silicon Mac?** FastVideo runs FastMetal-QAD through an MLX
|
||||
> runtime. Install with `uv pip install -e '.[mlx]'`, download
|
||||
> [`FastVideo/FastMetal-1.3B-QAD`](https://huggingface.co/FastVideo/FastMetal-1.3B-QAD),
|
||||
> and follow the
|
||||
> [Apple Silicon guide](https://hao-ai-lab.github.io/FastVideo/getting_started/installation/mps/).
|
||||
|
||||
Please see our [docs](https://hao-ai-lab.github.io/FastVideo/getting_started/installation/) for more detailed installation instructions.
|
||||
|
||||
@@ -87,7 +88,7 @@ Install FastVideo (https://github.com/hao-ai-lab/FastVideo) into a fresh uv virt
|
||||
https://hao-ai-lab.github.io/FastVideo/getting_started/installation/):
|
||||
- NVIDIA GPU, x86_64 -> docs/getting_started/installation/gpu.md
|
||||
- NVIDIA DGX Spark / GB10, aarch64, CUDA 13 -> docs/getting_started/installation/spark.md
|
||||
- Apple Silicon, macOS -> docs/getting_started/installation/mlx.md
|
||||
- Apple Silicon, macOS -> docs/getting_started/installation/mps.md
|
||||
3. Use uv for every step. If a command fails, debug it and tell me what you changed.
|
||||
4. Verify the result:
|
||||
python -c "import fastvideo, torch; print('cuda', torch.cuda.is_available())"
|
||||
@@ -135,20 +136,18 @@ def main():
|
||||
# Create a video generator with a pre-trained model
|
||||
generator = VideoGenerator.from_pretrained(
|
||||
"FastVideo/FastWan2.1-T2V-1.3B-Diffusers",
|
||||
{"engine": {"num_gpus": 1}}, # Adjust based on your hardware
|
||||
num_gpus=1, # Adjust based on your hardware
|
||||
)
|
||||
|
||||
# Define a prompt for your video
|
||||
prompt = "A curious raccoon peers through a vibrant field of yellow sunflowers, its eyes wide with interest."
|
||||
|
||||
# Generate the video
|
||||
video = generator.generate({
|
||||
"prompt": prompt,
|
||||
"output": {
|
||||
"output_path": "my_videos/", # Controls where videos are saved
|
||||
"save_video": True,
|
||||
},
|
||||
})
|
||||
video = generator.generate_video(
|
||||
prompt,
|
||||
output_path="my_videos/", # Controls where videos are saved
|
||||
save_video=True
|
||||
)
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
|
||||
+33
-4
@@ -42,7 +42,7 @@ Near-term OSS note:
|
||||
- `apps/dreamverse/dreamverse/main.py`: websocket endpoint, request handling,
|
||||
session state machine, rewrite orchestration, REST routes, and stream relay
|
||||
- `apps/dreamverse/dreamverse/gpu_pool.py`: GPU worker processes, warmup, model
|
||||
loading, and `generate()` calls through FastVideo
|
||||
loading, and `generate_video()` calls through FastVideo
|
||||
- `apps/dreamverse/dreamverse/prompt_enhancer.py`: prompt enhancement, rollout
|
||||
rewrite execution, provider selection, and timeout/fallback behavior
|
||||
- `apps/dreamverse/dreamverse/rewrite_prompt_payload.py`: canonical rewrite request payload
|
||||
@@ -299,18 +299,47 @@ There are three related prompt paths in the current system:
|
||||
|
||||
## Initial Image And Segment Handling
|
||||
|
||||
The frontend currently sends `initial_image` as part of session init or
|
||||
The frontend sends `initial_image` and, for first/last frame mode,
|
||||
`last_frame_image` as part of `session_init_v2`, `project_init_v1`, or
|
||||
`simple_generate`.
|
||||
|
||||
The server:
|
||||
|
||||
- validates and persists the image
|
||||
- uses it only for segment 1 when present
|
||||
- validates and persists the images
|
||||
- uses `initial_image` only for segment 1 when present
|
||||
- keeps continuation state for later segments in the GPU worker
|
||||
|
||||
This means the runtime, not the frontend, decides how segment 1 image
|
||||
conditioning and later continuation conditioning are applied.
|
||||
|
||||
## Creation Studio Config
|
||||
|
||||
The lobby creation studio sends model, mode, aspect ratio, resolution, and
|
||||
duration with session init. The server parses these fields into a per-session
|
||||
creation config and echoes the resolved values back on `gpu_assigned` and
|
||||
`ltx2_stream_start` as `creation_config`.
|
||||
|
||||
Incoming fields on `session_init_v2` and `project_init_v1`:
|
||||
|
||||
- `generation_mode`: `t2va`, `fl2va`, or `ref2va` (canonical upstream IDs from #1834)
|
||||
- `model_id`: `fast-ltx2`, `fast-ltx23`, or `fast-h3`
|
||||
- `aspect_ratio`: one of `21:9`, `16:9`, `4:3`, `1:1`, `3:4`, `9:16`
|
||||
- `resolution`: one of `480p`, `720p`, `1080p`, `4k`
|
||||
- `duration_sec`: `5`, `10`, or `15`
|
||||
- `initial_image`: optional image payload for reference / first-frame modes
|
||||
- `last_frame_image`: optional image payload for first/last frame mode
|
||||
|
||||
Echoed `creation_config` includes the resolved frame size,
|
||||
`num_frames`, and `generation_segment_cap` derived from `duration_sec`.
|
||||
|
||||
Mode validation:
|
||||
|
||||
- `ref2va` requires `initial_image`
|
||||
- `fl2va` requires both `initial_image` and `last_frame_image`
|
||||
|
||||
Per-step generation uses the resolved `frame_width`, `frame_height`, and
|
||||
`num_frames` from the session creation config.
|
||||
|
||||
## Websocket Contract
|
||||
|
||||
The websocket is the main integration surface between UI and runtime.
|
||||
|
||||
@@ -52,8 +52,7 @@ import torch # noqa: E402
|
||||
|
||||
from fastvideo import VideoGenerator # noqa: E402
|
||||
from fastvideo.api import ( # noqa: E402
|
||||
ComponentConfig, CompileConfig, EngineConfig, GenerationResult, GeneratorConfig, OffloadConfig, PipelineSelection,
|
||||
QuantizationConfig,
|
||||
ComponentConfig, CompileConfig, EngineConfig, GeneratorConfig, OffloadConfig, PipelineSelection, QuantizationConfig,
|
||||
)
|
||||
|
||||
DEFAULT_PROMPT = ("A cinematic drone shot over coastal cliffs at sunrise, golden "
|
||||
@@ -129,9 +128,9 @@ def _build_generator_config(model_path: str, enable_compile: bool, num_gpus: int
|
||||
)
|
||||
|
||||
|
||||
def _extract_stage_times(result: GenerationResult) -> OrderedDict[str, float]:
|
||||
def _extract_stage_times(result: dict) -> OrderedDict[str, float]:
|
||||
out: OrderedDict[str, float] = OrderedDict()
|
||||
info = result.logging_info if isinstance(result, GenerationResult) else None
|
||||
info = result.get("logging_info") if isinstance(result, dict) else None
|
||||
if info is None:
|
||||
return out
|
||||
stages = getattr(info, "stages", None)
|
||||
@@ -163,25 +162,19 @@ def _do_one_run(generator: VideoGenerator, prompt: str, *, height: int, width: i
|
||||
_reset_peak_gpu()
|
||||
t0 = time.perf_counter()
|
||||
try:
|
||||
result = generator.generate({
|
||||
"prompt": prompt,
|
||||
"negative_prompt": "",
|
||||
"sampling": {
|
||||
"height": height,
|
||||
"width": width,
|
||||
"num_frames": num_frames,
|
||||
"fps": 24,
|
||||
"num_inference_steps": num_inference_steps,
|
||||
"guidance_scale": 1.0,
|
||||
"seed": seed,
|
||||
},
|
||||
"output": {
|
||||
"save_video": False
|
||||
},
|
||||
"extensions": {
|
||||
"ltx2_image_crf": 0.0
|
||||
},
|
||||
})
|
||||
result = generator.generate_video(
|
||||
prompt=prompt,
|
||||
negative_prompt="",
|
||||
save_video=False,
|
||||
height=height,
|
||||
width=width,
|
||||
num_frames=num_frames,
|
||||
fps=24,
|
||||
num_inference_steps=num_inference_steps,
|
||||
guidance_scale=1.0,
|
||||
seed=seed,
|
||||
ltx2_image_crf=0.0,
|
||||
)
|
||||
if torch.cuda.is_available():
|
||||
torch.cuda.synchronize()
|
||||
except Exception as exc:
|
||||
|
||||
@@ -0,0 +1,137 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
from dreamverse.config import MODEL_REGISTRY
|
||||
|
||||
# Canonical upstream wire IDs. FL2VA is tracked in #1834 but not wired on Dreamverse
|
||||
# streaming backends yet.
|
||||
LTX_LOBBY_GENERATION_MODES = frozenset({"t2va", "ref2va"})
|
||||
H3_LOBBY_GENERATION_MODES = frozenset({"t2va", "ref2va"})
|
||||
|
||||
LTX_LOBBY_ASPECT_RATIOS = frozenset({"21:9", "16:9", "4:3", "1:1", "3:4", "9:16"})
|
||||
|
||||
# Realtime FastLTX serving is validated through 1080p-class outputs; 4K is rejected
|
||||
# until the runtime path is tested on Dreamverse GPUs.
|
||||
LTX_LOBBY_RESOLUTIONS = frozenset({"480p", "720p", "1080p"})
|
||||
|
||||
# FastH3 serves a fixed 768x1344 (16:9-class) output; lobby resolution is nominal.
|
||||
H3_LOBBY_ASPECT_RATIOS = frozenset({"16:9"})
|
||||
H3_LOBBY_RESOLUTIONS = frozenset({"720p"})
|
||||
|
||||
LOBBY_DURATION_SEC = frozenset({5, 10, 15})
|
||||
|
||||
UNSUPPORTED_GENERATION_MODE_MESSAGES = {
|
||||
"fl2va": "First/last frame mode (FL2VA) is not supported yet.",
|
||||
}
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ModelCreationCapabilities:
|
||||
generation_modes: frozenset[str]
|
||||
aspect_ratios: frozenset[str]
|
||||
resolutions: frozenset[str]
|
||||
duration_sec: frozenset[int]
|
||||
unsupported_generation_modes: frozenset[str] = frozenset({"fl2va"})
|
||||
|
||||
def as_dict(self) -> dict[str, object]:
|
||||
unsupported = {
|
||||
mode: UNSUPPORTED_GENERATION_MODE_MESSAGES[mode]
|
||||
for mode in sorted(self.unsupported_generation_modes)
|
||||
if mode in UNSUPPORTED_GENERATION_MODE_MESSAGES
|
||||
}
|
||||
return {
|
||||
"generation_modes": sorted(self.generation_modes),
|
||||
"aspect_ratios": sorted(self.aspect_ratios),
|
||||
"resolutions": sorted(self.resolutions),
|
||||
"duration_sec": sorted(self.duration_sec),
|
||||
"unsupported_generation_modes": unsupported,
|
||||
"reference_assets": {
|
||||
"mime_types": ["image/png", "image/jpeg", "image/webp"],
|
||||
"max_bytes": 15 * 1024 * 1024,
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
LTX_MODEL_CREATION_CAPABILITIES = ModelCreationCapabilities(
|
||||
generation_modes=LTX_LOBBY_GENERATION_MODES,
|
||||
aspect_ratios=LTX_LOBBY_ASPECT_RATIOS,
|
||||
resolutions=LTX_LOBBY_RESOLUTIONS,
|
||||
duration_sec=LOBBY_DURATION_SEC,
|
||||
)
|
||||
|
||||
H3_MODEL_CREATION_CAPABILITIES = ModelCreationCapabilities(
|
||||
generation_modes=H3_LOBBY_GENERATION_MODES,
|
||||
aspect_ratios=H3_LOBBY_ASPECT_RATIOS,
|
||||
resolutions=H3_LOBBY_RESOLUTIONS,
|
||||
duration_sec=LOBBY_DURATION_SEC,
|
||||
)
|
||||
|
||||
MODEL_CREATION_CAPABILITIES: dict[str, ModelCreationCapabilities] = {
|
||||
"fast-ltx2": LTX_MODEL_CREATION_CAPABILITIES,
|
||||
"fast-ltx23": LTX_MODEL_CREATION_CAPABILITIES,
|
||||
"fast-h3": H3_MODEL_CREATION_CAPABILITIES,
|
||||
}
|
||||
|
||||
|
||||
def capabilities_for_model(model_id: str) -> ModelCreationCapabilities:
|
||||
if model_id not in MODEL_REGISTRY:
|
||||
raise ValueError(f"Unknown model_id: {model_id}")
|
||||
return MODEL_CREATION_CAPABILITIES.get(model_id, LTX_MODEL_CREATION_CAPABILITIES)
|
||||
|
||||
|
||||
def lobby_capabilities_as_dict() -> dict[str, object]:
|
||||
model_ids = sorted(MODEL_REGISTRY.keys())
|
||||
models = {model_id: capabilities_for_model(model_id).as_dict() for model_id in model_ids}
|
||||
union_modes: set[str] = set()
|
||||
union_aspects: set[str] = set()
|
||||
union_resolutions: set[str] = set()
|
||||
union_durations: set[int] = set()
|
||||
for caps in MODEL_CREATION_CAPABILITIES.values():
|
||||
union_modes.update(caps.generation_modes)
|
||||
union_aspects.update(caps.aspect_ratios)
|
||||
union_resolutions.update(caps.resolutions)
|
||||
union_durations.update(caps.duration_sec)
|
||||
return {
|
||||
"model_ids": model_ids,
|
||||
"models": models,
|
||||
"generation_modes": sorted(union_modes),
|
||||
"aspect_ratios": sorted(union_aspects),
|
||||
"resolutions": sorted(union_resolutions),
|
||||
"duration_sec": sorted(union_durations),
|
||||
"unsupported_generation_modes": dict(UNSUPPORTED_GENERATION_MODE_MESSAGES),
|
||||
"reference_assets": {
|
||||
"mime_types": ["image/png", "image/jpeg", "image/webp"],
|
||||
"max_bytes": 15 * 1024 * 1024,
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
# Backward-compatible alias used in tests.
|
||||
LOBBY_CREATION_CAPABILITIES = lobby_capabilities_as_dict()
|
||||
|
||||
|
||||
def validate_lobby_creation_config(
|
||||
*,
|
||||
model_id: str,
|
||||
generation_mode: str,
|
||||
aspect_ratio: str,
|
||||
resolution: str,
|
||||
duration_sec: int,
|
||||
) -> None:
|
||||
if model_id not in MODEL_REGISTRY:
|
||||
raise ValueError(f"Unknown model_id: {model_id}")
|
||||
|
||||
caps = capabilities_for_model(model_id)
|
||||
|
||||
if generation_mode in caps.unsupported_generation_modes:
|
||||
raise ValueError(UNSUPPORTED_GENERATION_MODE_MESSAGES[generation_mode])
|
||||
if generation_mode not in caps.generation_modes:
|
||||
raise ValueError(f"Unsupported generation_mode: {generation_mode}")
|
||||
|
||||
if aspect_ratio not in caps.aspect_ratios:
|
||||
raise ValueError(f"Unsupported aspect_ratio: {aspect_ratio}")
|
||||
if resolution not in caps.resolutions:
|
||||
raise ValueError(f"Unsupported resolution: {resolution}")
|
||||
if duration_sec not in caps.duration_sec:
|
||||
raise ValueError("duration_sec must be 5, 10, or 15.")
|
||||
@@ -36,6 +36,10 @@ class GenerationBackend(Protocol):
|
||||
segment_idx: int,
|
||||
image_path: str | None,
|
||||
reset_conditioning: bool,
|
||||
*,
|
||||
frame_width: int | None = None,
|
||||
frame_height: int | None = None,
|
||||
num_frames: int | None = None,
|
||||
) -> StepResult:
|
||||
...
|
||||
|
||||
|
||||
@@ -80,6 +80,10 @@ class VideoGenerationWorker:
|
||||
segment_idx: int,
|
||||
image_path: str | None,
|
||||
reset_conditioning: bool,
|
||||
*,
|
||||
frame_width: int | None = None,
|
||||
frame_height: int | None = None,
|
||||
num_frames: int | None = None,
|
||||
) -> StepResult:
|
||||
"""Generate one segment through the selected model backend."""
|
||||
return self._require_backend().generate_step(
|
||||
@@ -87,6 +91,9 @@ class VideoGenerationWorker:
|
||||
segment_idx,
|
||||
image_path,
|
||||
reset_conditioning,
|
||||
frame_width=frame_width,
|
||||
frame_height=frame_height,
|
||||
num_frames=num_frames,
|
||||
)
|
||||
|
||||
def warmup(self, prompt: str) -> dict[str, float]:
|
||||
|
||||
@@ -189,6 +189,9 @@ def gpu_worker_process(
|
||||
segment_idx,
|
||||
image_path=payload.image_path,
|
||||
reset_conditioning=payload.reset_conditioning,
|
||||
frame_width=payload.frame_width,
|
||||
frame_height=payload.frame_height,
|
||||
num_frames=payload.num_frames,
|
||||
)
|
||||
head_trim_frames = step_result.head_trim_frames
|
||||
head_trim_audio_frames = step_result.head_trim_audio_frames
|
||||
@@ -753,6 +756,10 @@ class GPUSlot:
|
||||
segment_idx: int = 1,
|
||||
image_path: str | None = None,
|
||||
reset_conditioning: bool = False,
|
||||
*,
|
||||
frame_width: int | None = None,
|
||||
frame_height: int | None = None,
|
||||
num_frames: int | None = None,
|
||||
) -> dict[str, float]:
|
||||
"""Execute a generation step for a specific user.
|
||||
|
||||
@@ -766,6 +773,9 @@ class GPUSlot:
|
||||
segment_idx=segment_idx,
|
||||
image_path=image_path,
|
||||
reset_conditioning=bool(reset_conditioning),
|
||||
frame_width=frame_width,
|
||||
frame_height=frame_height,
|
||||
num_frames=num_frames,
|
||||
)
|
||||
response = await self._send_command_tagged(Command(CommandType.USER_STEP, payload=payload, user_id=user_id),
|
||||
timeout=1800.0)
|
||||
|
||||
@@ -14,7 +14,6 @@ import gc
|
||||
import os
|
||||
import re
|
||||
import time
|
||||
from typing import TYPE_CHECKING
|
||||
import numpy as np
|
||||
import torch
|
||||
|
||||
@@ -35,9 +34,6 @@ from dreamverse.config import (
|
||||
)
|
||||
from dreamverse.generation_contracts import StepResult
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from fastvideo.api import GenerationResult
|
||||
|
||||
# Multi-frame decoded continuation defaults from
|
||||
# examples/inference/basic/basic_ltx2_distilled_video_continuation.py.
|
||||
# Overridable via environment variables.
|
||||
@@ -99,8 +95,8 @@ class ContinuationState:
|
||||
self.video_images = None
|
||||
self.audio_latents = None
|
||||
|
||||
def apply_video(self, request: dict, segment_idx: int) -> None:
|
||||
"""Seed the next-segment request with the cached tail frames."""
|
||||
def apply_video(self, request_kwargs: dict, segment_idx: int) -> None:
|
||||
"""Seed next-segment kwargs with the cached tail frames."""
|
||||
if segment_idx <= 1 or not self.video_images:
|
||||
return
|
||||
from PIL import Image
|
||||
@@ -114,21 +110,21 @@ class ContinuationState:
|
||||
arr = np.clip(arr, 0, 255).astype(np.uint8)
|
||||
noisy.append(Image.fromarray(arr))
|
||||
cond_images = noisy
|
||||
request["extensions"]["ltx2_video_conditions"] = [(
|
||||
request_kwargs["ltx2_video_conditions"] = [(
|
||||
cond_images,
|
||||
LTX2_VIDEO_CONDITIONING_FRAME_IDX,
|
||||
LTX2_VIDEO_CONDITIONING_STRENGTH,
|
||||
)]
|
||||
request["extensions"]["ltx2_images"] = None
|
||||
request["inputs"]["image_path"] = None
|
||||
request_kwargs["ltx2_images"] = None
|
||||
request_kwargs["image_path"] = None
|
||||
|
||||
def apply_audio(
|
||||
self,
|
||||
request: dict,
|
||||
request_kwargs: dict,
|
||||
segment_idx: int,
|
||||
audio_lps: float,
|
||||
) -> None:
|
||||
"""Seed the next-segment request with clean audio latents + denoise mask.
|
||||
"""Seed next-segment kwargs with clean audio latents + denoise mask.
|
||||
|
||||
When audio conditioning is longer than video, extend audio
|
||||
generation and shift video RoPE forward so the audio prefix
|
||||
@@ -145,9 +141,9 @@ class ContinuationState:
|
||||
audio_extra = max(0, AUDIO_CONDITIONING_NUM_FRAMES - LTX2_VIDEO_CONDITIONING_NUM_FRAMES)
|
||||
if audio_extra > 0:
|
||||
audio_num_frames = NUM_FRAMES + audio_extra
|
||||
request["extensions"]["audio_num_frames"] = (audio_num_frames)
|
||||
request_kwargs["audio_num_frames"] = (audio_num_frames)
|
||||
prefix_sec = float(audio_extra) / 24.0
|
||||
request["extensions"]["video_position_offset_sec"] = prefix_sec
|
||||
request_kwargs["video_position_offset_sec"] = prefix_sec
|
||||
|
||||
new_duration = float(NUM_FRAMES + audio_extra) / 24.0
|
||||
total_T = max(
|
||||
@@ -165,8 +161,8 @@ class ContinuationState:
|
||||
mask = torch.ones((B, 1, total_T, 1), dtype=torch.float32)
|
||||
mask[:, :, :audio_cond_T, :] = (1.0 - AUDIO_CONDITIONING_STRENGTH)
|
||||
|
||||
request["extensions"]["ltx2_audio_clean_latent"] = clean
|
||||
request["extensions"]["ltx2_audio_denoise_mask"] = mask
|
||||
request_kwargs["ltx2_audio_clean_latent"] = clean
|
||||
request_kwargs["ltx2_audio_denoise_mask"] = mask
|
||||
|
||||
def save_video(self, frames: list) -> None:
|
||||
"""Snapshot trailing N frames as PIL images for next-segment conditioning."""
|
||||
@@ -310,7 +306,7 @@ class LTX2GenerationBackend:
|
||||
),
|
||||
)
|
||||
|
||||
self.generator = VideoGenerator.from_config(generator_config)
|
||||
self.generator = VideoGenerator.from_pretrained(config=generator_config)
|
||||
print(f"[GPU {self.gpu_id}] After model load: {self._gpu_mem()}")
|
||||
|
||||
lora_stack = DREAMVERSE_LORA_STACK or ([(DREAMVERSE_LORA_PATH,
|
||||
@@ -407,7 +403,7 @@ class LTX2GenerationBackend:
|
||||
return
|
||||
|
||||
loader = ComponentLoader.for_module_type("audio_encoder", "diffusers")
|
||||
enc = loader.load(audio_vae_path, self.generator.resolved_config)
|
||||
enc = loader.load(audio_vae_path, self.generator.fastvideo_args)
|
||||
target = getattr(enc, "model", enc)
|
||||
|
||||
proc = AudioProcessor(
|
||||
@@ -458,65 +454,61 @@ class LTX2GenerationBackend:
|
||||
segment_idx: int,
|
||||
image_path: str | None,
|
||||
reset_conditioning: bool,
|
||||
*,
|
||||
frame_width: int | None = None,
|
||||
frame_height: int | None = None,
|
||||
num_frames: int | None = None,
|
||||
) -> StepResult:
|
||||
"""Execute one generation step; snapshot state for the next segment."""
|
||||
timings: dict = {}
|
||||
|
||||
prompt = self._inject_style_trigger(prompt)
|
||||
|
||||
request = {
|
||||
"prompt": prompt,
|
||||
"negative_prompt": "",
|
||||
"inputs": {
|
||||
"image_path": image_path if segment_idx == 1 else None
|
||||
},
|
||||
"sampling": {
|
||||
"height": FRAME_HEIGHT,
|
||||
"width": FRAME_WIDTH,
|
||||
"num_frames": NUM_FRAMES,
|
||||
"fps": 24,
|
||||
"num_inference_steps": NUM_INFERENCE_STEPS,
|
||||
"guidance_scale": 1.0,
|
||||
"seed": 10,
|
||||
},
|
||||
"output": {
|
||||
"save_video": False
|
||||
},
|
||||
"extensions": {
|
||||
"ltx2_image_crf": 0.0,
|
||||
"return_continuation_state": False,
|
||||
},
|
||||
}
|
||||
request_kwargs = dict(
|
||||
prompt=prompt,
|
||||
negative_prompt="",
|
||||
save_video=False,
|
||||
height=frame_height or FRAME_HEIGHT,
|
||||
width=frame_width or FRAME_WIDTH,
|
||||
num_frames=num_frames or NUM_FRAMES,
|
||||
fps=24,
|
||||
num_inference_steps=NUM_INFERENCE_STEPS,
|
||||
guidance_scale=1.0,
|
||||
seed=10,
|
||||
ltx2_image_crf=0.0,
|
||||
image_path=image_path if segment_idx == 1 else None,
|
||||
return_continuation_state=False,
|
||||
)
|
||||
|
||||
if reset_conditioning:
|
||||
self.continuation.clear()
|
||||
|
||||
audio_lps = (DEFAULT_LTX2_AUDIO_SAMPLE_RATE / DEFAULT_LTX2_AUDIO_HOP_LENGTH / DEFAULT_LTX2_AUDIO_DOWNSAMPLE)
|
||||
|
||||
# Phase 1: seed the request with prior-segment conditioning.
|
||||
self.continuation.apply_video(request, segment_idx)
|
||||
self.continuation.apply_audio(request, segment_idx, audio_lps)
|
||||
# Phase 1: seed kwargs with prior-segment conditioning.
|
||||
self.continuation.apply_video(request_kwargs, segment_idx)
|
||||
self.continuation.apply_audio(request_kwargs, segment_idx, audio_lps)
|
||||
|
||||
# Phase 2: generate.
|
||||
t0 = time.perf_counter()
|
||||
result = self.generator.generate(request)
|
||||
result = self.generator.generate_video(**request_kwargs)
|
||||
torch.cuda.synchronize()
|
||||
timings["generation_ms"] = (time.perf_counter() - t0) * 1000
|
||||
|
||||
if isinstance(result, list):
|
||||
raise RuntimeError("Expected a single GenerationResult from generate.")
|
||||
frames = result.frames
|
||||
if not isinstance(result, dict):
|
||||
raise RuntimeError("Expected dictionary output from generate_video.")
|
||||
frames = result.get("frames")
|
||||
if not isinstance(frames, list) or len(frames) == 0:
|
||||
raise RuntimeError("Generation did not return frames.")
|
||||
audio = result.audio
|
||||
audio_sample_rate = result.audio_sample_rate
|
||||
audio = result.get("audio")
|
||||
audio_sample_rate = result.get("audio_sample_rate")
|
||||
if audio is not None and audio_sample_rate is None:
|
||||
# LTX2 audio decoding stage uses 24kHz output by default.
|
||||
audio_sample_rate = 24000
|
||||
print(f"[GPU {self.gpu_id}] audio_sample_rate missing from result; "
|
||||
f"defaulting to {audio_sample_rate}Hz")
|
||||
|
||||
timings["generation_time_ms"] = (result.generation_time or 0.0) * 1000
|
||||
timings["generation_time_ms"] = result.get("generation_time", 0.0) * 1000
|
||||
|
||||
# Phase 3: snapshot continuation state for the next segment.
|
||||
t_save_start = time.perf_counter()
|
||||
@@ -554,7 +546,7 @@ class LTX2GenerationBackend:
|
||||
self,
|
||||
audio: object,
|
||||
audio_sample_rate: int | None,
|
||||
result: "GenerationResult",
|
||||
result: dict,
|
||||
segment_idx: int,
|
||||
) -> torch.Tensor | None:
|
||||
"""Pick which tensor to cache for next-segment audio conditioning."""
|
||||
@@ -569,7 +561,7 @@ class LTX2GenerationBackend:
|
||||
f"for segment {segment_idx + 1}")
|
||||
return re_encoded
|
||||
return None
|
||||
audio_latents = result.extra.get("ltx2_audio_latents")
|
||||
audio_latents = result.get("ltx2_audio_latents")
|
||||
if audio_latents is not None:
|
||||
print(f"[GPU {self.gpu_id}] Cached audio latents "
|
||||
f"shape={tuple(audio_latents.shape)} "
|
||||
|
||||
@@ -33,6 +33,7 @@ from dreamverse.routes.presets import (
|
||||
prompt_config_router,
|
||||
curated_presets_router,
|
||||
)
|
||||
from dreamverse.routes.creation import creation_router
|
||||
from dreamverse.session.controller import SessionController
|
||||
|
||||
|
||||
@@ -92,6 +93,7 @@ app.add_middleware(
|
||||
app.include_router(build_health_router(lambda: runtime.gpu_pool))
|
||||
app.include_router(internal_monitor_router)
|
||||
app.include_router(prompt_config_router)
|
||||
app.include_router(creation_router)
|
||||
if DEVTOOLS_ENABLED:
|
||||
app.include_router(curated_presets_router)
|
||||
|
||||
|
||||
@@ -79,24 +79,32 @@ class MiniMaxH3GenerationBackend:
|
||||
|
||||
from fastvideo import VideoGenerator
|
||||
from fastvideo.api import (
|
||||
AttentionConfig,
|
||||
CompileConfig,
|
||||
ComponentConfig,
|
||||
EngineConfig,
|
||||
GeneratorConfig,
|
||||
MiniMaxH3Options,
|
||||
OffloadConfig,
|
||||
ParallelismConfig,
|
||||
PipelineSelection,
|
||||
)
|
||||
|
||||
adapter_path = hf_hub_download(repo_id=adapter_repo, filename=adapter_filename)
|
||||
use_vsa = attention_backend == "VIDEO_SPARSE_ATTN_H3"
|
||||
experimental = {
|
||||
"attention_backend": attention_backend,
|
||||
"inference_torch_compile": attention_backend == "FLASH_ATTN",
|
||||
"vae_parallel_decode": True,
|
||||
"vae_parallel_decode_strategy": "gather",
|
||||
}
|
||||
if attention_backend == "VIDEO_SPARSE_ATTN_H3":
|
||||
experimental.update({
|
||||
"VSA_sparsity": 0.9,
|
||||
"VSA_tile_size": 64,
|
||||
})
|
||||
generator_config = GeneratorConfig(
|
||||
model_path=model_path,
|
||||
pipeline=PipelineSelection(
|
||||
components=ComponentConfig(lora_path=adapter_path, lora_strength=1.0),
|
||||
model=MiniMaxH3Options(vae_parallel_decode=True, vae_parallel_decode_strategy="gather"),
|
||||
experimental=experimental,
|
||||
),
|
||||
engine=EngineConfig(
|
||||
num_gpus=DREAMVERSE_SP_SIZE,
|
||||
@@ -109,12 +117,7 @@ class MiniMaxH3GenerationBackend:
|
||||
vae=True,
|
||||
pin_cpu_memory=True,
|
||||
),
|
||||
compile=CompileConfig(enabled=False, vae_enabled=True, regional=attention_backend == "FLASH_ATTN"),
|
||||
attention=AttentionConfig(
|
||||
backend=attention_backend,
|
||||
vsa_sparsity=0.9 if use_vsa else None,
|
||||
vsa_tile_size=64 if use_vsa else None,
|
||||
),
|
||||
compile=CompileConfig(enabled=False, vae_enabled=True),
|
||||
use_fsdp_inference=False,
|
||||
),
|
||||
)
|
||||
@@ -197,6 +200,10 @@ class MiniMaxH3GenerationBackend:
|
||||
segment_idx: int,
|
||||
image_path: str | None,
|
||||
reset_conditioning: bool,
|
||||
*,
|
||||
frame_width: int | None = None,
|
||||
frame_height: int | None = None,
|
||||
num_frames: int | None = None,
|
||||
) -> StepResult:
|
||||
"""Generate one synchronized FastH3 segment and retain its last frame.
|
||||
|
||||
@@ -204,6 +211,7 @@ class MiniMaxH3GenerationBackend:
|
||||
conditioned frame and its matching audio duration are trimmed before
|
||||
streaming so adjacent segments do not duplicate media.
|
||||
"""
|
||||
del frame_width, frame_height, num_frames
|
||||
if self.generator is None:
|
||||
raise RuntimeError("FastH3 generator is not initialized.")
|
||||
conditioning_image, uses_continuation = self._select_conditioning_image(
|
||||
|
||||
@@ -31,6 +31,8 @@ from fastapi.staticfiles import StaticFiles
|
||||
|
||||
from dreamverse._deps import require_dreamverse_runtime_deps
|
||||
from dreamverse.config import FRONTEND_STATIC_DIR_CANDIDATES, GENERATION_SEGMENT_CAP
|
||||
from dreamverse.creation_capabilities import lobby_capabilities_as_dict
|
||||
from dreamverse.session_creation_config import parse_session_creation_config, validate_generation_mode_assets
|
||||
from dreamverse.session_init_image import cleanup_session_init_image, persist_session_init_image
|
||||
|
||||
LATENCY_MS = 200
|
||||
@@ -225,6 +227,11 @@ async def prompt_system_config():
|
||||
}
|
||||
|
||||
|
||||
@app.get("/creation-capabilities")
|
||||
async def creation_capabilities():
|
||||
return lobby_capabilities_as_dict()
|
||||
|
||||
|
||||
@app.get("/curated-presets")
|
||||
async def curated_presets():
|
||||
presets = [
|
||||
@@ -290,6 +297,8 @@ async def websocket_endpoint(websocket: WebSocket):
|
||||
send_lock = asyncio.Lock()
|
||||
stop_event = asyncio.Event()
|
||||
session_init_image = None
|
||||
session_last_frame_image = None
|
||||
session_creation_config = None
|
||||
|
||||
async def ws_send_json(payload: dict) -> None:
|
||||
async with send_lock:
|
||||
@@ -348,6 +357,7 @@ async def websocket_endpoint(websocket: WebSocket):
|
||||
|
||||
try:
|
||||
session_init_image = persist_session_init_image(init_data.get("initial_image"))
|
||||
session_last_frame_image = persist_session_init_image(init_data.get("last_frame_image"))
|
||||
except ValueError as exc:
|
||||
await ws_send_json({
|
||||
"type": "error",
|
||||
@@ -356,13 +366,31 @@ async def websocket_endpoint(websocket: WebSocket):
|
||||
await websocket.close(code=1003, reason="Invalid initial image")
|
||||
return
|
||||
|
||||
try:
|
||||
session_creation_config = parse_session_creation_config(init_data)
|
||||
validate_generation_mode_assets(
|
||||
session_creation_config.generation_mode,
|
||||
has_initial_image=session_init_image is not None,
|
||||
has_last_frame_image=session_last_frame_image is not None,
|
||||
)
|
||||
except ValueError as exc:
|
||||
await ws_send_json({
|
||||
"type": "error",
|
||||
"message": str(exc),
|
||||
})
|
||||
await websocket.close(code=1003, reason="Invalid creation config")
|
||||
return
|
||||
|
||||
timeout_task = asyncio.create_task(session_timeout())
|
||||
|
||||
await ws_send_json({
|
||||
gpu_assigned_payload: dict[str, object] = {
|
||||
"type": "gpu_assigned",
|
||||
"gpu_id": 0,
|
||||
"session_timeout": SESSION_TIMEOUT_SECONDS,
|
||||
})
|
||||
}
|
||||
if session_creation_config is not None:
|
||||
gpu_assigned_payload["creation_config"] = session_creation_config.as_dict()
|
||||
await ws_send_json(gpu_assigned_payload)
|
||||
|
||||
raw_prompt_queue: asyncio.Queue[PromptSubmission] = asyncio.Queue()
|
||||
ready_prompt_queue: asyncio.Queue[ReadyPrompt] = asyncio.Queue()
|
||||
@@ -391,8 +419,16 @@ async def websocket_endpoint(websocket: WebSocket):
|
||||
if previous_session_image is not None:
|
||||
cleanup_session_init_image(previous_session_image)
|
||||
|
||||
def replace_last_frame_image(last_frame_payload: object) -> None:
|
||||
nonlocal session_last_frame_image
|
||||
next_last_frame_image = persist_session_init_image(last_frame_payload)
|
||||
previous_last_frame_image = session_last_frame_image
|
||||
session_last_frame_image = next_last_frame_image
|
||||
if previous_last_frame_image is not None:
|
||||
cleanup_session_init_image(previous_last_frame_image)
|
||||
|
||||
async def send_stream_start(seed_reason: str) -> None:
|
||||
await ws_send_json({
|
||||
stream_start_payload: dict[str, object] = {
|
||||
"type": "ltx2_stream_start",
|
||||
"total_segments": len(curated_prompts),
|
||||
"preset_id": preset_id,
|
||||
@@ -400,8 +436,15 @@ async def websocket_endpoint(websocket: WebSocket):
|
||||
"live_mode": True,
|
||||
"loop_generation_enabled": loop_generation_enabled,
|
||||
"loop_iteration": loop_iteration,
|
||||
"generation_segment_cap": 0,
|
||||
})
|
||||
"generation_segment_cap": (
|
||||
session_creation_config.generation_segment_cap
|
||||
if session_creation_config is not None
|
||||
else GENERATION_SEGMENT_CAP
|
||||
),
|
||||
}
|
||||
if session_creation_config is not None:
|
||||
stream_start_payload["creation_config"] = session_creation_config.as_dict()
|
||||
await ws_send_json(stream_start_payload)
|
||||
if seed_reason == "init":
|
||||
await ws_send_json({
|
||||
"type": "seed_prompts_updated",
|
||||
@@ -509,6 +552,7 @@ async def websocket_endpoint(websocket: WebSocket):
|
||||
nonlocal project_active
|
||||
nonlocal project_stream_started
|
||||
nonlocal pending_project_end
|
||||
nonlocal session_creation_config
|
||||
|
||||
next_initial_rollout_prompt = str(payload.get("initial_rollout_prompt") or "").strip()
|
||||
next_preset_id = str(payload.get("preset_id") or "").strip()
|
||||
@@ -520,6 +564,21 @@ async def websocket_endpoint(websocket: WebSocket):
|
||||
|
||||
try:
|
||||
replace_session_image(payload.get("initial_image"))
|
||||
replace_last_frame_image(payload.get("last_frame_image"))
|
||||
except ValueError as exc:
|
||||
await ws_send_json({
|
||||
"type": "error",
|
||||
"message": str(exc),
|
||||
})
|
||||
return False
|
||||
|
||||
try:
|
||||
session_creation_config = parse_session_creation_config(payload)
|
||||
validate_generation_mode_assets(
|
||||
session_creation_config.generation_mode,
|
||||
has_initial_image=session_init_image is not None,
|
||||
has_last_frame_image=session_last_frame_image is not None,
|
||||
)
|
||||
except ValueError as exc:
|
||||
await ws_send_json({
|
||||
"type": "error",
|
||||
@@ -1182,6 +1241,7 @@ async def websocket_endpoint(websocket: WebSocket):
|
||||
finally:
|
||||
stop_event.set()
|
||||
cleanup_session_init_image(session_init_image)
|
||||
cleanup_session_init_image(session_last_frame_image)
|
||||
|
||||
|
||||
for static_dir in FRONTEND_STATIC_DIR_CANDIDATES:
|
||||
|
||||
@@ -0,0 +1,14 @@
|
||||
"""Creation studio capability routes."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from fastapi import APIRouter
|
||||
|
||||
from dreamverse.creation_capabilities import lobby_capabilities_as_dict
|
||||
|
||||
creation_router = APIRouter(tags=["creation"])
|
||||
|
||||
|
||||
@creation_router.get("/creation-capabilities")
|
||||
async def creation_capabilities() -> dict[str, object]:
|
||||
return lobby_capabilities_as_dict()
|
||||
@@ -27,6 +27,7 @@ from typing import TYPE_CHECKING
|
||||
from fastapi import WebSocket, WebSocketDisconnect
|
||||
from dreamverse.gpu_pool import GPUSlot
|
||||
from dreamverse.session_init_image import cleanup_session_init_image, persist_session_init_image
|
||||
from dreamverse.session_creation_config import parse_session_creation_config, validate_generation_mode_assets
|
||||
from dreamverse.worker_ipc import MediaChunk, MediaComplete, MediaInit
|
||||
|
||||
from dreamverse.config import (
|
||||
@@ -156,6 +157,9 @@ class SessionController:
|
||||
prompt_worker_task: asyncio.Task | None = None
|
||||
rewrite_seed_prompts_task: asyncio.Task | None = None
|
||||
session_init_image = None
|
||||
session_last_frame_image = None
|
||||
session_creation_config = None
|
||||
session_generation_segment_cap = GENERATION_SEGMENT_CAP
|
||||
|
||||
async def session_timeout():
|
||||
"""Close the session after timeout."""
|
||||
@@ -237,6 +241,7 @@ class SessionController:
|
||||
|
||||
try:
|
||||
session_init_image = persist_session_init_image(init_data.get("initial_image"))
|
||||
session_last_frame_image = persist_session_init_image(init_data.get("last_frame_image"))
|
||||
except ValueError as exc:
|
||||
await ws_send_json({
|
||||
"type": "error",
|
||||
@@ -245,6 +250,22 @@ class SessionController:
|
||||
await websocket.close(code=1003, reason="Invalid initial image")
|
||||
return
|
||||
|
||||
try:
|
||||
session_creation_config = parse_session_creation_config(init_data)
|
||||
session_generation_segment_cap = session_creation_config.generation_segment_cap
|
||||
validate_generation_mode_assets(
|
||||
session_creation_config.generation_mode,
|
||||
has_initial_image=session_init_image is not None,
|
||||
has_last_frame_image=session_last_frame_image is not None,
|
||||
)
|
||||
except ValueError as exc:
|
||||
await ws_send_json({
|
||||
"type": "error",
|
||||
"message": str(exc),
|
||||
})
|
||||
await websocket.close(code=1003, reason="Invalid creation config")
|
||||
return
|
||||
|
||||
if preset_id:
|
||||
print(f"Client {client_id[:8]} selected preset: {preset_id} "
|
||||
f"label={preset_label or '(unset)'} "
|
||||
@@ -256,6 +277,16 @@ class SessionController:
|
||||
if session_init_image is not None:
|
||||
print(f"Client {client_id[:8]} uploaded initial image: "
|
||||
f"{session_init_image.display_name}")
|
||||
if session_last_frame_image is not None:
|
||||
print(f"Client {client_id[:8]} uploaded last frame image: "
|
||||
f"{session_last_frame_image.display_name}")
|
||||
if session_creation_config is not None:
|
||||
print(f"Client {client_id[:8]} creation config: "
|
||||
f"model={session_creation_config.model_id}, "
|
||||
f"mode={session_creation_config.generation_mode}, "
|
||||
f"size={session_creation_config.frame_width}x{session_creation_config.frame_height}, "
|
||||
f"duration={session_creation_config.duration_sec}s, "
|
||||
f"segment_cap={session_creation_config.generation_segment_cap}")
|
||||
|
||||
# Acquire a GPU slot.
|
||||
gpu_id, slot = await self.gpu_pool.acquire(client_id, websocket)
|
||||
@@ -264,14 +295,20 @@ class SessionController:
|
||||
timeout_task = asyncio.create_task(session_timeout())
|
||||
|
||||
# Join the engine on this GPU.
|
||||
await slot.join_user(client_id, model_id=ACTIVE_MODEL_ID)
|
||||
await slot.join_user(
|
||||
client_id,
|
||||
model_id=session_creation_config.model_id if session_creation_config is not None else ACTIVE_MODEL_ID,
|
||||
)
|
||||
|
||||
# Notify client they're connected to a GPU.
|
||||
await ws_send_json({
|
||||
gpu_assigned_payload: dict[str, object] = {
|
||||
"type": "gpu_assigned",
|
||||
"gpu_id": gpu_id,
|
||||
"session_timeout": SESSION_TIMEOUT_SECONDS,
|
||||
})
|
||||
}
|
||||
if session_creation_config is not None:
|
||||
gpu_assigned_payload["creation_config"] = session_creation_config.as_dict()
|
||||
await ws_send_json(gpu_assigned_payload)
|
||||
await log_event(
|
||||
"gpu_assigned",
|
||||
{
|
||||
@@ -315,6 +352,14 @@ class SessionController:
|
||||
if previous_session_init_image is not None:
|
||||
cleanup_session_init_image(previous_session_init_image)
|
||||
|
||||
def replace_last_frame_image(last_frame_payload: object) -> None:
|
||||
nonlocal session_last_frame_image
|
||||
next_last_frame_image = persist_session_init_image(last_frame_payload)
|
||||
previous_last_frame_image = session_last_frame_image
|
||||
session_last_frame_image = next_last_frame_image
|
||||
if previous_last_frame_image is not None:
|
||||
cleanup_session_init_image(previous_last_frame_image)
|
||||
|
||||
async def schedule_simple_generate_request(payload: dict[str, object]) -> None:
|
||||
nonlocal preset_id
|
||||
nonlocal preset_label
|
||||
@@ -452,6 +497,8 @@ class SessionController:
|
||||
nonlocal project_active
|
||||
nonlocal project_stream_started
|
||||
nonlocal pending_project_end
|
||||
nonlocal session_creation_config
|
||||
nonlocal session_generation_segment_cap
|
||||
|
||||
next_initial_rollout_prompt = str(payload.get("initial_rollout_prompt") or "").strip()
|
||||
next_enhancement_enabled = bool(payload.get("enhancement_enabled", True))
|
||||
@@ -498,6 +545,22 @@ class SessionController:
|
||||
|
||||
try:
|
||||
replace_session_init_image(payload.get("initial_image"))
|
||||
replace_last_frame_image(payload.get("last_frame_image"))
|
||||
except ValueError as exc:
|
||||
await ws_send_json({
|
||||
"type": "error",
|
||||
"message": str(exc),
|
||||
})
|
||||
return False
|
||||
|
||||
try:
|
||||
session_creation_config = parse_session_creation_config(payload)
|
||||
session_generation_segment_cap = session_creation_config.generation_segment_cap
|
||||
validate_generation_mode_assets(
|
||||
session_creation_config.generation_mode,
|
||||
has_initial_image=session_init_image is not None,
|
||||
has_last_frame_image=session_last_frame_image is not None,
|
||||
)
|
||||
except ValueError as exc:
|
||||
await ws_send_json({
|
||||
"type": "error",
|
||||
@@ -941,7 +1004,7 @@ class SessionController:
|
||||
"segment_cap":
|
||||
_resolve_generation_segment_cap(
|
||||
single_clip_mode=single_clip_mode,
|
||||
cap=GENERATION_SEGMENT_CAP,
|
||||
cap=session_generation_segment_cap,
|
||||
),
|
||||
})
|
||||
continue
|
||||
@@ -1281,27 +1344,22 @@ class SessionController:
|
||||
))
|
||||
else:
|
||||
project_stream_started = True
|
||||
await ws_send_json({
|
||||
"type":
|
||||
"ltx2_stream_start",
|
||||
"total_segments":
|
||||
len(curated_prompts),
|
||||
"preset_id":
|
||||
preset_id,
|
||||
"stream_mode":
|
||||
"av_fmp4",
|
||||
"live_mode":
|
||||
True,
|
||||
"loop_generation_enabled":
|
||||
loop_generation_enabled,
|
||||
"loop_iteration":
|
||||
loop_iteration,
|
||||
"generation_segment_cap":
|
||||
_resolve_generation_segment_cap(
|
||||
stream_start_payload: dict[str, object] = {
|
||||
"type": "ltx2_stream_start",
|
||||
"total_segments": len(curated_prompts),
|
||||
"preset_id": preset_id,
|
||||
"stream_mode": "av_fmp4",
|
||||
"live_mode": True,
|
||||
"loop_generation_enabled": loop_generation_enabled,
|
||||
"loop_iteration": loop_iteration,
|
||||
"generation_segment_cap": _resolve_generation_segment_cap(
|
||||
single_clip_mode=single_clip_mode,
|
||||
cap=GENERATION_SEGMENT_CAP,
|
||||
cap=session_generation_segment_cap,
|
||||
),
|
||||
})
|
||||
}
|
||||
if session_creation_config is not None:
|
||||
stream_start_payload["creation_config"] = session_creation_config.as_dict()
|
||||
await ws_send_json(stream_start_payload)
|
||||
await ws_send_json({
|
||||
"type": "seed_prompts_updated",
|
||||
"prompts": seed_prompt_memory,
|
||||
@@ -1340,27 +1398,22 @@ class SessionController:
|
||||
|
||||
loop_iteration += 1
|
||||
project_stream_started = True
|
||||
await ws_send_json({
|
||||
"type":
|
||||
"ltx2_stream_start",
|
||||
"total_segments":
|
||||
len(curated_prompts),
|
||||
"preset_id":
|
||||
preset_id,
|
||||
"stream_mode":
|
||||
"av_fmp4",
|
||||
"live_mode":
|
||||
True,
|
||||
"loop_generation_enabled":
|
||||
loop_generation_enabled,
|
||||
"loop_iteration":
|
||||
loop_iteration,
|
||||
"generation_segment_cap":
|
||||
_resolve_generation_segment_cap(
|
||||
restart_stream_payload: dict[str, object] = {
|
||||
"type": "ltx2_stream_start",
|
||||
"total_segments": len(curated_prompts),
|
||||
"preset_id": preset_id,
|
||||
"stream_mode": "av_fmp4",
|
||||
"live_mode": True,
|
||||
"loop_generation_enabled": loop_generation_enabled,
|
||||
"loop_iteration": loop_iteration,
|
||||
"generation_segment_cap": _resolve_generation_segment_cap(
|
||||
single_clip_mode=single_clip_mode,
|
||||
cap=GENERATION_SEGMENT_CAP,
|
||||
cap=session_generation_segment_cap,
|
||||
),
|
||||
})
|
||||
}
|
||||
if session_creation_config is not None:
|
||||
restart_stream_payload["creation_config"] = session_creation_config.as_dict()
|
||||
await ws_send_json(restart_stream_payload)
|
||||
if nonlocal_reason == "loop_restart":
|
||||
await ws_send_json({
|
||||
"type": "loop_restarted",
|
||||
@@ -1389,13 +1442,14 @@ class SessionController:
|
||||
pending_simple_prompt_submission = None
|
||||
|
||||
if (not single_clip_mode and not generation_cap_blocked and not rollout_waiting_for_rewrite
|
||||
and GENERATION_SEGMENT_CAP > 0 and generated_segment_count >= GENERATION_SEGMENT_CAP):
|
||||
and session_generation_segment_cap > 0
|
||||
and generated_segment_count >= session_generation_segment_cap):
|
||||
loop_generation_enabled = False
|
||||
rollout_waiting_for_rewrite = True
|
||||
_main_print(
|
||||
"INFO",
|
||||
f"Segment cap reached for client {client_id[:8]} "
|
||||
f"(cap_segments={GENERATION_SEGMENT_CAP}, "
|
||||
f"(cap_segments={session_generation_segment_cap}, "
|
||||
f"generated_segments={generated_segment_count}); "
|
||||
"waiting for rollout rewrite",
|
||||
)
|
||||
@@ -1620,6 +1674,9 @@ class SessionController:
|
||||
pending_reset_conditioning = False
|
||||
step_image_path = (str(session_init_image.file_path)
|
||||
if segment_idx == 1 and session_init_image is not None else None)
|
||||
step_frame_width = session_creation_config.frame_width if session_creation_config is not None else None
|
||||
step_frame_height = session_creation_config.frame_height if session_creation_config is not None else None
|
||||
step_num_frames = session_creation_config.num_frames if session_creation_config is not None else None
|
||||
step_task = asyncio.create_task(
|
||||
slot.user_step(
|
||||
client_id,
|
||||
@@ -1627,6 +1684,9 @@ class SessionController:
|
||||
segment_idx=segment_idx,
|
||||
image_path=step_image_path,
|
||||
reset_conditioning=step_reset_conditioning,
|
||||
frame_width=step_frame_width,
|
||||
frame_height=step_frame_height,
|
||||
num_frames=step_num_frames,
|
||||
))
|
||||
segment_generation_active = True
|
||||
try:
|
||||
@@ -1808,3 +1868,4 @@ class SessionController:
|
||||
await self.gpu_pool.release(client_id)
|
||||
finally:
|
||||
cleanup_session_init_image(session_init_image)
|
||||
cleanup_session_init_image(session_last_frame_image)
|
||||
|
||||
@@ -0,0 +1,140 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
from dreamverse.config import FRAME_HEIGHT, FRAME_WIDTH, GENERATION_SEGMENT_CAP, MODEL_REGISTRY, NUM_FRAMES
|
||||
from dreamverse.creation_capabilities import validate_lobby_creation_config
|
||||
|
||||
LTX_LOBBY_MODEL_IDS = frozenset(MODEL_REGISTRY.keys())
|
||||
SUPPORTED_GENERATION_MODES = frozenset({"t2va", "fl2va", "ref2va"})
|
||||
SUPPORTED_ASPECT_RATIOS = frozenset({"21:9", "16:9", "4:3", "1:1", "3:4", "9:16"})
|
||||
SUPPORTED_RESOLUTIONS = frozenset({"480p", "720p", "1080p", "4k"})
|
||||
SEGMENT_DURATION_SEC = 5
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SessionCreationConfig:
|
||||
model_id: str
|
||||
generation_mode: str
|
||||
aspect_ratio: str
|
||||
resolution: str
|
||||
duration_sec: int
|
||||
frame_width: int
|
||||
frame_height: int
|
||||
num_frames: int
|
||||
generation_segment_cap: int
|
||||
|
||||
def as_dict(self) -> dict[str, object]:
|
||||
return {
|
||||
"model_id": self.model_id,
|
||||
"generation_mode": self.generation_mode,
|
||||
"aspect_ratio": self.aspect_ratio,
|
||||
"resolution": self.resolution,
|
||||
"duration_sec": self.duration_sec,
|
||||
"frame_width": self.frame_width,
|
||||
"frame_height": self.frame_height,
|
||||
"num_frames": self.num_frames,
|
||||
"generation_segment_cap": self.generation_segment_cap,
|
||||
}
|
||||
|
||||
|
||||
def _round_to_multiple(value: float, multiple: int = 32) -> int:
|
||||
rounded = int(round(value / multiple)) * multiple
|
||||
return max(multiple, rounded)
|
||||
|
||||
|
||||
def _resolution_base(resolution: str) -> int:
|
||||
return {
|
||||
"480p": 480,
|
||||
"720p": 720,
|
||||
"1080p": 1080,
|
||||
"4k": 2160,
|
||||
}.get(resolution, 720)
|
||||
|
||||
|
||||
def resolve_frame_size(aspect_ratio: str, resolution: str) -> tuple[int, int]:
|
||||
if aspect_ratio == "16:9" and resolution == "1080p":
|
||||
return FRAME_WIDTH, FRAME_HEIGHT
|
||||
|
||||
base = _resolution_base(resolution)
|
||||
width_ratio, height_ratio = {
|
||||
"21:9": (21, 9),
|
||||
"16:9": (16, 9),
|
||||
"4:3": (4, 3),
|
||||
"1:1": (1, 1),
|
||||
"3:4": (3, 4),
|
||||
"9:16": (9, 16),
|
||||
}.get(aspect_ratio, (16, 9))
|
||||
|
||||
if width_ratio >= height_ratio:
|
||||
height = _round_to_multiple(base)
|
||||
width = _round_to_multiple(height * width_ratio / height_ratio)
|
||||
else:
|
||||
width = _round_to_multiple(base)
|
||||
height = _round_to_multiple(width * height_ratio / width_ratio)
|
||||
return width, height
|
||||
|
||||
|
||||
def duration_sec_to_segment_cap(duration_sec: int, *, global_cap: int = GENERATION_SEGMENT_CAP) -> int:
|
||||
requested = max(1, int(round(duration_sec / SEGMENT_DURATION_SEC + 0.0001)))
|
||||
if global_cap <= 0:
|
||||
return requested
|
||||
return max(1, min(requested, global_cap))
|
||||
|
||||
|
||||
def parse_session_creation_config(payload: dict[str, object]) -> SessionCreationConfig:
|
||||
raw_model_id = str(payload.get("model_id") or "").strip()
|
||||
model_id = raw_model_id if raw_model_id in LTX_LOBBY_MODEL_IDS else "fast-ltx23"
|
||||
|
||||
generation_mode = str(payload.get("generation_mode") or "t2va").strip()
|
||||
if generation_mode not in SUPPORTED_GENERATION_MODES:
|
||||
raise ValueError(f"Unsupported generation_mode: {generation_mode}")
|
||||
|
||||
aspect_ratio = str(payload.get("aspect_ratio") or "16:9").strip()
|
||||
if aspect_ratio not in SUPPORTED_ASPECT_RATIOS:
|
||||
raise ValueError(f"Unsupported aspect_ratio: {aspect_ratio}")
|
||||
|
||||
resolution = str(payload.get("resolution") or "720p").strip()
|
||||
if resolution not in SUPPORTED_RESOLUTIONS:
|
||||
raise ValueError(f"Unsupported resolution: {resolution}")
|
||||
|
||||
try:
|
||||
duration_sec = int(payload.get("duration_sec") or SEGMENT_DURATION_SEC)
|
||||
except (TypeError, ValueError) as exc:
|
||||
raise ValueError("duration_sec must be an integer.") from exc
|
||||
if duration_sec not in {5, 10, 15}:
|
||||
raise ValueError("duration_sec must be 5, 10, or 15.")
|
||||
|
||||
validate_lobby_creation_config(
|
||||
model_id=model_id,
|
||||
generation_mode=generation_mode,
|
||||
aspect_ratio=aspect_ratio,
|
||||
resolution=resolution,
|
||||
duration_sec=duration_sec,
|
||||
)
|
||||
|
||||
if model_id not in MODEL_REGISTRY:
|
||||
raise ValueError(f"Unsupported model_id: {model_id}")
|
||||
|
||||
frame_width, frame_height = resolve_frame_size(aspect_ratio, resolution)
|
||||
return SessionCreationConfig(
|
||||
model_id=model_id,
|
||||
generation_mode=generation_mode,
|
||||
aspect_ratio=aspect_ratio,
|
||||
resolution=resolution,
|
||||
duration_sec=duration_sec,
|
||||
frame_width=frame_width,
|
||||
frame_height=frame_height,
|
||||
num_frames=NUM_FRAMES,
|
||||
generation_segment_cap=duration_sec_to_segment_cap(duration_sec),
|
||||
)
|
||||
|
||||
|
||||
def validate_generation_mode_assets(
|
||||
generation_mode: str,
|
||||
*,
|
||||
has_initial_image: bool,
|
||||
has_last_frame_image: bool,
|
||||
) -> None:
|
||||
if generation_mode == "ref2va" and not has_initial_image:
|
||||
raise ValueError("Ref2VA mode requires a reference image.")
|
||||
@@ -0,0 +1,74 @@
|
||||
import pytest
|
||||
|
||||
from dreamverse.creation_capabilities import (
|
||||
capabilities_for_model,
|
||||
lobby_capabilities_as_dict,
|
||||
validate_lobby_creation_config,
|
||||
)
|
||||
|
||||
|
||||
def test_lobby_capabilities_include_all_registry_models():
|
||||
caps = lobby_capabilities_as_dict()
|
||||
assert set(caps["model_ids"]) == {"fast-ltx2", "fast-ltx23", "fast-h3"}
|
||||
assert "fl2va" not in caps["generation_modes"]
|
||||
assert "4k" not in caps["resolutions"]
|
||||
|
||||
|
||||
def test_fast_h3_capabilities_use_fixed_geometry():
|
||||
h3_caps = capabilities_for_model("fast-h3")
|
||||
assert h3_caps.generation_modes == frozenset({"t2va", "ref2va"})
|
||||
assert h3_caps.aspect_ratios == frozenset({"16:9"})
|
||||
assert h3_caps.resolutions == frozenset({"720p"})
|
||||
|
||||
|
||||
def test_validate_lobby_creation_config_accepts_supported_t2va():
|
||||
validate_lobby_creation_config(
|
||||
model_id="fast-ltx23",
|
||||
generation_mode="t2va",
|
||||
aspect_ratio="16:9",
|
||||
resolution="1080p",
|
||||
duration_sec=5,
|
||||
)
|
||||
|
||||
|
||||
def test_validate_lobby_creation_config_accepts_fast_h3():
|
||||
validate_lobby_creation_config(
|
||||
model_id="fast-h3",
|
||||
generation_mode="ref2va",
|
||||
aspect_ratio="16:9",
|
||||
resolution="720p",
|
||||
duration_sec=10,
|
||||
)
|
||||
|
||||
|
||||
def test_validate_lobby_creation_config_rejects_fl2va():
|
||||
with pytest.raises(ValueError, match="FL2VA"):
|
||||
validate_lobby_creation_config(
|
||||
model_id="fast-ltx23",
|
||||
generation_mode="fl2va",
|
||||
aspect_ratio="16:9",
|
||||
resolution="720p",
|
||||
duration_sec=5,
|
||||
)
|
||||
|
||||
|
||||
def test_validate_lobby_creation_config_rejects_4k():
|
||||
with pytest.raises(ValueError, match="Unsupported resolution"):
|
||||
validate_lobby_creation_config(
|
||||
model_id="fast-ltx2",
|
||||
generation_mode="t2va",
|
||||
aspect_ratio="16:9",
|
||||
resolution="4k",
|
||||
duration_sec=10,
|
||||
)
|
||||
|
||||
|
||||
def test_validate_lobby_creation_config_rejects_invalid_h3_aspect():
|
||||
with pytest.raises(ValueError, match="Unsupported aspect_ratio"):
|
||||
validate_lobby_creation_config(
|
||||
model_id="fast-h3",
|
||||
generation_mode="t2va",
|
||||
aspect_ratio="9:16",
|
||||
resolution="720p",
|
||||
duration_sec=5,
|
||||
)
|
||||
@@ -13,6 +13,7 @@ FORBIDDEN_PREFIXES = (
|
||||
"fastvideo.models",
|
||||
"fastvideo.layers",
|
||||
"fastvideo.worker",
|
||||
"fastvideo.fastvideo_args",
|
||||
)
|
||||
ALLOWED_INTERNAL_IMPORTS = {
|
||||
(
|
||||
|
||||
@@ -88,13 +88,14 @@ def test_initialize_builds_vsa_datafree_fasth3_generator(monkeypatch):
|
||||
assert config.model_path == "MiniMaxAI/MiniMax-H3"
|
||||
assert config.pipeline.components.lora_path.endswith("vsa-datafree/adapter_model.safetensors")
|
||||
assert config.pipeline.components.lora_strength == 1.0
|
||||
assert config.pipeline.experimental == {}
|
||||
assert config.engine.attention.backend == "VIDEO_SPARSE_ATTN_H3"
|
||||
assert config.engine.attention.vsa_sparsity == 0.9
|
||||
assert config.engine.attention.vsa_tile_size == 64
|
||||
assert config.engine.compile.regional is False
|
||||
assert config.pipeline.model.vae_parallel_decode is True
|
||||
assert config.pipeline.model.vae_parallel_decode_strategy == "gather"
|
||||
assert config.pipeline.experimental == {
|
||||
"attention_backend": "VIDEO_SPARSE_ATTN_H3",
|
||||
"inference_torch_compile": False,
|
||||
"vae_parallel_decode": True,
|
||||
"vae_parallel_decode_strategy": "gather",
|
||||
"VSA_sparsity": 0.9,
|
||||
"VSA_tile_size": 64,
|
||||
}
|
||||
assert config.engine.num_gpus == 4
|
||||
assert config.engine.parallelism.tp_size == 1
|
||||
assert config.engine.parallelism.sp_size == 4
|
||||
|
||||
@@ -0,0 +1,89 @@
|
||||
import pytest
|
||||
|
||||
from dreamverse.session_creation_config import (
|
||||
duration_sec_to_segment_cap,
|
||||
parse_session_creation_config,
|
||||
resolve_frame_size,
|
||||
validate_generation_mode_assets,
|
||||
)
|
||||
|
||||
|
||||
def test_parse_session_creation_config_defaults():
|
||||
config = parse_session_creation_config({})
|
||||
assert config.model_id == "fast-ltx23"
|
||||
assert config.generation_mode == "t2va"
|
||||
assert config.aspect_ratio == "16:9"
|
||||
assert config.resolution == "720p"
|
||||
assert config.duration_sec == 5
|
||||
assert config.generation_segment_cap == 1
|
||||
|
||||
|
||||
def test_parse_session_creation_config_maps_duration_to_segment_cap():
|
||||
config = parse_session_creation_config(
|
||||
{
|
||||
"model_id": "fast-ltx2",
|
||||
"generation_mode": "ref2va",
|
||||
"aspect_ratio": "9:16",
|
||||
"resolution": "480p",
|
||||
"duration_sec": 15,
|
||||
},
|
||||
)
|
||||
assert config.model_id == "fast-ltx2"
|
||||
assert config.generation_mode == "ref2va"
|
||||
assert config.generation_segment_cap == 3
|
||||
assert config.frame_width >= 480
|
||||
assert config.frame_height >= 480
|
||||
|
||||
|
||||
def test_resolve_frame_size_uses_model_default_for_1080p_landscape():
|
||||
width, height = resolve_frame_size("16:9", "1080p")
|
||||
assert (width, height) == (1920, 1088)
|
||||
|
||||
|
||||
def test_duration_sec_to_segment_cap_respects_global_cap():
|
||||
assert duration_sec_to_segment_cap(15, global_cap=2) == 2
|
||||
|
||||
|
||||
def test_parse_session_creation_config_accepts_fast_h3():
|
||||
config = parse_session_creation_config(
|
||||
{
|
||||
"model_id": "fast-h3",
|
||||
"generation_mode": "t2va",
|
||||
"aspect_ratio": "16:9",
|
||||
"resolution": "720p",
|
||||
"duration_sec": 10,
|
||||
},
|
||||
)
|
||||
assert config.model_id == "fast-h3"
|
||||
assert config.generation_mode == "t2va"
|
||||
assert config.generation_segment_cap == 2
|
||||
|
||||
|
||||
def test_parse_session_creation_config_rejects_fl2va():
|
||||
with pytest.raises(ValueError, match="FL2VA"):
|
||||
parse_session_creation_config(
|
||||
{
|
||||
"generation_mode": "fl2va",
|
||||
"aspect_ratio": "16:9",
|
||||
"resolution": "720p",
|
||||
"duration_sec": 5,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def test_parse_session_creation_config_rejects_4k():
|
||||
with pytest.raises(ValueError, match="Unsupported resolution"):
|
||||
parse_session_creation_config(
|
||||
{
|
||||
"generation_mode": "t2va",
|
||||
"aspect_ratio": "16:9",
|
||||
"resolution": "4k",
|
||||
"duration_sec": 5,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def test_validate_generation_mode_assets():
|
||||
validate_generation_mode_assets("t2va", has_initial_image=False, has_last_frame_image=False)
|
||||
with pytest.raises(ValueError, match="Ref2VA"):
|
||||
validate_generation_mode_assets("ref2va", has_initial_image=False, has_last_frame_image=False)
|
||||
@@ -147,6 +147,9 @@ class UserStepPayload:
|
||||
segment_idx: int
|
||||
image_path: str | None
|
||||
reset_conditioning: bool
|
||||
frame_width: int | None = None
|
||||
frame_height: int | None = None
|
||||
num_frames: int | None = None
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
|
||||
@@ -207,7 +207,7 @@
|
||||
</Array>
|
||||
</mxGeometry>
|
||||
</mxCell>
|
||||
<mxCell id="e_dsg" value="generator.generate()" style="edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;strokeColor=#9673a6;endArrow=classic;fontSize=10;exitX=0.5;exitY=1;exitDx=0;exitDy=0;entryX=0.5;entryY=0;entryDx=0;entryDy=0;" parent="1" source="do_step" target="generator" edge="1">
|
||||
<mxCell id="e_dsg" value="generator.generate_video()" style="edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;strokeColor=#9673a6;endArrow=classic;fontSize=10;exitX=0.5;exitY=1;exitDx=0;exitDy=0;entryX=0.5;entryY=0;entryDx=0;entryDy=0;" parent="1" source="do_step" target="generator" edge="1">
|
||||
<mxGeometry relative="1" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="e_dscache" value="read / write" style="edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;strokeColor=#d6b656;endArrow=classic;startArrow=classic;fontSize=10;exitX=0;exitY=0.8;exitDx=0;exitDy=0;entryX=1;entryY=0.2;entryDx=0;entryDy=0;" parent="1" source="do_step" target="caches" edge="1">
|
||||
@@ -534,7 +534,7 @@
|
||||
<mxPoint x="1040" y="1610" as="targetPoint"/>
|
||||
</mxGeometry>
|
||||
</mxCell>
|
||||
<mxCell id="dm11a" value="10a. worker runs:
VideoGenerationWorker.generate_step()
 (ltx2_generation.py:380)
 → generator.generate()
 → updates ContinuationState
then stream_fmp4() (av_streaming.py:121)
 → ffmpeg (rawvideo+wav → fmp4)" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#ffe0b2;strokeColor=#d79b00;fontSize=10;align=left;spacingLeft=8;fontFamily=monospace;" parent="1" vertex="1">
|
||||
<mxCell id="dm11a" value="10a. worker runs:
VideoGenerationWorker.generate_step()
 (ltx2_generation.py:380)
 → generator.generate_video()
 → updates ContinuationState
then stream_fmp4() (av_streaming.py:121)
 → ffmpeg (rawvideo+wav → fmp4)" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#ffe0b2;strokeColor=#d79b00;fontSize=10;align=left;spacingLeft=8;fontFamily=monospace;" parent="1" vertex="1">
|
||||
<mxGeometry x="955" y="1640" width="180" height="70" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="dm11" value="10b. resp_q.put(MediaInit / MediaChunk / MediaComplete / StepComplete)" style="endArrow=classic;html=1;strokeColor=#b85450;fontSize=10;labelBackgroundColor=#ffffff;" parent="1" edge="1">
|
||||
|
||||
File diff suppressed because one or more lines are too long
|
Before Width: | Height: | Size: 85 KiB After Width: | Height: | Size: 85 KiB |
@@ -64,8 +64,8 @@ generator:
|
||||
# internal: pipeline_config.dit_config.quant_config = FP4Config()
|
||||
# set in gpu_pool.py:280 (via the legacy in-place mutation). The
|
||||
# public typed surface resolves "NVFP4" to NVFP4Config() and pins
|
||||
# it on dit_config when resolution materializes the PipelineConfig.
|
||||
# Comment this block out on hosts without flashinfer / NVFP4 hardware.
|
||||
# it on dit_config in FastVideoArgs.__post_init__. Comment this
|
||||
# block out on hosts without flashinfer / NVFP4 hardware.
|
||||
quantization:
|
||||
transformer_quant: NVFP4
|
||||
|
||||
|
||||
@@ -17,25 +17,17 @@ test.describe('frontend shell', () => {
|
||||
});
|
||||
});
|
||||
|
||||
test('composer hydrates with curated preset cards', async ({ page }) => {
|
||||
test('composer hydrates with creation studio controls', async ({ page }) => {
|
||||
await page.goto('/');
|
||||
|
||||
// The Continuation prompt textarea + Generate button render once
|
||||
// the FE has hydrated against the public-FastVideo-backed
|
||||
// dreamverse-server. Their presence proves the integration handshake
|
||||
// (CORS, /curated-presets, /prompt-system-config) completed.
|
||||
const continuation = page.getByLabel('Continuation prompt');
|
||||
await expect(continuation).toBeVisible({ timeout: 30_000 });
|
||||
|
||||
const generate = page.getByRole('button', { name: /^generate$/i });
|
||||
await expect(generate).toBeVisible({ timeout: 30_000 });
|
||||
|
||||
// Curated presets render as buttons; verify at least one is
|
||||
// available — that's the only way the user can populate the
|
||||
// Continuation textarea in the default composer.
|
||||
const presetCard = page.getByRole('button', {
|
||||
name: /LEGO Stormtroopers|Clay Stop-Motion|Boy & Dog|School Prank|Gamer Gets Banned|Small Town Oil Strike|Grandpa's Wing Costume/i,
|
||||
}).first();
|
||||
await expect(presetCard).toBeVisible({ timeout: 30_000 });
|
||||
await expect(page.getByText('Direct scenes in seconds')).toBeVisible({ timeout: 30_000 });
|
||||
await expect(page.getByRole('button', { name: /FastLTX/i }).first()).toBeVisible({ timeout: 30_000 });
|
||||
await expect(continuation).toHaveAttribute('placeholder', /Describe your video or mention elements/i);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -67,7 +67,7 @@ test.describe('preset prompt generation', () => {
|
||||
// on a B200 plus encode/transfer time. The "Continuation flipped
|
||||
// to Generating + Leave button rendered" pair above is the proof
|
||||
// the integration works: FE → /readyz → /curated-presets → WS
|
||||
// /ws → BE → GPU pool → VideoGenerator.generate, all green.
|
||||
// /ws → BE → GPU pool → VideoGenerator.generate_video, all green.
|
||||
const video = page.locator('video').first();
|
||||
await expect(video).toHaveCount(1);
|
||||
});
|
||||
|
||||
@@ -42,6 +42,10 @@ const nextConfig: NextConfig = {
|
||||
source: '/prompt-system-config',
|
||||
destination: `${backendUrl}/prompt-system-config`,
|
||||
},
|
||||
{
|
||||
source: '/creation-capabilities',
|
||||
destination: `${backendUrl}/creation-capabilities`,
|
||||
},
|
||||
{
|
||||
source: '/curated-presets',
|
||||
destination: `${backendUrl}/curated-presets`,
|
||||
|
||||
Generated
+2262
-113
File diff suppressed because it is too large
Load Diff
@@ -27,11 +27,15 @@
|
||||
"@radix-ui/react-accordion": "^1.2.12",
|
||||
"@radix-ui/react-checkbox": "^1.3.3",
|
||||
"@radix-ui/react-collapsible": "^1.1.12",
|
||||
"@radix-ui/react-dropdown-menu": "^2.1.24",
|
||||
"@radix-ui/react-label": "^2.1.8",
|
||||
"@radix-ui/react-popover": "^1.1.23",
|
||||
"@radix-ui/react-scroll-area": "^1.2.10",
|
||||
"@radix-ui/react-select": "^2.2.6",
|
||||
"@radix-ui/react-separator": "^1.1.8",
|
||||
"@radix-ui/react-slider": "^1.4.7",
|
||||
"@radix-ui/react-slot": "^1.2.4",
|
||||
"@radix-ui/react-tabs": "^1.1.21",
|
||||
"class-variance-authority": "^0.7.1",
|
||||
"clsx": "^2.1.1",
|
||||
"framer-motion": "^12.36.0",
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
@import "tailwindcss";
|
||||
|
||||
@custom-variant dark (&:is(.dark *));
|
||||
@custom-variant hover-capable (@media (hover: hover) and (pointer: fine));
|
||||
|
||||
:root {
|
||||
color-scheme: light;
|
||||
@@ -201,7 +202,7 @@ summary::-webkit-details-marker {
|
||||
@apply border-border;
|
||||
}
|
||||
body {
|
||||
@apply bg-background text-foreground;
|
||||
@apply bg-background text-foreground antialiased;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -220,6 +221,14 @@ summary::-webkit-details-marker {
|
||||
.stroke-dash-anim {
|
||||
animation: stroke-dash-animation 2s linear infinite;
|
||||
}
|
||||
|
||||
.text-pretty {
|
||||
text-wrap: pretty;
|
||||
}
|
||||
|
||||
.text-balance {
|
||||
text-wrap: balance;
|
||||
}
|
||||
}
|
||||
|
||||
@keyframes stroke-dash-animation {
|
||||
@@ -287,6 +296,22 @@ html.theme-transition *::after {
|
||||
/* —————————————— CUSTOM TAILWIND —————————————— */
|
||||
|
||||
@layer components {
|
||||
.studio-control {
|
||||
@apply transition-[border-color,background-color,box-shadow,color,transform] duration-150 focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-accent-blue/40;
|
||||
}
|
||||
|
||||
.studio-control-press {
|
||||
@apply active:scale-[0.96] motion-reduce:active:scale-100;
|
||||
}
|
||||
|
||||
.studio-hover-surface {
|
||||
@apply hover-capable:hover:border-border hover-capable:hover:bg-accent/50;
|
||||
}
|
||||
|
||||
.studio-media-outline {
|
||||
@apply outline outline-1 -outline-offset-1 outline-black/10 dark:outline-white/10;
|
||||
}
|
||||
|
||||
.debug {
|
||||
@apply border border-rose-500;
|
||||
}
|
||||
|
||||
@@ -1,10 +1,20 @@
|
||||
"use client";
|
||||
import { Fragment, useCallback, useEffect, useLayoutEffect, useMemo, useRef, useState } from "react";
|
||||
import { Fragment, useCallback, useEffect, useLayoutEffect, useMemo, useRef, useState, type Dispatch, type SetStateAction } from "react";
|
||||
import { AnimatePresence, motion } from "framer-motion";
|
||||
import { Download, Share2 } from "lucide-react";
|
||||
import DevtoolsShell from "@/components/devtools/DevtoolsShell";
|
||||
import MonitorPage from "@/components/MonitorPage";
|
||||
import ChatBar from "@/components/ChatBar";
|
||||
import CreationStudio from "@/components/creation/CreationStudio";
|
||||
import {
|
||||
buildMentionOptions,
|
||||
type AspectRatioId,
|
||||
type CreationModeId,
|
||||
type CreationModelId,
|
||||
type ResolutionId,
|
||||
} from "@/lib/creationConfig";
|
||||
import { toGenerationMode } from "@/lib/generationMode";
|
||||
import type { SessionCreationConfig } from "@/components/creation/SessionCreationConfigPills";
|
||||
import SessionTimeoutModal from "@/components/SessionTimeoutModal";
|
||||
import Sidebar from "@/components/Sidebar";
|
||||
import Header from "@/components/Header";
|
||||
@@ -22,6 +32,18 @@ import {
|
||||
buildRewritePromptWindowSnapshotFromPrompts,
|
||||
normalizePromptWindowSnapshot,
|
||||
} from "@/lib/prompts/promptWindowSnapshot";
|
||||
import {
|
||||
DEFAULT_LOBBY_CAPABILITIES_BUNDLE,
|
||||
clampLobbySelectionToCapabilities,
|
||||
parseLobbyCapabilitiesBundle,
|
||||
resolveModelCapabilities,
|
||||
validateLobbyCreationSelection,
|
||||
type LobbyCapabilitiesBundle,
|
||||
} from "@/lib/creationCapabilities";
|
||||
import {
|
||||
buildCreationInitPayload,
|
||||
parseEchoedCreationConfig,
|
||||
} from "@/lib/creationPayload";
|
||||
import rawPresets from "@/lib/storyPresetsData";
|
||||
import { cn } from "@/lib/utils";
|
||||
import { createWebSocketConnection, detachAndCloseWebSocket } from "@/lib/ws/client";
|
||||
@@ -78,98 +100,6 @@ function yieldToEventLoop(): Promise<void> {
|
||||
return new Promise((r) => setTimeout(r, 0));
|
||||
}
|
||||
|
||||
const HERO_WAVE_LIGHT = ["#2A4A98", "#4878E5", "#6FA0F2", "#B0BCC8", "#E8D99E", "#D8C844", "#C2A620"];
|
||||
const HERO_WAVE_DARK = ["#143468", "#1E58B8", "#3892F0", "#80B8E8", "#B8D0EA", "#E2D498", "#DABB50"];
|
||||
const HERO_TEXT = "Direct scenes in seconds";
|
||||
|
||||
function HeroTagline() {
|
||||
const ref = useRef<HTMLHeadingElement>(null);
|
||||
|
||||
useEffect(() => {
|
||||
const el = ref.current;
|
||||
if (!el) return;
|
||||
|
||||
let rafId = 0;
|
||||
|
||||
function play() {
|
||||
const chars = el!.querySelectorAll<HTMLSpanElement>("[data-char]");
|
||||
if (!chars.length) return;
|
||||
cancelAnimationFrame(rafId);
|
||||
|
||||
const isDark = document.documentElement.classList.contains("dark");
|
||||
const colors = isDark ? HERO_WAVE_DARK : HERO_WAVE_LIGHT;
|
||||
const waveLen = 10;
|
||||
const total = chars.length + waveLen;
|
||||
const duration = 1200;
|
||||
const maxBlur = 3.5;
|
||||
const start = performance.now();
|
||||
|
||||
function tick() {
|
||||
const t = Math.min((performance.now() - start) / duration, 1);
|
||||
const pos = t * total;
|
||||
chars.forEach((ch, i) => {
|
||||
const rel = pos - i;
|
||||
if (rel >= 0 && rel < waveLen) {
|
||||
const norm = rel / waveLen;
|
||||
const ci = Math.floor(norm * colors.length);
|
||||
ch.style.color = colors[Math.min(colors.length - 1, ci)];
|
||||
|
||||
let blur = 0;
|
||||
if (norm < 0.25) {
|
||||
blur = maxBlur * (1 - norm / 0.25);
|
||||
} else if (norm > 0.75) {
|
||||
blur = maxBlur * ((norm - 0.75) / 0.25);
|
||||
}
|
||||
ch.style.filter = blur > 0.1 ? `blur(${blur.toFixed(1)}px)` : "";
|
||||
} else {
|
||||
ch.style.color = "";
|
||||
ch.style.filter = "";
|
||||
}
|
||||
});
|
||||
if (t < 1) {
|
||||
rafId = requestAnimationFrame(tick);
|
||||
} else {
|
||||
chars.forEach((ch) => {
|
||||
ch.style.color = "";
|
||||
ch.style.filter = "";
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
rafId = requestAnimationFrame(tick);
|
||||
}
|
||||
|
||||
const initialDelay = setTimeout(play, 400);
|
||||
const interval = setInterval(play, 5000);
|
||||
return () => {
|
||||
clearTimeout(initialDelay);
|
||||
clearInterval(interval);
|
||||
cancelAnimationFrame(rafId);
|
||||
};
|
||||
}, []);
|
||||
|
||||
return (
|
||||
<h1 ref={ref} className="text-center text-3xl font-medium text-[#343537] dark:text-[#FAFAFB] sm:text-4xl">
|
||||
{HERO_TEXT.split(" ").map((word, wi) => (
|
||||
<Fragment key={wi}>
|
||||
{wi > 0 && (
|
||||
<span data-char className="transition-[color,filter] duration-150">
|
||||
{" "}
|
||||
</span>
|
||||
)}
|
||||
<span className="inline-flex">
|
||||
{word.split("").map((char, ci) => (
|
||||
<span key={ci} data-char className="inline-block transition-[color,filter] duration-150">
|
||||
{char}
|
||||
</span>
|
||||
))}
|
||||
</span>
|
||||
</Fragment>
|
||||
))}
|
||||
</h1>
|
||||
);
|
||||
}
|
||||
|
||||
export default function Page() {
|
||||
const storesRef = useRef<PageStores | null>(null);
|
||||
if (!storesRef.current) {
|
||||
@@ -323,8 +253,33 @@ export default function Page() {
|
||||
const [ttffValueMs, setTtffValueMs] = useState<number | null>(null);
|
||||
const ttffIntervalRef = useRef<ReturnType<typeof setInterval> | null>(null);
|
||||
const pendingInitialPromptRef = useRef("");
|
||||
const referenceFileRef = useRef<File | null>(null);
|
||||
const firstFrameFileRef = useRef<File | null>(null);
|
||||
const lastFrameFileRef = useRef<File | null>(null);
|
||||
const lastArchivedReplayKeyRef = useRef("");
|
||||
const [sidebarOpen, setSidebarOpen] = useState(false);
|
||||
const [creationModelId, setCreationModelId] = useState<CreationModelId>("fast-ltx23");
|
||||
const [creationModeId, setCreationModeId] = useState<CreationModeId>("t2v");
|
||||
const [creationAspectRatio, setCreationAspectRatio] = useState<AspectRatioId>("16:9");
|
||||
const [creationResolution, setCreationResolution] = useState<ResolutionId>("720p");
|
||||
const [creationDurationSec, setCreationDurationSec] = useState(5);
|
||||
const [lobbyCapabilitiesBundle, setLobbyCapabilitiesBundle] = useState<LobbyCapabilitiesBundle>(
|
||||
DEFAULT_LOBBY_CAPABILITIES_BUNDLE,
|
||||
);
|
||||
const activeModelCapabilities = useMemo(
|
||||
() => resolveModelCapabilities(lobbyCapabilitiesBundle, creationModelId),
|
||||
[lobbyCapabilitiesBundle, creationModelId],
|
||||
);
|
||||
const [sessionCreationConfig, setSessionCreationConfig] = useState<SessionCreationConfig>({
|
||||
modelId: "fast-ltx23",
|
||||
modeId: "t2v",
|
||||
aspectRatio: "16:9",
|
||||
resolution: "720p",
|
||||
durationSec: 5,
|
||||
});
|
||||
const [referencePreviewUrl, setReferencePreviewUrl] = useState<string | null>(null);
|
||||
const [firstFramePreviewUrl, setFirstFramePreviewUrl] = useState<string | null>(null);
|
||||
const [lastFramePreviewUrl, setLastFramePreviewUrl] = useState<string | null>(null);
|
||||
const [currentThumbnail, setCurrentThumbnail] = useState<string | null>(null);
|
||||
const currentProjectIdRef = useRef("");
|
||||
const currentProjectCreatedAtRef = useRef(0);
|
||||
@@ -345,6 +300,57 @@ export default function Page() {
|
||||
setIsMobileShareCapable(typeof navigator.canShare === "function" && window.matchMedia("(pointer: coarse)").matches);
|
||||
}, []);
|
||||
|
||||
useEffect(() => {
|
||||
return () => {
|
||||
if (referencePreviewUrl) {
|
||||
URL.revokeObjectURL(referencePreviewUrl);
|
||||
}
|
||||
if (firstFramePreviewUrl) {
|
||||
URL.revokeObjectURL(firstFramePreviewUrl);
|
||||
}
|
||||
if (lastFramePreviewUrl) {
|
||||
URL.revokeObjectURL(lastFramePreviewUrl);
|
||||
}
|
||||
};
|
||||
}, [referencePreviewUrl, firstFramePreviewUrl, lastFramePreviewUrl]);
|
||||
|
||||
function setPreviewUrl(setter: Dispatch<SetStateAction<string | null>>, file: File | null) {
|
||||
setter((current) => {
|
||||
if (current) URL.revokeObjectURL(current);
|
||||
return file ? URL.createObjectURL(file) : null;
|
||||
});
|
||||
}
|
||||
|
||||
function handleReferenceSelect(file: File | null) {
|
||||
referenceFileRef.current = file;
|
||||
setPreviewUrl(setReferencePreviewUrl, file);
|
||||
}
|
||||
|
||||
function handleFirstFrameSelect(file: File | null) {
|
||||
firstFrameFileRef.current = file;
|
||||
setPreviewUrl(setFirstFramePreviewUrl, file);
|
||||
}
|
||||
|
||||
function handleLastFrameSelect(file: File | null) {
|
||||
lastFrameFileRef.current = file;
|
||||
setPreviewUrl(setLastFramePreviewUrl, file);
|
||||
}
|
||||
|
||||
const mentionOptions = useMemo(() => buildMentionOptions(storyPresets as Array<{ id?: string; label?: string; description?: string }>), [storyPresets]);
|
||||
|
||||
const lobbyStoryPresets = useMemo(
|
||||
() =>
|
||||
(storyPresets as Array<{ id?: string; label?: string; description?: string; segment_prompts?: unknown }>)
|
||||
.filter((preset) => typeof preset.id === "string" && typeof preset.label === "string")
|
||||
.map((preset) => ({
|
||||
id: String(preset.id),
|
||||
label: String(preset.label),
|
||||
description: typeof preset.description === "string" ? preset.description : undefined,
|
||||
segmentCount: Array.isArray(preset.segment_prompts) ? preset.segment_prompts.length : undefined,
|
||||
})),
|
||||
[storyPresets],
|
||||
);
|
||||
|
||||
const videoElRef = useRef<HTMLVideoElement | null>(null);
|
||||
const archivedPlaybackElRef = useRef<HTMLVideoElement | null>(null);
|
||||
const viewingModePlaybackStateRef = useRef<{
|
||||
@@ -524,6 +530,65 @@ export default function Page() {
|
||||
setRuntimeReady(true);
|
||||
}, []);
|
||||
|
||||
function applyLobbyCapabilitiesBundle(bundle: LobbyCapabilitiesBundle) {
|
||||
setLobbyCapabilitiesBundle(bundle);
|
||||
const clamped = clampLobbySelectionToCapabilities({
|
||||
capabilities: resolveModelCapabilities(bundle, creationModelId),
|
||||
modelId: creationModelId,
|
||||
modeId: creationModeId,
|
||||
aspectRatio: creationAspectRatio,
|
||||
resolution: creationResolution,
|
||||
durationSec: creationDurationSec,
|
||||
});
|
||||
setCreationModelId(clamped.modelId);
|
||||
setCreationModeId(clamped.modeId);
|
||||
setCreationAspectRatio(clamped.aspectRatio);
|
||||
setCreationResolution(clamped.resolution);
|
||||
setCreationDurationSec(clamped.durationSec);
|
||||
}
|
||||
|
||||
function handleCreationModelChange(modelId: CreationModelId) {
|
||||
const clamped = clampLobbySelectionToCapabilities({
|
||||
capabilities: resolveModelCapabilities(lobbyCapabilitiesBundle, modelId),
|
||||
modelId,
|
||||
modeId: creationModeId,
|
||||
aspectRatio: creationAspectRatio,
|
||||
resolution: creationResolution,
|
||||
durationSec: creationDurationSec,
|
||||
});
|
||||
setCreationModelId(clamped.modelId);
|
||||
setCreationModeId(clamped.modeId);
|
||||
setCreationAspectRatio(clamped.aspectRatio);
|
||||
setCreationResolution(clamped.resolution);
|
||||
setCreationDurationSec(clamped.durationSec);
|
||||
}
|
||||
|
||||
useEffect(() => {
|
||||
if (!runtimeReady) return;
|
||||
let cancelled = false;
|
||||
void fetch("/creation-capabilities", {
|
||||
headers: { Accept: "application/json" },
|
||||
cache: "no-store",
|
||||
})
|
||||
.then(async (response) => {
|
||||
if (!response.ok) return DEFAULT_LOBBY_CAPABILITIES_BUNDLE;
|
||||
return parseLobbyCapabilitiesBundle(await response.json());
|
||||
})
|
||||
.then((bundle) => {
|
||||
if (!cancelled) {
|
||||
applyLobbyCapabilitiesBundle(bundle);
|
||||
}
|
||||
})
|
||||
.catch(() => {
|
||||
if (!cancelled) {
|
||||
applyLobbyCapabilitiesBundle(DEFAULT_LOBBY_CAPABILITIES_BUNDLE);
|
||||
}
|
||||
});
|
||||
return () => {
|
||||
cancelled = true;
|
||||
};
|
||||
}, [runtimeReady]);
|
||||
|
||||
useEffect(() => {
|
||||
if (!runtimeReady || initializedRef.current) return;
|
||||
initializedRef.current = true;
|
||||
@@ -1759,33 +1824,44 @@ export default function Page() {
|
||||
resetPlaybackState();
|
||||
}
|
||||
|
||||
function buildProjectInitPayload(type: "session_init_v2" | "project_init_v1") {
|
||||
async function buildProjectInitPayload(type: "session_init_v2" | "project_init_v1") {
|
||||
const segmentPrompts = getSessionInitPrompts();
|
||||
setSeedPrompts(segmentPrompts);
|
||||
const creationPayload = await buildCreationInitPayload({
|
||||
modelId: creationModelId,
|
||||
modeId: creationModeId,
|
||||
aspectRatio: creationAspectRatio,
|
||||
resolution: creationResolution,
|
||||
durationSec: creationDurationSec,
|
||||
referenceFile: referenceFileRef.current,
|
||||
firstFrameFile: firstFrameFileRef.current,
|
||||
lastFrameFile: lastFrameFileRef.current,
|
||||
});
|
||||
return {
|
||||
type,
|
||||
generation_mode: toGenerationMode(creationModeId),
|
||||
preset_id: getInitialPresetId(),
|
||||
preset_label: getInitialPresetLabel(),
|
||||
curated_prompts: segmentPrompts,
|
||||
initial_rollout_prompt: normalizeInitialPrompt(pendingInitialPromptRef.current),
|
||||
initial_image: null,
|
||||
single_clip_mode: false,
|
||||
enhancement_enabled: sessionStore.get().enhancementEnabled,
|
||||
auto_extension_enabled: sessionStore.get().autoExtensionEnabled,
|
||||
loop_generation_enabled: sessionStore.get().loopGenerationEnabled,
|
||||
...creationPayload,
|
||||
};
|
||||
}
|
||||
|
||||
function sendSessionInitMessage() {
|
||||
async function sendSessionInitMessage() {
|
||||
const ws = wsRef.current;
|
||||
if (!ws) return;
|
||||
ws.send(JSON.stringify(buildProjectInitPayload("session_init_v2")));
|
||||
ws.send(JSON.stringify(await buildProjectInitPayload("session_init_v2")));
|
||||
}
|
||||
|
||||
function sendProjectInitMessage() {
|
||||
async function sendProjectInitMessage() {
|
||||
const ws = wsRef.current;
|
||||
if (!ws || ws.readyState !== WebSocket.OPEN) return;
|
||||
ws.send(JSON.stringify(buildProjectInitPayload("project_init_v1")));
|
||||
ws.send(JSON.stringify(await buildProjectInitPayload("project_init_v1")));
|
||||
}
|
||||
|
||||
function sendEndProjectKeepSession() {
|
||||
@@ -1822,6 +1898,9 @@ export default function Page() {
|
||||
return;
|
||||
}
|
||||
const normalizedEvent = normalizeSocketMessage(decoded.data);
|
||||
if (decoded.data?.type === "gpu_assigned" || decoded.data?.type === "ltx2_stream_start") {
|
||||
applyEchoedCreationConfig(decoded.data);
|
||||
}
|
||||
await applyNormalizedSocketEvent(normalizedEvent, {
|
||||
sessionStore,
|
||||
promptWindowStore,
|
||||
@@ -1864,7 +1943,12 @@ export default function Page() {
|
||||
onOpen: () => {
|
||||
opened = true;
|
||||
sessionStore.patch({ connected: true, connecting: false });
|
||||
sendSessionInitMessage();
|
||||
void sendSessionInitMessage().catch((error) => {
|
||||
console.error("Failed to send session init payload:", error);
|
||||
recoverFailedSessionStart(
|
||||
error instanceof Error ? error.message : "Failed to prepare session settings.",
|
||||
);
|
||||
});
|
||||
},
|
||||
onMessage: (event: MessageEvent) => {
|
||||
wsMessageQueueRef.current = wsMessageQueueRef.current
|
||||
@@ -1937,10 +2021,29 @@ export default function Page() {
|
||||
}
|
||||
}
|
||||
|
||||
function syncSessionCreationConfigFromLobby() {
|
||||
setSessionCreationConfig({
|
||||
modelId: creationModelId,
|
||||
modeId: creationModeId,
|
||||
aspectRatio: creationAspectRatio,
|
||||
resolution: creationResolution,
|
||||
durationSec: creationDurationSec,
|
||||
});
|
||||
}
|
||||
|
||||
function applyEchoedCreationConfig(data: unknown) {
|
||||
const echoed = parseEchoedCreationConfig(data);
|
||||
if (!echoed) {
|
||||
return;
|
||||
}
|
||||
setSessionCreationConfig(echoed);
|
||||
}
|
||||
|
||||
function beginProjectLocally({ force = false } = {}) {
|
||||
if (!force && !canStartSession) return;
|
||||
if (sessionStore.get().sessionStarted || sessionStore.get().projectResetPending) return false;
|
||||
setTimeoutModalOpen(false);
|
||||
syncSessionCreationConfigFromLobby();
|
||||
// Unmute during the user gesture so iOS Safari permits audio playback.
|
||||
setVideoMuted(false);
|
||||
if (viewingProject) closeViewingProject();
|
||||
@@ -1999,13 +2102,34 @@ export default function Page() {
|
||||
}
|
||||
|
||||
async function joinSession({ force = false } = {}) {
|
||||
const validationError = validateLobbyCreationSelection({
|
||||
capabilities: activeModelCapabilities,
|
||||
modelId: creationModelId,
|
||||
modeId: creationModeId,
|
||||
aspectRatio: creationAspectRatio,
|
||||
resolution: creationResolution,
|
||||
durationSec: creationDurationSec,
|
||||
referenceFile: referenceFileRef.current,
|
||||
firstFrameFile: firstFrameFileRef.current,
|
||||
lastFrameFile: lastFrameFileRef.current,
|
||||
});
|
||||
if (validationError) {
|
||||
showPreSessionNotice(validationError);
|
||||
return;
|
||||
}
|
||||
|
||||
if (
|
||||
wsRef.current
|
||||
&& wsRef.current.readyState === WebSocket.OPEN
|
||||
&& sessionStore.get().connected
|
||||
) {
|
||||
if (!beginProjectLocally({ force })) return;
|
||||
sendProjectInitMessage();
|
||||
try {
|
||||
await sendProjectInitMessage();
|
||||
} catch (error) {
|
||||
console.error("Failed to send project init payload:", error);
|
||||
showPreSessionNotice(error instanceof Error ? error.message : "Failed to prepare session settings.");
|
||||
}
|
||||
return;
|
||||
}
|
||||
showPreSessionNotice("");
|
||||
@@ -2027,7 +2151,12 @@ export default function Page() {
|
||||
&& wsRef.current.readyState === WebSocket.OPEN
|
||||
&& sessionStore.get().connected
|
||||
) {
|
||||
sendProjectInitMessage();
|
||||
try {
|
||||
await sendProjectInitMessage();
|
||||
} catch (error) {
|
||||
console.error("Failed to send project init payload:", error);
|
||||
showPreSessionNotice(error instanceof Error ? error.message : "Failed to prepare session settings.");
|
||||
}
|
||||
return;
|
||||
}
|
||||
connectWebSocket();
|
||||
@@ -2641,7 +2770,7 @@ export default function Page() {
|
||||
/>
|
||||
<Header timeLeft={headerTimeLeft} formatTime={formatTime} onToggleSidebar={() => setSidebarOpen((prev) => !prev)} />
|
||||
|
||||
<div className="relative flex flex-1 min-h-0 flex-col justify-center px-4 pb-2 sm:px-6 sm:pb-12">
|
||||
<div className={cn("relative flex flex-1 min-h-0 flex-col", showActiveProject || isViewingMode ? "justify-center px-4 pb-2 sm:px-6 sm:pb-12" : "overflow-hidden")}>
|
||||
{isViewingMode && (
|
||||
<>
|
||||
{viewingSelectedClip && (
|
||||
@@ -2758,44 +2887,79 @@ export default function Page() {
|
||||
/>
|
||||
</section>
|
||||
|
||||
<AnimatePresence>
|
||||
{!showActiveProject && (
|
||||
<motion.div
|
||||
key="hero-tagline"
|
||||
initial={{ opacity: 0 }}
|
||||
animate={{ opacity: 1 }}
|
||||
exit={{ opacity: 0, transition: { duration: 0.2, ease: "easeIn" } }}
|
||||
transition={{ duration: 0.5, ease: "easeOut" }}
|
||||
className="pointer-events-none absolute inset-x-0 top-0 bottom-1/2 z-10 flex items-center justify-center px-4"
|
||||
>
|
||||
<HeroTagline />
|
||||
</motion.div>
|
||||
)}
|
||||
</AnimatePresence>
|
||||
|
||||
<motion.div layout="position" className="mx-auto w-full max-w-2xl shrink-0" transition={{ type: "spring", stiffness: 200, damping: 25 }}>
|
||||
<ChatBar
|
||||
sessionStarted={sessionStarted as boolean}
|
||||
rewritingSeedPrompts={rewritingSeedPrompts as boolean}
|
||||
{!showActiveProject ? (
|
||||
<CreationStudio
|
||||
value={livePromptDraft as string}
|
||||
disabled={projectResetPending as boolean}
|
||||
isGenerating={loadingAnimation as boolean}
|
||||
storyPresets={storyPresets as any[]}
|
||||
continuationDraft={livePromptDraft as string}
|
||||
canJoinSession={canStartSession}
|
||||
canSubmitContinuation={canSubmitContinuation}
|
||||
sessionExpired={sessionExpired as boolean}
|
||||
sessionNotice={sessionNotice as string}
|
||||
projectResetPending={projectResetPending as boolean}
|
||||
canSubmit={canStartSession}
|
||||
modelId={creationModelId}
|
||||
modeId={creationModeId}
|
||||
aspectRatio={creationAspectRatio}
|
||||
resolution={creationResolution}
|
||||
durationSec={creationDurationSec}
|
||||
referencePreviewUrl={referencePreviewUrl}
|
||||
firstFramePreviewUrl={firstFramePreviewUrl}
|
||||
lastFramePreviewUrl={lastFramePreviewUrl}
|
||||
mentionOptions={mentionOptions}
|
||||
storyPresets={lobbyStoryPresets}
|
||||
capabilities={activeModelCapabilities}
|
||||
onValueChange={(value) => sessionStore.patch({ livePromptDraft: value })}
|
||||
onSubmit={() => void joinSession()}
|
||||
onKeyDown={handleLivePromptKeydown}
|
||||
onModelChange={handleCreationModelChange}
|
||||
onModeChange={setCreationModeId}
|
||||
onAspectRatioChange={setCreationAspectRatio}
|
||||
onResolutionChange={setCreationResolution}
|
||||
onDurationChange={setCreationDurationSec}
|
||||
onReferenceSelect={handleReferenceSelect}
|
||||
onFirstFrameSelect={handleFirstFrameSelect}
|
||||
onLastFrameSelect={handleLastFrameSelect}
|
||||
onPresetGenerate={handlePresetGenerate}
|
||||
onContinuationInput={handleLivePromptInput}
|
||||
onContinuationKeydown={handleLivePromptKeydown}
|
||||
onGenerate={joinSession}
|
||||
onSubmitContinuation={submitLivePrompt}
|
||||
onLeave={leaveSession}
|
||||
onStartNewProject={handleStartNewProject}
|
||||
onSpeechTranscript={handleLivePromptSpeechTranscript}
|
||||
onSpeechInterimChange={handleLivePromptSpeechInterim}
|
||||
onOpenProjects={() => setSidebarOpen(true)}
|
||||
/>
|
||||
</motion.div>
|
||||
) : (
|
||||
<motion.div layout="position" className="mx-auto w-full max-w-2xl shrink-0" transition={{ type: "spring", stiffness: 200, damping: 25 }}>
|
||||
<ChatBar
|
||||
sessionStarted={sessionStarted as boolean}
|
||||
rewritingSeedPrompts={rewritingSeedPrompts as boolean}
|
||||
isGenerating={loadingAnimation as boolean}
|
||||
storyPresets={storyPresets as any[]}
|
||||
continuationDraft={livePromptDraft as string}
|
||||
canJoinSession={canStartSession}
|
||||
canSubmitContinuation={canSubmitContinuation}
|
||||
sessionExpired={sessionExpired as boolean}
|
||||
sessionNotice={sessionNotice as string}
|
||||
projectResetPending={projectResetPending as boolean}
|
||||
sessionCreationConfig={sessionCreationConfig}
|
||||
configPillsReadOnly
|
||||
onSessionModelChange={(modelId) => setSessionCreationConfig((current) => ({ ...current, modelId }))}
|
||||
onSessionModeChange={(modeId) => setSessionCreationConfig((current) => ({ ...current, modeId }))}
|
||||
onSessionAspectRatioChange={(aspectRatio) => setSessionCreationConfig((current) => ({ ...current, aspectRatio }))}
|
||||
onSessionResolutionChange={(resolution) => setSessionCreationConfig((current) => ({ ...current, resolution }))}
|
||||
onSessionDurationChange={(durationSec) => setSessionCreationConfig((current) => ({ ...current, durationSec }))}
|
||||
onPresetGenerate={handlePresetGenerate}
|
||||
onContinuationInput={handleLivePromptInput}
|
||||
onContinuationKeydown={handleLivePromptKeydown}
|
||||
onGenerate={joinSession}
|
||||
onSubmitContinuation={submitLivePrompt}
|
||||
onLeave={leaveSession}
|
||||
onStartNewProject={handleStartNewProject}
|
||||
onSpeechTranscript={handleLivePromptSpeechTranscript}
|
||||
onSpeechInterimChange={handleLivePromptSpeechInterim}
|
||||
/>
|
||||
</motion.div>
|
||||
)}
|
||||
|
||||
{sessionNotice && !showActiveProject && (
|
||||
<div className="mx-auto mt-2 w-full max-w-3xl px-4">
|
||||
<div className="rounded-xl border border-rose-500/20 bg-rose-500/10 px-4 py-2.5 text-center text-xs text-rose-700 dark:text-rose-300">
|
||||
{sessionNotice}
|
||||
</div>
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
</main>
|
||||
|
||||
@@ -2,10 +2,13 @@
|
||||
|
||||
import React, { useRef, useState, useCallback, useEffect } from "react";
|
||||
import Image from "next/image";
|
||||
import { Film, ArrowUp, X, Loader2, ArrowLeft } from "lucide-react";
|
||||
import { ArrowUp, X, Loader2, ArrowLeft } from "lucide-react";
|
||||
import { Button } from "@/components/ui/button";
|
||||
import LeaveSessionModal, { shouldShowLeaveWarning } from "@/components/LeaveSessionModal";
|
||||
import PresetQuickLaunchRail from "@/components/creation/PresetQuickLaunchRail";
|
||||
import SessionCreationConfigPills, { type SessionCreationConfig } from "@/components/creation/SessionCreationConfigPills";
|
||||
import SpeechToTextButton from "@/components/SpeechToTextButton";
|
||||
import type { AspectRatioId, CreationModeId, CreationModelId, ResolutionId } from "@/lib/creationConfig";
|
||||
import { cn } from "@/lib/utils";
|
||||
|
||||
const PROMPT_MAX_LENGTH = 500;
|
||||
@@ -32,6 +35,13 @@ interface Props {
|
||||
onBackFromViewing?: () => void;
|
||||
onSpeechTranscript?: (text: string) => void;
|
||||
onSpeechInterimChange?: (text: string) => void;
|
||||
sessionCreationConfig?: SessionCreationConfig | null;
|
||||
configPillsReadOnly?: boolean;
|
||||
onSessionModelChange?: (modelId: CreationModelId) => void;
|
||||
onSessionModeChange?: (modeId: CreationModeId) => void;
|
||||
onSessionAspectRatioChange?: (aspectRatio: AspectRatioId) => void;
|
||||
onSessionResolutionChange?: (resolution: ResolutionId) => void;
|
||||
onSessionDurationChange?: (durationSec: number) => void;
|
||||
}
|
||||
|
||||
export default function ChatBar({
|
||||
@@ -56,6 +66,13 @@ export default function ChatBar({
|
||||
onBackFromViewing = () => {},
|
||||
onSpeechTranscript,
|
||||
onSpeechInterimChange,
|
||||
sessionCreationConfig = null,
|
||||
configPillsReadOnly = false,
|
||||
onSessionModelChange,
|
||||
onSessionModeChange,
|
||||
onSessionAspectRatioChange,
|
||||
onSessionResolutionChange,
|
||||
onSessionDurationChange,
|
||||
}: Props) {
|
||||
const [sttBusy, setSttBusy] = useState(false);
|
||||
const [leaveModalOpen, setLeaveModalOpen] = useState(false);
|
||||
@@ -66,134 +83,11 @@ export default function ChatBar({
|
||||
: isBusy
|
||||
? "Generating video\u2026"
|
||||
: !sessionStarted
|
||||
? "What video are you imagining?"
|
||||
? "Describe your video"
|
||||
: "What do you want to edit?";
|
||||
const actionLabel = !sessionStarted ? "Generate" : "Rewrite rollout";
|
||||
|
||||
const inputRef = useRef<HTMLTextAreaElement>(null);
|
||||
const scrollRef = useRef<HTMLDivElement>(null);
|
||||
const [canScrollLeft, setCanScrollLeft] = useState(false);
|
||||
const [canScrollRight, setCanScrollRight] = useState(false);
|
||||
const [presetRailDragging, setPresetRailDragging] = useState(false);
|
||||
const presetDragStateRef = useRef({
|
||||
pointerId: null as number | null,
|
||||
startX: 0,
|
||||
startScrollLeft: 0,
|
||||
moved: false,
|
||||
});
|
||||
const suppressPresetClickRef = useRef(false);
|
||||
|
||||
const updateScrollState = useCallback(() => {
|
||||
const el = scrollRef.current;
|
||||
if (!el) return;
|
||||
setCanScrollLeft(el.scrollLeft > 2);
|
||||
setCanScrollRight(el.scrollLeft + el.clientWidth < el.scrollWidth - 2);
|
||||
}, []);
|
||||
|
||||
const handlePresetWheel = useCallback(
|
||||
(event: React.WheelEvent<HTMLDivElement>) => {
|
||||
const el = scrollRef.current;
|
||||
if (!el) return;
|
||||
if (el.scrollWidth <= el.clientWidth + 1) return;
|
||||
|
||||
const dominantDelta = Math.abs(event.deltaX) > Math.abs(event.deltaY)
|
||||
? event.deltaX
|
||||
: event.deltaY;
|
||||
if (!dominantDelta) return;
|
||||
|
||||
const maxScrollLeft = Math.max(el.scrollWidth - el.clientWidth, 0);
|
||||
const nextScrollLeft = Math.min(
|
||||
Math.max(el.scrollLeft + dominantDelta, 0),
|
||||
maxScrollLeft,
|
||||
);
|
||||
if (nextScrollLeft === el.scrollLeft) return;
|
||||
|
||||
event.preventDefault();
|
||||
el.scrollLeft = nextScrollLeft;
|
||||
updateScrollState();
|
||||
},
|
||||
[updateScrollState],
|
||||
);
|
||||
|
||||
const finishPresetDrag = useCallback(() => {
|
||||
presetDragStateRef.current = {
|
||||
pointerId: null,
|
||||
startX: 0,
|
||||
startScrollLeft: 0,
|
||||
moved: false,
|
||||
};
|
||||
setPresetRailDragging(false);
|
||||
}, []);
|
||||
|
||||
const handlePresetPointerDown = useCallback(
|
||||
(event: React.PointerEvent<HTMLDivElement>) => {
|
||||
const el = scrollRef.current;
|
||||
if (!el) return;
|
||||
if (event.pointerType !== "mouse" || event.button !== 0) return;
|
||||
if (el.scrollWidth <= el.clientWidth + 1) return;
|
||||
|
||||
suppressPresetClickRef.current = false;
|
||||
presetDragStateRef.current = {
|
||||
pointerId: event.pointerId,
|
||||
startX: event.clientX,
|
||||
startScrollLeft: el.scrollLeft,
|
||||
moved: false,
|
||||
};
|
||||
},
|
||||
[],
|
||||
);
|
||||
|
||||
const handlePresetPointerMove = useCallback(
|
||||
(event: React.PointerEvent<HTMLDivElement>) => {
|
||||
const el = scrollRef.current;
|
||||
const dragState = presetDragStateRef.current;
|
||||
if (!el || dragState.pointerId !== event.pointerId) return;
|
||||
|
||||
const deltaX = event.clientX - dragState.startX;
|
||||
if (!dragState.moved && Math.abs(deltaX) > 4) {
|
||||
dragState.moved = true;
|
||||
suppressPresetClickRef.current = true;
|
||||
setPresetRailDragging(true);
|
||||
el.setPointerCapture?.(event.pointerId);
|
||||
}
|
||||
if (!dragState.moved) return;
|
||||
|
||||
event.preventDefault();
|
||||
const maxScrollLeft = Math.max(el.scrollWidth - el.clientWidth, 0);
|
||||
el.scrollLeft = Math.min(
|
||||
Math.max(dragState.startScrollLeft - deltaX, 0),
|
||||
maxScrollLeft,
|
||||
);
|
||||
updateScrollState();
|
||||
},
|
||||
[updateScrollState],
|
||||
);
|
||||
|
||||
const handlePresetPointerUp = useCallback(
|
||||
(event: React.PointerEvent<HTMLDivElement>) => {
|
||||
const el = scrollRef.current;
|
||||
if (!el || presetDragStateRef.current.pointerId !== event.pointerId) return;
|
||||
if (el.hasPointerCapture?.(event.pointerId)) {
|
||||
el.releasePointerCapture(event.pointerId);
|
||||
}
|
||||
finishPresetDrag();
|
||||
},
|
||||
[finishPresetDrag],
|
||||
);
|
||||
|
||||
const handlePresetClickCapture = useCallback(
|
||||
(event: React.MouseEvent<HTMLDivElement>) => {
|
||||
if (!suppressPresetClickRef.current) return;
|
||||
suppressPresetClickRef.current = false;
|
||||
event.preventDefault();
|
||||
event.stopPropagation();
|
||||
},
|
||||
[],
|
||||
);
|
||||
|
||||
useEffect(() => {
|
||||
updateScrollState();
|
||||
}, [storyPresets, updateScrollState]);
|
||||
|
||||
useEffect(() => {
|
||||
if (!isBusy && !sttBusy && !window.matchMedia("(pointer: coarse)").matches) {
|
||||
@@ -239,7 +133,7 @@ export default function ChatBar({
|
||||
<div className="flex flex-col items-center gap-3 rounded-2xl border border-border bg-card/80 px-6 py-4 text-center shadow-md backdrop-blur-sm">
|
||||
<div className="flex flex-col gap-1">
|
||||
<p className="text-sm font-semibold text-foreground">View-only project</p>
|
||||
<p className="max-w-md text-xs text-muted-foreground">Project sessions are currently limited to 5 minutes. Start a new project to create more videos.</p>
|
||||
<p className="max-w-md text-xs text-muted-foreground">Sessions are limited to 5 minutes. Start a new project to keep creating.</p>
|
||||
</div>
|
||||
<div className="mt-1 flex items-center gap-2">
|
||||
<Button onClick={onBackFromViewing} variant="outline" size="sm" className="gap-1.5 rounded-full px-4">
|
||||
@@ -261,7 +155,7 @@ export default function ChatBar({
|
||||
<div className="flex flex-col items-center gap-3 rounded-2xl border border-border bg-card/80 px-8 py-5 text-center shadow-md backdrop-blur-sm">
|
||||
<div className="flex flex-col gap-1">
|
||||
<p className="text-sm font-semibold text-foreground">Session ended</p>
|
||||
<p className="max-w-xs text-xs text-muted-foreground">Each project currently has a 5-minute session. Start a new project to continue creating videos.</p>
|
||||
<p className="max-w-xs text-xs text-muted-foreground">Sessions are limited to 5 minutes. Start a new project to continue.</p>
|
||||
</div>
|
||||
<div className="mt-1 flex items-center gap-2">
|
||||
<Button onClick={onStartNewProject} size="sm" className="rounded-full px-5">
|
||||
@@ -280,51 +174,8 @@ export default function ChatBar({
|
||||
|
||||
return (
|
||||
<section className="mx-auto flex w-full max-w-2xl shrink-0 flex-col gap-4">
|
||||
{storyPresets.length > 0 && !sessionStarted && (
|
||||
<div className={cn("relative transition-opacity duration-200", isGenerating && "pointer-events-none opacity-40")}>
|
||||
<div
|
||||
ref={scrollRef}
|
||||
onScroll={updateScrollState}
|
||||
onWheel={handlePresetWheel}
|
||||
onPointerDown={handlePresetPointerDown}
|
||||
onPointerMove={handlePresetPointerMove}
|
||||
onPointerUp={handlePresetPointerUp}
|
||||
onPointerCancel={handlePresetPointerUp}
|
||||
onLostPointerCapture={finishPresetDrag}
|
||||
onClickCapture={handlePresetClickCapture}
|
||||
className={cn(
|
||||
"scrollbar-hidden flex gap-3 overflow-x-auto px-1 select-none",
|
||||
presetRailDragging ? "cursor-grabbing" : "cursor-grab",
|
||||
)}
|
||||
>
|
||||
{storyPresets.map((preset) => (
|
||||
<button
|
||||
key={preset.id}
|
||||
type="button"
|
||||
disabled={isGenerating}
|
||||
onClick={() => onPresetGenerate(preset.id)}
|
||||
className="flex flex-col sm:flex-row items-start gap-1.5 shrink-0 rounded-xl border p-2.5 text-left backdrop-blur-sm transition-colors max-w-42 sm:max-w-[215px] border-input bg-card/80 text-muted-foreground hover:bg-slate-200/60 hover:border-slate-400 hover:text-slate-700 dark:bg-slate-800/80 dark:text-slate-300 dark:hover:bg-slate-700/50 dark:hover:border-slate-500 dark:hover:text-slate-200"
|
||||
>
|
||||
<Film className="mt-0.5 size-4 shrink-0 opacity-60" />
|
||||
<span className="flex flex-col gap-1 min-w-0">
|
||||
<span className="text-[14px] font-medium line-clamp-1">{preset.label}</span>
|
||||
{preset.description && <span className="text-xs leading-tight opacity-70 line-clamp-3 sm:line-clamp-2">{preset.description}</span>}
|
||||
</span>
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
|
||||
<div
|
||||
className={cn("pointer-events-none absolute inset-y-0 left-0 w-8 bg-background transition-opacity duration-150", canScrollLeft ? "opacity-100" : "opacity-0")}
|
||||
style={{ maskImage: "linear-gradient(to right, black, transparent)", WebkitMaskImage: "linear-gradient(to right, black, transparent)" }}
|
||||
aria-hidden="true"
|
||||
/>
|
||||
<div
|
||||
className={cn("pointer-events-none absolute inset-y-0 right-0 w-8 bg-background transition-opacity duration-150", canScrollRight ? "opacity-100" : "opacity-0")}
|
||||
style={{ maskImage: "linear-gradient(to left, black, transparent)", WebkitMaskImage: "linear-gradient(to left, black, transparent)" }}
|
||||
aria-hidden="true"
|
||||
/>
|
||||
</div>
|
||||
{!sessionStarted && (
|
||||
<PresetQuickLaunchRail storyPresets={storyPresets} disabled={isGenerating} onPresetGenerate={onPresetGenerate} />
|
||||
)}
|
||||
|
||||
{sessionNotice && (
|
||||
@@ -342,17 +193,30 @@ export default function ChatBar({
|
||||
|
||||
{projectResetPending && sessionStarted && (
|
||||
<div className="rounded-xl border border-sky-500/20 bg-sky-500/10 px-4 py-2.5 text-center text-xs text-sky-700 dark:text-sky-300">
|
||||
Starting a new project after the current shot finishes. Your GPU session stays active.
|
||||
Starting a new project when this shot finishes. GPU session stays open.
|
||||
</div>
|
||||
)}
|
||||
|
||||
<div
|
||||
className={cn(
|
||||
"flex min-w-0 items-center gap-1.5 rounded-4xl border py-2.5 pl-5 pr-2.5 shadow-md backdrop-blur-sm transition-all duration-200",
|
||||
"flex min-w-0 flex-col gap-2 rounded-4xl border py-2.5 pl-5 pr-2.5 shadow-md backdrop-blur-sm transition-all duration-200",
|
||||
isBusy ? "border-input/60 bg-card/40" : "border-input bg-card/65",
|
||||
)}
|
||||
>
|
||||
<textarea
|
||||
{sessionStarted && sessionCreationConfig && (
|
||||
<SessionCreationConfigPills
|
||||
{...sessionCreationConfig}
|
||||
disabled={isBusy}
|
||||
readOnly={configPillsReadOnly}
|
||||
onModelChange={onSessionModelChange}
|
||||
onModeChange={onSessionModeChange}
|
||||
onAspectRatioChange={onSessionAspectRatioChange}
|
||||
onResolutionChange={onSessionResolutionChange}
|
||||
onDurationChange={onSessionDurationChange}
|
||||
/>
|
||||
)}
|
||||
<div className="flex min-w-0 items-center gap-1.5">
|
||||
<textarea
|
||||
ref={inputRef}
|
||||
id="continuation-prompt"
|
||||
aria-label="Continuation prompt"
|
||||
@@ -367,36 +231,37 @@ export default function ChatBar({
|
||||
"min-w-0 flex-1 resize-none bg-transparent text-foreground outline-none placeholder:text-muted-foreground transition-opacity duration-200 scrollbar-thin leading-snug",
|
||||
(isBusy || sttBusy) && "cursor-not-allowed opacity-50",
|
||||
)}
|
||||
/>
|
||||
{onSpeechTranscript && <SpeechToTextButton disabled={isBusy} onTranscript={onSpeechTranscript} onInterimChange={onSpeechInterimChange} onBusyChange={setSttBusy} />}
|
||||
{!sessionStarted ? (
|
||||
<Button
|
||||
aria-label={actionLabel}
|
||||
title={actionLabel}
|
||||
onClick={onGenerate}
|
||||
disabled={!canJoinSession || isGenerating || !continuationDraft.trim()}
|
||||
size="icon-sm"
|
||||
className="shrink-0 rounded-full"
|
||||
>
|
||||
{showSpinner ? <Loader2 className="size-5 animate-spin" /> : <ArrowUp className="size-5" />}
|
||||
</Button>
|
||||
) : (
|
||||
<>
|
||||
/>
|
||||
{onSpeechTranscript && <SpeechToTextButton disabled={isBusy} onTranscript={onSpeechTranscript} onInterimChange={onSpeechInterimChange} onBusyChange={setSttBusy} />}
|
||||
{!sessionStarted ? (
|
||||
<Button
|
||||
aria-label={actionLabel}
|
||||
title={actionLabel}
|
||||
onClick={onSubmitContinuation}
|
||||
disabled={!canSubmitContinuation || showSpinner || projectResetPending || !continuationDraft.trim()}
|
||||
onClick={onGenerate}
|
||||
disabled={!canJoinSession || isGenerating || !continuationDraft.trim()}
|
||||
size="icon-sm"
|
||||
className="shrink-0 rounded-full"
|
||||
>
|
||||
{showSpinner ? <Loader2 className="size-5 animate-spin" /> : <ArrowUp className="size-5" />}
|
||||
</Button>
|
||||
<Button variant="outline" aria-label="Leave" title="Leave" onClick={() => { if (shouldShowLeaveWarning()) setLeaveModalOpen(true); else onLeave(); }} disabled={isGenerating || projectResetPending} size="icon-sm" className="shrink-0 rounded-full">
|
||||
<X className="size-5" />
|
||||
</Button>
|
||||
</>
|
||||
)}
|
||||
) : (
|
||||
<>
|
||||
<Button
|
||||
aria-label={actionLabel}
|
||||
title={actionLabel}
|
||||
onClick={onSubmitContinuation}
|
||||
disabled={!canSubmitContinuation || showSpinner || projectResetPending || !continuationDraft.trim()}
|
||||
size="icon-sm"
|
||||
className="shrink-0 rounded-full"
|
||||
>
|
||||
{showSpinner ? <Loader2 className="size-5 animate-spin" /> : <ArrowUp className="size-5" />}
|
||||
</Button>
|
||||
<Button variant="outline" aria-label="Leave" title="Leave" onClick={() => { if (shouldShowLeaveWarning()) setLeaveModalOpen(true); else onLeave(); }} disabled={isGenerating || projectResetPending} size="icon-sm" className="shrink-0 rounded-full">
|
||||
<X className="size-5" />
|
||||
</Button>
|
||||
</>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
<p className="px-2 text-center text-[11px] text-muted-foreground">
|
||||
LLM powered by{" "}
|
||||
|
||||
@@ -0,0 +1,95 @@
|
||||
"use client";
|
||||
|
||||
import { Fragment, useEffect, useRef } from "react";
|
||||
|
||||
const HERO_WAVE_LIGHT = ["#2A4A98", "#4878E5", "#6FA0F2", "#B0BCC8", "#E8D99E", "#D8C844", "#C2A620"];
|
||||
const HERO_WAVE_DARK = ["#143468", "#1E58B8", "#3892F0", "#80B8E8", "#B8D0EA", "#E2D498", "#DABB50"];
|
||||
const HERO_TEXT = "Direct scenes in seconds";
|
||||
|
||||
export default function HeroTagline() {
|
||||
const ref = useRef<HTMLHeadingElement>(null);
|
||||
|
||||
useEffect(() => {
|
||||
const el = ref.current;
|
||||
if (!el) return;
|
||||
|
||||
let rafId = 0;
|
||||
|
||||
function play() {
|
||||
const chars = el!.querySelectorAll<HTMLSpanElement>("[data-char]");
|
||||
if (!chars.length) return;
|
||||
cancelAnimationFrame(rafId);
|
||||
|
||||
const isDark = document.documentElement.classList.contains("dark");
|
||||
const colors = isDark ? HERO_WAVE_DARK : HERO_WAVE_LIGHT;
|
||||
const waveLen = 10;
|
||||
const total = chars.length + waveLen;
|
||||
const duration = 1200;
|
||||
const maxBlur = 3.5;
|
||||
const start = performance.now();
|
||||
|
||||
function tick() {
|
||||
const t = Math.min((performance.now() - start) / duration, 1);
|
||||
const pos = t * total;
|
||||
chars.forEach((ch, i) => {
|
||||
const rel = pos - i;
|
||||
if (rel >= 0 && rel < waveLen) {
|
||||
const norm = rel / waveLen;
|
||||
const ci = Math.floor(norm * colors.length);
|
||||
ch.style.color = colors[Math.min(colors.length - 1, ci)];
|
||||
|
||||
let blur = 0;
|
||||
if (norm < 0.25) {
|
||||
blur = maxBlur * (1 - norm / 0.25);
|
||||
} else if (norm > 0.75) {
|
||||
blur = maxBlur * ((norm - 0.75) / 0.25);
|
||||
}
|
||||
ch.style.filter = blur > 0.1 ? `blur(${blur.toFixed(1)}px)` : "";
|
||||
} else {
|
||||
ch.style.color = "";
|
||||
ch.style.filter = "";
|
||||
}
|
||||
});
|
||||
if (t < 1) {
|
||||
rafId = requestAnimationFrame(tick);
|
||||
} else {
|
||||
chars.forEach((ch) => {
|
||||
ch.style.color = "";
|
||||
ch.style.filter = "";
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
rafId = requestAnimationFrame(tick);
|
||||
}
|
||||
|
||||
const initialDelay = setTimeout(play, 400);
|
||||
const interval = setInterval(play, 5000);
|
||||
return () => {
|
||||
clearTimeout(initialDelay);
|
||||
clearInterval(interval);
|
||||
cancelAnimationFrame(rafId);
|
||||
};
|
||||
}, []);
|
||||
|
||||
return (
|
||||
<h1 ref={ref} className="text-balance text-center text-3xl font-medium text-[#343537] dark:text-[#FAFAFB] sm:text-4xl">
|
||||
{HERO_TEXT.split(" ").map((word, wi) => (
|
||||
<Fragment key={wi}>
|
||||
{wi > 0 && (
|
||||
<span data-char className="transition-[color,filter] duration-150">
|
||||
{" "}
|
||||
</span>
|
||||
)}
|
||||
<span className="inline-flex">
|
||||
{word.split("").map((char, ci) => (
|
||||
<span key={ci} data-char className="inline-block transition-[color,filter] duration-150">
|
||||
{char}
|
||||
</span>
|
||||
))}
|
||||
</span>
|
||||
</Fragment>
|
||||
))}
|
||||
</h1>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,66 @@
|
||||
"use client";
|
||||
|
||||
import React from "react";
|
||||
import { FolderOpen, Home, Sparkles } from "lucide-react";
|
||||
|
||||
import { cn } from "@/lib/utils";
|
||||
|
||||
export type AppNavSection = "explore" | "create" | "assets";
|
||||
|
||||
interface AppNavRailProps {
|
||||
activeSection?: AppNavSection;
|
||||
onSectionChange?: (section: AppNavSection) => void;
|
||||
onOpenProjects?: () => void;
|
||||
className?: string;
|
||||
}
|
||||
|
||||
const NAV_ITEMS: Array<{ id: AppNavSection; label: string; icon: typeof Home }> = [
|
||||
{ id: "explore", label: "Explore", icon: Home },
|
||||
{ id: "create", label: "Create", icon: Sparkles },
|
||||
{ id: "assets", label: "Assets", icon: FolderOpen },
|
||||
];
|
||||
|
||||
export default function AppNavRail({
|
||||
activeSection = "create",
|
||||
onSectionChange = () => {},
|
||||
onOpenProjects,
|
||||
className,
|
||||
}: AppNavRailProps) {
|
||||
return (
|
||||
<aside
|
||||
className={cn(
|
||||
"hidden shrink-0 flex-col items-center gap-2 border-r border-border/40 bg-background/30 px-2.5 py-5 lg:flex",
|
||||
className,
|
||||
)}
|
||||
aria-label="Primary navigation"
|
||||
>
|
||||
{NAV_ITEMS.map((item) => {
|
||||
const Icon = item.icon;
|
||||
const isActive = item.id === activeSection;
|
||||
return (
|
||||
<button
|
||||
key={item.id}
|
||||
type="button"
|
||||
aria-label={item.label}
|
||||
aria-current={isActive ? "page" : undefined}
|
||||
onClick={() => {
|
||||
if (item.id === "assets") {
|
||||
onOpenProjects?.();
|
||||
}
|
||||
onSectionChange(item.id);
|
||||
}}
|
||||
className={cn(
|
||||
"studio-control studio-control-press flex w-[4.5rem] min-h-11 flex-col items-center gap-1 rounded-xl px-2 py-2.5 text-[10px] font-medium tracking-wide",
|
||||
isActive
|
||||
? "bg-secondary/90 text-foreground shadow-sm ring-1 ring-border/60"
|
||||
: "text-muted-foreground hover-capable:hover:bg-secondary/50 hover-capable:hover:text-foreground",
|
||||
)}
|
||||
>
|
||||
<Icon className={cn("size-[18px]", isActive && "text-accent-blue")} />
|
||||
{item.label}
|
||||
</button>
|
||||
);
|
||||
})}
|
||||
</aside>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,24 @@
|
||||
"use client";
|
||||
|
||||
import React from "react";
|
||||
|
||||
import { cn } from "@/lib/utils";
|
||||
|
||||
export default function ConfigPill({
|
||||
children,
|
||||
className,
|
||||
...props
|
||||
}: React.ButtonHTMLAttributes<HTMLButtonElement>) {
|
||||
return (
|
||||
<button
|
||||
type="button"
|
||||
className={cn(
|
||||
"studio-control studio-control-press studio-hover-surface inline-flex h-9 min-h-9 shrink-0 items-center gap-1 rounded-full border border-border/50 bg-background/80 px-2.5 text-[11px] font-medium text-foreground/90",
|
||||
className,
|
||||
)}
|
||||
{...props}
|
||||
>
|
||||
{children}
|
||||
</button>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,466 @@
|
||||
"use client";
|
||||
|
||||
import React, { useMemo, useRef, useState } from "react";
|
||||
import { ArrowUp, Box, ChevronDown, Clock, Monitor, Wand2 } from "lucide-react";
|
||||
|
||||
import ConfigPill from "@/components/creation/ConfigPill";
|
||||
import HeroTagline from "@/components/HeroTagline";
|
||||
import ReferenceUploadSlot from "@/components/creation/ReferenceUploadSlot";
|
||||
import { Button } from "@/components/ui/button";
|
||||
import {
|
||||
DropdownMenu,
|
||||
DropdownMenuContent,
|
||||
DropdownMenuItem,
|
||||
DropdownMenuLabel,
|
||||
DropdownMenuSeparator,
|
||||
DropdownMenuTrigger,
|
||||
} from "@/components/ui/dropdown-menu";
|
||||
import { Popover, PopoverContent, PopoverTrigger } from "@/components/ui/popover";
|
||||
import { Slider } from "@/components/ui/slider";
|
||||
import SpeechToTextButton from "@/components/SpeechToTextButton";
|
||||
import {
|
||||
ASPECT_RATIOS,
|
||||
CREATION_MODELS,
|
||||
CREATION_MODES,
|
||||
RESOLUTIONS,
|
||||
UNSUPPORTED_CREATION_MODES,
|
||||
UNSUPPORTED_RESOLUTIONS,
|
||||
modeRequiresReference,
|
||||
modeUsesDualFrames,
|
||||
type AspectRatioId,
|
||||
type CreationModeId,
|
||||
type CreationModelId,
|
||||
type MentionOption,
|
||||
type ResolutionId,
|
||||
formatDurationLabel,
|
||||
formatResolutionLabel,
|
||||
} from "@/lib/creationConfig";
|
||||
import {
|
||||
DEFAULT_LOBBY_CAPABILITIES_BUNDLE,
|
||||
isSupportedCreationMode,
|
||||
isSupportedResolution,
|
||||
resolveModelCapabilities,
|
||||
unsupportedModeNotice,
|
||||
type LobbyCreationCapabilities,
|
||||
} from "@/lib/creationCapabilities";
|
||||
import { cn } from "@/lib/utils";
|
||||
|
||||
const PROMPT_MAX_LENGTH = 500;
|
||||
|
||||
interface CreationComposerProps {
|
||||
value: string;
|
||||
disabled?: boolean;
|
||||
isGenerating?: boolean;
|
||||
canSubmit?: boolean;
|
||||
modelId: CreationModelId;
|
||||
modeId: CreationModeId;
|
||||
aspectRatio: AspectRatioId;
|
||||
resolution: ResolutionId;
|
||||
durationSec: number;
|
||||
referencePreviewUrl?: string | null;
|
||||
firstFramePreviewUrl?: string | null;
|
||||
lastFramePreviewUrl?: string | null;
|
||||
mentionOptions?: MentionOption[];
|
||||
onValueChange: (value: string) => void;
|
||||
onSubmit: () => void;
|
||||
onKeyDown?: (event: React.KeyboardEvent<HTMLTextAreaElement>) => void;
|
||||
onModelChange: (modelId: CreationModelId) => void;
|
||||
onModeChange: (modeId: CreationModeId) => void;
|
||||
onAspectRatioChange: (aspectRatio: AspectRatioId) => void;
|
||||
onResolutionChange: (resolution: ResolutionId) => void;
|
||||
onDurationChange: (durationSec: number) => void;
|
||||
onReferenceSelect?: (file: File | null) => void;
|
||||
onFirstFrameSelect?: (file: File | null) => void;
|
||||
onLastFrameSelect?: (file: File | null) => void;
|
||||
onSpeechTranscript?: (text: string) => void;
|
||||
onSpeechInterimChange?: (text: string) => void;
|
||||
capabilities?: LobbyCreationCapabilities;
|
||||
}
|
||||
|
||||
export default function CreationComposer({
|
||||
value,
|
||||
disabled = false,
|
||||
isGenerating = false,
|
||||
canSubmit = false,
|
||||
modelId,
|
||||
modeId,
|
||||
aspectRatio,
|
||||
resolution,
|
||||
durationSec,
|
||||
referencePreviewUrl = null,
|
||||
firstFramePreviewUrl = null,
|
||||
lastFramePreviewUrl = null,
|
||||
mentionOptions = [],
|
||||
onValueChange,
|
||||
onSubmit,
|
||||
onKeyDown,
|
||||
onModelChange,
|
||||
onModeChange,
|
||||
onAspectRatioChange,
|
||||
onResolutionChange,
|
||||
onDurationChange,
|
||||
onReferenceSelect,
|
||||
onFirstFrameSelect,
|
||||
onLastFrameSelect,
|
||||
onSpeechTranscript,
|
||||
onSpeechInterimChange,
|
||||
capabilities = resolveModelCapabilities(DEFAULT_LOBBY_CAPABILITIES_BUNDLE, modelId),
|
||||
}: CreationComposerProps) {
|
||||
const inputRef = useRef<HTMLTextAreaElement>(null);
|
||||
const [sttBusy, setSttBusy] = useState(false);
|
||||
const [mentionQuery, setMentionQuery] = useState("");
|
||||
const [mentionOpen, setMentionOpen] = useState(false);
|
||||
const [mentionStart, setMentionStart] = useState<number | null>(null);
|
||||
|
||||
const availableModels = useMemo(
|
||||
() => CREATION_MODELS.filter((model) => capabilities.model_ids.includes(model.id)),
|
||||
[capabilities.model_ids],
|
||||
);
|
||||
const availableModes = useMemo(
|
||||
() => CREATION_MODES.filter((mode) => isSupportedCreationMode(mode.id, capabilities)),
|
||||
[capabilities],
|
||||
);
|
||||
const unavailableModes = useMemo(
|
||||
() =>
|
||||
UNSUPPORTED_CREATION_MODES.filter(
|
||||
(mode) => unsupportedModeNotice(mode.id, capabilities) !== null,
|
||||
),
|
||||
[capabilities],
|
||||
);
|
||||
const availableAspectRatios = useMemo(
|
||||
() => ASPECT_RATIOS.filter((ratio) => capabilities.aspect_ratios.includes(ratio)),
|
||||
[capabilities.aspect_ratios],
|
||||
);
|
||||
const availableResolutions = useMemo(
|
||||
() => RESOLUTIONS.filter((item) => isSupportedResolution(item, capabilities)),
|
||||
[capabilities],
|
||||
);
|
||||
const unavailableResolutions = useMemo(
|
||||
() => UNSUPPORTED_RESOLUTIONS.filter((item) => !isSupportedResolution(item, capabilities)),
|
||||
[capabilities],
|
||||
);
|
||||
const durationMin = capabilities.duration_sec[0] ?? 5;
|
||||
const durationMax = capabilities.duration_sec[capabilities.duration_sec.length - 1] ?? 15;
|
||||
|
||||
const selectedModel = availableModels.find((model) => model.id === modelId) ?? availableModels[0];
|
||||
const selectedMode = availableModes.find((mode) => mode.id === modeId) ?? availableModes[0];
|
||||
const usesDualFrames = modeUsesDualFrames(modeId);
|
||||
const requiresReference = modeRequiresReference(modeId);
|
||||
const referenceMissing = requiresReference && !referencePreviewUrl;
|
||||
const submitDisabled = !canSubmit || disabled || isGenerating || !value.trim() || referenceMissing;
|
||||
|
||||
const filteredMentions = useMemo(() => {
|
||||
const query = mentionQuery.trim().toLowerCase();
|
||||
if (!query) return mentionOptions.slice(0, 6);
|
||||
return mentionOptions
|
||||
.filter((option) => option.label.toLowerCase().includes(query) || option.description?.toLowerCase().includes(query))
|
||||
.slice(0, 6);
|
||||
}, [mentionOptions, mentionQuery]);
|
||||
|
||||
function autoResize() {
|
||||
const el = inputRef.current;
|
||||
if (!el) return;
|
||||
el.style.height = "auto";
|
||||
const lineHeight = parseFloat(getComputedStyle(el).lineHeight) || 22;
|
||||
const maxHeight = lineHeight * 4;
|
||||
el.style.height = `${Math.min(el.scrollHeight, maxHeight)}px`;
|
||||
el.style.overflowY = el.scrollHeight > maxHeight ? "auto" : "hidden";
|
||||
}
|
||||
|
||||
function updateMentionState(nextValue: string, cursorPosition: number) {
|
||||
const beforeCursor = nextValue.slice(0, cursorPosition);
|
||||
const atIndex = beforeCursor.lastIndexOf("@");
|
||||
if (atIndex === -1 || (atIndex > 0 && !/\s/.test(beforeCursor[atIndex - 1] ?? ""))) {
|
||||
setMentionOpen(false);
|
||||
setMentionStart(null);
|
||||
setMentionQuery("");
|
||||
return;
|
||||
}
|
||||
const query = beforeCursor.slice(atIndex + 1);
|
||||
if (/\s/.test(query)) {
|
||||
setMentionOpen(false);
|
||||
setMentionStart(null);
|
||||
setMentionQuery("");
|
||||
return;
|
||||
}
|
||||
setMentionStart(atIndex);
|
||||
setMentionQuery(query);
|
||||
setMentionOpen(true);
|
||||
}
|
||||
|
||||
function insertMention(option: MentionOption) {
|
||||
if (mentionStart === null) return;
|
||||
const before = value.slice(0, mentionStart);
|
||||
const after = value.slice(inputRef.current?.selectionStart ?? value.length);
|
||||
const mentionText = `@${option.label} `;
|
||||
const nextValue = `${before}${mentionText}${after}`.slice(0, PROMPT_MAX_LENGTH);
|
||||
onValueChange(nextValue);
|
||||
setMentionOpen(false);
|
||||
setMentionStart(null);
|
||||
setMentionQuery("");
|
||||
requestAnimationFrame(() => {
|
||||
const el = inputRef.current;
|
||||
if (!el) return;
|
||||
const cursor = before.length + mentionText.length;
|
||||
el.focus();
|
||||
el.setSelectionRange(cursor, cursor);
|
||||
autoResize();
|
||||
});
|
||||
}
|
||||
|
||||
function handleInputChange(event: React.ChangeEvent<HTMLTextAreaElement>) {
|
||||
const nextValue = event.target.value.slice(0, PROMPT_MAX_LENGTH);
|
||||
onValueChange(nextValue);
|
||||
updateMentionState(nextValue, event.target.selectionStart ?? nextValue.length);
|
||||
requestAnimationFrame(autoResize);
|
||||
}
|
||||
|
||||
function handleKeyDown(event: React.KeyboardEvent<HTMLTextAreaElement>) {
|
||||
if (mentionOpen && filteredMentions.length > 0) {
|
||||
if (event.key === "Tab" || (event.key === "Enter" && !event.shiftKey)) {
|
||||
event.preventDefault();
|
||||
insertMention(filteredMentions[0]);
|
||||
return;
|
||||
}
|
||||
if (event.key === "Escape") {
|
||||
setMentionOpen(false);
|
||||
return;
|
||||
}
|
||||
}
|
||||
onKeyDown?.(event);
|
||||
}
|
||||
|
||||
return (
|
||||
<section className="mx-auto flex w-full max-w-3xl flex-col gap-5">
|
||||
<HeroTagline />
|
||||
|
||||
<div className="rounded-[32px] border border-border/40 bg-secondary/95 p-4 shadow-[0_24px_80px_-32px_rgba(0,0,0,0.72)] backdrop-blur-xl sm:p-5">
|
||||
<div className="flex gap-3.5">
|
||||
{usesDualFrames ? (
|
||||
<div className="flex shrink-0 gap-2">
|
||||
<ReferenceUploadSlot
|
||||
label="Asset"
|
||||
sublabel="First"
|
||||
previewUrl={firstFramePreviewUrl}
|
||||
disabled={disabled}
|
||||
onSelect={onFirstFrameSelect}
|
||||
/>
|
||||
<ReferenceUploadSlot
|
||||
label="Asset"
|
||||
sublabel="Last"
|
||||
previewUrl={lastFramePreviewUrl}
|
||||
disabled={disabled}
|
||||
onSelect={onLastFrameSelect}
|
||||
/>
|
||||
</div>
|
||||
) : (
|
||||
<ReferenceUploadSlot
|
||||
label="Reference"
|
||||
previewUrl={referencePreviewUrl}
|
||||
required={requiresReference}
|
||||
optional={!requiresReference}
|
||||
disabled={disabled}
|
||||
onSelect={onReferenceSelect}
|
||||
/>
|
||||
)}
|
||||
|
||||
<div className="relative min-w-0 flex-1">
|
||||
<textarea
|
||||
ref={inputRef}
|
||||
id="continuation-prompt"
|
||||
aria-label="Continuation prompt"
|
||||
value={value}
|
||||
onChange={handleInputChange}
|
||||
onKeyDown={handleKeyDown}
|
||||
onClick={(event) => updateMentionState(value, event.currentTarget.selectionStart ?? value.length)}
|
||||
placeholder="Describe your video or mention elements"
|
||||
disabled={disabled || sttBusy}
|
||||
rows={3}
|
||||
className={cn(
|
||||
"min-h-[92px] w-full resize-none bg-transparent px-0.5 text-base leading-6 text-foreground outline-none placeholder:text-muted-foreground/80 sm:text-sm",
|
||||
(disabled || sttBusy) && "cursor-not-allowed opacity-50",
|
||||
)}
|
||||
/>
|
||||
{mentionOpen && filteredMentions.length > 0 && (
|
||||
<div className="absolute left-0 right-0 top-full z-20 mt-2 overflow-hidden rounded-2xl border border-border bg-popover/95 p-1 shadow-xl backdrop-blur-md">
|
||||
<p className="px-2 py-1 text-[11px] font-medium uppercase tracking-wide text-muted-foreground">Mention</p>
|
||||
{filteredMentions.map((option) => (
|
||||
<button
|
||||
key={option.id}
|
||||
type="button"
|
||||
onMouseDown={(event) => {
|
||||
event.preventDefault();
|
||||
insertMention(option);
|
||||
}}
|
||||
className="studio-control studio-hover-surface flex w-full items-start gap-2 rounded-xl px-2.5 py-2 text-left"
|
||||
>
|
||||
<span className="mt-0.5 rounded-md bg-accent px-1.5 py-0.5 text-[10px] font-semibold uppercase tracking-wide text-muted-foreground">
|
||||
{option.kind}
|
||||
</span>
|
||||
<span className="min-w-0">
|
||||
<span className="block truncate text-sm font-medium text-foreground">{option.label}</span>
|
||||
{option.description && <span className="block truncate text-xs text-muted-foreground">{option.description}</span>}
|
||||
</span>
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div className="mt-4 flex flex-wrap items-center gap-1.5 rounded-2xl bg-muted/35 p-1.5 ring-1 ring-border/25">
|
||||
<DropdownMenu>
|
||||
<DropdownMenuTrigger asChild>
|
||||
<ConfigPill disabled={disabled}>
|
||||
<Box className="size-3.5" />
|
||||
{selectedModel.label}
|
||||
<ChevronDown className="size-3 opacity-60" />
|
||||
</ConfigPill>
|
||||
</DropdownMenuTrigger>
|
||||
<DropdownMenuContent align="start" className="w-72">
|
||||
<DropdownMenuLabel>Model</DropdownMenuLabel>
|
||||
<DropdownMenuSeparator />
|
||||
{availableModels.map((model) => (
|
||||
<DropdownMenuItem key={model.id} onClick={() => onModelChange(model.id)} className="flex-col items-start gap-1 py-2.5">
|
||||
<span className="flex items-center gap-2 text-sm font-medium">
|
||||
{model.label}
|
||||
{model.badge && <span className="rounded-full bg-accent-blue/15 px-1.5 py-0.5 text-[10px] text-accent-blue">{model.badge}</span>}
|
||||
</span>
|
||||
<span className="text-xs text-muted-foreground">{model.description}</span>
|
||||
</DropdownMenuItem>
|
||||
))}
|
||||
</DropdownMenuContent>
|
||||
</DropdownMenu>
|
||||
|
||||
<DropdownMenu>
|
||||
<DropdownMenuTrigger asChild>
|
||||
<ConfigPill disabled={disabled}>
|
||||
<Wand2 className="size-3.5" />
|
||||
{selectedMode.label}
|
||||
<ChevronDown className="size-3 opacity-60" />
|
||||
</ConfigPill>
|
||||
</DropdownMenuTrigger>
|
||||
<DropdownMenuContent align="start" className="w-64">
|
||||
<DropdownMenuLabel>Mode</DropdownMenuLabel>
|
||||
<DropdownMenuSeparator />
|
||||
{availableModes.map((mode) => (
|
||||
<DropdownMenuItem key={mode.id} onClick={() => onModeChange(mode.id)} className="flex-col items-start gap-1 py-2.5">
|
||||
<span className="text-sm font-medium">{mode.label}</span>
|
||||
<span className="text-xs text-muted-foreground">{mode.description}</span>
|
||||
</DropdownMenuItem>
|
||||
))}
|
||||
{unavailableModes.length > 0 && <DropdownMenuSeparator />}
|
||||
{unavailableModes.map((mode) => (
|
||||
<DropdownMenuItem key={mode.id} disabled className="flex-col items-start gap-1 py-2.5 opacity-60">
|
||||
<span className="text-sm font-medium">{mode.label}</span>
|
||||
<span className="text-xs text-muted-foreground">
|
||||
{unsupportedModeNotice(mode.id, capabilities) ?? mode.description}
|
||||
</span>
|
||||
</DropdownMenuItem>
|
||||
))}
|
||||
</DropdownMenuContent>
|
||||
</DropdownMenu>
|
||||
|
||||
<Popover>
|
||||
<PopoverTrigger asChild>
|
||||
<ConfigPill disabled={disabled}>
|
||||
<Monitor className="size-3.5" />
|
||||
{aspectRatio} {formatResolutionLabel(resolution)}
|
||||
</ConfigPill>
|
||||
</PopoverTrigger>
|
||||
<PopoverContent align="start" className="w-80">
|
||||
<p className="mb-3 text-xs font-medium text-muted-foreground">Aspect ratio</p>
|
||||
<div className="grid grid-cols-3 gap-2">
|
||||
{availableAspectRatios.map((ratio) => (
|
||||
<button
|
||||
key={ratio}
|
||||
type="button"
|
||||
onClick={() => onAspectRatioChange(ratio)}
|
||||
className={cn(
|
||||
"studio-control studio-control-press studio-hover-surface flex flex-col items-center gap-2 rounded-xl border px-2 py-3 text-xs",
|
||||
aspectRatio === ratio ? "border-accent-blue bg-accent-blue/10 text-foreground" : "border-border",
|
||||
)}
|
||||
>
|
||||
<span className={cn("rounded-sm border border-current/40 bg-muted/40", ratio === "9:16" && "h-7 w-4", ratio === "16:9" && "h-4 w-7", ratio === "1:1" && "size-5", ratio === "4:3" && "h-5 w-6", ratio === "3:4" && "h-6 w-5", ratio === "21:9" && "h-3 w-8")} />
|
||||
{ratio}
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
<p className="mb-2 mt-4 text-xs font-medium text-muted-foreground">Resolution</p>
|
||||
<div className="flex flex-wrap gap-2">
|
||||
{availableResolutions.map((item) => (
|
||||
<button
|
||||
key={item}
|
||||
type="button"
|
||||
onClick={() => onResolutionChange(item)}
|
||||
className={cn(
|
||||
"studio-control studio-control-press studio-hover-surface rounded-full border px-3 py-1.5 text-xs font-medium",
|
||||
resolution === item ? "border-accent-blue bg-accent-blue/10 text-foreground" : "border-border",
|
||||
)}
|
||||
>
|
||||
{formatResolutionLabel(item)}
|
||||
</button>
|
||||
))}
|
||||
{unavailableResolutions.map((item) => (
|
||||
<button
|
||||
key={item}
|
||||
type="button"
|
||||
disabled
|
||||
className="studio-control rounded-full border border-border px-3 py-1.5 text-xs font-medium text-muted-foreground opacity-50"
|
||||
title="Not supported on FastLTX models yet"
|
||||
>
|
||||
{formatResolutionLabel(item)}
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
</PopoverContent>
|
||||
</Popover>
|
||||
|
||||
<Popover>
|
||||
<PopoverTrigger asChild>
|
||||
<ConfigPill disabled={disabled}>
|
||||
<Clock className="size-3.5" />
|
||||
{formatDurationLabel(durationSec)}
|
||||
</ConfigPill>
|
||||
</PopoverTrigger>
|
||||
<PopoverContent align="start" className="w-72">
|
||||
<p className="mb-3 text-xs font-medium text-muted-foreground">Total duration</p>
|
||||
<Slider min={durationMin} max={durationMax} step={5} value={[durationSec]} onValueChange={(values) => onDurationChange(values[0] ?? durationMin)} />
|
||||
<div className="mt-3 flex items-center justify-between text-[11px] text-muted-foreground">
|
||||
<span>{formatDurationLabel(durationMin)}</span>
|
||||
<span className="rounded-md border border-border px-2 py-1 text-xs font-medium text-foreground">{formatDurationLabel(durationSec)}</span>
|
||||
<span>{formatDurationLabel(durationMax)}</span>
|
||||
</div>
|
||||
</PopoverContent>
|
||||
</Popover>
|
||||
|
||||
<div className="ml-auto flex items-center gap-1.5">
|
||||
{onSpeechTranscript && (
|
||||
<SpeechToTextButton
|
||||
disabled={disabled || isGenerating}
|
||||
onTranscript={onSpeechTranscript}
|
||||
onInterimChange={onSpeechInterimChange}
|
||||
onBusyChange={setSttBusy}
|
||||
/>
|
||||
)}
|
||||
<Button
|
||||
aria-label="Generate"
|
||||
onClick={onSubmit}
|
||||
disabled={submitDisabled}
|
||||
size="icon"
|
||||
className="studio-control-press rounded-full bg-accent-blue text-white shadow-sm hover-capable:hover:bg-accent-blue/90 disabled:bg-muted disabled:text-muted-foreground"
|
||||
>
|
||||
<ArrowUp className="size-5" />
|
||||
</Button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{referenceMissing && value.trim() && (
|
||||
<p className="mt-3 text-center text-xs leading-5 text-amber-700 dark:text-amber-400">
|
||||
Upload a reference asset to use Omni reference mode.
|
||||
</p>
|
||||
)}
|
||||
</div>
|
||||
</section>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,73 @@
|
||||
"use client";
|
||||
|
||||
import React from "react";
|
||||
|
||||
import AppNavRail, { type AppNavSection } from "@/components/creation/AppNavRail";
|
||||
import CreationComposer from "@/components/creation/CreationComposer";
|
||||
import PresetQuickLaunchRail, { type StoryPresetLike } from "@/components/creation/PresetQuickLaunchRail";
|
||||
import {
|
||||
type AspectRatioId,
|
||||
type CreationModeId,
|
||||
type CreationModelId,
|
||||
type MentionOption,
|
||||
type ResolutionId,
|
||||
} from "@/lib/creationConfig";
|
||||
import type { LobbyCreationCapabilities } from "@/lib/creationCapabilities";
|
||||
|
||||
interface CreationStudioProps {
|
||||
value: string;
|
||||
disabled?: boolean;
|
||||
isGenerating?: boolean;
|
||||
canSubmit?: boolean;
|
||||
modelId: CreationModelId;
|
||||
modeId: CreationModeId;
|
||||
aspectRatio: AspectRatioId;
|
||||
resolution: ResolutionId;
|
||||
durationSec: number;
|
||||
referencePreviewUrl?: string | null;
|
||||
firstFramePreviewUrl?: string | null;
|
||||
lastFramePreviewUrl?: string | null;
|
||||
mentionOptions?: MentionOption[];
|
||||
storyPresets?: StoryPresetLike[];
|
||||
activeSection?: AppNavSection;
|
||||
onValueChange: (value: string) => void;
|
||||
onSubmit: () => void;
|
||||
onKeyDown?: (event: React.KeyboardEvent<HTMLTextAreaElement>) => void;
|
||||
onModelChange: (modelId: CreationModelId) => void;
|
||||
onModeChange: (modeId: CreationModeId) => void;
|
||||
onAspectRatioChange: (aspectRatio: AspectRatioId) => void;
|
||||
onResolutionChange: (resolution: ResolutionId) => void;
|
||||
onDurationChange: (durationSec: number) => void;
|
||||
onReferenceSelect?: (file: File | null) => void;
|
||||
onFirstFrameSelect?: (file: File | null) => void;
|
||||
onLastFrameSelect?: (file: File | null) => void;
|
||||
onPresetGenerate?: (presetId: string) => void;
|
||||
onSpeechTranscript?: (text: string) => void;
|
||||
onSpeechInterimChange?: (text: string) => void;
|
||||
onOpenProjects?: () => void;
|
||||
capabilities?: LobbyCreationCapabilities;
|
||||
}
|
||||
|
||||
export default function CreationStudio({
|
||||
activeSection = "create",
|
||||
onOpenProjects,
|
||||
storyPresets = [],
|
||||
onPresetGenerate,
|
||||
isGenerating = false,
|
||||
capabilities,
|
||||
...composerProps
|
||||
}: CreationStudioProps) {
|
||||
return (
|
||||
<div className="flex min-h-0 flex-1">
|
||||
<AppNavRail activeSection={activeSection} onOpenProjects={onOpenProjects} />
|
||||
<div className="min-w-0 flex-1 overflow-y-auto">
|
||||
<div className="mx-auto flex w-full max-w-5xl flex-col gap-5 px-4 py-7 sm:px-6 sm:py-8">
|
||||
<CreationComposer {...composerProps} isGenerating={isGenerating} capabilities={capabilities} />
|
||||
{storyPresets.length > 0 && onPresetGenerate && (
|
||||
<PresetQuickLaunchRail storyPresets={storyPresets} disabled={isGenerating} onPresetGenerate={onPresetGenerate} />
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,240 @@
|
||||
"use client";
|
||||
|
||||
import React, { useCallback, useEffect, useRef, useState } from "react";
|
||||
import { ChevronLeft, ChevronRight } from "lucide-react";
|
||||
|
||||
import { cn } from "@/lib/utils";
|
||||
|
||||
export interface StoryPresetLike {
|
||||
id: string;
|
||||
label: string;
|
||||
description?: string;
|
||||
segmentCount?: number;
|
||||
styleTag?: string;
|
||||
}
|
||||
|
||||
interface PresetQuickLaunchRailProps {
|
||||
storyPresets: StoryPresetLike[];
|
||||
disabled?: boolean;
|
||||
onPresetGenerate: (presetId: string) => void;
|
||||
}
|
||||
|
||||
export default function PresetQuickLaunchRail({
|
||||
storyPresets,
|
||||
disabled = false,
|
||||
onPresetGenerate,
|
||||
}: PresetQuickLaunchRailProps) {
|
||||
const scrollRef = useRef<HTMLDivElement>(null);
|
||||
const [canScrollLeft, setCanScrollLeft] = useState(false);
|
||||
const [canScrollRight, setCanScrollRight] = useState(false);
|
||||
const [presetRailDragging, setPresetRailDragging] = useState(false);
|
||||
const presetDragStateRef = useRef({
|
||||
pointerId: null as number | null,
|
||||
startX: 0,
|
||||
startScrollLeft: 0,
|
||||
moved: false,
|
||||
});
|
||||
const suppressPresetClickRef = useRef(false);
|
||||
|
||||
const updateScrollState = useCallback(() => {
|
||||
const el = scrollRef.current;
|
||||
if (!el) return;
|
||||
setCanScrollLeft(el.scrollLeft > 2);
|
||||
setCanScrollRight(el.scrollLeft + el.clientWidth < el.scrollWidth - 2);
|
||||
}, []);
|
||||
|
||||
const scrollByAmount = useCallback(
|
||||
(direction: "left" | "right") => {
|
||||
const el = scrollRef.current;
|
||||
if (!el) return;
|
||||
const delta = direction === "left" ? -220 : 220;
|
||||
el.scrollBy({ left: delta, behavior: "smooth" });
|
||||
window.setTimeout(updateScrollState, 220);
|
||||
},
|
||||
[updateScrollState],
|
||||
);
|
||||
|
||||
const handlePresetWheel = useCallback(
|
||||
(event: React.WheelEvent<HTMLDivElement>) => {
|
||||
const el = scrollRef.current;
|
||||
if (!el) return;
|
||||
if (el.scrollWidth <= el.clientWidth + 1) return;
|
||||
|
||||
const dominantDelta = Math.abs(event.deltaX) > Math.abs(event.deltaY) ? event.deltaX : event.deltaY;
|
||||
if (!dominantDelta) return;
|
||||
|
||||
const maxScrollLeft = Math.max(el.scrollWidth - el.clientWidth, 0);
|
||||
const nextScrollLeft = Math.min(Math.max(el.scrollLeft + dominantDelta, 0), maxScrollLeft);
|
||||
if (nextScrollLeft === el.scrollLeft) return;
|
||||
|
||||
event.preventDefault();
|
||||
el.scrollLeft = nextScrollLeft;
|
||||
updateScrollState();
|
||||
},
|
||||
[updateScrollState],
|
||||
);
|
||||
|
||||
const finishPresetDrag = useCallback(() => {
|
||||
presetDragStateRef.current = {
|
||||
pointerId: null,
|
||||
startX: 0,
|
||||
startScrollLeft: 0,
|
||||
moved: false,
|
||||
};
|
||||
setPresetRailDragging(false);
|
||||
}, []);
|
||||
|
||||
const handlePresetPointerDown = useCallback((event: React.PointerEvent<HTMLDivElement>) => {
|
||||
const el = scrollRef.current;
|
||||
if (!el) return;
|
||||
if (event.pointerType !== "mouse" || event.button !== 0) return;
|
||||
if (el.scrollWidth <= el.clientWidth + 1) return;
|
||||
|
||||
suppressPresetClickRef.current = false;
|
||||
presetDragStateRef.current = {
|
||||
pointerId: event.pointerId,
|
||||
startX: event.clientX,
|
||||
startScrollLeft: el.scrollLeft,
|
||||
moved: false,
|
||||
};
|
||||
}, []);
|
||||
|
||||
const handlePresetPointerMove = useCallback(
|
||||
(event: React.PointerEvent<HTMLDivElement>) => {
|
||||
const el = scrollRef.current;
|
||||
const dragState = presetDragStateRef.current;
|
||||
if (!el || dragState.pointerId !== event.pointerId) return;
|
||||
|
||||
const deltaX = event.clientX - dragState.startX;
|
||||
if (!dragState.moved && Math.abs(deltaX) > 4) {
|
||||
dragState.moved = true;
|
||||
suppressPresetClickRef.current = true;
|
||||
setPresetRailDragging(true);
|
||||
el.setPointerCapture?.(event.pointerId);
|
||||
}
|
||||
if (!dragState.moved) return;
|
||||
|
||||
event.preventDefault();
|
||||
const maxScrollLeft = Math.max(el.scrollWidth - el.clientWidth, 0);
|
||||
el.scrollLeft = Math.min(Math.max(dragState.startScrollLeft - deltaX, 0), maxScrollLeft);
|
||||
updateScrollState();
|
||||
},
|
||||
[updateScrollState],
|
||||
);
|
||||
|
||||
const handlePresetPointerUp = useCallback(
|
||||
(event: React.PointerEvent<HTMLDivElement>) => {
|
||||
const el = scrollRef.current;
|
||||
if (!el || presetDragStateRef.current.pointerId !== event.pointerId) return;
|
||||
if (el.hasPointerCapture?.(event.pointerId)) {
|
||||
el.releasePointerCapture(event.pointerId);
|
||||
}
|
||||
finishPresetDrag();
|
||||
},
|
||||
[finishPresetDrag],
|
||||
);
|
||||
|
||||
const handlePresetClickCapture = useCallback((event: React.MouseEvent<HTMLDivElement>) => {
|
||||
if (!suppressPresetClickRef.current) return;
|
||||
suppressPresetClickRef.current = false;
|
||||
event.preventDefault();
|
||||
event.stopPropagation();
|
||||
}, []);
|
||||
|
||||
useEffect(() => {
|
||||
updateScrollState();
|
||||
}, [storyPresets, updateScrollState]);
|
||||
|
||||
useEffect(() => {
|
||||
const el = scrollRef.current;
|
||||
if (!el) return;
|
||||
|
||||
const observer = new ResizeObserver(() => updateScrollState());
|
||||
observer.observe(el);
|
||||
return () => observer.disconnect();
|
||||
}, [updateScrollState]);
|
||||
|
||||
if (storyPresets.length === 0) return null;
|
||||
|
||||
const scrollMaskStyle =
|
||||
canScrollLeft && canScrollRight
|
||||
? {
|
||||
maskImage: "linear-gradient(to right, transparent, black 20px, black calc(100% - 20px), transparent)",
|
||||
WebkitMaskImage: "linear-gradient(to right, transparent, black 20px, black calc(100% - 20px), transparent)",
|
||||
}
|
||||
: canScrollLeft
|
||||
? {
|
||||
maskImage: "linear-gradient(to right, transparent, black 20px, black)",
|
||||
WebkitMaskImage: "linear-gradient(to right, transparent, black 20px, black)",
|
||||
}
|
||||
: canScrollRight
|
||||
? {
|
||||
maskImage: "linear-gradient(to right, black, black calc(100% - 20px), transparent)",
|
||||
WebkitMaskImage: "linear-gradient(to right, black, black calc(100% - 20px), transparent)",
|
||||
}
|
||||
: undefined;
|
||||
|
||||
return (
|
||||
<div className={cn("mx-auto w-full max-w-3xl transition-opacity duration-200", disabled && "pointer-events-none opacity-40")}>
|
||||
<div className="grid grid-cols-[auto_minmax(0,1fr)_auto] items-center gap-1 sm:gap-2">
|
||||
<div className="flex w-8 shrink-0 justify-center">
|
||||
{canScrollLeft ? (
|
||||
<button
|
||||
type="button"
|
||||
aria-label="Scroll suggested prompts left"
|
||||
onClick={() => scrollByAmount("left")}
|
||||
className="studio-control studio-control-press inline-flex size-8 items-center justify-center rounded-full text-muted-foreground hover-capable:hover:bg-muted/60 hover-capable:hover:text-foreground"
|
||||
>
|
||||
<ChevronLeft className="size-4" />
|
||||
</button>
|
||||
) : null}
|
||||
</div>
|
||||
|
||||
<div
|
||||
ref={scrollRef}
|
||||
onScroll={updateScrollState}
|
||||
onWheel={handlePresetWheel}
|
||||
onPointerDown={handlePresetPointerDown}
|
||||
onPointerMove={handlePresetPointerMove}
|
||||
onPointerUp={handlePresetPointerUp}
|
||||
onPointerCancel={handlePresetPointerUp}
|
||||
onLostPointerCapture={finishPresetDrag}
|
||||
onClickCapture={handlePresetClickCapture}
|
||||
style={scrollMaskStyle}
|
||||
className={cn(
|
||||
"scrollbar-hidden flex gap-2 overflow-x-auto overflow-y-visible py-0.5 select-none",
|
||||
presetRailDragging ? "cursor-grabbing" : "cursor-grab",
|
||||
)}
|
||||
>
|
||||
{storyPresets.map((preset) => (
|
||||
<button
|
||||
key={preset.id}
|
||||
type="button"
|
||||
disabled={disabled}
|
||||
onClick={() => onPresetGenerate(preset.id)}
|
||||
className="studio-control studio-control-press studio-hover-surface flex w-[12.5rem] shrink-0 flex-col gap-1 rounded-xl border border-border/50 bg-card/70 px-3 py-2.5 text-left"
|
||||
>
|
||||
<span className="line-clamp-1 text-sm font-medium text-foreground">{preset.label}</span>
|
||||
{preset.description && (
|
||||
<span className="text-pretty line-clamp-2 text-xs leading-5 text-muted-foreground">{preset.description}</span>
|
||||
)}
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
|
||||
<div className="flex w-8 shrink-0 justify-center">
|
||||
{canScrollRight ? (
|
||||
<button
|
||||
type="button"
|
||||
aria-label="Scroll suggested prompts right"
|
||||
onClick={() => scrollByAmount("right")}
|
||||
className="studio-control studio-control-press inline-flex size-8 items-center justify-center rounded-full text-muted-foreground hover-capable:hover:bg-muted/60 hover-capable:hover:text-foreground"
|
||||
>
|
||||
<ChevronRight className="size-4" />
|
||||
</button>
|
||||
) : null}
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,97 @@
|
||||
"use client";
|
||||
|
||||
import React, { useRef, useState } from "react";
|
||||
import { ImagePlus } from "lucide-react";
|
||||
|
||||
import { REFERENCE_ACCEPT, isReferenceMediaFile } from "@/lib/creationConfig";
|
||||
import { cn } from "@/lib/utils";
|
||||
|
||||
interface ReferenceUploadSlotProps {
|
||||
label: string;
|
||||
sublabel?: string;
|
||||
previewUrl?: string | null;
|
||||
required?: boolean;
|
||||
optional?: boolean;
|
||||
disabled?: boolean;
|
||||
onSelect?: (file: File | null) => void;
|
||||
}
|
||||
|
||||
export default function ReferenceUploadSlot({
|
||||
label,
|
||||
sublabel,
|
||||
previewUrl = null,
|
||||
required = false,
|
||||
optional = false,
|
||||
disabled = false,
|
||||
onSelect,
|
||||
}: ReferenceUploadSlotProps) {
|
||||
const fileInputRef = useRef<HTMLInputElement>(null);
|
||||
const [dragActive, setDragActive] = useState(false);
|
||||
|
||||
function handleFile(file: File | null) {
|
||||
if (!file || !isReferenceMediaFile(file)) return;
|
||||
onSelect?.(file);
|
||||
}
|
||||
|
||||
return (
|
||||
<div className="flex flex-col gap-1">
|
||||
<button
|
||||
type="button"
|
||||
aria-label={[label, sublabel].filter(Boolean).join(" ")}
|
||||
onClick={() => fileInputRef.current?.click()}
|
||||
disabled={disabled}
|
||||
onDragEnter={(event) => {
|
||||
event.preventDefault();
|
||||
event.stopPropagation();
|
||||
if (!disabled) setDragActive(true);
|
||||
}}
|
||||
onDragOver={(event) => {
|
||||
event.preventDefault();
|
||||
event.stopPropagation();
|
||||
if (!disabled) setDragActive(true);
|
||||
}}
|
||||
onDragLeave={(event) => {
|
||||
event.preventDefault();
|
||||
event.stopPropagation();
|
||||
setDragActive(false);
|
||||
}}
|
||||
onDrop={(event) => {
|
||||
event.preventDefault();
|
||||
event.stopPropagation();
|
||||
setDragActive(false);
|
||||
if (disabled) return;
|
||||
handleFile(event.dataTransfer.files?.[0] ?? null);
|
||||
}}
|
||||
className={cn(
|
||||
"studio-control studio-control-press studio-hover-surface relative flex size-[76px] shrink-0 flex-col items-center justify-center gap-1 overflow-hidden rounded-2xl border border-dashed bg-muted/50 px-1 text-center text-[11px] font-medium text-muted-foreground",
|
||||
required && !previewUrl ? "border-amber-500/50" : "border-border/60",
|
||||
dragActive && "border-accent-blue bg-accent-blue/10 ring-2 ring-accent-blue/30",
|
||||
disabled && "pointer-events-none opacity-50",
|
||||
)}
|
||||
>
|
||||
{previewUrl ? (
|
||||
<img src={previewUrl} alt="" className="studio-media-outline absolute inset-0 size-full object-cover" />
|
||||
) : (
|
||||
<>
|
||||
<ImagePlus className="size-4" />
|
||||
<span>{label}</span>
|
||||
{sublabel && <span className="text-[10px] font-normal opacity-70">{sublabel}</span>}
|
||||
</>
|
||||
)}
|
||||
</button>
|
||||
{(required || optional) && (
|
||||
<span className="text-center text-[10px] text-muted-foreground">{required ? "Required" : "Optional"}</span>
|
||||
)}
|
||||
<input
|
||||
ref={fileInputRef}
|
||||
type="file"
|
||||
accept={REFERENCE_ACCEPT}
|
||||
className="hidden"
|
||||
onChange={(event) => {
|
||||
handleFile(event.target.files?.[0] ?? null);
|
||||
event.target.value = "";
|
||||
}}
|
||||
/>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,213 @@
|
||||
"use client";
|
||||
|
||||
import React from "react";
|
||||
import { Box, ChevronDown, Clock, Monitor, Wand2 } from "lucide-react";
|
||||
|
||||
import ConfigPill from "@/components/creation/ConfigPill";
|
||||
import {
|
||||
DropdownMenu,
|
||||
DropdownMenuContent,
|
||||
DropdownMenuItem,
|
||||
DropdownMenuLabel,
|
||||
DropdownMenuSeparator,
|
||||
DropdownMenuTrigger,
|
||||
} from "@/components/ui/dropdown-menu";
|
||||
import { Popover, PopoverContent, PopoverTrigger } from "@/components/ui/popover";
|
||||
import { Slider } from "@/components/ui/slider";
|
||||
import {
|
||||
ASPECT_RATIOS,
|
||||
CREATION_MODELS,
|
||||
CREATION_MODES,
|
||||
RESOLUTIONS,
|
||||
type AspectRatioId,
|
||||
type CreationModeId,
|
||||
type CreationModelId,
|
||||
type ResolutionId,
|
||||
formatDurationLabel,
|
||||
formatResolutionLabel,
|
||||
} from "@/lib/creationConfig";
|
||||
import { cn } from "@/lib/utils";
|
||||
|
||||
export interface SessionCreationConfig {
|
||||
modelId: CreationModelId;
|
||||
modeId: CreationModeId;
|
||||
aspectRatio: AspectRatioId;
|
||||
resolution: ResolutionId;
|
||||
durationSec: number;
|
||||
}
|
||||
|
||||
interface SessionCreationConfigPillsProps extends SessionCreationConfig {
|
||||
disabled?: boolean;
|
||||
readOnly?: boolean;
|
||||
onModelChange?: (modelId: CreationModelId) => void;
|
||||
onModeChange?: (modeId: CreationModeId) => void;
|
||||
onAspectRatioChange?: (aspectRatio: AspectRatioId) => void;
|
||||
onResolutionChange?: (resolution: ResolutionId) => void;
|
||||
onDurationChange?: (durationSec: number) => void;
|
||||
}
|
||||
|
||||
export default function SessionCreationConfigPills({
|
||||
modelId,
|
||||
modeId,
|
||||
aspectRatio,
|
||||
resolution,
|
||||
durationSec,
|
||||
disabled = false,
|
||||
readOnly = false,
|
||||
onModelChange,
|
||||
onModeChange,
|
||||
onAspectRatioChange,
|
||||
onResolutionChange,
|
||||
onDurationChange,
|
||||
}: SessionCreationConfigPillsProps) {
|
||||
const selectedModel = CREATION_MODELS.find((model) => model.id === modelId) ?? CREATION_MODELS[0];
|
||||
const selectedMode = CREATION_MODES.find((mode) => mode.id === modeId) ?? CREATION_MODES[0];
|
||||
const isInteractive = !readOnly && !disabled;
|
||||
|
||||
const pillClassName = cn(
|
||||
"h-9 min-h-9 px-2 text-[11px]",
|
||||
!isInteractive && "pointer-events-none opacity-70",
|
||||
);
|
||||
|
||||
if (readOnly) {
|
||||
return (
|
||||
<div className="flex flex-wrap items-center gap-1.5">
|
||||
<ConfigPill disabled className={pillClassName} aria-label="Model">
|
||||
<Box className="size-3" />
|
||||
{selectedModel.label}
|
||||
</ConfigPill>
|
||||
<ConfigPill disabled className={pillClassName} aria-label="Mode">
|
||||
<Wand2 className="size-3" />
|
||||
{selectedMode.label}
|
||||
</ConfigPill>
|
||||
<ConfigPill disabled className={pillClassName} aria-label="Aspect ratio and resolution">
|
||||
<Monitor className="size-3" />
|
||||
{aspectRatio} {formatResolutionLabel(resolution)}
|
||||
</ConfigPill>
|
||||
<ConfigPill disabled className={pillClassName} aria-label="Duration">
|
||||
<Clock className="size-3" />
|
||||
{formatDurationLabel(durationSec)}
|
||||
</ConfigPill>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
return (
|
||||
<div className="flex flex-wrap items-center gap-1.5">
|
||||
<DropdownMenu>
|
||||
<DropdownMenuTrigger asChild>
|
||||
<ConfigPill disabled={disabled} className={pillClassName} aria-label="Model">
|
||||
<Box className="size-3" />
|
||||
{selectedModel.label}
|
||||
<ChevronDown className="size-2.5 opacity-60" />
|
||||
</ConfigPill>
|
||||
</DropdownMenuTrigger>
|
||||
<DropdownMenuContent align="start" className="w-72">
|
||||
<DropdownMenuLabel>Model</DropdownMenuLabel>
|
||||
<DropdownMenuSeparator />
|
||||
{CREATION_MODELS.map((model) => (
|
||||
<DropdownMenuItem key={model.id} onClick={() => onModelChange?.(model.id)} className="flex-col items-start gap-1 py-2.5">
|
||||
<span className="flex items-center gap-2 text-sm font-medium">
|
||||
{model.label}
|
||||
{model.badge && <span className="rounded-full bg-accent-blue/15 px-1.5 py-0.5 text-[10px] text-accent-blue">{model.badge}</span>}
|
||||
</span>
|
||||
<span className="text-xs text-muted-foreground">{model.description}</span>
|
||||
</DropdownMenuItem>
|
||||
))}
|
||||
</DropdownMenuContent>
|
||||
</DropdownMenu>
|
||||
|
||||
<DropdownMenu>
|
||||
<DropdownMenuTrigger asChild>
|
||||
<ConfigPill disabled={disabled} className={pillClassName} aria-label="Mode">
|
||||
<Wand2 className="size-3" />
|
||||
{selectedMode.label}
|
||||
<ChevronDown className="size-2.5 opacity-60" />
|
||||
</ConfigPill>
|
||||
</DropdownMenuTrigger>
|
||||
<DropdownMenuContent align="start" className="w-64">
|
||||
<DropdownMenuLabel>Mode</DropdownMenuLabel>
|
||||
<DropdownMenuSeparator />
|
||||
{CREATION_MODES.map((mode) => (
|
||||
<DropdownMenuItem key={mode.id} onClick={() => onModeChange?.(mode.id)} className="flex-col items-start gap-1 py-2.5">
|
||||
<span className="text-sm font-medium">{mode.label}</span>
|
||||
<span className="text-xs text-muted-foreground">{mode.description}</span>
|
||||
</DropdownMenuItem>
|
||||
))}
|
||||
</DropdownMenuContent>
|
||||
</DropdownMenu>
|
||||
|
||||
<Popover>
|
||||
<PopoverTrigger asChild>
|
||||
<ConfigPill disabled={disabled} className={pillClassName} aria-label="Aspect ratio and resolution">
|
||||
<Monitor className="size-3" />
|
||||
{aspectRatio} {formatResolutionLabel(resolution)}
|
||||
</ConfigPill>
|
||||
</PopoverTrigger>
|
||||
<PopoverContent align="start" className="w-80">
|
||||
<p className="mb-3 text-xs font-medium text-muted-foreground">Aspect ratio</p>
|
||||
<div className="grid grid-cols-3 gap-2">
|
||||
{ASPECT_RATIOS.map((ratio) => (
|
||||
<button
|
||||
key={ratio}
|
||||
type="button"
|
||||
onClick={() => onAspectRatioChange?.(ratio)}
|
||||
className={cn(
|
||||
"studio-control studio-control-press studio-hover-surface flex flex-col items-center gap-2 rounded-xl border px-2 py-3 text-xs",
|
||||
aspectRatio === ratio ? "border-accent-blue bg-accent-blue/10 text-foreground" : "border-border",
|
||||
)}
|
||||
>
|
||||
<span
|
||||
className={cn(
|
||||
"rounded-sm border border-current/40 bg-muted/40",
|
||||
ratio === "9:16" && "h-7 w-4",
|
||||
ratio === "16:9" && "h-4 w-7",
|
||||
ratio === "1:1" && "size-5",
|
||||
ratio === "4:3" && "h-5 w-6",
|
||||
ratio === "3:4" && "h-6 w-5",
|
||||
ratio === "21:9" && "h-3 w-8",
|
||||
)}
|
||||
/>
|
||||
{ratio}
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
<p className="mb-2 mt-4 text-xs font-medium text-muted-foreground">Resolution</p>
|
||||
<div className="flex flex-wrap gap-2">
|
||||
{RESOLUTIONS.map((item) => (
|
||||
<button
|
||||
key={item}
|
||||
type="button"
|
||||
onClick={() => onResolutionChange?.(item)}
|
||||
className={cn(
|
||||
"studio-control studio-control-press studio-hover-surface rounded-full border px-3 py-1.5 text-xs font-medium",
|
||||
resolution === item ? "border-accent-blue bg-accent-blue/10 text-foreground" : "border-border",
|
||||
)}
|
||||
>
|
||||
{formatResolutionLabel(item)}
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
</PopoverContent>
|
||||
</Popover>
|
||||
|
||||
<Popover>
|
||||
<PopoverTrigger asChild>
|
||||
<ConfigPill disabled={disabled} className={pillClassName} aria-label="Duration">
|
||||
<Clock className="size-3" />
|
||||
{formatDurationLabel(durationSec)}
|
||||
</ConfigPill>
|
||||
</PopoverTrigger>
|
||||
<PopoverContent align="start" className="w-72">
|
||||
<p className="mb-3 text-xs font-medium text-muted-foreground">Total duration</p>
|
||||
<Slider min={5} max={15} step={5} value={[durationSec]} onValueChange={(values) => onDurationChange?.(values[0] ?? 5)} />
|
||||
<div className="mt-3 flex items-center justify-between text-[11px] text-muted-foreground">
|
||||
<span>5s</span>
|
||||
<span className="rounded-md border border-border px-2 py-1 text-xs font-medium text-foreground">{formatDurationLabel(durationSec)}</span>
|
||||
<span>15s</span>
|
||||
</div>
|
||||
</PopoverContent>
|
||||
</Popover>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,141 @@
|
||||
"use client";
|
||||
|
||||
import * as React from "react";
|
||||
import * as DropdownMenuPrimitive from "@radix-ui/react-dropdown-menu";
|
||||
import { Check, ChevronRight } from "lucide-react";
|
||||
|
||||
import { cn } from "@/lib/utils";
|
||||
|
||||
const DropdownMenu = DropdownMenuPrimitive.Root;
|
||||
const DropdownMenuTrigger = DropdownMenuPrimitive.Trigger;
|
||||
const DropdownMenuGroup = DropdownMenuPrimitive.Group;
|
||||
const DropdownMenuPortal = DropdownMenuPrimitive.Portal;
|
||||
const DropdownMenuSub = DropdownMenuPrimitive.Sub;
|
||||
const DropdownMenuRadioGroup = DropdownMenuPrimitive.RadioGroup;
|
||||
|
||||
const DropdownMenuSubTrigger = React.forwardRef<
|
||||
React.ElementRef<typeof DropdownMenuPrimitive.SubTrigger>,
|
||||
React.ComponentPropsWithoutRef<typeof DropdownMenuPrimitive.SubTrigger> & { inset?: boolean }
|
||||
>(({ className, inset, children, ...props }, ref) => (
|
||||
<DropdownMenuPrimitive.SubTrigger
|
||||
ref={ref}
|
||||
className={cn(
|
||||
"flex cursor-default select-none items-center rounded-xl px-2 py-1.5 text-sm outline-none data-[state=open]:bg-accent focus:bg-accent",
|
||||
inset && "pl-8",
|
||||
className,
|
||||
)}
|
||||
{...props}
|
||||
>
|
||||
{children}
|
||||
<ChevronRight className="ml-auto size-4" />
|
||||
</DropdownMenuPrimitive.SubTrigger>
|
||||
));
|
||||
DropdownMenuSubTrigger.displayName = DropdownMenuPrimitive.SubTrigger.displayName;
|
||||
|
||||
const DropdownMenuSubContent = React.forwardRef<
|
||||
React.ElementRef<typeof DropdownMenuPrimitive.SubContent>,
|
||||
React.ComponentPropsWithoutRef<typeof DropdownMenuPrimitive.SubContent>
|
||||
>(({ className, ...props }, ref) => (
|
||||
<DropdownMenuPrimitive.SubContent
|
||||
ref={ref}
|
||||
className={cn(
|
||||
"z-50 min-w-[8rem] overflow-hidden rounded-2xl border border-border bg-popover/95 p-1 text-popover-foreground shadow-xl backdrop-blur-md",
|
||||
"data-[state=open]:animate-in data-[state=closed]:animate-out data-[state=closed]:fade-out-0 data-[state=open]:fade-in-0 data-[state=closed]:zoom-out-95 data-[state=open]:zoom-in-95",
|
||||
"data-[side=bottom]:slide-in-from-top-2 data-[side=top]:slide-in-from-bottom-2 data-[side=left]:slide-in-from-right-2 data-[side=right]:slide-in-from-left-2",
|
||||
className,
|
||||
)}
|
||||
{...props}
|
||||
/>
|
||||
));
|
||||
DropdownMenuSubContent.displayName = DropdownMenuPrimitive.SubContent.displayName;
|
||||
|
||||
const DropdownMenuContent = React.forwardRef<
|
||||
React.ElementRef<typeof DropdownMenuPrimitive.Content>,
|
||||
React.ComponentPropsWithoutRef<typeof DropdownMenuPrimitive.Content>
|
||||
>(({ className, sideOffset = 6, ...props }, ref) => (
|
||||
<DropdownMenuPrimitive.Portal>
|
||||
<DropdownMenuPrimitive.Content
|
||||
ref={ref}
|
||||
sideOffset={sideOffset}
|
||||
className={cn(
|
||||
"z-50 min-w-[12rem] overflow-hidden rounded-2xl border border-border bg-popover/95 p-1.5 text-popover-foreground shadow-xl backdrop-blur-md",
|
||||
"data-[state=open]:animate-in data-[state=closed]:animate-out data-[state=closed]:fade-out-0 data-[state=open]:fade-in-0 data-[state=closed]:zoom-out-95 data-[state=open]:zoom-in-95",
|
||||
"data-[side=bottom]:slide-in-from-top-2 data-[side=top]:slide-in-from-bottom-2 data-[side=left]:slide-in-from-right-2 data-[side=right]:slide-in-from-left-2",
|
||||
className,
|
||||
)}
|
||||
{...props}
|
||||
/>
|
||||
</DropdownMenuPrimitive.Portal>
|
||||
));
|
||||
DropdownMenuContent.displayName = DropdownMenuPrimitive.Content.displayName;
|
||||
|
||||
const DropdownMenuItem = React.forwardRef<
|
||||
React.ElementRef<typeof DropdownMenuPrimitive.Item>,
|
||||
React.ComponentPropsWithoutRef<typeof DropdownMenuPrimitive.Item> & { inset?: boolean }
|
||||
>(({ className, inset, ...props }, ref) => (
|
||||
<DropdownMenuPrimitive.Item
|
||||
ref={ref}
|
||||
className={cn(
|
||||
"relative flex cursor-default select-none items-center gap-2 rounded-xl px-2.5 py-2 text-sm outline-none transition-colors data-[disabled]:pointer-events-none data-[disabled]:opacity-50 focus:bg-accent focus:text-accent-foreground",
|
||||
inset && "pl-8",
|
||||
className,
|
||||
)}
|
||||
{...props}
|
||||
/>
|
||||
));
|
||||
DropdownMenuItem.displayName = DropdownMenuPrimitive.Item.displayName;
|
||||
|
||||
const DropdownMenuCheckboxItem = React.forwardRef<
|
||||
React.ElementRef<typeof DropdownMenuPrimitive.CheckboxItem>,
|
||||
React.ComponentPropsWithoutRef<typeof DropdownMenuPrimitive.CheckboxItem>
|
||||
>(({ className, children, checked, ...props }, ref) => (
|
||||
<DropdownMenuPrimitive.CheckboxItem
|
||||
ref={ref}
|
||||
className={cn(
|
||||
"relative flex cursor-default select-none items-center rounded-xl py-2 pl-8 pr-2 text-sm outline-none transition-colors data-[disabled]:pointer-events-none data-[disabled]:opacity-50 focus:bg-accent focus:text-accent-foreground",
|
||||
className,
|
||||
)}
|
||||
checked={checked}
|
||||
{...props}
|
||||
>
|
||||
<span className="absolute left-2 flex size-3.5 items-center justify-center">
|
||||
<DropdownMenuPrimitive.ItemIndicator>
|
||||
<Check className="size-4 text-accent-blue" />
|
||||
</DropdownMenuPrimitive.ItemIndicator>
|
||||
</span>
|
||||
{children}
|
||||
</DropdownMenuPrimitive.CheckboxItem>
|
||||
));
|
||||
DropdownMenuCheckboxItem.displayName = DropdownMenuPrimitive.CheckboxItem.displayName;
|
||||
|
||||
const DropdownMenuLabel = React.forwardRef<
|
||||
React.ElementRef<typeof DropdownMenuPrimitive.Label>,
|
||||
React.ComponentPropsWithoutRef<typeof DropdownMenuPrimitive.Label> & { inset?: boolean }
|
||||
>(({ className, inset, ...props }, ref) => (
|
||||
<DropdownMenuPrimitive.Label ref={ref} className={cn("px-2 py-1.5 text-xs font-semibold text-muted-foreground", inset && "pl-8", className)} {...props} />
|
||||
));
|
||||
DropdownMenuLabel.displayName = DropdownMenuPrimitive.Label.displayName;
|
||||
|
||||
const DropdownMenuSeparator = React.forwardRef<
|
||||
React.ElementRef<typeof DropdownMenuPrimitive.Separator>,
|
||||
React.ComponentPropsWithoutRef<typeof DropdownMenuPrimitive.Separator>
|
||||
>(({ className, ...props }, ref) => (
|
||||
<DropdownMenuPrimitive.Separator ref={ref} className={cn("-mx-1 my-1 h-px bg-border", className)} {...props} />
|
||||
));
|
||||
DropdownMenuSeparator.displayName = DropdownMenuPrimitive.Separator.displayName;
|
||||
|
||||
export {
|
||||
DropdownMenu,
|
||||
DropdownMenuTrigger,
|
||||
DropdownMenuContent,
|
||||
DropdownMenuItem,
|
||||
DropdownMenuCheckboxItem,
|
||||
DropdownMenuLabel,
|
||||
DropdownMenuSeparator,
|
||||
DropdownMenuGroup,
|
||||
DropdownMenuPortal,
|
||||
DropdownMenuSub,
|
||||
DropdownMenuSubContent,
|
||||
DropdownMenuSubTrigger,
|
||||
DropdownMenuRadioGroup,
|
||||
};
|
||||
@@ -0,0 +1,33 @@
|
||||
"use client";
|
||||
|
||||
import * as React from "react";
|
||||
import * as PopoverPrimitive from "@radix-ui/react-popover";
|
||||
|
||||
import { cn } from "@/lib/utils";
|
||||
|
||||
const Popover = PopoverPrimitive.Root;
|
||||
const PopoverTrigger = PopoverPrimitive.Trigger;
|
||||
const PopoverAnchor = PopoverPrimitive.Anchor;
|
||||
|
||||
const PopoverContent = React.forwardRef<
|
||||
React.ElementRef<typeof PopoverPrimitive.Content>,
|
||||
React.ComponentPropsWithoutRef<typeof PopoverPrimitive.Content>
|
||||
>(({ className, align = "center", sideOffset = 6, ...props }, ref) => (
|
||||
<PopoverPrimitive.Portal>
|
||||
<PopoverPrimitive.Content
|
||||
ref={ref}
|
||||
align={align}
|
||||
sideOffset={sideOffset}
|
||||
className={cn(
|
||||
"z-50 w-72 rounded-2xl border border-border bg-popover/95 p-3 text-popover-foreground shadow-xl backdrop-blur-md outline-none",
|
||||
"data-[state=open]:animate-in data-[state=closed]:animate-out data-[state=closed]:fade-out-0 data-[state=open]:fade-in-0 data-[state=closed]:zoom-out-95 data-[state=open]:zoom-in-95",
|
||||
"data-[side=bottom]:slide-in-from-top-2 data-[side=top]:slide-in-from-bottom-2 data-[side=left]:slide-in-from-right-2 data-[side=right]:slide-in-from-left-2",
|
||||
className,
|
||||
)}
|
||||
{...props}
|
||||
/>
|
||||
</PopoverPrimitive.Portal>
|
||||
));
|
||||
PopoverContent.displayName = PopoverPrimitive.Content.displayName;
|
||||
|
||||
export { Popover, PopoverTrigger, PopoverContent, PopoverAnchor };
|
||||
@@ -0,0 +1,25 @@
|
||||
"use client";
|
||||
|
||||
import * as React from "react";
|
||||
import * as SliderPrimitive from "@radix-ui/react-slider";
|
||||
|
||||
import { cn } from "@/lib/utils";
|
||||
|
||||
const Slider = React.forwardRef<
|
||||
React.ElementRef<typeof SliderPrimitive.Root>,
|
||||
React.ComponentPropsWithoutRef<typeof SliderPrimitive.Root>
|
||||
>(({ className, ...props }, ref) => (
|
||||
<SliderPrimitive.Root
|
||||
ref={ref}
|
||||
className={cn("relative flex w-full touch-none select-none items-center", className)}
|
||||
{...props}
|
||||
>
|
||||
<SliderPrimitive.Track className="relative h-1.5 w-full grow overflow-hidden rounded-full bg-muted">
|
||||
<SliderPrimitive.Range className="absolute h-full bg-accent-blue" />
|
||||
</SliderPrimitive.Track>
|
||||
<SliderPrimitive.Thumb className="block size-4 rounded-full border border-accent-blue/40 bg-background shadow transition-colors focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-accent-blue/40 disabled:pointer-events-none disabled:opacity-50" />
|
||||
</SliderPrimitive.Root>
|
||||
));
|
||||
Slider.displayName = SliderPrimitive.Root.displayName;
|
||||
|
||||
export { Slider };
|
||||
@@ -0,0 +1,48 @@
|
||||
"use client";
|
||||
|
||||
import * as React from "react";
|
||||
import * as TabsPrimitive from "@radix-ui/react-tabs";
|
||||
|
||||
import { cn } from "@/lib/utils";
|
||||
|
||||
const Tabs = TabsPrimitive.Root;
|
||||
|
||||
const TabsList = React.forwardRef<
|
||||
React.ElementRef<typeof TabsPrimitive.List>,
|
||||
React.ComponentPropsWithoutRef<typeof TabsPrimitive.List>
|
||||
>(({ className, ...props }, ref) => (
|
||||
<TabsPrimitive.List
|
||||
ref={ref}
|
||||
className={cn("inline-flex items-center gap-1 rounded-full bg-muted/60 p-1 text-muted-foreground", className)}
|
||||
{...props}
|
||||
/>
|
||||
));
|
||||
TabsList.displayName = TabsPrimitive.List.displayName;
|
||||
|
||||
const TabsTrigger = React.forwardRef<
|
||||
React.ElementRef<typeof TabsPrimitive.Trigger>,
|
||||
React.ComponentPropsWithoutRef<typeof TabsPrimitive.Trigger>
|
||||
>(({ className, ...props }, ref) => (
|
||||
<TabsPrimitive.Trigger
|
||||
ref={ref}
|
||||
className={cn(
|
||||
"inline-flex items-center justify-center rounded-full px-3 py-1.5 text-xs font-medium whitespace-nowrap transition-all",
|
||||
"focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-accent-blue/40",
|
||||
"disabled:pointer-events-none disabled:opacity-50",
|
||||
"data-[state=active]:bg-card data-[state=active]:text-foreground data-[state=active]:shadow-sm",
|
||||
className,
|
||||
)}
|
||||
{...props}
|
||||
/>
|
||||
));
|
||||
TabsTrigger.displayName = TabsPrimitive.Trigger.displayName;
|
||||
|
||||
const TabsContent = React.forwardRef<
|
||||
React.ElementRef<typeof TabsPrimitive.Content>,
|
||||
React.ComponentPropsWithoutRef<typeof TabsPrimitive.Content>
|
||||
>(({ className, ...props }, ref) => (
|
||||
<TabsPrimitive.Content ref={ref} className={cn("mt-4 outline-none", className)} {...props} />
|
||||
));
|
||||
TabsContent.displayName = TabsPrimitive.Content.displayName;
|
||||
|
||||
export { Tabs, TabsList, TabsTrigger, TabsContent };
|
||||
@@ -0,0 +1,80 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
import {
|
||||
DEFAULT_LOBBY_CAPABILITIES_BUNDLE,
|
||||
clampLobbySelectionToCapabilities,
|
||||
parseLobbyCapabilitiesBundle,
|
||||
resolveModelCapabilities,
|
||||
validateLobbyCreationSelection,
|
||||
} from "@/lib/creationCapabilities";
|
||||
|
||||
describe("creationCapabilities", () => {
|
||||
it("parses backend capability payloads with per-model caps", () => {
|
||||
const bundle = parseLobbyCapabilitiesBundle({
|
||||
model_ids: ["fast-ltx2", "fast-h3"],
|
||||
models: {
|
||||
"fast-ltx2": {
|
||||
generation_modes: ["t2va"],
|
||||
resolutions: ["480p", "720p"],
|
||||
duration_sec: [5, 10],
|
||||
},
|
||||
"fast-h3": {
|
||||
generation_modes: ["t2va", "ref2va"],
|
||||
aspect_ratios: ["16:9"],
|
||||
resolutions: ["720p"],
|
||||
},
|
||||
},
|
||||
});
|
||||
expect(bundle.model_ids).toEqual(["fast-ltx2", "fast-h3"]);
|
||||
expect(bundle.models["fast-h3"]?.aspect_ratios).toEqual(["16:9"]);
|
||||
});
|
||||
|
||||
it("includes fast-h3 in default lobby models", () => {
|
||||
expect(DEFAULT_LOBBY_CAPABILITIES_BUNDLE.model_ids).toContain("fast-h3");
|
||||
});
|
||||
|
||||
it("clamps unsupported lobby selections to model-specific defaults", () => {
|
||||
expect(
|
||||
clampLobbySelectionToCapabilities({
|
||||
capabilities: resolveModelCapabilities(DEFAULT_LOBBY_CAPABILITIES_BUNDLE, "fast-h3"),
|
||||
modelId: "fast-h3",
|
||||
modeId: "fl2av",
|
||||
aspectRatio: "9:16",
|
||||
resolution: "4k",
|
||||
durationSec: 99,
|
||||
}),
|
||||
).toEqual({
|
||||
modelId: "fast-h3",
|
||||
modeId: "t2v",
|
||||
aspectRatio: "16:9",
|
||||
resolution: "720p",
|
||||
durationSec: 5,
|
||||
});
|
||||
});
|
||||
|
||||
it("rejects unsupported generation modes with a clear message", () => {
|
||||
expect(
|
||||
validateLobbyCreationSelection({
|
||||
capabilities: resolveModelCapabilities(DEFAULT_LOBBY_CAPABILITIES_BUNDLE, "fast-ltx23"),
|
||||
modelId: "fast-ltx23",
|
||||
modeId: "fl2av",
|
||||
aspectRatio: "16:9",
|
||||
resolution: "720p",
|
||||
durationSec: 5,
|
||||
}),
|
||||
).toMatch(/FL2VA/i);
|
||||
});
|
||||
|
||||
it("rejects unsupported resolutions for ltx models", () => {
|
||||
expect(
|
||||
validateLobbyCreationSelection({
|
||||
capabilities: resolveModelCapabilities(DEFAULT_LOBBY_CAPABILITIES_BUNDLE, "fast-ltx23"),
|
||||
modelId: "fast-ltx23",
|
||||
modeId: "t2v",
|
||||
aspectRatio: "16:9",
|
||||
resolution: "4k",
|
||||
durationSec: 5,
|
||||
}),
|
||||
).toMatch(/resolution/i);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,263 @@
|
||||
import type {
|
||||
AspectRatioId,
|
||||
CreationModeId,
|
||||
CreationModelId,
|
||||
ResolutionId,
|
||||
} from "@/lib/creationConfig";
|
||||
import { fromGenerationMode, toGenerationMode, type GenerationMode } from "@/lib/generationMode";
|
||||
|
||||
const ALL_MODEL_IDS: CreationModelId[] = ["fast-ltx23", "fast-ltx2", "fast-h3"];
|
||||
const ALL_GENERATION_MODES: GenerationMode[] = ["t2va", "fl2va", "ref2va"];
|
||||
const ALL_ASPECT_RATIOS: AspectRatioId[] = ["21:9", "16:9", "4:3", "1:1", "3:4", "9:16"];
|
||||
const ALL_RESOLUTIONS: ResolutionId[] = ["480p", "720p", "1080p", "4k"];
|
||||
|
||||
export interface ModelCreationCapabilities {
|
||||
generation_modes: GenerationMode[];
|
||||
aspect_ratios: AspectRatioId[];
|
||||
resolutions: ResolutionId[];
|
||||
duration_sec: number[];
|
||||
unsupported_generation_modes: Record<string, string>;
|
||||
reference_assets: {
|
||||
mime_types: string[];
|
||||
max_bytes: number;
|
||||
};
|
||||
}
|
||||
|
||||
export interface LobbyCreationCapabilities extends ModelCreationCapabilities {
|
||||
model_ids: CreationModelId[];
|
||||
}
|
||||
|
||||
export interface LobbyCapabilitiesBundle {
|
||||
model_ids: CreationModelId[];
|
||||
models: Partial<Record<CreationModelId, ModelCreationCapabilities>>;
|
||||
generation_modes: GenerationMode[];
|
||||
aspect_ratios: AspectRatioId[];
|
||||
resolutions: ResolutionId[];
|
||||
duration_sec: number[];
|
||||
unsupported_generation_modes: Record<string, string>;
|
||||
reference_assets: {
|
||||
mime_types: string[];
|
||||
max_bytes: number;
|
||||
};
|
||||
}
|
||||
|
||||
const DEFAULT_LTX_MODEL_CAPABILITIES: ModelCreationCapabilities = {
|
||||
generation_modes: ["t2va", "ref2va"],
|
||||
aspect_ratios: ["21:9", "16:9", "4:3", "1:1", "3:4", "9:16"],
|
||||
resolutions: ["480p", "720p", "1080p"],
|
||||
duration_sec: [5, 10, 15],
|
||||
unsupported_generation_modes: {
|
||||
fl2va: "First/last frame mode (FL2VA) is not supported yet.",
|
||||
},
|
||||
reference_assets: {
|
||||
mime_types: ["image/png", "image/jpeg", "image/webp"],
|
||||
max_bytes: 15 * 1024 * 1024,
|
||||
},
|
||||
};
|
||||
|
||||
const DEFAULT_H3_MODEL_CAPABILITIES: ModelCreationCapabilities = {
|
||||
generation_modes: ["t2va", "ref2va"],
|
||||
aspect_ratios: ["16:9"],
|
||||
resolutions: ["720p"],
|
||||
duration_sec: [5, 10, 15],
|
||||
unsupported_generation_modes: {
|
||||
fl2va: "First/last frame mode (FL2VA) is not supported yet.",
|
||||
},
|
||||
reference_assets: DEFAULT_LTX_MODEL_CAPABILITIES.reference_assets,
|
||||
};
|
||||
|
||||
export const DEFAULT_LOBBY_CAPABILITIES_BUNDLE: LobbyCapabilitiesBundle = {
|
||||
model_ids: ALL_MODEL_IDS,
|
||||
models: {
|
||||
"fast-ltx2": DEFAULT_LTX_MODEL_CAPABILITIES,
|
||||
"fast-ltx23": DEFAULT_LTX_MODEL_CAPABILITIES,
|
||||
"fast-h3": DEFAULT_H3_MODEL_CAPABILITIES,
|
||||
},
|
||||
generation_modes: ["t2va", "ref2va"],
|
||||
aspect_ratios: ["21:9", "16:9", "4:3", "1:1", "3:4", "9:16"],
|
||||
resolutions: ["480p", "720p", "1080p"],
|
||||
duration_sec: [5, 10, 15],
|
||||
unsupported_generation_modes: DEFAULT_LTX_MODEL_CAPABILITIES.unsupported_generation_modes,
|
||||
reference_assets: DEFAULT_LTX_MODEL_CAPABILITIES.reference_assets,
|
||||
};
|
||||
|
||||
function pickStrings<T extends string>(value: unknown, allowed: readonly T[], fallback: readonly T[]): T[] {
|
||||
if (!Array.isArray(value)) return [...fallback];
|
||||
return value.filter((item): item is T => typeof item === "string" && allowed.includes(item as T));
|
||||
}
|
||||
|
||||
function parseReferenceAssets(
|
||||
value: unknown,
|
||||
fallback: ModelCreationCapabilities["reference_assets"],
|
||||
): ModelCreationCapabilities["reference_assets"] {
|
||||
if (!value || typeof value !== "object") return fallback;
|
||||
const data = value as Record<string, unknown>;
|
||||
return {
|
||||
mime_types: Array.isArray(data.mime_types)
|
||||
? (data.mime_types as string[])
|
||||
: fallback.mime_types,
|
||||
max_bytes: typeof data.max_bytes === "number" ? data.max_bytes : fallback.max_bytes,
|
||||
};
|
||||
}
|
||||
|
||||
function parseModelCreationCapabilities(
|
||||
value: unknown,
|
||||
fallback: ModelCreationCapabilities,
|
||||
): ModelCreationCapabilities {
|
||||
if (!value || typeof value !== "object") return fallback;
|
||||
const data = value as Record<string, unknown>;
|
||||
return {
|
||||
generation_modes: pickStrings(data.generation_modes, ALL_GENERATION_MODES, fallback.generation_modes),
|
||||
aspect_ratios: pickStrings(data.aspect_ratios, ALL_ASPECT_RATIOS, fallback.aspect_ratios),
|
||||
resolutions: pickStrings(data.resolutions, ALL_RESOLUTIONS, fallback.resolutions),
|
||||
duration_sec: Array.isArray(data.duration_sec)
|
||||
? data.duration_sec.filter((item): item is number => typeof item === "number")
|
||||
: fallback.duration_sec,
|
||||
unsupported_generation_modes:
|
||||
typeof data.unsupported_generation_modes === "object" && data.unsupported_generation_modes
|
||||
? (data.unsupported_generation_modes as Record<string, string>)
|
||||
: fallback.unsupported_generation_modes,
|
||||
reference_assets: parseReferenceAssets(data.reference_assets, fallback.reference_assets),
|
||||
};
|
||||
}
|
||||
|
||||
export function parseLobbyCapabilitiesBundle(payload: unknown): LobbyCapabilitiesBundle {
|
||||
if (!payload || typeof payload !== "object") {
|
||||
return DEFAULT_LOBBY_CAPABILITIES_BUNDLE;
|
||||
}
|
||||
const data = payload as Record<string, unknown>;
|
||||
const modelIds = pickStrings(data.model_ids, ALL_MODEL_IDS, DEFAULT_LOBBY_CAPABILITIES_BUNDLE.model_ids);
|
||||
const rawModels = typeof data.models === "object" && data.models ? (data.models as Record<string, unknown>) : {};
|
||||
const models: Partial<Record<CreationModelId, ModelCreationCapabilities>> = {};
|
||||
for (const modelId of modelIds) {
|
||||
const fallback =
|
||||
DEFAULT_LOBBY_CAPABILITIES_BUNDLE.models[modelId] ??
|
||||
(modelId === "fast-h3" ? DEFAULT_H3_MODEL_CAPABILITIES : DEFAULT_LTX_MODEL_CAPABILITIES);
|
||||
models[modelId] = parseModelCreationCapabilities(rawModels[modelId], fallback);
|
||||
}
|
||||
const unionFallback = parseModelCreationCapabilities(payload, DEFAULT_LTX_MODEL_CAPABILITIES);
|
||||
return {
|
||||
model_ids: modelIds,
|
||||
models,
|
||||
generation_modes: unionFallback.generation_modes,
|
||||
aspect_ratios: unionFallback.aspect_ratios,
|
||||
resolutions: unionFallback.resolutions,
|
||||
duration_sec: unionFallback.duration_sec,
|
||||
unsupported_generation_modes: unionFallback.unsupported_generation_modes,
|
||||
reference_assets: unionFallback.reference_assets,
|
||||
};
|
||||
}
|
||||
|
||||
export function resolveModelCapabilities(
|
||||
bundle: LobbyCapabilitiesBundle,
|
||||
modelId: CreationModelId,
|
||||
): LobbyCreationCapabilities {
|
||||
const modelCaps =
|
||||
bundle.models[modelId] ??
|
||||
(modelId === "fast-h3" ? DEFAULT_H3_MODEL_CAPABILITIES : DEFAULT_LTX_MODEL_CAPABILITIES);
|
||||
return {
|
||||
model_ids: bundle.model_ids,
|
||||
...modelCaps,
|
||||
};
|
||||
}
|
||||
|
||||
export function supportedCreationModes(capabilities: LobbyCreationCapabilities) {
|
||||
return capabilities.generation_modes.map((wireMode) => ({
|
||||
wireMode,
|
||||
modeId: fromGenerationMode(wireMode),
|
||||
}));
|
||||
}
|
||||
|
||||
export function isSupportedCreationMode(modeId: CreationModeId, capabilities: LobbyCreationCapabilities): boolean {
|
||||
return capabilities.generation_modes.includes(toGenerationMode(modeId));
|
||||
}
|
||||
|
||||
export function isSupportedResolution(resolution: ResolutionId, capabilities: LobbyCreationCapabilities): boolean {
|
||||
return capabilities.resolutions.includes(resolution);
|
||||
}
|
||||
|
||||
export function isSupportedReferenceImage(file: File, capabilities: LobbyCreationCapabilities): boolean {
|
||||
return capabilities.reference_assets.mime_types.includes(file.type);
|
||||
}
|
||||
|
||||
export function unsupportedModeNotice(modeId: CreationModeId, capabilities: LobbyCreationCapabilities): string | null {
|
||||
const wireMode = toGenerationMode(modeId);
|
||||
return capabilities.unsupported_generation_modes[wireMode] ?? null;
|
||||
}
|
||||
|
||||
export function clampLobbySelectionToCapabilities(input: {
|
||||
capabilities: LobbyCreationCapabilities;
|
||||
modelId: CreationModelId;
|
||||
modeId: CreationModeId;
|
||||
aspectRatio: AspectRatioId;
|
||||
resolution: ResolutionId;
|
||||
durationSec: number;
|
||||
}): {
|
||||
modelId: CreationModelId;
|
||||
modeId: CreationModeId;
|
||||
aspectRatio: AspectRatioId;
|
||||
resolution: ResolutionId;
|
||||
durationSec: number;
|
||||
} {
|
||||
const { capabilities } = input;
|
||||
const modelId = capabilities.model_ids.includes(input.modelId)
|
||||
? input.modelId
|
||||
: (capabilities.model_ids[0] ?? "fast-ltx23");
|
||||
const supportedModes = supportedCreationModes(capabilities);
|
||||
const modeId = isSupportedCreationMode(input.modeId, capabilities)
|
||||
? input.modeId
|
||||
: (supportedModes[0]?.modeId ?? "t2v");
|
||||
const aspectRatio = capabilities.aspect_ratios.includes(input.aspectRatio)
|
||||
? input.aspectRatio
|
||||
: (capabilities.aspect_ratios[0] ?? "16:9");
|
||||
const resolution = isSupportedResolution(input.resolution, capabilities)
|
||||
? input.resolution
|
||||
: (capabilities.resolutions[0] ?? "720p");
|
||||
const durationSec = capabilities.duration_sec.includes(input.durationSec)
|
||||
? input.durationSec
|
||||
: (capabilities.duration_sec[0] ?? 5);
|
||||
return { modelId, modeId, aspectRatio, resolution, durationSec };
|
||||
}
|
||||
|
||||
export function validateLobbyCreationSelection(input: {
|
||||
capabilities: LobbyCreationCapabilities;
|
||||
modelId: CreationModelId;
|
||||
modeId: CreationModeId;
|
||||
aspectRatio: AspectRatioId;
|
||||
resolution: ResolutionId;
|
||||
durationSec: number;
|
||||
referenceFile?: File | null;
|
||||
firstFrameFile?: File | null;
|
||||
lastFrameFile?: File | null;
|
||||
}): string | null {
|
||||
const unsupportedMode = unsupportedModeNotice(input.modeId, input.capabilities);
|
||||
if (unsupportedMode) return unsupportedMode;
|
||||
if (!input.capabilities.model_ids.includes(input.modelId)) {
|
||||
return "Selected model is not supported yet.";
|
||||
}
|
||||
if (!isSupportedCreationMode(input.modeId, input.capabilities)) {
|
||||
return "Selected mode is not supported yet.";
|
||||
}
|
||||
if (!input.capabilities.aspect_ratios.includes(input.aspectRatio)) {
|
||||
return "Selected aspect ratio is not supported for this model yet.";
|
||||
}
|
||||
if (!isSupportedResolution(input.resolution, input.capabilities)) {
|
||||
return "Selected resolution is not supported for this model yet.";
|
||||
}
|
||||
if (!input.capabilities.duration_sec.includes(input.durationSec)) {
|
||||
return "Selected duration is not supported yet.";
|
||||
}
|
||||
if (input.modeId === "ref2av" && !input.referenceFile) {
|
||||
return "Upload a reference image to use reference-guided mode.";
|
||||
}
|
||||
if (input.referenceFile && !isSupportedReferenceImage(input.referenceFile, input.capabilities)) {
|
||||
return "Reference assets must be PNG, JPEG, or WebP images.";
|
||||
}
|
||||
if (input.firstFrameFile && !isSupportedReferenceImage(input.firstFrameFile, input.capabilities)) {
|
||||
return "First frame must be a PNG, JPEG, or WebP image.";
|
||||
}
|
||||
if (input.lastFrameFile && !isSupportedReferenceImage(input.lastFrameFile, input.capabilities)) {
|
||||
return "Last frame must be a PNG, JPEG, or WebP image.";
|
||||
}
|
||||
return null;
|
||||
}
|
||||
@@ -0,0 +1,62 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
import {
|
||||
CREATION_MODELS,
|
||||
buildMentionOptions,
|
||||
formatDurationLabel,
|
||||
formatResolutionLabel,
|
||||
isReferenceMediaFile,
|
||||
modeRequiresReference,
|
||||
modeUsesDualFrames,
|
||||
} from "@/lib/creationConfig";
|
||||
|
||||
describe("creationConfig", () => {
|
||||
it("formats resolution labels", () => {
|
||||
expect(formatResolutionLabel("480p")).toBe("480P");
|
||||
expect(formatResolutionLabel("720p")).toBe("720P");
|
||||
expect(formatResolutionLabel("4k")).toBe("4K");
|
||||
});
|
||||
|
||||
it("formats duration labels", () => {
|
||||
expect(formatDurationLabel(5)).toBe("5s");
|
||||
});
|
||||
|
||||
it("includes all Dreamverse lobby models", () => {
|
||||
expect(CREATION_MODELS.map((model) => model.id)).toEqual(["fast-ltx23", "fast-ltx2", "fast-h3"]);
|
||||
});
|
||||
|
||||
it("builds mention options from presets", () => {
|
||||
expect(
|
||||
buildMentionOptions([
|
||||
{ id: "preset-a", label: "Preset A", description: "A short preset" },
|
||||
{ label: "Missing id" },
|
||||
]),
|
||||
).toEqual([
|
||||
{
|
||||
id: "preset-a",
|
||||
label: "Preset A",
|
||||
kind: "preset",
|
||||
description: "A short preset",
|
||||
},
|
||||
{
|
||||
id: "Missing id",
|
||||
label: "Missing id",
|
||||
kind: "preset",
|
||||
description: undefined,
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
it("derives mode-specific reference requirements", () => {
|
||||
expect(modeRequiresReference("ref2av")).toBe(true);
|
||||
expect(modeRequiresReference("t2v")).toBe(false);
|
||||
expect(modeUsesDualFrames("fl2av")).toBe(true);
|
||||
expect(modeUsesDualFrames("t2v")).toBe(false);
|
||||
});
|
||||
|
||||
it("accepts image reference files only", () => {
|
||||
expect(isReferenceMediaFile(new File(["x"], "a.png", { type: "image/png" }))).toBe(true);
|
||||
expect(isReferenceMediaFile(new File(["x"], "a.mp4", { type: "video/mp4" }))).toBe(false);
|
||||
expect(isReferenceMediaFile(new File(["x"], "a.txt", { type: "text/plain" }))).toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,96 @@
|
||||
export type CreationModeId = "t2v" | "fl2av" | "ref2av";
|
||||
|
||||
export type CreationModelId = "fast-ltx2" | "fast-ltx23" | "fast-h3";
|
||||
|
||||
export type AspectRatioId = "21:9" | "16:9" | "4:3" | "1:1" | "3:4" | "9:16";
|
||||
|
||||
export type ResolutionId = "480p" | "720p" | "1080p" | "4k";
|
||||
|
||||
export interface CreationModeOption {
|
||||
id: CreationModeId;
|
||||
label: string;
|
||||
description: string;
|
||||
}
|
||||
|
||||
export interface CreationModelOption {
|
||||
id: CreationModelId;
|
||||
label: string;
|
||||
description: string;
|
||||
badge?: string;
|
||||
}
|
||||
|
||||
export interface MentionOption {
|
||||
id: string;
|
||||
label: string;
|
||||
kind: "preset" | "asset" | "character";
|
||||
description?: string;
|
||||
}
|
||||
|
||||
export const CREATION_MODES: CreationModeOption[] = [
|
||||
{ id: "t2v", label: "Text to video", description: "Generate from a text prompt" },
|
||||
{ id: "ref2av", label: "Image to video", description: "Guide the first segment with a reference image" },
|
||||
];
|
||||
|
||||
export const UNSUPPORTED_CREATION_MODES: CreationModeOption[] = [
|
||||
{ id: "fl2av", label: "First and last frame", description: "Coming soon on FastLTX models" },
|
||||
];
|
||||
|
||||
export const CREATION_MODELS: CreationModelOption[] = [
|
||||
{
|
||||
id: "fast-ltx23",
|
||||
label: "FastLTX 2.3",
|
||||
description: "LTX 2.3 with OmniNFT LoRA",
|
||||
badge: "New",
|
||||
},
|
||||
{
|
||||
id: "fast-ltx2",
|
||||
label: "FastLTX 2",
|
||||
description: "FastLTX 2 for streaming",
|
||||
},
|
||||
{
|
||||
id: "fast-h3",
|
||||
label: "FastH3",
|
||||
description: "MiniMax H3 with VSA data-free adapter",
|
||||
},
|
||||
];
|
||||
|
||||
export const ASPECT_RATIOS: AspectRatioId[] = ["21:9", "16:9", "4:3", "1:1", "3:4", "9:16"];
|
||||
|
||||
export const RESOLUTIONS: ResolutionId[] = ["480p", "720p", "1080p"];
|
||||
|
||||
export const UNSUPPORTED_RESOLUTIONS: ResolutionId[] = ["4k"];
|
||||
|
||||
export const DURATION_MARKS = [5, 10, 15] as const;
|
||||
|
||||
export const REFERENCE_ACCEPT = "image/png,image/jpeg,image/webp";
|
||||
|
||||
export function formatResolutionLabel(resolution: ResolutionId): string {
|
||||
return resolution === "4k" ? "4K" : resolution.toUpperCase();
|
||||
}
|
||||
|
||||
export function formatDurationLabel(seconds: number): string {
|
||||
return `${seconds}s`;
|
||||
}
|
||||
|
||||
export function modeRequiresReference(modeId: CreationModeId): boolean {
|
||||
return modeId === "ref2av";
|
||||
}
|
||||
|
||||
export function modeUsesDualFrames(modeId: CreationModeId): boolean {
|
||||
return modeId === "fl2av";
|
||||
}
|
||||
|
||||
export function isReferenceMediaFile(file: File): boolean {
|
||||
return file.type === "image/png" || file.type === "image/jpeg" || file.type === "image/webp";
|
||||
}
|
||||
|
||||
export function buildMentionOptions(storyPresets: Array<{ id?: string; label?: string; description?: string }>): MentionOption[] {
|
||||
return storyPresets
|
||||
.filter((preset) => typeof preset.label === "string" && preset.label.trim())
|
||||
.map((preset) => ({
|
||||
id: String(preset.id || preset.label),
|
||||
label: String(preset.label),
|
||||
kind: "preset" as const,
|
||||
description: typeof preset.description === "string" ? preset.description : undefined,
|
||||
}));
|
||||
}
|
||||
@@ -0,0 +1,66 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
import { parseEchoedCreationConfig, validateCreationInputs } from "@/lib/creationPayload";
|
||||
|
||||
describe("creationPayload", () => {
|
||||
it("requires a reference asset for omni reference mode", () => {
|
||||
expect(
|
||||
validateCreationInputs({
|
||||
modeId: "ref2av",
|
||||
referenceFile: null,
|
||||
}),
|
||||
).toMatch(/reference asset/i);
|
||||
});
|
||||
|
||||
it("requires both frames for first and last frame mode", () => {
|
||||
expect(
|
||||
validateCreationInputs({
|
||||
modeId: "fl2av",
|
||||
firstFrameFile: new File(["a"], "first.png", { type: "image/png" }),
|
||||
lastFrameFile: null,
|
||||
}),
|
||||
).toMatch(/both first and last/i);
|
||||
});
|
||||
|
||||
it("accepts text to video without references", () => {
|
||||
expect(
|
||||
validateCreationInputs({
|
||||
modeId: "t2v",
|
||||
}),
|
||||
).toBeNull();
|
||||
});
|
||||
|
||||
it("parses echoed creation config from server payloads", () => {
|
||||
expect(
|
||||
parseEchoedCreationConfig({
|
||||
type: "gpu_assigned",
|
||||
creation_config: {
|
||||
model_id: "fast-ltx2",
|
||||
generation_mode: "ref2va",
|
||||
aspect_ratio: "9:16",
|
||||
resolution: "480p",
|
||||
duration_sec: 10,
|
||||
},
|
||||
}),
|
||||
).toEqual({
|
||||
modelId: "fast-ltx2",
|
||||
modeId: "ref2av",
|
||||
aspectRatio: "9:16",
|
||||
resolution: "480p",
|
||||
durationSec: 10,
|
||||
});
|
||||
});
|
||||
|
||||
it("ignores invalid echoed creation config", () => {
|
||||
expect(parseEchoedCreationConfig({ creation_config: { model_id: "unknown" } })).toBeNull();
|
||||
});
|
||||
|
||||
it("rejects unsupported reference mime types", () => {
|
||||
expect(
|
||||
validateCreationInputs({
|
||||
modeId: "t2v",
|
||||
referenceFile: new File(["a"], "clip.mp4", { type: "video/mp4" }),
|
||||
}),
|
||||
).toMatch(/PNG, JPEG, or WebP/i);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,172 @@
|
||||
import type {
|
||||
AspectRatioId,
|
||||
CreationModeId,
|
||||
CreationModelId,
|
||||
ResolutionId,
|
||||
} from "@/lib/creationConfig";
|
||||
import { fromGenerationMode, type GenerationMode } from "@/lib/generationMode";
|
||||
|
||||
const LOBBY_MODEL_IDS = new Set<CreationModelId>(["fast-ltx2", "fast-ltx23", "fast-h3"]);
|
||||
const ASPECT_RATIO_IDS = new Set<AspectRatioId>(["21:9", "16:9", "4:3", "1:1", "3:4", "9:16"]);
|
||||
const RESOLUTION_IDS = new Set<ResolutionId>(["480p", "720p", "1080p", "4k"]);
|
||||
const DURATION_SEC_VALUES = new Set([5, 10, 15]);
|
||||
|
||||
export interface EchoedSessionCreationConfig {
|
||||
modelId: CreationModelId;
|
||||
modeId: CreationModeId;
|
||||
aspectRatio: AspectRatioId;
|
||||
resolution: ResolutionId;
|
||||
durationSec: number;
|
||||
}
|
||||
|
||||
const MAX_IMAGE_BYTES = 15 * 1024 * 1024;
|
||||
const SUPPORTED_IMAGE_TYPES = new Set(["image/png", "image/jpeg", "image/webp"]);
|
||||
|
||||
export interface InitialImagePayload {
|
||||
name: string;
|
||||
mime_type: string;
|
||||
data_url: string;
|
||||
}
|
||||
|
||||
export interface CreationInitPayload {
|
||||
model_id: string;
|
||||
aspect_ratio: string;
|
||||
resolution: string;
|
||||
duration_sec: number;
|
||||
initial_image: InitialImagePayload | null;
|
||||
last_frame_image: InitialImagePayload | null;
|
||||
}
|
||||
|
||||
function readFileAsDataUrl(file: File): Promise<string> {
|
||||
return new Promise((resolve, reject) => {
|
||||
const reader = new FileReader();
|
||||
reader.onload = () => {
|
||||
if (typeof reader.result === "string") {
|
||||
resolve(reader.result);
|
||||
return;
|
||||
}
|
||||
reject(new Error("Failed to read reference image."));
|
||||
};
|
||||
reader.onerror = () => reject(new Error("Failed to read reference image."));
|
||||
reader.readAsDataURL(file);
|
||||
});
|
||||
}
|
||||
|
||||
export async function fileToInitialImagePayload(file: File): Promise<InitialImagePayload> {
|
||||
if (!SUPPORTED_IMAGE_TYPES.has(file.type)) {
|
||||
throw new Error("Reference assets must be PNG, JPEG, or WebP images.");
|
||||
}
|
||||
if (file.size > MAX_IMAGE_BYTES) {
|
||||
throw new Error("Reference image must be 15 MB or smaller.");
|
||||
}
|
||||
return {
|
||||
name: file.name,
|
||||
mime_type: file.type,
|
||||
data_url: await readFileAsDataUrl(file),
|
||||
};
|
||||
}
|
||||
|
||||
export async function resolveCreationImages(input: {
|
||||
modeId: CreationModeId;
|
||||
referenceFile?: File | null;
|
||||
firstFrameFile?: File | null;
|
||||
lastFrameFile?: File | null;
|
||||
}): Promise<Pick<CreationInitPayload, "initial_image" | "last_frame_image">> {
|
||||
if (input.modeId === "fl2av") {
|
||||
const firstFrame = input.firstFrameFile ? await fileToInitialImagePayload(input.firstFrameFile) : null;
|
||||
const lastFrame = input.lastFrameFile ? await fileToInitialImagePayload(input.lastFrameFile) : null;
|
||||
return {
|
||||
initial_image: firstFrame,
|
||||
last_frame_image: lastFrame,
|
||||
};
|
||||
}
|
||||
|
||||
const reference = input.referenceFile ? await fileToInitialImagePayload(input.referenceFile) : null;
|
||||
return {
|
||||
initial_image: reference,
|
||||
last_frame_image: null,
|
||||
};
|
||||
}
|
||||
|
||||
export function validateCreationInputs(input: {
|
||||
modeId: CreationModeId;
|
||||
referenceFile?: File | null;
|
||||
firstFrameFile?: File | null;
|
||||
lastFrameFile?: File | null;
|
||||
}): string | null {
|
||||
if (input.modeId === "ref2av" && !input.referenceFile) {
|
||||
return "Upload a reference asset to use Omni reference mode.";
|
||||
}
|
||||
if (input.modeId === "fl2av") {
|
||||
if (!input.firstFrameFile || !input.lastFrameFile) {
|
||||
return "Upload both first and last frame assets.";
|
||||
}
|
||||
}
|
||||
if (input.referenceFile && !SUPPORTED_IMAGE_TYPES.has(input.referenceFile.type)) {
|
||||
return "Reference assets must be PNG, JPEG, or WebP images.";
|
||||
}
|
||||
if (input.firstFrameFile && !SUPPORTED_IMAGE_TYPES.has(input.firstFrameFile.type)) {
|
||||
return "First frame must be a PNG, JPEG, or WebP image.";
|
||||
}
|
||||
if (input.lastFrameFile && !SUPPORTED_IMAGE_TYPES.has(input.lastFrameFile.type)) {
|
||||
return "Last frame must be a PNG, JPEG, or WebP image.";
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
export function parseEchoedCreationConfig(data: unknown): EchoedSessionCreationConfig | null {
|
||||
if (!data || typeof data !== "object") {
|
||||
return null;
|
||||
}
|
||||
const creationConfig = (data as Record<string, unknown>).creation_config;
|
||||
if (!creationConfig || typeof creationConfig !== "object") {
|
||||
return null;
|
||||
}
|
||||
const config = creationConfig as Record<string, unknown>;
|
||||
const modelId = typeof config.model_id === "string" && LOBBY_MODEL_IDS.has(config.model_id as CreationModelId)
|
||||
? (config.model_id as CreationModelId)
|
||||
: null;
|
||||
const generationMode = typeof config.generation_mode === "string" ? config.generation_mode as GenerationMode : null;
|
||||
const modeId = generationMode === "t2va" || generationMode === "fl2va" || generationMode === "ref2va"
|
||||
? fromGenerationMode(generationMode)
|
||||
: null;
|
||||
const aspectRatio = typeof config.aspect_ratio === "string" && ASPECT_RATIO_IDS.has(config.aspect_ratio as AspectRatioId)
|
||||
? (config.aspect_ratio as AspectRatioId)
|
||||
: null;
|
||||
const resolution = typeof config.resolution === "string" && RESOLUTION_IDS.has(config.resolution as ResolutionId)
|
||||
? (config.resolution as ResolutionId)
|
||||
: null;
|
||||
const durationSec = typeof config.duration_sec === "number" && DURATION_SEC_VALUES.has(config.duration_sec)
|
||||
? config.duration_sec
|
||||
: null;
|
||||
if (modelId === null || modeId === null || aspectRatio === null || resolution === null || durationSec === null) {
|
||||
return null;
|
||||
}
|
||||
return {
|
||||
modelId,
|
||||
modeId,
|
||||
aspectRatio,
|
||||
resolution,
|
||||
durationSec,
|
||||
};
|
||||
}
|
||||
|
||||
export async function buildCreationInitPayload(input: {
|
||||
modelId: string;
|
||||
modeId: CreationModeId;
|
||||
aspectRatio: string;
|
||||
resolution: string;
|
||||
durationSec: number;
|
||||
referenceFile?: File | null;
|
||||
firstFrameFile?: File | null;
|
||||
lastFrameFile?: File | null;
|
||||
}): Promise<CreationInitPayload> {
|
||||
const images = await resolveCreationImages(input);
|
||||
return {
|
||||
model_id: input.modelId,
|
||||
aspect_ratio: input.aspectRatio,
|
||||
resolution: input.resolution,
|
||||
duration_sec: input.durationSec,
|
||||
...images,
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,39 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
import {
|
||||
DEFAULT_GENERATION_MODE,
|
||||
GENERATION_MODES,
|
||||
fromGenerationMode,
|
||||
getGenerationMode,
|
||||
isGenerationMode,
|
||||
toGenerationMode,
|
||||
} from "./generationMode";
|
||||
|
||||
describe("generation modes", () => {
|
||||
it("exposes stable wire IDs in the expected product order", () => {
|
||||
expect(GENERATION_MODES.map((mode) => mode.id)).toEqual([
|
||||
"t2va",
|
||||
"fl2va",
|
||||
"ref2va",
|
||||
]);
|
||||
expect(DEFAULT_GENERATION_MODE).toBe("t2va");
|
||||
});
|
||||
|
||||
it("validates and resolves generation mode values", () => {
|
||||
expect(isGenerationMode("ref2va")).toBe(true);
|
||||
expect(isGenerationMode("unknown")).toBe(false);
|
||||
expect(getGenerationMode("fl2va").label).toBe("FL2VA");
|
||||
});
|
||||
|
||||
it("maps creation studio mode IDs to upstream wire values", () => {
|
||||
expect(toGenerationMode("t2v")).toBe("t2va");
|
||||
expect(toGenerationMode("fl2av")).toBe("fl2va");
|
||||
expect(toGenerationMode("ref2av")).toBe("ref2va");
|
||||
});
|
||||
|
||||
it("maps upstream wire values back to creation studio mode IDs", () => {
|
||||
expect(fromGenerationMode("t2va")).toBe("t2v");
|
||||
expect(fromGenerationMode("fl2va")).toBe("fl2av");
|
||||
expect(fromGenerationMode("ref2va")).toBe("ref2av");
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,54 @@
|
||||
import type { CreationModeId } from "@/lib/creationConfig";
|
||||
|
||||
export const GENERATION_MODES = [
|
||||
{
|
||||
id: "t2va",
|
||||
label: "T2VA",
|
||||
name: "Text to video + audio",
|
||||
description: "Start with a text prompt; no reference asset is required.",
|
||||
},
|
||||
{
|
||||
id: "fl2va",
|
||||
label: "FL2VA",
|
||||
name: "First/last frames to video + audio",
|
||||
description: "Provide first and last frame images to control the transition.",
|
||||
},
|
||||
{
|
||||
id: "ref2va",
|
||||
label: "Ref2VA",
|
||||
name: "References to video + audio",
|
||||
description: "Guide the result with ordered image, video, or audio references.",
|
||||
},
|
||||
] as const;
|
||||
|
||||
export type GenerationMode = (typeof GENERATION_MODES)[number]["id"];
|
||||
|
||||
export const DEFAULT_GENERATION_MODE: GenerationMode = "t2va";
|
||||
|
||||
const CREATION_MODE_TO_GENERATION_MODE: Record<CreationModeId, GenerationMode> = {
|
||||
t2v: "t2va",
|
||||
fl2av: "fl2va",
|
||||
ref2av: "ref2va",
|
||||
};
|
||||
|
||||
export function isGenerationMode(value: unknown): value is GenerationMode {
|
||||
return GENERATION_MODES.some((mode) => mode.id === value);
|
||||
}
|
||||
|
||||
export function getGenerationMode(value: GenerationMode) {
|
||||
return GENERATION_MODES.find((mode) => mode.id === value) ?? GENERATION_MODES[0];
|
||||
}
|
||||
|
||||
const GENERATION_MODE_TO_CREATION_MODE: Record<GenerationMode, CreationModeId> = {
|
||||
t2va: "t2v",
|
||||
fl2va: "fl2av",
|
||||
ref2va: "ref2av",
|
||||
};
|
||||
|
||||
export function fromGenerationMode(mode: GenerationMode): CreationModeId {
|
||||
return GENERATION_MODE_TO_CREATION_MODE[mode];
|
||||
}
|
||||
|
||||
export function toGenerationMode(modeId: CreationModeId): GenerationMode {
|
||||
return CREATION_MODE_TO_GENERATION_MODE[modeId];
|
||||
}
|
||||
@@ -1,63 +0,0 @@
|
||||
import { expect, test } from '@playwright/test';
|
||||
|
||||
import { skipWithoutMock } from './helpers';
|
||||
|
||||
test.describe('create job interactions', () => {
|
||||
skipWithoutMock();
|
||||
|
||||
for (const jobType of ['inference', 'finetuning', 'distillation']) {
|
||||
test(`${jobType} remains interactive after repeated dialog dismissals`, async ({ page }) => {
|
||||
await page.goto(`/${jobType}`);
|
||||
const trigger = page.getByRole('button', { name: 'Create Job', exact: true });
|
||||
const dialog = page.getByRole('dialog');
|
||||
|
||||
// Exercise both dismissal paths and reopen without reloading the page.
|
||||
for (const closeWithEscape of [false, true]) {
|
||||
await trigger.click();
|
||||
await page.getByRole('menuitem').first().click();
|
||||
await expect(dialog).toBeVisible();
|
||||
if (closeWithEscape) {
|
||||
await page.keyboard.press('Escape');
|
||||
} else {
|
||||
await dialog.getByRole('button', { name: 'Close', exact: true }).click();
|
||||
}
|
||||
await expect(dialog).toBeHidden();
|
||||
await expect(page.locator('body')).toHaveCSS('pointer-events', 'auto');
|
||||
await expect(trigger).toBeFocused();
|
||||
}
|
||||
|
||||
await page.getByRole('link', { name: 'Datasets', exact: true }).click();
|
||||
await expect(page).toHaveURL(/\/datasets$/);
|
||||
});
|
||||
}
|
||||
|
||||
test('preserves keyboard menu dismissal and dialog focus trapping', async ({ page }) => {
|
||||
await page.goto('/inference');
|
||||
const trigger = page.getByRole('button', { name: 'Create Job', exact: true });
|
||||
await trigger.focus();
|
||||
await page.keyboard.press('Enter');
|
||||
const firstItem = page.getByRole('menuitem').first();
|
||||
await expect(firstItem).toBeFocused();
|
||||
await page.keyboard.press('Escape');
|
||||
await expect(page.getByRole('menu')).toBeHidden();
|
||||
await expect(trigger).toBeFocused();
|
||||
await expect(page.locator('body')).toHaveCSS('pointer-events', 'auto');
|
||||
|
||||
await page.keyboard.press('Enter');
|
||||
await expect(firstItem).toBeFocused();
|
||||
await page.keyboard.press('Enter');
|
||||
const dialog = page.getByRole('dialog');
|
||||
await expect(dialog).toBeVisible();
|
||||
await expect(dialog.getByLabel('Name (optional)')).toBeFocused();
|
||||
|
||||
// Shift+Tab from the first field wraps to Close, then Tab wraps back.
|
||||
await page.keyboard.press('Shift+Tab');
|
||||
await expect(dialog.getByRole('button', { name: 'Close', exact: true })).toBeFocused();
|
||||
await page.keyboard.press('Tab');
|
||||
await expect(dialog.getByLabel('Name (optional)')).toBeFocused();
|
||||
await page.keyboard.press('Escape');
|
||||
await expect(dialog).toBeHidden();
|
||||
await expect(trigger).toBeFocused();
|
||||
await expect(page.locator('body')).toHaveCSS('pointer-events', 'auto');
|
||||
});
|
||||
});
|
||||
@@ -1,6 +1,6 @@
|
||||
import { expect, test } from '@playwright/test';
|
||||
|
||||
import { API_BASE, skipWithoutMock } from './helpers';
|
||||
import { skipWithoutMock } from './helpers';
|
||||
|
||||
/**
|
||||
* Create-job flow: open the Create Job modal on /inference, fill the prompt
|
||||
@@ -10,8 +10,7 @@ import { API_BASE, skipWithoutMock } from './helpers';
|
||||
test.describe('create inference job', () => {
|
||||
skipWithoutMock();
|
||||
|
||||
test('creates a T2V job and starts it without refreshing', async ({ page, request }) => {
|
||||
await request.put(`${API_BASE}/settings`, { data: { autoStartJob: false } });
|
||||
test('creates a T2V job and shows it in the queue', async ({ page }) => {
|
||||
await page.goto('/inference');
|
||||
|
||||
// The trigger opens a real menu on click, so this path works for touch,
|
||||
@@ -39,20 +38,5 @@ test.describe('create inference job', () => {
|
||||
// Modal closes and the queue refreshes with the newly created job.
|
||||
await expect(dialog).toBeHidden();
|
||||
await expect(page.getByText(prompt)).toBeVisible();
|
||||
await expect(page.locator('body')).toHaveCSS('pointer-events', 'auto');
|
||||
|
||||
const card = page.getByRole('article').filter({ hasText: prompt });
|
||||
await expect(card.getByText('pending', { exact: true })).toBeVisible();
|
||||
const started = page.waitForResponse((response) =>
|
||||
response.url().startsWith(`${API_BASE}/jobs/`) &&
|
||||
response.url().endsWith('/start') &&
|
||||
response.request().method() === 'POST',
|
||||
);
|
||||
await card.getByRole('button', { name: 'Start', exact: true }).click();
|
||||
expect((await started).ok()).toBe(true);
|
||||
await expect(card.getByText('running', { exact: true })).toBeVisible();
|
||||
|
||||
await page.getByRole('link', { name: 'Datasets', exact: true }).click();
|
||||
await expect(page).toHaveURL(/\/datasets$/);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -862,40 +862,23 @@ class JobRunner:
|
||||
sp_size,
|
||||
)
|
||||
|
||||
gen = VideoGenerator.from_config(
|
||||
{
|
||||
"model_path": model_id,
|
||||
"engine": {
|
||||
"num_gpus": num_gpus,
|
||||
"parallelism": {
|
||||
"tp_size": tp_size,
|
||||
"sp_size": sp_size,
|
||||
},
|
||||
"offload": {
|
||||
"dit": dit_cpu_offload,
|
||||
"dit_layerwise": dit_layerwise_offload,
|
||||
"text_encoder": text_encoder_cpu_offload,
|
||||
"image_encoder": image_encoder_cpu_offload,
|
||||
"vae": vae_cpu_offload,
|
||||
},
|
||||
"compile": {
|
||||
"enabled": enable_torch_compile
|
||||
},
|
||||
"attention": {
|
||||
"vsa_sparsity": vsa_sparsity
|
||||
},
|
||||
"use_fsdp_inference": use_fsdp_inference,
|
||||
},
|
||||
"pipeline": {
|
||||
"workload_type":
|
||||
workload_type,
|
||||
**({
|
||||
"components": {
|
||||
"override_pipeline_cls_name": override_pipeline_cls_name
|
||||
}
|
||||
} if override_pipeline_cls_name else {}),
|
||||
},
|
||||
},
|
||||
gen = VideoGenerator.from_pretrained(
|
||||
model_id,
|
||||
workload_type=workload_type,
|
||||
num_gpus=num_gpus,
|
||||
dit_layerwise_offload=dit_layerwise_offload,
|
||||
**({
|
||||
"override_pipeline_cls_name": override_pipeline_cls_name
|
||||
} if override_pipeline_cls_name else {}),
|
||||
dit_cpu_offload=dit_cpu_offload,
|
||||
text_encoder_cpu_offload=text_encoder_cpu_offload,
|
||||
vae_cpu_offload=vae_cpu_offload,
|
||||
image_encoder_cpu_offload=image_encoder_cpu_offload,
|
||||
use_fsdp_inference=use_fsdp_inference,
|
||||
enable_torch_compile=enable_torch_compile,
|
||||
VSA_sparsity=vsa_sparsity,
|
||||
tp_size=tp_size,
|
||||
sp_size=sp_size,
|
||||
log_queue=log_queue,
|
||||
)
|
||||
|
||||
@@ -1120,33 +1103,30 @@ class JobRunner:
|
||||
# Without a name FastVideo derives the filename from the prompt.
|
||||
safe_name = re.sub(r'[\\/:*?"<>|]+', "", job.name).strip().strip(".")
|
||||
output_target = (os.path.join(job_output_dir, f"{safe_name[:80]}.mp4") if safe_name else job_output_dir)
|
||||
request: dict[str, Any] = {
|
||||
gen_kwargs: dict[str, Any] = {
|
||||
"prompt": job.prompt,
|
||||
"output_path": output_target,
|
||||
"save_video": True,
|
||||
"num_inference_steps": job.num_inference_steps,
|
||||
"num_frames": job.num_frames,
|
||||
"height": job.height,
|
||||
"width": job.width,
|
||||
"guidance_scale": job.guidance_scale,
|
||||
"guidance_rescale": job.guidance_rescale,
|
||||
"fps": job.fps,
|
||||
"seed": job.seed,
|
||||
"negative_prompt": job.negative_prompt or "",
|
||||
"sampling": {
|
||||
"num_inference_steps": job.num_inference_steps,
|
||||
"num_frames": job.num_frames,
|
||||
"height": job.height,
|
||||
"width": job.width,
|
||||
"guidance_scale": job.guidance_scale,
|
||||
"guidance_rescale": job.guidance_rescale,
|
||||
"fps": job.fps,
|
||||
"seed": job.seed,
|
||||
},
|
||||
"output": {
|
||||
"output_path": output_target,
|
||||
"save_video": True,
|
||||
},
|
||||
"log_queue": log_queue,
|
||||
}
|
||||
if job.image_path:
|
||||
request.setdefault("inputs", {})["image_path"] = job.image_path
|
||||
gen_kwargs["image_path"] = job.image_path
|
||||
if job.references:
|
||||
request.setdefault("inputs", {})["references"] = _build_h3_references(job.references)
|
||||
gen_kwargs["references"] = _build_h3_references(job.references)
|
||||
if job.last_image_path:
|
||||
# _prepare_fl2va requires a PIL image, not a path.
|
||||
from PIL import Image as _PILImage
|
||||
request.setdefault("inputs", {})["last_image"] = _PILImage.open(job.last_image_path)
|
||||
generator.generate(request, log_queue=log_queue)
|
||||
gen_kwargs["last_image"] = _PILImage.open(job.last_image_path)
|
||||
generator.generate_video(**gen_kwargs)
|
||||
|
||||
buf.phase = "saving"
|
||||
logger.info("Generation completed, searching for output file...")
|
||||
|
||||
Generated
+829
-975
File diff suppressed because it is too large
Load Diff
@@ -17,11 +17,20 @@
|
||||
"start:all": "concurrently --kill-others-on-fail \"npm:start:api\" \"npm:start:web\""
|
||||
},
|
||||
"dependencies": {
|
||||
"@radix-ui/react-dialog": "^1.1.0",
|
||||
"@radix-ui/react-dropdown-menu": "^2.1.24",
|
||||
"@radix-ui/react-label": "^2.1.8",
|
||||
"@radix-ui/react-scroll-area": "^1.2.10",
|
||||
"@radix-ui/react-select": "^2.2.6",
|
||||
"@radix-ui/react-separator": "^1.1.8",
|
||||
"@radix-ui/react-slider": "^1.2.0",
|
||||
"@radix-ui/react-slot": "^1.2.4",
|
||||
"@radix-ui/react-switch": "^1.1.0",
|
||||
"@radix-ui/react-tabs": "^1.1.0",
|
||||
"class-variance-authority": "^0.7.1",
|
||||
"clsx": "^2.1.1",
|
||||
"lucide-react": "^0.577.0",
|
||||
"next": "15.5.18",
|
||||
"radix-ui": "^1.6.7",
|
||||
"react": "^19.1.0",
|
||||
"react-dom": "^19.1.0",
|
||||
"sonner": "^2.0.7",
|
||||
|
||||
@@ -1,26 +1,24 @@
|
||||
import { render, screen, waitFor, within } from '@testing-library/react';
|
||||
import { render, screen } from '@testing-library/react';
|
||||
import userEvent from '@testing-library/user-event';
|
||||
import { beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
import { describe, expect, it, vi } from 'vitest';
|
||||
|
||||
import CreateJobButton from './CreateJobButton';
|
||||
import { getDatasets, getModels } from '@/lib/api';
|
||||
|
||||
vi.mock('@/lib/api', () => ({
|
||||
createJob: vi.fn(),
|
||||
getModels: vi.fn(),
|
||||
getDatasets: vi.fn(),
|
||||
uploadImage: vi.fn(),
|
||||
getSettings: vi.fn(),
|
||||
updateSettings: vi.fn(),
|
||||
vi.mock('./CreateJobModal', () => ({
|
||||
default: ({
|
||||
isOpen,
|
||||
workloadType,
|
||||
}: {
|
||||
isOpen: boolean;
|
||||
workloadType: string;
|
||||
}) =>
|
||||
isOpen ? (
|
||||
<div role="dialog" data-workload-type={workloadType}>
|
||||
Create job form
|
||||
</div>
|
||||
) : null,
|
||||
}));
|
||||
|
||||
beforeEach(() => {
|
||||
vi.mocked(getModels).mockResolvedValue([
|
||||
{ id: 'wan/t2v-1.3b', label: 'Wan T2V' },
|
||||
]);
|
||||
vi.mocked(getDatasets).mockResolvedValue([]);
|
||||
});
|
||||
|
||||
describe('CreateJobButton', () => {
|
||||
it('opens the workload menu on click and selects an item', async () => {
|
||||
const user = userEvent.setup();
|
||||
@@ -29,9 +27,10 @@ describe('CreateJobButton', () => {
|
||||
await user.click(screen.getByRole('button', { name: 'Create Job' }));
|
||||
await user.click(screen.getByRole('menuitem', { name: /I2V/i }));
|
||||
|
||||
expect(
|
||||
screen.getByRole('dialog', { name: 'New Inference Job (I2V)' }),
|
||||
).toBeInTheDocument();
|
||||
expect(screen.getByRole('dialog')).toHaveAttribute(
|
||||
'data-workload-type',
|
||||
'i2v',
|
||||
);
|
||||
});
|
||||
|
||||
it('opens and operates the workload menu from the keyboard', async () => {
|
||||
@@ -46,42 +45,9 @@ describe('CreateJobButton', () => {
|
||||
expect(firstItem).toHaveFocus();
|
||||
await user.keyboard('{Enter}');
|
||||
|
||||
expect(
|
||||
screen.getByRole('dialog', { name: 'New Inference Job (T2V)' }),
|
||||
).toBeInTheDocument();
|
||||
await user.keyboard('{Escape}');
|
||||
await waitFor(() =>
|
||||
expect(screen.queryByRole('dialog')).not.toBeInTheDocument(),
|
||||
expect(screen.getByRole('dialog')).toHaveAttribute(
|
||||
'data-workload-type',
|
||||
't2v',
|
||||
);
|
||||
await waitFor(() =>
|
||||
expect(document.body.style.pointerEvents).not.toBe('none'),
|
||||
);
|
||||
expect(trigger).toHaveFocus();
|
||||
});
|
||||
|
||||
it.each(['inference', 'finetuning', 'distillation'] as const)(
|
||||
'restores page interaction after closing the real %s dialog',
|
||||
async (jobType) => {
|
||||
const user = userEvent.setup();
|
||||
render(<CreateJobButton jobType={jobType} />);
|
||||
const trigger = screen.getByRole('button', { name: 'Create Job' });
|
||||
|
||||
// Keep the real Dialog mounted: mocking it hides conflicting Radix layers.
|
||||
for (let attempt = 0; attempt < 2; attempt++) {
|
||||
await user.click(trigger);
|
||||
await user.click(screen.getAllByRole('menuitem')[0]);
|
||||
const dialog = screen.getByRole('dialog');
|
||||
await user.click(
|
||||
within(dialog).getByRole('button', { name: 'Close' }),
|
||||
);
|
||||
await waitFor(() =>
|
||||
expect(screen.queryByRole('dialog')).not.toBeInTheDocument(),
|
||||
);
|
||||
await waitFor(() =>
|
||||
expect(document.body.style.pointerEvents).not.toBe('none'),
|
||||
);
|
||||
expect(trigger).toHaveFocus();
|
||||
}
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
import * as React from 'react';
|
||||
import { ChevronDown } from 'lucide-react';
|
||||
import { DropdownMenu } from 'radix-ui';
|
||||
import * as DropdownMenu from '@radix-ui/react-dropdown-menu';
|
||||
|
||||
import CreateJobModal from '@/components/jobs/CreateJobModal';
|
||||
import { Button } from '@/components/ui/button';
|
||||
@@ -16,7 +16,6 @@ interface CreateJobButtonProps {
|
||||
|
||||
export default function CreateJobButton({ jobType }: CreateJobButtonProps) {
|
||||
const options = WORKLOAD_OPTIONS[jobType] ?? [];
|
||||
const triggerRef = React.useRef<HTMLButtonElement>(null);
|
||||
|
||||
const [modalOpen, setModalOpen] = React.useState(false);
|
||||
const [workloadType, setWorkloadType] = React.useState(
|
||||
@@ -37,7 +36,7 @@ export default function CreateJobButton({ jobType }: CreateJobButtonProps) {
|
||||
<>
|
||||
<DropdownMenu.Root>
|
||||
<DropdownMenu.Trigger asChild>
|
||||
<Button ref={triggerRef} type="button" className="gap-1.5">
|
||||
<Button type="button" className="gap-1.5">
|
||||
Create Job
|
||||
<ChevronDown className="size-3.5 opacity-85" aria-hidden />
|
||||
</Button>
|
||||
@@ -67,11 +66,6 @@ export default function CreateJobButton({ jobType }: CreateJobButtonProps) {
|
||||
<CreateJobModal
|
||||
isOpen={modalOpen}
|
||||
onClose={() => setModalOpen(false)}
|
||||
onCloseAutoFocus={(event) => {
|
||||
// This dialog opens from a menu item, so it has no DialogTrigger.
|
||||
event.preventDefault();
|
||||
triggerRef.current?.focus();
|
||||
}}
|
||||
onSuccess={handleSuccess}
|
||||
jobType={jobType}
|
||||
workloadType={workloadType}
|
||||
|
||||
@@ -55,9 +55,6 @@ import { jobToFormFields, type JobLike } from '@/lib/jobToFields';
|
||||
export interface CreateJobModalProps {
|
||||
isOpen: boolean;
|
||||
onClose: () => void;
|
||||
onCloseAutoFocus?: React.ComponentProps<
|
||||
typeof DialogContent
|
||||
>['onCloseAutoFocus'];
|
||||
onSuccess: () => void;
|
||||
jobType: JobType;
|
||||
workloadType: string;
|
||||
@@ -70,7 +67,6 @@ export interface CreateJobModalProps {
|
||||
export default function CreateJobModal({
|
||||
isOpen,
|
||||
onClose,
|
||||
onCloseAutoFocus,
|
||||
onSuccess,
|
||||
jobType,
|
||||
workloadType,
|
||||
@@ -131,9 +127,8 @@ export default function CreateJobModal({
|
||||
const editingJobId = editingJob?.id ?? null;
|
||||
const editingJobModelId = editingJob?.model_id ?? null;
|
||||
|
||||
// Layerwise offload and FSDP compete for the DiT weights and the device offload
|
||||
// policy (resolve_device_offload_conflicts in fastvideo/api/device_policy.py)
|
||||
// silently picks a winner; resolve it visibly here.
|
||||
// Layerwise offload and FSDP compete for the DiT weights and FastVideoArgs
|
||||
// silently picks a winner (fastvideo_args.py:859); resolve it visibly here.
|
||||
// dit_cpu_offload is deliberately not interlocked -- it is a modifier, not a
|
||||
// competing strategy.
|
||||
const handleDitLayerwiseOffloadChange = React.useCallback((next: boolean) => {
|
||||
@@ -649,7 +644,6 @@ export default function CreateJobModal({
|
||||
>
|
||||
<DialogContent
|
||||
className="max-h-[90vh] w-[90vw] max-w-[850px] overflow-y-auto"
|
||||
onCloseAutoFocus={onCloseAutoFocus}
|
||||
onEscapeKeyDown={(e) => {
|
||||
if (isSubmitting) e.preventDefault();
|
||||
}}
|
||||
|
||||
@@ -1,7 +1,5 @@
|
||||
import * as React from 'react';
|
||||
import { render, screen } from '@testing-library/react';
|
||||
import userEvent from '@testing-library/user-event';
|
||||
import { describe, expect, it, vi } from 'vitest';
|
||||
import { describe, expect, it } from 'vitest';
|
||||
|
||||
import { Button } from './button';
|
||||
import { Input } from './input';
|
||||
@@ -10,24 +8,6 @@ import { Slider } from './slider';
|
||||
import { Switch } from './switch';
|
||||
|
||||
describe('shared control accessibility', () => {
|
||||
it('forwards refs and click handlers to the asChild button', async () => {
|
||||
const user = userEvent.setup();
|
||||
const ref = React.createRef<HTMLButtonElement>();
|
||||
const onClick = vi.fn();
|
||||
render(
|
||||
<Button asChild ref={ref} onClick={onClick}>
|
||||
<button type="button">Slotted action</button>
|
||||
</Button>,
|
||||
);
|
||||
|
||||
const button = screen.getByRole('button', { name: 'Slotted action' });
|
||||
expect(screen.getAllByRole('button')).toHaveLength(1);
|
||||
expect(ref.current).toBe(button);
|
||||
await user.click(button);
|
||||
expect(onClick).toHaveBeenCalledTimes(1);
|
||||
expect(button).toHaveFocus();
|
||||
});
|
||||
|
||||
it('keeps button, input, and select targets at least 44px tall', () => {
|
||||
render(
|
||||
<>
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
"use client";
|
||||
|
||||
import * as React from "react";
|
||||
import { Slot } from "radix-ui";
|
||||
import { Slot } from "@radix-ui/react-slot";
|
||||
import { cva, type VariantProps } from "class-variance-authority";
|
||||
|
||||
import { cn } from "@/lib/utils";
|
||||
@@ -37,7 +37,7 @@ export interface ButtonProps extends React.ButtonHTMLAttributes<HTMLButtonElemen
|
||||
}
|
||||
|
||||
const Button = React.forwardRef<HTMLButtonElement, ButtonProps>(({ className, variant, size, asChild = false, ...props }, ref) => {
|
||||
const Comp = asChild ? Slot.Root : "button";
|
||||
const Comp = asChild ? Slot : "button";
|
||||
return <Comp className={cn(buttonVariants({ variant, size, className }))} ref={ref} {...props} />;
|
||||
});
|
||||
Button.displayName = "Button";
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
'use client';
|
||||
|
||||
import * as React from 'react';
|
||||
import { Dialog as DialogPrimitive } from 'radix-ui';
|
||||
import * as DialogPrimitive from '@radix-ui/react-dialog';
|
||||
import { X } from 'lucide-react';
|
||||
|
||||
import { cn } from '@/lib/utils';
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
'use client';
|
||||
|
||||
import * as React from 'react';
|
||||
import { Label as LabelPrimitive } from 'radix-ui';
|
||||
import * as LabelPrimitive from '@radix-ui/react-label';
|
||||
import { cva, type VariantProps } from 'class-variance-authority';
|
||||
|
||||
import { cn } from '@/lib/utils';
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
'use client';
|
||||
|
||||
import * as React from 'react';
|
||||
import { ScrollArea as ScrollAreaPrimitive } from 'radix-ui';
|
||||
import * as ScrollAreaPrimitive from '@radix-ui/react-scroll-area';
|
||||
|
||||
import { cn } from '@/lib/utils';
|
||||
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
'use client';
|
||||
|
||||
import * as React from 'react';
|
||||
import { Select as SelectPrimitive } from 'radix-ui';
|
||||
import * as SelectPrimitive from '@radix-ui/react-select';
|
||||
import { Check, ChevronDown, ChevronUp } from 'lucide-react';
|
||||
|
||||
import { cn } from '@/lib/utils';
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
'use client';
|
||||
|
||||
import * as React from 'react';
|
||||
import { Separator as SeparatorPrimitive } from 'radix-ui';
|
||||
import * as SeparatorPrimitive from '@radix-ui/react-separator';
|
||||
|
||||
import { cn } from '@/lib/utils';
|
||||
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
'use client';
|
||||
|
||||
import * as React from 'react';
|
||||
import { Slider as SliderPrimitive } from 'radix-ui';
|
||||
import * as SliderPrimitive from '@radix-ui/react-slider';
|
||||
|
||||
import { cn } from '@/lib/utils';
|
||||
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
'use client';
|
||||
|
||||
import * as React from 'react';
|
||||
import { Switch as SwitchPrimitives } from 'radix-ui';
|
||||
import * as SwitchPrimitives from '@radix-ui/react-switch';
|
||||
|
||||
import { cn } from '@/lib/utils';
|
||||
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
'use client';
|
||||
|
||||
import * as React from 'react';
|
||||
import { Tabs as TabsPrimitive } from 'radix-ui';
|
||||
import * as TabsPrimitive from '@radix-ui/react-tabs';
|
||||
|
||||
import { cn } from '@/lib/utils';
|
||||
|
||||
|
||||
@@ -15,9 +15,6 @@ from fastvideo import VideoGenerator as FastVideoGenerator
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))))
|
||||
|
||||
# InferenceArgs keys that are SamplingConfig fields of a GenerationRequest.
|
||||
_SAMPLING_INFERENCE_ARGS = ("height", "width", "num_frames", "num_inference_steps", "guidance_scale", "seed", "fps")
|
||||
|
||||
|
||||
# Custom exception for interruption
|
||||
class GenerationInterruptedException(Exception):
|
||||
@@ -157,17 +154,7 @@ class VideoGenerator:
|
||||
"""Thread function to run the generation"""
|
||||
try:
|
||||
if self.generator is not None:
|
||||
# Place each InferenceArgs value in the GenerationRequest section that owns it.
|
||||
request: dict[str, Any] = {"prompt": prompt, "output": {"output_path": output_path}}
|
||||
for key, value in inference_args.items():
|
||||
if key == "image_path":
|
||||
section = "inputs"
|
||||
elif key in _SAMPLING_INFERENCE_ARGS:
|
||||
section = "sampling"
|
||||
else:
|
||||
section = "extensions"
|
||||
request.setdefault(section, {})[key] = value
|
||||
self.generator.generate(request)
|
||||
self.generator.generate_video(prompt=prompt, output_path=output_path, **inference_args)
|
||||
self._generation_result = os.path.join(output_path, f"{prompt[:100]}.mp4")
|
||||
else:
|
||||
raise RuntimeError("Generator is not initialized")
|
||||
@@ -266,24 +253,9 @@ class VideoGenerator:
|
||||
if self.generator is None:
|
||||
print('generation_args', generation_args)
|
||||
print('pipeline_config', pipeline_config)
|
||||
# Place each generation argument at its GeneratorConfig engine path.
|
||||
engine_config: dict[str, Any] = {}
|
||||
if "num_gpus" in generation_args:
|
||||
engine_config["num_gpus"] = generation_args["num_gpus"]
|
||||
for parallelism_key in ("tp_size", "sp_size"):
|
||||
if parallelism_key in generation_args:
|
||||
engine_config.setdefault("parallelism", {})[parallelism_key] = generation_args[parallelism_key]
|
||||
if "dit_cpu_offload" in generation_args:
|
||||
engine_config["offload"] = {"dit": generation_args["dit_cpu_offload"]}
|
||||
self.generator = FastVideoGenerator.from_config({
|
||||
"model_path": model_path,
|
||||
"engine": engine_config,
|
||||
"pipeline": {
|
||||
"experimental": {
|
||||
"pipeline_config": pipeline_config
|
||||
}
|
||||
},
|
||||
})
|
||||
self.generator = FastVideoGenerator.from_pretrained(model_path=model_path,
|
||||
**generation_args,
|
||||
pipeline_config=pipeline_config)
|
||||
|
||||
print('inference_args', inference_args)
|
||||
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
{
|
||||
"version": 11,
|
||||
"version": 9,
|
||||
"recipes": [
|
||||
{
|
||||
"id": "fastwan21-t2v",
|
||||
@@ -445,13 +445,13 @@
|
||||
{
|
||||
"id": "fasth3-preview-cuda",
|
||||
"group": "fasth3-preview",
|
||||
"group_label": "FastH3 V1",
|
||||
"group_label": "FastH3 Preview",
|
||||
"group_task": "4-step text to video + audio",
|
||||
"family": "minimax_h3",
|
||||
"stage": "inference",
|
||||
"task": "Few-step text to video (with audio)",
|
||||
"label": "FastH3 V1 on CUDA",
|
||||
"summary": "Run FastH3 V1 with four DiT forwards, trained H3 sparse attention, compiled decode, and synchronized audio.",
|
||||
"label": "FastH3 Preview on CUDA",
|
||||
"summary": "Run the DMD2-distilled FastH3 Preview with four DiT forwards, trained H3 sparse attention, compiled decode, and synchronized audio.",
|
||||
"model": "FastVideo/FastVideo-Minimax-FastH3-Preview-v0.2",
|
||||
"source": "examples/inference/basic/basic_fasth3.py",
|
||||
"serving": {
|
||||
@@ -484,13 +484,13 @@
|
||||
"prepare": "hf download FastVideo/FastVideo-Minimax-FastH3-Preview-v0.2 --local-dir ./FastH3-Preview-v0.2\npython scripts/checkpoint_conversion/convert_minimax_h3_mlx.py --model-root ./FastH3-Preview-v0.2/transformer --out ./FastH3-MLX --formats \"int6\""
|
||||
},
|
||||
"group": "fasth3-preview",
|
||||
"group_label": "FastH3 V1",
|
||||
"group_label": "FastH3 Preview",
|
||||
"group_task": "4-step text to video + audio",
|
||||
"family": "minimax_h3",
|
||||
"stage": "inference",
|
||||
"task": "Few-step text to video (with audio)",
|
||||
"label": "FastH3 V1 on MLX",
|
||||
"summary": "Run FastH3 V1 on Apple Silicon with a locally converted INT6 DiT, streamed Qwen3-VL conditioning, and native MLX video and audio VAEs.",
|
||||
"label": "FastH3 Preview on MLX",
|
||||
"summary": "Run FastH3 Preview on Apple Silicon with a locally converted INT6 DiT, streamed Qwen3-VL conditioning, and native MLX video and audio VAEs.",
|
||||
"model": "FastVideo/FastVideo-Minimax-FastH3-Preview-v0.2",
|
||||
"source": "examples/inference/basic/mlx_fasth3.py",
|
||||
"command": "hf download FastVideo/FastVideo-Minimax-FastH3-Preview-v0.2 --local-dir ./FastH3-Preview-v0.2\npython scripts/checkpoint_conversion/convert_minimax_h3_mlx.py --model-root ./FastH3-Preview-v0.2/transformer --out ./FastH3-MLX --formats \"int6\"\npython examples/inference/basic/mlx_fasth3.py --model-root ./FastH3-Preview-v0.2 --mlx-checkpoint ./FastH3-MLX/int6 --prompt \"(S1) A presenter says <d>[English] FastVideo runs FastH3.</d>\" --height 480 --width 832 --num-frames 124 --seed 2026 --output-path ./outputs/fasth3_int6.mp4",
|
||||
@@ -516,13 +516,13 @@
|
||||
{
|
||||
"id": "fasth3-preview-spark",
|
||||
"group": "fasth3-preview",
|
||||
"group_label": "FastH3 V1",
|
||||
"group_label": "FastH3 Preview",
|
||||
"group_task": "4-step text to video + audio",
|
||||
"family": "minimax_h3",
|
||||
"stage": "inference",
|
||||
"task": "Few-step text to video (with audio)",
|
||||
"label": "FastH3 V1 on one DGX Spark",
|
||||
"summary": "Run FastH3 V1 on one GB10 with Triton VSA, FA4 off, and lazy module load. Height, width, frames, and steps in the YAML are examples.",
|
||||
"label": "FastH3 Preview on one DGX Spark",
|
||||
"summary": "Run FastH3 Preview on one GB10 with Triton VSA, FA4 off, and lazy module load. Height, width, frames, and steps in the YAML are examples.",
|
||||
"model": "FastVideo/FastVideo-FastH3-4-step-Preview-v1-VSA-DataFree",
|
||||
"source": "examples/inference/basic/basic_fasth3_spark.yaml",
|
||||
"serving": {
|
||||
@@ -544,19 +544,19 @@
|
||||
"limitations": [
|
||||
"Install from the DGX Spark guide, not the generic CUDA extra. GB10 has no FA4 / sm_100a VSA kernel; keep FASTVIDEO_FA4=0 and FASTVIDEO_VSA_SM100A=0.",
|
||||
"Legal num_frames values are 17n+5, capped at 345 (15 s). Native 16:9 sizes include 832x480 and 1344x768.",
|
||||
"Lazy module load reloads Qwen3-VL and the DiT between phases of each request. Do not set engine.offload.lazy_module_load to false on this box.",
|
||||
"Lazy module load reloads Qwen3-VL and the DiT between phases of each request. Do not pass --no-lazy-module-load on this box.",
|
||||
"A 345-frame request on one Spark can OOM. Prefer 124 or 243 frames, TAEH3 decode, or two Sparks over QSFP."
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "fasth3-spark-pair",
|
||||
"group": "fasth3-preview",
|
||||
"group_label": "FastH3 V1",
|
||||
"group_label": "FastH3 Preview",
|
||||
"group_task": "4-step text to video + audio",
|
||||
"family": "minimax_h3",
|
||||
"stage": "inference",
|
||||
"task": "Few-step text to video (with audio)",
|
||||
"label": "FastH3 V1 on two DGX Sparks",
|
||||
"label": "FastH3 Preview on two DGX Sparks",
|
||||
"summary": "Run one FastH3 clip across two GB10s with Ray sequence parallel over QSFP RoCE. Sequential load and lazy module load stay on because SP replicates the DiT on each node.",
|
||||
"model": "FastVideo/FastVideo-FastH3-4-step-Preview-v1-VSA-DataFree",
|
||||
"source": "examples/inference/basic/basic_fasth3_spark_pair.yaml",
|
||||
@@ -581,19 +581,15 @@
|
||||
{
|
||||
"id": "fasth3-8step-v2-cuda",
|
||||
"group": "fasth3-8step-v2",
|
||||
"group_label": "FastH3 V2",
|
||||
"group_label": "FastH3 8-Step V2",
|
||||
"group_task": "8-step text to video + audio",
|
||||
"family": "minimax_h3",
|
||||
"stage": "inference",
|
||||
"task": "Few-step text to video (with audio)",
|
||||
"label": "FastH3 V2 on CUDA",
|
||||
"summary": "Run FastH3 V2, the eight-forward checkpoint (video/audio shifts 10/3, VSA 0.8, 64-token tiles) with the trained DMD ladder loaded from the checkpoint's fastvideo_inference.json.",
|
||||
"label": "FastH3 8-Step V2 on CUDA",
|
||||
"summary": "Run the eight-forward FastH3 8-Step V2 checkpoint (video/audio shifts 10/3, VSA 0.8, 64-token tiles) with the trained DMD ladder loaded from the checkpoint's fastvideo_inference.json.",
|
||||
"model": "FastVideo/FastVideo-FastH3-8-Step-V2",
|
||||
"source": "examples/inference/basic/basic_fasth3_8step.py",
|
||||
"serving": {
|
||||
"source": "examples/serving/openai_fasth3_8step.yaml",
|
||||
"install": "UV_TORCH_BACKEND=cu130 uv pip install -e \".[fasth3]\""
|
||||
},
|
||||
"command": "UV_TORCH_BACKEND=cu130 uv pip install -e \".[fasth3]\"\npython examples/inference/basic/basic_fasth3_8step.py --prompt \"(S1) A presenter says <d>[English] FastVideo runs FastH3.</d>\" --profile strict --no-inference-torch-compile --no-compile-vae",
|
||||
"gpu_types": ["NVIDIA"],
|
||||
"hardware": {
|
||||
@@ -613,46 +609,7 @@
|
||||
"limitations": [
|
||||
"Nine sigma-grid points (eight transformer forwards) are fixed by the checkpoint's trained ladder; the example rejects any other --steps.",
|
||||
"Validated with the eager strict route (--profile strict --no-inference-torch-compile --no-compile-vae). The compiled all profile has not been measured for this checkpoint.",
|
||||
"T2AV only; no FL2VA/Ref2VA distillation and no matching LoRA. V2 uses eight forwards rather than V1's four."
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "fasth3-8step-v2-mlx",
|
||||
"serving": {
|
||||
"source": "examples/serving/mlx_fasth3_8step.yaml",
|
||||
"install": "uv pip install -e \".[mlx]\"",
|
||||
"prepare": "hf download FastVideo/FastVideo-FastH3-8-Step-V2 --local-dir ./FastH3-8-Step-V2\npython scripts/checkpoint_conversion/convert_minimax_h3_mlx.py --model-root ./FastH3-8-Step-V2/transformer --out ./FastH3-8-Step-V2-MLX --formats \"int8\" --include-vsa"
|
||||
},
|
||||
"group": "fasth3-8step-v2",
|
||||
"group_label": "FastH3 V2",
|
||||
"group_task": "8-step text to video + audio",
|
||||
"family": "minimax_h3",
|
||||
"stage": "inference",
|
||||
"task": "Few-step text to video (with audio)",
|
||||
"label": "FastH3 V2 on MLX",
|
||||
"summary": "Run FastH3 V2 on Apple Silicon with a locally converted INT8 DiT, the checkpoint's DMD contract ladder, and trained VSA (sparsity 0.8, 64-token tiles).",
|
||||
"model": "FastVideo/FastVideo-FastH3-8-Step-V2",
|
||||
"source": "examples/inference/basic/mlx_fasth3_8step.py",
|
||||
"command": "hf download FastVideo/FastVideo-FastH3-8-Step-V2 --local-dir ./FastH3-8-Step-V2\npython scripts/checkpoint_conversion/convert_minimax_h3_mlx.py --model-root ./FastH3-8-Step-V2/transformer --out ./FastH3-8-Step-V2-MLX --formats \"int8\" --include-vsa\npython examples/inference/basic/mlx_fasth3_8step.py --model-root ./FastH3-8-Step-V2 --mlx-checkpoint ./FastH3-8-Step-V2-MLX/int8 --prompt \"(S1) A presenter says <d>[English] FastVideo runs FastH3.</d>\" --height 480 --width 832 --num-frames 124 --seed 2026 --output-path ./outputs/fasth3_8step_int8.mp4",
|
||||
"gpu_types": ["Apple Silicon"],
|
||||
"hardware": {
|
||||
"platform": "mlx",
|
||||
"accelerator": "Apple M4 Max",
|
||||
"system_memory": "36 GB unified memory",
|
||||
"peak_memory": "27.39 GiB peak MLX memory during denoising",
|
||||
"evidence": "validated",
|
||||
"evidence_url": "https://github.com/hao-ai-lab/FastVideo/pull/1863"
|
||||
},
|
||||
"evidence": "Verified",
|
||||
"expected_artifact": "MP4 with H.264 video and stereo AAC audio at outputs/fasth3_8step_int8.mp4",
|
||||
"modes": ["T2VA", "8-step FastH3", "trained VSA"],
|
||||
"knobs": [
|
||||
{"key": "video_decode_backend", "label": "VAE decode", "hint": "Fidelity vs. speed", "flag": "--video-decode-backend", "options": [{"value": "h3-vae", "label": "Full H3 VAE"}, {"value": "taeh3", "label": "TAEH3 preview"}], "default": "h3-vae"}
|
||||
],
|
||||
"limitations": [
|
||||
"Convert with --include-vsa. mlx_fasth3_8step.py turns VSA on (sparsity 0.8, tile 64). A dense export fails at configure_vsa.",
|
||||
"--steps 8 or 9 both run the eight trained forwards. Reuse the preview VAE, audio VAE, text encoder, and tokenizer if those directories already exist.",
|
||||
"The MLX path supports T2VA only. FL2VA, Ref2VA, and two-pass refinement are not wired."
|
||||
"T2AV only; no FL2VA/Ref2VA distillation and no matching LoRA. Quality and latency were not compared against the four-step preview in this PR."
|
||||
]
|
||||
},
|
||||
{
|
||||
|
||||
@@ -136,7 +136,7 @@
|
||||
if (platform === "mps") {
|
||||
return {
|
||||
id: "mps",
|
||||
label: "Apple Silicon · PyTorch MPS",
|
||||
label: "Apple Silicon · MPS",
|
||||
hint: recipe.hardware?.minimum_memory || recipe.hardware?.system_memory || "Memory not recorded",
|
||||
};
|
||||
}
|
||||
@@ -519,10 +519,10 @@
|
||||
option.setAttribute("aria-pressed", String(selected));
|
||||
});
|
||||
servingAvailability.textContent = profile
|
||||
? "The playground and the OpenAI Python client share one server process. Both workflows can run on your own machine."
|
||||
? "The playground and API clients share one server process. Both workflows can run on your own machine."
|
||||
: servingLoadFailed
|
||||
? "Server examples could not be loaded. Open the H3 server guide below, or use Python directly."
|
||||
: "This recipe uses Python directly. FastH3 V1 and FastH3 V2 can also run a local server for the playground and the OpenAI Python client.";
|
||||
: "This recipe uses Python directly. For the playground and API clients, choose FastH3 Preview with CUDA, MLX, or one Spark.";
|
||||
servingPanel.hidden = !useServer;
|
||||
commandBlock.hidden = useServer;
|
||||
root.querySelector("[data-cookbook-python-note]").hidden = useServer;
|
||||
@@ -537,7 +537,7 @@
|
||||
? "Start once, then change prompts in the playground or your app. On a DGX Spark, lazy module load still reloads Qwen3-VL and the DiT between phases of each request, so later prompts are not a free hot cache."
|
||||
: "Start once, then change prompts in the playground or your app. CUDA requests reuse the loaded model. The Python SDK can also reuse a generator within one process.";
|
||||
servingPanel.querySelector("[data-cookbook-install-guide]").href = isMLX
|
||||
? "../../getting_started/installation/mlx/"
|
||||
? "../../getting_started/installation/mps/#run-fasth3-preview"
|
||||
: isSpark
|
||||
? "../../getting_started/installation/spark/"
|
||||
: "../../getting_started/installation/gpu/";
|
||||
@@ -574,7 +574,7 @@
|
||||
});
|
||||
|
||||
description.textContent = useServer
|
||||
? `${recipe.group_label || recipe.label} generates video with audio. Start the local server, then use the playground or the OpenAI Python client. This profile uses the checked-in ${runtime.label} configuration.`
|
||||
? `FastH3 Preview generates video with audio. This server profile uses the checked-in ${runtime.label} configuration.`
|
||||
: recipe.summary;
|
||||
label.textContent = useServer ? `${recipe.group_label || recipe.label} · Server` : recipe.label;
|
||||
model.textContent = recipe.model;
|
||||
|
||||
@@ -18,8 +18,8 @@ the CUDA `fastvideo-kernel` package:
|
||||
- **Dense-only checkpoints** (the default converter) drop the 50 gate
|
||||
matrices and keep fused SDPA. They remain valid for dense inference.
|
||||
- **VSA-capable checkpoints** retain those gates, quantize them on the same
|
||||
affine grid, and record `vsa.capable` in `mlx_h3_dit.json`. Preview leaves
|
||||
runtime VSA off until you pass `--vsa`. `mlx_fasth3_8step.py` turns it on.
|
||||
affine grid, and record `vsa.capable` in `mlx_h3_dit.json`. Runtime VSA is
|
||||
still off until you pass `--vsa`.
|
||||
- **Tile sizes** 64 `(4, 4, 4)` and 256 `(4, 8, 8)`. Prefix keys can be
|
||||
`exempt` or `compete`. `--vsa-dense-first-n-steps` and `--vsa-dense-layers`
|
||||
force dense SDPA on the selected steps or blocks.
|
||||
@@ -32,12 +32,9 @@ the CUDA `fastvideo-kernel` package:
|
||||
but does not yet match reference video. `--vsa-impl reference` is the same
|
||||
as `auto`.
|
||||
|
||||
See the [MLX install guide](../../getting_started/installation/mlx.md)
|
||||
and the [MiniMax H3 cookbook](../../cookbook/minimax-h3.md) for conversion
|
||||
and `mlx_fasth3.py` / `mlx_fasth3_8step.py` flags. Do not enable
|
||||
VSA on a dense-only checkpoint; reconvert with `--include-vsa` first. V1
|
||||
VSA is opt-in. FastH3 V2 converts with `--include-vsa` and turns VSA on
|
||||
by default.
|
||||
See the [Apple Silicon guide](../../getting_started/installation/mps.md) for
|
||||
conversion and `mlx_fasth3.py` flags. Do not enable VSA on a dense-only
|
||||
checkpoint; reconvert with `--include-vsa` first.
|
||||
|
||||
H3 uses fused MLX RMSNorm by default, including dense inference. This can
|
||||
change BF16 rounding relative to the older explicit normalization path.
|
||||
|
||||
@@ -30,7 +30,7 @@ FastVideo and the reference model first produce different numbers?"
|
||||
| General logging | `init_logger(__name__)` |
|
||||
| Per-stage timing | `FASTVIDEO_STAGE_LOGGING` |
|
||||
| Profiling kernel timings | `FASTVIDEO_TORCH_PROFILER_DIR` (see [Profiling](profiling.md)) |
|
||||
| Function-call tracing | `fastvideo.logger.enable_trace_function_call()` (heavy) |
|
||||
| Function-call tracing | `FASTVIDEO_TRACE_FUNCTION` (heavy) |
|
||||
|
||||
## Quickstart
|
||||
|
||||
|
||||
@@ -57,15 +57,19 @@ Minimal usage example (based on `examples/inference/basic/basic.py`):
|
||||
|
||||
```python
|
||||
from fastvideo import VideoGenerator
|
||||
from fastvideo.api.sampling_param import SamplingParam
|
||||
|
||||
model_id = "Wan-AI/Wan2.1-T2V-1.3B-Diffusers" # or official_weights/<model_name>/
|
||||
generator = VideoGenerator.from_pretrained(model_id, {"engine": {"num_gpus": 1}})
|
||||
generator = VideoGenerator.from_pretrained(model_id, num_gpus=1)
|
||||
|
||||
video = generator.generate({
|
||||
"prompt": "A vibrant city street at sunset.",
|
||||
"sampling": {"num_frames": 45},
|
||||
"output": {"output_path": "video_samples", "save_video": True},
|
||||
})
|
||||
sampling = SamplingParam.from_pretrained(model_id)
|
||||
sampling.num_frames = 45
|
||||
video = generator.generate_video(
|
||||
"A vibrant city street at sunset.",
|
||||
sampling_param=sampling,
|
||||
output_path="video_samples",
|
||||
save_video=True,
|
||||
)
|
||||
```
|
||||
|
||||
## Some questions to ask yourself before starting
|
||||
|
||||
@@ -1,294 +0,0 @@
|
||||
# Environment Variables
|
||||
|
||||
FastVideo reads environment variables for expert switches, debugging, profiling, and the settings that launchers such
|
||||
as `torchrun` provide. This page is the policy for those variables. The contract test
|
||||
`fastvideo/tests/contract/test_env_policy.py` enforces the policy in the unit CI lane, and the coding-agent skill
|
||||
`.agents/skills/env-var-conventions/SKILL.md` points here. When the policy changes, update this page and the contract
|
||||
test in the same pull request.
|
||||
|
||||
## Rules
|
||||
|
||||
1. **Register every FastVideo variable in `fastvideo/envs.py`.** Each entry declares a type, a default, a category,
|
||||
and a description. Variables that other tools own (CUDA, NCCL, PyTorch, launchers) are not registered; code reads
|
||||
them directly with `os.environ.get("NAME")`, and the name must be in the external-variable allowlist
|
||||
(`EXTERNAL_ALLOWLIST` in the contract test). When FastVideo sets such a variable for the other tool, it calls
|
||||
`envs.set_external`, `envs.setdefault_external`, or `envs.unset_external`, and the name must be in
|
||||
`EXTERNAL_WRITE_ALLOWLIST`. Variables that FastVideo's CI and CI tooling define (for example `TEST_SCOPE` and
|
||||
`PERF_RUN_SOURCE`) keep their names, and test code under `fastvideo/tests/` reads them directly; they are listed
|
||||
in `CI_ONLY_VARIABLES` in the contract test, together with the file that sets each one.
|
||||
2. **Read with `envs.NAME.get()`, write with `envs.NAME.set()`, and change a value in tests with
|
||||
`envs.NAME.override()`.** Each type has one parsing rule. A value that the rule rejects raises
|
||||
`fastvideo.envs.EnvVarError` instead of falling back to the default.
|
||||
3. **Name FastVideo variables with the `FASTVIDEO_` prefix.** The second word states the purpose where one applies:
|
||||
`ENABLE_`, `DISABLE_`, `USE_`, `FORCE_`, `DEBUG_`, `TEST_`. Variables that only tests read use
|
||||
`FASTVIDEO_TEST_`, for example `FASTVIDEO_TEST_SD35_MODEL_DIR`, and the category `test`.
|
||||
4. **Keep a renamed variable as a deprecated alias until the next minor release.** Setting the old name logs a
|
||||
warning. Delete a variable that no code reads, and list it in `DEPRECATED_VARIABLES` so that setting it logs a
|
||||
warning.
|
||||
5. **Give each setting one source: an argument or an environment variable.** Settings that users change per
|
||||
deployment are arguments (CLI or YAML). Expert switches, emergency off switches, and debugging and test switches
|
||||
are environment variables.
|
||||
6. **Read variables inside functions.** `envs.NAME.get()` runs when the function runs, so a changed value takes
|
||||
effect without re-importing a module. Module level, class bodies, decorators, and default argument values run at
|
||||
import time.
|
||||
7. **Do not write the environment to pass values between parts of FastVideo.** Pass an argument instead. Tests use
|
||||
`envs.NAME.override()`, and `envs.override_external()` for variables outside the registry; both restore the
|
||||
previous value.
|
||||
|
||||
## Field types
|
||||
|
||||
| Class | Value type | Parsing rule |
|
||||
| ------------ | ---------------- | ----------------------------------------------------------------------------------- |
|
||||
| `EnvBool` | `bool` | `1`, `true`, `yes`, `on` are true; `0`, `false`, `no`, `off`, and `""` are false. |
|
||||
| | | Case-insensitive; surrounding whitespace is ignored. |
|
||||
| `EnvInt` | `int` | `int(value)` |
|
||||
| `EnvFloat` | `float` | `float(value)` |
|
||||
| `EnvStr` | `str` or `None` | The raw string. A `None` default means that the variable has no default. |
|
||||
| `EnvPath` | `str` or `None` | The raw string with a leading `~` expanded. |
|
||||
| `EnvChoice` | `str` | Stripped and lower-cased, then checked against the declared `choices`. |
|
||||
|
||||
A default can be a zero-argument function; `get()` calls it on each read while the variable is unset. The path roots
|
||||
use this to follow `XDG_CONFIG_HOME` and `XDG_CACHE_HOME`.
|
||||
|
||||
Using a field without a method, as in `if envs.FASTVIDEO_FA4:`, raises `TypeError`.
|
||||
|
||||
## Add a variable
|
||||
|
||||
1. Declare the variable in the matching section of `fastvideo/envs.py`:
|
||||
|
||||
```python
|
||||
FASTVIDEO_DEBUG_MY_STAGE = EnvBool(False, category="debug", doc="Log the inputs of MyStage.")
|
||||
```
|
||||
|
||||
The category is one of the values in `envs.CATEGORIES`.
|
||||
|
||||
2. Read the variable inside a function:
|
||||
|
||||
```python
|
||||
import fastvideo.envs as envs
|
||||
|
||||
def forward(self, batch):
|
||||
if envs.FASTVIDEO_DEBUG_MY_STAGE.get():
|
||||
logger.info("MyStage inputs: %s", batch.keys())
|
||||
```
|
||||
|
||||
3. Regenerate the table at the end of this page:
|
||||
|
||||
```bash
|
||||
python fastvideo/tests/contract/test_env_policy.py
|
||||
```
|
||||
|
||||
4. Run the contract test:
|
||||
|
||||
```bash
|
||||
pytest fastvideo/tests/contract/test_env_policy.py
|
||||
```
|
||||
|
||||
In a test, change the value with `override`, which restores the previous value on exit:
|
||||
|
||||
```python
|
||||
with envs.FASTVIDEO_DEBUG_MY_STAGE.override(True):
|
||||
run_stage()
|
||||
```
|
||||
|
||||
For a variable outside the registry, such as `MASTER_PORT` or `TEST_SCOPE`, use `envs.override_external`. To keep
|
||||
an override until the end of a test, enter it through the `env_overrides` fixture from `fastvideo/tests/conftest.py`,
|
||||
which restores every value at teardown:
|
||||
|
||||
```python
|
||||
def test_my_stage(env_overrides):
|
||||
env_overrides.enter_context(envs.FASTVIDEO_DEBUG_MY_STAGE.override(True))
|
||||
env_overrides.enter_context(envs.override_external("MASTER_PORT", "29512"))
|
||||
run_stage()
|
||||
```
|
||||
|
||||
In test code under `fastvideo/tests/`, `override_external` accepts any name that code may read directly
|
||||
(`EXTERNAL_ALLOWLIST`, `CI_ONLY_VARIABLES`) or that is in `EXTERNAL_WRITE_ALLOWLIST`. Library code may write only
|
||||
the names in `EXTERNAL_WRITE_ALLOWLIST`.
|
||||
|
||||
## Rename or remove a variable
|
||||
|
||||
To rename a variable, declare it under the new name and list the old name in `deprecated_names`:
|
||||
|
||||
```python
|
||||
FASTVIDEO_LTX2_USE_DISTILLED_SIGMAS = EnvBool(True,
|
||||
category="sampling",
|
||||
doc="...",
|
||||
deprecated_names=("LTX2_USE_DISTILLED_SIGMAS", ))
|
||||
```
|
||||
|
||||
`get()` reads an old name only when the new name is unset, and logs a warning once. Update the uses of the old name
|
||||
in `examples/`, `scripts/`, `docs/`, `apps/`, and the tests in the same pull request. Delete the old name in the next
|
||||
minor release.
|
||||
|
||||
To remove a variable that no code reads, delete its entry and add the name to `DEPRECATED_VARIABLES` in
|
||||
`fastvideo/envs.py` with a reason. The config resolution step `warn_deprecated_environment_variables` in
|
||||
`fastvideo/api/inference_resolution.py` calls `envs.warn_deprecated_variables()`, which logs a warning for each listed
|
||||
variable that is set. Delete the entry in the next minor release.
|
||||
|
||||
## What the contract test checks
|
||||
|
||||
The test parses every Python file under `fastvideo/`, including `fastvideo/tests/`, with Python's `ast` module. It
|
||||
skips `fastvideo/third_party/`, which is copied from upstream projects, and the registry `fastvideo/envs.py`. It does
|
||||
not check `apps/`, `examples/`, `scripts/`, `fastvideo-kernel/`, or `docs/`.
|
||||
|
||||
It reports each violation as `<path>: <kind> <name>`:
|
||||
|
||||
| Kind | Code that triggers it | Fix |
|
||||
| ------------------ | ------------------------------------------------------------ | -------------------------------------------- |
|
||||
| `read` | `os.getenv`, `os.environ.get`, `os.environ[...]`, or | Register the variable and call |
|
||||
| | `"NAME" in os.environ` with a name outside the allowlist, or | `envs.NAME.get()`. For a variable that |
|
||||
| | with a name built at runtime (`<dynamic>`) | another tool owns, add it to |
|
||||
| | | `EXTERNAL_ALLOWLIST` with a reason. |
|
||||
| `write` | `os.environ[...] = ...`, `setdefault`, `pop`, `del`, | Pass an argument instead. In tests, use |
|
||||
| | `os.putenv`, `os.unsetenv`, `monkeypatch.setenv`/`delenv`, | `envs.NAME.override()`, or |
|
||||
| | or an `envs.*_external` call with a name that the | `envs.override_external()` for a variable |
|
||||
| | helper does not accept | outside the registry. For a variable that |
|
||||
| | | another tool reads, call an |
|
||||
| | | `envs.*_external` helper and add the name to |
|
||||
| | | `EXTERNAL_WRITE_ALLOWLIST` with a reason. |
|
||||
| `whole-environ` | `os.environ.copy()`, `dict(os.environ)`, iteration, | Read the specific variables that the code |
|
||||
| | `mock.patch.dict(os.environ, ...)`, `os.environ.update` | needs. |
|
||||
| `bare-field` | A registry field used without calling one of its methods, | Call `envs.NAME.get()`. |
|
||||
| | as in `envs.NAME == "auto"` or `getter = envs.NAME.get` | |
|
||||
| `import-time-read` | `envs.NAME.get()` outside a function | Move the read into the function that uses |
|
||||
| | | the value. |
|
||||
| `prefix` | A registry entry without the `FASTVIDEO_` prefix | Rename the variable and keep the old name in |
|
||||
| | | `deprecated_names`. |
|
||||
| `unread` | A registry entry that no code reads with `get()` or | Delete the variable and add it to |
|
||||
| | `is_set()` | `DEPRECATED_VARIABLES`. |
|
||||
|
||||
The test recognizes `os` imported under another name, `from os import environ, getenv`, and a name held in a
|
||||
module-level string constant. Code that reaches the environment through `importlib` or `getattr(os, "environ")` is
|
||||
left to code review.
|
||||
|
||||
The test also checks that every registry entry has a category from `envs.CATEGORIES` and a description, and that the
|
||||
table at the end of this page matches the registry.
|
||||
|
||||
**Known violations.** `KNOWN_VIOLATIONS` in the contract test lists the violations that existed when the policy was
|
||||
introduced. The list only shrinks. A violation that is not in the list fails the test. A listed violation that no
|
||||
longer exists also fails the test, so the fixing pull request deletes its entry.
|
||||
|
||||
## Registered variables
|
||||
|
||||
<!-- BEGIN GENERATED ENV TABLE: python fastvideo/tests/contract/test_env_policy.py -->
|
||||
| Variable | Type | Default | Category | Description |
|
||||
| ---------------------------------------------------------- | ------------------------------ | ----------------------------------------------------- | ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `FASTVIDEO_CONFIG_ROOT` | path | computed | path | Root directory for FastVideo configuration files, at runtime and at installation. Defaults to ~/.config/fastvideo, or $XDG_CONFIG_HOME/fastvideo when XDG_CONFIG_HOME is set. |
|
||||
| `FASTVIDEO_CACHE_ROOT` | path | computed | path | Root directory for FastVideo cache files. Defaults to ~/.cache/fastvideo, or $XDG_CACHE_HOME/fastvideo when XDG_CACHE_HOME is set. |
|
||||
| `FASTVIDEO_REASON1_WEIGHTS_PATH` | str | unset | path | Local path or Hugging Face id of Reason1 weights to load instead of the checkpoint's own. |
|
||||
| `FASTVIDEO_HOST_IP` | str | `""` | distributed | IP address of this node when the node has several network interfaces. Set it on each node for multi-node inference. |
|
||||
| `FASTVIDEO_LOOPBACK_IP` | str | `""` | distributed | Loopback IP address to use instead of the detected one. |
|
||||
| `FASTVIDEO_RAY_PER_WORKER_GPUS` | float | `1.0` | distributed | GPUs per Ray worker. A fraction lets Ray schedule several actors on one GPU, so other actors can share the GPUs with FastVideo. |
|
||||
| `FASTVIDEO_NCCL_SO_PATH` | str | unset | distributed | Path to the NCCL library file. Needed because the nccl>=2.19 that PyTorch ships has a bug (https://github.com/NVIDIA/nccl/issues/1234). |
|
||||
| `FASTVIDEO_HCCL_SO_PATH` | str | unset | distributed | Path to the HCCL library file on Ascend NPUs. Deprecated names: `HCCL_SO_PATH`. |
|
||||
| `FASTVIDEO_WORKER_MULTIPROC_METHOD` | one of spawn, fork, forkserver | `spawn` | distributed | Multiprocessing start method for worker processes. |
|
||||
| `FASTVIDEO_ULYSSES_A2A` | one of off, auto | `off` | distributed | Sequence-parallel all-to-all backend. off uses the NCCL path in DistributedAutograd.AllToAll4D. auto uses the fused NVLink kernel when the group is a load-store accessible mesh of 2, 4, 6, or 8 ranks in eager execution, and the NCCL path otherwise. |
|
||||
| `FASTVIDEO_CONFIGURE_LOGGING` | bool | `1` | logging | Configure logging at import. When true, FastVideo uses its default logging configuration or the file in FASTVIDEO_LOGGING_CONFIG_PATH. |
|
||||
| `FASTVIDEO_LOGGING_CONFIG_PATH` | str | unset | logging | Path to a JSON logging configuration file. |
|
||||
| `FASTVIDEO_LOGGING_LEVEL` | str | `INFO` | logging | Default logging level. |
|
||||
| `FASTVIDEO_LOGGING_PREFIX` | str | `""` | logging | Prefix prepended to every log message. |
|
||||
| `FASTVIDEO_STAGE_LOGGING` | bool | `0` | logging | Log the time that each pipeline stage takes. |
|
||||
| `FASTVIDEO_ATTENTION_BACKEND` | str | unset | attention | Attention backend, as an AttentionBackendEnum name such as TORCH_SDPA, FLASH_ATTN, VIDEO_SPARSE_ATTN, SAGE_ATTN, or SAGE_ATTN_THREE. Config resolution uses it when engine.attention.backend is unset. An unsupported name raises an error. |
|
||||
| `FASTVIDEO_FA4` | bool | `0` | attention | The FLASH_ATTN backend uses FlashAttention-4 (flash_attn.cute) instead of FA3 or FA2. |
|
||||
| `FASTVIDEO_MINIMAX_H3_FA4_PACKED_VARLEN` | bool | `0` | attention | MiniMax-H3 dense DiT self-attention uses the FlashAttention-4 packed-varlen entry point. This changes the floating-point reduction order, so it is an inference-only opt-in. |
|
||||
| `FASTVIDEO_VSA_SM100A` | bool | `0` | attention | VIDEO_SPARSE_ATTN_H3 sends no-grad tile-64 forwards to the data-center Blackwell (sm_100a) kernel. fastvideo-kernel reads the same variable with the same rule. |
|
||||
| `FASTVIDEO_NVFP4_FA4` | bool | `0` | attention | FlashAttention-4 quantizes Q and K to NVFP4. An explicit nvfp4_fa4 attention implementation argument takes precedence. |
|
||||
| `FASTVIDEO_DISABLE_ATTENTION_COMPILE` | bool | `1` | attention | Keep attention forward out of torch.compile graphs (torch.compiler.disable). Set it to 0 to let attention constructed under that setting be traced. Setting it explicitly to true also blocks regional compile. |
|
||||
| `FASTVIDEO_MLX_WINDOW` | int | `0` | attention | MLX FastWan windowed attention size in tokens. 0 uses full attention. |
|
||||
| `FASTVIDEO_MLX_WINDOW_SINK` | int | `0` | attention | Number of sink tokens that MLX windowed attention always attends to. |
|
||||
| `FASTVIDEO_INFERENCE_TORCH_COMPILE` | bool | `0` | performance | Compile each DiT transformer block with fullgraph torch.compile at inference. Same as engine.compile.regional=True. |
|
||||
| `FASTVIDEO_VAE_PARALLEL_DECODE` | bool | `0` | performance | MiniMax-H3 VAE decode splits its temporal chunks across the sequence-parallel ranks instead of running serially on the output rank. Same as pipeline.model.minimax_h3.vae_parallel_decode=True. |
|
||||
| `FASTVIDEO_VAE_PARALLEL_ENCODE` | bool | `0` | performance | MiniMax-H3 reference-video VAE encode splits its temporal chunks across the sequence-parallel ranks. Same as pipeline.model.minimax_h3.vae_parallel_encode=True. |
|
||||
| `FASTVIDEO_VAE_PARALLEL_DECODE_STRATEGY` | str | unset | performance | Collective that moves chunks in parallel VAE decode: gather (used when unset) or all_gather. |
|
||||
| `FASTVIDEO_MINIMAX_H3_FUSIONS` | str | `""` | performance | MiniMax-H3 inference-only Triton fusions: all, 1, or a comma-separated subset of modulate,qknorm_rope,swiglu. Empty, 0, or none keeps the eager implementation. |
|
||||
| `FASTVIDEO_FSDP2_AUTOWRAP` | bool | `0` | performance | FSDP2 shards modules by parameter count instead of the model's shard conditions. Not supported by self-forcing distillation. |
|
||||
| `FASTVIDEO_FSDP2_MIN_PARAMS` | int | `10000000` | performance | Minimum parameter count of a module that FASTVIDEO_FSDP2_AUTOWRAP shards. |
|
||||
| `FASTVIDEO_MLX_COMPILE` | bool | `0` | performance | Compile the MLX DiT forward with mx.compile. |
|
||||
| `FASTVIDEO_MLX_FAST_NORM` | bool | `0` | performance | Use MLX fast normalization kernels. |
|
||||
| `FASTVIDEO_MLX_DQ_GEMM` | str | `1` | performance | MLX dequantized GEMM for affine-quantized weights: 0 turns it off, 1 uses the measured minimum row count, and an integer sets the minimum row count. |
|
||||
| `FASTVIDEO_LTX2_VAE_CHANNELS_LAST_3D` | bool | `1` | performance | LTX-2 VAE uses the channels_last_3d memory format. |
|
||||
| `FASTVIDEO_LTX2_DISABLE_AUDIO_AUTOCAST` | bool | `1` | performance | LTX-2 audio decoding runs without CUDA autocast. Deprecated names: `LTX2_DISABLE_AUDIO_AUTOCAST`. |
|
||||
| `FASTVIDEO_FLUX2_DISABLE_BF16_REDUCED_PRECISION_REDUCTION` | bool | `0` | performance | Flux denoising disables reduced-precision reductions in bf16 matmuls, which tightens accumulation for the 4-step Klein model. |
|
||||
| `FASTVIDEO_FFMPEG_BIN` | str | `ffmpeg` | output | ffmpeg executable used to save video with audio. |
|
||||
| `FASTVIDEO_VIDEO_CODEC` | str | `libx264` | output | ffmpeg video codec for saved videos. |
|
||||
| `FASTVIDEO_NVENC_PRESET` | str | `p1` | output | NVENC preset when the codec is an \*_nvenc codec. |
|
||||
| `FASTVIDEO_NVENC_TUNE` | str | `ull` | output | NVENC tune option. |
|
||||
| `FASTVIDEO_NVENC_RC` | str | `constqp` | output | NVENC rate-control mode. |
|
||||
| `FASTVIDEO_NVENC_QP` | str | `28` | output | NVENC quantization parameter. |
|
||||
| `FASTVIDEO_NVENC_BF` | str | `0` | output | NVENC number of B-frames. |
|
||||
| `FASTVIDEO_X264_PRESET` | str | `ultrafast` | output | x264 preset for non-NVENC codecs. |
|
||||
| `FASTVIDEO_OUTPUT_PIX_FMT` | str | `yuv420p` | output | ffmpeg pixel format for saved videos. |
|
||||
| `FASTVIDEO_NVTX_PROFILE` | bool | `0` | profiling | Emit NVTX ranges for external profilers such as Nsight Systems. |
|
||||
| `FASTVIDEO_TORCH_PROFILER_DIR` | path | unset | profiling | Enables the torch profiler and sets the directory for its traces. Must be an absolute path. |
|
||||
| `FASTVIDEO_TORCH_PROFILER_RECORD_SHAPES` | bool | `0` | profiling | Torch profiler records shapes. |
|
||||
| `FASTVIDEO_TORCH_PROFILER_WITH_PROFILE_MEMORY` | bool | `0` | profiling | Torch profiler profiles memory. |
|
||||
| `FASTVIDEO_TORCH_PROFILER_WITH_STACK` | bool | `0` | profiling | Torch profiler captures stacks. Costs about 1.5x runtime and 1.4x trace size. |
|
||||
| `FASTVIDEO_TORCH_PROFILER_WITH_FLOPS` | bool | `0` | profiling | Torch profiler profiles FLOPs. |
|
||||
| `FASTVIDEO_TORCH_PROFILE_REGIONS` | str | `""` | profiling | Comma-separated profiler regions to record. The torch profiler requires at least one region. |
|
||||
| `FASTVIDEO_TRACE_ACTIVATIONS` | bool | `0` | debug | Enable activation trace hooks. |
|
||||
| `FASTVIDEO_TRACE_LAYERS` | str | `""` | debug | Regex filter for traced module names. Empty means all. |
|
||||
| `FASTVIDEO_TRACE_STATS` | str | `abs_mean,sum` | debug | Comma-separated activation statistics dumped for each output tensor. |
|
||||
| `FASTVIDEO_TRACE_OUTPUT` | str | `/tmp/fv_trace_<pid>.jsonl` | debug | JSONL path for activation traces. The literal <pid> is replaced at runtime. |
|
||||
| `FASTVIDEO_TRACE_STEPS` | str | `""` | debug | Comma-separated denoising step indices. Empty means all. |
|
||||
| `FASTVIDEO_H3_VSA_PROBE` | str | unset | debug | Output directory for the VSA-H3 attention-mass probe, which writes one .pt file per step, layer, and rank. Keeps the model out of regional compile. |
|
||||
| `FASTVIDEO_LTX2_GEMMA_LOG` | str | `""` | debug | Log file for LTX-2 Gemma text-encoder hidden states, used by parity tests. Deprecated names: `LTX2_FASTVIDEO_GEMMA_LOG`. |
|
||||
| `FASTVIDEO_COSMOS25_LOG_KNOBS` | bool | `0` | debug | Log the Cosmos 2.5 latent-preparation conditioning inputs. |
|
||||
| `FASTVIDEO_CFG_GATE_STEP` | float | `1.0` | sampling | CFG gating fraction in [0, 1]. Steps before len(timesteps) \* X run the conditional and unconditional forwards; later steps reuse the cached difference. 1.0 disables gating. |
|
||||
| `FASTVIDEO_LTX2_USE_DISTILLED_SIGMAS` | bool | `1` | sampling | LTX-2 uses the distilled sigma schedule when pipeline.model.ltx2.use_distilled_sigmas is also true. Deprecated names: `LTX2_USE_DISTILLED_SIGMAS`. |
|
||||
| `FASTVIDEO_EVAL_CACHE` | path | computed | eval | Cache directory for evaluation models and datasets. Defaults to $FASTVIDEO_CACHE_ROOT/eval. |
|
||||
| `FASTVIDEO_PHYSICS_IQ_BUCKET_URL` | str | `https://storage.googleapis.com/physics-iq-benchmark` | eval | Base URL of the Physics-IQ benchmark bucket. |
|
||||
| `FASTVIDEO_VBENCH_FULL_INFO_JSON` | str | unset | eval | Path to VBench_full_info.json, used instead of the vendored copy. Deprecated names: `VBENCH_FULL_INFO_JSON`. |
|
||||
| `FASTVIDEO_FVD_REF_FEATURES` | str | unset | eval | Cached reference-feature file for the FVD metric. |
|
||||
| `FASTVIDEO_FAD_REF_FEATURES` | str | unset | eval | Cached reference-feature file for the audio Frechet distance metric. |
|
||||
| `FASTVIDEO_TEST_LTX2_OVERFIT_DATA_DIR` | str | `data/cats` | test | Raw data directory for preprocess_ltx2_overfit.py. Deprecated names: `LTX2_OVERFIT_DATA_DIR`. |
|
||||
| `FASTVIDEO_TEST_LTX2_OVERFIT_CAPTION_JSON` | str | `videos2caption_1_sample.json` | test | Caption file, relative to the raw data directory. Deprecated names: `LTX2_OVERFIT_CAPTION_JSON`. |
|
||||
| `FASTVIDEO_TEST_LTX2_OVERFIT_VIDEO_SUBDIR` | str | `video` | test | Video subdirectory, relative to the raw data directory. Deprecated names: `LTX2_OVERFIT_VIDEO_SUBDIR`. |
|
||||
| `FASTVIDEO_TEST_LTX2_OVERFIT_OUTPUT_DIR` | str | `data/ltx2_overfit_preprocessed` | test | Output directory for preprocess_ltx2_overfit.py. Deprecated names: `LTX2_OVERFIT_OUTPUT_DIR`. |
|
||||
| `FASTVIDEO_TEST_LTX2_OVERFIT_MODEL` | str | `FastVideo/LTX2-Distilled-Diffusers` | test | Model repository whose encoders preprocess_ltx2_overfit.py uses. Deprecated names: `LTX2_OVERFIT_MODEL`. |
|
||||
| `FASTVIDEO_TEST_LTX2_OVERFIT_NUM_COPIES` | int | `4` | test | Number of copies of the overfit sample in the parquet file. Deprecated names: `LTX2_OVERFIT_NUM_COPIES`. |
|
||||
| `FASTVIDEO_TEST_KANDINSKY5_OVERFIT_DATA_DIR` | str | `data/kandinsky5_overfit` | test | Raw data directory for preprocess_kandinsky5_overfit.py. Deprecated names: `KANDINSKY5_OVERFIT_DATA_DIR`. |
|
||||
| `FASTVIDEO_TEST_KANDINSKY5_OVERFIT_OUTPUT_DIR` | str | `data/kandinsky5_overfit_preprocessed` | test | Output directory for preprocess_kandinsky5_overfit.py. Deprecated names: `KANDINSKY5_OVERFIT_OUTPUT_DIR`. |
|
||||
| `FASTVIDEO_TEST_SSIM_REFERENCE_HF_REPO` | str | `FastVideo/ssim-reference-videos` | test | Hugging Face repository that holds the SSIM reference videos. Deprecated names: `FASTVIDEO_SSIM_REFERENCE_HF_REPO`. |
|
||||
| `FASTVIDEO_TEST_SSIM_REFERENCE_HF_REPO_TYPE` | str | `dataset` | test | Repository type of FASTVIDEO_TEST_SSIM_REFERENCE_HF_REPO. Deprecated names: `FASTVIDEO_SSIM_REFERENCE_HF_REPO_TYPE`. |
|
||||
| `FASTVIDEO_TEST_SSIM_SKIP_REFERENCE_DOWNLOAD` | bool | `0` | test | SSIM tests use local reference videos without downloading. Deprecated names: `FASTVIDEO_SSIM_SKIP_REFERENCE_DOWNLOAD`. |
|
||||
| `FASTVIDEO_TEST_SSIM_FULL_QUALITY` | bool | `0` | test | SSIM tests use the full-quality sampling configurations. Deprecated names: `FASTVIDEO_SSIM_FULL_QUALITY`. |
|
||||
| `FASTVIDEO_TEST_NIGHTLY` | bool | `0` | test | Run the nightly end-to-end overfit tests. Deprecated names: `FASTVIDEO_NIGHTLY`. |
|
||||
| `FASTVIDEO_TEST_ULYSSES_FAULT_RANK` | str | unset | test | Rank that fails in the Ulysses fault-injection test. The test sets it for its worker processes. Deprecated names: `FASTVIDEO_ULYSSES_FAULT_RANK`. |
|
||||
| `FASTVIDEO_TEST_ULYSSES_FAULT_STAGE` | str | unset | test | Stage that fails in the Ulysses fault-injection test. The test sets it for its worker processes. Deprecated names: `FASTVIDEO_ULYSSES_FAULT_STAGE`. |
|
||||
| `FASTVIDEO_TEST_GOLDEN_GATE_DIR` | str | unset | test | Local directory of golden-gate reference tensors. Deprecated names: `FASTVIDEO_GOLDEN_GATE_DIR`. |
|
||||
| `FASTVIDEO_TEST_WAN22_5B_ALLOW_LOW_MEMORY` | bool | `0` | test | Run the MLX Wan2.2 5B real-weights parity test on hosts with little memory. Deprecated names: `FASTVIDEO_WAN22_5B_ALLOW_LOW_MEMORY`. |
|
||||
| `FASTVIDEO_TEST_WAN22_5B_ROOT` | str | unset | test | Local Wan2.2 5B checkpoint for the MLX real-weights parity test. Deprecated names: `FASTVIDEO_WAN22_5B_ROOT`. |
|
||||
| `FASTVIDEO_TEST_GRADNORM_UPDATE` | bool | `0` | test | Gradient-norm regression tests update their references. Deprecated names: `FASTVIDEO_GRADNORM_UPDATE`. |
|
||||
| `FASTVIDEO_TEST_DREAMX_WORLD_SSIM_MODEL_PATH` | str | `FastVideo/DreamX-World-5B-Cam-Diffusers` | test | Model for the DreamX-World camera SSIM test. Deprecated names: `DREAMX_WORLD_SSIM_MODEL_PATH`. |
|
||||
| `FASTVIDEO_TEST_DREAMX_WORLD_AR_SSIM_MODEL_PATH` | str | `FastVideo/DreamX-World-5B-Diffusers` | test | Model for the DreamX-World autoregressive SSIM test. Deprecated names: `DREAMX_WORLD_AR_SSIM_MODEL_PATH`. |
|
||||
| `FASTVIDEO_TEST_FLUX_T2I_MODEL_DIR` | str | `black-forest-labs/FLUX.1-dev` | test | Model for the Flux text-to-image SSIM test. Deprecated names: `FLUX_T2I_MODEL_DIR`. |
|
||||
| `FASTVIDEO_TEST_FLUX_TRANSFORMER_PATH` | str | unset | test | Local Flux transformer for the Flux transformer test. Deprecated names: `FLUX_TRANSFORMER_PATH`. |
|
||||
| `FASTVIDEO_TEST_GAMECRAFT_MODEL_PATH` | str | `FastVideo/HunyuanGameCraft-Diffusers` | test | Model for the HunyuanGameCraft SSIM test. Deprecated names: `GAMECRAFT_MODEL_PATH`. |
|
||||
| `FASTVIDEO_TEST_GEN3C_MODEL_PATH` | str | `FastVideo/GEN3C-Cosmos-7B-Diffusers` | test | Model for the GEN3C SSIM test. Deprecated names: `GEN3C_MODEL_PATH`. |
|
||||
| `FASTVIDEO_TEST_GEN3C_IMAGE_PATH` | str | unset | test | Input image for the GEN3C SSIM test. Deprecated names: `GEN3C_TEST_IMAGE_PATH`. |
|
||||
| `FASTVIDEO_TEST_GLM_IMAGE_LOCAL_WEIGHTS_DIR` | str | unset | test | Local official GLM-Image weights for the GLM-Image SSIM test. Deprecated names: `GLM_IMAGE_LOCAL_WEIGHTS_DIR`. |
|
||||
| `FASTVIDEO_TEST_GLM_IMAGE_MODEL_DIR` | str | unset | test | Model for the GLM-Image SSIM test. Deprecated names: `GLM_IMAGE_MODEL_DIR`. |
|
||||
| `FASTVIDEO_TEST_KANDINSKY5_E2E_NUM_GPUS` | int | `1` | test | GPUs for the Kandinsky5 nightly end-to-end overfit test. Deprecated names: `KANDINSKY5_E2E_NUM_GPUS`. |
|
||||
| `FASTVIDEO_TEST_KANDINSKY5_E2E_WRITE_REFERENCE` | bool | `0` | test | The Kandinsky5 nightly end-to-end test writes a missing reference video. Deprecated names: `KANDINSKY5_E2E_WRITE_REFERENCE`. |
|
||||
| `FASTVIDEO_TEST_LONGCAT_MODEL_ROOT` | str | unset | test | Local LongCat-Video checkpoint for the golden-gate test. Deprecated names: `LONGCAT_MODEL_ROOT`. |
|
||||
| `FASTVIDEO_TEST_MINIMAX_H3_GATE_GOLDEN_DIR` | str | unset | test | Local directory of MiniMax-H3 golden-gate tensors. Deprecated names: `MINIMAX_H3_GATE_GOLDEN_DIR`. |
|
||||
| `FASTVIDEO_TEST_MINIMAX_H3_GATE_LAYER` | int | `0` | test | Transformer layer that the MiniMax-H3 golden-gate test checks. Deprecated names: `MINIMAX_H3_GATE_LAYER`. |
|
||||
| `FASTVIDEO_TEST_MINIMAX_H3_MODEL_ROOT` | str | unset | test | Local MiniMax-H3 checkpoint for the golden-gate test. Deprecated names: `MINIMAX_H3_MODEL_ROOT`. |
|
||||
| `FASTVIDEO_TEST_SD35_MODEL_DIR` | str | `stabilityai/stable-diffusion-3.5-medium` | test | Model for the Stable Diffusion 3.5 SSIM test. Deprecated names: `SD35_MODEL_DIR`. |
|
||||
| `FASTVIDEO_TEST_TAEH3_REFERENCE_DIR` | str | unset | test | Upstream taehv checkout for the MLX TAEH3 parity test. Deprecated names: `TAEH3_REFERENCE_DIR`. |
|
||||
| `FASTVIDEO_TEST_ZIMAGE_MODEL_DIR` | str | `Tongyi-MAI/Z-Image-Turbo` | test | Model for the Z-Image SSIM test. Deprecated names: `ZIMAGE_MODEL_DIR`. |
|
||||
| `FASTVIDEO_TEST_ZIMAGE_MODEL_REVISION` | str | `f332072aa78be7aecdf3ee76d5c247082da564a6` | test | Hugging Face revision of the Z-Image model for its SSIM test. Deprecated names: `ZIMAGE_MODEL_REVISION`. |
|
||||
|
||||
Variables that FastVideo no longer reads; setting one logs a warning:
|
||||
|
||||
| Deprecated variable | Reason |
|
||||
| ----------------------------------------- | ---------------- |
|
||||
| `FASTVIDEO_TARGET_DEVICE` | no code reads it |
|
||||
| `FASTVIDEO_USE_PRECOMPILED` | no code reads it |
|
||||
| `FASTVIDEO_RINGBUFFER_WARNING_INTERVAL` | no code reads it |
|
||||
| `FASTVIDEO_ENGINE_ITERATION_TIMEOUT_S` | no code reads it |
|
||||
| `FASTVIDEO_SERVER_DEV_MODE` | no code reads it |
|
||||
| `FASTVIDEO_TEST_DYNAMO_FULLGRAPH_CAPTURE` | no code reads it |
|
||||
| `FASTVIDEO_TRACE_FUNCTION` | no code reads it |
|
||||
<!-- END GENERATED ENV TABLE -->
|
||||
@@ -112,7 +112,7 @@ per-metric policy with direction, percent threshold, absolute threshold, and a
|
||||
|
||||
`test_inference_performance.py` temporarily sets `FASTVIDEO_STAGE_LOGGING=1`
|
||||
while it runs so pipeline stage execution times are available in
|
||||
`generate(...).logging_info`. Stage logs use pipeline-unique keys such as
|
||||
`generate_video(...).logging_info`. Stage logs use pipeline-unique keys such as
|
||||
`prompt_encoding_stage` so duplicate stage classes do not collide. For
|
||||
`PipelineStage` entries, shared component stage bases emit a stable
|
||||
`component_metric`: text encoding stages map to `text_encoder_time_s`,
|
||||
|
||||
@@ -5,7 +5,7 @@ hide:
|
||||
|
||||
# Cosmos recipes
|
||||
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="cosmos" data-recipes="../../assets/cookbook-recipes.json?v=11">
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="cosmos" data-recipes="../../assets/cookbook-recipes.json?v=9">
|
||||
<header class="cookbook-family-header">
|
||||
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
|
||||
<div class="cookbook-family-header__body">
|
||||
|
||||
@@ -5,7 +5,7 @@ hide:
|
||||
|
||||
# FLUX recipes
|
||||
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="flux" data-recipes="../../assets/cookbook-recipes.json?v=11">
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="flux" data-recipes="../../assets/cookbook-recipes.json?v=9">
|
||||
<header class="cookbook-family-header">
|
||||
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
|
||||
<div class="cookbook-family-header__body">
|
||||
|
||||
@@ -5,7 +5,7 @@ hide:
|
||||
|
||||
# GLM-Image recipes
|
||||
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="glm_image" data-recipes="../../assets/cookbook-recipes.json?v=11">
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="glm_image" data-recipes="../../assets/cookbook-recipes.json?v=9">
|
||||
<header class="cookbook-family-header">
|
||||
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
|
||||
<div class="cookbook-family-header__body">
|
||||
|
||||
@@ -5,7 +5,7 @@ hide:
|
||||
|
||||
# Hunyuan recipes
|
||||
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="hunyuan" data-recipes="../../assets/cookbook-recipes.json?v=11">
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="hunyuan" data-recipes="../../assets/cookbook-recipes.json?v=9">
|
||||
<header class="cookbook-family-header">
|
||||
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
|
||||
<div class="cookbook-family-header__body">
|
||||
|
||||
@@ -41,7 +41,7 @@ hide:
|
||||
<span class="cookbook-family-tile__footer">
|
||||
<span class="cookbook-family-tile__footer-top">
|
||||
<span><strong>MiniMax H3</strong><small>Video + stereo audio</small></span>
|
||||
<span class="cookbook-count">9 recipes</span>
|
||||
<span class="cookbook-count">8 recipes</span>
|
||||
</span>
|
||||
<ul class="cookbook-mode-row">
|
||||
<li>T2VA</li>
|
||||
|
||||
@@ -5,7 +5,7 @@ hide:
|
||||
|
||||
# Kandinsky 5 recipes
|
||||
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="kandinsky5" data-recipes="../../assets/cookbook-recipes.json?v=11">
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="kandinsky5" data-recipes="../../assets/cookbook-recipes.json?v=9">
|
||||
<header class="cookbook-family-header">
|
||||
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
|
||||
<div class="cookbook-family-header__body">
|
||||
|
||||
@@ -5,7 +5,7 @@ hide:
|
||||
|
||||
# LongCat recipes
|
||||
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="longcat" data-recipes="../../assets/cookbook-recipes.json?v=11">
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="longcat" data-recipes="../../assets/cookbook-recipes.json?v=9">
|
||||
<header class="cookbook-family-header">
|
||||
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
|
||||
<div class="cookbook-family-header__body">
|
||||
|
||||
@@ -5,7 +5,7 @@ hide:
|
||||
|
||||
# LTX recipes
|
||||
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="ltx2" data-recipes="../../assets/cookbook-recipes.json?v=11">
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="ltx2" data-recipes="../../assets/cookbook-recipes.json?v=9">
|
||||
<header class="cookbook-family-header">
|
||||
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
|
||||
<div class="cookbook-family-header__body">
|
||||
|
||||
@@ -5,7 +5,7 @@ hide:
|
||||
|
||||
# Matrix Game recipes
|
||||
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="matrixgame" data-recipes="../../assets/cookbook-recipes.json?v=11">
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="matrixgame" data-recipes="../../assets/cookbook-recipes.json?v=9">
|
||||
<header class="cookbook-family-header">
|
||||
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
|
||||
<div class="cookbook-family-header__body">
|
||||
|
||||
+15
-15
@@ -5,13 +5,13 @@ hide:
|
||||
|
||||
# MiniMax H3 recipes
|
||||
|
||||
FastH3 is two distilled MiniMax-H3 checkpoints. **V1** is the four-step
|
||||
launch. Some Hub repo names still say Preview. That name is historical. V1 is
|
||||
a full model, not a demo. **V2** is the eight-step checkpoint. More forwards
|
||||
is why V2 is the higher-quality FastH3. The V2 schedule contract is in
|
||||
The **FastH3 8-Step V2** recipe below runs the eight-forward
|
||||
[`FastVideo/FastVideo-FastH3-8-Step-V2`](https://huggingface.co/FastVideo/FastVideo-FastH3-8-Step-V2)
|
||||
checkpoint; its schedule contract is documented in
|
||||
[FastH3 distilled checkpoint schedules](../inference/fasth3-distilled.md).
|
||||
The four-forward FastH3 Preview recipes are unchanged.
|
||||
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="minimax_h3" data-default-recipe="fasth3-preview-cuda" data-recipes="../../assets/cookbook-recipes.json?v=11">
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="minimax_h3" data-default-recipe="fasth3-preview-cuda" data-recipes="../../assets/cookbook-recipes.json?v=9">
|
||||
<header class="cookbook-family-header">
|
||||
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
|
||||
<div class="cookbook-family-header__body">
|
||||
@@ -22,7 +22,7 @@ is why V2 is the higher-quality FastH3. The V2 schedule contract is in
|
||||
<p class="cookbook-eyebrow">Primary focus · Inference</p>
|
||||
<h2>MiniMax H3 recipes</h2>
|
||||
<p>Generate video and audio with H3. Run a server on CUDA, one DGX Spark, or Apple Silicon MLX to iterate on prompts, or call the pipeline directly from Python.</p>
|
||||
<span class="cookbook-count" data-cookbook-count>9 maintained recipes</span>
|
||||
<span class="cookbook-count" data-cookbook-count>8 maintained recipes</span>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cookbook-lifecycle" aria-label="Lifecycle stages">
|
||||
@@ -47,9 +47,9 @@ is why V2 is the higher-quality FastH3. The V2 schedule contract is in
|
||||
<h2 id="h3-modes-heading">Supported modes</h2>
|
||||
<p>
|
||||
CUDA covers T2VA, FL2VA, and Ref2VA on the full checkpoint, plus FastH3
|
||||
V1 and FastH3 V2. FastH3 V1 also has a DGX Spark runtime with
|
||||
a 1-Spark or 2-Spark device row. MLX is T2VA only: V1 and V2.
|
||||
Temporal <code>--fast</code>, spatial <code>--fast-spatial</code>, and opt-in VSA are flags on the same
|
||||
Preview and FastH3 LoRA. FastH3 Preview also has a DGX Spark runtime with
|
||||
a 1-Spark or 2-Spark device row. MLX is T2VA only. Temporal <code>--fast</code>,
|
||||
spatial <code>--fast-spatial</code>, and opt-in VSA are flags on the same
|
||||
MLX script, not extra recipes.
|
||||
</p>
|
||||
<div class="cookbook-modes__table-wrap">
|
||||
@@ -64,8 +64,8 @@ is why V2 is the higher-quality FastH3. The V2 schedule contract is in
|
||||
<tbody>
|
||||
<tr>
|
||||
<td>T2VA</td>
|
||||
<td>Full H3, FastH3 V1, FastH3 LoRA, FastH3 V2</td>
|
||||
<td>FastH3 V1 or FastH3 V2 after a local DiT conversion</td>
|
||||
<td>Full H3, FastH3 Preview, FastH3 LoRA</td>
|
||||
<td>FastH3 Preview after a local DiT conversion</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>FL2VA</td>
|
||||
@@ -99,7 +99,7 @@ is why V2 is the higher-quality FastH3. The V2 schedule contract is in
|
||||
</tr>
|
||||
<tr>
|
||||
<td>DGX Spark</td>
|
||||
<td>FastH3 V1 on one GB10, or two Sparks with Ray sequence parallel (<code>sp_size=2</code>) over QSFP RoCE. Select NVIDIA DGX Spark, then 1 Spark or 2 Sparks.</td>
|
||||
<td>FastH3 Preview on one GB10, or two Sparks with Ray sequence parallel (<code>sp_size=2</code>) over QSFP RoCE. Select NVIDIA DGX Spark, then 1 Spark or 2 Sparks.</td>
|
||||
<td>Not wired</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
@@ -257,7 +257,7 @@ cd FastVideo</code></pre>
|
||||
<pre><code>UV_TORCH_BACKEND=cu130 uv pip install -e ".[fasth3]"</code></pre>
|
||||
<p class="cookbook-eyebrow">Apple Silicon</p>
|
||||
<pre><code>uv pip install -e ".[mlx]"</code></pre>
|
||||
<p>Follow the <a href="../../getting_started/installation/mlx/">MLX install guide</a> for the extra, <code>ffmpeg</code>, and a clone. Then pick FastH3 V1 or V2 in the builder above.</p>
|
||||
<p>Follow the <a href="../../getting_started/installation/mps/#run-fasth3-preview">Apple Silicon guide</a> for the download, conversion, and storage requirements.</p>
|
||||
<p class="cookbook-eyebrow">NVIDIA DGX Spark</p>
|
||||
<pre><code>UV_TORCH_BACKEND=cu130 uv pip install -e .</code></pre>
|
||||
<p>Follow the <a href="../../getting_started/installation/spark/">DGX Spark install guide</a> for ARM64 CUDA 13. One Spark is a local process. Two Sparks need Ray on the QSFP link:</p>
|
||||
@@ -272,9 +272,9 @@ cd FastVideo</code></pre>
|
||||
<ul>
|
||||
<li>The full CUDA H3 examples request four GPUs by default. Their sources do not claim a GPU model or memory minimum.</li>
|
||||
<li>The FastH3 CUDA performance profile was measured on four GB200 GPUs. Use its strict profile when exact operation order matters more than the measured performance configuration.</li>
|
||||
<li>The MLX source runtime supports T2VA, optional temporal <code>--fast</code>, optional spatial <code>--fast-spatial</code>, and opt-in VSA on <code>--include-vsa</code> checkpoints. FastH3 V2 MLX converts with <code>--include-vsa</code> and runs eight forwards. FL2VA, Ref2VA, and two-pass refinement are not wired.</li>
|
||||
<li>The MLX source runtime supports T2VA, optional temporal <code>--fast</code>, optional spatial <code>--fast-spatial</code>, and opt-in VSA on <code>--include-vsa</code> checkpoints. FL2VA, Ref2VA, and two-pass refinement are not wired.</li>
|
||||
<li>GPU count and VAE decode backend are configurable in the builder above for FastH3 CUDA recipes. Only the value shown by default has a recorded run; other supported values are unmeasured here.</li>
|
||||
<li>DGX Spark is a runtime on FastH3 V1, not a separate family card. Select NVIDIA DGX Spark, then 1 Spark or 2 Sparks. The CUDA GPU-count knob does not apply to Spark.</li>
|
||||
<li>DGX Spark is a runtime on FastH3 Preview, not a separate family card. Select NVIDIA DGX Spark, then 1 Spark or 2 Sparks. The CUDA GPU-count knob does not apply to Spark.</li>
|
||||
<li>GB10 has no FA4 / sm_100a VSA kernel. Keep <code>FASTVIDEO_FA4=0</code> and <code>FASTVIDEO_VSA_SM100A=0</code>. Legal <code>num_frames</code> values are <code>17n+5</code>, capped at 362 (15.08 s). A 345-frame request on one Spark can OOM.</li>
|
||||
<li>Gated or missing checkpoints: run <code>huggingface-cli login</code> and confirm you accepted the model's license on Hugging Face.</li>
|
||||
</ul>
|
||||
|
||||
@@ -5,7 +5,7 @@ hide:
|
||||
|
||||
# MMAudio recipes
|
||||
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="mmaudio" data-recipes="../../assets/cookbook-recipes.json?v=11">
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="mmaudio" data-recipes="../../assets/cookbook-recipes.json?v=9">
|
||||
<header class="cookbook-family-header">
|
||||
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
|
||||
<div class="cookbook-family-header__body">
|
||||
|
||||
@@ -8,9 +8,8 @@ JavaScript client examples use that interface, but requests go to FastVideo.
|
||||
You do not need an OpenAI account or cloud key.
|
||||
|
||||
The [H3 recipe selector](minimax-h3.md) provides the same workflow with runtime
|
||||
selection. This guide covers FastH3 V1 (four forwards) and FastH3 V2
|
||||
(eight forwards). Start a server, then iterate in the playground or with the
|
||||
OpenAI Python client. Other H3 recipes keep their direct Python commands.
|
||||
selection. This guide covers FastH3 Preview text-to-video/audio. Other H3
|
||||
recipes keep their direct Python commands.
|
||||
|
||||
CUDA requests reuse one loaded `VideoGenerator`. The Python SDK can do the same
|
||||
when you reuse the generator across `generate()` calls. MLX keeps one
|
||||
@@ -37,14 +36,6 @@ advertises it as `fasth3`. It configures four CUDA GPUs but does not record
|
||||
a GPU model or VRAM requirement. This is a source-backed server profile, not
|
||||
the measured GB200 Python performance profile. Compilation is disabled.
|
||||
|
||||
For FastH3 V2, use the same install and the 8-step config (nine sigma
|
||||
points, eight DiT forwards, VSA sparsity 0.8):
|
||||
|
||||
```bash
|
||||
UV_TORCH_BACKEND=cu130 uv pip install -e ".[fasth3]"
|
||||
fastvideo serve --config examples/serving/openai_fasth3_8step.yaml --server.host 127.0.0.1
|
||||
```
|
||||
|
||||
Keep the server running. In another terminal, check readiness:
|
||||
|
||||
```bash
|
||||
@@ -81,7 +72,7 @@ After model loading completes, the response is `{"status":"ok"}`.
|
||||
|
||||
### Apple Silicon MLX
|
||||
|
||||
Complete the [MLX install](../getting_started/installation/mlx.md),
|
||||
Complete the [Apple Silicon installation](../getting_started/installation/mps.md#run-fasth3-preview),
|
||||
including `ffmpeg`. From your FastVideo clone, install the MLX extra:
|
||||
|
||||
```bash
|
||||
@@ -120,22 +111,6 @@ The MLX server has no recorded device or unified-memory requirement. The
|
||||
direct Python recipe's M4 Max measurements are not a server benchmark or a
|
||||
minimum-memory claim.
|
||||
|
||||
For FastH3 V2, convert that checkpoint's transformer with `--include-vsa`
|
||||
and start the 8-step config. Do not overwrite a V1 export. The
|
||||
[MiniMax H3 cookbook](minimax-h3.md) has the same download and convert
|
||||
commands as the Python recipe.
|
||||
|
||||
```bash
|
||||
hf download FastVideo/FastVideo-FastH3-8-Step-V2 --local-dir ./FastH3-8-Step-V2
|
||||
python scripts/checkpoint_conversion/convert_minimax_h3_mlx.py --model-root ./FastH3-8-Step-V2/transformer --out ./FastH3-8-Step-V2-MLX --formats "int8" --include-vsa
|
||||
python -m fastvideo.entrypoints.openai.mlx_server --config examples/serving/mlx_fasth3_8step.yaml
|
||||
```
|
||||
|
||||
That adapter passes `num_steps=8` when the HTTP field is `num_inference_steps=9`,
|
||||
and it enables the trained VSA recipe (sparsity 0.8, 64-token tiles). Reuse the
|
||||
preview VAE, audio VAE, text encoder, and tokenizer if those directories already
|
||||
exist; edit the YAML paths if your files live elsewhere.
|
||||
|
||||
## Open the playground
|
||||
|
||||
Open [the local H3 playground](http://127.0.0.1:8000/playground/) after startup.
|
||||
@@ -161,11 +136,9 @@ or manage a GPU server for you.
|
||||
## Generate with cURL or an SDK
|
||||
|
||||
These examples use the server's resolution, frame count, and sampling defaults.
|
||||
Do not copy Sora-specific durations or resolutions onto H3. V1 configs use
|
||||
124 frames, 24 fps, and the five-point distilled sigma schedule with four DiT
|
||||
forwards. V2 configs use nine sigma points and eight DiT forwards. CUDA
|
||||
and one Spark use 1344 × 768; MLX uses 832 × 480. The OpenAI Python client is
|
||||
the same for every FastH3 server that advertises `fasth3`.
|
||||
Do not copy Sora-specific durations or resolutions onto H3. Both server configs
|
||||
use 124 frames, 24 fps, and the five-point distilled sigma schedule with four
|
||||
DiT forwards. CUDA and one Spark use 1344 × 768; MLX uses 832 × 480.
|
||||
|
||||
Each client submits a job, checks for completion or failure, and downloads an
|
||||
MP4 named after the job ID. Polling stops after 30 minutes; a timeout does not
|
||||
|
||||
@@ -5,7 +5,7 @@ hide:
|
||||
|
||||
# Stable Audio recipes
|
||||
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="stable_audio" data-recipes="../../assets/cookbook-recipes.json?v=11">
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="stable_audio" data-recipes="../../assets/cookbook-recipes.json?v=9">
|
||||
<header class="cookbook-family-header">
|
||||
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
|
||||
<div class="cookbook-family-header__body">
|
||||
|
||||
@@ -5,7 +5,7 @@ hide:
|
||||
|
||||
# Stable Diffusion recipes
|
||||
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="sd35" data-recipes="../../assets/cookbook-recipes.json?v=11">
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="sd35" data-recipes="../../assets/cookbook-recipes.json?v=9">
|
||||
<header class="cookbook-family-header">
|
||||
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
|
||||
<div class="cookbook-family-header__body">
|
||||
|
||||
@@ -5,7 +5,7 @@ hide:
|
||||
|
||||
# TurboDiffusion recipes
|
||||
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="turbodiffusion" data-recipes="../../assets/cookbook-recipes.json?v=11">
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="turbodiffusion" data-recipes="../../assets/cookbook-recipes.json?v=9">
|
||||
<header class="cookbook-family-header">
|
||||
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
|
||||
<div class="cookbook-family-header__body">
|
||||
|
||||
@@ -5,7 +5,7 @@ hide:
|
||||
|
||||
# Wan recipes
|
||||
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="wan" data-recipes="../../assets/cookbook-recipes.json?v=11">
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="wan" data-recipes="../../assets/cookbook-recipes.json?v=9">
|
||||
<header class="cookbook-family-header">
|
||||
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
|
||||
<div class="cookbook-family-header__body">
|
||||
@@ -186,7 +186,7 @@ cd FastVideo</code></pre>
|
||||
<ul>
|
||||
<li>Out of memory on the A14B recipes: the checked-in sources already enable CPU offload; see <a href="../../inference/configuration/">Configuration</a> for the offload surface before reducing resolution or frames.</li>
|
||||
<li>The FastWan2.1 recipe requires <code>VIDEO_SPARSE_ATTN</code>; confirm the environment variable in the command was set in the same shell.</li>
|
||||
<li>FastMetal MLX: install with the <a href="../../getting_started/installation/mlx/">MLX install guide</a>, then pick a FastMetal recipe in the builder. CUDA FastWan-QAD checkpoints are refused on the MLX runtime.</li>
|
||||
<li>FastMetal MLX: install with <code>uv pip install -e ".[mlx]"</code>, then follow the <a href="../../getting_started/installation/mps/">Apple Silicon guide</a>. CUDA FastWan-QAD checkpoints are refused on the MLX runtime.</li>
|
||||
<li>FastMetal 5B uses <code>mlx_wan22_generate.py</code>. 1.3B and 14B use <code>mlx_wan_prompt_to_video.py</code>.</li>
|
||||
<li>Gated or missing checkpoints: run <code>huggingface-cli login</code> and confirm you accepted the model's license on Hugging Face.</li>
|
||||
</ul>
|
||||
|
||||
@@ -5,7 +5,7 @@ hide:
|
||||
|
||||
# Z-Image recipes
|
||||
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="zimage" data-recipes="../../assets/cookbook-recipes.json?v=11">
|
||||
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="zimage" data-recipes="../../assets/cookbook-recipes.json?v=9">
|
||||
<header class="cookbook-family-header">
|
||||
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
|
||||
<div class="cookbook-family-header__body">
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user