Compare commits

..
Author SHA1 Message Date
0c16ec91b0 [feat]: connect Dreamverse creation settings to generation
Apply the backend-wiring changes beyond the UI uplift to
ds8/dreamversev2-dev for review and refactoring.

Source PR: hao-ai-lab/FastVideo#1854
Source range: 90d739a91892302edf37c4b23f807f402c83071d..8c5ee9b51cc3b75f4eba9cb3904fe08578c9dc9e
The 28-file patch is identical to that source range.

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Aryan Kumar <aryan5v@users.noreply.github.com>
2026-09-15 17:16:15 -07:00
e57543b79d [feat]: Dreamverse creation studio UI uplift (#1853)
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Aryan Kumar <aryan5v@users.noreply.github.com>
2026-09-15 16:45:08 -07:00
276 changed files with 8258 additions and 11474 deletions
@@ -1,76 +0,0 @@
---
name: env-var-conventions
description: Add, read, rename, or remove an environment variable in FastVideo, or change the environment-variable policy. Use before touching fastvideo/envs.py, os.environ, os.getenv, or monkeypatch.setenv in fastvideo/, and when fastvideo/tests/contract/test_env_policy.py fails.
---
# Environment Variable Conventions
## Purpose
FastVideo registers its environment variables as typed fields in
`fastvideo/envs.py`. The policy that governs them is
`docs/contributing/env_vars.md`, and the contract test
`fastvideo/tests/contract/test_env_policy.py` enforces the policy in the unit
CI lane. This skill routes an environment-variable change through that policy.
The policy doc is the single source of the rules; read it instead of relying
on a summary here.
## Prerequisites
- Read `docs/contributing/env_vars.md` in full.
- Decide whether the setting belongs in an environment variable or an argument
(rule 5 in the policy doc). Settings that users change per deployment are
arguments; add them through `fastvideo/fastvideo_args.py` instead.
## Inputs
| Parameter | Required | Description |
| ---------- | -------- | -------------------------------------------------------------- |
| `change` | Yes | Add, read, rename, or remove a variable, or change the policy. |
| `variable` | Yes | The variable name, with the `FASTVIDEO_` prefix. |
## Steps
1. **Declare or edit the variable in `fastvideo/envs.py`.**
- Pick the field type and category that the policy doc lists.
- Write a description that states what the variable does and its units.
- To rename, keep the old name in `deprecated_names`. To remove, add the
name to `DEPRECATED_VARIABLES`. Update the uses in `examples/`,
`scripts/`, `docs/`, `apps/`, and the tests.
2. **Read the variable with `envs.NAME.get()` inside a function.**
- In tests, change the value with `envs.NAME.override(value)`.
- Do not call `os.environ`, `os.getenv`, or `monkeypatch.setenv` for a
FastVideo variable.
- To set a variable that another tool reads, call `envs.set_external`,
`envs.setdefault_external`, or `envs.unset_external`.
3. **Regenerate the table in the policy doc.**
- Run `python fastvideo/tests/contract/test_env_policy.py`.
4. **Run the contract test.**
- Run `pytest fastvideo/tests/contract/test_env_policy.py`.
- When the test reports a fixed known violation, delete or lower its entry
in `KNOWN_VIOLATIONS`. Never add an entry to `KNOWN_VIOLATIONS`.
5. **When the policy itself changes, update the policy doc and the contract
test in the same pull request.**
- The rules in `docs/contributing/env_vars.md`, the checks and allowlist in
`fastvideo/tests/contract/test_env_policy.py`, and this skill must agree.
## Outputs
- A registry entry in `fastvideo/envs.py` and call sites that use
`envs.NAME.get()`.
- A regenerated table in `docs/contributing/env_vars.md`.
- A passing `fastvideo/tests/contract/test_env_policy.py`.
## Example Usage
```
Add a FASTVIDEO_DEBUG_MY_STAGE switch that logs MyStage inputs.
```
## References
- `docs/contributing/env_vars.md`: the policy, the field types, and the
violation kinds that the contract test reports.
- `fastvideo/envs.py`: the registry.
- `fastvideo/tests/contract/test_env_policy.py`: the contract test,
`EXTERNAL_ALLOWLIST`, and `KNOWN_VIOLATIONS`.
-6
View File
@@ -22,12 +22,6 @@ else
export PERF_UPLOAD_POLICY=never
fi
# Alternate GPU backends compare against references without publishing records.
# Their worker has read-only Hub credentials; publication is an operator task.
if [ "${FASTVIDEO_CI_LOCAL_ONLY:-0}" = 1 ]; then
export PERF_UPLOAD_POLICY=never
fi
nvidia-smi \
--query-gpu=index,timestamp,clocks.sm,clocks.max.sm,power.draw,power.limit,temperature.gpu \
--format=csv -l 10 > "$PERF_REPORTS_DIR/gpu_telemetry.csv" 2>/dev/null &
-1
View File
@@ -16,7 +16,6 @@ exec pytest \
./fastvideo/tests/worker/ \
./fastvideo/tests/training/test_trackers.py \
./fastvideo/tests/attention/test_sdpa_metadata_mask_contract.py \
./fastvideo/tests/attention/test_vsa_h3_tile_grad_safety.py \
./fastvideo/tests/modal/test_kernel_build_cache.py \
./fastvideo/tests/modal/test_pr_test.py \
./fastvideo/tests/modal/test_ssim_test.py \
@@ -11,7 +11,6 @@ jobs:
if: >-
github.event.context == 'direct-test-completed'
&& github.event.state == 'success'
&& (vars.CI_GPU_BACKEND == '' || vars.CI_GPU_BACKEND == 'slurm')
runs-on: ubuntu-latest
steps:
- name: Check and update aggregate status
@@ -1,62 +0,0 @@
name: Promote Selected GPU Backend Status
on:
status:
permissions:
statuses: write
concurrency:
group: gpu-ci-status-${{ github.event.sha }}-${{ vars.CI_GPU_BACKEND }}
cancel-in-progress: false
jobs:
promote:
if: >-
(vars.CI_GPU_BACKEND == 'modal' || vars.CI_GPU_BACKEND == 'vllm')
&& (github.event.context == format('gpu-ci/{0}/fastcheck-passed', vars.CI_GPU_BACKEND)
|| github.event.context == format('gpu-ci/{0}/full-suite-passed', vars.CI_GPU_BACKEND))
runs-on: ubuntu-latest
env:
SELECTED_BACKEND: ${{ vars.CI_GPU_BACKEND }}
steps:
- name: Mirror the selected backend's latest suite results
uses: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea # v7.0.1
with:
script: |
const backend = process.env.SELECTED_BACKEND;
if (!['modal', 'vllm'].includes(backend)) {
throw new Error('Unsupported selected GPU backend');
}
const sha = context.payload.sha;
// Read current state after entering the serialized workflow. A
// delayed event must not overwrite a newer failure with success.
const statuses = await github.paginate(github.rest.repos.listCommitStatusesForRef, {
owner: context.repo.owner,
repo: context.repo.repo,
ref: sha,
per_page: 100,
});
for (const suffix of ['fastcheck-passed', 'full-suite-passed']) {
const sourceContext = `gpu-ci/${backend}/${suffix}`;
const matches = statuses.filter(status => status.context === sourceContext);
matches.sort((a, b) =>
Date.parse(b.updated_at) - Date.parse(a.updated_at) || b.id - a.id
);
const latest = matches[0];
const state = latest ? latest.state : 'pending';
if (!['pending', 'success', 'failure', 'error'].includes(state)) {
throw new Error(`Unsupported status state for ${sourceContext}`);
}
await github.rest.repos.createCommitStatus({
owner: context.repo.owner,
repo: context.repo.repo,
sha,
context: suffix,
state,
description: latest
? `${backend} ${suffix}: ${state}`
: `Waiting for ${backend} ${suffix}`,
...(latest && latest.target_url ? {target_url: latest.target_url} : {}),
});
}
-2
View File
@@ -92,7 +92,6 @@ jobs:
fastvideo/tests/mlx/test_frame_upsample.py \
fastvideo/tests/mlx/test_mlx_fast_spatial.py \
fastvideo/tests/mlx/test_mlx_refine.py \
fastvideo/tests/mlx/test_mlx_prompt_enhance.py \
fastvideo/tests/mlx/test_mlx_prompt_to_video_decode.py \
fastvideo/tests/mlx/test_mlx_wan22_prompt_cache_fingerprint.py \
fastvideo/tests/mlx/test_wan22_sample.py \
@@ -157,7 +156,6 @@ jobs:
fastvideo/tests/mlx/test_frame_upsample.py \
fastvideo/tests/mlx/test_mlx_fast_spatial.py \
fastvideo/tests/mlx/test_mlx_refine.py \
fastvideo/tests/mlx/test_mlx_prompt_enhance.py \
fastvideo/tests/mlx/test_mlx_prompt_to_video_decode.py \
fastvideo/tests/mlx/test_mlx_wan22_prompt_cache_fingerprint.py \
fastvideo/tests/mlx/test_wan22_sample.py \
+2 -3
View File
@@ -203,8 +203,7 @@ jobs:
docker buildx imagetools create "${TAG_ARGS[@]}" "${IMAGE_REFS[@]}"
docker buildx imagetools inspect "${TAGS[0]}"
# The CI runner is ARM64 like DGX Spark, but targets sm_100a rather than sm_121.
# The architecture-specific target includes the GB200 VSA CUDA extensions.
# The CI runner is ARM64 like DGX Spark, but targets sm_100 rather than sm_121.
# Publish a single-architecture variant so the self-hosted CI runner can reuse
# the exact prebuilt kernel instead of compiling it in every job.
build-ci-runner-image:
@@ -220,7 +219,7 @@ jobs:
PYTHON_VERSION=3.12
CUDA_VERSION=13.0.0
UV_TORCH_BACKEND=cu130
TORCH_CUDA_ARCH_LIST=10.0a
TORCH_CUDA_ARCH_LIST=10.0
CMAKE_BUILD_PARALLEL_LEVEL=1
FLASH_ATTN_WHEEL_TAG=cu130torch2.12
FLASH_ATTN_WHEEL_RELEASE_ARM64=https://github.com/mjun0812/flash-attention-prebuild-wheels/releases/download/v0.9.22
+8 -7
View File
@@ -12,7 +12,7 @@
- `2026/09/15`: Release [FastH3 8-Step V2](https://huggingface.co/FastVideo/FastVideo-FastH3-8-Step-V2), an eight-forward data-free DMD2 checkpoint distilled from MiniMax-H3 with 80% Video Sparse Attention. Run it with `examples/inference/basic/basic_fasth3_8step.py` or the [FastH3 8-Step V2 recipe](https://haoailab.com/FastVideo/cookbook/minimax-h3/).
- `2026/09/01`: FastH3 now runs locally on Apple Silicon through MLX and on NVIDIA DGX Spark through CUDA 13, including two-Spark inference. Follow the [FastH3 recipes](https://haoailab.com/FastVideo/cookbook/minimax-h3/) and read the [Blog](https://haoailab.com/blogs/fasth3-local/).
- `2026/08/27`: [FastH3 Preview v1](https://haoailab.com/blogs/fasth3-preview/) is an open-weight 4-step sparse-distilled MiniMax-H3 model for synchronized video-and-audio generation, developed in collaboration with [Nuva Lab](https://nuvalab.ai/) and the [NVIDIA FastGen team](https://github.com/NVlabs/FastGen). Download the recommended [VSA / Data-Free weights](https://huggingface.co/FastVideo/FastVideo-FastH3-4-step-Preview-v1-VSA-DataFree), or see the [full FastH3 collection](https://huggingface.co/collections/FastVideo/fastvideo-fasth3).
- `2026/08/19`: FastVideo now supports MLX on Apple Silicon with [FastMetal-QAD](https://huggingface.co/collections/FastVideo/fastmetal), a family of 1.3B, 5B, and 14B models optimized for Mac. Follow the [MLX install guide](https://hao-ai-lab.github.io/FastVideo/getting_started/installation/mlx/) and read the [Blog](https://haoailab.com/blogs/fastmetal/).
- `2026/08/19`: FastVideo now supports MLX on Apple Silicon with [FastMetal-QAD](https://huggingface.co/collections/FastVideo/fastmetal), a family of 1.3B, 5B, and 14B models optimized for Mac—follow the [Apple Silicon guide](https://hao-ai-lab.github.io/FastVideo/getting_started/installation/mps/) and read the [Blog](https://haoailab.com/blogs/fastmetal/).
- `2026/06/23`: Release FastWan-QAD: 5s of Video generated in 1.8s E2E. See the [FastWan-QAD models](https://huggingface.co/FastVideo/FastWan-QAD-FP8-1.3B), [Attn-QAT training guide](https://haoailab.com/FastVideo/training/attn_qat/), and [blog](https://haoailab.com/blogs/fastwan-qad/).
- `2026/03/17`: Release demo: Into the Dreamverse: Vibe Directing in FastVideo, check out the [Blog](https://haoailab.com/blogs/dreamverse/).
- `2026/03/13`: Release demo: Create a 5s 1080p Video in 4.5s with FastVideo on a Single GPU, check out the [Blog](https://haoailab.com/blogs/fastvideo_realtime_1080p/).
@@ -64,12 +64,13 @@ UV_TORCH_BACKEND=cu126 uv pip install fastvideo
```
Use `UV_TORCH_BACKEND=cu130` on CUDA 13. Apple silicon users should follow the
[MLX install guide](https://hao-ai-lab.github.io/FastVideo/getting_started/installation/mlx/).
[MPS installation guide](https://hao-ai-lab.github.io/FastVideo/getting_started/installation/mps/).
> **On an Apple Silicon Mac?** Install with `uv pip install -e '.[mlx]'` from
> a clone, then pick a recipe in the
> [cookbook](https://haoailab.com/FastVideo/cookbook/). See the
> [MLX install guide](https://hao-ai-lab.github.io/FastVideo/getting_started/installation/mlx/).
> **On an Apple Silicon Mac?** FastVideo runs FastMetal-QAD through an MLX
> runtime. Install with `uv pip install -e '.[mlx]'`, download
> [`FastVideo/FastMetal-1.3B-QAD`](https://huggingface.co/FastVideo/FastMetal-1.3B-QAD),
> and follow the
> [Apple Silicon guide](https://hao-ai-lab.github.io/FastVideo/getting_started/installation/mps/).
Please see our [docs](https://hao-ai-lab.github.io/FastVideo/getting_started/installation/) for more detailed installation instructions.
@@ -87,7 +88,7 @@ Install FastVideo (https://github.com/hao-ai-lab/FastVideo) into a fresh uv virt
https://hao-ai-lab.github.io/FastVideo/getting_started/installation/):
- NVIDIA GPU, x86_64 -> docs/getting_started/installation/gpu.md
- NVIDIA DGX Spark / GB10, aarch64, CUDA 13 -> docs/getting_started/installation/spark.md
- Apple Silicon, macOS -> docs/getting_started/installation/mlx.md
- Apple Silicon, macOS -> docs/getting_started/installation/mps.md
3. Use uv for every step. If a command fails, debug it and tell me what you changed.
4. Verify the result:
python -c "import fastvideo, torch; print('cuda', torch.cuda.is_available())"
+32 -3
View File
@@ -299,18 +299,47 @@ There are three related prompt paths in the current system:
## Initial Image And Segment Handling
The frontend currently sends `initial_image` as part of session init or
The frontend sends `initial_image` and, for first/last frame mode,
`last_frame_image` as part of `session_init_v2`, `project_init_v1`, or
`simple_generate`.
The server:
- validates and persists the image
- uses it only for segment 1 when present
- validates and persists the images
- uses `initial_image` only for segment 1 when present
- keeps continuation state for later segments in the GPU worker
This means the runtime, not the frontend, decides how segment 1 image
conditioning and later continuation conditioning are applied.
## Creation Studio Config
The lobby creation studio sends model, mode, aspect ratio, resolution, and
duration with session init. The server parses these fields into a per-session
creation config and echoes the resolved values back on `gpu_assigned` and
`ltx2_stream_start` as `creation_config`.
Incoming fields on `session_init_v2` and `project_init_v1`:
- `generation_mode`: `t2va`, `fl2va`, or `ref2va` (canonical upstream IDs from #1834)
- `model_id`: `fast-ltx2`, `fast-ltx23`, or `fast-h3`
- `aspect_ratio`: one of `21:9`, `16:9`, `4:3`, `1:1`, `3:4`, `9:16`
- `resolution`: one of `480p`, `720p`, `1080p`, `4k`
- `duration_sec`: `5`, `10`, or `15`
- `initial_image`: optional image payload for reference / first-frame modes
- `last_frame_image`: optional image payload for first/last frame mode
Echoed `creation_config` includes the resolved frame size,
`num_frames`, and `generation_segment_cap` derived from `duration_sec`.
Mode validation:
- `ref2va` requires `initial_image`
- `fl2va` requires both `initial_image` and `last_frame_image`
Per-step generation uses the resolved `frame_width`, `frame_height`, and
`num_frames` from the session creation config.
## Websocket Contract
The websocket is the main integration surface between UI and runtime.
@@ -0,0 +1,137 @@
from __future__ import annotations
from dataclasses import dataclass
from dreamverse.config import MODEL_REGISTRY
# Canonical upstream wire IDs. FL2VA is tracked in #1834 but not wired on Dreamverse
# streaming backends yet.
LTX_LOBBY_GENERATION_MODES = frozenset({"t2va", "ref2va"})
H3_LOBBY_GENERATION_MODES = frozenset({"t2va", "ref2va"})
LTX_LOBBY_ASPECT_RATIOS = frozenset({"21:9", "16:9", "4:3", "1:1", "3:4", "9:16"})
# Realtime FastLTX serving is validated through 1080p-class outputs; 4K is rejected
# until the runtime path is tested on Dreamverse GPUs.
LTX_LOBBY_RESOLUTIONS = frozenset({"480p", "720p", "1080p"})
# FastH3 serves a fixed 768x1344 (16:9-class) output; lobby resolution is nominal.
H3_LOBBY_ASPECT_RATIOS = frozenset({"16:9"})
H3_LOBBY_RESOLUTIONS = frozenset({"720p"})
LOBBY_DURATION_SEC = frozenset({5, 10, 15})
UNSUPPORTED_GENERATION_MODE_MESSAGES = {
"fl2va": "First/last frame mode (FL2VA) is not supported yet.",
}
@dataclass(frozen=True)
class ModelCreationCapabilities:
generation_modes: frozenset[str]
aspect_ratios: frozenset[str]
resolutions: frozenset[str]
duration_sec: frozenset[int]
unsupported_generation_modes: frozenset[str] = frozenset({"fl2va"})
def as_dict(self) -> dict[str, object]:
unsupported = {
mode: UNSUPPORTED_GENERATION_MODE_MESSAGES[mode]
for mode in sorted(self.unsupported_generation_modes)
if mode in UNSUPPORTED_GENERATION_MODE_MESSAGES
}
return {
"generation_modes": sorted(self.generation_modes),
"aspect_ratios": sorted(self.aspect_ratios),
"resolutions": sorted(self.resolutions),
"duration_sec": sorted(self.duration_sec),
"unsupported_generation_modes": unsupported,
"reference_assets": {
"mime_types": ["image/png", "image/jpeg", "image/webp"],
"max_bytes": 15 * 1024 * 1024,
},
}
LTX_MODEL_CREATION_CAPABILITIES = ModelCreationCapabilities(
generation_modes=LTX_LOBBY_GENERATION_MODES,
aspect_ratios=LTX_LOBBY_ASPECT_RATIOS,
resolutions=LTX_LOBBY_RESOLUTIONS,
duration_sec=LOBBY_DURATION_SEC,
)
H3_MODEL_CREATION_CAPABILITIES = ModelCreationCapabilities(
generation_modes=H3_LOBBY_GENERATION_MODES,
aspect_ratios=H3_LOBBY_ASPECT_RATIOS,
resolutions=H3_LOBBY_RESOLUTIONS,
duration_sec=LOBBY_DURATION_SEC,
)
MODEL_CREATION_CAPABILITIES: dict[str, ModelCreationCapabilities] = {
"fast-ltx2": LTX_MODEL_CREATION_CAPABILITIES,
"fast-ltx23": LTX_MODEL_CREATION_CAPABILITIES,
"fast-h3": H3_MODEL_CREATION_CAPABILITIES,
}
def capabilities_for_model(model_id: str) -> ModelCreationCapabilities:
if model_id not in MODEL_REGISTRY:
raise ValueError(f"Unknown model_id: {model_id}")
return MODEL_CREATION_CAPABILITIES.get(model_id, LTX_MODEL_CREATION_CAPABILITIES)
def lobby_capabilities_as_dict() -> dict[str, object]:
model_ids = sorted(MODEL_REGISTRY.keys())
models = {model_id: capabilities_for_model(model_id).as_dict() for model_id in model_ids}
union_modes: set[str] = set()
union_aspects: set[str] = set()
union_resolutions: set[str] = set()
union_durations: set[int] = set()
for caps in MODEL_CREATION_CAPABILITIES.values():
union_modes.update(caps.generation_modes)
union_aspects.update(caps.aspect_ratios)
union_resolutions.update(caps.resolutions)
union_durations.update(caps.duration_sec)
return {
"model_ids": model_ids,
"models": models,
"generation_modes": sorted(union_modes),
"aspect_ratios": sorted(union_aspects),
"resolutions": sorted(union_resolutions),
"duration_sec": sorted(union_durations),
"unsupported_generation_modes": dict(UNSUPPORTED_GENERATION_MODE_MESSAGES),
"reference_assets": {
"mime_types": ["image/png", "image/jpeg", "image/webp"],
"max_bytes": 15 * 1024 * 1024,
},
}
# Backward-compatible alias used in tests.
LOBBY_CREATION_CAPABILITIES = lobby_capabilities_as_dict()
def validate_lobby_creation_config(
*,
model_id: str,
generation_mode: str,
aspect_ratio: str,
resolution: str,
duration_sec: int,
) -> None:
if model_id not in MODEL_REGISTRY:
raise ValueError(f"Unknown model_id: {model_id}")
caps = capabilities_for_model(model_id)
if generation_mode in caps.unsupported_generation_modes:
raise ValueError(UNSUPPORTED_GENERATION_MODE_MESSAGES[generation_mode])
if generation_mode not in caps.generation_modes:
raise ValueError(f"Unsupported generation_mode: {generation_mode}")
if aspect_ratio not in caps.aspect_ratios:
raise ValueError(f"Unsupported aspect_ratio: {aspect_ratio}")
if resolution not in caps.resolutions:
raise ValueError(f"Unsupported resolution: {resolution}")
if duration_sec not in caps.duration_sec:
raise ValueError("duration_sec must be 5, 10, or 15.")
@@ -36,6 +36,10 @@ class GenerationBackend(Protocol):
segment_idx: int,
image_path: str | None,
reset_conditioning: bool,
*,
frame_width: int | None = None,
frame_height: int | None = None,
num_frames: int | None = None,
) -> StepResult:
...
@@ -80,6 +80,10 @@ class VideoGenerationWorker:
segment_idx: int,
image_path: str | None,
reset_conditioning: bool,
*,
frame_width: int | None = None,
frame_height: int | None = None,
num_frames: int | None = None,
) -> StepResult:
"""Generate one segment through the selected model backend."""
return self._require_backend().generate_step(
@@ -87,6 +91,9 @@ class VideoGenerationWorker:
segment_idx,
image_path,
reset_conditioning,
frame_width=frame_width,
frame_height=frame_height,
num_frames=num_frames,
)
def warmup(self, prompt: str) -> dict[str, float]:
+10
View File
@@ -189,6 +189,9 @@ def gpu_worker_process(
segment_idx,
image_path=payload.image_path,
reset_conditioning=payload.reset_conditioning,
frame_width=payload.frame_width,
frame_height=payload.frame_height,
num_frames=payload.num_frames,
)
head_trim_frames = step_result.head_trim_frames
head_trim_audio_frames = step_result.head_trim_audio_frames
@@ -753,6 +756,10 @@ class GPUSlot:
segment_idx: int = 1,
image_path: str | None = None,
reset_conditioning: bool = False,
*,
frame_width: int | None = None,
frame_height: int | None = None,
num_frames: int | None = None,
) -> dict[str, float]:
"""Execute a generation step for a specific user.
@@ -766,6 +773,9 @@ class GPUSlot:
segment_idx=segment_idx,
image_path=image_path,
reset_conditioning=bool(reset_conditioning),
frame_width=frame_width,
frame_height=frame_height,
num_frames=num_frames,
)
response = await self._send_command_tagged(Command(CommandType.USER_STEP, payload=payload, user_id=user_id),
timeout=1800.0)
@@ -454,6 +454,10 @@ class LTX2GenerationBackend:
segment_idx: int,
image_path: str | None,
reset_conditioning: bool,
*,
frame_width: int | None = None,
frame_height: int | None = None,
num_frames: int | None = None,
) -> StepResult:
"""Execute one generation step; snapshot state for the next segment."""
timings: dict = {}
@@ -464,9 +468,9 @@ class LTX2GenerationBackend:
prompt=prompt,
negative_prompt="",
save_video=False,
height=FRAME_HEIGHT,
width=FRAME_WIDTH,
num_frames=NUM_FRAMES,
height=frame_height or FRAME_HEIGHT,
width=frame_width or FRAME_WIDTH,
num_frames=num_frames or NUM_FRAMES,
fps=24,
num_inference_steps=NUM_INFERENCE_STEPS,
guidance_scale=1.0,
+2
View File
@@ -33,6 +33,7 @@ from dreamverse.routes.presets import (
prompt_config_router,
curated_presets_router,
)
from dreamverse.routes.creation import creation_router
from dreamverse.session.controller import SessionController
@@ -92,6 +93,7 @@ app.add_middleware(
app.include_router(build_health_router(lambda: runtime.gpu_pool))
app.include_router(internal_monitor_router)
app.include_router(prompt_config_router)
app.include_router(creation_router)
if DEVTOOLS_ENABLED:
app.include_router(curated_presets_router)
@@ -200,6 +200,10 @@ class MiniMaxH3GenerationBackend:
segment_idx: int,
image_path: str | None,
reset_conditioning: bool,
*,
frame_width: int | None = None,
frame_height: int | None = None,
num_frames: int | None = None,
) -> StepResult:
"""Generate one synchronized FastH3 segment and retain its last frame.
@@ -207,6 +211,7 @@ class MiniMaxH3GenerationBackend:
conditioned frame and its matching audio duration are trimmed before
streaming so adjacent segments do not duplicate media.
"""
del frame_width, frame_height, num_frames
if self.generator is None:
raise RuntimeError("FastH3 generator is not initialized.")
conditioning_image, uses_continuation = self._select_conditioning_image(
+65 -5
View File
@@ -31,6 +31,8 @@ from fastapi.staticfiles import StaticFiles
from dreamverse._deps import require_dreamverse_runtime_deps
from dreamverse.config import FRONTEND_STATIC_DIR_CANDIDATES, GENERATION_SEGMENT_CAP
from dreamverse.creation_capabilities import lobby_capabilities_as_dict
from dreamverse.session_creation_config import parse_session_creation_config, validate_generation_mode_assets
from dreamverse.session_init_image import cleanup_session_init_image, persist_session_init_image
LATENCY_MS = 200
@@ -225,6 +227,11 @@ async def prompt_system_config():
}
@app.get("/creation-capabilities")
async def creation_capabilities():
return lobby_capabilities_as_dict()
@app.get("/curated-presets")
async def curated_presets():
presets = [
@@ -290,6 +297,8 @@ async def websocket_endpoint(websocket: WebSocket):
send_lock = asyncio.Lock()
stop_event = asyncio.Event()
session_init_image = None
session_last_frame_image = None
session_creation_config = None
async def ws_send_json(payload: dict) -> None:
async with send_lock:
@@ -348,6 +357,7 @@ async def websocket_endpoint(websocket: WebSocket):
try:
session_init_image = persist_session_init_image(init_data.get("initial_image"))
session_last_frame_image = persist_session_init_image(init_data.get("last_frame_image"))
except ValueError as exc:
await ws_send_json({
"type": "error",
@@ -356,13 +366,31 @@ async def websocket_endpoint(websocket: WebSocket):
await websocket.close(code=1003, reason="Invalid initial image")
return
try:
session_creation_config = parse_session_creation_config(init_data)
validate_generation_mode_assets(
session_creation_config.generation_mode,
has_initial_image=session_init_image is not None,
has_last_frame_image=session_last_frame_image is not None,
)
except ValueError as exc:
await ws_send_json({
"type": "error",
"message": str(exc),
})
await websocket.close(code=1003, reason="Invalid creation config")
return
timeout_task = asyncio.create_task(session_timeout())
await ws_send_json({
gpu_assigned_payload: dict[str, object] = {
"type": "gpu_assigned",
"gpu_id": 0,
"session_timeout": SESSION_TIMEOUT_SECONDS,
})
}
if session_creation_config is not None:
gpu_assigned_payload["creation_config"] = session_creation_config.as_dict()
await ws_send_json(gpu_assigned_payload)
raw_prompt_queue: asyncio.Queue[PromptSubmission] = asyncio.Queue()
ready_prompt_queue: asyncio.Queue[ReadyPrompt] = asyncio.Queue()
@@ -391,8 +419,16 @@ async def websocket_endpoint(websocket: WebSocket):
if previous_session_image is not None:
cleanup_session_init_image(previous_session_image)
def replace_last_frame_image(last_frame_payload: object) -> None:
nonlocal session_last_frame_image
next_last_frame_image = persist_session_init_image(last_frame_payload)
previous_last_frame_image = session_last_frame_image
session_last_frame_image = next_last_frame_image
if previous_last_frame_image is not None:
cleanup_session_init_image(previous_last_frame_image)
async def send_stream_start(seed_reason: str) -> None:
await ws_send_json({
stream_start_payload: dict[str, object] = {
"type": "ltx2_stream_start",
"total_segments": len(curated_prompts),
"preset_id": preset_id,
@@ -400,8 +436,15 @@ async def websocket_endpoint(websocket: WebSocket):
"live_mode": True,
"loop_generation_enabled": loop_generation_enabled,
"loop_iteration": loop_iteration,
"generation_segment_cap": 0,
})
"generation_segment_cap": (
session_creation_config.generation_segment_cap
if session_creation_config is not None
else GENERATION_SEGMENT_CAP
),
}
if session_creation_config is not None:
stream_start_payload["creation_config"] = session_creation_config.as_dict()
await ws_send_json(stream_start_payload)
if seed_reason == "init":
await ws_send_json({
"type": "seed_prompts_updated",
@@ -509,6 +552,7 @@ async def websocket_endpoint(websocket: WebSocket):
nonlocal project_active
nonlocal project_stream_started
nonlocal pending_project_end
nonlocal session_creation_config
next_initial_rollout_prompt = str(payload.get("initial_rollout_prompt") or "").strip()
next_preset_id = str(payload.get("preset_id") or "").strip()
@@ -520,6 +564,21 @@ async def websocket_endpoint(websocket: WebSocket):
try:
replace_session_image(payload.get("initial_image"))
replace_last_frame_image(payload.get("last_frame_image"))
except ValueError as exc:
await ws_send_json({
"type": "error",
"message": str(exc),
})
return False
try:
session_creation_config = parse_session_creation_config(payload)
validate_generation_mode_assets(
session_creation_config.generation_mode,
has_initial_image=session_init_image is not None,
has_last_frame_image=session_last_frame_image is not None,
)
except ValueError as exc:
await ws_send_json({
"type": "error",
@@ -1182,6 +1241,7 @@ async def websocket_endpoint(websocket: WebSocket):
finally:
stop_event.set()
cleanup_session_init_image(session_init_image)
cleanup_session_init_image(session_last_frame_image)
for static_dir in FRONTEND_STATIC_DIR_CANDIDATES:
@@ -0,0 +1,14 @@
"""Creation studio capability routes."""
from __future__ import annotations
from fastapi import APIRouter
from dreamverse.creation_capabilities import lobby_capabilities_as_dict
creation_router = APIRouter(tags=["creation"])
@creation_router.get("/creation-capabilities")
async def creation_capabilities() -> dict[str, object]:
return lobby_capabilities_as_dict()
+105 -44
View File
@@ -27,6 +27,7 @@ from typing import TYPE_CHECKING
from fastapi import WebSocket, WebSocketDisconnect
from dreamverse.gpu_pool import GPUSlot
from dreamverse.session_init_image import cleanup_session_init_image, persist_session_init_image
from dreamverse.session_creation_config import parse_session_creation_config, validate_generation_mode_assets
from dreamverse.worker_ipc import MediaChunk, MediaComplete, MediaInit
from dreamverse.config import (
@@ -156,6 +157,9 @@ class SessionController:
prompt_worker_task: asyncio.Task | None = None
rewrite_seed_prompts_task: asyncio.Task | None = None
session_init_image = None
session_last_frame_image = None
session_creation_config = None
session_generation_segment_cap = GENERATION_SEGMENT_CAP
async def session_timeout():
"""Close the session after timeout."""
@@ -237,6 +241,7 @@ class SessionController:
try:
session_init_image = persist_session_init_image(init_data.get("initial_image"))
session_last_frame_image = persist_session_init_image(init_data.get("last_frame_image"))
except ValueError as exc:
await ws_send_json({
"type": "error",
@@ -245,6 +250,22 @@ class SessionController:
await websocket.close(code=1003, reason="Invalid initial image")
return
try:
session_creation_config = parse_session_creation_config(init_data)
session_generation_segment_cap = session_creation_config.generation_segment_cap
validate_generation_mode_assets(
session_creation_config.generation_mode,
has_initial_image=session_init_image is not None,
has_last_frame_image=session_last_frame_image is not None,
)
except ValueError as exc:
await ws_send_json({
"type": "error",
"message": str(exc),
})
await websocket.close(code=1003, reason="Invalid creation config")
return
if preset_id:
print(f"Client {client_id[:8]} selected preset: {preset_id} "
f"label={preset_label or '(unset)'} "
@@ -256,6 +277,16 @@ class SessionController:
if session_init_image is not None:
print(f"Client {client_id[:8]} uploaded initial image: "
f"{session_init_image.display_name}")
if session_last_frame_image is not None:
print(f"Client {client_id[:8]} uploaded last frame image: "
f"{session_last_frame_image.display_name}")
if session_creation_config is not None:
print(f"Client {client_id[:8]} creation config: "
f"model={session_creation_config.model_id}, "
f"mode={session_creation_config.generation_mode}, "
f"size={session_creation_config.frame_width}x{session_creation_config.frame_height}, "
f"duration={session_creation_config.duration_sec}s, "
f"segment_cap={session_creation_config.generation_segment_cap}")
# Acquire a GPU slot.
gpu_id, slot = await self.gpu_pool.acquire(client_id, websocket)
@@ -264,14 +295,20 @@ class SessionController:
timeout_task = asyncio.create_task(session_timeout())
# Join the engine on this GPU.
await slot.join_user(client_id, model_id=ACTIVE_MODEL_ID)
await slot.join_user(
client_id,
model_id=session_creation_config.model_id if session_creation_config is not None else ACTIVE_MODEL_ID,
)
# Notify client they're connected to a GPU.
await ws_send_json({
gpu_assigned_payload: dict[str, object] = {
"type": "gpu_assigned",
"gpu_id": gpu_id,
"session_timeout": SESSION_TIMEOUT_SECONDS,
})
}
if session_creation_config is not None:
gpu_assigned_payload["creation_config"] = session_creation_config.as_dict()
await ws_send_json(gpu_assigned_payload)
await log_event(
"gpu_assigned",
{
@@ -315,6 +352,14 @@ class SessionController:
if previous_session_init_image is not None:
cleanup_session_init_image(previous_session_init_image)
def replace_last_frame_image(last_frame_payload: object) -> None:
nonlocal session_last_frame_image
next_last_frame_image = persist_session_init_image(last_frame_payload)
previous_last_frame_image = session_last_frame_image
session_last_frame_image = next_last_frame_image
if previous_last_frame_image is not None:
cleanup_session_init_image(previous_last_frame_image)
async def schedule_simple_generate_request(payload: dict[str, object]) -> None:
nonlocal preset_id
nonlocal preset_label
@@ -452,6 +497,8 @@ class SessionController:
nonlocal project_active
nonlocal project_stream_started
nonlocal pending_project_end
nonlocal session_creation_config
nonlocal session_generation_segment_cap
next_initial_rollout_prompt = str(payload.get("initial_rollout_prompt") or "").strip()
next_enhancement_enabled = bool(payload.get("enhancement_enabled", True))
@@ -498,6 +545,22 @@ class SessionController:
try:
replace_session_init_image(payload.get("initial_image"))
replace_last_frame_image(payload.get("last_frame_image"))
except ValueError as exc:
await ws_send_json({
"type": "error",
"message": str(exc),
})
return False
try:
session_creation_config = parse_session_creation_config(payload)
session_generation_segment_cap = session_creation_config.generation_segment_cap
validate_generation_mode_assets(
session_creation_config.generation_mode,
has_initial_image=session_init_image is not None,
has_last_frame_image=session_last_frame_image is not None,
)
except ValueError as exc:
await ws_send_json({
"type": "error",
@@ -941,7 +1004,7 @@ class SessionController:
"segment_cap":
_resolve_generation_segment_cap(
single_clip_mode=single_clip_mode,
cap=GENERATION_SEGMENT_CAP,
cap=session_generation_segment_cap,
),
})
continue
@@ -1281,27 +1344,22 @@ class SessionController:
))
else:
project_stream_started = True
await ws_send_json({
"type":
"ltx2_stream_start",
"total_segments":
len(curated_prompts),
"preset_id":
preset_id,
"stream_mode":
"av_fmp4",
"live_mode":
True,
"loop_generation_enabled":
loop_generation_enabled,
"loop_iteration":
loop_iteration,
"generation_segment_cap":
_resolve_generation_segment_cap(
stream_start_payload: dict[str, object] = {
"type": "ltx2_stream_start",
"total_segments": len(curated_prompts),
"preset_id": preset_id,
"stream_mode": "av_fmp4",
"live_mode": True,
"loop_generation_enabled": loop_generation_enabled,
"loop_iteration": loop_iteration,
"generation_segment_cap": _resolve_generation_segment_cap(
single_clip_mode=single_clip_mode,
cap=GENERATION_SEGMENT_CAP,
cap=session_generation_segment_cap,
),
})
}
if session_creation_config is not None:
stream_start_payload["creation_config"] = session_creation_config.as_dict()
await ws_send_json(stream_start_payload)
await ws_send_json({
"type": "seed_prompts_updated",
"prompts": seed_prompt_memory,
@@ -1340,27 +1398,22 @@ class SessionController:
loop_iteration += 1
project_stream_started = True
await ws_send_json({
"type":
"ltx2_stream_start",
"total_segments":
len(curated_prompts),
"preset_id":
preset_id,
"stream_mode":
"av_fmp4",
"live_mode":
True,
"loop_generation_enabled":
loop_generation_enabled,
"loop_iteration":
loop_iteration,
"generation_segment_cap":
_resolve_generation_segment_cap(
restart_stream_payload: dict[str, object] = {
"type": "ltx2_stream_start",
"total_segments": len(curated_prompts),
"preset_id": preset_id,
"stream_mode": "av_fmp4",
"live_mode": True,
"loop_generation_enabled": loop_generation_enabled,
"loop_iteration": loop_iteration,
"generation_segment_cap": _resolve_generation_segment_cap(
single_clip_mode=single_clip_mode,
cap=GENERATION_SEGMENT_CAP,
cap=session_generation_segment_cap,
),
})
}
if session_creation_config is not None:
restart_stream_payload["creation_config"] = session_creation_config.as_dict()
await ws_send_json(restart_stream_payload)
if nonlocal_reason == "loop_restart":
await ws_send_json({
"type": "loop_restarted",
@@ -1389,13 +1442,14 @@ class SessionController:
pending_simple_prompt_submission = None
if (not single_clip_mode and not generation_cap_blocked and not rollout_waiting_for_rewrite
and GENERATION_SEGMENT_CAP > 0 and generated_segment_count >= GENERATION_SEGMENT_CAP):
and session_generation_segment_cap > 0
and generated_segment_count >= session_generation_segment_cap):
loop_generation_enabled = False
rollout_waiting_for_rewrite = True
_main_print(
"INFO",
f"Segment cap reached for client {client_id[:8]} "
f"(cap_segments={GENERATION_SEGMENT_CAP}, "
f"(cap_segments={session_generation_segment_cap}, "
f"generated_segments={generated_segment_count}); "
"waiting for rollout rewrite",
)
@@ -1620,6 +1674,9 @@ class SessionController:
pending_reset_conditioning = False
step_image_path = (str(session_init_image.file_path)
if segment_idx == 1 and session_init_image is not None else None)
step_frame_width = session_creation_config.frame_width if session_creation_config is not None else None
step_frame_height = session_creation_config.frame_height if session_creation_config is not None else None
step_num_frames = session_creation_config.num_frames if session_creation_config is not None else None
step_task = asyncio.create_task(
slot.user_step(
client_id,
@@ -1627,6 +1684,9 @@ class SessionController:
segment_idx=segment_idx,
image_path=step_image_path,
reset_conditioning=step_reset_conditioning,
frame_width=step_frame_width,
frame_height=step_frame_height,
num_frames=step_num_frames,
))
segment_generation_active = True
try:
@@ -1808,3 +1868,4 @@ class SessionController:
await self.gpu_pool.release(client_id)
finally:
cleanup_session_init_image(session_init_image)
cleanup_session_init_image(session_last_frame_image)
@@ -0,0 +1,140 @@
from __future__ import annotations
from dataclasses import dataclass
from dreamverse.config import FRAME_HEIGHT, FRAME_WIDTH, GENERATION_SEGMENT_CAP, MODEL_REGISTRY, NUM_FRAMES
from dreamverse.creation_capabilities import validate_lobby_creation_config
LTX_LOBBY_MODEL_IDS = frozenset(MODEL_REGISTRY.keys())
SUPPORTED_GENERATION_MODES = frozenset({"t2va", "fl2va", "ref2va"})
SUPPORTED_ASPECT_RATIOS = frozenset({"21:9", "16:9", "4:3", "1:1", "3:4", "9:16"})
SUPPORTED_RESOLUTIONS = frozenset({"480p", "720p", "1080p", "4k"})
SEGMENT_DURATION_SEC = 5
@dataclass(frozen=True)
class SessionCreationConfig:
model_id: str
generation_mode: str
aspect_ratio: str
resolution: str
duration_sec: int
frame_width: int
frame_height: int
num_frames: int
generation_segment_cap: int
def as_dict(self) -> dict[str, object]:
return {
"model_id": self.model_id,
"generation_mode": self.generation_mode,
"aspect_ratio": self.aspect_ratio,
"resolution": self.resolution,
"duration_sec": self.duration_sec,
"frame_width": self.frame_width,
"frame_height": self.frame_height,
"num_frames": self.num_frames,
"generation_segment_cap": self.generation_segment_cap,
}
def _round_to_multiple(value: float, multiple: int = 32) -> int:
rounded = int(round(value / multiple)) * multiple
return max(multiple, rounded)
def _resolution_base(resolution: str) -> int:
return {
"480p": 480,
"720p": 720,
"1080p": 1080,
"4k": 2160,
}.get(resolution, 720)
def resolve_frame_size(aspect_ratio: str, resolution: str) -> tuple[int, int]:
if aspect_ratio == "16:9" and resolution == "1080p":
return FRAME_WIDTH, FRAME_HEIGHT
base = _resolution_base(resolution)
width_ratio, height_ratio = {
"21:9": (21, 9),
"16:9": (16, 9),
"4:3": (4, 3),
"1:1": (1, 1),
"3:4": (3, 4),
"9:16": (9, 16),
}.get(aspect_ratio, (16, 9))
if width_ratio >= height_ratio:
height = _round_to_multiple(base)
width = _round_to_multiple(height * width_ratio / height_ratio)
else:
width = _round_to_multiple(base)
height = _round_to_multiple(width * height_ratio / width_ratio)
return width, height
def duration_sec_to_segment_cap(duration_sec: int, *, global_cap: int = GENERATION_SEGMENT_CAP) -> int:
requested = max(1, int(round(duration_sec / SEGMENT_DURATION_SEC + 0.0001)))
if global_cap <= 0:
return requested
return max(1, min(requested, global_cap))
def parse_session_creation_config(payload: dict[str, object]) -> SessionCreationConfig:
raw_model_id = str(payload.get("model_id") or "").strip()
model_id = raw_model_id if raw_model_id in LTX_LOBBY_MODEL_IDS else "fast-ltx23"
generation_mode = str(payload.get("generation_mode") or "t2va").strip()
if generation_mode not in SUPPORTED_GENERATION_MODES:
raise ValueError(f"Unsupported generation_mode: {generation_mode}")
aspect_ratio = str(payload.get("aspect_ratio") or "16:9").strip()
if aspect_ratio not in SUPPORTED_ASPECT_RATIOS:
raise ValueError(f"Unsupported aspect_ratio: {aspect_ratio}")
resolution = str(payload.get("resolution") or "720p").strip()
if resolution not in SUPPORTED_RESOLUTIONS:
raise ValueError(f"Unsupported resolution: {resolution}")
try:
duration_sec = int(payload.get("duration_sec") or SEGMENT_DURATION_SEC)
except (TypeError, ValueError) as exc:
raise ValueError("duration_sec must be an integer.") from exc
if duration_sec not in {5, 10, 15}:
raise ValueError("duration_sec must be 5, 10, or 15.")
validate_lobby_creation_config(
model_id=model_id,
generation_mode=generation_mode,
aspect_ratio=aspect_ratio,
resolution=resolution,
duration_sec=duration_sec,
)
if model_id not in MODEL_REGISTRY:
raise ValueError(f"Unsupported model_id: {model_id}")
frame_width, frame_height = resolve_frame_size(aspect_ratio, resolution)
return SessionCreationConfig(
model_id=model_id,
generation_mode=generation_mode,
aspect_ratio=aspect_ratio,
resolution=resolution,
duration_sec=duration_sec,
frame_width=frame_width,
frame_height=frame_height,
num_frames=NUM_FRAMES,
generation_segment_cap=duration_sec_to_segment_cap(duration_sec),
)
def validate_generation_mode_assets(
generation_mode: str,
*,
has_initial_image: bool,
has_last_frame_image: bool,
) -> None:
if generation_mode == "ref2va" and not has_initial_image:
raise ValueError("Ref2VA mode requires a reference image.")
@@ -0,0 +1,74 @@
import pytest
from dreamverse.creation_capabilities import (
capabilities_for_model,
lobby_capabilities_as_dict,
validate_lobby_creation_config,
)
def test_lobby_capabilities_include_all_registry_models():
caps = lobby_capabilities_as_dict()
assert set(caps["model_ids"]) == {"fast-ltx2", "fast-ltx23", "fast-h3"}
assert "fl2va" not in caps["generation_modes"]
assert "4k" not in caps["resolutions"]
def test_fast_h3_capabilities_use_fixed_geometry():
h3_caps = capabilities_for_model("fast-h3")
assert h3_caps.generation_modes == frozenset({"t2va", "ref2va"})
assert h3_caps.aspect_ratios == frozenset({"16:9"})
assert h3_caps.resolutions == frozenset({"720p"})
def test_validate_lobby_creation_config_accepts_supported_t2va():
validate_lobby_creation_config(
model_id="fast-ltx23",
generation_mode="t2va",
aspect_ratio="16:9",
resolution="1080p",
duration_sec=5,
)
def test_validate_lobby_creation_config_accepts_fast_h3():
validate_lobby_creation_config(
model_id="fast-h3",
generation_mode="ref2va",
aspect_ratio="16:9",
resolution="720p",
duration_sec=10,
)
def test_validate_lobby_creation_config_rejects_fl2va():
with pytest.raises(ValueError, match="FL2VA"):
validate_lobby_creation_config(
model_id="fast-ltx23",
generation_mode="fl2va",
aspect_ratio="16:9",
resolution="720p",
duration_sec=5,
)
def test_validate_lobby_creation_config_rejects_4k():
with pytest.raises(ValueError, match="Unsupported resolution"):
validate_lobby_creation_config(
model_id="fast-ltx2",
generation_mode="t2va",
aspect_ratio="16:9",
resolution="4k",
duration_sec=10,
)
def test_validate_lobby_creation_config_rejects_invalid_h3_aspect():
with pytest.raises(ValueError, match="Unsupported aspect_ratio"):
validate_lobby_creation_config(
model_id="fast-h3",
generation_mode="t2va",
aspect_ratio="9:16",
resolution="720p",
duration_sec=5,
)
@@ -0,0 +1,89 @@
import pytest
from dreamverse.session_creation_config import (
duration_sec_to_segment_cap,
parse_session_creation_config,
resolve_frame_size,
validate_generation_mode_assets,
)
def test_parse_session_creation_config_defaults():
config = parse_session_creation_config({})
assert config.model_id == "fast-ltx23"
assert config.generation_mode == "t2va"
assert config.aspect_ratio == "16:9"
assert config.resolution == "720p"
assert config.duration_sec == 5
assert config.generation_segment_cap == 1
def test_parse_session_creation_config_maps_duration_to_segment_cap():
config = parse_session_creation_config(
{
"model_id": "fast-ltx2",
"generation_mode": "ref2va",
"aspect_ratio": "9:16",
"resolution": "480p",
"duration_sec": 15,
},
)
assert config.model_id == "fast-ltx2"
assert config.generation_mode == "ref2va"
assert config.generation_segment_cap == 3
assert config.frame_width >= 480
assert config.frame_height >= 480
def test_resolve_frame_size_uses_model_default_for_1080p_landscape():
width, height = resolve_frame_size("16:9", "1080p")
assert (width, height) == (1920, 1088)
def test_duration_sec_to_segment_cap_respects_global_cap():
assert duration_sec_to_segment_cap(15, global_cap=2) == 2
def test_parse_session_creation_config_accepts_fast_h3():
config = parse_session_creation_config(
{
"model_id": "fast-h3",
"generation_mode": "t2va",
"aspect_ratio": "16:9",
"resolution": "720p",
"duration_sec": 10,
},
)
assert config.model_id == "fast-h3"
assert config.generation_mode == "t2va"
assert config.generation_segment_cap == 2
def test_parse_session_creation_config_rejects_fl2va():
with pytest.raises(ValueError, match="FL2VA"):
parse_session_creation_config(
{
"generation_mode": "fl2va",
"aspect_ratio": "16:9",
"resolution": "720p",
"duration_sec": 5,
},
)
def test_parse_session_creation_config_rejects_4k():
with pytest.raises(ValueError, match="Unsupported resolution"):
parse_session_creation_config(
{
"generation_mode": "t2va",
"aspect_ratio": "16:9",
"resolution": "4k",
"duration_sec": 5,
},
)
def test_validate_generation_mode_assets():
validate_generation_mode_assets("t2va", has_initial_image=False, has_last_frame_image=False)
with pytest.raises(ValueError, match="Ref2VA"):
validate_generation_mode_assets("ref2va", has_initial_image=False, has_last_frame_image=False)
+3
View File
@@ -147,6 +147,9 @@ class UserStepPayload:
segment_idx: int
image_path: str | None
reset_conditioning: bool
frame_width: int | None = None
frame_height: int | None = None
num_frames: int | None = None
@dataclass(frozen=True)
+4 -12
View File
@@ -17,25 +17,17 @@ test.describe('frontend shell', () => {
});
});
test('composer hydrates with curated preset cards', async ({ page }) => {
test('composer hydrates with creation studio controls', async ({ page }) => {
await page.goto('/');
// The Continuation prompt textarea + Generate button render once
// the FE has hydrated against the public-FastVideo-backed
// dreamverse-server. Their presence proves the integration handshake
// (CORS, /curated-presets, /prompt-system-config) completed.
const continuation = page.getByLabel('Continuation prompt');
await expect(continuation).toBeVisible({ timeout: 30_000 });
const generate = page.getByRole('button', { name: /^generate$/i });
await expect(generate).toBeVisible({ timeout: 30_000 });
// Curated presets render as buttons; verify at least one is
// available — that's the only way the user can populate the
// Continuation textarea in the default composer.
const presetCard = page.getByRole('button', {
name: /LEGO Stormtroopers|Clay Stop-Motion|Boy & Dog|School Prank|Gamer Gets Banned|Small Town Oil Strike|Grandpa's Wing Costume/i,
}).first();
await expect(presetCard).toBeVisible({ timeout: 30_000 });
await expect(page.getByText('Direct scenes in seconds')).toBeVisible({ timeout: 30_000 });
await expect(page.getByRole('button', { name: /FastLTX/i }).first()).toBeVisible({ timeout: 30_000 });
await expect(continuation).toHaveAttribute('placeholder', /Describe your video or mention elements/i);
});
});
+4
View File
@@ -42,6 +42,10 @@ const nextConfig: NextConfig = {
source: '/prompt-system-config',
destination: `${backendUrl}/prompt-system-config`,
},
{
source: '/creation-capabilities',
destination: `${backendUrl}/creation-capabilities`,
},
{
source: '/curated-presets',
destination: `${backendUrl}/curated-presets`,
+2262 -113
View File
File diff suppressed because it is too large Load Diff
+4
View File
@@ -27,11 +27,15 @@
"@radix-ui/react-accordion": "^1.2.12",
"@radix-ui/react-checkbox": "^1.3.3",
"@radix-ui/react-collapsible": "^1.1.12",
"@radix-ui/react-dropdown-menu": "^2.1.24",
"@radix-ui/react-label": "^2.1.8",
"@radix-ui/react-popover": "^1.1.23",
"@radix-ui/react-scroll-area": "^1.2.10",
"@radix-ui/react-select": "^2.2.6",
"@radix-ui/react-separator": "^1.1.8",
"@radix-ui/react-slider": "^1.4.7",
"@radix-ui/react-slot": "^1.2.4",
"@radix-ui/react-tabs": "^1.1.21",
"class-variance-authority": "^0.7.1",
"clsx": "^2.1.1",
"framer-motion": "^12.36.0",
+26 -1
View File
@@ -2,6 +2,7 @@
@import "tailwindcss";
@custom-variant dark (&:is(.dark *));
@custom-variant hover-capable (@media (hover: hover) and (pointer: fine));
:root {
color-scheme: light;
@@ -201,7 +202,7 @@ summary::-webkit-details-marker {
@apply border-border;
}
body {
@apply bg-background text-foreground;
@apply bg-background text-foreground antialiased;
}
}
@@ -220,6 +221,14 @@ summary::-webkit-details-marker {
.stroke-dash-anim {
animation: stroke-dash-animation 2s linear infinite;
}
.text-pretty {
text-wrap: pretty;
}
.text-balance {
text-wrap: balance;
}
}
@keyframes stroke-dash-animation {
@@ -287,6 +296,22 @@ html.theme-transition *::after {
/* —————————————— CUSTOM TAILWIND —————————————— */
@layer components {
.studio-control {
@apply transition-[border-color,background-color,box-shadow,color,transform] duration-150 focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-accent-blue/40;
}
.studio-control-press {
@apply active:scale-[0.96] motion-reduce:active:scale-100;
}
.studio-hover-surface {
@apply hover-capable:hover:border-border hover-capable:hover:bg-accent/50;
}
.studio-media-outline {
@apply outline outline-1 -outline-offset-1 outline-black/10 dark:outline-white/10;
}
.debug {
@apply border border-rose-500;
}
+300 -136
View File
@@ -1,10 +1,20 @@
"use client";
import { Fragment, useCallback, useEffect, useLayoutEffect, useMemo, useRef, useState } from "react";
import { Fragment, useCallback, useEffect, useLayoutEffect, useMemo, useRef, useState, type Dispatch, type SetStateAction } from "react";
import { AnimatePresence, motion } from "framer-motion";
import { Download, Share2 } from "lucide-react";
import DevtoolsShell from "@/components/devtools/DevtoolsShell";
import MonitorPage from "@/components/MonitorPage";
import ChatBar from "@/components/ChatBar";
import CreationStudio from "@/components/creation/CreationStudio";
import {
buildMentionOptions,
type AspectRatioId,
type CreationModeId,
type CreationModelId,
type ResolutionId,
} from "@/lib/creationConfig";
import { toGenerationMode } from "@/lib/generationMode";
import type { SessionCreationConfig } from "@/components/creation/SessionCreationConfigPills";
import SessionTimeoutModal from "@/components/SessionTimeoutModal";
import Sidebar from "@/components/Sidebar";
import Header from "@/components/Header";
@@ -22,6 +32,18 @@ import {
buildRewritePromptWindowSnapshotFromPrompts,
normalizePromptWindowSnapshot,
} from "@/lib/prompts/promptWindowSnapshot";
import {
DEFAULT_LOBBY_CAPABILITIES_BUNDLE,
clampLobbySelectionToCapabilities,
parseLobbyCapabilitiesBundle,
resolveModelCapabilities,
validateLobbyCreationSelection,
type LobbyCapabilitiesBundle,
} from "@/lib/creationCapabilities";
import {
buildCreationInitPayload,
parseEchoedCreationConfig,
} from "@/lib/creationPayload";
import rawPresets from "@/lib/storyPresetsData";
import { cn } from "@/lib/utils";
import { createWebSocketConnection, detachAndCloseWebSocket } from "@/lib/ws/client";
@@ -78,98 +100,6 @@ function yieldToEventLoop(): Promise<void> {
return new Promise((r) => setTimeout(r, 0));
}
const HERO_WAVE_LIGHT = ["#2A4A98", "#4878E5", "#6FA0F2", "#B0BCC8", "#E8D99E", "#D8C844", "#C2A620"];
const HERO_WAVE_DARK = ["#143468", "#1E58B8", "#3892F0", "#80B8E8", "#B8D0EA", "#E2D498", "#DABB50"];
const HERO_TEXT = "Direct scenes in seconds";
function HeroTagline() {
const ref = useRef<HTMLHeadingElement>(null);
useEffect(() => {
const el = ref.current;
if (!el) return;
let rafId = 0;
function play() {
const chars = el!.querySelectorAll<HTMLSpanElement>("[data-char]");
if (!chars.length) return;
cancelAnimationFrame(rafId);
const isDark = document.documentElement.classList.contains("dark");
const colors = isDark ? HERO_WAVE_DARK : HERO_WAVE_LIGHT;
const waveLen = 10;
const total = chars.length + waveLen;
const duration = 1200;
const maxBlur = 3.5;
const start = performance.now();
function tick() {
const t = Math.min((performance.now() - start) / duration, 1);
const pos = t * total;
chars.forEach((ch, i) => {
const rel = pos - i;
if (rel >= 0 && rel < waveLen) {
const norm = rel / waveLen;
const ci = Math.floor(norm * colors.length);
ch.style.color = colors[Math.min(colors.length - 1, ci)];
let blur = 0;
if (norm < 0.25) {
blur = maxBlur * (1 - norm / 0.25);
} else if (norm > 0.75) {
blur = maxBlur * ((norm - 0.75) / 0.25);
}
ch.style.filter = blur > 0.1 ? `blur(${blur.toFixed(1)}px)` : "";
} else {
ch.style.color = "";
ch.style.filter = "";
}
});
if (t < 1) {
rafId = requestAnimationFrame(tick);
} else {
chars.forEach((ch) => {
ch.style.color = "";
ch.style.filter = "";
});
}
}
rafId = requestAnimationFrame(tick);
}
const initialDelay = setTimeout(play, 400);
const interval = setInterval(play, 5000);
return () => {
clearTimeout(initialDelay);
clearInterval(interval);
cancelAnimationFrame(rafId);
};
}, []);
return (
<h1 ref={ref} className="text-center text-3xl font-medium text-[#343537] dark:text-[#FAFAFB] sm:text-4xl">
{HERO_TEXT.split(" ").map((word, wi) => (
<Fragment key={wi}>
{wi > 0 && (
<span data-char className="transition-[color,filter] duration-150">
{" "}
</span>
)}
<span className="inline-flex">
{word.split("").map((char, ci) => (
<span key={ci} data-char className="inline-block transition-[color,filter] duration-150">
{char}
</span>
))}
</span>
</Fragment>
))}
</h1>
);
}
export default function Page() {
const storesRef = useRef<PageStores | null>(null);
if (!storesRef.current) {
@@ -323,8 +253,33 @@ export default function Page() {
const [ttffValueMs, setTtffValueMs] = useState<number | null>(null);
const ttffIntervalRef = useRef<ReturnType<typeof setInterval> | null>(null);
const pendingInitialPromptRef = useRef("");
const referenceFileRef = useRef<File | null>(null);
const firstFrameFileRef = useRef<File | null>(null);
const lastFrameFileRef = useRef<File | null>(null);
const lastArchivedReplayKeyRef = useRef("");
const [sidebarOpen, setSidebarOpen] = useState(false);
const [creationModelId, setCreationModelId] = useState<CreationModelId>("fast-ltx23");
const [creationModeId, setCreationModeId] = useState<CreationModeId>("t2v");
const [creationAspectRatio, setCreationAspectRatio] = useState<AspectRatioId>("16:9");
const [creationResolution, setCreationResolution] = useState<ResolutionId>("720p");
const [creationDurationSec, setCreationDurationSec] = useState(5);
const [lobbyCapabilitiesBundle, setLobbyCapabilitiesBundle] = useState<LobbyCapabilitiesBundle>(
DEFAULT_LOBBY_CAPABILITIES_BUNDLE,
);
const activeModelCapabilities = useMemo(
() => resolveModelCapabilities(lobbyCapabilitiesBundle, creationModelId),
[lobbyCapabilitiesBundle, creationModelId],
);
const [sessionCreationConfig, setSessionCreationConfig] = useState<SessionCreationConfig>({
modelId: "fast-ltx23",
modeId: "t2v",
aspectRatio: "16:9",
resolution: "720p",
durationSec: 5,
});
const [referencePreviewUrl, setReferencePreviewUrl] = useState<string | null>(null);
const [firstFramePreviewUrl, setFirstFramePreviewUrl] = useState<string | null>(null);
const [lastFramePreviewUrl, setLastFramePreviewUrl] = useState<string | null>(null);
const [currentThumbnail, setCurrentThumbnail] = useState<string | null>(null);
const currentProjectIdRef = useRef("");
const currentProjectCreatedAtRef = useRef(0);
@@ -345,6 +300,57 @@ export default function Page() {
setIsMobileShareCapable(typeof navigator.canShare === "function" && window.matchMedia("(pointer: coarse)").matches);
}, []);
useEffect(() => {
return () => {
if (referencePreviewUrl) {
URL.revokeObjectURL(referencePreviewUrl);
}
if (firstFramePreviewUrl) {
URL.revokeObjectURL(firstFramePreviewUrl);
}
if (lastFramePreviewUrl) {
URL.revokeObjectURL(lastFramePreviewUrl);
}
};
}, [referencePreviewUrl, firstFramePreviewUrl, lastFramePreviewUrl]);
function setPreviewUrl(setter: Dispatch<SetStateAction<string | null>>, file: File | null) {
setter((current) => {
if (current) URL.revokeObjectURL(current);
return file ? URL.createObjectURL(file) : null;
});
}
function handleReferenceSelect(file: File | null) {
referenceFileRef.current = file;
setPreviewUrl(setReferencePreviewUrl, file);
}
function handleFirstFrameSelect(file: File | null) {
firstFrameFileRef.current = file;
setPreviewUrl(setFirstFramePreviewUrl, file);
}
function handleLastFrameSelect(file: File | null) {
lastFrameFileRef.current = file;
setPreviewUrl(setLastFramePreviewUrl, file);
}
const mentionOptions = useMemo(() => buildMentionOptions(storyPresets as Array<{ id?: string; label?: string; description?: string }>), [storyPresets]);
const lobbyStoryPresets = useMemo(
() =>
(storyPresets as Array<{ id?: string; label?: string; description?: string; segment_prompts?: unknown }>)
.filter((preset) => typeof preset.id === "string" && typeof preset.label === "string")
.map((preset) => ({
id: String(preset.id),
label: String(preset.label),
description: typeof preset.description === "string" ? preset.description : undefined,
segmentCount: Array.isArray(preset.segment_prompts) ? preset.segment_prompts.length : undefined,
})),
[storyPresets],
);
const videoElRef = useRef<HTMLVideoElement | null>(null);
const archivedPlaybackElRef = useRef<HTMLVideoElement | null>(null);
const viewingModePlaybackStateRef = useRef<{
@@ -524,6 +530,65 @@ export default function Page() {
setRuntimeReady(true);
}, []);
function applyLobbyCapabilitiesBundle(bundle: LobbyCapabilitiesBundle) {
setLobbyCapabilitiesBundle(bundle);
const clamped = clampLobbySelectionToCapabilities({
capabilities: resolveModelCapabilities(bundle, creationModelId),
modelId: creationModelId,
modeId: creationModeId,
aspectRatio: creationAspectRatio,
resolution: creationResolution,
durationSec: creationDurationSec,
});
setCreationModelId(clamped.modelId);
setCreationModeId(clamped.modeId);
setCreationAspectRatio(clamped.aspectRatio);
setCreationResolution(clamped.resolution);
setCreationDurationSec(clamped.durationSec);
}
function handleCreationModelChange(modelId: CreationModelId) {
const clamped = clampLobbySelectionToCapabilities({
capabilities: resolveModelCapabilities(lobbyCapabilitiesBundle, modelId),
modelId,
modeId: creationModeId,
aspectRatio: creationAspectRatio,
resolution: creationResolution,
durationSec: creationDurationSec,
});
setCreationModelId(clamped.modelId);
setCreationModeId(clamped.modeId);
setCreationAspectRatio(clamped.aspectRatio);
setCreationResolution(clamped.resolution);
setCreationDurationSec(clamped.durationSec);
}
useEffect(() => {
if (!runtimeReady) return;
let cancelled = false;
void fetch("/creation-capabilities", {
headers: { Accept: "application/json" },
cache: "no-store",
})
.then(async (response) => {
if (!response.ok) return DEFAULT_LOBBY_CAPABILITIES_BUNDLE;
return parseLobbyCapabilitiesBundle(await response.json());
})
.then((bundle) => {
if (!cancelled) {
applyLobbyCapabilitiesBundle(bundle);
}
})
.catch(() => {
if (!cancelled) {
applyLobbyCapabilitiesBundle(DEFAULT_LOBBY_CAPABILITIES_BUNDLE);
}
});
return () => {
cancelled = true;
};
}, [runtimeReady]);
useEffect(() => {
if (!runtimeReady || initializedRef.current) return;
initializedRef.current = true;
@@ -1759,33 +1824,44 @@ export default function Page() {
resetPlaybackState();
}
function buildProjectInitPayload(type: "session_init_v2" | "project_init_v1") {
async function buildProjectInitPayload(type: "session_init_v2" | "project_init_v1") {
const segmentPrompts = getSessionInitPrompts();
setSeedPrompts(segmentPrompts);
const creationPayload = await buildCreationInitPayload({
modelId: creationModelId,
modeId: creationModeId,
aspectRatio: creationAspectRatio,
resolution: creationResolution,
durationSec: creationDurationSec,
referenceFile: referenceFileRef.current,
firstFrameFile: firstFrameFileRef.current,
lastFrameFile: lastFrameFileRef.current,
});
return {
type,
generation_mode: toGenerationMode(creationModeId),
preset_id: getInitialPresetId(),
preset_label: getInitialPresetLabel(),
curated_prompts: segmentPrompts,
initial_rollout_prompt: normalizeInitialPrompt(pendingInitialPromptRef.current),
initial_image: null,
single_clip_mode: false,
enhancement_enabled: sessionStore.get().enhancementEnabled,
auto_extension_enabled: sessionStore.get().autoExtensionEnabled,
loop_generation_enabled: sessionStore.get().loopGenerationEnabled,
...creationPayload,
};
}
function sendSessionInitMessage() {
async function sendSessionInitMessage() {
const ws = wsRef.current;
if (!ws) return;
ws.send(JSON.stringify(buildProjectInitPayload("session_init_v2")));
ws.send(JSON.stringify(await buildProjectInitPayload("session_init_v2")));
}
function sendProjectInitMessage() {
async function sendProjectInitMessage() {
const ws = wsRef.current;
if (!ws || ws.readyState !== WebSocket.OPEN) return;
ws.send(JSON.stringify(buildProjectInitPayload("project_init_v1")));
ws.send(JSON.stringify(await buildProjectInitPayload("project_init_v1")));
}
function sendEndProjectKeepSession() {
@@ -1822,6 +1898,9 @@ export default function Page() {
return;
}
const normalizedEvent = normalizeSocketMessage(decoded.data);
if (decoded.data?.type === "gpu_assigned" || decoded.data?.type === "ltx2_stream_start") {
applyEchoedCreationConfig(decoded.data);
}
await applyNormalizedSocketEvent(normalizedEvent, {
sessionStore,
promptWindowStore,
@@ -1864,7 +1943,12 @@ export default function Page() {
onOpen: () => {
opened = true;
sessionStore.patch({ connected: true, connecting: false });
sendSessionInitMessage();
void sendSessionInitMessage().catch((error) => {
console.error("Failed to send session init payload:", error);
recoverFailedSessionStart(
error instanceof Error ? error.message : "Failed to prepare session settings.",
);
});
},
onMessage: (event: MessageEvent) => {
wsMessageQueueRef.current = wsMessageQueueRef.current
@@ -1937,10 +2021,29 @@ export default function Page() {
}
}
function syncSessionCreationConfigFromLobby() {
setSessionCreationConfig({
modelId: creationModelId,
modeId: creationModeId,
aspectRatio: creationAspectRatio,
resolution: creationResolution,
durationSec: creationDurationSec,
});
}
function applyEchoedCreationConfig(data: unknown) {
const echoed = parseEchoedCreationConfig(data);
if (!echoed) {
return;
}
setSessionCreationConfig(echoed);
}
function beginProjectLocally({ force = false } = {}) {
if (!force && !canStartSession) return;
if (sessionStore.get().sessionStarted || sessionStore.get().projectResetPending) return false;
setTimeoutModalOpen(false);
syncSessionCreationConfigFromLobby();
// Unmute during the user gesture so iOS Safari permits audio playback.
setVideoMuted(false);
if (viewingProject) closeViewingProject();
@@ -1999,13 +2102,34 @@ export default function Page() {
}
async function joinSession({ force = false } = {}) {
const validationError = validateLobbyCreationSelection({
capabilities: activeModelCapabilities,
modelId: creationModelId,
modeId: creationModeId,
aspectRatio: creationAspectRatio,
resolution: creationResolution,
durationSec: creationDurationSec,
referenceFile: referenceFileRef.current,
firstFrameFile: firstFrameFileRef.current,
lastFrameFile: lastFrameFileRef.current,
});
if (validationError) {
showPreSessionNotice(validationError);
return;
}
if (
wsRef.current
&& wsRef.current.readyState === WebSocket.OPEN
&& sessionStore.get().connected
) {
if (!beginProjectLocally({ force })) return;
sendProjectInitMessage();
try {
await sendProjectInitMessage();
} catch (error) {
console.error("Failed to send project init payload:", error);
showPreSessionNotice(error instanceof Error ? error.message : "Failed to prepare session settings.");
}
return;
}
showPreSessionNotice("");
@@ -2027,7 +2151,12 @@ export default function Page() {
&& wsRef.current.readyState === WebSocket.OPEN
&& sessionStore.get().connected
) {
sendProjectInitMessage();
try {
await sendProjectInitMessage();
} catch (error) {
console.error("Failed to send project init payload:", error);
showPreSessionNotice(error instanceof Error ? error.message : "Failed to prepare session settings.");
}
return;
}
connectWebSocket();
@@ -2641,7 +2770,7 @@ export default function Page() {
/>
<Header timeLeft={headerTimeLeft} formatTime={formatTime} onToggleSidebar={() => setSidebarOpen((prev) => !prev)} />
<div className="relative flex flex-1 min-h-0 flex-col justify-center px-4 pb-2 sm:px-6 sm:pb-12">
<div className={cn("relative flex flex-1 min-h-0 flex-col", showActiveProject || isViewingMode ? "justify-center px-4 pb-2 sm:px-6 sm:pb-12" : "overflow-hidden")}>
{isViewingMode && (
<>
{viewingSelectedClip && (
@@ -2758,44 +2887,79 @@ export default function Page() {
/>
</section>
<AnimatePresence>
{!showActiveProject && (
<motion.div
key="hero-tagline"
initial={{ opacity: 0 }}
animate={{ opacity: 1 }}
exit={{ opacity: 0, transition: { duration: 0.2, ease: "easeIn" } }}
transition={{ duration: 0.5, ease: "easeOut" }}
className="pointer-events-none absolute inset-x-0 top-0 bottom-1/2 z-10 flex items-center justify-center px-4"
>
<HeroTagline />
</motion.div>
)}
</AnimatePresence>
<motion.div layout="position" className="mx-auto w-full max-w-2xl shrink-0" transition={{ type: "spring", stiffness: 200, damping: 25 }}>
<ChatBar
sessionStarted={sessionStarted as boolean}
rewritingSeedPrompts={rewritingSeedPrompts as boolean}
{!showActiveProject ? (
<CreationStudio
value={livePromptDraft as string}
disabled={projectResetPending as boolean}
isGenerating={loadingAnimation as boolean}
storyPresets={storyPresets as any[]}
continuationDraft={livePromptDraft as string}
canJoinSession={canStartSession}
canSubmitContinuation={canSubmitContinuation}
sessionExpired={sessionExpired as boolean}
sessionNotice={sessionNotice as string}
projectResetPending={projectResetPending as boolean}
canSubmit={canStartSession}
modelId={creationModelId}
modeId={creationModeId}
aspectRatio={creationAspectRatio}
resolution={creationResolution}
durationSec={creationDurationSec}
referencePreviewUrl={referencePreviewUrl}
firstFramePreviewUrl={firstFramePreviewUrl}
lastFramePreviewUrl={lastFramePreviewUrl}
mentionOptions={mentionOptions}
storyPresets={lobbyStoryPresets}
capabilities={activeModelCapabilities}
onValueChange={(value) => sessionStore.patch({ livePromptDraft: value })}
onSubmit={() => void joinSession()}
onKeyDown={handleLivePromptKeydown}
onModelChange={handleCreationModelChange}
onModeChange={setCreationModeId}
onAspectRatioChange={setCreationAspectRatio}
onResolutionChange={setCreationResolution}
onDurationChange={setCreationDurationSec}
onReferenceSelect={handleReferenceSelect}
onFirstFrameSelect={handleFirstFrameSelect}
onLastFrameSelect={handleLastFrameSelect}
onPresetGenerate={handlePresetGenerate}
onContinuationInput={handleLivePromptInput}
onContinuationKeydown={handleLivePromptKeydown}
onGenerate={joinSession}
onSubmitContinuation={submitLivePrompt}
onLeave={leaveSession}
onStartNewProject={handleStartNewProject}
onSpeechTranscript={handleLivePromptSpeechTranscript}
onSpeechInterimChange={handleLivePromptSpeechInterim}
onOpenProjects={() => setSidebarOpen(true)}
/>
</motion.div>
) : (
<motion.div layout="position" className="mx-auto w-full max-w-2xl shrink-0" transition={{ type: "spring", stiffness: 200, damping: 25 }}>
<ChatBar
sessionStarted={sessionStarted as boolean}
rewritingSeedPrompts={rewritingSeedPrompts as boolean}
isGenerating={loadingAnimation as boolean}
storyPresets={storyPresets as any[]}
continuationDraft={livePromptDraft as string}
canJoinSession={canStartSession}
canSubmitContinuation={canSubmitContinuation}
sessionExpired={sessionExpired as boolean}
sessionNotice={sessionNotice as string}
projectResetPending={projectResetPending as boolean}
sessionCreationConfig={sessionCreationConfig}
configPillsReadOnly
onSessionModelChange={(modelId) => setSessionCreationConfig((current) => ({ ...current, modelId }))}
onSessionModeChange={(modeId) => setSessionCreationConfig((current) => ({ ...current, modeId }))}
onSessionAspectRatioChange={(aspectRatio) => setSessionCreationConfig((current) => ({ ...current, aspectRatio }))}
onSessionResolutionChange={(resolution) => setSessionCreationConfig((current) => ({ ...current, resolution }))}
onSessionDurationChange={(durationSec) => setSessionCreationConfig((current) => ({ ...current, durationSec }))}
onPresetGenerate={handlePresetGenerate}
onContinuationInput={handleLivePromptInput}
onContinuationKeydown={handleLivePromptKeydown}
onGenerate={joinSession}
onSubmitContinuation={submitLivePrompt}
onLeave={leaveSession}
onStartNewProject={handleStartNewProject}
onSpeechTranscript={handleLivePromptSpeechTranscript}
onSpeechInterimChange={handleLivePromptSpeechInterim}
/>
</motion.div>
)}
{sessionNotice && !showActiveProject && (
<div className="mx-auto mt-2 w-full max-w-3xl px-4">
<div className="rounded-xl border border-rose-500/20 bg-rose-500/10 px-4 py-2.5 text-center text-xs text-rose-700 dark:text-rose-300">
{sessionNotice}
</div>
</div>
)}
</div>
</div>
</main>
+62 -197
View File
@@ -2,10 +2,13 @@
import React, { useRef, useState, useCallback, useEffect } from "react";
import Image from "next/image";
import { Film, ArrowUp, X, Loader2, ArrowLeft } from "lucide-react";
import { ArrowUp, X, Loader2, ArrowLeft } from "lucide-react";
import { Button } from "@/components/ui/button";
import LeaveSessionModal, { shouldShowLeaveWarning } from "@/components/LeaveSessionModal";
import PresetQuickLaunchRail from "@/components/creation/PresetQuickLaunchRail";
import SessionCreationConfigPills, { type SessionCreationConfig } from "@/components/creation/SessionCreationConfigPills";
import SpeechToTextButton from "@/components/SpeechToTextButton";
import type { AspectRatioId, CreationModeId, CreationModelId, ResolutionId } from "@/lib/creationConfig";
import { cn } from "@/lib/utils";
const PROMPT_MAX_LENGTH = 500;
@@ -32,6 +35,13 @@ interface Props {
onBackFromViewing?: () => void;
onSpeechTranscript?: (text: string) => void;
onSpeechInterimChange?: (text: string) => void;
sessionCreationConfig?: SessionCreationConfig | null;
configPillsReadOnly?: boolean;
onSessionModelChange?: (modelId: CreationModelId) => void;
onSessionModeChange?: (modeId: CreationModeId) => void;
onSessionAspectRatioChange?: (aspectRatio: AspectRatioId) => void;
onSessionResolutionChange?: (resolution: ResolutionId) => void;
onSessionDurationChange?: (durationSec: number) => void;
}
export default function ChatBar({
@@ -56,6 +66,13 @@ export default function ChatBar({
onBackFromViewing = () => {},
onSpeechTranscript,
onSpeechInterimChange,
sessionCreationConfig = null,
configPillsReadOnly = false,
onSessionModelChange,
onSessionModeChange,
onSessionAspectRatioChange,
onSessionResolutionChange,
onSessionDurationChange,
}: Props) {
const [sttBusy, setSttBusy] = useState(false);
const [leaveModalOpen, setLeaveModalOpen] = useState(false);
@@ -66,134 +83,11 @@ export default function ChatBar({
: isBusy
? "Generating video\u2026"
: !sessionStarted
? "What video are you imagining?"
? "Describe your video"
: "What do you want to edit?";
const actionLabel = !sessionStarted ? "Generate" : "Rewrite rollout";
const inputRef = useRef<HTMLTextAreaElement>(null);
const scrollRef = useRef<HTMLDivElement>(null);
const [canScrollLeft, setCanScrollLeft] = useState(false);
const [canScrollRight, setCanScrollRight] = useState(false);
const [presetRailDragging, setPresetRailDragging] = useState(false);
const presetDragStateRef = useRef({
pointerId: null as number | null,
startX: 0,
startScrollLeft: 0,
moved: false,
});
const suppressPresetClickRef = useRef(false);
const updateScrollState = useCallback(() => {
const el = scrollRef.current;
if (!el) return;
setCanScrollLeft(el.scrollLeft > 2);
setCanScrollRight(el.scrollLeft + el.clientWidth < el.scrollWidth - 2);
}, []);
const handlePresetWheel = useCallback(
(event: React.WheelEvent<HTMLDivElement>) => {
const el = scrollRef.current;
if (!el) return;
if (el.scrollWidth <= el.clientWidth + 1) return;
const dominantDelta = Math.abs(event.deltaX) > Math.abs(event.deltaY)
? event.deltaX
: event.deltaY;
if (!dominantDelta) return;
const maxScrollLeft = Math.max(el.scrollWidth - el.clientWidth, 0);
const nextScrollLeft = Math.min(
Math.max(el.scrollLeft + dominantDelta, 0),
maxScrollLeft,
);
if (nextScrollLeft === el.scrollLeft) return;
event.preventDefault();
el.scrollLeft = nextScrollLeft;
updateScrollState();
},
[updateScrollState],
);
const finishPresetDrag = useCallback(() => {
presetDragStateRef.current = {
pointerId: null,
startX: 0,
startScrollLeft: 0,
moved: false,
};
setPresetRailDragging(false);
}, []);
const handlePresetPointerDown = useCallback(
(event: React.PointerEvent<HTMLDivElement>) => {
const el = scrollRef.current;
if (!el) return;
if (event.pointerType !== "mouse" || event.button !== 0) return;
if (el.scrollWidth <= el.clientWidth + 1) return;
suppressPresetClickRef.current = false;
presetDragStateRef.current = {
pointerId: event.pointerId,
startX: event.clientX,
startScrollLeft: el.scrollLeft,
moved: false,
};
},
[],
);
const handlePresetPointerMove = useCallback(
(event: React.PointerEvent<HTMLDivElement>) => {
const el = scrollRef.current;
const dragState = presetDragStateRef.current;
if (!el || dragState.pointerId !== event.pointerId) return;
const deltaX = event.clientX - dragState.startX;
if (!dragState.moved && Math.abs(deltaX) > 4) {
dragState.moved = true;
suppressPresetClickRef.current = true;
setPresetRailDragging(true);
el.setPointerCapture?.(event.pointerId);
}
if (!dragState.moved) return;
event.preventDefault();
const maxScrollLeft = Math.max(el.scrollWidth - el.clientWidth, 0);
el.scrollLeft = Math.min(
Math.max(dragState.startScrollLeft - deltaX, 0),
maxScrollLeft,
);
updateScrollState();
},
[updateScrollState],
);
const handlePresetPointerUp = useCallback(
(event: React.PointerEvent<HTMLDivElement>) => {
const el = scrollRef.current;
if (!el || presetDragStateRef.current.pointerId !== event.pointerId) return;
if (el.hasPointerCapture?.(event.pointerId)) {
el.releasePointerCapture(event.pointerId);
}
finishPresetDrag();
},
[finishPresetDrag],
);
const handlePresetClickCapture = useCallback(
(event: React.MouseEvent<HTMLDivElement>) => {
if (!suppressPresetClickRef.current) return;
suppressPresetClickRef.current = false;
event.preventDefault();
event.stopPropagation();
},
[],
);
useEffect(() => {
updateScrollState();
}, [storyPresets, updateScrollState]);
useEffect(() => {
if (!isBusy && !sttBusy && !window.matchMedia("(pointer: coarse)").matches) {
@@ -239,7 +133,7 @@ export default function ChatBar({
<div className="flex flex-col items-center gap-3 rounded-2xl border border-border bg-card/80 px-6 py-4 text-center shadow-md backdrop-blur-sm">
<div className="flex flex-col gap-1">
<p className="text-sm font-semibold text-foreground">View-only project</p>
<p className="max-w-md text-xs text-muted-foreground">Project sessions are currently limited to 5 minutes. Start a new project to create more videos.</p>
<p className="max-w-md text-xs text-muted-foreground">Sessions are limited to 5 minutes. Start a new project to keep creating.</p>
</div>
<div className="mt-1 flex items-center gap-2">
<Button onClick={onBackFromViewing} variant="outline" size="sm" className="gap-1.5 rounded-full px-4">
@@ -261,7 +155,7 @@ export default function ChatBar({
<div className="flex flex-col items-center gap-3 rounded-2xl border border-border bg-card/80 px-8 py-5 text-center shadow-md backdrop-blur-sm">
<div className="flex flex-col gap-1">
<p className="text-sm font-semibold text-foreground">Session ended</p>
<p className="max-w-xs text-xs text-muted-foreground">Each project currently has a 5-minute session. Start a new project to continue creating videos.</p>
<p className="max-w-xs text-xs text-muted-foreground">Sessions are limited to 5 minutes. Start a new project to continue.</p>
</div>
<div className="mt-1 flex items-center gap-2">
<Button onClick={onStartNewProject} size="sm" className="rounded-full px-5">
@@ -280,51 +174,8 @@ export default function ChatBar({
return (
<section className="mx-auto flex w-full max-w-2xl shrink-0 flex-col gap-4">
{storyPresets.length > 0 && !sessionStarted && (
<div className={cn("relative transition-opacity duration-200", isGenerating && "pointer-events-none opacity-40")}>
<div
ref={scrollRef}
onScroll={updateScrollState}
onWheel={handlePresetWheel}
onPointerDown={handlePresetPointerDown}
onPointerMove={handlePresetPointerMove}
onPointerUp={handlePresetPointerUp}
onPointerCancel={handlePresetPointerUp}
onLostPointerCapture={finishPresetDrag}
onClickCapture={handlePresetClickCapture}
className={cn(
"scrollbar-hidden flex gap-3 overflow-x-auto px-1 select-none",
presetRailDragging ? "cursor-grabbing" : "cursor-grab",
)}
>
{storyPresets.map((preset) => (
<button
key={preset.id}
type="button"
disabled={isGenerating}
onClick={() => onPresetGenerate(preset.id)}
className="flex flex-col sm:flex-row items-start gap-1.5 shrink-0 rounded-xl border p-2.5 text-left backdrop-blur-sm transition-colors max-w-42 sm:max-w-[215px] border-input bg-card/80 text-muted-foreground hover:bg-slate-200/60 hover:border-slate-400 hover:text-slate-700 dark:bg-slate-800/80 dark:text-slate-300 dark:hover:bg-slate-700/50 dark:hover:border-slate-500 dark:hover:text-slate-200"
>
<Film className="mt-0.5 size-4 shrink-0 opacity-60" />
<span className="flex flex-col gap-1 min-w-0">
<span className="text-[14px] font-medium line-clamp-1">{preset.label}</span>
{preset.description && <span className="text-xs leading-tight opacity-70 line-clamp-3 sm:line-clamp-2">{preset.description}</span>}
</span>
</button>
))}
</div>
<div
className={cn("pointer-events-none absolute inset-y-0 left-0 w-8 bg-background transition-opacity duration-150", canScrollLeft ? "opacity-100" : "opacity-0")}
style={{ maskImage: "linear-gradient(to right, black, transparent)", WebkitMaskImage: "linear-gradient(to right, black, transparent)" }}
aria-hidden="true"
/>
<div
className={cn("pointer-events-none absolute inset-y-0 right-0 w-8 bg-background transition-opacity duration-150", canScrollRight ? "opacity-100" : "opacity-0")}
style={{ maskImage: "linear-gradient(to left, black, transparent)", WebkitMaskImage: "linear-gradient(to left, black, transparent)" }}
aria-hidden="true"
/>
</div>
{!sessionStarted && (
<PresetQuickLaunchRail storyPresets={storyPresets} disabled={isGenerating} onPresetGenerate={onPresetGenerate} />
)}
{sessionNotice && (
@@ -342,17 +193,30 @@ export default function ChatBar({
{projectResetPending && sessionStarted && (
<div className="rounded-xl border border-sky-500/20 bg-sky-500/10 px-4 py-2.5 text-center text-xs text-sky-700 dark:text-sky-300">
Starting a new project after the current shot finishes. Your GPU session stays active.
Starting a new project when this shot finishes. GPU session stays open.
</div>
)}
<div
className={cn(
"flex min-w-0 items-center gap-1.5 rounded-4xl border py-2.5 pl-5 pr-2.5 shadow-md backdrop-blur-sm transition-all duration-200",
"flex min-w-0 flex-col gap-2 rounded-4xl border py-2.5 pl-5 pr-2.5 shadow-md backdrop-blur-sm transition-all duration-200",
isBusy ? "border-input/60 bg-card/40" : "border-input bg-card/65",
)}
>
<textarea
{sessionStarted && sessionCreationConfig && (
<SessionCreationConfigPills
{...sessionCreationConfig}
disabled={isBusy}
readOnly={configPillsReadOnly}
onModelChange={onSessionModelChange}
onModeChange={onSessionModeChange}
onAspectRatioChange={onSessionAspectRatioChange}
onResolutionChange={onSessionResolutionChange}
onDurationChange={onSessionDurationChange}
/>
)}
<div className="flex min-w-0 items-center gap-1.5">
<textarea
ref={inputRef}
id="continuation-prompt"
aria-label="Continuation prompt"
@@ -367,36 +231,37 @@ export default function ChatBar({
"min-w-0 flex-1 resize-none bg-transparent text-foreground outline-none placeholder:text-muted-foreground transition-opacity duration-200 scrollbar-thin leading-snug",
(isBusy || sttBusy) && "cursor-not-allowed opacity-50",
)}
/>
{onSpeechTranscript && <SpeechToTextButton disabled={isBusy} onTranscript={onSpeechTranscript} onInterimChange={onSpeechInterimChange} onBusyChange={setSttBusy} />}
{!sessionStarted ? (
<Button
aria-label={actionLabel}
title={actionLabel}
onClick={onGenerate}
disabled={!canJoinSession || isGenerating || !continuationDraft.trim()}
size="icon-sm"
className="shrink-0 rounded-full"
>
{showSpinner ? <Loader2 className="size-5 animate-spin" /> : <ArrowUp className="size-5" />}
</Button>
) : (
<>
/>
{onSpeechTranscript && <SpeechToTextButton disabled={isBusy} onTranscript={onSpeechTranscript} onInterimChange={onSpeechInterimChange} onBusyChange={setSttBusy} />}
{!sessionStarted ? (
<Button
aria-label={actionLabel}
title={actionLabel}
onClick={onSubmitContinuation}
disabled={!canSubmitContinuation || showSpinner || projectResetPending || !continuationDraft.trim()}
onClick={onGenerate}
disabled={!canJoinSession || isGenerating || !continuationDraft.trim()}
size="icon-sm"
className="shrink-0 rounded-full"
>
{showSpinner ? <Loader2 className="size-5 animate-spin" /> : <ArrowUp className="size-5" />}
</Button>
<Button variant="outline" aria-label="Leave" title="Leave" onClick={() => { if (shouldShowLeaveWarning()) setLeaveModalOpen(true); else onLeave(); }} disabled={isGenerating || projectResetPending} size="icon-sm" className="shrink-0 rounded-full">
<X className="size-5" />
</Button>
</>
)}
) : (
<>
<Button
aria-label={actionLabel}
title={actionLabel}
onClick={onSubmitContinuation}
disabled={!canSubmitContinuation || showSpinner || projectResetPending || !continuationDraft.trim()}
size="icon-sm"
className="shrink-0 rounded-full"
>
{showSpinner ? <Loader2 className="size-5 animate-spin" /> : <ArrowUp className="size-5" />}
</Button>
<Button variant="outline" aria-label="Leave" title="Leave" onClick={() => { if (shouldShowLeaveWarning()) setLeaveModalOpen(true); else onLeave(); }} disabled={isGenerating || projectResetPending} size="icon-sm" className="shrink-0 rounded-full">
<X className="size-5" />
</Button>
</>
)}
</div>
</div>
<p className="px-2 text-center text-[11px] text-muted-foreground">
LLM powered by{" "}
@@ -0,0 +1,95 @@
"use client";
import { Fragment, useEffect, useRef } from "react";
const HERO_WAVE_LIGHT = ["#2A4A98", "#4878E5", "#6FA0F2", "#B0BCC8", "#E8D99E", "#D8C844", "#C2A620"];
const HERO_WAVE_DARK = ["#143468", "#1E58B8", "#3892F0", "#80B8E8", "#B8D0EA", "#E2D498", "#DABB50"];
const HERO_TEXT = "Direct scenes in seconds";
export default function HeroTagline() {
const ref = useRef<HTMLHeadingElement>(null);
useEffect(() => {
const el = ref.current;
if (!el) return;
let rafId = 0;
function play() {
const chars = el!.querySelectorAll<HTMLSpanElement>("[data-char]");
if (!chars.length) return;
cancelAnimationFrame(rafId);
const isDark = document.documentElement.classList.contains("dark");
const colors = isDark ? HERO_WAVE_DARK : HERO_WAVE_LIGHT;
const waveLen = 10;
const total = chars.length + waveLen;
const duration = 1200;
const maxBlur = 3.5;
const start = performance.now();
function tick() {
const t = Math.min((performance.now() - start) / duration, 1);
const pos = t * total;
chars.forEach((ch, i) => {
const rel = pos - i;
if (rel >= 0 && rel < waveLen) {
const norm = rel / waveLen;
const ci = Math.floor(norm * colors.length);
ch.style.color = colors[Math.min(colors.length - 1, ci)];
let blur = 0;
if (norm < 0.25) {
blur = maxBlur * (1 - norm / 0.25);
} else if (norm > 0.75) {
blur = maxBlur * ((norm - 0.75) / 0.25);
}
ch.style.filter = blur > 0.1 ? `blur(${blur.toFixed(1)}px)` : "";
} else {
ch.style.color = "";
ch.style.filter = "";
}
});
if (t < 1) {
rafId = requestAnimationFrame(tick);
} else {
chars.forEach((ch) => {
ch.style.color = "";
ch.style.filter = "";
});
}
}
rafId = requestAnimationFrame(tick);
}
const initialDelay = setTimeout(play, 400);
const interval = setInterval(play, 5000);
return () => {
clearTimeout(initialDelay);
clearInterval(interval);
cancelAnimationFrame(rafId);
};
}, []);
return (
<h1 ref={ref} className="text-balance text-center text-3xl font-medium text-[#343537] dark:text-[#FAFAFB] sm:text-4xl">
{HERO_TEXT.split(" ").map((word, wi) => (
<Fragment key={wi}>
{wi > 0 && (
<span data-char className="transition-[color,filter] duration-150">
{" "}
</span>
)}
<span className="inline-flex">
{word.split("").map((char, ci) => (
<span key={ci} data-char className="inline-block transition-[color,filter] duration-150">
{char}
</span>
))}
</span>
</Fragment>
))}
</h1>
);
}
@@ -0,0 +1,66 @@
"use client";
import React from "react";
import { FolderOpen, Home, Sparkles } from "lucide-react";
import { cn } from "@/lib/utils";
export type AppNavSection = "explore" | "create" | "assets";
interface AppNavRailProps {
activeSection?: AppNavSection;
onSectionChange?: (section: AppNavSection) => void;
onOpenProjects?: () => void;
className?: string;
}
const NAV_ITEMS: Array<{ id: AppNavSection; label: string; icon: typeof Home }> = [
{ id: "explore", label: "Explore", icon: Home },
{ id: "create", label: "Create", icon: Sparkles },
{ id: "assets", label: "Assets", icon: FolderOpen },
];
export default function AppNavRail({
activeSection = "create",
onSectionChange = () => {},
onOpenProjects,
className,
}: AppNavRailProps) {
return (
<aside
className={cn(
"hidden shrink-0 flex-col items-center gap-2 border-r border-border/40 bg-background/30 px-2.5 py-5 lg:flex",
className,
)}
aria-label="Primary navigation"
>
{NAV_ITEMS.map((item) => {
const Icon = item.icon;
const isActive = item.id === activeSection;
return (
<button
key={item.id}
type="button"
aria-label={item.label}
aria-current={isActive ? "page" : undefined}
onClick={() => {
if (item.id === "assets") {
onOpenProjects?.();
}
onSectionChange(item.id);
}}
className={cn(
"studio-control studio-control-press flex w-[4.5rem] min-h-11 flex-col items-center gap-1 rounded-xl px-2 py-2.5 text-[10px] font-medium tracking-wide",
isActive
? "bg-secondary/90 text-foreground shadow-sm ring-1 ring-border/60"
: "text-muted-foreground hover-capable:hover:bg-secondary/50 hover-capable:hover:text-foreground",
)}
>
<Icon className={cn("size-[18px]", isActive && "text-accent-blue")} />
{item.label}
</button>
);
})}
</aside>
);
}
@@ -0,0 +1,24 @@
"use client";
import React from "react";
import { cn } from "@/lib/utils";
export default function ConfigPill({
children,
className,
...props
}: React.ButtonHTMLAttributes<HTMLButtonElement>) {
return (
<button
type="button"
className={cn(
"studio-control studio-control-press studio-hover-surface inline-flex h-9 min-h-9 shrink-0 items-center gap-1 rounded-full border border-border/50 bg-background/80 px-2.5 text-[11px] font-medium text-foreground/90",
className,
)}
{...props}
>
{children}
</button>
);
}
@@ -0,0 +1,466 @@
"use client";
import React, { useMemo, useRef, useState } from "react";
import { ArrowUp, Box, ChevronDown, Clock, Monitor, Wand2 } from "lucide-react";
import ConfigPill from "@/components/creation/ConfigPill";
import HeroTagline from "@/components/HeroTagline";
import ReferenceUploadSlot from "@/components/creation/ReferenceUploadSlot";
import { Button } from "@/components/ui/button";
import {
DropdownMenu,
DropdownMenuContent,
DropdownMenuItem,
DropdownMenuLabel,
DropdownMenuSeparator,
DropdownMenuTrigger,
} from "@/components/ui/dropdown-menu";
import { Popover, PopoverContent, PopoverTrigger } from "@/components/ui/popover";
import { Slider } from "@/components/ui/slider";
import SpeechToTextButton from "@/components/SpeechToTextButton";
import {
ASPECT_RATIOS,
CREATION_MODELS,
CREATION_MODES,
RESOLUTIONS,
UNSUPPORTED_CREATION_MODES,
UNSUPPORTED_RESOLUTIONS,
modeRequiresReference,
modeUsesDualFrames,
type AspectRatioId,
type CreationModeId,
type CreationModelId,
type MentionOption,
type ResolutionId,
formatDurationLabel,
formatResolutionLabel,
} from "@/lib/creationConfig";
import {
DEFAULT_LOBBY_CAPABILITIES_BUNDLE,
isSupportedCreationMode,
isSupportedResolution,
resolveModelCapabilities,
unsupportedModeNotice,
type LobbyCreationCapabilities,
} from "@/lib/creationCapabilities";
import { cn } from "@/lib/utils";
const PROMPT_MAX_LENGTH = 500;
interface CreationComposerProps {
value: string;
disabled?: boolean;
isGenerating?: boolean;
canSubmit?: boolean;
modelId: CreationModelId;
modeId: CreationModeId;
aspectRatio: AspectRatioId;
resolution: ResolutionId;
durationSec: number;
referencePreviewUrl?: string | null;
firstFramePreviewUrl?: string | null;
lastFramePreviewUrl?: string | null;
mentionOptions?: MentionOption[];
onValueChange: (value: string) => void;
onSubmit: () => void;
onKeyDown?: (event: React.KeyboardEvent<HTMLTextAreaElement>) => void;
onModelChange: (modelId: CreationModelId) => void;
onModeChange: (modeId: CreationModeId) => void;
onAspectRatioChange: (aspectRatio: AspectRatioId) => void;
onResolutionChange: (resolution: ResolutionId) => void;
onDurationChange: (durationSec: number) => void;
onReferenceSelect?: (file: File | null) => void;
onFirstFrameSelect?: (file: File | null) => void;
onLastFrameSelect?: (file: File | null) => void;
onSpeechTranscript?: (text: string) => void;
onSpeechInterimChange?: (text: string) => void;
capabilities?: LobbyCreationCapabilities;
}
export default function CreationComposer({
value,
disabled = false,
isGenerating = false,
canSubmit = false,
modelId,
modeId,
aspectRatio,
resolution,
durationSec,
referencePreviewUrl = null,
firstFramePreviewUrl = null,
lastFramePreviewUrl = null,
mentionOptions = [],
onValueChange,
onSubmit,
onKeyDown,
onModelChange,
onModeChange,
onAspectRatioChange,
onResolutionChange,
onDurationChange,
onReferenceSelect,
onFirstFrameSelect,
onLastFrameSelect,
onSpeechTranscript,
onSpeechInterimChange,
capabilities = resolveModelCapabilities(DEFAULT_LOBBY_CAPABILITIES_BUNDLE, modelId),
}: CreationComposerProps) {
const inputRef = useRef<HTMLTextAreaElement>(null);
const [sttBusy, setSttBusy] = useState(false);
const [mentionQuery, setMentionQuery] = useState("");
const [mentionOpen, setMentionOpen] = useState(false);
const [mentionStart, setMentionStart] = useState<number | null>(null);
const availableModels = useMemo(
() => CREATION_MODELS.filter((model) => capabilities.model_ids.includes(model.id)),
[capabilities.model_ids],
);
const availableModes = useMemo(
() => CREATION_MODES.filter((mode) => isSupportedCreationMode(mode.id, capabilities)),
[capabilities],
);
const unavailableModes = useMemo(
() =>
UNSUPPORTED_CREATION_MODES.filter(
(mode) => unsupportedModeNotice(mode.id, capabilities) !== null,
),
[capabilities],
);
const availableAspectRatios = useMemo(
() => ASPECT_RATIOS.filter((ratio) => capabilities.aspect_ratios.includes(ratio)),
[capabilities.aspect_ratios],
);
const availableResolutions = useMemo(
() => RESOLUTIONS.filter((item) => isSupportedResolution(item, capabilities)),
[capabilities],
);
const unavailableResolutions = useMemo(
() => UNSUPPORTED_RESOLUTIONS.filter((item) => !isSupportedResolution(item, capabilities)),
[capabilities],
);
const durationMin = capabilities.duration_sec[0] ?? 5;
const durationMax = capabilities.duration_sec[capabilities.duration_sec.length - 1] ?? 15;
const selectedModel = availableModels.find((model) => model.id === modelId) ?? availableModels[0];
const selectedMode = availableModes.find((mode) => mode.id === modeId) ?? availableModes[0];
const usesDualFrames = modeUsesDualFrames(modeId);
const requiresReference = modeRequiresReference(modeId);
const referenceMissing = requiresReference && !referencePreviewUrl;
const submitDisabled = !canSubmit || disabled || isGenerating || !value.trim() || referenceMissing;
const filteredMentions = useMemo(() => {
const query = mentionQuery.trim().toLowerCase();
if (!query) return mentionOptions.slice(0, 6);
return mentionOptions
.filter((option) => option.label.toLowerCase().includes(query) || option.description?.toLowerCase().includes(query))
.slice(0, 6);
}, [mentionOptions, mentionQuery]);
function autoResize() {
const el = inputRef.current;
if (!el) return;
el.style.height = "auto";
const lineHeight = parseFloat(getComputedStyle(el).lineHeight) || 22;
const maxHeight = lineHeight * 4;
el.style.height = `${Math.min(el.scrollHeight, maxHeight)}px`;
el.style.overflowY = el.scrollHeight > maxHeight ? "auto" : "hidden";
}
function updateMentionState(nextValue: string, cursorPosition: number) {
const beforeCursor = nextValue.slice(0, cursorPosition);
const atIndex = beforeCursor.lastIndexOf("@");
if (atIndex === -1 || (atIndex > 0 && !/\s/.test(beforeCursor[atIndex - 1] ?? ""))) {
setMentionOpen(false);
setMentionStart(null);
setMentionQuery("");
return;
}
const query = beforeCursor.slice(atIndex + 1);
if (/\s/.test(query)) {
setMentionOpen(false);
setMentionStart(null);
setMentionQuery("");
return;
}
setMentionStart(atIndex);
setMentionQuery(query);
setMentionOpen(true);
}
function insertMention(option: MentionOption) {
if (mentionStart === null) return;
const before = value.slice(0, mentionStart);
const after = value.slice(inputRef.current?.selectionStart ?? value.length);
const mentionText = `@${option.label} `;
const nextValue = `${before}${mentionText}${after}`.slice(0, PROMPT_MAX_LENGTH);
onValueChange(nextValue);
setMentionOpen(false);
setMentionStart(null);
setMentionQuery("");
requestAnimationFrame(() => {
const el = inputRef.current;
if (!el) return;
const cursor = before.length + mentionText.length;
el.focus();
el.setSelectionRange(cursor, cursor);
autoResize();
});
}
function handleInputChange(event: React.ChangeEvent<HTMLTextAreaElement>) {
const nextValue = event.target.value.slice(0, PROMPT_MAX_LENGTH);
onValueChange(nextValue);
updateMentionState(nextValue, event.target.selectionStart ?? nextValue.length);
requestAnimationFrame(autoResize);
}
function handleKeyDown(event: React.KeyboardEvent<HTMLTextAreaElement>) {
if (mentionOpen && filteredMentions.length > 0) {
if (event.key === "Tab" || (event.key === "Enter" && !event.shiftKey)) {
event.preventDefault();
insertMention(filteredMentions[0]);
return;
}
if (event.key === "Escape") {
setMentionOpen(false);
return;
}
}
onKeyDown?.(event);
}
return (
<section className="mx-auto flex w-full max-w-3xl flex-col gap-5">
<HeroTagline />
<div className="rounded-[32px] border border-border/40 bg-secondary/95 p-4 shadow-[0_24px_80px_-32px_rgba(0,0,0,0.72)] backdrop-blur-xl sm:p-5">
<div className="flex gap-3.5">
{usesDualFrames ? (
<div className="flex shrink-0 gap-2">
<ReferenceUploadSlot
label="Asset"
sublabel="First"
previewUrl={firstFramePreviewUrl}
disabled={disabled}
onSelect={onFirstFrameSelect}
/>
<ReferenceUploadSlot
label="Asset"
sublabel="Last"
previewUrl={lastFramePreviewUrl}
disabled={disabled}
onSelect={onLastFrameSelect}
/>
</div>
) : (
<ReferenceUploadSlot
label="Reference"
previewUrl={referencePreviewUrl}
required={requiresReference}
optional={!requiresReference}
disabled={disabled}
onSelect={onReferenceSelect}
/>
)}
<div className="relative min-w-0 flex-1">
<textarea
ref={inputRef}
id="continuation-prompt"
aria-label="Continuation prompt"
value={value}
onChange={handleInputChange}
onKeyDown={handleKeyDown}
onClick={(event) => updateMentionState(value, event.currentTarget.selectionStart ?? value.length)}
placeholder="Describe your video or mention elements"
disabled={disabled || sttBusy}
rows={3}
className={cn(
"min-h-[92px] w-full resize-none bg-transparent px-0.5 text-base leading-6 text-foreground outline-none placeholder:text-muted-foreground/80 sm:text-sm",
(disabled || sttBusy) && "cursor-not-allowed opacity-50",
)}
/>
{mentionOpen && filteredMentions.length > 0 && (
<div className="absolute left-0 right-0 top-full z-20 mt-2 overflow-hidden rounded-2xl border border-border bg-popover/95 p-1 shadow-xl backdrop-blur-md">
<p className="px-2 py-1 text-[11px] font-medium uppercase tracking-wide text-muted-foreground">Mention</p>
{filteredMentions.map((option) => (
<button
key={option.id}
type="button"
onMouseDown={(event) => {
event.preventDefault();
insertMention(option);
}}
className="studio-control studio-hover-surface flex w-full items-start gap-2 rounded-xl px-2.5 py-2 text-left"
>
<span className="mt-0.5 rounded-md bg-accent px-1.5 py-0.5 text-[10px] font-semibold uppercase tracking-wide text-muted-foreground">
{option.kind}
</span>
<span className="min-w-0">
<span className="block truncate text-sm font-medium text-foreground">{option.label}</span>
{option.description && <span className="block truncate text-xs text-muted-foreground">{option.description}</span>}
</span>
</button>
))}
</div>
)}
</div>
</div>
<div className="mt-4 flex flex-wrap items-center gap-1.5 rounded-2xl bg-muted/35 p-1.5 ring-1 ring-border/25">
<DropdownMenu>
<DropdownMenuTrigger asChild>
<ConfigPill disabled={disabled}>
<Box className="size-3.5" />
{selectedModel.label}
<ChevronDown className="size-3 opacity-60" />
</ConfigPill>
</DropdownMenuTrigger>
<DropdownMenuContent align="start" className="w-72">
<DropdownMenuLabel>Model</DropdownMenuLabel>
<DropdownMenuSeparator />
{availableModels.map((model) => (
<DropdownMenuItem key={model.id} onClick={() => onModelChange(model.id)} className="flex-col items-start gap-1 py-2.5">
<span className="flex items-center gap-2 text-sm font-medium">
{model.label}
{model.badge && <span className="rounded-full bg-accent-blue/15 px-1.5 py-0.5 text-[10px] text-accent-blue">{model.badge}</span>}
</span>
<span className="text-xs text-muted-foreground">{model.description}</span>
</DropdownMenuItem>
))}
</DropdownMenuContent>
</DropdownMenu>
<DropdownMenu>
<DropdownMenuTrigger asChild>
<ConfigPill disabled={disabled}>
<Wand2 className="size-3.5" />
{selectedMode.label}
<ChevronDown className="size-3 opacity-60" />
</ConfigPill>
</DropdownMenuTrigger>
<DropdownMenuContent align="start" className="w-64">
<DropdownMenuLabel>Mode</DropdownMenuLabel>
<DropdownMenuSeparator />
{availableModes.map((mode) => (
<DropdownMenuItem key={mode.id} onClick={() => onModeChange(mode.id)} className="flex-col items-start gap-1 py-2.5">
<span className="text-sm font-medium">{mode.label}</span>
<span className="text-xs text-muted-foreground">{mode.description}</span>
</DropdownMenuItem>
))}
{unavailableModes.length > 0 && <DropdownMenuSeparator />}
{unavailableModes.map((mode) => (
<DropdownMenuItem key={mode.id} disabled className="flex-col items-start gap-1 py-2.5 opacity-60">
<span className="text-sm font-medium">{mode.label}</span>
<span className="text-xs text-muted-foreground">
{unsupportedModeNotice(mode.id, capabilities) ?? mode.description}
</span>
</DropdownMenuItem>
))}
</DropdownMenuContent>
</DropdownMenu>
<Popover>
<PopoverTrigger asChild>
<ConfigPill disabled={disabled}>
<Monitor className="size-3.5" />
{aspectRatio} {formatResolutionLabel(resolution)}
</ConfigPill>
</PopoverTrigger>
<PopoverContent align="start" className="w-80">
<p className="mb-3 text-xs font-medium text-muted-foreground">Aspect ratio</p>
<div className="grid grid-cols-3 gap-2">
{availableAspectRatios.map((ratio) => (
<button
key={ratio}
type="button"
onClick={() => onAspectRatioChange(ratio)}
className={cn(
"studio-control studio-control-press studio-hover-surface flex flex-col items-center gap-2 rounded-xl border px-2 py-3 text-xs",
aspectRatio === ratio ? "border-accent-blue bg-accent-blue/10 text-foreground" : "border-border",
)}
>
<span className={cn("rounded-sm border border-current/40 bg-muted/40", ratio === "9:16" && "h-7 w-4", ratio === "16:9" && "h-4 w-7", ratio === "1:1" && "size-5", ratio === "4:3" && "h-5 w-6", ratio === "3:4" && "h-6 w-5", ratio === "21:9" && "h-3 w-8")} />
{ratio}
</button>
))}
</div>
<p className="mb-2 mt-4 text-xs font-medium text-muted-foreground">Resolution</p>
<div className="flex flex-wrap gap-2">
{availableResolutions.map((item) => (
<button
key={item}
type="button"
onClick={() => onResolutionChange(item)}
className={cn(
"studio-control studio-control-press studio-hover-surface rounded-full border px-3 py-1.5 text-xs font-medium",
resolution === item ? "border-accent-blue bg-accent-blue/10 text-foreground" : "border-border",
)}
>
{formatResolutionLabel(item)}
</button>
))}
{unavailableResolutions.map((item) => (
<button
key={item}
type="button"
disabled
className="studio-control rounded-full border border-border px-3 py-1.5 text-xs font-medium text-muted-foreground opacity-50"
title="Not supported on FastLTX models yet"
>
{formatResolutionLabel(item)}
</button>
))}
</div>
</PopoverContent>
</Popover>
<Popover>
<PopoverTrigger asChild>
<ConfigPill disabled={disabled}>
<Clock className="size-3.5" />
{formatDurationLabel(durationSec)}
</ConfigPill>
</PopoverTrigger>
<PopoverContent align="start" className="w-72">
<p className="mb-3 text-xs font-medium text-muted-foreground">Total duration</p>
<Slider min={durationMin} max={durationMax} step={5} value={[durationSec]} onValueChange={(values) => onDurationChange(values[0] ?? durationMin)} />
<div className="mt-3 flex items-center justify-between text-[11px] text-muted-foreground">
<span>{formatDurationLabel(durationMin)}</span>
<span className="rounded-md border border-border px-2 py-1 text-xs font-medium text-foreground">{formatDurationLabel(durationSec)}</span>
<span>{formatDurationLabel(durationMax)}</span>
</div>
</PopoverContent>
</Popover>
<div className="ml-auto flex items-center gap-1.5">
{onSpeechTranscript && (
<SpeechToTextButton
disabled={disabled || isGenerating}
onTranscript={onSpeechTranscript}
onInterimChange={onSpeechInterimChange}
onBusyChange={setSttBusy}
/>
)}
<Button
aria-label="Generate"
onClick={onSubmit}
disabled={submitDisabled}
size="icon"
className="studio-control-press rounded-full bg-accent-blue text-white shadow-sm hover-capable:hover:bg-accent-blue/90 disabled:bg-muted disabled:text-muted-foreground"
>
<ArrowUp className="size-5" />
</Button>
</div>
</div>
{referenceMissing && value.trim() && (
<p className="mt-3 text-center text-xs leading-5 text-amber-700 dark:text-amber-400">
Upload a reference asset to use Omni reference mode.
</p>
)}
</div>
</section>
);
}
@@ -0,0 +1,73 @@
"use client";
import React from "react";
import AppNavRail, { type AppNavSection } from "@/components/creation/AppNavRail";
import CreationComposer from "@/components/creation/CreationComposer";
import PresetQuickLaunchRail, { type StoryPresetLike } from "@/components/creation/PresetQuickLaunchRail";
import {
type AspectRatioId,
type CreationModeId,
type CreationModelId,
type MentionOption,
type ResolutionId,
} from "@/lib/creationConfig";
import type { LobbyCreationCapabilities } from "@/lib/creationCapabilities";
interface CreationStudioProps {
value: string;
disabled?: boolean;
isGenerating?: boolean;
canSubmit?: boolean;
modelId: CreationModelId;
modeId: CreationModeId;
aspectRatio: AspectRatioId;
resolution: ResolutionId;
durationSec: number;
referencePreviewUrl?: string | null;
firstFramePreviewUrl?: string | null;
lastFramePreviewUrl?: string | null;
mentionOptions?: MentionOption[];
storyPresets?: StoryPresetLike[];
activeSection?: AppNavSection;
onValueChange: (value: string) => void;
onSubmit: () => void;
onKeyDown?: (event: React.KeyboardEvent<HTMLTextAreaElement>) => void;
onModelChange: (modelId: CreationModelId) => void;
onModeChange: (modeId: CreationModeId) => void;
onAspectRatioChange: (aspectRatio: AspectRatioId) => void;
onResolutionChange: (resolution: ResolutionId) => void;
onDurationChange: (durationSec: number) => void;
onReferenceSelect?: (file: File | null) => void;
onFirstFrameSelect?: (file: File | null) => void;
onLastFrameSelect?: (file: File | null) => void;
onPresetGenerate?: (presetId: string) => void;
onSpeechTranscript?: (text: string) => void;
onSpeechInterimChange?: (text: string) => void;
onOpenProjects?: () => void;
capabilities?: LobbyCreationCapabilities;
}
export default function CreationStudio({
activeSection = "create",
onOpenProjects,
storyPresets = [],
onPresetGenerate,
isGenerating = false,
capabilities,
...composerProps
}: CreationStudioProps) {
return (
<div className="flex min-h-0 flex-1">
<AppNavRail activeSection={activeSection} onOpenProjects={onOpenProjects} />
<div className="min-w-0 flex-1 overflow-y-auto">
<div className="mx-auto flex w-full max-w-5xl flex-col gap-5 px-4 py-7 sm:px-6 sm:py-8">
<CreationComposer {...composerProps} isGenerating={isGenerating} capabilities={capabilities} />
{storyPresets.length > 0 && onPresetGenerate && (
<PresetQuickLaunchRail storyPresets={storyPresets} disabled={isGenerating} onPresetGenerate={onPresetGenerate} />
)}
</div>
</div>
</div>
);
}
@@ -0,0 +1,240 @@
"use client";
import React, { useCallback, useEffect, useRef, useState } from "react";
import { ChevronLeft, ChevronRight } from "lucide-react";
import { cn } from "@/lib/utils";
export interface StoryPresetLike {
id: string;
label: string;
description?: string;
segmentCount?: number;
styleTag?: string;
}
interface PresetQuickLaunchRailProps {
storyPresets: StoryPresetLike[];
disabled?: boolean;
onPresetGenerate: (presetId: string) => void;
}
export default function PresetQuickLaunchRail({
storyPresets,
disabled = false,
onPresetGenerate,
}: PresetQuickLaunchRailProps) {
const scrollRef = useRef<HTMLDivElement>(null);
const [canScrollLeft, setCanScrollLeft] = useState(false);
const [canScrollRight, setCanScrollRight] = useState(false);
const [presetRailDragging, setPresetRailDragging] = useState(false);
const presetDragStateRef = useRef({
pointerId: null as number | null,
startX: 0,
startScrollLeft: 0,
moved: false,
});
const suppressPresetClickRef = useRef(false);
const updateScrollState = useCallback(() => {
const el = scrollRef.current;
if (!el) return;
setCanScrollLeft(el.scrollLeft > 2);
setCanScrollRight(el.scrollLeft + el.clientWidth < el.scrollWidth - 2);
}, []);
const scrollByAmount = useCallback(
(direction: "left" | "right") => {
const el = scrollRef.current;
if (!el) return;
const delta = direction === "left" ? -220 : 220;
el.scrollBy({ left: delta, behavior: "smooth" });
window.setTimeout(updateScrollState, 220);
},
[updateScrollState],
);
const handlePresetWheel = useCallback(
(event: React.WheelEvent<HTMLDivElement>) => {
const el = scrollRef.current;
if (!el) return;
if (el.scrollWidth <= el.clientWidth + 1) return;
const dominantDelta = Math.abs(event.deltaX) > Math.abs(event.deltaY) ? event.deltaX : event.deltaY;
if (!dominantDelta) return;
const maxScrollLeft = Math.max(el.scrollWidth - el.clientWidth, 0);
const nextScrollLeft = Math.min(Math.max(el.scrollLeft + dominantDelta, 0), maxScrollLeft);
if (nextScrollLeft === el.scrollLeft) return;
event.preventDefault();
el.scrollLeft = nextScrollLeft;
updateScrollState();
},
[updateScrollState],
);
const finishPresetDrag = useCallback(() => {
presetDragStateRef.current = {
pointerId: null,
startX: 0,
startScrollLeft: 0,
moved: false,
};
setPresetRailDragging(false);
}, []);
const handlePresetPointerDown = useCallback((event: React.PointerEvent<HTMLDivElement>) => {
const el = scrollRef.current;
if (!el) return;
if (event.pointerType !== "mouse" || event.button !== 0) return;
if (el.scrollWidth <= el.clientWidth + 1) return;
suppressPresetClickRef.current = false;
presetDragStateRef.current = {
pointerId: event.pointerId,
startX: event.clientX,
startScrollLeft: el.scrollLeft,
moved: false,
};
}, []);
const handlePresetPointerMove = useCallback(
(event: React.PointerEvent<HTMLDivElement>) => {
const el = scrollRef.current;
const dragState = presetDragStateRef.current;
if (!el || dragState.pointerId !== event.pointerId) return;
const deltaX = event.clientX - dragState.startX;
if (!dragState.moved && Math.abs(deltaX) > 4) {
dragState.moved = true;
suppressPresetClickRef.current = true;
setPresetRailDragging(true);
el.setPointerCapture?.(event.pointerId);
}
if (!dragState.moved) return;
event.preventDefault();
const maxScrollLeft = Math.max(el.scrollWidth - el.clientWidth, 0);
el.scrollLeft = Math.min(Math.max(dragState.startScrollLeft - deltaX, 0), maxScrollLeft);
updateScrollState();
},
[updateScrollState],
);
const handlePresetPointerUp = useCallback(
(event: React.PointerEvent<HTMLDivElement>) => {
const el = scrollRef.current;
if (!el || presetDragStateRef.current.pointerId !== event.pointerId) return;
if (el.hasPointerCapture?.(event.pointerId)) {
el.releasePointerCapture(event.pointerId);
}
finishPresetDrag();
},
[finishPresetDrag],
);
const handlePresetClickCapture = useCallback((event: React.MouseEvent<HTMLDivElement>) => {
if (!suppressPresetClickRef.current) return;
suppressPresetClickRef.current = false;
event.preventDefault();
event.stopPropagation();
}, []);
useEffect(() => {
updateScrollState();
}, [storyPresets, updateScrollState]);
useEffect(() => {
const el = scrollRef.current;
if (!el) return;
const observer = new ResizeObserver(() => updateScrollState());
observer.observe(el);
return () => observer.disconnect();
}, [updateScrollState]);
if (storyPresets.length === 0) return null;
const scrollMaskStyle =
canScrollLeft && canScrollRight
? {
maskImage: "linear-gradient(to right, transparent, black 20px, black calc(100% - 20px), transparent)",
WebkitMaskImage: "linear-gradient(to right, transparent, black 20px, black calc(100% - 20px), transparent)",
}
: canScrollLeft
? {
maskImage: "linear-gradient(to right, transparent, black 20px, black)",
WebkitMaskImage: "linear-gradient(to right, transparent, black 20px, black)",
}
: canScrollRight
? {
maskImage: "linear-gradient(to right, black, black calc(100% - 20px), transparent)",
WebkitMaskImage: "linear-gradient(to right, black, black calc(100% - 20px), transparent)",
}
: undefined;
return (
<div className={cn("mx-auto w-full max-w-3xl transition-opacity duration-200", disabled && "pointer-events-none opacity-40")}>
<div className="grid grid-cols-[auto_minmax(0,1fr)_auto] items-center gap-1 sm:gap-2">
<div className="flex w-8 shrink-0 justify-center">
{canScrollLeft ? (
<button
type="button"
aria-label="Scroll suggested prompts left"
onClick={() => scrollByAmount("left")}
className="studio-control studio-control-press inline-flex size-8 items-center justify-center rounded-full text-muted-foreground hover-capable:hover:bg-muted/60 hover-capable:hover:text-foreground"
>
<ChevronLeft className="size-4" />
</button>
) : null}
</div>
<div
ref={scrollRef}
onScroll={updateScrollState}
onWheel={handlePresetWheel}
onPointerDown={handlePresetPointerDown}
onPointerMove={handlePresetPointerMove}
onPointerUp={handlePresetPointerUp}
onPointerCancel={handlePresetPointerUp}
onLostPointerCapture={finishPresetDrag}
onClickCapture={handlePresetClickCapture}
style={scrollMaskStyle}
className={cn(
"scrollbar-hidden flex gap-2 overflow-x-auto overflow-y-visible py-0.5 select-none",
presetRailDragging ? "cursor-grabbing" : "cursor-grab",
)}
>
{storyPresets.map((preset) => (
<button
key={preset.id}
type="button"
disabled={disabled}
onClick={() => onPresetGenerate(preset.id)}
className="studio-control studio-control-press studio-hover-surface flex w-[12.5rem] shrink-0 flex-col gap-1 rounded-xl border border-border/50 bg-card/70 px-3 py-2.5 text-left"
>
<span className="line-clamp-1 text-sm font-medium text-foreground">{preset.label}</span>
{preset.description && (
<span className="text-pretty line-clamp-2 text-xs leading-5 text-muted-foreground">{preset.description}</span>
)}
</button>
))}
</div>
<div className="flex w-8 shrink-0 justify-center">
{canScrollRight ? (
<button
type="button"
aria-label="Scroll suggested prompts right"
onClick={() => scrollByAmount("right")}
className="studio-control studio-control-press inline-flex size-8 items-center justify-center rounded-full text-muted-foreground hover-capable:hover:bg-muted/60 hover-capable:hover:text-foreground"
>
<ChevronRight className="size-4" />
</button>
) : null}
</div>
</div>
</div>
);
}
@@ -0,0 +1,97 @@
"use client";
import React, { useRef, useState } from "react";
import { ImagePlus } from "lucide-react";
import { REFERENCE_ACCEPT, isReferenceMediaFile } from "@/lib/creationConfig";
import { cn } from "@/lib/utils";
interface ReferenceUploadSlotProps {
label: string;
sublabel?: string;
previewUrl?: string | null;
required?: boolean;
optional?: boolean;
disabled?: boolean;
onSelect?: (file: File | null) => void;
}
export default function ReferenceUploadSlot({
label,
sublabel,
previewUrl = null,
required = false,
optional = false,
disabled = false,
onSelect,
}: ReferenceUploadSlotProps) {
const fileInputRef = useRef<HTMLInputElement>(null);
const [dragActive, setDragActive] = useState(false);
function handleFile(file: File | null) {
if (!file || !isReferenceMediaFile(file)) return;
onSelect?.(file);
}
return (
<div className="flex flex-col gap-1">
<button
type="button"
aria-label={[label, sublabel].filter(Boolean).join(" ")}
onClick={() => fileInputRef.current?.click()}
disabled={disabled}
onDragEnter={(event) => {
event.preventDefault();
event.stopPropagation();
if (!disabled) setDragActive(true);
}}
onDragOver={(event) => {
event.preventDefault();
event.stopPropagation();
if (!disabled) setDragActive(true);
}}
onDragLeave={(event) => {
event.preventDefault();
event.stopPropagation();
setDragActive(false);
}}
onDrop={(event) => {
event.preventDefault();
event.stopPropagation();
setDragActive(false);
if (disabled) return;
handleFile(event.dataTransfer.files?.[0] ?? null);
}}
className={cn(
"studio-control studio-control-press studio-hover-surface relative flex size-[76px] shrink-0 flex-col items-center justify-center gap-1 overflow-hidden rounded-2xl border border-dashed bg-muted/50 px-1 text-center text-[11px] font-medium text-muted-foreground",
required && !previewUrl ? "border-amber-500/50" : "border-border/60",
dragActive && "border-accent-blue bg-accent-blue/10 ring-2 ring-accent-blue/30",
disabled && "pointer-events-none opacity-50",
)}
>
{previewUrl ? (
<img src={previewUrl} alt="" className="studio-media-outline absolute inset-0 size-full object-cover" />
) : (
<>
<ImagePlus className="size-4" />
<span>{label}</span>
{sublabel && <span className="text-[10px] font-normal opacity-70">{sublabel}</span>}
</>
)}
</button>
{(required || optional) && (
<span className="text-center text-[10px] text-muted-foreground">{required ? "Required" : "Optional"}</span>
)}
<input
ref={fileInputRef}
type="file"
accept={REFERENCE_ACCEPT}
className="hidden"
onChange={(event) => {
handleFile(event.target.files?.[0] ?? null);
event.target.value = "";
}}
/>
</div>
);
}
@@ -0,0 +1,213 @@
"use client";
import React from "react";
import { Box, ChevronDown, Clock, Monitor, Wand2 } from "lucide-react";
import ConfigPill from "@/components/creation/ConfigPill";
import {
DropdownMenu,
DropdownMenuContent,
DropdownMenuItem,
DropdownMenuLabel,
DropdownMenuSeparator,
DropdownMenuTrigger,
} from "@/components/ui/dropdown-menu";
import { Popover, PopoverContent, PopoverTrigger } from "@/components/ui/popover";
import { Slider } from "@/components/ui/slider";
import {
ASPECT_RATIOS,
CREATION_MODELS,
CREATION_MODES,
RESOLUTIONS,
type AspectRatioId,
type CreationModeId,
type CreationModelId,
type ResolutionId,
formatDurationLabel,
formatResolutionLabel,
} from "@/lib/creationConfig";
import { cn } from "@/lib/utils";
export interface SessionCreationConfig {
modelId: CreationModelId;
modeId: CreationModeId;
aspectRatio: AspectRatioId;
resolution: ResolutionId;
durationSec: number;
}
interface SessionCreationConfigPillsProps extends SessionCreationConfig {
disabled?: boolean;
readOnly?: boolean;
onModelChange?: (modelId: CreationModelId) => void;
onModeChange?: (modeId: CreationModeId) => void;
onAspectRatioChange?: (aspectRatio: AspectRatioId) => void;
onResolutionChange?: (resolution: ResolutionId) => void;
onDurationChange?: (durationSec: number) => void;
}
export default function SessionCreationConfigPills({
modelId,
modeId,
aspectRatio,
resolution,
durationSec,
disabled = false,
readOnly = false,
onModelChange,
onModeChange,
onAspectRatioChange,
onResolutionChange,
onDurationChange,
}: SessionCreationConfigPillsProps) {
const selectedModel = CREATION_MODELS.find((model) => model.id === modelId) ?? CREATION_MODELS[0];
const selectedMode = CREATION_MODES.find((mode) => mode.id === modeId) ?? CREATION_MODES[0];
const isInteractive = !readOnly && !disabled;
const pillClassName = cn(
"h-9 min-h-9 px-2 text-[11px]",
!isInteractive && "pointer-events-none opacity-70",
);
if (readOnly) {
return (
<div className="flex flex-wrap items-center gap-1.5">
<ConfigPill disabled className={pillClassName} aria-label="Model">
<Box className="size-3" />
{selectedModel.label}
</ConfigPill>
<ConfigPill disabled className={pillClassName} aria-label="Mode">
<Wand2 className="size-3" />
{selectedMode.label}
</ConfigPill>
<ConfigPill disabled className={pillClassName} aria-label="Aspect ratio and resolution">
<Monitor className="size-3" />
{aspectRatio} {formatResolutionLabel(resolution)}
</ConfigPill>
<ConfigPill disabled className={pillClassName} aria-label="Duration">
<Clock className="size-3" />
{formatDurationLabel(durationSec)}
</ConfigPill>
</div>
);
}
return (
<div className="flex flex-wrap items-center gap-1.5">
<DropdownMenu>
<DropdownMenuTrigger asChild>
<ConfigPill disabled={disabled} className={pillClassName} aria-label="Model">
<Box className="size-3" />
{selectedModel.label}
<ChevronDown className="size-2.5 opacity-60" />
</ConfigPill>
</DropdownMenuTrigger>
<DropdownMenuContent align="start" className="w-72">
<DropdownMenuLabel>Model</DropdownMenuLabel>
<DropdownMenuSeparator />
{CREATION_MODELS.map((model) => (
<DropdownMenuItem key={model.id} onClick={() => onModelChange?.(model.id)} className="flex-col items-start gap-1 py-2.5">
<span className="flex items-center gap-2 text-sm font-medium">
{model.label}
{model.badge && <span className="rounded-full bg-accent-blue/15 px-1.5 py-0.5 text-[10px] text-accent-blue">{model.badge}</span>}
</span>
<span className="text-xs text-muted-foreground">{model.description}</span>
</DropdownMenuItem>
))}
</DropdownMenuContent>
</DropdownMenu>
<DropdownMenu>
<DropdownMenuTrigger asChild>
<ConfigPill disabled={disabled} className={pillClassName} aria-label="Mode">
<Wand2 className="size-3" />
{selectedMode.label}
<ChevronDown className="size-2.5 opacity-60" />
</ConfigPill>
</DropdownMenuTrigger>
<DropdownMenuContent align="start" className="w-64">
<DropdownMenuLabel>Mode</DropdownMenuLabel>
<DropdownMenuSeparator />
{CREATION_MODES.map((mode) => (
<DropdownMenuItem key={mode.id} onClick={() => onModeChange?.(mode.id)} className="flex-col items-start gap-1 py-2.5">
<span className="text-sm font-medium">{mode.label}</span>
<span className="text-xs text-muted-foreground">{mode.description}</span>
</DropdownMenuItem>
))}
</DropdownMenuContent>
</DropdownMenu>
<Popover>
<PopoverTrigger asChild>
<ConfigPill disabled={disabled} className={pillClassName} aria-label="Aspect ratio and resolution">
<Monitor className="size-3" />
{aspectRatio} {formatResolutionLabel(resolution)}
</ConfigPill>
</PopoverTrigger>
<PopoverContent align="start" className="w-80">
<p className="mb-3 text-xs font-medium text-muted-foreground">Aspect ratio</p>
<div className="grid grid-cols-3 gap-2">
{ASPECT_RATIOS.map((ratio) => (
<button
key={ratio}
type="button"
onClick={() => onAspectRatioChange?.(ratio)}
className={cn(
"studio-control studio-control-press studio-hover-surface flex flex-col items-center gap-2 rounded-xl border px-2 py-3 text-xs",
aspectRatio === ratio ? "border-accent-blue bg-accent-blue/10 text-foreground" : "border-border",
)}
>
<span
className={cn(
"rounded-sm border border-current/40 bg-muted/40",
ratio === "9:16" && "h-7 w-4",
ratio === "16:9" && "h-4 w-7",
ratio === "1:1" && "size-5",
ratio === "4:3" && "h-5 w-6",
ratio === "3:4" && "h-6 w-5",
ratio === "21:9" && "h-3 w-8",
)}
/>
{ratio}
</button>
))}
</div>
<p className="mb-2 mt-4 text-xs font-medium text-muted-foreground">Resolution</p>
<div className="flex flex-wrap gap-2">
{RESOLUTIONS.map((item) => (
<button
key={item}
type="button"
onClick={() => onResolutionChange?.(item)}
className={cn(
"studio-control studio-control-press studio-hover-surface rounded-full border px-3 py-1.5 text-xs font-medium",
resolution === item ? "border-accent-blue bg-accent-blue/10 text-foreground" : "border-border",
)}
>
{formatResolutionLabel(item)}
</button>
))}
</div>
</PopoverContent>
</Popover>
<Popover>
<PopoverTrigger asChild>
<ConfigPill disabled={disabled} className={pillClassName} aria-label="Duration">
<Clock className="size-3" />
{formatDurationLabel(durationSec)}
</ConfigPill>
</PopoverTrigger>
<PopoverContent align="start" className="w-72">
<p className="mb-3 text-xs font-medium text-muted-foreground">Total duration</p>
<Slider min={5} max={15} step={5} value={[durationSec]} onValueChange={(values) => onDurationChange?.(values[0] ?? 5)} />
<div className="mt-3 flex items-center justify-between text-[11px] text-muted-foreground">
<span>5s</span>
<span className="rounded-md border border-border px-2 py-1 text-xs font-medium text-foreground">{formatDurationLabel(durationSec)}</span>
<span>15s</span>
</div>
</PopoverContent>
</Popover>
</div>
);
}
@@ -0,0 +1,141 @@
"use client";
import * as React from "react";
import * as DropdownMenuPrimitive from "@radix-ui/react-dropdown-menu";
import { Check, ChevronRight } from "lucide-react";
import { cn } from "@/lib/utils";
const DropdownMenu = DropdownMenuPrimitive.Root;
const DropdownMenuTrigger = DropdownMenuPrimitive.Trigger;
const DropdownMenuGroup = DropdownMenuPrimitive.Group;
const DropdownMenuPortal = DropdownMenuPrimitive.Portal;
const DropdownMenuSub = DropdownMenuPrimitive.Sub;
const DropdownMenuRadioGroup = DropdownMenuPrimitive.RadioGroup;
const DropdownMenuSubTrigger = React.forwardRef<
React.ElementRef<typeof DropdownMenuPrimitive.SubTrigger>,
React.ComponentPropsWithoutRef<typeof DropdownMenuPrimitive.SubTrigger> & { inset?: boolean }
>(({ className, inset, children, ...props }, ref) => (
<DropdownMenuPrimitive.SubTrigger
ref={ref}
className={cn(
"flex cursor-default select-none items-center rounded-xl px-2 py-1.5 text-sm outline-none data-[state=open]:bg-accent focus:bg-accent",
inset && "pl-8",
className,
)}
{...props}
>
{children}
<ChevronRight className="ml-auto size-4" />
</DropdownMenuPrimitive.SubTrigger>
));
DropdownMenuSubTrigger.displayName = DropdownMenuPrimitive.SubTrigger.displayName;
const DropdownMenuSubContent = React.forwardRef<
React.ElementRef<typeof DropdownMenuPrimitive.SubContent>,
React.ComponentPropsWithoutRef<typeof DropdownMenuPrimitive.SubContent>
>(({ className, ...props }, ref) => (
<DropdownMenuPrimitive.SubContent
ref={ref}
className={cn(
"z-50 min-w-[8rem] overflow-hidden rounded-2xl border border-border bg-popover/95 p-1 text-popover-foreground shadow-xl backdrop-blur-md",
"data-[state=open]:animate-in data-[state=closed]:animate-out data-[state=closed]:fade-out-0 data-[state=open]:fade-in-0 data-[state=closed]:zoom-out-95 data-[state=open]:zoom-in-95",
"data-[side=bottom]:slide-in-from-top-2 data-[side=top]:slide-in-from-bottom-2 data-[side=left]:slide-in-from-right-2 data-[side=right]:slide-in-from-left-2",
className,
)}
{...props}
/>
));
DropdownMenuSubContent.displayName = DropdownMenuPrimitive.SubContent.displayName;
const DropdownMenuContent = React.forwardRef<
React.ElementRef<typeof DropdownMenuPrimitive.Content>,
React.ComponentPropsWithoutRef<typeof DropdownMenuPrimitive.Content>
>(({ className, sideOffset = 6, ...props }, ref) => (
<DropdownMenuPrimitive.Portal>
<DropdownMenuPrimitive.Content
ref={ref}
sideOffset={sideOffset}
className={cn(
"z-50 min-w-[12rem] overflow-hidden rounded-2xl border border-border bg-popover/95 p-1.5 text-popover-foreground shadow-xl backdrop-blur-md",
"data-[state=open]:animate-in data-[state=closed]:animate-out data-[state=closed]:fade-out-0 data-[state=open]:fade-in-0 data-[state=closed]:zoom-out-95 data-[state=open]:zoom-in-95",
"data-[side=bottom]:slide-in-from-top-2 data-[side=top]:slide-in-from-bottom-2 data-[side=left]:slide-in-from-right-2 data-[side=right]:slide-in-from-left-2",
className,
)}
{...props}
/>
</DropdownMenuPrimitive.Portal>
));
DropdownMenuContent.displayName = DropdownMenuPrimitive.Content.displayName;
const DropdownMenuItem = React.forwardRef<
React.ElementRef<typeof DropdownMenuPrimitive.Item>,
React.ComponentPropsWithoutRef<typeof DropdownMenuPrimitive.Item> & { inset?: boolean }
>(({ className, inset, ...props }, ref) => (
<DropdownMenuPrimitive.Item
ref={ref}
className={cn(
"relative flex cursor-default select-none items-center gap-2 rounded-xl px-2.5 py-2 text-sm outline-none transition-colors data-[disabled]:pointer-events-none data-[disabled]:opacity-50 focus:bg-accent focus:text-accent-foreground",
inset && "pl-8",
className,
)}
{...props}
/>
));
DropdownMenuItem.displayName = DropdownMenuPrimitive.Item.displayName;
const DropdownMenuCheckboxItem = React.forwardRef<
React.ElementRef<typeof DropdownMenuPrimitive.CheckboxItem>,
React.ComponentPropsWithoutRef<typeof DropdownMenuPrimitive.CheckboxItem>
>(({ className, children, checked, ...props }, ref) => (
<DropdownMenuPrimitive.CheckboxItem
ref={ref}
className={cn(
"relative flex cursor-default select-none items-center rounded-xl py-2 pl-8 pr-2 text-sm outline-none transition-colors data-[disabled]:pointer-events-none data-[disabled]:opacity-50 focus:bg-accent focus:text-accent-foreground",
className,
)}
checked={checked}
{...props}
>
<span className="absolute left-2 flex size-3.5 items-center justify-center">
<DropdownMenuPrimitive.ItemIndicator>
<Check className="size-4 text-accent-blue" />
</DropdownMenuPrimitive.ItemIndicator>
</span>
{children}
</DropdownMenuPrimitive.CheckboxItem>
));
DropdownMenuCheckboxItem.displayName = DropdownMenuPrimitive.CheckboxItem.displayName;
const DropdownMenuLabel = React.forwardRef<
React.ElementRef<typeof DropdownMenuPrimitive.Label>,
React.ComponentPropsWithoutRef<typeof DropdownMenuPrimitive.Label> & { inset?: boolean }
>(({ className, inset, ...props }, ref) => (
<DropdownMenuPrimitive.Label ref={ref} className={cn("px-2 py-1.5 text-xs font-semibold text-muted-foreground", inset && "pl-8", className)} {...props} />
));
DropdownMenuLabel.displayName = DropdownMenuPrimitive.Label.displayName;
const DropdownMenuSeparator = React.forwardRef<
React.ElementRef<typeof DropdownMenuPrimitive.Separator>,
React.ComponentPropsWithoutRef<typeof DropdownMenuPrimitive.Separator>
>(({ className, ...props }, ref) => (
<DropdownMenuPrimitive.Separator ref={ref} className={cn("-mx-1 my-1 h-px bg-border", className)} {...props} />
));
DropdownMenuSeparator.displayName = DropdownMenuPrimitive.Separator.displayName;
export {
DropdownMenu,
DropdownMenuTrigger,
DropdownMenuContent,
DropdownMenuItem,
DropdownMenuCheckboxItem,
DropdownMenuLabel,
DropdownMenuSeparator,
DropdownMenuGroup,
DropdownMenuPortal,
DropdownMenuSub,
DropdownMenuSubContent,
DropdownMenuSubTrigger,
DropdownMenuRadioGroup,
};
@@ -0,0 +1,33 @@
"use client";
import * as React from "react";
import * as PopoverPrimitive from "@radix-ui/react-popover";
import { cn } from "@/lib/utils";
const Popover = PopoverPrimitive.Root;
const PopoverTrigger = PopoverPrimitive.Trigger;
const PopoverAnchor = PopoverPrimitive.Anchor;
const PopoverContent = React.forwardRef<
React.ElementRef<typeof PopoverPrimitive.Content>,
React.ComponentPropsWithoutRef<typeof PopoverPrimitive.Content>
>(({ className, align = "center", sideOffset = 6, ...props }, ref) => (
<PopoverPrimitive.Portal>
<PopoverPrimitive.Content
ref={ref}
align={align}
sideOffset={sideOffset}
className={cn(
"z-50 w-72 rounded-2xl border border-border bg-popover/95 p-3 text-popover-foreground shadow-xl backdrop-blur-md outline-none",
"data-[state=open]:animate-in data-[state=closed]:animate-out data-[state=closed]:fade-out-0 data-[state=open]:fade-in-0 data-[state=closed]:zoom-out-95 data-[state=open]:zoom-in-95",
"data-[side=bottom]:slide-in-from-top-2 data-[side=top]:slide-in-from-bottom-2 data-[side=left]:slide-in-from-right-2 data-[side=right]:slide-in-from-left-2",
className,
)}
{...props}
/>
</PopoverPrimitive.Portal>
));
PopoverContent.displayName = PopoverPrimitive.Content.displayName;
export { Popover, PopoverTrigger, PopoverContent, PopoverAnchor };
@@ -0,0 +1,25 @@
"use client";
import * as React from "react";
import * as SliderPrimitive from "@radix-ui/react-slider";
import { cn } from "@/lib/utils";
const Slider = React.forwardRef<
React.ElementRef<typeof SliderPrimitive.Root>,
React.ComponentPropsWithoutRef<typeof SliderPrimitive.Root>
>(({ className, ...props }, ref) => (
<SliderPrimitive.Root
ref={ref}
className={cn("relative flex w-full touch-none select-none items-center", className)}
{...props}
>
<SliderPrimitive.Track className="relative h-1.5 w-full grow overflow-hidden rounded-full bg-muted">
<SliderPrimitive.Range className="absolute h-full bg-accent-blue" />
</SliderPrimitive.Track>
<SliderPrimitive.Thumb className="block size-4 rounded-full border border-accent-blue/40 bg-background shadow transition-colors focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-accent-blue/40 disabled:pointer-events-none disabled:opacity-50" />
</SliderPrimitive.Root>
));
Slider.displayName = SliderPrimitive.Root.displayName;
export { Slider };
@@ -0,0 +1,48 @@
"use client";
import * as React from "react";
import * as TabsPrimitive from "@radix-ui/react-tabs";
import { cn } from "@/lib/utils";
const Tabs = TabsPrimitive.Root;
const TabsList = React.forwardRef<
React.ElementRef<typeof TabsPrimitive.List>,
React.ComponentPropsWithoutRef<typeof TabsPrimitive.List>
>(({ className, ...props }, ref) => (
<TabsPrimitive.List
ref={ref}
className={cn("inline-flex items-center gap-1 rounded-full bg-muted/60 p-1 text-muted-foreground", className)}
{...props}
/>
));
TabsList.displayName = TabsPrimitive.List.displayName;
const TabsTrigger = React.forwardRef<
React.ElementRef<typeof TabsPrimitive.Trigger>,
React.ComponentPropsWithoutRef<typeof TabsPrimitive.Trigger>
>(({ className, ...props }, ref) => (
<TabsPrimitive.Trigger
ref={ref}
className={cn(
"inline-flex items-center justify-center rounded-full px-3 py-1.5 text-xs font-medium whitespace-nowrap transition-all",
"focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-accent-blue/40",
"disabled:pointer-events-none disabled:opacity-50",
"data-[state=active]:bg-card data-[state=active]:text-foreground data-[state=active]:shadow-sm",
className,
)}
{...props}
/>
));
TabsTrigger.displayName = TabsPrimitive.Trigger.displayName;
const TabsContent = React.forwardRef<
React.ElementRef<typeof TabsPrimitive.Content>,
React.ComponentPropsWithoutRef<typeof TabsPrimitive.Content>
>(({ className, ...props }, ref) => (
<TabsPrimitive.Content ref={ref} className={cn("mt-4 outline-none", className)} {...props} />
));
TabsContent.displayName = TabsPrimitive.Content.displayName;
export { Tabs, TabsList, TabsTrigger, TabsContent };
@@ -0,0 +1,80 @@
import { describe, expect, it } from "vitest";
import {
DEFAULT_LOBBY_CAPABILITIES_BUNDLE,
clampLobbySelectionToCapabilities,
parseLobbyCapabilitiesBundle,
resolveModelCapabilities,
validateLobbyCreationSelection,
} from "@/lib/creationCapabilities";
describe("creationCapabilities", () => {
it("parses backend capability payloads with per-model caps", () => {
const bundle = parseLobbyCapabilitiesBundle({
model_ids: ["fast-ltx2", "fast-h3"],
models: {
"fast-ltx2": {
generation_modes: ["t2va"],
resolutions: ["480p", "720p"],
duration_sec: [5, 10],
},
"fast-h3": {
generation_modes: ["t2va", "ref2va"],
aspect_ratios: ["16:9"],
resolutions: ["720p"],
},
},
});
expect(bundle.model_ids).toEqual(["fast-ltx2", "fast-h3"]);
expect(bundle.models["fast-h3"]?.aspect_ratios).toEqual(["16:9"]);
});
it("includes fast-h3 in default lobby models", () => {
expect(DEFAULT_LOBBY_CAPABILITIES_BUNDLE.model_ids).toContain("fast-h3");
});
it("clamps unsupported lobby selections to model-specific defaults", () => {
expect(
clampLobbySelectionToCapabilities({
capabilities: resolveModelCapabilities(DEFAULT_LOBBY_CAPABILITIES_BUNDLE, "fast-h3"),
modelId: "fast-h3",
modeId: "fl2av",
aspectRatio: "9:16",
resolution: "4k",
durationSec: 99,
}),
).toEqual({
modelId: "fast-h3",
modeId: "t2v",
aspectRatio: "16:9",
resolution: "720p",
durationSec: 5,
});
});
it("rejects unsupported generation modes with a clear message", () => {
expect(
validateLobbyCreationSelection({
capabilities: resolveModelCapabilities(DEFAULT_LOBBY_CAPABILITIES_BUNDLE, "fast-ltx23"),
modelId: "fast-ltx23",
modeId: "fl2av",
aspectRatio: "16:9",
resolution: "720p",
durationSec: 5,
}),
).toMatch(/FL2VA/i);
});
it("rejects unsupported resolutions for ltx models", () => {
expect(
validateLobbyCreationSelection({
capabilities: resolveModelCapabilities(DEFAULT_LOBBY_CAPABILITIES_BUNDLE, "fast-ltx23"),
modelId: "fast-ltx23",
modeId: "t2v",
aspectRatio: "16:9",
resolution: "4k",
durationSec: 5,
}),
).toMatch(/resolution/i);
});
});
@@ -0,0 +1,263 @@
import type {
AspectRatioId,
CreationModeId,
CreationModelId,
ResolutionId,
} from "@/lib/creationConfig";
import { fromGenerationMode, toGenerationMode, type GenerationMode } from "@/lib/generationMode";
const ALL_MODEL_IDS: CreationModelId[] = ["fast-ltx23", "fast-ltx2", "fast-h3"];
const ALL_GENERATION_MODES: GenerationMode[] = ["t2va", "fl2va", "ref2va"];
const ALL_ASPECT_RATIOS: AspectRatioId[] = ["21:9", "16:9", "4:3", "1:1", "3:4", "9:16"];
const ALL_RESOLUTIONS: ResolutionId[] = ["480p", "720p", "1080p", "4k"];
export interface ModelCreationCapabilities {
generation_modes: GenerationMode[];
aspect_ratios: AspectRatioId[];
resolutions: ResolutionId[];
duration_sec: number[];
unsupported_generation_modes: Record<string, string>;
reference_assets: {
mime_types: string[];
max_bytes: number;
};
}
export interface LobbyCreationCapabilities extends ModelCreationCapabilities {
model_ids: CreationModelId[];
}
export interface LobbyCapabilitiesBundle {
model_ids: CreationModelId[];
models: Partial<Record<CreationModelId, ModelCreationCapabilities>>;
generation_modes: GenerationMode[];
aspect_ratios: AspectRatioId[];
resolutions: ResolutionId[];
duration_sec: number[];
unsupported_generation_modes: Record<string, string>;
reference_assets: {
mime_types: string[];
max_bytes: number;
};
}
const DEFAULT_LTX_MODEL_CAPABILITIES: ModelCreationCapabilities = {
generation_modes: ["t2va", "ref2va"],
aspect_ratios: ["21:9", "16:9", "4:3", "1:1", "3:4", "9:16"],
resolutions: ["480p", "720p", "1080p"],
duration_sec: [5, 10, 15],
unsupported_generation_modes: {
fl2va: "First/last frame mode (FL2VA) is not supported yet.",
},
reference_assets: {
mime_types: ["image/png", "image/jpeg", "image/webp"],
max_bytes: 15 * 1024 * 1024,
},
};
const DEFAULT_H3_MODEL_CAPABILITIES: ModelCreationCapabilities = {
generation_modes: ["t2va", "ref2va"],
aspect_ratios: ["16:9"],
resolutions: ["720p"],
duration_sec: [5, 10, 15],
unsupported_generation_modes: {
fl2va: "First/last frame mode (FL2VA) is not supported yet.",
},
reference_assets: DEFAULT_LTX_MODEL_CAPABILITIES.reference_assets,
};
export const DEFAULT_LOBBY_CAPABILITIES_BUNDLE: LobbyCapabilitiesBundle = {
model_ids: ALL_MODEL_IDS,
models: {
"fast-ltx2": DEFAULT_LTX_MODEL_CAPABILITIES,
"fast-ltx23": DEFAULT_LTX_MODEL_CAPABILITIES,
"fast-h3": DEFAULT_H3_MODEL_CAPABILITIES,
},
generation_modes: ["t2va", "ref2va"],
aspect_ratios: ["21:9", "16:9", "4:3", "1:1", "3:4", "9:16"],
resolutions: ["480p", "720p", "1080p"],
duration_sec: [5, 10, 15],
unsupported_generation_modes: DEFAULT_LTX_MODEL_CAPABILITIES.unsupported_generation_modes,
reference_assets: DEFAULT_LTX_MODEL_CAPABILITIES.reference_assets,
};
function pickStrings<T extends string>(value: unknown, allowed: readonly T[], fallback: readonly T[]): T[] {
if (!Array.isArray(value)) return [...fallback];
return value.filter((item): item is T => typeof item === "string" && allowed.includes(item as T));
}
function parseReferenceAssets(
value: unknown,
fallback: ModelCreationCapabilities["reference_assets"],
): ModelCreationCapabilities["reference_assets"] {
if (!value || typeof value !== "object") return fallback;
const data = value as Record<string, unknown>;
return {
mime_types: Array.isArray(data.mime_types)
? (data.mime_types as string[])
: fallback.mime_types,
max_bytes: typeof data.max_bytes === "number" ? data.max_bytes : fallback.max_bytes,
};
}
function parseModelCreationCapabilities(
value: unknown,
fallback: ModelCreationCapabilities,
): ModelCreationCapabilities {
if (!value || typeof value !== "object") return fallback;
const data = value as Record<string, unknown>;
return {
generation_modes: pickStrings(data.generation_modes, ALL_GENERATION_MODES, fallback.generation_modes),
aspect_ratios: pickStrings(data.aspect_ratios, ALL_ASPECT_RATIOS, fallback.aspect_ratios),
resolutions: pickStrings(data.resolutions, ALL_RESOLUTIONS, fallback.resolutions),
duration_sec: Array.isArray(data.duration_sec)
? data.duration_sec.filter((item): item is number => typeof item === "number")
: fallback.duration_sec,
unsupported_generation_modes:
typeof data.unsupported_generation_modes === "object" && data.unsupported_generation_modes
? (data.unsupported_generation_modes as Record<string, string>)
: fallback.unsupported_generation_modes,
reference_assets: parseReferenceAssets(data.reference_assets, fallback.reference_assets),
};
}
export function parseLobbyCapabilitiesBundle(payload: unknown): LobbyCapabilitiesBundle {
if (!payload || typeof payload !== "object") {
return DEFAULT_LOBBY_CAPABILITIES_BUNDLE;
}
const data = payload as Record<string, unknown>;
const modelIds = pickStrings(data.model_ids, ALL_MODEL_IDS, DEFAULT_LOBBY_CAPABILITIES_BUNDLE.model_ids);
const rawModels = typeof data.models === "object" && data.models ? (data.models as Record<string, unknown>) : {};
const models: Partial<Record<CreationModelId, ModelCreationCapabilities>> = {};
for (const modelId of modelIds) {
const fallback =
DEFAULT_LOBBY_CAPABILITIES_BUNDLE.models[modelId] ??
(modelId === "fast-h3" ? DEFAULT_H3_MODEL_CAPABILITIES : DEFAULT_LTX_MODEL_CAPABILITIES);
models[modelId] = parseModelCreationCapabilities(rawModels[modelId], fallback);
}
const unionFallback = parseModelCreationCapabilities(payload, DEFAULT_LTX_MODEL_CAPABILITIES);
return {
model_ids: modelIds,
models,
generation_modes: unionFallback.generation_modes,
aspect_ratios: unionFallback.aspect_ratios,
resolutions: unionFallback.resolutions,
duration_sec: unionFallback.duration_sec,
unsupported_generation_modes: unionFallback.unsupported_generation_modes,
reference_assets: unionFallback.reference_assets,
};
}
export function resolveModelCapabilities(
bundle: LobbyCapabilitiesBundle,
modelId: CreationModelId,
): LobbyCreationCapabilities {
const modelCaps =
bundle.models[modelId] ??
(modelId === "fast-h3" ? DEFAULT_H3_MODEL_CAPABILITIES : DEFAULT_LTX_MODEL_CAPABILITIES);
return {
model_ids: bundle.model_ids,
...modelCaps,
};
}
export function supportedCreationModes(capabilities: LobbyCreationCapabilities) {
return capabilities.generation_modes.map((wireMode) => ({
wireMode,
modeId: fromGenerationMode(wireMode),
}));
}
export function isSupportedCreationMode(modeId: CreationModeId, capabilities: LobbyCreationCapabilities): boolean {
return capabilities.generation_modes.includes(toGenerationMode(modeId));
}
export function isSupportedResolution(resolution: ResolutionId, capabilities: LobbyCreationCapabilities): boolean {
return capabilities.resolutions.includes(resolution);
}
export function isSupportedReferenceImage(file: File, capabilities: LobbyCreationCapabilities): boolean {
return capabilities.reference_assets.mime_types.includes(file.type);
}
export function unsupportedModeNotice(modeId: CreationModeId, capabilities: LobbyCreationCapabilities): string | null {
const wireMode = toGenerationMode(modeId);
return capabilities.unsupported_generation_modes[wireMode] ?? null;
}
export function clampLobbySelectionToCapabilities(input: {
capabilities: LobbyCreationCapabilities;
modelId: CreationModelId;
modeId: CreationModeId;
aspectRatio: AspectRatioId;
resolution: ResolutionId;
durationSec: number;
}): {
modelId: CreationModelId;
modeId: CreationModeId;
aspectRatio: AspectRatioId;
resolution: ResolutionId;
durationSec: number;
} {
const { capabilities } = input;
const modelId = capabilities.model_ids.includes(input.modelId)
? input.modelId
: (capabilities.model_ids[0] ?? "fast-ltx23");
const supportedModes = supportedCreationModes(capabilities);
const modeId = isSupportedCreationMode(input.modeId, capabilities)
? input.modeId
: (supportedModes[0]?.modeId ?? "t2v");
const aspectRatio = capabilities.aspect_ratios.includes(input.aspectRatio)
? input.aspectRatio
: (capabilities.aspect_ratios[0] ?? "16:9");
const resolution = isSupportedResolution(input.resolution, capabilities)
? input.resolution
: (capabilities.resolutions[0] ?? "720p");
const durationSec = capabilities.duration_sec.includes(input.durationSec)
? input.durationSec
: (capabilities.duration_sec[0] ?? 5);
return { modelId, modeId, aspectRatio, resolution, durationSec };
}
export function validateLobbyCreationSelection(input: {
capabilities: LobbyCreationCapabilities;
modelId: CreationModelId;
modeId: CreationModeId;
aspectRatio: AspectRatioId;
resolution: ResolutionId;
durationSec: number;
referenceFile?: File | null;
firstFrameFile?: File | null;
lastFrameFile?: File | null;
}): string | null {
const unsupportedMode = unsupportedModeNotice(input.modeId, input.capabilities);
if (unsupportedMode) return unsupportedMode;
if (!input.capabilities.model_ids.includes(input.modelId)) {
return "Selected model is not supported yet.";
}
if (!isSupportedCreationMode(input.modeId, input.capabilities)) {
return "Selected mode is not supported yet.";
}
if (!input.capabilities.aspect_ratios.includes(input.aspectRatio)) {
return "Selected aspect ratio is not supported for this model yet.";
}
if (!isSupportedResolution(input.resolution, input.capabilities)) {
return "Selected resolution is not supported for this model yet.";
}
if (!input.capabilities.duration_sec.includes(input.durationSec)) {
return "Selected duration is not supported yet.";
}
if (input.modeId === "ref2av" && !input.referenceFile) {
return "Upload a reference image to use reference-guided mode.";
}
if (input.referenceFile && !isSupportedReferenceImage(input.referenceFile, input.capabilities)) {
return "Reference assets must be PNG, JPEG, or WebP images.";
}
if (input.firstFrameFile && !isSupportedReferenceImage(input.firstFrameFile, input.capabilities)) {
return "First frame must be a PNG, JPEG, or WebP image.";
}
if (input.lastFrameFile && !isSupportedReferenceImage(input.lastFrameFile, input.capabilities)) {
return "Last frame must be a PNG, JPEG, or WebP image.";
}
return null;
}
@@ -0,0 +1,62 @@
import { describe, expect, it } from "vitest";
import {
CREATION_MODELS,
buildMentionOptions,
formatDurationLabel,
formatResolutionLabel,
isReferenceMediaFile,
modeRequiresReference,
modeUsesDualFrames,
} from "@/lib/creationConfig";
describe("creationConfig", () => {
it("formats resolution labels", () => {
expect(formatResolutionLabel("480p")).toBe("480P");
expect(formatResolutionLabel("720p")).toBe("720P");
expect(formatResolutionLabel("4k")).toBe("4K");
});
it("formats duration labels", () => {
expect(formatDurationLabel(5)).toBe("5s");
});
it("includes all Dreamverse lobby models", () => {
expect(CREATION_MODELS.map((model) => model.id)).toEqual(["fast-ltx23", "fast-ltx2", "fast-h3"]);
});
it("builds mention options from presets", () => {
expect(
buildMentionOptions([
{ id: "preset-a", label: "Preset A", description: "A short preset" },
{ label: "Missing id" },
]),
).toEqual([
{
id: "preset-a",
label: "Preset A",
kind: "preset",
description: "A short preset",
},
{
id: "Missing id",
label: "Missing id",
kind: "preset",
description: undefined,
},
]);
});
it("derives mode-specific reference requirements", () => {
expect(modeRequiresReference("ref2av")).toBe(true);
expect(modeRequiresReference("t2v")).toBe(false);
expect(modeUsesDualFrames("fl2av")).toBe(true);
expect(modeUsesDualFrames("t2v")).toBe(false);
});
it("accepts image reference files only", () => {
expect(isReferenceMediaFile(new File(["x"], "a.png", { type: "image/png" }))).toBe(true);
expect(isReferenceMediaFile(new File(["x"], "a.mp4", { type: "video/mp4" }))).toBe(false);
expect(isReferenceMediaFile(new File(["x"], "a.txt", { type: "text/plain" }))).toBe(false);
});
});
@@ -0,0 +1,96 @@
export type CreationModeId = "t2v" | "fl2av" | "ref2av";
export type CreationModelId = "fast-ltx2" | "fast-ltx23" | "fast-h3";
export type AspectRatioId = "21:9" | "16:9" | "4:3" | "1:1" | "3:4" | "9:16";
export type ResolutionId = "480p" | "720p" | "1080p" | "4k";
export interface CreationModeOption {
id: CreationModeId;
label: string;
description: string;
}
export interface CreationModelOption {
id: CreationModelId;
label: string;
description: string;
badge?: string;
}
export interface MentionOption {
id: string;
label: string;
kind: "preset" | "asset" | "character";
description?: string;
}
export const CREATION_MODES: CreationModeOption[] = [
{ id: "t2v", label: "Text to video", description: "Generate from a text prompt" },
{ id: "ref2av", label: "Image to video", description: "Guide the first segment with a reference image" },
];
export const UNSUPPORTED_CREATION_MODES: CreationModeOption[] = [
{ id: "fl2av", label: "First and last frame", description: "Coming soon on FastLTX models" },
];
export const CREATION_MODELS: CreationModelOption[] = [
{
id: "fast-ltx23",
label: "FastLTX 2.3",
description: "LTX 2.3 with OmniNFT LoRA",
badge: "New",
},
{
id: "fast-ltx2",
label: "FastLTX 2",
description: "FastLTX 2 for streaming",
},
{
id: "fast-h3",
label: "FastH3",
description: "MiniMax H3 with VSA data-free adapter",
},
];
export const ASPECT_RATIOS: AspectRatioId[] = ["21:9", "16:9", "4:3", "1:1", "3:4", "9:16"];
export const RESOLUTIONS: ResolutionId[] = ["480p", "720p", "1080p"];
export const UNSUPPORTED_RESOLUTIONS: ResolutionId[] = ["4k"];
export const DURATION_MARKS = [5, 10, 15] as const;
export const REFERENCE_ACCEPT = "image/png,image/jpeg,image/webp";
export function formatResolutionLabel(resolution: ResolutionId): string {
return resolution === "4k" ? "4K" : resolution.toUpperCase();
}
export function formatDurationLabel(seconds: number): string {
return `${seconds}s`;
}
export function modeRequiresReference(modeId: CreationModeId): boolean {
return modeId === "ref2av";
}
export function modeUsesDualFrames(modeId: CreationModeId): boolean {
return modeId === "fl2av";
}
export function isReferenceMediaFile(file: File): boolean {
return file.type === "image/png" || file.type === "image/jpeg" || file.type === "image/webp";
}
export function buildMentionOptions(storyPresets: Array<{ id?: string; label?: string; description?: string }>): MentionOption[] {
return storyPresets
.filter((preset) => typeof preset.label === "string" && preset.label.trim())
.map((preset) => ({
id: String(preset.id || preset.label),
label: String(preset.label),
kind: "preset" as const,
description: typeof preset.description === "string" ? preset.description : undefined,
}));
}
@@ -0,0 +1,66 @@
import { describe, expect, it } from "vitest";
import { parseEchoedCreationConfig, validateCreationInputs } from "@/lib/creationPayload";
describe("creationPayload", () => {
it("requires a reference asset for omni reference mode", () => {
expect(
validateCreationInputs({
modeId: "ref2av",
referenceFile: null,
}),
).toMatch(/reference asset/i);
});
it("requires both frames for first and last frame mode", () => {
expect(
validateCreationInputs({
modeId: "fl2av",
firstFrameFile: new File(["a"], "first.png", { type: "image/png" }),
lastFrameFile: null,
}),
).toMatch(/both first and last/i);
});
it("accepts text to video without references", () => {
expect(
validateCreationInputs({
modeId: "t2v",
}),
).toBeNull();
});
it("parses echoed creation config from server payloads", () => {
expect(
parseEchoedCreationConfig({
type: "gpu_assigned",
creation_config: {
model_id: "fast-ltx2",
generation_mode: "ref2va",
aspect_ratio: "9:16",
resolution: "480p",
duration_sec: 10,
},
}),
).toEqual({
modelId: "fast-ltx2",
modeId: "ref2av",
aspectRatio: "9:16",
resolution: "480p",
durationSec: 10,
});
});
it("ignores invalid echoed creation config", () => {
expect(parseEchoedCreationConfig({ creation_config: { model_id: "unknown" } })).toBeNull();
});
it("rejects unsupported reference mime types", () => {
expect(
validateCreationInputs({
modeId: "t2v",
referenceFile: new File(["a"], "clip.mp4", { type: "video/mp4" }),
}),
).toMatch(/PNG, JPEG, or WebP/i);
});
});
@@ -0,0 +1,172 @@
import type {
AspectRatioId,
CreationModeId,
CreationModelId,
ResolutionId,
} from "@/lib/creationConfig";
import { fromGenerationMode, type GenerationMode } from "@/lib/generationMode";
const LOBBY_MODEL_IDS = new Set<CreationModelId>(["fast-ltx2", "fast-ltx23", "fast-h3"]);
const ASPECT_RATIO_IDS = new Set<AspectRatioId>(["21:9", "16:9", "4:3", "1:1", "3:4", "9:16"]);
const RESOLUTION_IDS = new Set<ResolutionId>(["480p", "720p", "1080p", "4k"]);
const DURATION_SEC_VALUES = new Set([5, 10, 15]);
export interface EchoedSessionCreationConfig {
modelId: CreationModelId;
modeId: CreationModeId;
aspectRatio: AspectRatioId;
resolution: ResolutionId;
durationSec: number;
}
const MAX_IMAGE_BYTES = 15 * 1024 * 1024;
const SUPPORTED_IMAGE_TYPES = new Set(["image/png", "image/jpeg", "image/webp"]);
export interface InitialImagePayload {
name: string;
mime_type: string;
data_url: string;
}
export interface CreationInitPayload {
model_id: string;
aspect_ratio: string;
resolution: string;
duration_sec: number;
initial_image: InitialImagePayload | null;
last_frame_image: InitialImagePayload | null;
}
function readFileAsDataUrl(file: File): Promise<string> {
return new Promise((resolve, reject) => {
const reader = new FileReader();
reader.onload = () => {
if (typeof reader.result === "string") {
resolve(reader.result);
return;
}
reject(new Error("Failed to read reference image."));
};
reader.onerror = () => reject(new Error("Failed to read reference image."));
reader.readAsDataURL(file);
});
}
export async function fileToInitialImagePayload(file: File): Promise<InitialImagePayload> {
if (!SUPPORTED_IMAGE_TYPES.has(file.type)) {
throw new Error("Reference assets must be PNG, JPEG, or WebP images.");
}
if (file.size > MAX_IMAGE_BYTES) {
throw new Error("Reference image must be 15 MB or smaller.");
}
return {
name: file.name,
mime_type: file.type,
data_url: await readFileAsDataUrl(file),
};
}
export async function resolveCreationImages(input: {
modeId: CreationModeId;
referenceFile?: File | null;
firstFrameFile?: File | null;
lastFrameFile?: File | null;
}): Promise<Pick<CreationInitPayload, "initial_image" | "last_frame_image">> {
if (input.modeId === "fl2av") {
const firstFrame = input.firstFrameFile ? await fileToInitialImagePayload(input.firstFrameFile) : null;
const lastFrame = input.lastFrameFile ? await fileToInitialImagePayload(input.lastFrameFile) : null;
return {
initial_image: firstFrame,
last_frame_image: lastFrame,
};
}
const reference = input.referenceFile ? await fileToInitialImagePayload(input.referenceFile) : null;
return {
initial_image: reference,
last_frame_image: null,
};
}
export function validateCreationInputs(input: {
modeId: CreationModeId;
referenceFile?: File | null;
firstFrameFile?: File | null;
lastFrameFile?: File | null;
}): string | null {
if (input.modeId === "ref2av" && !input.referenceFile) {
return "Upload a reference asset to use Omni reference mode.";
}
if (input.modeId === "fl2av") {
if (!input.firstFrameFile || !input.lastFrameFile) {
return "Upload both first and last frame assets.";
}
}
if (input.referenceFile && !SUPPORTED_IMAGE_TYPES.has(input.referenceFile.type)) {
return "Reference assets must be PNG, JPEG, or WebP images.";
}
if (input.firstFrameFile && !SUPPORTED_IMAGE_TYPES.has(input.firstFrameFile.type)) {
return "First frame must be a PNG, JPEG, or WebP image.";
}
if (input.lastFrameFile && !SUPPORTED_IMAGE_TYPES.has(input.lastFrameFile.type)) {
return "Last frame must be a PNG, JPEG, or WebP image.";
}
return null;
}
export function parseEchoedCreationConfig(data: unknown): EchoedSessionCreationConfig | null {
if (!data || typeof data !== "object") {
return null;
}
const creationConfig = (data as Record<string, unknown>).creation_config;
if (!creationConfig || typeof creationConfig !== "object") {
return null;
}
const config = creationConfig as Record<string, unknown>;
const modelId = typeof config.model_id === "string" && LOBBY_MODEL_IDS.has(config.model_id as CreationModelId)
? (config.model_id as CreationModelId)
: null;
const generationMode = typeof config.generation_mode === "string" ? config.generation_mode as GenerationMode : null;
const modeId = generationMode === "t2va" || generationMode === "fl2va" || generationMode === "ref2va"
? fromGenerationMode(generationMode)
: null;
const aspectRatio = typeof config.aspect_ratio === "string" && ASPECT_RATIO_IDS.has(config.aspect_ratio as AspectRatioId)
? (config.aspect_ratio as AspectRatioId)
: null;
const resolution = typeof config.resolution === "string" && RESOLUTION_IDS.has(config.resolution as ResolutionId)
? (config.resolution as ResolutionId)
: null;
const durationSec = typeof config.duration_sec === "number" && DURATION_SEC_VALUES.has(config.duration_sec)
? config.duration_sec
: null;
if (modelId === null || modeId === null || aspectRatio === null || resolution === null || durationSec === null) {
return null;
}
return {
modelId,
modeId,
aspectRatio,
resolution,
durationSec,
};
}
export async function buildCreationInitPayload(input: {
modelId: string;
modeId: CreationModeId;
aspectRatio: string;
resolution: string;
durationSec: number;
referenceFile?: File | null;
firstFrameFile?: File | null;
lastFrameFile?: File | null;
}): Promise<CreationInitPayload> {
const images = await resolveCreationImages(input);
return {
model_id: input.modelId,
aspect_ratio: input.aspectRatio,
resolution: input.resolution,
duration_sec: input.durationSec,
...images,
};
}
@@ -0,0 +1,39 @@
import { describe, expect, it } from "vitest";
import {
DEFAULT_GENERATION_MODE,
GENERATION_MODES,
fromGenerationMode,
getGenerationMode,
isGenerationMode,
toGenerationMode,
} from "./generationMode";
describe("generation modes", () => {
it("exposes stable wire IDs in the expected product order", () => {
expect(GENERATION_MODES.map((mode) => mode.id)).toEqual([
"t2va",
"fl2va",
"ref2va",
]);
expect(DEFAULT_GENERATION_MODE).toBe("t2va");
});
it("validates and resolves generation mode values", () => {
expect(isGenerationMode("ref2va")).toBe(true);
expect(isGenerationMode("unknown")).toBe(false);
expect(getGenerationMode("fl2va").label).toBe("FL2VA");
});
it("maps creation studio mode IDs to upstream wire values", () => {
expect(toGenerationMode("t2v")).toBe("t2va");
expect(toGenerationMode("fl2av")).toBe("fl2va");
expect(toGenerationMode("ref2av")).toBe("ref2va");
});
it("maps upstream wire values back to creation studio mode IDs", () => {
expect(fromGenerationMode("t2va")).toBe("t2v");
expect(fromGenerationMode("fl2va")).toBe("fl2av");
expect(fromGenerationMode("ref2va")).toBe("ref2av");
});
});
@@ -0,0 +1,54 @@
import type { CreationModeId } from "@/lib/creationConfig";
export const GENERATION_MODES = [
{
id: "t2va",
label: "T2VA",
name: "Text to video + audio",
description: "Start with a text prompt; no reference asset is required.",
},
{
id: "fl2va",
label: "FL2VA",
name: "First/last frames to video + audio",
description: "Provide first and last frame images to control the transition.",
},
{
id: "ref2va",
label: "Ref2VA",
name: "References to video + audio",
description: "Guide the result with ordered image, video, or audio references.",
},
] as const;
export type GenerationMode = (typeof GENERATION_MODES)[number]["id"];
export const DEFAULT_GENERATION_MODE: GenerationMode = "t2va";
const CREATION_MODE_TO_GENERATION_MODE: Record<CreationModeId, GenerationMode> = {
t2v: "t2va",
fl2av: "fl2va",
ref2av: "ref2va",
};
export function isGenerationMode(value: unknown): value is GenerationMode {
return GENERATION_MODES.some((mode) => mode.id === value);
}
export function getGenerationMode(value: GenerationMode) {
return GENERATION_MODES.find((mode) => mode.id === value) ?? GENERATION_MODES[0];
}
const GENERATION_MODE_TO_CREATION_MODE: Record<GenerationMode, CreationModeId> = {
t2va: "t2v",
fl2va: "fl2av",
ref2va: "ref2av",
};
export function fromGenerationMode(mode: GenerationMode): CreationModeId {
return GENERATION_MODE_TO_CREATION_MODE[mode];
}
export function toGenerationMode(modeId: CreationModeId): GenerationMode {
return CREATION_MODE_TO_GENERATION_MODE[modeId];
}
@@ -1,63 +0,0 @@
import { expect, test } from '@playwright/test';
import { skipWithoutMock } from './helpers';
test.describe('create job interactions', () => {
skipWithoutMock();
for (const jobType of ['inference', 'finetuning', 'distillation']) {
test(`${jobType} remains interactive after repeated dialog dismissals`, async ({ page }) => {
await page.goto(`/${jobType}`);
const trigger = page.getByRole('button', { name: 'Create Job', exact: true });
const dialog = page.getByRole('dialog');
// Exercise both dismissal paths and reopen without reloading the page.
for (const closeWithEscape of [false, true]) {
await trigger.click();
await page.getByRole('menuitem').first().click();
await expect(dialog).toBeVisible();
if (closeWithEscape) {
await page.keyboard.press('Escape');
} else {
await dialog.getByRole('button', { name: 'Close', exact: true }).click();
}
await expect(dialog).toBeHidden();
await expect(page.locator('body')).toHaveCSS('pointer-events', 'auto');
await expect(trigger).toBeFocused();
}
await page.getByRole('link', { name: 'Datasets', exact: true }).click();
await expect(page).toHaveURL(/\/datasets$/);
});
}
test('preserves keyboard menu dismissal and dialog focus trapping', async ({ page }) => {
await page.goto('/inference');
const trigger = page.getByRole('button', { name: 'Create Job', exact: true });
await trigger.focus();
await page.keyboard.press('Enter');
const firstItem = page.getByRole('menuitem').first();
await expect(firstItem).toBeFocused();
await page.keyboard.press('Escape');
await expect(page.getByRole('menu')).toBeHidden();
await expect(trigger).toBeFocused();
await expect(page.locator('body')).toHaveCSS('pointer-events', 'auto');
await page.keyboard.press('Enter');
await expect(firstItem).toBeFocused();
await page.keyboard.press('Enter');
const dialog = page.getByRole('dialog');
await expect(dialog).toBeVisible();
await expect(dialog.getByLabel('Name (optional)')).toBeFocused();
// Shift+Tab from the first field wraps to Close, then Tab wraps back.
await page.keyboard.press('Shift+Tab');
await expect(dialog.getByRole('button', { name: 'Close', exact: true })).toBeFocused();
await page.keyboard.press('Tab');
await expect(dialog.getByLabel('Name (optional)')).toBeFocused();
await page.keyboard.press('Escape');
await expect(dialog).toBeHidden();
await expect(trigger).toBeFocused();
await expect(page.locator('body')).toHaveCSS('pointer-events', 'auto');
});
});
+2 -18
View File
@@ -1,6 +1,6 @@
import { expect, test } from '@playwright/test';
import { API_BASE, skipWithoutMock } from './helpers';
import { skipWithoutMock } from './helpers';
/**
* Create-job flow: open the Create Job modal on /inference, fill the prompt
@@ -10,8 +10,7 @@ import { API_BASE, skipWithoutMock } from './helpers';
test.describe('create inference job', () => {
skipWithoutMock();
test('creates a T2V job and starts it without refreshing', async ({ page, request }) => {
await request.put(`${API_BASE}/settings`, { data: { autoStartJob: false } });
test('creates a T2V job and shows it in the queue', async ({ page }) => {
await page.goto('/inference');
// The trigger opens a real menu on click, so this path works for touch,
@@ -39,20 +38,5 @@ test.describe('create inference job', () => {
// Modal closes and the queue refreshes with the newly created job.
await expect(dialog).toBeHidden();
await expect(page.getByText(prompt)).toBeVisible();
await expect(page.locator('body')).toHaveCSS('pointer-events', 'auto');
const card = page.getByRole('article').filter({ hasText: prompt });
await expect(card.getByText('pending', { exact: true })).toBeVisible();
const started = page.waitForResponse((response) =>
response.url().startsWith(`${API_BASE}/jobs/`) &&
response.url().endsWith('/start') &&
response.request().method() === 'POST',
);
await card.getByRole('button', { name: 'Start', exact: true }).click();
expect((await started).ok()).toBe(true);
await expect(card.getByText('running', { exact: true })).toBeVisible();
await page.getByRole('link', { name: 'Datasets', exact: true }).click();
await expect(page).toHaveURL(/\/datasets$/);
});
});
+829 -975
View File
File diff suppressed because it is too large Load Diff
+10 -1
View File
@@ -17,11 +17,20 @@
"start:all": "concurrently --kill-others-on-fail \"npm:start:api\" \"npm:start:web\""
},
"dependencies": {
"@radix-ui/react-dialog": "^1.1.0",
"@radix-ui/react-dropdown-menu": "^2.1.24",
"@radix-ui/react-label": "^2.1.8",
"@radix-ui/react-scroll-area": "^1.2.10",
"@radix-ui/react-select": "^2.2.6",
"@radix-ui/react-separator": "^1.1.8",
"@radix-ui/react-slider": "^1.2.0",
"@radix-ui/react-slot": "^1.2.4",
"@radix-ui/react-switch": "^1.1.0",
"@radix-ui/react-tabs": "^1.1.0",
"class-variance-authority": "^0.7.1",
"clsx": "^2.1.1",
"lucide-react": "^0.577.0",
"next": "15.5.18",
"radix-ui": "^1.6.7",
"react": "^19.1.0",
"react-dom": "^19.1.0",
"sonner": "^2.0.7",
@@ -1,26 +1,24 @@
import { render, screen, waitFor, within } from '@testing-library/react';
import { render, screen } from '@testing-library/react';
import userEvent from '@testing-library/user-event';
import { beforeEach, describe, expect, it, vi } from 'vitest';
import { describe, expect, it, vi } from 'vitest';
import CreateJobButton from './CreateJobButton';
import { getDatasets, getModels } from '@/lib/api';
vi.mock('@/lib/api', () => ({
createJob: vi.fn(),
getModels: vi.fn(),
getDatasets: vi.fn(),
uploadImage: vi.fn(),
getSettings: vi.fn(),
updateSettings: vi.fn(),
vi.mock('./CreateJobModal', () => ({
default: ({
isOpen,
workloadType,
}: {
isOpen: boolean;
workloadType: string;
}) =>
isOpen ? (
<div role="dialog" data-workload-type={workloadType}>
Create job form
</div>
) : null,
}));
beforeEach(() => {
vi.mocked(getModels).mockResolvedValue([
{ id: 'wan/t2v-1.3b', label: 'Wan T2V' },
]);
vi.mocked(getDatasets).mockResolvedValue([]);
});
describe('CreateJobButton', () => {
it('opens the workload menu on click and selects an item', async () => {
const user = userEvent.setup();
@@ -29,9 +27,10 @@ describe('CreateJobButton', () => {
await user.click(screen.getByRole('button', { name: 'Create Job' }));
await user.click(screen.getByRole('menuitem', { name: /I2V/i }));
expect(
screen.getByRole('dialog', { name: 'New Inference Job (I2V)' }),
).toBeInTheDocument();
expect(screen.getByRole('dialog')).toHaveAttribute(
'data-workload-type',
'i2v',
);
});
it('opens and operates the workload menu from the keyboard', async () => {
@@ -46,42 +45,9 @@ describe('CreateJobButton', () => {
expect(firstItem).toHaveFocus();
await user.keyboard('{Enter}');
expect(
screen.getByRole('dialog', { name: 'New Inference Job (T2V)' }),
).toBeInTheDocument();
await user.keyboard('{Escape}');
await waitFor(() =>
expect(screen.queryByRole('dialog')).not.toBeInTheDocument(),
expect(screen.getByRole('dialog')).toHaveAttribute(
'data-workload-type',
't2v',
);
await waitFor(() =>
expect(document.body.style.pointerEvents).not.toBe('none'),
);
expect(trigger).toHaveFocus();
});
it.each(['inference', 'finetuning', 'distillation'] as const)(
'restores page interaction after closing the real %s dialog',
async (jobType) => {
const user = userEvent.setup();
render(<CreateJobButton jobType={jobType} />);
const trigger = screen.getByRole('button', { name: 'Create Job' });
// Keep the real Dialog mounted: mocking it hides conflicting Radix layers.
for (let attempt = 0; attempt < 2; attempt++) {
await user.click(trigger);
await user.click(screen.getAllByRole('menuitem')[0]);
const dialog = screen.getByRole('dialog');
await user.click(
within(dialog).getByRole('button', { name: 'Close' }),
);
await waitFor(() =>
expect(screen.queryByRole('dialog')).not.toBeInTheDocument(),
);
await waitFor(() =>
expect(document.body.style.pointerEvents).not.toBe('none'),
);
expect(trigger).toHaveFocus();
}
},
);
});
@@ -2,7 +2,7 @@
import * as React from 'react';
import { ChevronDown } from 'lucide-react';
import { DropdownMenu } from 'radix-ui';
import * as DropdownMenu from '@radix-ui/react-dropdown-menu';
import CreateJobModal from '@/components/jobs/CreateJobModal';
import { Button } from '@/components/ui/button';
@@ -16,7 +16,6 @@ interface CreateJobButtonProps {
export default function CreateJobButton({ jobType }: CreateJobButtonProps) {
const options = WORKLOAD_OPTIONS[jobType] ?? [];
const triggerRef = React.useRef<HTMLButtonElement>(null);
const [modalOpen, setModalOpen] = React.useState(false);
const [workloadType, setWorkloadType] = React.useState(
@@ -37,7 +36,7 @@ export default function CreateJobButton({ jobType }: CreateJobButtonProps) {
<>
<DropdownMenu.Root>
<DropdownMenu.Trigger asChild>
<Button ref={triggerRef} type="button" className="gap-1.5">
<Button type="button" className="gap-1.5">
Create Job
<ChevronDown className="size-3.5 opacity-85" aria-hidden />
</Button>
@@ -67,11 +66,6 @@ export default function CreateJobButton({ jobType }: CreateJobButtonProps) {
<CreateJobModal
isOpen={modalOpen}
onClose={() => setModalOpen(false)}
onCloseAutoFocus={(event) => {
// This dialog opens from a menu item, so it has no DialogTrigger.
event.preventDefault();
triggerRef.current?.focus();
}}
onSuccess={handleSuccess}
jobType={jobType}
workloadType={workloadType}
@@ -55,9 +55,6 @@ import { jobToFormFields, type JobLike } from '@/lib/jobToFields';
export interface CreateJobModalProps {
isOpen: boolean;
onClose: () => void;
onCloseAutoFocus?: React.ComponentProps<
typeof DialogContent
>['onCloseAutoFocus'];
onSuccess: () => void;
jobType: JobType;
workloadType: string;
@@ -70,7 +67,6 @@ export interface CreateJobModalProps {
export default function CreateJobModal({
isOpen,
onClose,
onCloseAutoFocus,
onSuccess,
jobType,
workloadType,
@@ -648,7 +644,6 @@ export default function CreateJobModal({
>
<DialogContent
className="max-h-[90vh] w-[90vw] max-w-[850px] overflow-y-auto"
onCloseAutoFocus={onCloseAutoFocus}
onEscapeKeyDown={(e) => {
if (isSubmitting) e.preventDefault();
}}
@@ -1,7 +1,5 @@
import * as React from 'react';
import { render, screen } from '@testing-library/react';
import userEvent from '@testing-library/user-event';
import { describe, expect, it, vi } from 'vitest';
import { describe, expect, it } from 'vitest';
import { Button } from './button';
import { Input } from './input';
@@ -10,24 +8,6 @@ import { Slider } from './slider';
import { Switch } from './switch';
describe('shared control accessibility', () => {
it('forwards refs and click handlers to the asChild button', async () => {
const user = userEvent.setup();
const ref = React.createRef<HTMLButtonElement>();
const onClick = vi.fn();
render(
<Button asChild ref={ref} onClick={onClick}>
<button type="button">Slotted action</button>
</Button>,
);
const button = screen.getByRole('button', { name: 'Slotted action' });
expect(screen.getAllByRole('button')).toHaveLength(1);
expect(ref.current).toBe(button);
await user.click(button);
expect(onClick).toHaveBeenCalledTimes(1);
expect(button).toHaveFocus();
});
it('keeps button, input, and select targets at least 44px tall', () => {
render(
<>
@@ -1,7 +1,7 @@
"use client";
import * as React from "react";
import { Slot } from "radix-ui";
import { Slot } from "@radix-ui/react-slot";
import { cva, type VariantProps } from "class-variance-authority";
import { cn } from "@/lib/utils";
@@ -37,7 +37,7 @@ export interface ButtonProps extends React.ButtonHTMLAttributes<HTMLButtonElemen
}
const Button = React.forwardRef<HTMLButtonElement, ButtonProps>(({ className, variant, size, asChild = false, ...props }, ref) => {
const Comp = asChild ? Slot.Root : "button";
const Comp = asChild ? Slot : "button";
return <Comp className={cn(buttonVariants({ variant, size, className }))} ref={ref} {...props} />;
});
Button.displayName = "Button";
@@ -1,7 +1,7 @@
'use client';
import * as React from 'react';
import { Dialog as DialogPrimitive } from 'radix-ui';
import * as DialogPrimitive from '@radix-ui/react-dialog';
import { X } from 'lucide-react';
import { cn } from '@/lib/utils';
@@ -1,7 +1,7 @@
'use client';
import * as React from 'react';
import { Label as LabelPrimitive } from 'radix-ui';
import * as LabelPrimitive from '@radix-ui/react-label';
import { cva, type VariantProps } from 'class-variance-authority';
import { cn } from '@/lib/utils';
@@ -1,7 +1,7 @@
'use client';
import * as React from 'react';
import { ScrollArea as ScrollAreaPrimitive } from 'radix-ui';
import * as ScrollAreaPrimitive from '@radix-ui/react-scroll-area';
import { cn } from '@/lib/utils';
@@ -1,7 +1,7 @@
'use client';
import * as React from 'react';
import { Select as SelectPrimitive } from 'radix-ui';
import * as SelectPrimitive from '@radix-ui/react-select';
import { Check, ChevronDown, ChevronUp } from 'lucide-react';
import { cn } from '@/lib/utils';
@@ -1,7 +1,7 @@
'use client';
import * as React from 'react';
import { Separator as SeparatorPrimitive } from 'radix-ui';
import * as SeparatorPrimitive from '@radix-ui/react-separator';
import { cn } from '@/lib/utils';
@@ -1,7 +1,7 @@
'use client';
import * as React from 'react';
import { Slider as SliderPrimitive } from 'radix-ui';
import * as SliderPrimitive from '@radix-ui/react-slider';
import { cn } from '@/lib/utils';
@@ -1,7 +1,7 @@
'use client';
import * as React from 'react';
import { Switch as SwitchPrimitives } from 'radix-ui';
import * as SwitchPrimitives from '@radix-ui/react-switch';
import { cn } from '@/lib/utils';
@@ -1,7 +1,7 @@
'use client';
import * as React from 'react';
import { Tabs as TabsPrimitive } from 'radix-ui';
import * as TabsPrimitive from '@radix-ui/react-tabs';
import { cn } from '@/lib/utils';
+16 -59
View File
@@ -1,5 +1,5 @@
{
"version": 11,
"version": 9,
"recipes": [
{
"id": "fastwan21-t2v",
@@ -445,13 +445,13 @@
{
"id": "fasth3-preview-cuda",
"group": "fasth3-preview",
"group_label": "FastH3 V1",
"group_label": "FastH3 Preview",
"group_task": "4-step text to video + audio",
"family": "minimax_h3",
"stage": "inference",
"task": "Few-step text to video (with audio)",
"label": "FastH3 V1 on CUDA",
"summary": "Run FastH3 V1 with four DiT forwards, trained H3 sparse attention, compiled decode, and synchronized audio.",
"label": "FastH3 Preview on CUDA",
"summary": "Run the DMD2-distilled FastH3 Preview with four DiT forwards, trained H3 sparse attention, compiled decode, and synchronized audio.",
"model": "FastVideo/FastVideo-Minimax-FastH3-Preview-v0.2",
"source": "examples/inference/basic/basic_fasth3.py",
"serving": {
@@ -484,13 +484,13 @@
"prepare": "hf download FastVideo/FastVideo-Minimax-FastH3-Preview-v0.2 --local-dir ./FastH3-Preview-v0.2\npython scripts/checkpoint_conversion/convert_minimax_h3_mlx.py --model-root ./FastH3-Preview-v0.2/transformer --out ./FastH3-MLX --formats \"int6\""
},
"group": "fasth3-preview",
"group_label": "FastH3 V1",
"group_label": "FastH3 Preview",
"group_task": "4-step text to video + audio",
"family": "minimax_h3",
"stage": "inference",
"task": "Few-step text to video (with audio)",
"label": "FastH3 V1 on MLX",
"summary": "Run FastH3 V1 on Apple Silicon with a locally converted INT6 DiT, streamed Qwen3-VL conditioning, and native MLX video and audio VAEs.",
"label": "FastH3 Preview on MLX",
"summary": "Run FastH3 Preview on Apple Silicon with a locally converted INT6 DiT, streamed Qwen3-VL conditioning, and native MLX video and audio VAEs.",
"model": "FastVideo/FastVideo-Minimax-FastH3-Preview-v0.2",
"source": "examples/inference/basic/mlx_fasth3.py",
"command": "hf download FastVideo/FastVideo-Minimax-FastH3-Preview-v0.2 --local-dir ./FastH3-Preview-v0.2\npython scripts/checkpoint_conversion/convert_minimax_h3_mlx.py --model-root ./FastH3-Preview-v0.2/transformer --out ./FastH3-MLX --formats \"int6\"\npython examples/inference/basic/mlx_fasth3.py --model-root ./FastH3-Preview-v0.2 --mlx-checkpoint ./FastH3-MLX/int6 --prompt \"(S1) A presenter says <d>[English] FastVideo runs FastH3.</d>\" --height 480 --width 832 --num-frames 124 --seed 2026 --output-path ./outputs/fasth3_int6.mp4",
@@ -516,13 +516,13 @@
{
"id": "fasth3-preview-spark",
"group": "fasth3-preview",
"group_label": "FastH3 V1",
"group_label": "FastH3 Preview",
"group_task": "4-step text to video + audio",
"family": "minimax_h3",
"stage": "inference",
"task": "Few-step text to video (with audio)",
"label": "FastH3 V1 on one DGX Spark",
"summary": "Run FastH3 V1 on one GB10 with Triton VSA, FA4 off, and lazy module load. Height, width, frames, and steps in the YAML are examples.",
"label": "FastH3 Preview on one DGX Spark",
"summary": "Run FastH3 Preview on one GB10 with Triton VSA, FA4 off, and lazy module load. Height, width, frames, and steps in the YAML are examples.",
"model": "FastVideo/FastVideo-FastH3-4-step-Preview-v1-VSA-DataFree",
"source": "examples/inference/basic/basic_fasth3_spark.yaml",
"serving": {
@@ -551,12 +551,12 @@
{
"id": "fasth3-spark-pair",
"group": "fasth3-preview",
"group_label": "FastH3 V1",
"group_label": "FastH3 Preview",
"group_task": "4-step text to video + audio",
"family": "minimax_h3",
"stage": "inference",
"task": "Few-step text to video (with audio)",
"label": "FastH3 V1 on two DGX Sparks",
"label": "FastH3 Preview on two DGX Sparks",
"summary": "Run one FastH3 clip across two GB10s with Ray sequence parallel over QSFP RoCE. Sequential load and lazy module load stay on because SP replicates the DiT on each node.",
"model": "FastVideo/FastVideo-FastH3-4-step-Preview-v1-VSA-DataFree",
"source": "examples/inference/basic/basic_fasth3_spark_pair.yaml",
@@ -581,19 +581,15 @@
{
"id": "fasth3-8step-v2-cuda",
"group": "fasth3-8step-v2",
"group_label": "FastH3 V2",
"group_label": "FastH3 8-Step V2",
"group_task": "8-step text to video + audio",
"family": "minimax_h3",
"stage": "inference",
"task": "Few-step text to video (with audio)",
"label": "FastH3 V2 on CUDA",
"summary": "Run FastH3 V2, the eight-forward checkpoint (video/audio shifts 10/3, VSA 0.8, 64-token tiles) with the trained DMD ladder loaded from the checkpoint's fastvideo_inference.json.",
"label": "FastH3 8-Step V2 on CUDA",
"summary": "Run the eight-forward FastH3 8-Step V2 checkpoint (video/audio shifts 10/3, VSA 0.8, 64-token tiles) with the trained DMD ladder loaded from the checkpoint's fastvideo_inference.json.",
"model": "FastVideo/FastVideo-FastH3-8-Step-V2",
"source": "examples/inference/basic/basic_fasth3_8step.py",
"serving": {
"source": "examples/serving/openai_fasth3_8step.yaml",
"install": "UV_TORCH_BACKEND=cu130 uv pip install -e \".[fasth3]\""
},
"command": "UV_TORCH_BACKEND=cu130 uv pip install -e \".[fasth3]\"\npython examples/inference/basic/basic_fasth3_8step.py --prompt \"(S1) A presenter says <d>[English] FastVideo runs FastH3.</d>\" --profile strict --no-inference-torch-compile --no-compile-vae",
"gpu_types": ["NVIDIA"],
"hardware": {
@@ -613,46 +609,7 @@
"limitations": [
"Nine sigma-grid points (eight transformer forwards) are fixed by the checkpoint's trained ladder; the example rejects any other --steps.",
"Validated with the eager strict route (--profile strict --no-inference-torch-compile --no-compile-vae). The compiled all profile has not been measured for this checkpoint.",
"T2AV only; no FL2VA/Ref2VA distillation and no matching LoRA. V2 uses eight forwards rather than V1's four."
]
},
{
"id": "fasth3-8step-v2-mlx",
"serving": {
"source": "examples/serving/mlx_fasth3_8step.yaml",
"install": "uv pip install -e \".[mlx]\"",
"prepare": "hf download FastVideo/FastVideo-FastH3-8-Step-V2 --local-dir ./FastH3-8-Step-V2\npython scripts/checkpoint_conversion/convert_minimax_h3_mlx.py --model-root ./FastH3-8-Step-V2/transformer --out ./FastH3-8-Step-V2-MLX --formats \"int8\" --include-vsa"
},
"group": "fasth3-8step-v2",
"group_label": "FastH3 V2",
"group_task": "8-step text to video + audio",
"family": "minimax_h3",
"stage": "inference",
"task": "Few-step text to video (with audio)",
"label": "FastH3 V2 on MLX",
"summary": "Run FastH3 V2 on Apple Silicon with a locally converted INT8 DiT, the checkpoint's DMD contract ladder, and trained VSA (sparsity 0.8, 64-token tiles).",
"model": "FastVideo/FastVideo-FastH3-8-Step-V2",
"source": "examples/inference/basic/mlx_fasth3_8step.py",
"command": "hf download FastVideo/FastVideo-FastH3-8-Step-V2 --local-dir ./FastH3-8-Step-V2\npython scripts/checkpoint_conversion/convert_minimax_h3_mlx.py --model-root ./FastH3-8-Step-V2/transformer --out ./FastH3-8-Step-V2-MLX --formats \"int8\" --include-vsa\npython examples/inference/basic/mlx_fasth3_8step.py --model-root ./FastH3-8-Step-V2 --mlx-checkpoint ./FastH3-8-Step-V2-MLX/int8 --prompt \"(S1) A presenter says <d>[English] FastVideo runs FastH3.</d>\" --height 480 --width 832 --num-frames 124 --seed 2026 --output-path ./outputs/fasth3_8step_int8.mp4",
"gpu_types": ["Apple Silicon"],
"hardware": {
"platform": "mlx",
"accelerator": "Apple M4 Max",
"system_memory": "36 GB unified memory",
"peak_memory": "27.39 GiB peak MLX memory during denoising",
"evidence": "validated",
"evidence_url": "https://github.com/hao-ai-lab/FastVideo/pull/1863"
},
"evidence": "Verified",
"expected_artifact": "MP4 with H.264 video and stereo AAC audio at outputs/fasth3_8step_int8.mp4",
"modes": ["T2VA", "8-step FastH3", "trained VSA"],
"knobs": [
{"key": "video_decode_backend", "label": "VAE decode", "hint": "Fidelity vs. speed", "flag": "--video-decode-backend", "options": [{"value": "h3-vae", "label": "Full H3 VAE"}, {"value": "taeh3", "label": "TAEH3 preview"}], "default": "h3-vae"}
],
"limitations": [
"Convert with --include-vsa. mlx_fasth3_8step.py turns VSA on (sparsity 0.8, tile 64). A dense export fails at configure_vsa.",
"--steps 8 or 9 both run the eight trained forwards. Reuse the preview VAE, audio VAE, text encoder, and tokenizer if those directories already exist.",
"The MLX path supports T2VA only. FL2VA, Ref2VA, and two-pass refinement are not wired."
"T2AV only; no FL2VA/Ref2VA distillation and no matching LoRA. Quality and latency were not compared against the four-step preview in this PR."
]
},
{
+5 -5
View File
@@ -136,7 +136,7 @@
if (platform === "mps") {
return {
id: "mps",
label: "Apple Silicon · PyTorch MPS",
label: "Apple Silicon · MPS",
hint: recipe.hardware?.minimum_memory || recipe.hardware?.system_memory || "Memory not recorded",
};
}
@@ -519,10 +519,10 @@
option.setAttribute("aria-pressed", String(selected));
});
servingAvailability.textContent = profile
? "The playground and the OpenAI Python client share one server process. Both workflows can run on your own machine."
? "The playground and API clients share one server process. Both workflows can run on your own machine."
: servingLoadFailed
? "Server examples could not be loaded. Open the H3 server guide below, or use Python directly."
: "This recipe uses Python directly. FastH3 V1 and FastH3 V2 can also run a local server for the playground and the OpenAI Python client.";
: "This recipe uses Python directly. For the playground and API clients, choose FastH3 Preview with CUDA, MLX, or one Spark.";
servingPanel.hidden = !useServer;
commandBlock.hidden = useServer;
root.querySelector("[data-cookbook-python-note]").hidden = useServer;
@@ -537,7 +537,7 @@
? "Start once, then change prompts in the playground or your app. On a DGX Spark, lazy module load still reloads Qwen3-VL and the DiT between phases of each request, so later prompts are not a free hot cache."
: "Start once, then change prompts in the playground or your app. CUDA requests reuse the loaded model. The Python SDK can also reuse a generator within one process.";
servingPanel.querySelector("[data-cookbook-install-guide]").href = isMLX
? "../../getting_started/installation/mlx/"
? "../../getting_started/installation/mps/#run-fasth3-preview"
: isSpark
? "../../getting_started/installation/spark/"
: "../../getting_started/installation/gpu/";
@@ -574,7 +574,7 @@
});
description.textContent = useServer
? `${recipe.group_label || recipe.label} generates video with audio. Start the local server, then use the playground or the OpenAI Python client. This profile uses the checked-in ${runtime.label} configuration.`
? `FastH3 Preview generates video with audio. This server profile uses the checked-in ${runtime.label} configuration.`
: recipe.summary;
label.textContent = useServer ? `${recipe.group_label || recipe.label} · Server` : recipe.label;
model.textContent = recipe.model;
+5 -8
View File
@@ -18,8 +18,8 @@ the CUDA `fastvideo-kernel` package:
- **Dense-only checkpoints** (the default converter) drop the 50 gate
matrices and keep fused SDPA. They remain valid for dense inference.
- **VSA-capable checkpoints** retain those gates, quantize them on the same
affine grid, and record `vsa.capable` in `mlx_h3_dit.json`. Preview leaves
runtime VSA off until you pass `--vsa`. `mlx_fasth3_8step.py` turns it on.
affine grid, and record `vsa.capable` in `mlx_h3_dit.json`. Runtime VSA is
still off until you pass `--vsa`.
- **Tile sizes** 64 `(4, 4, 4)` and 256 `(4, 8, 8)`. Prefix keys can be
`exempt` or `compete`. `--vsa-dense-first-n-steps` and `--vsa-dense-layers`
force dense SDPA on the selected steps or blocks.
@@ -32,12 +32,9 @@ the CUDA `fastvideo-kernel` package:
but does not yet match reference video. `--vsa-impl reference` is the same
as `auto`.
See the [MLX install guide](../../getting_started/installation/mlx.md)
and the [MiniMax H3 cookbook](../../cookbook/minimax-h3.md) for conversion
and `mlx_fasth3.py` / `mlx_fasth3_8step.py` flags. Do not enable
VSA on a dense-only checkpoint; reconvert with `--include-vsa` first. V1
VSA is opt-in. FastH3 V2 converts with `--include-vsa` and turns VSA on
by default.
See the [Apple Silicon guide](../../getting_started/installation/mps.md) for
conversion and `mlx_fasth3.py` flags. Do not enable VSA on a dense-only
checkpoint; reconvert with `--include-vsa` first.
H3 uses fused MLX RMSNorm by default, including dense inference. This can
change BF16 rounding relative to the older explicit normalization path.
+1 -1
View File
@@ -30,7 +30,7 @@ FastVideo and the reference model first produce different numbers?"
| General logging | `init_logger(__name__)` |
| Per-stage timing | `FASTVIDEO_STAGE_LOGGING` |
| Profiling kernel timings | `FASTVIDEO_TORCH_PROFILER_DIR` (see [Profiling](profiling.md)) |
| Function-call tracing | `fastvideo.logger.enable_trace_function_call()` (heavy) |
| Function-call tracing | `FASTVIDEO_TRACE_FUNCTION` (heavy) |
## Quickstart
+2 -10
View File
@@ -4,11 +4,6 @@ This is the canonical reference for FastVideo's CI/CD system. Contributor-facing
PR steps live in [Pull Requests](pull_requests.md), and test-authoring guidance
lives in [Testing](testing.md).
The existing Slurm route below remains the default. Operators can also install
the [selectable GPU dispatcher](gpu_ci_backends.md) to run the same lane scripts
on Modal or Kubernetes in the `vllm` namespace. That opt-in path enforces two
active PRs, four GPUs per PR, and eight total, with separate backend statuses.
## Overview
FastVideo splits validation across GitHub Actions, Buildkite, Slinky Slurm,
@@ -440,11 +435,8 @@ before later jobs consume the updated image pin.
The same workflow publishes a single-architecture ARM64, CUDA 13, SM100 image
for the self-hosted CI runner under the
`py3.12-cuda13.0.0-sm100-{latest,sha-*}` tags. It carries the matching prebuilt
kernel wheel compiled with `TORCH_CUDA_ARCH_LIST=10.0a` to include the GB200
VSA CUDA extensions. Runtime kernel detection uses the same target, so runner
jobs can validate and install the exact source and ABI match instead of
recompiling it in every lane. Older artifacts built for `10.0` have a different
cache key and trigger a local rebuild when the worker detects `10.0a`.
kernel wheel so runner jobs can validate and install the exact source and ABI
match instead of recompiling it in every lane.
The optional Dreamverse matrix builds backend and UI images for CUDA 12.6 and
CUDA 13 on `amd64`. Dreamverse remains `amd64`-only because its FA4 dependency
-242
View File
@@ -1,242 +0,0 @@
# Environment Variables
FastVideo reads environment variables for expert switches, debugging, profiling, and the settings that launchers such
as `torchrun` provide. This page is the policy for those variables. The contract test
`fastvideo/tests/contract/test_env_policy.py` enforces the policy in the unit CI lane, and the coding-agent skill
`.agents/skills/env-var-conventions/SKILL.md` points here. When the policy changes, update this page and the contract
test in the same pull request.
## Rules
1. **Register every FastVideo variable in `fastvideo/envs.py`.** Each entry declares a type, a default, a category,
and a description. Variables that other tools own (CUDA, NCCL, PyTorch, launchers) are not registered; code reads
them directly with `os.environ.get("NAME")`, and the name must be in the external-variable allowlist
(`EXTERNAL_ALLOWLIST` in the contract test). When FastVideo sets such a variable for the other tool, it calls
`envs.set_external`, `envs.setdefault_external`, or `envs.unset_external`, and the name must be in
`EXTERNAL_WRITE_ALLOWLIST`.
2. **Read with `envs.NAME.get()`, write with `envs.NAME.set()`, and change a value in tests with
`envs.NAME.override()`.** Each type has one parsing rule. A value that the rule rejects raises
`fastvideo.envs.EnvVarError` instead of falling back to the default.
3. **Name FastVideo variables with the `FASTVIDEO_` prefix.** The second word states the purpose where one applies:
`ENABLE_`, `DISABLE_`, `USE_`, `FORCE_`, `DEBUG_`, `TEST_`.
4. **Keep a renamed variable as a deprecated alias until the next minor release.** Setting the old name logs a
warning. Delete a variable that no code reads, and list it in `DEPRECATED_VARIABLES` so that setting it logs a
warning.
5. **Give each setting one source: an argument or an environment variable.** Settings that users change per
deployment are arguments (CLI or YAML). Expert switches, emergency off switches, and debugging and test switches
are environment variables.
6. **Read variables inside functions.** `envs.NAME.get()` runs when the function runs, so a changed value takes
effect without re-importing a module. Module level, class bodies, decorators, and default argument values run at
import time.
7. **Do not write the environment to pass values between parts of FastVideo.** Pass an argument instead. Tests use
`envs.NAME.override()`.
## Field types
| Class | Value type | Parsing rule |
| ------------ | ---------------- | ----------------------------------------------------------------------------------- |
| `EnvBool` | `bool` | `1`, `true`, `yes`, `on` are true; `0`, `false`, `no`, `off`, and `""` are false. |
| | | Case-insensitive; surrounding whitespace is ignored. |
| `EnvInt` | `int` | `int(value)` |
| `EnvFloat` | `float` | `float(value)` |
| `EnvStr` | `str` or `None` | The raw string. A `None` default means that the variable has no default. |
| `EnvPath` | `str` or `None` | The raw string with a leading `~` expanded. |
| `EnvChoice` | `str` | Stripped and lower-cased, then checked against the declared `choices`. |
A default can be a zero-argument function; `get()` calls it on each read while the variable is unset. The path roots
use this to follow `XDG_CONFIG_HOME` and `XDG_CACHE_HOME`.
Using a field without a method, as in `if envs.FASTVIDEO_FA4:`, raises `TypeError`.
## Add a variable
1. Declare the variable in the matching section of `fastvideo/envs.py`:
```python
FASTVIDEO_DEBUG_MY_STAGE = EnvBool(False, category="debug", doc="Log the inputs of MyStage.")
```
The category is one of the values in `envs.CATEGORIES`.
2. Read the variable inside a function:
```python
import fastvideo.envs as envs
def forward(self, batch):
if envs.FASTVIDEO_DEBUG_MY_STAGE.get():
logger.info("MyStage inputs: %s", batch.keys())
```
3. Regenerate the table at the end of this page:
```bash
python fastvideo/tests/contract/test_env_policy.py
```
4. Run the contract test:
```bash
pytest fastvideo/tests/contract/test_env_policy.py
```
In a test, change the value with `override`, which restores the previous value on exit:
```python
with envs.FASTVIDEO_DEBUG_MY_STAGE.override(True):
run_stage()
```
## Rename or remove a variable
To rename a variable, declare it under the new name and list the old name in `deprecated_names`:
```python
FASTVIDEO_LTX2_USE_DISTILLED_SIGMAS = EnvBool(True,
category="sampling",
doc="...",
deprecated_names=("LTX2_USE_DISTILLED_SIGMAS", ))
```
`get()` reads an old name only when the new name is unset, and logs a warning once. Update the uses of the old name
in `examples/`, `scripts/`, `docs/`, `apps/`, and the tests in the same pull request. Delete the old name in the next
minor release.
To remove a variable that no code reads, delete its entry and add the name to `DEPRECATED_VARIABLES` in
`fastvideo/envs.py` with a reason. `FastVideoArgs` calls `envs.warn_deprecated_variables()`, which logs a warning for
each listed variable that is set. Delete the entry in the next minor release.
## What the contract test checks
The test parses every Python file under `fastvideo/`, including `fastvideo/tests/`, with Python's `ast` module. It
skips `fastvideo/third_party/`, which is copied from upstream projects, and the registry `fastvideo/envs.py`. It does
not check `apps/`, `examples/`, `scripts/`, `fastvideo-kernel/`, or `docs/`.
It reports each violation as `<path>: <kind> <name>`:
| Kind | Code that triggers it | Fix |
| ------------------ | ------------------------------------------------------------ | -------------------------------------------- |
| `read` | `os.getenv`, `os.environ.get`, `os.environ[...]`, or | Register the variable and call |
| | `"NAME" in os.environ` with a name outside the allowlist, or | `envs.NAME.get()`. For a variable that |
| | with a name built at runtime (`<dynamic>`) | another tool owns, add it to |
| | | `EXTERNAL_ALLOWLIST` with a reason. |
| `write` | `os.environ[...] = ...`, `setdefault`, `pop`, `del`, | Pass an argument instead. In tests, use |
| | `os.putenv`, `os.unsetenv`, `monkeypatch.setenv`/`delenv`, | `envs.NAME.override()`. For a variable that |
| | or an `envs.*_external` call with a name outside | another tool reads, call an |
| | `EXTERNAL_WRITE_ALLOWLIST` | `envs.*_external` helper and add the name to |
| | | `EXTERNAL_WRITE_ALLOWLIST` with a reason. |
| `whole-environ` | `os.environ.copy()`, `dict(os.environ)`, iteration, | Read the specific variables that the code |
| | `mock.patch.dict(os.environ, ...)`, `os.environ.update` | needs. |
| `bare-field` | A registry field used without calling one of its methods, | Call `envs.NAME.get()`. |
| | as in `envs.NAME == "auto"` or `getter = envs.NAME.get` | |
| `import-time-read` | `envs.NAME.get()` outside a function | Move the read into the function that uses |
| | | the value. |
| `prefix` | A registry entry without the `FASTVIDEO_` prefix | Rename the variable and keep the old name in |
| | | `deprecated_names`. |
| `unread` | A registry entry that no code reads with `get()` or | Delete the variable and add it to |
| | `is_set()` | `DEPRECATED_VARIABLES`. |
The test recognizes `os` imported under another name, `from os import environ, getenv`, and a name held in a
module-level string constant. Code that reaches the environment through `importlib` or `getattr(os, "environ")` is
left to code review.
The test also checks that every registry entry has a category from `envs.CATEGORIES` and a description, and that the
table at the end of this page matches the registry.
**Known violations.** `KNOWN_VIOLATIONS` in the contract test lists the violations that existed when the policy was
introduced. The list only shrinks. A violation that is not in the list fails the test. A listed violation that no
longer exists also fails the test, so the fixing pull request deletes its entry.
## Registered variables
<!-- BEGIN GENERATED ENV TABLE: python fastvideo/tests/contract/test_env_policy.py -->
| Variable | Type | Default | Category | Description |
| ---------------------------------------------------------- | ------------------------------ | ----------------------------------------------------- | ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `FASTVIDEO_CONFIG_ROOT` | path | computed | path | Root directory for FastVideo configuration files, at runtime and at installation. Defaults to ~/.config/fastvideo, or $XDG_CONFIG_HOME/fastvideo when XDG_CONFIG_HOME is set. |
| `FASTVIDEO_CACHE_ROOT` | path | computed | path | Root directory for FastVideo cache files. Defaults to ~/.cache/fastvideo, or $XDG_CACHE_HOME/fastvideo when XDG_CACHE_HOME is set. |
| `FASTVIDEO_REASON1_WEIGHTS_PATH` | str | unset | path | Local path or Hugging Face id of Reason1 weights to load instead of the checkpoint's own. |
| `FASTVIDEO_HOST_IP` | str | `""` | distributed | IP address of this node when the node has several network interfaces. Set it on each node for multi-node inference. |
| `FASTVIDEO_LOOPBACK_IP` | str | `""` | distributed | Loopback IP address to use instead of the detected one. |
| `FASTVIDEO_RAY_PER_WORKER_GPUS` | float | `1.0` | distributed | GPUs per Ray worker. A fraction lets Ray schedule several actors on one GPU, so other actors can share the GPUs with FastVideo. |
| `FASTVIDEO_NCCL_SO_PATH` | str | unset | distributed | Path to the NCCL library file. Needed because the nccl&gt;=2.19 that PyTorch ships has a bug (https://github.com/NVIDIA/nccl/issues/1234). |
| `FASTVIDEO_HCCL_SO_PATH` | str | unset | distributed | Path to the HCCL library file on Ascend NPUs. Deprecated names: `HCCL_SO_PATH`. |
| `FASTVIDEO_WORKER_MULTIPROC_METHOD` | one of spawn, fork, forkserver | `spawn` | distributed | Multiprocessing start method for worker processes. |
| `FASTVIDEO_ULYSSES_A2A` | one of off, auto | `off` | distributed | Sequence-parallel all-to-all backend. off uses the NCCL path in DistributedAutograd.AllToAll4D. auto uses the fused NVLink kernel when the group is a load-store accessible mesh of 2, 4, 6, or 8 ranks in eager execution, and the NCCL path otherwise. |
| `FASTVIDEO_CONFIGURE_LOGGING` | bool | `1` | logging | Configure logging at import. When true, FastVideo uses its default logging configuration or the file in FASTVIDEO_LOGGING_CONFIG_PATH. |
| `FASTVIDEO_LOGGING_CONFIG_PATH` | str | unset | logging | Path to a JSON logging configuration file. |
| `FASTVIDEO_LOGGING_LEVEL` | str | `INFO` | logging | Default logging level. |
| `FASTVIDEO_LOGGING_PREFIX` | str | `""` | logging | Prefix prepended to every log message. |
| `FASTVIDEO_STAGE_LOGGING` | bool | `0` | logging | Log the time that each pipeline stage takes. |
| `FASTVIDEO_ATTENTION_BACKEND` | str | unset | attention | Attention backend, as an AttentionBackendEnum name such as TORCH_SDPA, FLASH_ATTN, VIDEO_SPARSE_ATTN, SAGE_ATTN, or SAGE_ATTN_THREE. FastVideoArgs uses it when FastVideoArgs.attention_backend is unset. |
| `FASTVIDEO_FA4` | bool | `0` | attention | The FLASH_ATTN backend uses FlashAttention-4 (flash_attn.cute) instead of FA3 or FA2. |
| `FASTVIDEO_MINIMAX_H3_FA4_PACKED_VARLEN` | bool | `0` | attention | MiniMax-H3 dense DiT self-attention uses the FlashAttention-4 packed-varlen entry point. This changes the floating-point reduction order, so it is an inference-only opt-in. |
| `FASTVIDEO_VSA_SM100A` | bool | `0` | attention | VIDEO_SPARSE_ATTN_H3 sends no-grad tile-64 forwards to the data-center Blackwell (sm_100a) kernel. fastvideo-kernel reads the same variable with the same rule. |
| `FASTVIDEO_NVFP4_FA4` | bool | `0` | attention | FlashAttention-4 quantizes Q and K to NVFP4. An explicit nvfp4_fa4 attention implementation argument takes precedence. |
| `FASTVIDEO_DISABLE_ATTENTION_COMPILE` | bool | `1` | attention | Keep attention forward out of torch.compile graphs (torch.compiler.disable). Set it to 0 to let attention constructed under that setting be traced. Setting it explicitly to true also blocks regional compile. |
| `FASTVIDEO_MLX_WINDOW` | int | `0` | attention | MLX FastWan windowed attention size in tokens. 0 uses full attention. |
| `FASTVIDEO_MLX_WINDOW_SINK` | int | `0` | attention | Number of sink tokens that MLX windowed attention always attends to. |
| `FASTVIDEO_INFERENCE_TORCH_COMPILE` | bool | `0` | performance | Compile each DiT transformer block with fullgraph torch.compile at inference. Same as FastVideoArgs.inference_torch_compile=True. |
| `FASTVIDEO_VAE_PARALLEL_DECODE` | bool | `0` | performance | MiniMax-H3 VAE decode splits its temporal chunks across the sequence-parallel ranks instead of running serially on the output rank. Same as FastVideoArgs.vae_parallel_decode=True. |
| `FASTVIDEO_VAE_PARALLEL_ENCODE` | bool | `0` | performance | MiniMax-H3 reference-video VAE encode splits its temporal chunks across the sequence-parallel ranks. Same as FastVideoArgs.vae_parallel_encode=True. |
| `FASTVIDEO_VAE_PARALLEL_DECODE_STRATEGY` | str | unset | performance | Collective that moves chunks in parallel VAE decode: gather (used when unset) or all_gather. |
| `FASTVIDEO_MINIMAX_H3_FUSIONS` | str | `""` | performance | MiniMax-H3 inference-only Triton fusions: all, 1, or a comma-separated subset of modulate,qknorm_rope,swiglu. Empty, 0, or none keeps the eager implementation. |
| `FASTVIDEO_FSDP2_AUTOWRAP` | bool | `0` | performance | FSDP2 shards modules by parameter count instead of the model's shard conditions. Not supported by self-forcing distillation. |
| `FASTVIDEO_FSDP2_MIN_PARAMS` | int | `10000000` | performance | Minimum parameter count of a module that FASTVIDEO_FSDP2_AUTOWRAP shards. |
| `FASTVIDEO_MLX_COMPILE` | bool | `0` | performance | Compile the MLX DiT forward with mx.compile. |
| `FASTVIDEO_MLX_FAST_NORM` | bool | `0` | performance | Use MLX fast normalization kernels. |
| `FASTVIDEO_MLX_DQ_GEMM` | str | `1` | performance | MLX dequantized GEMM for affine-quantized weights: 0 turns it off, 1 uses the measured minimum row count, and an integer sets the minimum row count. |
| `FASTVIDEO_LTX2_VAE_CHANNELS_LAST_3D` | bool | `1` | performance | LTX-2 VAE uses the channels_last_3d memory format. |
| `FASTVIDEO_LTX2_DISABLE_AUDIO_AUTOCAST` | bool | `1` | performance | LTX-2 audio decoding runs without CUDA autocast. Deprecated names: `LTX2_DISABLE_AUDIO_AUTOCAST`. |
| `FASTVIDEO_FLUX2_DISABLE_BF16_REDUCED_PRECISION_REDUCTION` | bool | `0` | performance | Flux denoising disables reduced-precision reductions in bf16 matmuls, which tightens accumulation for the 4-step Klein model. |
| `FASTVIDEO_FFMPEG_BIN` | str | `ffmpeg` | output | ffmpeg executable used to save video with audio. |
| `FASTVIDEO_VIDEO_CODEC` | str | `libx264` | output | ffmpeg video codec for saved videos. |
| `FASTVIDEO_NVENC_PRESET` | str | `p1` | output | NVENC preset when the codec is an \*_nvenc codec. |
| `FASTVIDEO_NVENC_TUNE` | str | `ull` | output | NVENC tune option. |
| `FASTVIDEO_NVENC_RC` | str | `constqp` | output | NVENC rate-control mode. |
| `FASTVIDEO_NVENC_QP` | str | `28` | output | NVENC quantization parameter. |
| `FASTVIDEO_NVENC_BF` | str | `0` | output | NVENC number of B-frames. |
| `FASTVIDEO_X264_PRESET` | str | `ultrafast` | output | x264 preset for non-NVENC codecs. |
| `FASTVIDEO_OUTPUT_PIX_FMT` | str | `yuv420p` | output | ffmpeg pixel format for saved videos. |
| `FASTVIDEO_NVTX_PROFILE` | bool | `0` | profiling | Emit NVTX ranges for external profilers such as Nsight Systems. |
| `FASTVIDEO_TORCH_PROFILER_DIR` | path | unset | profiling | Enables the torch profiler and sets the directory for its traces. Must be an absolute path. |
| `FASTVIDEO_TORCH_PROFILER_RECORD_SHAPES` | bool | `0` | profiling | Torch profiler records shapes. |
| `FASTVIDEO_TORCH_PROFILER_WITH_PROFILE_MEMORY` | bool | `0` | profiling | Torch profiler profiles memory. |
| `FASTVIDEO_TORCH_PROFILER_WITH_STACK` | bool | `0` | profiling | Torch profiler captures stacks. Costs about 1.5x runtime and 1.4x trace size. |
| `FASTVIDEO_TORCH_PROFILER_WITH_FLOPS` | bool | `0` | profiling | Torch profiler profiles FLOPs. |
| `FASTVIDEO_TORCH_PROFILE_REGIONS` | str | `""` | profiling | Comma-separated profiler regions to record. The torch profiler requires at least one region. |
| `FASTVIDEO_TRACE_ACTIVATIONS` | bool | `0` | debug | Enable activation trace hooks. |
| `FASTVIDEO_TRACE_LAYERS` | str | `""` | debug | Regex filter for traced module names. Empty means all. |
| `FASTVIDEO_TRACE_STATS` | str | `abs_mean,sum` | debug | Comma-separated activation statistics dumped for each output tensor. |
| `FASTVIDEO_TRACE_OUTPUT` | str | `/tmp/fv_trace_<pid>.jsonl` | debug | JSONL path for activation traces. The literal &lt;pid&gt; is replaced at runtime. |
| `FASTVIDEO_TRACE_STEPS` | str | `""` | debug | Comma-separated denoising step indices. Empty means all. |
| `FASTVIDEO_H3_VSA_PROBE` | str | unset | debug | Output directory for the VSA-H3 attention-mass probe, which writes one .pt file per step, layer, and rank. Keeps the model out of regional compile. |
| `FASTVIDEO_LTX2_GEMMA_LOG` | str | `""` | debug | Log file for LTX-2 Gemma text-encoder hidden states, used by parity tests. Deprecated names: `LTX2_FASTVIDEO_GEMMA_LOG`. |
| `FASTVIDEO_COSMOS25_LOG_KNOBS` | bool | `0` | debug | Log the Cosmos 2.5 latent-preparation conditioning inputs. |
| `FASTVIDEO_CFG_GATE_STEP` | float | `1.0` | sampling | CFG gating fraction in [0, 1]. Steps before len(timesteps) \* X run the conditional and unconditional forwards; later steps reuse the cached difference. 1.0 disables gating. |
| `FASTVIDEO_LTX2_USE_DISTILLED_SIGMAS` | bool | `1` | sampling | LTX-2 uses the distilled sigma schedule when FastVideoArgs.ltx2_use_distilled_sigmas is also true. Deprecated names: `LTX2_USE_DISTILLED_SIGMAS`. |
| `FASTVIDEO_EVAL_CACHE` | path | computed | eval | Cache directory for evaluation models and datasets. Defaults to $FASTVIDEO_CACHE_ROOT/eval. |
| `FASTVIDEO_PHYSICS_IQ_BUCKET_URL` | str | `https://storage.googleapis.com/physics-iq-benchmark` | eval | Base URL of the Physics-IQ benchmark bucket. |
| `FASTVIDEO_VBENCH_FULL_INFO_JSON` | str | unset | eval | Path to VBench_full_info.json, used instead of the vendored copy. Deprecated names: `VBENCH_FULL_INFO_JSON`. |
| `FASTVIDEO_FVD_REF_FEATURES` | str | unset | eval | Cached reference-feature file for the FVD metric. |
| `FASTVIDEO_FAD_REF_FEATURES` | str | unset | eval | Cached reference-feature file for the audio Frechet distance metric. |
| `FASTVIDEO_TEST_LTX2_OVERFIT_DATA_DIR` | str | `data/cats` | test | Raw data directory for preprocess_ltx2_overfit.py. Deprecated names: `LTX2_OVERFIT_DATA_DIR`. |
| `FASTVIDEO_TEST_LTX2_OVERFIT_CAPTION_JSON` | str | `videos2caption_1_sample.json` | test | Caption file, relative to the raw data directory. Deprecated names: `LTX2_OVERFIT_CAPTION_JSON`. |
| `FASTVIDEO_TEST_LTX2_OVERFIT_VIDEO_SUBDIR` | str | `video` | test | Video subdirectory, relative to the raw data directory. Deprecated names: `LTX2_OVERFIT_VIDEO_SUBDIR`. |
| `FASTVIDEO_TEST_LTX2_OVERFIT_OUTPUT_DIR` | str | `data/ltx2_overfit_preprocessed` | test | Output directory for preprocess_ltx2_overfit.py. Deprecated names: `LTX2_OVERFIT_OUTPUT_DIR`. |
| `FASTVIDEO_TEST_LTX2_OVERFIT_MODEL` | str | `FastVideo/LTX2-Distilled-Diffusers` | test | Model repository whose encoders preprocess_ltx2_overfit.py uses. Deprecated names: `LTX2_OVERFIT_MODEL`. |
| `FASTVIDEO_TEST_LTX2_OVERFIT_NUM_COPIES` | int | `4` | test | Number of copies of the overfit sample in the parquet file. Deprecated names: `LTX2_OVERFIT_NUM_COPIES`. |
| `FASTVIDEO_TEST_KANDINSKY5_OVERFIT_DATA_DIR` | str | `data/kandinsky5_overfit` | test | Raw data directory for preprocess_kandinsky5_overfit.py. Deprecated names: `KANDINSKY5_OVERFIT_DATA_DIR`. |
| `FASTVIDEO_TEST_KANDINSKY5_OVERFIT_OUTPUT_DIR` | str | `data/kandinsky5_overfit_preprocessed` | test | Output directory for preprocess_kandinsky5_overfit.py. Deprecated names: `KANDINSKY5_OVERFIT_OUTPUT_DIR`. |
Variables that FastVideo no longer reads; setting one logs a warning:
| Deprecated variable | Reason |
| ----------------------------------------- | ---------------- |
| `FASTVIDEO_TARGET_DEVICE` | no code reads it |
| `FASTVIDEO_USE_PRECOMPILED` | no code reads it |
| `FASTVIDEO_RINGBUFFER_WARNING_INTERVAL` | no code reads it |
| `FASTVIDEO_ENGINE_ITERATION_TIMEOUT_S` | no code reads it |
| `FASTVIDEO_SERVER_DEV_MODE` | no code reads it |
| `FASTVIDEO_TEST_DYNAMO_FULLGRAPH_CAPTURE` | no code reads it |
| `FASTVIDEO_TRACE_FUNCTION` | no code reads it |
<!-- END GENERATED ENV TABLE -->
-252
View File
@@ -1,252 +0,0 @@
# Selectable GPU CI Backends
GPU CI can use the existing Slurm dispatcher, Modal, or GB200 Kubernetes Jobs
in the `vllm` namespace. GitHub and Buildkite remain the trigger and reporting
systems. `vllm` names the cluster namespace here; tests run FastVideo's existing
lane scripts rather than a vLLM inference server.
The default remains Slurm. The existing `.buildkite/pipeline.yml`, private
Slurm uploader, and dormant Modal launchers are preserved. Deploying the new
trusted uploader is an operator step; merging these files does not change the
live Buildkite pipelines or create cluster resources. See
[CI/CD Architecture](ci_architecture.md) for the existing installation.
## Execution and Limits
```text
GitHub PR / slash command / schedule
-> trusted Buildkite bootstrap
-> slurm: existing validated graph and private dispatcher
-> modal or vllm: one trusted suite coordinator
-> shared PR admission and GPU reservations
-> isolated GPU worker per existing lane
-> lane results, logs, and backend-specific suite status
```
All new Modal and `vllm` coordinators share one admission database. It enforces:
- At most two distinct PRs with admitted GPU work.
- At most four reserved GPUs per PR across its concurrent builds and lanes.
- At most eight reserved GPUs across both backends combined.
- Additional PRs wait without creating GPU workers.
A PR is identified by repository and PR number, so Fastcheck, merge builds,
manual reruns, and separate attempts share its allowance. Its slot remains
occupied between lanes until all admitted builds for that PR finish. A
non-PR build, including a schedule on `main`, consumes its own slot and the
same GPU budget. The preserved Slurm dispatcher has its existing independent
limits; this new admission database does not control legacy Slurm jobs.
Lanes retain their existing one-, two-, or four-GPU requirements. Four-GPU
SSIM and training lanes wait for that PR's other lanes to release capacity.
Selected integration lanes wait for the golden gate and are skipped if it
fails. CPU-only GitHub checks are unaffected. Pending admission is bounded by
`queue_timeout_seconds`, initially six hours. The coordinator's Buildkite
command timeout is eight hours, including admission and lane waits after the
command starts; time waiting for a Buildkite agent is outside that timeout.
Reservations persist before a worker is created. Cancellation releases them
only after worker termination is confirmed. A lost coordinator, uncertain
create response, or unavailable backend retains its reservations until
recovery confirms cleanup. There is no heartbeat expiry that silently frees
GPUs while an old worker could still be running.
## Operator Installation
Use an operator-reviewed immutable checkout under
`/opt/fastvideo-gpu-ci/source`. Install the supplied `scripts/gpu_ci/run` and
`scripts/gpu_ci/upload` wrappers as `/opt/fastvideo-gpu-ci/run` and
`/opt/fastvideo-gpu-ci/upload`. They invoke the reviewed Python entrypoint
with isolated import mode. Install the supplied files instead of generating
shell wrappers from build metadata:
```bash
install -m 755 /opt/fastvideo-gpu-ci/source/scripts/gpu_ci/run /opt/fastvideo-gpu-ci/run
install -m 755 /opt/fastvideo-gpu-ci/source/scripts/gpu_ci/upload /opt/fastvideo-gpu-ci/upload
```
The controller needs Python 3.10 or newer, PyYAML,
the Buildkite agent, and `kubectl`; Modal additionally needs the reviewed
Modal SDK and a controller-side Modal credential.
Copy [the configuration example](../../scripts/gpu_ci/config.example.json)
to `/etc/fastvideo-gpu-ci.json`. Keep the installation and configuration
operator-owned and unwritable by workers. Configure the actual legacy
`slurm_uploader` command, replace each enabled backend's image placeholder
with a reviewed registry digest, and leave `default_backend` as `slurm`
during canaries. The checked-in placeholders deliberately cannot start jobs.
The permanent dispatcher must reach the Kubernetes API independently of a
developer laptop, SSH tunnel, or Tailscale session. The existing CPU development
pod is useful for investigating access, but is not a production CI service.
Use a dedicated controller identity and a separate `gpu-ci-dispatch` Buildkite
queue. The trusted bootstrap also needs access to the existing Slurm uploader
while that backend remains enabled.
Run all coordinators on **one host**, with `state_path` and `artifacts_dir` on
its persistent local disk. The implementation uses SQLite transactions and
local file locks. Do not place the database or locks on Lustre/NFS, run
independent database copies, or scale controller replicas across nodes. Those
configurations would invalidate the global limits. Multiple Buildkite agent
processes on the same host can share the installation and ledger; enough
agents are needed for concurrent suite coordinators and queued work.
Configure agent-owned hooks to skip repository checkout and reject commands
other than the trusted uploader and coordinator. Disable repository hooks
and plugins on this queue. Do not use a PR checkout to load `scripts/gpu_ci`,
its configuration, or its worker entrypoint. Build metadata is validated by
the dispatcher; arbitrary environment variables are not forwarded into GPU
workers. Buildkite, Kubernetes, and Modal control-plane credentials stay on
the dispatcher.
For Kubernetes, provision namespace-scoped permissions to create/get/delete
Jobs and read Pods and their logs. Worker Pods disable service-account token
mounting and request the exact GPU count on ARM64 GB200 nodes. The image must
include the expected `/opt/venv` runtime, CUDA/SM100 support, and the reviewed
FA4 dependencies. Use a separate AMD64 image digest for Modal. The Modal
profile preserves H100 for encoder, custom-kernel, and VSA lanes, and L40S
for the remaining lanes. Its reviewed image must support both SM89 and
SM90a. Attention settings are selected per backend and lane to preserve the
existing Modal and GB200 FA4 profiles; do not replace them with one global
attention override. Validate the images against their respective hardware
before enabling merge gates.
Optional Kubernetes configuration includes `context`, `hf_secret`,
`hf_secret_key`, `cache_pvc`, `cache_subpath`, `artifacts_pvc`, and
`artifacts_subpath`. Only provide a read-only Hugging Face credential when
private/gated downloads require it. PR workers are untrusted and can access
any credential supplied to them, so never provide a reference-publication
or other write-capable token. Prefer a pre-populated read-only cache.
Use dedicated CI PVCs and relative CI subpaths. The personal
`lustre-pvc-vllm` and `nfs-pvc-vllm` claims are rejected. The cache is mounted
read-only; mutable references and locks stay inside the worker. The artifact
init container prepares each Job's dedicated artifact subdirectory before
the worker mounts it. An
operator quota or admission policy scoped to CI can provide a second GPU
ceiling; do not apply an eight-GPU quota to the shared `vllm` namespace if it
would also cap other users' work.
## Wire Every Buildkite Entry Pipeline
Set each pipeline's operator-owned bootstrap command to
`/opt/fastvideo-gpu-ci/upload`. Apply this to all three entry pipelines:
| Pipeline | Trigger and scope |
|---|---|
| `pr-fastcheck` | Automatic PR webhook; `TEST_SCOPE=fastcheck` or unset. |
| `ci` | Existing API triggers for Fastcheck, full, merge, direct, and scheduled SSIM. Keep its incoming PR webhook disabled to avoid duplicate builds. |
| `fastvideo-performance-lane` | Existing schedule; `TEST_SCOPE=direct`, `TEST_TYPE=performance`. |
The wrapper resolves `CI_GPU_BACKEND` from the build environment, falling
back to `default_backend` in the trusted configuration. Allowed values are
exactly `slurm`, `modal`, and `vllm`; unknown values fail. For Slurm, upload
delegates to the unchanged private uploader. For Modal or `vllm`, it uploads
one fixed `/opt/fastvideo-gpu-ci/run` command on the dedicated queue, with
the selected backend and scope pinned in step environment.
Set `CI_GPU_BACKEND=vllm` in the Buildkite build environment for a rack canary,
or `modal` for a Modal canary. The existing API build payload can carry the
same string in its `env` object. Automatic PR webhook builds inherit the
operator's configured default unless pipeline/build configuration explicitly
overrides it. Do not put backend selection inside a PR-controlled command.
Keep the existing exact `BUILDKITE_COMMIT`, repository, PR identity, and
`TEST_SCOPE` metadata. Direct runs also need an allowlisted `TEST_TYPE`.
Merge runs need the trusted base-branch planner's `MERGE_TEST_PLAN`,
`MERGE_GOLDEN_TESTS`, and `MERGE_SSIM_TESTS`. Full/direct/scheduled quality
runs keep their complete matrices. The worker fetches and verifies the
immutable commit before installing dependencies or invoking a lane script.
The new Modal adapter uses bounded one-to-four-GPU sandboxes and the same
lane scripts as Kubernetes. It does not reactivate `pr_test.sh` or the old
Modal SSIM fan-out, which cannot enforce this shared four-GPU-per-PR budget.
The legacy files remain available for their existing manual workflows.
## Statuses and Default-Backend Cutover
The selected adapter reports separate suite contexts:
| Scope | GitHub context |
|---|---|
| Fastcheck | `gpu-ci/<backend>/fastcheck-passed` |
| Merge or explicit full suite | `gpu-ci/<backend>/full-suite-passed` |
| Direct lane | `gpu-ci/<backend>/direct-test-completed` |
| Scheduled SSIM | `gpu-ci/<backend>/scheduled-ssim-passed` |
The repository variable `CI_GPU_BACKEND` controls which new backend can
publish the existing required `fastcheck-passed` and `full-suite-passed`
contexts. Empty or `slurm` preserves the existing status behavior. `modal`
or `vllm` enables `ci-gpu-backend-status.yml`, which reads the latest statuses
for that one backend and mirrors both required contexts. A missing result
becomes pending; failure and error remain failures. It never combines one
backend's Fastcheck with another backend's full-suite result. Late status
events re-read current state instead of replaying stale event payloads.
Direct tests are diagnostic and do not promote a whole suite on the new
backends. Rerun the matching Fastcheck or merge/full suite to clear its gate.
Per-build tests on the other new backend remain separate diagnostics. The
legacy direct-test aggregation workflow is disabled while a new backend is
selected. These workflows retain the existing trust assumption that only
authorized status-writing integrations can publish CI status contexts.
For production cutover, drain existing Slurm builds: its preserved pipeline
still emits canonical status contexts. The new uploader rejects a Slurm
override when `default_backend` is `modal` or `vllm`, preventing later legacy
builds from overwriting the promoted backend's checks. Ensure no old bootstrap
bypasses the new uploader. Synchronize the operator `default_backend` and
the GitHub repository variable, then clear or rerun required checks for all
open PRs. Old green contexts do not become new-backend validation merely
because a setting changed. Run both Fastcheck and the merge/full suite on
the promoted backend before allowing merge. Apply the same drain and rerun
procedure when rolling back to Slurm.
## Validation and Recovery
Inspect the rendered opt-in pipeline without submitting a build:
```bash
CI_GPU_BACKEND=vllm TEST_SCOPE=fastcheck \
/opt/fastvideo-gpu-ci/venv/bin/python -I \
/opt/fastvideo-gpu-ci/source/scripts/gpu_ci/entrypoint.py render \
--config /etc/fastvideo-gpu-ci.json
```
Validate a one-GPU lane, then a two-/four-GPU lane and the complete Fastcheck
suite. Exercise two PRs plus a third waiter, concurrent builds of the same
PR, cancellation, retries, and coordinator restart. Confirm observed GPU
reservations never exceed two PRs, four per PR, or eight in total. A GPU
reservation includes a pending worker, so unavailable nodes cannot cause
the controller to submit more work than its allowance.
Run SSIM, training, and performance canaries separately. References must
match the effective GPU/runtime/attention backend; do not silently reuse
L40S performance results as GB200 baselines or reseed references as part of
routine CI. Workers keep W&B offline and disable reference publication.
Inspect the persistent ledger and recover an abandoned attempt with:
```bash
/opt/fastvideo-gpu-ci/venv/bin/python -I \
/opt/fastvideo-gpu-ci/source/scripts/gpu_ci/entrypoint.py status \
--config /etc/fastvideo-gpu-ci.json
/opt/fastvideo-gpu-ci/venv/bin/python -I \
/opt/fastvideo-gpu-ci/source/scripts/gpu_ci/entrypoint.py recover \
--config /etc/fastvideo-gpu-ci.json --build-id BUILD_ID.JOB_ID
```
Use the exact ledger ID from `status`; each Buildkite retry has a distinct
job ID. Recovery refuses a live coordinator, persists cancellation, stops
owned resources, and releases reservations only after confirming termination.
If creation or cleanup is ambiguous, investigate the recorded handle on its
backend and retain the reservation until the outcome is known. Do not delete
the database or manually zero counters to unblock the queue.
The dispatcher uploads controller logs, per-lane numeric results, request
metadata, and the suite summary to Buildkite. These are the sources for its
exit status. Generated videos and JUnit files stay in the worker unless a
dedicated Kubernetes artifact PVC is configured; automatic publication of
those worker files and Modal worker artifacts is not implemented. This is
a deployment limitation to account for before replacing existing artifact
review workflows.
+1 -1
View File
@@ -5,7 +5,7 @@ hide:
# Cosmos recipes
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="cosmos" data-recipes="../../assets/cookbook-recipes.json?v=11">
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="cosmos" data-recipes="../../assets/cookbook-recipes.json?v=9">
<header class="cookbook-family-header">
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
<div class="cookbook-family-header__body">
+1 -1
View File
@@ -5,7 +5,7 @@ hide:
# FLUX recipes
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="flux" data-recipes="../../assets/cookbook-recipes.json?v=11">
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="flux" data-recipes="../../assets/cookbook-recipes.json?v=9">
<header class="cookbook-family-header">
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
<div class="cookbook-family-header__body">
+1 -1
View File
@@ -5,7 +5,7 @@ hide:
# GLM-Image recipes
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="glm_image" data-recipes="../../assets/cookbook-recipes.json?v=11">
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="glm_image" data-recipes="../../assets/cookbook-recipes.json?v=9">
<header class="cookbook-family-header">
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
<div class="cookbook-family-header__body">
+1 -1
View File
@@ -5,7 +5,7 @@ hide:
# Hunyuan recipes
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="hunyuan" data-recipes="../../assets/cookbook-recipes.json?v=11">
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="hunyuan" data-recipes="../../assets/cookbook-recipes.json?v=9">
<header class="cookbook-family-header">
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
<div class="cookbook-family-header__body">
+1 -1
View File
@@ -41,7 +41,7 @@ hide:
<span class="cookbook-family-tile__footer">
<span class="cookbook-family-tile__footer-top">
<span><strong>MiniMax H3</strong><small>Video + stereo audio</small></span>
<span class="cookbook-count">9 recipes</span>
<span class="cookbook-count">8 recipes</span>
</span>
<ul class="cookbook-mode-row">
<li>T2VA</li>
+1 -1
View File
@@ -5,7 +5,7 @@ hide:
# Kandinsky 5 recipes
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="kandinsky5" data-recipes="../../assets/cookbook-recipes.json?v=11">
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="kandinsky5" data-recipes="../../assets/cookbook-recipes.json?v=9">
<header class="cookbook-family-header">
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
<div class="cookbook-family-header__body">
+1 -1
View File
@@ -5,7 +5,7 @@ hide:
# LongCat recipes
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="longcat" data-recipes="../../assets/cookbook-recipes.json?v=11">
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="longcat" data-recipes="../../assets/cookbook-recipes.json?v=9">
<header class="cookbook-family-header">
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
<div class="cookbook-family-header__body">
+1 -1
View File
@@ -5,7 +5,7 @@ hide:
# LTX recipes
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="ltx2" data-recipes="../../assets/cookbook-recipes.json?v=11">
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="ltx2" data-recipes="../../assets/cookbook-recipes.json?v=9">
<header class="cookbook-family-header">
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
<div class="cookbook-family-header__body">
+1 -1
View File
@@ -5,7 +5,7 @@ hide:
# Matrix Game recipes
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="matrixgame" data-recipes="../../assets/cookbook-recipes.json?v=11">
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="matrixgame" data-recipes="../../assets/cookbook-recipes.json?v=9">
<header class="cookbook-family-header">
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
<div class="cookbook-family-header__body">
+15 -15
View File
@@ -5,13 +5,13 @@ hide:
# MiniMax H3 recipes
FastH3 is two distilled MiniMax-H3 checkpoints. **V1** is the four-step
launch. Some Hub repo names still say Preview. That name is historical. V1 is
a full model, not a demo. **V2** is the eight-step checkpoint. More forwards
is why V2 is the higher-quality FastH3. The V2 schedule contract is in
The **FastH3 8-Step V2** recipe below runs the eight-forward
[`FastVideo/FastVideo-FastH3-8-Step-V2`](https://huggingface.co/FastVideo/FastVideo-FastH3-8-Step-V2)
checkpoint; its schedule contract is documented in
[FastH3 distilled checkpoint schedules](../inference/fasth3-distilled.md).
The four-forward FastH3 Preview recipes are unchanged.
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="minimax_h3" data-default-recipe="fasth3-preview-cuda" data-recipes="../../assets/cookbook-recipes.json?v=11">
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="minimax_h3" data-default-recipe="fasth3-preview-cuda" data-recipes="../../assets/cookbook-recipes.json?v=9">
<header class="cookbook-family-header">
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
<div class="cookbook-family-header__body">
@@ -22,7 +22,7 @@ is why V2 is the higher-quality FastH3. The V2 schedule contract is in
<p class="cookbook-eyebrow">Primary focus · Inference</p>
<h2>MiniMax H3 recipes</h2>
<p>Generate video and audio with H3. Run a server on CUDA, one DGX Spark, or Apple Silicon MLX to iterate on prompts, or call the pipeline directly from Python.</p>
<span class="cookbook-count" data-cookbook-count>9 maintained recipes</span>
<span class="cookbook-count" data-cookbook-count>8 maintained recipes</span>
</div>
</div>
<div class="cookbook-lifecycle" aria-label="Lifecycle stages">
@@ -47,9 +47,9 @@ is why V2 is the higher-quality FastH3. The V2 schedule contract is in
<h2 id="h3-modes-heading">Supported modes</h2>
<p>
CUDA covers T2VA, FL2VA, and Ref2VA on the full checkpoint, plus FastH3
V1 and FastH3 V2. FastH3 V1 also has a DGX Spark runtime with
a 1-Spark or 2-Spark device row. MLX is T2VA only: V1 and V2.
Temporal <code>--fast</code>, spatial <code>--fast-spatial</code>, and opt-in VSA are flags on the same
Preview and FastH3 LoRA. FastH3 Preview also has a DGX Spark runtime with
a 1-Spark or 2-Spark device row. MLX is T2VA only. Temporal <code>--fast</code>,
spatial <code>--fast-spatial</code>, and opt-in VSA are flags on the same
MLX script, not extra recipes.
</p>
<div class="cookbook-modes__table-wrap">
@@ -64,8 +64,8 @@ is why V2 is the higher-quality FastH3. The V2 schedule contract is in
<tbody>
<tr>
<td>T2VA</td>
<td>Full H3, FastH3 V1, FastH3 LoRA, FastH3 V2</td>
<td>FastH3 V1 or FastH3 V2 after a local DiT conversion</td>
<td>Full H3, FastH3 Preview, FastH3 LoRA</td>
<td>FastH3 Preview after a local DiT conversion</td>
</tr>
<tr>
<td>FL2VA</td>
@@ -99,7 +99,7 @@ is why V2 is the higher-quality FastH3. The V2 schedule contract is in
</tr>
<tr>
<td>DGX Spark</td>
<td>FastH3 V1 on one GB10, or two Sparks with Ray sequence parallel (<code>sp_size=2</code>) over QSFP RoCE. Select NVIDIA DGX Spark, then 1 Spark or 2 Sparks.</td>
<td>FastH3 Preview on one GB10, or two Sparks with Ray sequence parallel (<code>sp_size=2</code>) over QSFP RoCE. Select NVIDIA DGX Spark, then 1 Spark or 2 Sparks.</td>
<td>Not wired</td>
</tr>
</tbody>
@@ -257,7 +257,7 @@ cd FastVideo</code></pre>
<pre><code>UV_TORCH_BACKEND=cu130 uv pip install -e ".[fasth3]"</code></pre>
<p class="cookbook-eyebrow">Apple Silicon</p>
<pre><code>uv pip install -e ".[mlx]"</code></pre>
<p>Follow the <a href="../../getting_started/installation/mlx/">MLX install guide</a> for the extra, <code>ffmpeg</code>, and a clone. Then pick FastH3 V1 or V2 in the builder above.</p>
<p>Follow the <a href="../../getting_started/installation/mps/#run-fasth3-preview">Apple Silicon guide</a> for the download, conversion, and storage requirements.</p>
<p class="cookbook-eyebrow">NVIDIA DGX Spark</p>
<pre><code>UV_TORCH_BACKEND=cu130 uv pip install -e .</code></pre>
<p>Follow the <a href="../../getting_started/installation/spark/">DGX Spark install guide</a> for ARM64 CUDA 13. One Spark is a local process. Two Sparks need Ray on the QSFP link:</p>
@@ -272,9 +272,9 @@ cd FastVideo</code></pre>
<ul>
<li>The full CUDA H3 examples request four GPUs by default. Their sources do not claim a GPU model or memory minimum.</li>
<li>The FastH3 CUDA performance profile was measured on four GB200 GPUs. Use its strict profile when exact operation order matters more than the measured performance configuration.</li>
<li>The MLX source runtime supports T2VA, optional temporal <code>--fast</code>, optional spatial <code>--fast-spatial</code>, and opt-in VSA on <code>--include-vsa</code> checkpoints. FastH3 V2 MLX converts with <code>--include-vsa</code> and runs eight forwards. FL2VA, Ref2VA, and two-pass refinement are not wired.</li>
<li>The MLX source runtime supports T2VA, optional temporal <code>--fast</code>, optional spatial <code>--fast-spatial</code>, and opt-in VSA on <code>--include-vsa</code> checkpoints. FL2VA, Ref2VA, and two-pass refinement are not wired.</li>
<li>GPU count and VAE decode backend are configurable in the builder above for FastH3 CUDA recipes. Only the value shown by default has a recorded run; other supported values are unmeasured here.</li>
<li>DGX Spark is a runtime on FastH3 V1, not a separate family card. Select NVIDIA DGX Spark, then 1 Spark or 2 Sparks. The CUDA GPU-count knob does not apply to Spark.</li>
<li>DGX Spark is a runtime on FastH3 Preview, not a separate family card. Select NVIDIA DGX Spark, then 1 Spark or 2 Sparks. The CUDA GPU-count knob does not apply to Spark.</li>
<li>GB10 has no FA4 / sm_100a VSA kernel. Keep <code>FASTVIDEO_FA4=0</code> and <code>FASTVIDEO_VSA_SM100A=0</code>. Legal <code>num_frames</code> values are <code>17n+5</code>, capped at 362 (15.08 s). A 345-frame request on one Spark can OOM.</li>
<li>Gated or missing checkpoints: run <code>huggingface-cli login</code> and confirm you accepted the model's license on Hugging Face.</li>
</ul>
+1 -1
View File
@@ -5,7 +5,7 @@ hide:
# MMAudio recipes
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="mmaudio" data-recipes="../../assets/cookbook-recipes.json?v=11">
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="mmaudio" data-recipes="../../assets/cookbook-recipes.json?v=9">
<header class="cookbook-family-header">
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
<div class="cookbook-family-header__body">
+6 -33
View File
@@ -8,9 +8,8 @@ JavaScript client examples use that interface, but requests go to FastVideo.
You do not need an OpenAI account or cloud key.
The [H3 recipe selector](minimax-h3.md) provides the same workflow with runtime
selection. This guide covers FastH3 V1 (four forwards) and FastH3 V2
(eight forwards). Start a server, then iterate in the playground or with the
OpenAI Python client. Other H3 recipes keep their direct Python commands.
selection. This guide covers FastH3 Preview text-to-video/audio. Other H3
recipes keep their direct Python commands.
CUDA requests reuse one loaded `VideoGenerator`. The Python SDK can do the same
when you reuse the generator across `generate()` calls. MLX keeps one
@@ -37,14 +36,6 @@ advertises it as `fasth3`. It configures four CUDA GPUs but does not record
a GPU model or VRAM requirement. This is a source-backed server profile, not
the measured GB200 Python performance profile. Compilation is disabled.
For FastH3 V2, use the same install and the 8-step config (nine sigma
points, eight DiT forwards, VSA sparsity 0.8):
```bash
UV_TORCH_BACKEND=cu130 uv pip install -e ".[fasth3]"
fastvideo serve --config examples/serving/openai_fasth3_8step.yaml --server.host 127.0.0.1
```
Keep the server running. In another terminal, check readiness:
```bash
@@ -81,7 +72,7 @@ After model loading completes, the response is `{"status":"ok"}`.
### Apple Silicon MLX
Complete the [MLX install](../getting_started/installation/mlx.md),
Complete the [Apple Silicon installation](../getting_started/installation/mps.md#run-fasth3-preview),
including `ffmpeg`. From your FastVideo clone, install the MLX extra:
```bash
@@ -120,22 +111,6 @@ The MLX server has no recorded device or unified-memory requirement. The
direct Python recipe's M4 Max measurements are not a server benchmark or a
minimum-memory claim.
For FastH3 V2, convert that checkpoint's transformer with `--include-vsa`
and start the 8-step config. Do not overwrite a V1 export. The
[MiniMax H3 cookbook](minimax-h3.md) has the same download and convert
commands as the Python recipe.
```bash
hf download FastVideo/FastVideo-FastH3-8-Step-V2 --local-dir ./FastH3-8-Step-V2
python scripts/checkpoint_conversion/convert_minimax_h3_mlx.py --model-root ./FastH3-8-Step-V2/transformer --out ./FastH3-8-Step-V2-MLX --formats "int8" --include-vsa
python -m fastvideo.entrypoints.openai.mlx_server --config examples/serving/mlx_fasth3_8step.yaml
```
That adapter passes `num_steps=8` when the HTTP field is `num_inference_steps=9`,
and it enables the trained VSA recipe (sparsity 0.8, 64-token tiles). Reuse the
preview VAE, audio VAE, text encoder, and tokenizer if those directories already
exist; edit the YAML paths if your files live elsewhere.
## Open the playground
Open [the local H3 playground](http://127.0.0.1:8000/playground/) after startup.
@@ -161,11 +136,9 @@ or manage a GPU server for you.
## Generate with cURL or an SDK
These examples use the server's resolution, frame count, and sampling defaults.
Do not copy Sora-specific durations or resolutions onto H3. V1 configs use
124 frames, 24 fps, and the five-point distilled sigma schedule with four DiT
forwards. V2 configs use nine sigma points and eight DiT forwards. CUDA
and one Spark use 1344 × 768; MLX uses 832 × 480. The OpenAI Python client is
the same for every FastH3 server that advertises `fasth3`.
Do not copy Sora-specific durations or resolutions onto H3. Both server configs
use 124 frames, 24 fps, and the five-point distilled sigma schedule with four
DiT forwards. CUDA and one Spark use 1344 × 768; MLX uses 832 × 480.
Each client submits a job, checks for completion or failure, and downloads an
MP4 named after the job ID. Polling stops after 30 minutes; a timeout does not
+1 -1
View File
@@ -5,7 +5,7 @@ hide:
# Stable Audio recipes
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="stable_audio" data-recipes="../../assets/cookbook-recipes.json?v=11">
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="stable_audio" data-recipes="../../assets/cookbook-recipes.json?v=9">
<header class="cookbook-family-header">
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
<div class="cookbook-family-header__body">
+1 -1
View File
@@ -5,7 +5,7 @@ hide:
# Stable Diffusion recipes
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="sd35" data-recipes="../../assets/cookbook-recipes.json?v=11">
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="sd35" data-recipes="../../assets/cookbook-recipes.json?v=9">
<header class="cookbook-family-header">
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
<div class="cookbook-family-header__body">
+1 -1
View File
@@ -5,7 +5,7 @@ hide:
# TurboDiffusion recipes
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="turbodiffusion" data-recipes="../../assets/cookbook-recipes.json?v=11">
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="turbodiffusion" data-recipes="../../assets/cookbook-recipes.json?v=9">
<header class="cookbook-family-header">
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
<div class="cookbook-family-header__body">
+2 -2
View File
@@ -5,7 +5,7 @@ hide:
# Wan recipes
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="wan" data-recipes="../../assets/cookbook-recipes.json?v=11">
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="wan" data-recipes="../../assets/cookbook-recipes.json?v=9">
<header class="cookbook-family-header">
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
<div class="cookbook-family-header__body">
@@ -186,7 +186,7 @@ cd FastVideo</code></pre>
<ul>
<li>Out of memory on the A14B recipes: the checked-in sources already enable CPU offload; see <a href="../../inference/configuration/">Configuration</a> for the offload surface before reducing resolution or frames.</li>
<li>The FastWan2.1 recipe requires <code>VIDEO_SPARSE_ATTN</code>; confirm the environment variable in the command was set in the same shell.</li>
<li>FastMetal MLX: install with the <a href="../../getting_started/installation/mlx/">MLX install guide</a>, then pick a FastMetal recipe in the builder. CUDA FastWan-QAD checkpoints are refused on the MLX runtime.</li>
<li>FastMetal MLX: install with <code>uv pip install -e ".[mlx]"</code>, then follow the <a href="../../getting_started/installation/mps/">Apple Silicon guide</a>. CUDA FastWan-QAD checkpoints are refused on the MLX runtime.</li>
<li>FastMetal 5B uses <code>mlx_wan22_generate.py</code>. 1.3B and 14B use <code>mlx_wan_prompt_to_video.py</code>.</li>
<li>Gated or missing checkpoints: run <code>huggingface-cli login</code> and confirm you accepted the model's license on Hugging Face.</li>
</ul>
+1 -1
View File
@@ -5,7 +5,7 @@ hide:
# Z-Image recipes
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="zimage" data-recipes="../../assets/cookbook-recipes.json?v=11">
<div class="cookbook-shell cookbook-family-page" data-cookbook data-family="zimage" data-recipes="../../assets/cookbook-recipes.json?v=9">
<header class="cookbook-family-header">
<a class="cookbook-back-link" href="../"><span aria-hidden="true">←</span> All model families</a>
<div class="cookbook-family-header__body">
+2 -2
View File
@@ -7,7 +7,7 @@ FastVideo supports the following hardware platforms:
- **NVIDIA DGX Spark / GB10 (ARM64 + CUDA 13)** — [install](installation/spark.md),
[performance](installation/spark_performance.md),
[pair two Sparks](installation/spark_pair.md)
- [Apple silicon (MLX)](installation/mlx.md)
- [Apple silicon](installation/mps.md)
## Quick Installation
@@ -15,7 +15,7 @@ FastVideo supports the following hardware platforms:
Use uv as the default environment manager for faster and more stable installs.
The commands below target NVIDIA CUDA 12; use `UV_TORCH_BACKEND=cu130` on
CUDA 13. Apple silicon users should follow the [MLX install guide](installation/mlx.md).
CUDA 13. Apple silicon users should follow the [MPS guide](installation/mps.md).
```bash
# Create and activate a new uv environment
-65
View File
@@ -1,65 +0,0 @@
# Install FastVideo with MLX
Install FastVideo on a Mac, then generate from the cookbook. Local video uses
the native MLX runtime, not CUDA, and not the old PyTorch MPS demo at
`examples/inference/basic/basic_mps.py`.
## Requirements
- macOS 14 or newer
- Python 3.12
- `ffmpeg` (`brew install ffmpeg`)
## Install
Cookbook commands run from a clone.
```bash
git clone https://github.com/hao-ai-lab/FastVideo.git && cd FastVideo
uv venv --python 3.12 --seed
source .venv/bin/activate
brew install ffmpeg
uv pip install -e ".[mlx]"
```
Conda is optional. After you activate a Conda env, still install with
`uv pip` as above.
`uv pip install "fastvideo[mlx]"` from PyPI installs the extra only. It does
not ship the example scripts the cookbook copies.
## Generate a video
Open the cookbook. Select Apple Silicon as the runtime. Each recipe has a
Python command. FastH3 also has a server path for the playground and the
OpenAI Python client.
- [Wan recipes](../../cookbook/wan.md) for FastMetal 1.3B, 5B, and 14B
- [MiniMax H3 recipes](../../cookbook/minimax-h3.md) for FastH3 V1 and FastH3 V2
- [H3 server guide](../../cookbook/openai-api.md) for the playground, cURL, and SDKs
FastH3 is two distilled MiniMax-H3 checkpoints. V1 is the four-step launch.
Some Hub repo names still say Preview. That name is historical. V1 is a full
model, not a demo. V2 is the eight-step checkpoint. More forwards is why V2
is the higher-quality FastH3.
Recorded shapes and evidence live in the
[support matrix](../../inference/support_matrix.md#apple-silicon-native-runtime).
## Hardware
- FastMetal 1.3B and 5B: 16 GB unified memory and up
- FastMetal 14B: 36 GB unified memory and up
- FastH3 V1 and V2: validated on an M4 Max with 36 GB unified memory
## Troubleshooting
- **`basic_mps.py` is the wrong path.** That script is PyTorch MPS. Use an
Apple Silicon recipe in the cookbook.
- **Muxing fails.** Install `ffmpeg` with Homebrew.
- **A cookbook command cannot find a script.** Run it from the FastVideo
clone after `uv pip install -e ".[mlx]"`.
If that does not match what you see, open an issue on the
[GitHub repository](https://github.com/hao-ai-lab/FastVideo) or ask in the
[Slack community](https://join.slack.com/t/fastvideo/shared_invite/zt-3f4lao1uq-u~Ipx6Lt4J27AlD2y~IdLQ).
+260
View File
@@ -0,0 +1,260 @@
# MPS (Apple Silicon)
Install FastVideo on Apple Silicon and run FastMetal-QAD or FastH3 Preview.
Apple Silicon uses the MLX runtime. FastMetal-QAD ships ready-to-run MLX
checkpoints; FastH3 Preview currently requires a local MLX DiT conversion.
See the [FastMetal-QAD blog](https://haoailab.com/blogs/fastmetal/) and the
[FastMetal collection](https://huggingface.co/collections/FastVideo/fastmetal).
## Requirements
- **OS: macOS 14 or newer**
- **Python: 3.12.4**
## Set up using Python
### Create a new Python environment
#### uv
Recommended default: use [uv](https://docs.astral.sh/uv/) for faster and more stable environment setup.
Please follow the [documentation](https://docs.astral.sh/uv/#getting-started) to install `uv`. After installing `uv`, create a new environment using:
```console
# (Recommended) Create a new uv environment. Use `--seed` to install `pip` and `setuptools`.
uv venv --python 3.12 --seed
source .venv/bin/activate
```
#### Conda (alternative)
You can also create a Python environment using [Conda](https://docs.conda.io/projects/conda/en/stable/user-guide/getting-started.html).
##### 1. Install Miniconda (if not already installed)
```bash
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-arm64.sh
bash Miniconda3-latest-MacOSX-arm64.sh
source ~/.zshrc
```
##### 2. Create and activate a Conda environment for FastVideo
```bash
conda create -n fastvideo python=3.12.4 -y
conda activate fastvideo
```
### Dependencies
```
brew install ffmpeg
```
### Installation
FastMetal's native Apple Silicon runtime requires the `mlx` extra.
#### With uv (recommended)
```bash
uv pip install "fastvideo[mlx]"
```
#### With Conda environment (alternative)
`uv` works inside an active conda env too, so prefer `uv pip` for the actual install:
```bash
uv pip install "fastvideo[mlx]"
```
### Installation from Source
#### 1. Clone the FastVideo repository
```bash
git clone https://github.com/hao-ai-lab/FastVideo.git && cd FastVideo
```
#### 2. Install FastVideo
Basic installation:
```bash
uv pip install -e ".[mlx]"
```
Alternative with Conda environment:
```bash
uv pip install -e ".[mlx]"
```
## Run FastMetal-QAD
Each release is self-contained. Download one checkpoint and point both
`--model-root` and `--mlx-checkpoint` at it (the example also auto-detects
`mlx_dit.json` under `--model-root`).
| Checkpoint | Script | Mac tier |
| --- | --- | --- |
| [`FastVideo/FastMetal-1.3B-QAD`](https://huggingface.co/FastVideo/FastMetal-1.3B-QAD) | `mlx_wan_prompt_to_video.py` | 16 GB+ |
| [`FastVideo/FastMetal-5B-QAD`](https://huggingface.co/FastVideo/FastMetal-5B-QAD) | `mlx_wan22_generate.py` | 16 GB+ |
| [`FastVideo/FastMetal-14B-QAD`](https://huggingface.co/FastVideo/FastMetal-14B-QAD) | `mlx_wan_prompt_to_video.py` | 36 GB+ |
```bash
hf download FastVideo/FastMetal-1.3B-QAD --local-dir ./FastMetal-1.3B-QAD
python examples/inference/basic/mlx_wan_prompt_to_video.py \
--model-root ./FastMetal-1.3B-QAD \
--mlx-checkpoint ./FastMetal-1.3B-QAD \
--height 480 --width 832 --num-frames 81 \
--prompt "A bird's-eye view of a misty forest valley at dawn."
```
14B uses the same script. Point both flags at `./FastMetal-14B-QAD`. That repo also ships an EMA variant: keep `--model-root` at the repo root and set `--mlx-checkpoint ./FastMetal-14B-QAD/ema`.
Wan2.2 5B uses a different latent layout, so it has its own entrypoint:
```bash
hf download FastVideo/FastMetal-5B-QAD --local-dir ./FastMetal-5B-QAD
python examples/inference/basic/mlx_wan22_generate.py \
--mlx-checkpoint ./FastMetal-5B-QAD \
--text-encoder-root ./FastMetal-5B-QAD \
--vae-root ./FastMetal-5B-QAD/vae \
--height 704 --width 1280 --num-frames 81 \
--prompt "A cinematic portrait with soft neon lighting and smooth camera motion."
```
CUDA FastWan-QAD (`FastVideo/FastWan-QAD-1.3B`, `FastVideo/FastWan-QAD-FP8-1.3B`) is a separate NVIDIA release. The MLX examples look for FastMetal packed weights (`mlx_dit.json`).
`basic_mps.py` is a generic PyTorch MPS demo. For local video on Mac, use the FastMetal commands above.
## Run FastH3 Preview
FastH3 Preview uses the existing MLX runtime for text-to-video-with-audio
(T2VA). The runtime streams the Qwen3-VL text conditioner, loads one
heavyweight component at a time, denoises synchronized video and audio
latents with a converted INT8, INT6, or INT4 DiT, and decodes both modalities
with native MLX VAEs.
Download the FastH3 snapshot, then convert one or more DiT formats:
```bash
hf download FastVideo/FastVideo-Minimax-FastH3-Preview-v0.2 \
--local-dir ./FastH3-Preview-v0.2
python scripts/checkpoint_conversion/convert_minimax_h3_mlx.py \
--model-root ./FastH3-Preview-v0.2/transformer \
--out ./FastH3-MLX \
--formats "int6"
```
Dense conversion drops the trained VSA gate projections. To keep them (INT6
weight-only, same affine grid as the other linear matrices) write a **new**
directory:
```bash
python scripts/checkpoint_conversion/convert_minimax_h3_mlx.py \
--model-root ./FastH3-Preview-v0.2/transformer \
--out ./FastH3-MLX-vsa \
--formats "int6" \
--include-vsa
```
Do not overwrite an existing dense export such as `./FastH3-MLX/int6`.
Run the baseline path:
```bash
python examples/inference/basic/mlx_fasth3.py \
--model-root ./FastH3-Preview-v0.2 \
--mlx-checkpoint ./FastH3-MLX/int6 \
--prompt "(S1) A presenter says <d>[English] Fast H3 is amazing.</d>" \
--height 480 --width 832 --num-frames 124 --seed 2026 \
--output-path ./outputs/fasth3_int6.mp4
```
Add `--fast` for temporal fast mode. It denoises a shorter video sequence,
uses MLX RIFE to restore the requested frame count, and keeps the audio
sequence at full duration:
```bash
python examples/inference/basic/mlx_fasth3.py \
--model-root ./FastH3-Preview-v0.2 \
--mlx-checkpoint ./FastH3-MLX/int6 \
--prompt "(S1) A presenter says <d>[English] Fast H3 is even faster.</d>" \
--height 720 --width 1280 --num-frames 124 --seed 2027 \
--fast \
--output-path ./outputs/fasth3_int6_fast_720p.mp4
```
Add `--fast-spatial` for spatial fast mode, `--fast`'s spatial twin. It
denoises and decodes on the smallest 32px-aligned canvas covering the
requested size divided by `--fast-spatial-scale` (a 480x832 request runs on a
256x416 canvas), then resamples the decoded frames up to the requested size
in pixel space. It composes with `--fast`. This is a speed/quality trade-off
and stays off by default: the output carries the reduced canvas's detail
budget, so it reads softer than a native-resolution render, with the unsharp
pass countering some but not all of the difference:
```bash
python examples/inference/basic/mlx_fasth3.py \
--model-root ./FastH3-Preview-v0.2 \
--mlx-checkpoint ./FastH3-MLX/int6 \
--prompt "(S1) A presenter says <d>[English] Fast H3 is fastest.</d>" \
--height 480 --width 832 --num-frames 124 --seed 2028 \
--fast --fast-spatial \
--output-path ./outputs/fasth3_int6_fast_spatial.mp4
```
VSA is off by default. A dense-only checkpoint (no `--include-vsa`) keeps the
existing fused-SDPA path. After converting with `--include-vsa`, enable the
sparse path explicitly:
```bash
python examples/inference/basic/mlx_fasth3.py \
--model-root ./FastH3-Preview-v0.2 \
--mlx-checkpoint ./FastH3-MLX-vsa/int6 \
--vsa --vsa-sparsity 0.9 --vsa-tile-size 64 --vsa-prefix-mode exempt \
--prompt "(S1) A presenter says <d>[English] Fast H3 is amazing.</d>" \
--height 720 --width 1280 --num-frames 124 --seed 2026 \
--output-path ./outputs/fasth3_int6_vsa_720p.mp4
```
`--vsa-impl auto` uses the chunked gather+SDPA **reference** path.
`--vsa-impl simd` is an opt-in SIMD-group kernel (tile 64, head dim 128) that
falls back to reference on unsupported shapes. It is not the default.
`--vsa-impl reference` is the same as `auto`.
!!! note "Current MLX scope"
This source runtime supports T2VA, temporal `--fast`, spatial
`--fast-spatial`, and opt-in VSA. FL2VA, Ref2VA, two-pass refinement, and
`VideoGenerator` registry dispatch are not wired yet. INT8/INT6/INT4
quantization is **weight-only**; VSA attention Q/K/V stay BF16. Old dense
MLX checkpoints remain valid for dense inference and raise a reconvert
error if `--vsa` is set. The checkpoint uses the MiniMax H3
Community License; review the model card before use or redistribution.
## Development Environment Setup
If you're planning to contribute to FastVideo please see the following page:
[Contributor Guide](../../contributing/overview.md)
## Hardware Requirements
- **1.3B / 5B:** 16 GB unified memory and up (M1 and later)
- **14B:** 36 GB unified memory and up
- **FastH3 Preview:** validated on an M4 Max with 36 GB unified memory; use one
converted DiT format at a time and leave substantial free disk space for the
source snapshot plus the converted checkpoint
- Fanless 13-inch MacBook Air can run 1.3B and 5B at the same resolutions
## Troubleshooting
If you encounter any issues during installation, please open an issue on our [GitHub repository](https://github.com/hao-ai-lab/FastVideo).
You can also join our [Slack community](https://join.slack.com/t/fastvideo/shared_invite/zt-3f4lao1uq-u~Ipx6Lt4J27AlD2y~IdLQ) for additional support.
+1 -1
View File
@@ -147,7 +147,7 @@ that can't move here.
Two Sparks with QSFP cables: [Pair two NVIDIA DGX Sparks](spark_pair.md) for
one FastH3 clip across both GPUs (`sp_size=2` over Ray). Copy-paste commands
for one or two Sparks also live on the
[MiniMax H3 cookbook](../../cookbook/minimax-h3.md): pick FastH3 V1,
[MiniMax H3 cookbook](../../cookbook/minimax-h3.md): pick FastH3 Preview,
then NVIDIA DGX Spark, then 1 Spark or 2 Sparks.
## Development Environment Setup
@@ -7,7 +7,7 @@ by the QSFP ConnectX-7 cables can run **one clip faster** and can hold a
**longer clip** (up to the FastH3 15 s cap).
Copy-paste commands for both counts are on the
[MiniMax H3 cookbook](../../cookbook/minimax-h3.md): FastH3 V1 → NVIDIA DGX
[MiniMax H3 cookbook](../../cookbook/minimax-h3.md): FastH3 Preview → NVIDIA DGX
Spark → 1 Spark or 2 Sparks.
This is FastVideo sequence parallel (`sp_size=2`) over Ray, not a third-party
@@ -181,9 +181,6 @@ is power-cycled. To avoid it:
and frame counts are valid. Weights stay replicated, so lazy module load
(auto on GB10) is still required on each box. Bring-up and knobs:
[Pair two NVIDIA DGX Sparks](spark_pair.md).
- A worker's SIGTERM log and traceback show where it was interrupted, not why it
was selected; confirm the cause in the `earlyoom` service or system logs. A
later SIGKILL or kernel OOM kill cannot be caught and reported by Python.
## Gotchas specific to the GB10
-15
View File
@@ -73,18 +73,3 @@ unset: H3 has separate video and audio shifts, not one shared shift.
This documents execution support for the published checkpoint. It is not a
quality claim: compare video/audio output against base MiniMax-H3 on your own
prompts before adopting it.
## Apple Silicon
`mlx_fasth3.py` stays on FastH3 V1 and its uniform AdaLN cache.
`mlx_fasth3_8step.py` is the eight-forward MLX recipe for FastH3 V2. It
reads the same `fastvideo_inference.json` rungs and shifts, and it expects an
MLX DiT whose AdaLN cache was converted from that contract. Reuse the V1
snapshot's VAE, audio VAE, text encoder, and tokenizer; only the DiT and the
sidecar change. Rank-reduced AdaLN checkpoints are unchanged and are not
produced by the MLX converter.
Install is in the
[MLX install guide](../getting_started/installation/mlx.md). Generation and
serving are in the [MiniMax H3 cookbook](../cookbook/minimax-h3.md) and the
[H3 server guide](../cookbook/openai-api.md).
+2 -2
View File
@@ -5,10 +5,10 @@ This page contains step-by-step instructions to get you quickly started with vid
## Requirements
- **OS**: Linux (tested on Ubuntu 22.04+), or macOS on Apple silicon via the
[MLX install guide](../getting_started/installation/mlx.md)
[MPS installation guide](../getting_started/installation/mps.md)
- **Python**: 3.10-3.12
- **CUDA**: 12.6 or 13.0 (NVIDIA GPUs)
- **GPU**: At least one NVIDIA GPU, or an Apple silicon chip with the MLX runtime
- **GPU**: At least one NVIDIA GPU, or an Apple silicon chip with MPS
## Installation
+1 -3
View File
@@ -10,9 +10,7 @@ that tradeoff. It is not a lossless acceleration of the full VAE.
## Generate a video
Use your existing MLX FastH3 environment and converted checkpoint. The same
`--video-decode-backend taeh3` flag works on `mlx_fasth3.py` (V1) and
`mlx_fasth3_8step.py` (V2):
Use your existing MLX FastH3 environment and converted checkpoint:
```bash
python examples/inference/basic/mlx_fasth3.py \

Some files were not shown because too many files have changed in this diff Show More