Compare commits

..
Author SHA1 Message Date
SolitaryThinker 59cf9d6fa2 [fix] dreamverse: steering session-flow, mock parity, and init-image validation
Steering protocol/UI fixes that interlock via one contract change — the
frontend now sends initial_rollout_prompt_id (its own prompt id for the
opening scene) in session-init and reset payloads, and both backend pre-seed
sites use it, so prompt lifecycle events (enhancing/ready/fallback) finally
match the frontend's records instead of no-oping against a backend uuid.

- controller: suppress prompt_sources_blocked while a submission is queued or
  enhancing (fixes the blank player at typed-prompt session start); segment
  prompt logging is now opt-in via DREAMVERSE_SEGMENT_PROMPT_LOG, written off
  the event loop, warning on failure (drops the hardcoded author-machine path
  and bare except).
- web: 'Generating next scene' overlay driven by explicit generation state
  instead of waitingForSegmentPrompt prop edges, so auto_extension_updated
  while idle can't strand it over the video; steering scene history is no
  longer truncated by the 24-event prompt feed cap and keeps stable numbers
  for 30+ scene sessions.
- mock server: manual_continuation_mode parity (no rewrite-flow wait, honors
  initial_rollout_prompt_id, no segment cap in manual mode) so the GPU-less
  dev backend works with the steering-only frontend.
- init image: client-side type/size validation on picker and paste paths
  (png/jpeg/webp, 15MB) with a visible error, and ws_max_size raised to 32MiB
  on both servers so a legitimate ~15MB image reaches the backend's own
  validation instead of tripping uvicorn's 16MiB frame cap.
2026-07-16 20:39:32 -07:00
SolitaryThinker 0c90c328c5 [fix] ltx2: drop unreachable video_position_offset_sec kwargs fallback
video_position_offset_sec is a declared parameter of forward, so a caller's
keyword binds to it and never lands in **kwargs — the added
kwargs.get("video_position_offset_sec", 0.0) block was unreachable dead code
with a false comment, and it rebound the local to 0.0. The pre-existing
application of the offset is the only live path; behavior is unchanged.
2026-07-16 20:39:32 -07:00
SolitaryThinker cfb54a3b2a [fix] dreamverse: env-tunable session timeout, restore warmup watchdog, derive SP size in launch script
- SESSION_TIMEOUT_SECONDS: default back to 300 and now reads the
  DREAMVERSE_SESSION_TIMEOUT_SECONDS env var that launch-dreamverse.sh was
  already exporting (previously nothing read it, and the hardcoded 1800 made
  every deployment hold idle GPU-pool slots 6x longer). Fixes the stale
  five-minute-timeout test and adds an override test.
- STARTUP_WARMUP_TIMEOUT_SECONDS: default back to 2400 so the warmup watchdog
  works again; launch-dreamverse.sh exports the existing
  FASTVIDEO_STARTUP_WARMUP_TIMEOUT_SECONDS override (24000) for the slow
  GB200 max-autotune boot.
- launch-dreamverse.sh: derive the DREAMVERSE_SP_SIZE default from the number
  of visible GPUs so the documented CUDA_VISIBLE_DEVICES=0 invocation no
  longer crashes gpu_pool with 'Not enough GPUs'; explicit env still wins.
  setup-dreamverse-env.sh's printed instruction now matches. gpu_pool raises
  a friendly error for non-integer CUDA_VISIBLE_DEVICES entries.
- video_generation.py: drop unused ParallelismConfig import (F401).
2026-07-16 20:39:13 -07:00
SolitaryThinker deb51f1dcc [fix] dreamverse: return last outer JSON object when parsing enhancer replies
The free-form scanner attempted raw_decode at every '{', including braces
inside an already-decoded object, so 'return the last decodable object'
(deliberate: chain-of-thought models emit draft JSON before the final answer)
actually returned the innermost/trailing nested fragment — e.g.
{"next_prompt": ..., "style": {"mood": "noir"}} parsed to {"mood": "noir"}
and enhancement fell back, failing steering requests.

Scan with a position cursor instead: skip past the consumed span of every
successfully decoded object, and skip malformed/truncated spans wholesale via
a balanced-brace scan so their nested fragments can't displace an earlier
complete object. Regression tests cover nested values, sequential drafts,
truncated tails, and the nested rollout shape.
2026-07-16 20:38:57 -07:00
alexzms 79a5930340 [fix] dreamverse: drop steering hint line that overlapped the hero title on mobile
The two-line 'Drive each scene yourself...' hint grew the vertically-centered
landing content, pushing the preset cards up into the absolutely-positioned
hero title on short mobile viewports. Removing it restores the clean spacing.
2026-07-15 23:20:23 +00:00
alexzms 3a0ce7c20d [misc] dreamverse: bump steering generating progress bar to 10.5s 2026-07-15 23:20:23 +00:00
alexzms 0b006a9e46 [feat] dreamverse: robust steering 'Generating next scene' overlay
Drive the generating indicator off whether the new segment has actually
landed (buffered timeline grows past the boundary captured at generation
start) instead of bare playback events, so scrubbing back and replaying to
the end keeps it visible and it clears promptly once frames arrive. Gate the
overlay on playbackReachedEnd like 'Segment complete', and bump the progress
bar ETA to 8s.
2026-07-15 23:20:23 +00:00
alexzms fb9acac514 [feat] dreamverse: steering-only UI — drop auto-rollout mode selector
Remove the pre-session Auto rollout / Steering segmented control and make
manual continuation (steering) the sole mode: default it on in the session
store and keep it on across project/lobby resets. Leaves a short steering
hint in its place.
2026-07-15 23:20:22 +00:00
kevin314 2d78957219 6+2 2026-07-15 23:20:22 +00:00
kevin314 63991d2017 Test 2026-07-15 23:20:22 +00:00
alexzms b6eafbea50 [fix] dreamverse: recover steering after a blocked or failed prompt
When a steering prompt is refused by the enhancer (e.g. content-policy) or enhancement
fails for all providers, the backend enqueues nothing and won't re-emit prompt_sources_
blocked, leaving the UI stuck on the generating overlay. On prompt/fallback_used and
session/error in steering, return to the 'describe the next scene' state, drop the failed
scene from the history (steeringFailed), and surface a retry notice; clear the notice on
the next submit.
2026-07-15 23:20:22 +00:00
alexzms 7b84041642 [feat] dreamverse: steering scene history of per-segment user prompts
Steering mode now shows an elegant list of each scene's prompt above the player. The
text comes from the user's own words, captured stably at submit time as rawText (the
backend later overwrites text/source with the enhanced prompt, so those are never read);
a preset's opening scene with no user prompt falls back to promptHistory. The redundant
ChatBar 'Segment complete' banner is dropped (the video overlay already says it, and it
was squeezing the list), and a ResizeObserver re-pins the list to the latest scene when
the area resizes.
2026-07-15 23:20:22 +00:00
kevin314 48801c29c7 Add image input UI 2026-07-15 23:20:22 +00:00
alexzms cb8be0d3f3 [feat] dreamverse: main-UI steering mode switch + first-segment-only seeding
Add a user-facing Auto rollout / Steering segmented control to the main composer (not just
devtools), wired to manualContinuationMode and authoritative at session start. In steering
mode a preset seeds only its first segment and auto/loop are forced off, so the backend waits
for the user to describe each subsequent scene by hand.
2026-07-15 23:20:22 +00:00
alexzms 1bc7ff79e6 [ui] dreamverse: logo links home, replace Join Waitlist with Blog
The FastVideo logo now navigates to the app home (most intuitive), and the
Join-Waitlist buttons (header desktop/mobile + session-ended card) become a Blog
link pointing at the Dreamverse blog.
2026-07-15 23:20:21 +00:00
alexzms 76fbe472ae [feat] dreamverse: allow video download at any time during playback
handleDownloadVideo already remuxes live (including in-progress) segments, but the button
was gated on a finalized clip blob. Surface it as soon as playback starts (avPlaybackStarted)
so the user can grab the in-progress video at any point — important for unbounded steering
sessions.
2026-07-15 23:20:21 +00:00
alexzms eba74c43ef [feat] dreamverse: graceful segment-complete & generating overlays in steering playback
When a segment finishes in steering mode the player no longer spins. Instead it shows a
soft 'Segment complete' prompt over the frozen last frame (gated on the playhead actually
reaching the buffered end, and hidden again when the user scrubs back). After the user
submits the next scene, a ~4.5s progress bar covers the generation latency so the wait has
a visible ETA.
2026-07-15 23:20:21 +00:00
alexzms 7744a74c13 [feat] dreamverse: steering-mode toggle in devtools composer
Add a 'Steering mode' checkbox to the devtools composer and thread the
manualContinuationEnabled / onManualContinuationToggle props through DevtoolsShell.
2026-07-15 23:20:21 +00:00
alexzms d746b8b956 [feat] dreamverse: unlimited segments in steering mode
Steering (manual continuation) lets the user drive the rollout segment-by-segment
indefinitely. Treat it like single-clip mode for the generation cap:
_resolve_generation_segment_cap returns 0 (unlimited) and the cap-reached guard is
skipped when manual_continuation_mode is on.
2026-07-15 23:20:21 +00:00
alexzms 65f12605d8 [bugfix] ltx2: apply video_position_offset_sec RoPE offset
The DiT forward swallowed video_position_offset_sec via **kwargs, so multi-segment
rollouts never advanced the temporal RoPE phase between segments, causing ~1s audio/
video desync at each seam. Add the offset to the temporal position coords (mirrors
hao-ai-lab/FastVideo#1422).
2026-07-15 23:20:21 +00:00
kevin314 1e1ac08cd0 Add manual continuation 2026-07-15 23:20:21 +00:00
1174 changed files with 32536 additions and 103821 deletions
@@ -10,7 +10,9 @@ from pathlib import Path
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description="Clone a reference repo for FastVideo parity tests.")
parser = argparse.ArgumentParser(
description="Clone a reference repo for FastVideo parity tests."
)
parser.add_argument("repo_url", help="Official reference repository URL")
parser.add_argument("target_dir", help="Directory to clone into")
parser.add_argument("--branch", help="Branch or tag to clone")
@@ -60,7 +62,9 @@ def gitignore_entry_for(target: Path) -> str:
try:
relative = resolved.relative_to(root)
except ValueError as exc:
raise ValueError("--update-gitignore requires target_dir to be under the current directory") from exc
raise ValueError(
"--update-gitignore requires target_dir to be under the current directory"
) from exc
text = relative.as_posix().rstrip("/")
return "/" + text + "/"
@@ -8,12 +8,14 @@ import os
import sys
from pathlib import Path
HF_TOKEN_ENV_KEYS = ("HF_TOKEN", "HUGGINGFACE_HUB_TOKEN", "HF_API_KEY")
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(
description="Download a HF model snapshot or selected files into a local directory.")
description="Download a HF model snapshot or selected files into a local directory."
)
parser.add_argument("repo_id", help="HF repo id, for example Org/Model")
parser.add_argument("local_dir", help="Destination directory")
parser.add_argument("--repo-type", default="model", help="HF repo type (default: model)")
@@ -10,6 +10,7 @@ import sys
from pathlib import Path
from typing import Any
HF_TOKEN_ENV_KEYS = ("HF_TOKEN", "HUGGINGFACE_HUB_TOKEN", "HF_API_KEY")
RAW_WEIGHT_SUFFIXES = (".safetensors", ".pt", ".pth", ".ckpt", ".bin")
KNOWN_COMPONENTS = {
@@ -33,7 +34,8 @@ KNOWN_COMPONENTS = {
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(
description="Classify a HF repo or local directory as Diffusers, raw, custom, or unknown.")
description="Classify a HF repo or local directory as Diffusers, raw, custom, or unknown."
)
parser.add_argument("source", help="HF repo id or local weights directory")
parser.add_argument("--repo-type", default="model", help="HF repo type (default: model)")
parser.add_argument("--revision", help="HF revision to inspect")
@@ -92,12 +94,14 @@ def load_remote_files(
) -> list[str]:
from huggingface_hub import list_repo_files
return sorted(list_repo_files(
repo_id,
repo_type=repo_type,
revision=revision,
token=token,
))
return sorted(
list_repo_files(
repo_id,
repo_type=repo_type,
revision=revision,
token=token,
)
)
def load_remote_model_index(
@@ -211,24 +215,24 @@ def build_result(args: argparse.Namespace) -> dict[str, Any]:
"components_seen": components,
"file_count": len(files),
"file_scan_truncated": truncated,
"files_sample": files[:args.sample_limit],
"files_sample": files[: args.sample_limit],
}
def print_human(result: dict[str, Any]) -> None:
for key in (
"source",
"source_kind",
"repo_type",
"revision",
"token_env",
"source_layout",
"needs_conversion",
"model_index_class",
"model_index_diffusers_version",
"model_index_error",
"file_count",
"file_scan_truncated",
"source",
"source_kind",
"repo_type",
"revision",
"token_env",
"source_layout",
"needs_conversion",
"model_index_class",
"model_index_diffusers_version",
"model_index_error",
"file_count",
"file_scan_truncated",
):
value = result.get(key)
if value is not None:
@@ -18,6 +18,7 @@ import pytest
import torch
from torch.testing import assert_close
os.environ.setdefault("MASTER_ADDR", "localhost")
os.environ.setdefault("MASTER_PORT", "29519")
os.environ.setdefault("DISABLE_SP", "1")
@@ -34,10 +35,15 @@ FASTVIDEO_CONFIG_CLASS = "<FastVideoConfig>" # TODO.
FASTVIDEO_MODEL_MODULE = "fastvideo.models.<bucket>.<module>" # TODO.
FASTVIDEO_MODEL_CLASS = "<FastVideoModel>" # TODO.
OFFICIAL_REF_DIR = Path(os.getenv("<FAMILY_UPPER>_OFFICIAL_REF_DIR", REPO_ROOT / "<ReferenceDir>"))
LOCAL_WEIGHTS_DIR = Path(os.getenv("<FAMILY_UPPER>_LOCAL_WEIGHTS_DIR", REPO_ROOT / "official_weights" / FAMILY))
CONVERTED_WEIGHTS_DIR = Path(os.getenv("<FAMILY_UPPER>_CONVERTED_WEIGHTS_DIR",
REPO_ROOT / "converted_weights" / FAMILY))
OFFICIAL_REF_DIR = Path(
os.getenv("<FAMILY_UPPER>_OFFICIAL_REF_DIR", REPO_ROOT / "<ReferenceDir>")
)
LOCAL_WEIGHTS_DIR = Path(
os.getenv("<FAMILY_UPPER>_LOCAL_WEIGHTS_DIR", REPO_ROOT / "official_weights" / FAMILY)
)
CONVERTED_WEIGHTS_DIR = Path(
os.getenv("<FAMILY_UPPER>_CONVERTED_WEIGHTS_DIR", REPO_ROOT / "converted_weights" / FAMILY)
)
def _resolve_hf_token() -> str | None:
@@ -93,14 +99,18 @@ def _load_official_model(device: torch.device, dtype: torch.dtype) -> torch.nn.M
model = OfficialClass() # TODO: pass official config kwargs.
state_dict = {} # TODO: load official state dict from LOCAL_WEIGHTS_DIR.
missing, unexpected = model.load_state_dict(state_dict, strict=True)
assert not missing and not unexpected, (f"official load mismatch missing={missing[:5]} unexpected={unexpected[:5]}")
assert not missing and not unexpected, (
f"official load mismatch missing={missing[:5]} unexpected={unexpected[:5]}"
)
return model.to(device=device, dtype=dtype).eval()
def _load_fastvideo_model(device: torch.device, dtype: torch.dtype) -> torch.nn.Module:
"""Load the FastVideo component with the same tensor content."""
if not CONVERTED_WEIGHTS_DIR.exists() and not LOCAL_WEIGHTS_DIR.exists():
pytest.skip(f"No FastVideo loadable weights: {CONVERTED_WEIGHTS_DIR} or {LOCAL_WEIGHTS_DIR}")
pytest.skip(
f"No FastVideo loadable weights: {CONVERTED_WEIGHTS_DIR} or {LOCAL_WEIGHTS_DIR}"
)
# TODO: replace with the bucket-specific FastVideo config/class/loader.
# DiT examples:
@@ -117,7 +127,8 @@ def _load_fastvideo_model(device: torch.device, dtype: torch.dtype) -> torch.nn.
state_dict = {} # TODO: load converted or directly mapped state dict.
missing, unexpected = model.load_state_dict(state_dict, strict=True)
assert not missing and not unexpected, (
f"FastVideo load mismatch missing={missing[:5]} unexpected={unexpected[:5]}")
f"FastVideo load mismatch missing={missing[:5]} unexpected={unexpected[:5]}"
)
return model.to(device=device, dtype=dtype).eval()
@@ -176,9 +187,11 @@ def test_component_parity():
assert official_out.shape == fastvideo_out.shape
diff = (official_out - fastvideo_out).abs()
print(f"official abs_mean={official_out.abs().mean().item():.6f} "
f"fastvideo abs_mean={fastvideo_out.abs().mean().item():.6f} "
f"diff_max={diff.max().item():.6f} diff_mean={diff.mean().item():.6f}")
print(
f"official abs_mean={official_out.abs().mean().item():.6f} "
f"fastvideo abs_mean={fastvideo_out.abs().mean().item():.6f} "
f"diff_max={diff.max().item():.6f} diff_mean={diff.mean().item():.6f}"
)
# TODO: pick tolerance by scope:
# - single block / same kernel: 1e-4
@@ -27,6 +27,7 @@ try:
except ImportError: # pragma: no cover - optional local conversion dependency
snapshot_download = None
# TODO: fill with authoritative component prefixes for monolithic checkpoints.
# Example: {"model.model.": "transformer", "pretransform.model.": "vae"}
COMPONENT_PREFIXES: dict[str, str] = {}
@@ -46,7 +47,10 @@ SKIP_PATTERNS: tuple[str, ...] = ()
def _hf_token() -> str | None:
return (os.environ.get("HF_TOKEN") or os.environ.get("HUGGINGFACE_HUB_TOKEN") or os.environ.get("HF_API_KEY"))
return (
os.environ.get("HF_TOKEN") or os.environ.get("HUGGINGFACE_HUB_TOKEN")
or os.environ.get("HF_API_KEY")
)
def resolve_src(src: str, revision: str | None) -> Path:
@@ -91,10 +95,11 @@ def apply_mapping(key: str) -> str | None:
return key
def split_monolithic(state: dict[str, torch.Tensor], ) -> dict[str, OrderedDict[str, torch.Tensor]]:
def split_monolithic(
state: dict[str, torch.Tensor],
) -> dict[str, OrderedDict[str, torch.Tensor]]:
components: dict[str, OrderedDict[str, torch.Tensor]] = {
name: OrderedDict()
for name in set(COMPONENT_PREFIXES.values())
name: OrderedDict() for name in set(COMPONENT_PREFIXES.values())
}
intentionally_skipped: list[str] = []
unowned: list[str] = []
@@ -112,8 +117,10 @@ def split_monolithic(state: dict[str, torch.Tensor], ) -> dict[str, OrderedDict[
unowned.append(key)
if unowned:
sample = ", ".join(unowned[:10])
raise ValueError(f"Unowned monolithic keys: {len(unowned)}. "
f"Add COMPONENT_PREFIXES or SKIP_PATTERNS entries. Sample: {sample}")
raise ValueError(
f"Unowned monolithic keys: {len(unowned)}. "
f"Add COMPONENT_PREFIXES or SKIP_PATTERNS entries. Sample: {sample}"
)
if intentionally_skipped:
print(f"Intentionally skipped {len(intentionally_skipped)} keys")
return {name: weights for name, weights in components.items() if weights}
@@ -136,12 +143,8 @@ def build_component_configs(_src_dir: Path) -> dict[str, dict[str, Any]]:
# TODO: emit config content accepted by FastVideo loaders. Most components use
# config.json; schedulers use scheduler_config.json.
return {
"transformer": {
"_class_name": "<FastVideoTransformerClass>"
},
"vae": {
"_class_name": "<FastVideoVAEClass>"
},
"transformer": {"_class_name": "<FastVideoTransformerClass>"},
"vae": {"_class_name": "<FastVideoVAEClass>"},
}
@@ -174,13 +177,19 @@ def build_model_index(
}
if revision:
index["_fastvideo_converted_revision"] = revision
return {key: value for key, value in index.items() if key.startswith("_") or key in available_components}
return {
key: value
for key, value in index.items()
if key.startswith("_") or key in available_components
}
def validate_component_configs(configs: dict[str, dict[str, Any]]) -> None:
# TODO: instantiate each FastVideo config and call update_model_arch(...) or
# update_model_config(...) with this JSON so unknown emitted keys fail here.
placeholder_configs = [name for name, config in configs.items() if "<" in json.dumps(config)]
placeholder_configs = [
name for name, config in configs.items() if "<" in json.dumps(config)
]
if placeholder_configs:
raise ValueError(f"Replace config placeholders for: {placeholder_configs}")
@@ -192,7 +201,9 @@ def verify_conversion(
del dst_dir, components
# TODO: load each emitted stateful component through its production loader and
# assert strict load, or document exact allowed missing/unexpected keys.
raise NotImplementedError("Implement production config validation and strict-load checks")
raise NotImplementedError(
"Implement production config validation and strict-load checks"
)
def write_component(
@@ -205,7 +216,9 @@ def write_component(
if component_dir.exists() and any(component_dir.iterdir()):
shutil.rmtree(component_dir)
component_dir.mkdir(parents=True, exist_ok=True)
save_file(dict(state), str(component_dir / "diffusion_pytorch_model.safetensors"))
save_file(
dict(state), str(component_dir / "diffusion_pytorch_model.safetensors")
)
if config is not None:
config_path = component_dir / config_filename(name)
with config_path.open("w", encoding="utf-8") as f:
@@ -248,7 +261,9 @@ def convert(
if layout in {"monolithic", "raw_official"}:
# TODO: replace model.safetensors with the official monolithic file name.
components = split_monolithic(load_checkpoint(default_monolithic_checkpoint(src_path)))
components = split_monolithic(
load_checkpoint(default_monolithic_checkpoint(src_path))
)
elif layout in {"separate_components", "mixed"}:
if not src_path.is_dir():
raise ValueError(f"{layout} layout requires a source directory: {src_path}")
@@ -256,7 +271,9 @@ def convert(
else:
raise ValueError(f"Unsupported template layout: {layout}")
copied = (copy_passthrough(src_path, dst_dir) if src_path.is_dir() else [])
copied = (
copy_passthrough(src_path, dst_dir) if src_path.is_dir() else []
)
configs = build_component_configs(src_path if src_path.is_dir() else src_path.parent)
validate_component_configs(configs)
for name, state in components.items():
@@ -272,7 +289,9 @@ def convert(
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--src", required=True, help="HF repo id, local dir, or checkpoint path")
parser.add_argument(
"--src", required=True, help="HF repo id, local dir, or checkpoint path"
)
parser.add_argument("--revision", help="HF branch, tag, or commit for repo sources")
parser.add_argument(
"--dst",
@@ -24,8 +24,8 @@ from typing import Any
import torch
FAMILY: str = "<family>" # e.g. "magi_human", "ltx2", "wan"
COMPONENT: str = "<component>" # e.g. "dit", "vae", "encoder"
FAMILY: str = "<family>" # e.g. "magi_human", "ltx2", "wan"
COMPONENT: str = "<component>" # e.g. "dit", "vae", "encoder"
DRILL_LAYER_ENV: str = "<FAMILY>_DEBUG_DRILL_LAYER"
HYPOTHESIS_ENV: str = "<FAMILY>_DEBUG_PATCH_<HYPOTHESIS>"
REL_THRESHOLD: float = 0.005 # 0.5% abs_mean drift flags a block as divergent
@@ -94,7 +94,6 @@ def _attach_block_hooks(
handles: list[Any] = []
def _hook(name: str):
def fn(_module, _inputs, outputs):
t = outputs[0] if isinstance(outputs, tuple) else outputs
if not torch.is_tensor(t):
@@ -102,7 +101,6 @@ def _attach_block_hooks(
log.append({"side": label, **_stat(name, t)})
if tensors is not None:
tensors[name] = t.detach().float().cpu()
return fn
def _pre_hook(name: str):
@@ -116,7 +114,6 @@ def _attach_block_hooks(
log.append({"side": label, **_stat(key, t)})
if tensors is not None:
tensors[key] = t.detach().float().cpu()
return fn
# TODO: adapt attribute paths to your model. Remove adapter block if absent.
@@ -134,21 +131,43 @@ def _attach_block_hooks(
# magi-human uses: attention, mlp.pre_norm, mlp.up_gate_proj,
# mlp.down_proj (pre+post), mlp, attn_post_norm, mlp_post_norm.
if hasattr(layer, "attention"):
handles.append(layer.attention.register_forward_hook(_hook(f"{tag}.attention")))
handles.append(
layer.attention.register_forward_hook(_hook(f"{tag}.attention"))
)
if hasattr(layer, "mlp"):
mlp = layer.mlp
if hasattr(mlp, "pre_norm"):
handles.append(mlp.pre_norm.register_forward_hook(_hook(f"{tag}.mlp.pre_norm")))
handles.append(
mlp.pre_norm.register_forward_hook(_hook(f"{tag}.mlp.pre_norm"))
)
if hasattr(mlp, "up_gate_proj"):
handles.append(mlp.up_gate_proj.register_forward_hook(_hook(f"{tag}.mlp.up_gate_proj")))
handles.append(
mlp.up_gate_proj.register_forward_hook(
_hook(f"{tag}.mlp.up_gate_proj")
)
)
if hasattr(mlp, "down_proj"):
handles.append(mlp.down_proj.register_forward_pre_hook(_pre_hook(f"{tag}.mlp.down_proj")))
handles.append(mlp.down_proj.register_forward_hook(_hook(f"{tag}.mlp.down_proj")))
handles.append(
mlp.down_proj.register_forward_pre_hook(
_pre_hook(f"{tag}.mlp.down_proj")
)
)
handles.append(
mlp.down_proj.register_forward_hook(_hook(f"{tag}.mlp.down_proj"))
)
handles.append(mlp.register_forward_hook(_hook(f"{tag}.mlp")))
if hasattr(layer, "attn_post_norm"):
handles.append(layer.attn_post_norm.register_forward_hook(_hook(f"{tag}.attn_post_norm")))
handles.append(
layer.attn_post_norm.register_forward_hook(
_hook(f"{tag}.attn_post_norm")
)
)
if hasattr(layer, "mlp_post_norm"):
handles.append(layer.mlp_post_norm.register_forward_hook(_hook(f"{tag}.mlp_post_norm")))
handles.append(
layer.mlp_post_norm.register_forward_hook(
_hook(f"{tag}.mlp_post_norm")
)
)
return handles
@@ -174,9 +193,11 @@ def _write_log(entries: list[dict], path: Path) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
with open(path, "w") as f:
for e in entries:
f.write(f"{e['name']} {e['shape']} "
f"{e['abs_mean']:.8f} {e['sum']:.4f} "
f"{e['min']:.6f} {e['max']:.6f}\n")
f.write(
f"{e['name']} {e['shape']} "
f"{e['abs_mean']:.8f} {e['sum']:.4f} "
f"{e['min']:.6f} {e['max']:.6f}\n"
)
def _sort_key(name: str, drill_layer: int) -> tuple:
@@ -184,14 +205,9 @@ def _sort_key(name: str, drill_layer: int) -> tuple:
return (0, "")
if name.startswith(f"L{drill_layer:02d}."):
sub_order = {
"attention": 0,
"attn_post_norm": 1,
"mlp.pre_norm": 2,
"mlp.up_gate_proj": 3,
"mlp.down_proj<in>": 4,
"mlp.down_proj": 5,
"mlp": 6,
"mlp_post_norm": 7,
"attention": 0, "attn_post_norm": 1, "mlp.pre_norm": 2,
"mlp.up_gate_proj": 3, "mlp.down_proj<in>": 4,
"mlp.down_proj": 5, "mlp": 6, "mlp_post_norm": 7,
}.get(name.split(".", 1)[1], 9)
return (1, f"block[{drill_layer:02d}]", sub_order)
if name.startswith("block["):
@@ -200,8 +216,10 @@ def _sort_key(name: str, drill_layer: int) -> tuple:
def _print_table(by_name: dict[str, dict], drill_layer: int) -> int | None:
hdr = (f"{'name':<18} {'up_shape':<22} {'up_absmean':>12} {'fv_absmean':>12} "
f"{'absmean_diff':>14} {'rel%':>8} {'up_sum':>14} {'fv_sum':>14} {'sum_diff':>12}")
hdr = (
f"{'name':<18} {'up_shape':<22} {'up_absmean':>12} {'fv_absmean':>12} "
f"{'absmean_diff':>14} {'rel%':>8} {'up_sum':>14} {'fv_sum':>14} {'sum_diff':>12}"
)
print(f"\n{hdr}\n{'-' * len(hdr)}")
first_div: int | None = None
for name in sorted(by_name.keys(), key=lambda n: _sort_key(n, drill_layer)):
@@ -217,9 +235,11 @@ def _print_table(by_name: dict[str, dict], drill_layer: int) -> int | None:
flag = " <<< DIVERGE"
if first_div is None:
first_div = int(name[len("block["):-1])
print(f"{name:<18} {str(up['shape']):<22} {up['abs_mean']:>12.6f} "
f"{fv['abs_mean']:>12.6f} {am_diff:>14.6f} {am_rel * 100:>7.3f}% "
f"{up['sum']:>14.4f} {fv['sum']:>14.4f} {sum_diff:>12.4f}{flag}")
print(
f"{name:<18} {str(up['shape']):<22} {up['abs_mean']:>12.6f} "
f"{fv['abs_mean']:>12.6f} {am_diff:>14.6f} {am_rel * 100:>7.3f}% "
f"{up['sum']:>14.4f} {fv['sum']:>14.4f} {sum_diff:>12.4f}{flag}"
)
return first_div
@@ -235,8 +255,10 @@ def _print_elementwise(up_t: dict[str, torch.Tensor], fv_t: dict[str, torch.Tens
continue
diff = (a - b).abs()
rel = (diff.mean().item() / max(a.abs().mean().item(), 1e-9)) * 100
print(f"{name:<30} {str(tuple(a.shape)):<22} "
f"{diff.max().item():>12.6f} {diff.mean().item():>12.6f} {rel:>9.4f}%")
print(
f"{name:<30} {str(tuple(a.shape)):<22} "
f"{diff.max().item():>12.6f} {diff.mean().item():>12.6f} {rel:>9.4f}%"
)
def main() -> None:
@@ -43,10 +43,12 @@ def _add_official_to_path() -> Path:
def _log_tensor_stats(label: str, tensor: torch.Tensor) -> None:
value = tensor.detach().float()
print(f"[{_MODEL_FAMILY} PIPELINE] {label}: shape={tuple(tensor.shape)} "
f"dtype={tensor.dtype} device={tensor.device} "
f"min={value.min().item():.6f} max={value.max().item():.6f} "
f"mean={value.mean().item():.6f} std={value.std().item():.6f}")
print(
f"[{_MODEL_FAMILY} PIPELINE] {label}: shape={tuple(tensor.shape)} "
f"dtype={tensor.dtype} device={tensor.device} "
f"min={value.min().item():.6f} max={value.max().item():.6f} "
f"mean={value.mean().item():.6f} std={value.std().item():.6f}"
)
def _extract_tensor(output: Any, key: str) -> torch.Tensor:
@@ -71,8 +73,10 @@ def _run_official_pipeline(
device: torch.device,
) -> Any:
del official_path, params, device
pytest.skip("TODO: import the official pipeline/factory, load official weights, "
"run with params, and return the comparison target.")
pytest.skip(
"TODO: import the official pipeline/factory, load official weights, "
"run with params, and return the comparison target."
)
def _run_fastvideo_pipeline(model_path: Path, params: dict[str, Any]) -> Any:
@@ -142,6 +146,8 @@ def test_todo_model_family_pipeline_official_parity() -> None:
assert official_tensor.shape == fastvideo_tensor.shape
diff = (official_tensor - fastvideo_tensor).abs()
print(f"diff max={diff.max().item():.6f} "
f"mean={diff.mean().item():.6f} median={diff.median().item():.6f}")
print(
f"diff max={diff.max().item():.6f} "
f"mean={diff.mean().item():.6f} median={diff.median().item():.6f}"
)
assert_close(fastvideo_tensor, official_tensor, atol=1e-2, rtol=1e-2)
+2 -15
View File
@@ -1,8 +1,6 @@
env:
IMAGE_VERSION: "py3.12-latest"
BUILDKITE_CLEAN_CHECKOUT: true
# Buildkite only launches Modal; remote jobs initialize their own submodules.
BUILDKITE_GIT_SUBMODULES: false
notify:
- github_commit_status:
@@ -68,17 +66,6 @@ steps:
limit: 2
agents:
queue: "default"
- label: ":vertical_traffic_light: Golden-Gate Tests"
if: build.env("TEST_SCOPE") == "direct" && build.env("TEST_TYPE") == "golden_gate"
command: "timeout 90m .buildkite/scripts/pr_test.sh"
retry:
automatic:
- exit_status: 128
limit: 3
- exit_status: -1
limit: 2
agents:
queue: "default"
- label: ":microscope: Unit Tests"
if: build.env("TEST_SCOPE") == "direct" && build.env("TEST_TYPE") == "unit_test"
command: "timeout 90m .buildkite/scripts/pr_test.sh"
@@ -415,7 +402,7 @@ steps:
- "pyproject.toml"
- "docker/Dockerfile"
config:
command: "timeout 25m .buildkite/scripts/pr_test.sh"
command: "timeout 15m .buildkite/scripts/pr_test.sh"
label: ":test_tube: Training Tests"
env:
- TEST_TYPE=training
@@ -426,7 +413,7 @@ steps:
- "pyproject.toml"
- "docker/Dockerfile"
config:
command: "timeout 25m .buildkite/scripts/pr_test.sh"
command: "timeout 15m .buildkite/scripts/pr_test.sh"
label: ":test_tube: Distillation DMD Tests"
env:
- TEST_TYPE=distillation_dmd
-4
View File
@@ -187,10 +187,6 @@ case "$TEST_TYPE" in
log "Running transformer tests..."
MODAL_COMMAND="$MODAL_ENV HF_API_KEY=$HF_API_KEY python3 -m modal run $MODAL_TEST_FILE::run_transformer_tests"
;;
"golden_gate")
log "Running golden-gate tests..."
MODAL_COMMAND="$MODAL_ENV HF_API_KEY=$HF_API_KEY python3 -m modal run $MODAL_TEST_FILE::run_golden_gate_tests"
;;
"ssim")
log "Running SSIM tests..."
SSIM_BOOTSTRAP_ARGS=$(ssim_bootstrap_args)
-149
View File
@@ -1,149 +0,0 @@
name: macOS MLX Smoke
on:
pull_request:
branches: [main]
paths:
- ".github/workflows/ci-macos-mlx.yml"
- "fastvideo/mlx_runtime/**"
- "fastvideo/tests/mlx/**"
- "fastvideo/tests/platforms/test_mps_vsa_error.py"
- "fastvideo/platforms/mps.py"
- "fastvideo/platforms/__init__.py"
- "fastvideo/__init__.py"
- "examples/inference/basic/mlx_*.py"
- "fastvideo/benchmarks/mlx_*.py"
- "pyproject.toml"
workflow_dispatch:
permissions:
contents: read
concurrency:
group: macos-mlx-${{ github.ref }}
cancel-in-progress: true
jobs:
mlx-smoke:
if: github.event_name == 'workflow_dispatch' || github.event.pull_request.draft != true
runs-on: macos-15
timeout-minutes: 25
env:
FASTVIDEO_ATTENTION_BACKEND: TORCH_SDPA
TOKENIZERS_PARALLELISM: "false"
MASTER_ADDR: localhost
MASTER_PORT: "29513"
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
cache: pip
- uses: astral-sh/setup-uv@v3
- name: Install lightweight MLX smoke dependencies
run: |
uv pip install --system \
--index-url https://download.pytorch.org/whl/cpu \
torch==2.11.0 torchvision torchaudio
uv pip install --system \
pytest numpy scipy pillow imageio einops cloudpickle filelock \
PyYAML diffusers huggingface_hub remote-pdb safetensors loguru mlx \
"ftfy>=6.3.1" "opencv-python>=4.10.0.84" psutil "transformers>=5.0.0"
- name: Show Apple runtime
run: |
python - <<'PY'
import platform
import mlx.core as mx
import torch
print("machine:", platform.machine())
print("processor:", platform.processor())
print("mlx default device:", mx.default_device())
memory_size = mx.metal.device_info().get("memory_size") if mx.metal.is_available() else "metal unavailable"
print("mlx memory_size:", memory_size)
print("torch:", torch.__version__)
print("torch mps available:", torch.backends.mps.is_available())
PY
- name: Run MLX smoke tests
run: |
python -m pytest \
fastvideo/tests/mlx/test_dmd_sampling.py \
fastvideo/tests/mlx/test_memory_limits.py \
fastvideo/tests/mlx/test_quant_capability.py \
fastvideo/tests/mlx/test_mlx_dit_parity.py \
fastvideo/tests/mlx/test_mlx_compile_parity.py \
fastvideo/tests/mlx/test_mlx_checkpoint.py \
fastvideo/tests/mlx/test_mlx_fastwan_benchmark.py \
fastvideo/tests/mlx/test_taehv_decode.py \
fastvideo/tests/mlx/test_frame_upsample.py \
fastvideo/tests/mlx/test_mlx_fast_spatial.py \
fastvideo/tests/mlx/test_mlx_refine.py \
fastvideo/tests/mlx/test_mlx_prompt_to_video_decode.py \
fastvideo/tests/mlx/test_mlx_wan22_prompt_cache_fingerprint.py \
fastvideo/tests/mlx/test_wan22_sample.py \
fastvideo/tests/mlx/test_windowed_attention.py \
fastvideo/tests/mlx/test_mlx_rife_interpolation.py::test_rife_download_unavailable_has_specific_error \
fastvideo/tests/mlx/test_mlx_rife_interpolation.py::test_rife_backend_regression_is_not_skip_eligible \
fastvideo/tests/platforms/test_mps_vsa_error.py \
-q
# Same tests on MLX's CPU backend. Hosted macOS runners are scarce and
# slower to schedule; this Linux job gives fast PR signal on the identical
# graph (the parity tests were designed to be backend-agnostic), while the
# macOS job above stays the source of truth for Metal behavior.
mlx-smoke-linux-cpu:
if: github.event_name == 'workflow_dispatch' || github.event.pull_request.draft != true
runs-on: ubuntu-latest
timeout-minutes: 20
env:
FASTVIDEO_ATTENTION_BACKEND: TORCH_SDPA
TOKENIZERS_PARALLELISM: "false"
MASTER_ADDR: localhost
MASTER_PORT: "29513"
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
cache: pip
- uses: astral-sh/setup-uv@v3
- name: Install lightweight MLX smoke dependencies (CPU backend)
run: |
uv pip install --system \
--index-url https://download.pytorch.org/whl/cpu \
torch==2.11.0 torchvision torchaudio
uv pip install --system \
pytest numpy scipy pillow imageio einops cloudpickle filelock \
PyYAML diffusers huggingface_hub remote-pdb safetensors loguru "mlx[cpu]" \
"ftfy>=6.3.1" "opencv-python>=4.10.0.84" psutil "transformers>=5.0.0"
- name: Run MLX smoke tests (CPU backend)
run: |
python -m pytest \
fastvideo/tests/mlx/test_dmd_sampling.py \
fastvideo/tests/mlx/test_memory_limits.py \
fastvideo/tests/mlx/test_quant_capability.py \
fastvideo/tests/mlx/test_mlx_dit_parity.py \
fastvideo/tests/mlx/test_mlx_compile_parity.py \
fastvideo/tests/mlx/test_mlx_checkpoint.py \
fastvideo/tests/mlx/test_mlx_fastwan_benchmark.py \
fastvideo/tests/mlx/test_taehv_decode.py \
fastvideo/tests/mlx/test_frame_upsample.py \
fastvideo/tests/mlx/test_mlx_fast_spatial.py \
fastvideo/tests/mlx/test_mlx_refine.py \
fastvideo/tests/mlx/test_mlx_prompt_to_video_decode.py \
fastvideo/tests/mlx/test_mlx_wan22_prompt_cache_fingerprint.py \
fastvideo/tests/mlx/test_wan22_sample.py \
fastvideo/tests/mlx/test_windowed_attention.py \
fastvideo/tests/mlx/test_mlx_rife_interpolation.py::test_rife_download_unavailable_has_specific_error \
fastvideo/tests/mlx/test_mlx_rife_interpolation.py::test_rife_backend_regression_is_not_skip_eligible \
fastvideo/tests/platforms/test_mps_vsa_error.py \
-q
+4 -17
View File
@@ -27,25 +27,14 @@ jobs:
ref: ${{ inputs.ref || '' }}
# For PR events, lint the PR head — but keep the hook definitions from
# the base branch so an untrusted PR cannot alter what gets executed.
# The gate scripts are saved too: the self-test step below executes them,
# so it must run the base-branch copies, not the PR head's.
- name: Save trusted hook config and gate scripts
- name: Save trusted hook config
if: github.event_name == 'pull_request_target'
run: |
cp .pre-commit-config.yaml "$RUNNER_TEMP/trusted-pre-commit-config.yaml"
cp -a .github/scripts "$RUNNER_TEMP/trusted-scripts"
echo "GATE_SCRIPTS_DIR=$RUNNER_TEMP/trusted-scripts" >> "$GITHUB_ENV"
# allow-unsafe-pr-checkout acknowledges checkout's pull_request_target
# guard: the head is data for the trusted hooks to lint; nothing from it
# is executed (config and gate scripts are pinned to the base branch
# above) and credentials are not persisted. SHA-pinned to v4.4.0 because
# actionlint's action schema does not know the new input yet.
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
run: cp .pre-commit-config.yaml "$RUNNER_TEMP/trusted-pre-commit-config.yaml"
- uses: actions/checkout@v4
if: github.event_name == 'pull_request_target'
with:
ref: ${{ github.event.pull_request.head.sha }}
persist-credentials: false
allow-unsafe-pr-checkout: true
- name: Restore trusted hook config
if: github.event_name == 'pull_request_target'
run: cp "$RUNNER_TEMP/trusted-pre-commit-config.yaml" .pre-commit-config.yaml
@@ -59,7 +48,5 @@ jobs:
with:
extra_args: --all-files --hook-stage manual
# After pre-commit so a self-test failure cannot mask lint failures.
# GATE_SCRIPTS_DIR points at the base-branch copy on fork PRs (set above);
# push / workflow_call runs use the checked-out tree directly.
- name: Full-suite gate self-test
run: bash "${GATE_SCRIPTS_DIR:-.github/scripts}/test_gate_full_suite.sh"
run: bash .github/scripts/test_gate_full_suite.sh
+2 -2
View File
@@ -129,7 +129,7 @@ jobs:
set -euo pipefail
TEST_NAME=$(echo "$COMMENT" | grep -oP '(?<=/test\s)\S+' | head -1 || true)
VALID="encoder vae transformer kernel unit dreamverse ssim golden-gate training lora-inference lora-training lora-extraction distillation self-forcing vsa vmoba performance api train-framework eval full fastcheck pre-commit"
VALID="encoder vae transformer kernel unit dreamverse ssim training lora-inference lora-training lora-extraction distillation self-forcing vsa vmoba performance api train-framework eval full fastcheck pre-commit"
if [ -z "$TEST_NAME" ] || ! echo "$VALID" | grep -qw "$TEST_NAME"; then
echo "Unknown test: '$TEST_NAME'. Valid: $VALID"
exit 1
@@ -138,7 +138,7 @@ jobs:
declare -A MAP=(
[encoder]=encoder [vae]=vae [transformer]=transformer
[kernel]=kernel_tests [unit]=unit_test [dreamverse]=dreamverse_app
[ssim]=ssim [golden-gate]=golden_gate [training]=training
[ssim]=ssim [training]=training
[lora-inference]=inference_lora [lora-training]=training_lora
[lora-extraction]=lora_extraction
[distillation]=distillation_dmd [self-forcing]=self_forcing
+7 -14
View File
@@ -13,23 +13,16 @@ on:
required: false
default: false
type: boolean
# Auto-rebuild the CUDA images when a repository-controlled image input
# changes on main. This includes the trusted SM89 kernel artifact's source,
# metadata/key helper, ABI dependency metadata, and build orchestration.
# Dreamverse (apps/dreamverse/docker/Dockerfile) and the ROCm Dockerfile stay
# manual-dispatch only.
# Auto-rebuild the CUDA images when their Dockerfile changes on main. The CUDA
# matrix is the only lane that builds from docker/Dockerfile, so a path-scoped
# push trigger is a sufficient change detector on its own -- no separate
# detect-changes/paths-filter job is needed now that there is a single
# in-scope Dockerfile. Dreamverse (apps/dreamverse/docker/Dockerfile) and the
# rocm Dockerfile stay manual-dispatch only.
push:
branches: [main]
paths:
- '.dockerignore'
- '.github/workflows/_template-build-image.yml'
- '.github/workflows/infra-build-image.yml'
- '.gitmodules'
- 'docker/Dockerfile'
- 'docker/uv-excludes'
- 'fastvideo-kernel/**'
- 'fastvideo/tests/modal/kernel_build_cache.py'
- 'pyproject.toml'
permissions:
@@ -57,7 +50,7 @@ jobs:
# 2.8.3 comes from the architecture-specific prebuilt releases.
build-cuda-images:
# Runs on a manual dispatch when build_cuda_matrix is set, or automatically
# on an in-scope main push (inputs are null on push). The
# on a push that changed docker/Dockerfile (inputs are null on push). The
# repository guard keeps fork syncs from auto-building; manual dispatch
# still works in forks.
if: ${{ (github.event_name == 'push' && github.repository == 'hao-ai-lab/FastVideo') || github.event.inputs.build_cuda_matrix == 'true' }}
-2
View File
@@ -6,7 +6,6 @@ on:
paths:
- 'docs/**'
- 'examples/**'
- 'scripts/inference/**'
- 'mkdocs.yml'
- 'requirements-mkdocs.in'
- 'requirements-mkdocs.txt'
@@ -17,7 +16,6 @@ on:
paths:
- 'docs/**'
- 'examples/**'
- 'scripts/inference/**'
- 'mkdocs.yml'
- 'requirements-mkdocs.in'
- 'requirements-mkdocs.txt'
-4
View File
@@ -6,7 +6,6 @@ results/
wandb/
*.ipynb
*.jpg
!examples/datasets/lingbotworld2/image.jpg
*.safetensors
*.mp4
*.png
@@ -23,7 +22,6 @@ Miniconda3-latest-Linux-x86_64.sh
*validation/
data/
outputs/
outputs_audio/
outputs_video
checkpoints/
sbatch.sh
@@ -36,7 +34,6 @@ env
*.log
weights/
logs/
/Z-Image/
official_weights/
converted_weights/
@@ -76,7 +73,6 @@ docs/distillation/examples/
*.pkl
# Reference videos (negations must come after the catch-all on line below)
!fastvideo/tests/nightly/reference_video_*.mp4
# Static images
!docs/assets/images/**/*.png
-1
View File
@@ -22,7 +22,6 @@ repos:
hooks:
- id: yapf
args: [--in-place, --verbose]
language_version: python3.12
additional_dependencies: [toml] # TODO: Remove when yapf is upgraded
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.11.12
+2 -8
View File
@@ -9,8 +9,7 @@
**FastVideo is a unified post-training and real-time inference framework for accelerated video generation.**
## NEWS
- `2026/08/19`: FastVideo now supports MLX on Apple Silicon with [FastMetal-QAD](https://huggingface.co/collections/FastVideo/fastmetal), a family of 1.3B, 5B, and 14B models optimized for Mac—follow the [Apple Silicon guide](https://hao-ai-lab.github.io/FastVideo/getting_started/installation/mps/) and read the [Blog](https://haoailab.com/blogs/fastmetal/).
- `2026/06/23`: Release FastWan-QAD: 5s of Video generated in 1.8s E2E. See the [FastWan-QAD models](https://huggingface.co/FastVideo/FastWan-QAD-FP8-1.3B), [Attn-QAT training guide](https://haoailab.com/FastVideo/training/attn_qat/), and [blog](https://haoailab.com/blogs/fastwan-qad/).
- `2026/06/23`: Release FastWan-QAD: 5s of Video generated in 1.8s E2E. [FastWan-QAD models](https://huggingface.co/FastVideo/FastWan-QAD-FP8-1.3B), check out the [Blog](https://haoailab.com/blogs/fastwan-qad/).
- `2026/03/17`: Release demo: Into the Dreamverse: Vibe Directing in FastVideo, check out the [Blog](https://haoailab.com/blogs/dreamverse/).
- `2026/03/13`: Release demo: Create a 5s 1080p Video in 4.5s with FastVideo on a Single GPU, check out the [Blog](https://haoailab.com/blogs/fastvideo_realtime_1080p/).
- `2025/11/19`: Release [CausalWan2.2 I2V A14B Preview](https://huggingface.co/FastVideo/CausalWan2.2-I2V-A14B-Preview-Diffusers) models, [Blog](https://hao-ai-lab.github.io/blogs/fastvideo_causalwan_preview/) and [Inference Code!](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_self_forcing_causal_wan2_2_i2v.py).
@@ -34,7 +33,7 @@ FastVideo has the following features:
- [Sparse distillation](https://hao-ai-lab.github.io/blogs/fastvideo_post_training/) to achieve >50x denoising speedup
- Scalable training with FSDP2, sequence parallelism, and selective activation checkpointing.
- Causal distillation through Self-Forcing
- See this [page](https://hao-ai-lab.github.io/FastVideo/training/overview/) for the supported training workflows, and the [support matrix](https://hao-ai-lab.github.io/FastVideo/inference/support_matrix/) for supported models.
- See this [page](https://hao-ai-lab.github.io/FastVideo/training/overview/) for full list of supported models and recipes.
- State-of-the-art performance optimizations for inference
- Sequence Parallelism for distributed inference
- Multiple state-of-the-art attention backends
@@ -63,11 +62,6 @@ UV_TORCH_BACKEND=cu126 uv pip install fastvideo
Use `UV_TORCH_BACKEND=cu130` on CUDA 13. Apple silicon users should follow the
[MPS installation guide](https://hao-ai-lab.github.io/FastVideo/getting_started/installation/mps/).
> **On an Apple Silicon Mac?** FastVideo runs FastWan text-to-video natively
> through an MLX runtime — a 5-second 480p clip generated locally, no cloud,
> no discrete GPU. Install with `uv pip install -e '.[mlx]'` and follow the
> [Apple Silicon guide](https://hao-ai-lab.github.io/FastVideo/getting_started/installation/mps/).
Please see our [docs](https://hao-ai-lab.github.io/FastVideo/getting_started/installation/) for more detailed installation instructions.
> **On an NVIDIA DGX Spark (GB10 / ARM64 + CUDA 13)?** There's no prebuilt ARM wheel for the FastVideo CUDA kernel, so it's an editable from-source install (`UV_TORCH_BACKEND=cu130 uv pip install -e .`, which compiles that kernel for you) rather than `UV_TORCH_BACKEND=cu130 uv pip install fastvideo`. A compatible prebuilt ARM64 FlashAttention wheel is available separately. Follow the [DGX Spark install guide](https://hao-ai-lab.github.io/FastVideo/getting_started/installation/spark/).
+5 -5
View File
@@ -67,7 +67,7 @@ MODEL_REGISTRY = {
},
}
DEFAULT_MODEL_ID = "fast-ltx2"
DEFAULT_MODEL_ID = "fast-ltx23"
ACTIVE_MODEL_ID = (os.getenv("DREAMVERSE_MODEL_ID", "").strip() or DEFAULT_MODEL_ID)
if ACTIVE_MODEL_ID not in MODEL_REGISTRY:
@@ -76,14 +76,11 @@ if ACTIVE_MODEL_ID not in MODEL_REGISTRY:
# Active model configuration
MODEL_CONFIG = MODEL_REGISTRY[ACTIVE_MODEL_ID]
# Generation limits
SESSION_TIMEOUT_SECONDS = 300
# Frame settings
NUM_FRAMES = 121
FRAME_HEIGHT = 1088
FRAME_WIDTH = 1920
NUM_INFERENCE_STEPS = 5
NUM_INFERENCE_STEPS = 6
JPEG_QUALITY = 100
BATCH_SIZE = 3
@@ -168,6 +165,9 @@ def _optional_env(*names: str) -> str | None:
return None
# Generation limits
SESSION_TIMEOUT_SECONDS = _env_int("DREAMVERSE_SESSION_TIMEOUT_SECONDS", 300)
DEVTOOLS_ENABLED = _env_bool("FASTVIDEO_ENABLE_DEVTOOLS", False)
PROMPT_SAFETY_ENABLED = _env_bool("FASTVIDEO_ENABLE_PROMPT_SAFETY", False)
DREAMVERSE_MAX_AUTOTUNE = _env_bool("DREAMVERSE_MAX_AUTOTUNE", True)
+5 -1
View File
@@ -1023,7 +1023,11 @@ def get_available_gpus() -> list[int]:
"""Get list of available GPU IDs from environment or auto-detect."""
cuda_visible = os.environ.get("CUDA_VISIBLE_DEVICES", "")
if cuda_visible:
visible_gpu_ids = [int(x.strip()) for x in cuda_visible.split(",") if x.strip()]
try:
visible_gpu_ids = [int(x.strip()) for x in cuda_visible.split(",") if x.strip()]
except ValueError as exc:
raise RuntimeError("CUDA_VISIBLE_DEVICES must be a comma-separated list of integer GPU "
f"indices (got {cuda_visible!r}); GPU UUIDs are not supported.") from exc
return _limit_gpu_ids(visible_gpu_ids)
# Auto-detect available GPUs
+3 -1
View File
@@ -206,7 +206,9 @@ def cli() -> None:
args = parser.parse_args()
_install_heartbeat_log_filter()
uvicorn.run(app, host=args.host, port=args.port)
# A 15MB init image (session_init_image.MAX_SESSION_INIT_IMAGE_BYTES) is ~20MB
# as a base64 ws message, above uvicorn's default 16MiB frame cap.
uvicorn.run(app, host=args.host, port=args.port, ws_max_size=32 * 1024 * 1024)
if __name__ == "__main__":
+47 -21
View File
@@ -30,11 +30,11 @@ from fastapi.middleware.cors import CORSMiddleware
from fastapi.staticfiles import StaticFiles
from dreamverse._deps import require_dreamverse_runtime_deps
from dreamverse.config import FRONTEND_STATIC_DIR_CANDIDATES, GENERATION_SEGMENT_CAP
from dreamverse.config import FRONTEND_STATIC_DIR_CANDIDATES, GENERATION_SEGMENT_CAP, SESSION_TIMEOUT_SECONDS
from dreamverse.session_init_image import cleanup_session_init_image, persist_session_init_image
from dreamverse.utils import _resolve_generation_segment_cap
LATENCY_MS = 200
SESSION_TIMEOUT_SECONDS = 300
MOCK_FRAME_WIDTH = 640
MOCK_FRAME_HEIGHT = 352
MOCK_FPS = 24
@@ -334,6 +334,7 @@ async def websocket_endpoint(websocket: WebSocket):
auto_extension_enabled = bool(init_data.get("auto_extension_enabled", False))
loop_generation_enabled = bool(init_data.get("loop_generation_enabled", False))
single_clip_mode = bool(init_data.get("single_clip_mode", False))
manual_continuation_mode = bool(init_data.get("manual_continuation_mode", False))
generation_paused = False
if init_type == "session_init_v2":
@@ -344,7 +345,8 @@ async def websocket_endpoint(websocket: WebSocket):
incoming_prompts = []
curated_prompts = [prompt.strip() for prompt in incoming_prompts if isinstance(prompt, str) and prompt.strip()]
generation_paused = bool(initial_rollout_prompt and not single_clip_mode and len(curated_prompts) == 0)
generation_paused = bool(not manual_continuation_mode and initial_rollout_prompt and not single_clip_mode
and len(curated_prompts) == 0)
try:
session_init_image = persist_session_init_image(init_data.get("initial_image"))
@@ -373,7 +375,10 @@ async def websocket_endpoint(websocket: WebSocket):
prompt_sources_blocked = False
pending_seed_reset = False
pending_seed_reset_reason = ""
pending_simple_submission: PromptSubmission | None = None
pending_simple_submission: PromptSubmission | None = (PromptSubmission(
prompt_id=str(init_data.get("initial_rollout_prompt_id") or uuid.uuid4()),
raw_prompt=initial_rollout_prompt,
) if manual_continuation_mode and initial_rollout_prompt else None)
single_clip_waiting_for_request = False
rollout_waiting_for_rewrite = False
initial_rollout_waiting_for_rewrite = generation_paused
@@ -393,14 +398,26 @@ async def websocket_endpoint(websocket: WebSocket):
async def send_stream_start(seed_reason: str) -> None:
await ws_send_json({
"type": "ltx2_stream_start",
"total_segments": len(curated_prompts),
"preset_id": preset_id,
"stream_mode": "av_fmp4",
"live_mode": True,
"loop_generation_enabled": loop_generation_enabled,
"loop_iteration": loop_iteration,
"generation_segment_cap": 0,
"type":
"ltx2_stream_start",
"total_segments":
len(curated_prompts),
"preset_id":
preset_id,
"stream_mode":
"av_fmp4",
"live_mode":
True,
"loop_generation_enabled":
loop_generation_enabled,
"loop_iteration":
loop_iteration,
"generation_segment_cap":
_resolve_generation_segment_cap(
single_clip_mode=single_clip_mode,
cap=GENERATION_SEGMENT_CAP,
manual_continuation_mode=manual_continuation_mode,
),
})
if seed_reason == "init":
await ws_send_json({
@@ -493,6 +510,7 @@ async def websocket_endpoint(websocket: WebSocket):
nonlocal auto_extension_enabled
nonlocal loop_generation_enabled
nonlocal single_clip_mode
nonlocal manual_continuation_mode
nonlocal generation_paused
nonlocal seed_prompt_memory
nonlocal curated_prompts
@@ -537,6 +555,7 @@ async def websocket_endpoint(websocket: WebSocket):
auto_extension_enabled = bool(payload.get("auto_extension_enabled", False))
loop_generation_enabled = bool(payload.get("loop_generation_enabled", False))
single_clip_mode = bool(payload.get("single_clip_mode", False))
manual_continuation_mode = bool(payload.get("manual_continuation_mode", False))
seed_prompt_memory = list(next_curated_prompts)
curated_prompts = list(seed_prompt_memory)
@@ -544,10 +563,14 @@ async def websocket_endpoint(websocket: WebSocket):
segment_idx = 0
pending_seed_reset = False
pending_seed_reset_reason = ""
pending_simple_submission = None
pending_simple_submission = (PromptSubmission(
prompt_id=str(payload.get("initial_rollout_prompt_id") or uuid.uuid4()),
raw_prompt=initial_rollout_prompt,
) if manual_continuation_mode and initial_rollout_prompt else None)
single_clip_waiting_for_request = False
rollout_waiting_for_rewrite = False
generation_paused = bool(initial_rollout_prompt and not single_clip_mode and len(curated_prompts) == 0)
generation_paused = bool(not manual_continuation_mode and initial_rollout_prompt and not single_clip_mode
and len(curated_prompts) == 0)
initial_rollout_waiting_for_rewrite = generation_paused
rewrite_restart_pending = False
loop_iteration = 0
@@ -968,10 +991,11 @@ async def websocket_endpoint(websocket: WebSocket):
project_stream_started = True
await send_stream_start(pending_seed_reset_reason)
pending_seed_reset_reason = ""
if pending_simple_submission is not None:
submission = pending_simple_submission
pending_simple_submission = None
await promote_submission_to_ready(submission)
if pending_simple_submission is not None:
submission = pending_simple_submission
pending_simple_submission = None
await promote_submission_to_ready(submission)
if generation_paused:
await asyncio.sleep(0.05)
@@ -981,8 +1005,8 @@ async def websocket_endpoint(websocket: WebSocket):
await asyncio.sleep(0.05)
continue
if (not single_clip_mode and not rollout_waiting_for_rewrite and GENERATION_SEGMENT_CAP > 0
and segment_idx >= GENERATION_SEGMENT_CAP):
if (not single_clip_mode and not manual_continuation_mode and not rollout_waiting_for_rewrite
and GENERATION_SEGMENT_CAP > 0 and segment_idx >= GENERATION_SEGMENT_CAP):
rollout_waiting_for_rewrite = True
loop_generation_enabled = False
project_stream_started = False
@@ -1217,7 +1241,9 @@ def cli() -> None:
print(f"Starting mock server with {LATENCY_MS}ms latency on port {args.port}")
_install_heartbeat_log_filter()
uvicorn.run(app, host="0.0.0.0", port=args.port)
# A 15MB init image (session_init_image.MAX_SESSION_INIT_IMAGE_BYTES) is ~20MB
# as a base64 ws message, above uvicorn's default 16MiB frame cap.
uvicorn.run(app, host="0.0.0.0", port=args.port, ws_max_size=32 * 1024 * 1024)
if __name__ == "__main__":
+60 -18
View File
@@ -313,6 +313,33 @@ def _extract_content_or_empty(response_json: dict[str, Any]) -> str:
return ""
def _find_balanced_object_end(text: str, start: int) -> int:
"""Return the index just past the brace-balanced span opening at
``text[start] == '{'``, honoring JSON string literals and escapes, or -1
if the braces never balance (i.e. the object was truncated)."""
depth = 0
in_string = False
escaped = False
for i in range(start, len(text)):
ch = text[i]
if in_string:
if escaped:
escaped = False
elif ch == "\\":
escaped = True
elif ch == '"':
in_string = False
elif ch == '"':
in_string = True
elif ch == "{":
depth += 1
elif ch == "}":
depth -= 1
if depth == 0:
return i + 1
return -1
def _parse_json_response(content: str) -> dict[str, Any]:
text = content.strip()
if not text:
@@ -330,6 +357,7 @@ def _parse_json_response(content: str) -> dict[str, Any]:
r"```(?:json)?\s*([\s\S]*?)```",
flags=re.IGNORECASE,
)
last_fenced: dict[str, Any] | None = None
for match in fence_pattern.finditer(text):
block = match.group(1).strip()
if not block:
@@ -337,21 +365,37 @@ def _parse_json_response(content: str) -> dict[str, Any]:
try:
parsed = json.loads(block)
if isinstance(parsed, dict):
return parsed
last_fenced = parsed
except json.JSONDecodeError:
continue
if last_fenced is not None:
return last_fenced
# Fall back to scanning for the first decodable JSON object in free-form text.
# Scan for all decodable JSON objects and return the last — chain-of-thought
# models emit draft JSON mid-reasoning; the final answer is always last.
decoder = json.JSONDecoder()
for idx, char in enumerate(text):
if char != "{":
continue
last_parsed: dict[str, Any] | None = None
pos = 0
while (idx := text.find("{", pos)) != -1:
try:
parsed, _ = decoder.raw_decode(text[idx:])
parsed, consumed = decoder.raw_decode(text[idx:])
except json.JSONDecodeError:
# Skip the whole failed object rather than rescanning inside it:
# fragments nested in a malformed or truncated (finish_reason=
# length) object must not override an earlier complete object.
span_end = _find_balanced_object_end(text, idx)
if span_end == -1:
break
pos = span_end
continue
# Skip past the consumed span so nested braces inside a decoded
# object are not re-parsed as standalone objects.
pos = idx + consumed
if isinstance(parsed, dict):
return parsed
last_parsed = parsed
if last_parsed is not None:
return last_parsed
raise ValueError("No JSON object found in assistant response.")
@@ -381,7 +425,7 @@ def _format_locked_segments(locked_segments: list[str]) -> str:
class PromptEnhancer:
def __init__(self):
def __init__(self) -> None:
self.provider = PROMPT_PROVIDER
self.provider_label = _resolve_provider_label(PROMPT_PROVIDER)
self.api_key = PROMPT_API_KEY
@@ -1417,15 +1461,12 @@ class PromptEnhancer:
locked_text = _format_locked_segments(locked_segments_clean)
request_system_prompt = self.enhance_system_prompt
user_payload = {
"request": (
"<locked_segments>\n"
f"{locked_text}\n"
"</locked_segments>\n\n"
f"<conditioning_prompt>{cleaned}</conditioning_prompt>\n\n"
f"Write exactly one new segment ({next_segment_key}) "
"continuing from the locked segments. "
'Respond with valid JSON only as {"next_prompt": "..."}.' # noqa: E501
),
"request": ("<locked_segments>\n"
f"{locked_text}\n"
"</locked_segments>\n\n"
f"<conditioning_prompt>{cleaned}</conditioning_prompt>\n\n"
f"Write exactly one new segment ({next_segment_key}) "
"continuing from the locked segments."),
}
t0 = time.perf_counter()
@@ -1457,6 +1498,7 @@ class PromptEnhancer:
body=request_body,
timeout_seconds=timeout_seconds,
)
_enhance_print("INFO", f"raw_response: {response_content}")
if is_single_clip_mode:
prompt = self._extract_single_clip_prompt(response_content)
else:
@@ -1539,7 +1581,7 @@ class PromptEnhancer:
f"Write exactly one new segment ({next_segment_key}) "
"that continues linearly from the locked segments. "
"Infer the next narrative beat from this history. "
'Respond with valid JSON only as {"next_prompt": "..."}.' # noqa: E501
'Respond with valid JSON only: {"next_prompt": "<your segment description here>"}.' # noqa: E501
),
}
@@ -20,6 +20,7 @@ from __future__ import annotations
# mypy: ignore-errors
import asyncio
import os
import time
import uuid
from typing import TYPE_CHECKING
@@ -52,6 +53,14 @@ if TYPE_CHECKING:
from dreamverse.prompt_enhancer import PromptEnhancer
from dreamverse.prompt_safety import PromptSafetyFilter
# Optional append-only log of every generated segment prompt; unset disables it.
SEGMENT_PROMPT_LOG_PATH = os.environ.get("DREAMVERSE_SEGMENT_PROMPT_LOG", "")
def _append_segment_prompt_log(path: str, text: str) -> None:
with open(path, "a") as f:
f.write(text)
class SessionController:
"""Runs one WebSocket session from accept() through disconnect."""
@@ -198,6 +207,7 @@ class SessionController:
auto_extension_enabled = bool(init_data.get("auto_extension_enabled", False))
loop_generation_enabled = bool(init_data.get("loop_generation_enabled", False))
single_clip_mode = bool(init_data.get("single_clip_mode", False))
manual_continuation_mode = bool(init_data.get("manual_continuation_mode", False))
rewrite_model = self.prompt_enhancer.resolve_rewrite_model(init_data.get("rewrite_model"))
rewrite_system_prompt_override = str(init_data.get("rewrite_window_system_prompt") or "").strip()
rewrite_user_system_prompt_override = str(init_data.get("rewrite_user_system_prompt") or "").strip()
@@ -282,6 +292,9 @@ class SessionController:
# Session queues and mutable state.
raw_prompt_queue: asyncio.Queue[PromptSubmission] = asyncio.Queue()
ready_prompt_queue: asyncio.Queue[ReadyPrompt] = asyncio.Queue()
# Submissions dequeued by prompt_worker_loop but not yet resolved; while
# non-zero the prompt sources are busy, not drained.
prompt_enhancement_inflight = 0
curated_idx = 0
segment_idx = 0
@@ -291,13 +304,20 @@ class SessionController:
generation_cap_blocked = False
auto_extension_blocked_segment_idx: int | None = None
prompt_sources_drained_logged = False
generation_paused = bool(initial_rollout_prompt and not single_clip_mode and len(curated_prompts) == 0)
generation_paused = bool(not manual_continuation_mode and initial_rollout_prompt and not single_clip_mode
and len(curated_prompts) == 0)
pending_seed_reset = False
pending_seed_reset_reason = ""
pending_reset_conditioning = False
loop_iteration = 0 if generation_paused else 1
force_curated_restart_segment = False
pending_simple_prompt_submission: PromptSubmission | None = None
# The frontend records the opening scene under this id; reuse it so
# prompt lifecycle events for the opening prompt reach that record.
pending_simple_prompt_submission: PromptSubmission | None = (PromptSubmission(
prompt_id=str(init_data.get("initial_rollout_prompt_id") or uuid.uuid4()),
raw_prompt=initial_rollout_prompt,
created_at_s=time.time(),
) if manual_continuation_mode and initial_rollout_prompt else None)
single_clip_waiting_for_request = False
rollout_waiting_for_rewrite = False
initial_rollout_waiting_for_rewrite = generation_paused
@@ -306,6 +326,7 @@ class SessionController:
project_active = True
project_stream_started = False
pending_project_end = False
segment_prompt_log_warned = False
def replace_session_init_image(initial_image_payload: object) -> None:
nonlocal session_init_image
@@ -424,6 +445,7 @@ class SessionController:
nonlocal auto_extension_enabled
nonlocal loop_generation_enabled
nonlocal single_clip_mode
nonlocal manual_continuation_mode
nonlocal generation_paused
nonlocal curated_prompts
nonlocal seed_prompt_memory
@@ -458,6 +480,7 @@ class SessionController:
next_auto_extension_enabled = bool(payload.get("auto_extension_enabled", False))
next_loop_generation_enabled = bool(payload.get("loop_generation_enabled", False))
next_single_clip_mode = bool(payload.get("single_clip_mode", False))
next_manual_continuation_mode = bool(payload.get("manual_continuation_mode", False))
next_preset_id = str(payload.get("preset_id") or "").strip()
if next_preset_id:
@@ -510,6 +533,7 @@ class SessionController:
auto_extension_enabled = next_auto_extension_enabled
loop_generation_enabled = next_loop_generation_enabled
single_clip_mode = next_single_clip_mode
manual_continuation_mode = next_manual_continuation_mode
rewrite_model = next_rewrite_model
rewrite_system_prompt_override = (next_rewrite_system_prompt_override)
rewrite_user_system_prompt_override = (next_rewrite_user_system_prompt_override)
@@ -530,10 +554,15 @@ class SessionController:
generation_cap_blocked = False
auto_extension_blocked_segment_idx = None
prompt_sources_drained_logged = False
pending_simple_prompt_submission = None
pending_simple_prompt_submission = (PromptSubmission(
prompt_id=str(payload.get("initial_rollout_prompt_id") or uuid.uuid4()),
raw_prompt=initial_rollout_prompt,
created_at_s=time.time(),
) if manual_continuation_mode and initial_rollout_prompt else None)
single_clip_waiting_for_request = False
rollout_waiting_for_rewrite = False
generation_paused = bool(initial_rollout_prompt and not single_clip_mode and len(curated_prompts) == 0)
generation_paused = bool(not manual_continuation_mode and initial_rollout_prompt
and not single_clip_mode and len(curated_prompts) == 0)
initial_rollout_waiting_for_rewrite = generation_paused
rewrite_restart_pending = False
loop_iteration = 0
@@ -942,6 +971,7 @@ class SessionController:
_resolve_generation_segment_cap(
single_clip_mode=single_clip_mode,
cap=GENERATION_SEGMENT_CAP,
manual_continuation_mode=manual_continuation_mode,
),
})
continue
@@ -1005,11 +1035,9 @@ class SessionController:
continue
async def prompt_worker_loop():
while not stop_event.is_set():
try:
submission = await asyncio.wait_for(raw_prompt_queue.get(), timeout=0.1)
except asyncio.TimeoutError:
continue
nonlocal prompt_enhancement_inflight
async def process_submission(submission: PromptSubmission) -> None:
_main_print("INFO", f"Received user prompt for enhancement: {submission.raw_prompt}")
prompt_id = submission.prompt_id
raw_prompt = submission.raw_prompt
@@ -1032,8 +1060,9 @@ class SessionController:
await ws_send_json({
"type": "error",
"message": blocked_raw_prompt_error,
"prompt_id": prompt_id,
})
continue
return
await log_event(
"enhance_request",
{
@@ -1093,8 +1122,9 @@ class SessionController:
await ws_send_json({
"type": "error",
"message": blocked_final_prompt_error,
"prompt_id": prompt_id,
})
continue
return
if result.fallback_used or not final_prompt:
source = "user_enhancement_failed"
_main_print(
@@ -1113,7 +1143,7 @@ class SessionController:
})
# Enhancement is strict JSON-only; do not enqueue raw
# prompt when enhancement fails.
continue
return
else:
source = "user_enhanced"
await ws_send_json({
@@ -1130,6 +1160,7 @@ class SessionController:
source=source,
fallback_used=result.fallback_used,
loop_iteration=loop_iteration,
raw_prompt=raw_prompt,
))
else:
await ready_prompt_queue.put(
@@ -1139,6 +1170,7 @@ class SessionController:
source="user_raw",
fallback_used=False,
loop_iteration=loop_iteration,
raw_prompt=raw_prompt,
))
await ws_send_json({
"type": "prompt_ready",
@@ -1148,6 +1180,21 @@ class SessionController:
"latency_ms": 0.0,
})
while not stop_event.is_set():
try:
submission = raw_prompt_queue.get_nowait()
except asyncio.QueueEmpty:
await asyncio.sleep(0.1)
continue
# Dequeue and increment without an await in between so the
# generation loop never sees an empty queue with zero in flight
# while this submission is still being enhanced.
prompt_enhancement_inflight += 1
try:
await process_submission(submission)
finally:
prompt_enhancement_inflight -= 1
def queue_snapshot() -> dict[str, object]:
return {
"user_ready": ready_prompt_queue.qsize(),
@@ -1300,6 +1347,7 @@ class SessionController:
_resolve_generation_segment_cap(
single_clip_mode=single_clip_mode,
cap=GENERATION_SEGMENT_CAP,
manual_continuation_mode=manual_continuation_mode,
),
})
await ws_send_json({
@@ -1359,6 +1407,7 @@ class SessionController:
_resolve_generation_segment_cap(
single_clip_mode=single_clip_mode,
cap=GENERATION_SEGMENT_CAP,
manual_continuation_mode=manual_continuation_mode,
),
})
if nonlocal_reason == "loop_restart":
@@ -1388,8 +1437,9 @@ class SessionController:
await raw_prompt_queue.put(pending_simple_prompt_submission)
pending_simple_prompt_submission = None
if (not single_clip_mode and not generation_cap_blocked and not rollout_waiting_for_rewrite
and GENERATION_SEGMENT_CAP > 0 and generated_segment_count >= GENERATION_SEGMENT_CAP):
if (not single_clip_mode and not manual_continuation_mode and not generation_cap_blocked
and not rollout_waiting_for_rewrite and GENERATION_SEGMENT_CAP > 0
and generated_segment_count >= GENERATION_SEGMENT_CAP):
loop_generation_enabled = False
rollout_waiting_for_rewrite = True
_main_print(
@@ -1542,7 +1592,10 @@ class SessionController:
if single_clip_mode:
await asyncio.sleep(PROMPT_AUTO_SLEEP_MS / 1000.0)
continue
if not prompt_sources_drained_logged:
# A raw submission still queued or being enhanced will produce a
# ready prompt shortly; that is not a drained/blocked state.
enhancement_pending = (raw_prompt_queue.qsize() > 0 or prompt_enhancement_inflight > 0)
if not prompt_sources_drained_logged and not enhancement_pending:
snapshot = queue_snapshot()
_main_print(
"WARN",
@@ -1581,6 +1634,24 @@ class SessionController:
total_segments_hint = max(segment_idx, len(curated_prompts))
prompt = selected.prompt
locked_segment_prompts.append(prompt)
if SEGMENT_PROMPT_LOG_PATH:
_ts = time.strftime("%Y-%m-%d %H:%M:%S")
_lines = [
f"\n=== Segment {segment_idx} [{_ts}] source={selected.source} client={client_id[:8]} ===",
]
if selected.raw_prompt and selected.raw_prompt != prompt:
_lines.append(f"User: {selected.raw_prompt}")
_lines.append(f"Rewritten: {prompt}")
try:
await asyncio.to_thread(_append_segment_prompt_log, SEGMENT_PROMPT_LOG_PATH,
"\n".join(_lines) + "\n")
except Exception as exc:
if not segment_prompt_log_warned:
segment_prompt_log_warned = True
_main_print(
"WARN",
f"Failed to write segment prompt log {SEGMENT_PROMPT_LOG_PATH}: {exc}",
)
if (auto_extension_blocked_segment_idx is not None
and auto_extension_blocked_segment_idx <= segment_idx):
auto_extension_blocked_segment_idx = None
@@ -19,3 +19,4 @@ class ReadyPrompt:
fallback_used: bool = False
seed_prompt_index: int | None = None
loop_iteration: int | None = None
raw_prompt: str | None = None
@@ -3,6 +3,7 @@ from __future__ import annotations
import sys
from pathlib import Path
TESTS_DIR = Path(__file__).resolve().parent
DREAMVERSE_PACKAGE_DIR = TESTS_DIR.parent
DREAMVERSE_APP_DIR = DREAMVERSE_PACKAGE_DIR.parent
@@ -5,6 +5,7 @@ from pathlib import Path
import pytest
SERVER_DIR = Path(__file__).resolve().parents[1]
@@ -52,7 +53,9 @@ def test_config_defaults_to_cerebras_with_parallel_groq_fallback_stage(monkeypat
module = _load_config_module()
assert module.PROMPT_PROVIDER == "cerebras"
assert module.PROMPT_PROVIDER_RUNTIME_STAGES == (("cerebras", "groq"), )
assert module.PROMPT_PROVIDER_RUNTIME_STAGES == (
("cerebras", "groq"),
)
assert module.PROMPT_PROVIDER_PRIORITY == (
"cerebras",
"groq",
@@ -83,7 +86,9 @@ def test_config_ignores_legacy_groq_primary_override(monkeypatch):
module = _load_config_module()
assert module.PROMPT_PROVIDER == "cerebras"
assert module.PROMPT_PROVIDER_RUNTIME_STAGES == (("cerebras", "groq"), )
assert module.PROMPT_PROVIDER_RUNTIME_STAGES == (
("cerebras", "groq"),
)
assert module.PROMPT_PROVIDER_PRIORITY == (
"cerebras",
"groq",
@@ -101,17 +106,24 @@ def test_config_uses_local_overlay_paths_when_devtools_enabled(monkeypatch, tmp_
assert module.DEVTOOLS_ENABLED is True
assert module.FRONTEND_ROOT.as_posix().endswith("apps/dreamverse/web")
assert module.PROMPT_ENHANCE_SYSTEM_PROMPT_PATH.endswith("dreamverse/prompts.local/next_segment_system_prompt.md")
assert module.PROMPT_ENHANCE_SYSTEM_PROMPT_PATH.endswith(
"dreamverse/prompts.local/next_segment_system_prompt.md"
)
assert module.PROMPT_ENHANCE_SYSTEM_PROMPT_FALLBACK_PATH.endswith(
"dreamverse/prompts/next_segment_system_prompt.md")
"dreamverse/prompts/next_segment_system_prompt.md"
)
assert module.PROMPT_REWRITE_USER_SYSTEM_PROMPT_PATH.endswith(
"dreamverse/prompts.local/rewrite_user_system_prompt.md")
"dreamverse/prompts.local/rewrite_user_system_prompt.md"
)
assert module.PROMPT_REWRITE_USER_SYSTEM_PROMPT_FALLBACK_PATH.endswith(
"dreamverse/prompts/rewrite_user_system_prompt.md")
"dreamverse/prompts/rewrite_user_system_prompt.md"
)
assert module.CURATED_PRESETS_FILE_PATH.endswith(
"apps/dreamverse/web/prompts.local/selected_ltx2_continuation_story_presets.json")
"apps/dreamverse/web/prompts.local/selected_ltx2_continuation_story_presets.json"
)
assert module.CURATED_PRESETS_FALLBACK_FILE_PATH.endswith(
"apps/dreamverse/web/prompts/selected_ltx2_continuation_story_presets.json")
"apps/dreamverse/web/prompts/selected_ltx2_continuation_story_presets.json"
)
assert module.FRONTEND_STATIC_DIR_CANDIDATES[:2] == (
str(module.FRONTEND_ROOT / "out"),
str(module.FRONTEND_ROOT / "dist"),
@@ -138,12 +150,22 @@ def test_config_enables_prompt_safety_when_requested(monkeypatch):
def test_config_uses_five_minute_session_timeout(monkeypatch):
_set_required_prompt_keys(monkeypatch)
monkeypatch.delenv("DREAMVERSE_SESSION_TIMEOUT_SECONDS", raising=False)
module = _load_config_module()
assert module.SESSION_TIMEOUT_SECONDS == 300
def test_config_session_timeout_env_override(monkeypatch):
_set_required_prompt_keys(monkeypatch)
monkeypatch.setenv("DREAMVERSE_SESSION_TIMEOUT_SECONDS", "1800")
module = _load_config_module()
assert module.SESSION_TIMEOUT_SECONDS == 1800
def test_config_rejects_invalid_prompt_provider(monkeypatch):
monkeypatch.setenv("FASTVIDEO_PROMPT_PROVIDER", "unsupported")
_set_required_prompt_keys(monkeypatch)
@@ -9,7 +9,6 @@ from fastapi.testclient import TestClient
import fastvideo.entrypoints.streaming as streaming_entrypoints
import pytest
def _install_stack03_import_stubs(monkeypatch):
"""Keep entrypoint tests focused while later-stack runtime modules are absent."""
if not hasattr(streaming_entrypoints, "build_health_router"):
@@ -18,7 +17,6 @@ def _install_stack03_import_stubs(monkeypatch):
gpu_pool_stub = types.ModuleType("dreamverse.gpu_pool")
class GPUPool:
def __init__(self, _gpu_ids):
pass
@@ -51,7 +49,6 @@ def _install_stack03_import_stubs(monkeypatch):
controller_stub = types.ModuleType("dreamverse.session.controller")
class SessionController:
def __init__(self, **_kwargs):
pass
@@ -78,12 +75,15 @@ def _run_cli(module, monkeypatch, argv: list[str]) -> list[dict[str, object]]:
calls: list[dict[str, object]] = []
uvicorn_stub = types.ModuleType("uvicorn")
def run(app, host: str, port: int) -> None:
calls.append({
"app": app,
"host": host,
"port": port,
})
def run(app, host: str, port: int, **kwargs) -> None:
calls.append(
{
"app": app,
"host": host,
"port": port,
**kwargs,
}
)
uvicorn_stub.run = run
monkeypatch.setitem(sys.modules, "uvicorn", uvicorn_stub)
@@ -100,11 +100,14 @@ def test_server_cli_defaults_to_local_web_port(monkeypatch):
server_main = _import_server_main(monkeypatch)
calls = _run_cli(server_main, monkeypatch, ["dreamverse-server"])
assert calls == [{
"app": server_main.app,
"host": "0.0.0.0",
"port": 8009,
}]
assert calls == [
{
"app": server_main.app,
"host": "0.0.0.0",
"port": 8009,
"ws_max_size": 32 * 1024 * 1024,
}
]
def test_server_cli_allows_explicit_host_and_port(monkeypatch):
@@ -115,11 +118,14 @@ def test_server_cli_allows_explicit_host_and_port(monkeypatch):
["dreamverse-server", "--host", "127.0.0.1", "--port", "8123"],
)
assert calls == [{
"app": server_main.app,
"host": "127.0.0.1",
"port": 8123,
}]
assert calls == [
{
"app": server_main.app,
"host": "127.0.0.1",
"port": 8123,
"ws_max_size": 32 * 1024 * 1024,
}
]
def test_server_does_not_expose_backend_source_as_static_assets(monkeypatch):
@@ -139,11 +145,14 @@ def test_mock_server_cli_defaults_to_local_web_port(monkeypatch):
["dreamverse-mock-server"],
)
assert calls == [{
"app": mock_server.app,
"host": "0.0.0.0",
"port": 8009,
}]
assert calls == [
{
"app": mock_server.app,
"host": "0.0.0.0",
"port": 8009,
"ws_max_size": 32 * 1024 * 1024,
}
]
def test_mock_server_cli_updates_latency(monkeypatch):
@@ -156,11 +165,14 @@ def test_mock_server_cli_updates_latency(monkeypatch):
["dreamverse-mock-server", "--latency", "321", "--port", "8111"],
)
assert calls == [{
"app": mock_server.app,
"host": "0.0.0.0",
"port": 8111,
}]
assert calls == [
{
"app": mock_server.app,
"host": "0.0.0.0",
"port": 8111,
"ws_max_size": 32 * 1024 * 1024,
}
]
assert mock_server.LATENCY_MS == 321
finally:
mock_server.LATENCY_MS = old_latency_ms
@@ -7,6 +7,7 @@ from types import SimpleNamespace
import pytest
import dreamverse.gpu_pool as gpu_pool
@@ -84,7 +85,9 @@ def test_send_command_raises_on_worker_death():
cmd_q = ctx.Queue()
resp_q = ctx.Queue()
proc = ctx.Process(target=_child_consume_and_exit, args=(cmd_q, resp_q))
proc = ctx.Process(
target=_child_consume_and_exit, args=(cmd_q, resp_q)
)
proc.start()
# Wait for the spawn child to fully boot. Allow generous time —
@@ -7,7 +7,7 @@ ALLOWED_PREFIXES = (
"fastvideo.entrypoints.video_generator",
"fastvideo.configs",
)
ALLOWED_EXACT = ("fastvideo", )
ALLOWED_EXACT = ("fastvideo",)
FORBIDDEN_PREFIXES = (
"fastvideo.pipelines",
"fastvideo.models",
@@ -38,13 +38,19 @@ def test_dreamverse_server_imports_only_public_fastvideo_surfaces() -> None:
except SyntaxError as task_exc:
raise AssertionError(f"Failed to parse {path}") from task_exc
for node in ast.walk(tree):
names = ([a.name for a in node.names] if isinstance(node, ast.Import) else
[node.module] if isinstance(node, ast.ImportFrom) and node.module else [])
names = (
[a.name for a in node.names] if isinstance(node, ast.Import)
else [node.module] if isinstance(node, ast.ImportFrom) and node.module
else []
)
for name in names:
if not name:
continue
rel_path = str(path.relative_to(root))
if (name.startswith(FORBIDDEN_PREFIXES) and (rel_path, name) not in ALLOWED_INTERNAL_IMPORTS):
if (
name.startswith(FORBIDDEN_PREFIXES)
and (rel_path, name) not in ALLOWED_INTERNAL_IMPORTS
):
bad.append((str(path.relative_to(root)), getattr(node, "lineno", 0), name))
assert bad == [], f"Forbidden internal imports: {bad}"
@@ -6,6 +6,7 @@ import os
from fastapi import WebSocketDisconnect
os.environ.setdefault("CEREBRAS_API_KEY", "dummy")
os.environ.setdefault("GROQ_API_KEY", "dummy")
@@ -13,7 +14,6 @@ import dreamverse.mock_server as mock_server
class _FakeWebSocket:
def __init__(self, messages: list[tuple[float, dict[str, object]]]):
self._messages = messages
self._index = 0
@@ -49,34 +49,34 @@ def test_mock_server_matches_current_single5s_protocol():
mock_server.MOCK_SEGMENT_BYTES = b"mock-fmp4-bytes"
mock_server.LATENCY_MS = 1
ws = _FakeWebSocket([
(
0.0,
{
"type": "session_init_v2",
"preset_id": "simple_prompt_1",
"curated_prompts": ["selected prompt"],
"single_clip_mode": True,
"enhancement_enabled": False,
"auto_extension_enabled": False,
"loop_generation_enabled": False,
},
),
(
0.01,
{
"type": "simple_generate",
"preset_id": "simple_custom_prompt",
"prompt_id": "simple_custom_prompt",
"prompt": "custom prompt",
"enhancement_enabled": True,
"initial_image": None,
},
),
(0.20, {
"type": "leave"
}),
])
ws = _FakeWebSocket(
[
(
0.0,
{
"type": "session_init_v2",
"preset_id": "simple_prompt_1",
"curated_prompts": ["selected prompt"],
"single_clip_mode": True,
"enhancement_enabled": False,
"auto_extension_enabled": False,
"loop_generation_enabled": False,
},
),
(
0.01,
{
"type": "simple_generate",
"preset_id": "simple_custom_prompt",
"prompt_id": "simple_custom_prompt",
"prompt": "custom prompt",
"enhancement_enabled": True,
"initial_image": None,
},
),
(0.20, {"type": "leave"}),
]
)
asyncio.run(mock_server.websocket_endpoint(ws))
@@ -92,14 +92,24 @@ def test_mock_server_matches_current_single5s_protocol():
assert message_types.count("ltx2_stream_complete") == 2
assert "prompt_sources_blocked" not in message_types
segment_start_events = [payload for payload in ws.sent_json if payload["type"] == "ltx2_segment_start"]
gpu_assigned_event = next(payload for payload in ws.sent_json if payload["type"] == "gpu_assigned")
segment_start_events = [
payload
for payload in ws.sent_json
if payload["type"] == "ltx2_segment_start"
]
gpu_assigned_event = next(
payload for payload in ws.sent_json if payload["type"] == "gpu_assigned"
)
assert gpu_assigned_event["session_timeout"] == mock_server.SESSION_TIMEOUT_SECONDS
assert [payload["segment_idx"] for payload in segment_start_events] == [1, 1]
assert segment_start_events[0]["prompt"] == "selected prompt"
assert segment_start_events[1]["prompt"] == "custom prompt"
step_complete_events = [payload for payload in ws.sent_json if payload["type"] == "step_complete"]
step_complete_events = [
payload
for payload in ws.sent_json
if payload["type"] == "step_complete"
]
assert len(step_complete_events) == 2
assert step_complete_events[0]["latency_ms"] == {
"total": 121.0,
@@ -124,29 +134,29 @@ def test_mock_server_regular_cap_waits_for_rewrite_rollout():
mock_server.LATENCY_MS = 1
mock_server.GENERATION_SEGMENT_CAP = 1
ws = _FakeWebSocket([
(
0.0,
{
"type": "session_init_v2",
"preset_id": "test_preset",
"curated_prompts": ["segment one"],
"enhancement_enabled": True,
"auto_extension_enabled": False,
"loop_generation_enabled": False,
},
),
(
0.02,
{
"type": "rewrite_seed_prompts",
"rewrite_instruction": "start a new rollout",
},
),
(0.20, {
"type": "leave"
}),
])
ws = _FakeWebSocket(
[
(
0.0,
{
"type": "session_init_v2",
"preset_id": "test_preset",
"curated_prompts": ["segment one"],
"enhancement_enabled": True,
"auto_extension_enabled": False,
"loop_generation_enabled": False,
},
),
(
0.02,
{
"type": "rewrite_seed_prompts",
"rewrite_instruction": "start a new rollout",
},
),
(0.20, {"type": "leave"}),
]
)
asyncio.run(mock_server.websocket_endpoint(ws))
@@ -156,7 +166,11 @@ def test_mock_server_regular_cap_waits_for_rewrite_rollout():
assert "generation_cap_reached" not in message_types
assert "prompt_sources_blocked" not in message_types
segment_start_events = [payload for payload in ws.sent_json if payload["type"] == "ltx2_segment_start"]
segment_start_events = [
payload
for payload in ws.sent_json
if payload["type"] == "ltx2_segment_start"
]
assert [payload["segment_idx"] for payload in segment_start_events] == [1, 1]
assert segment_start_events[0]["prompt"] == "segment one"
assert segment_start_events[1]["prompt"] == "segment one [start a new rollout]"
@@ -173,40 +187,54 @@ def test_mock_server_rewrite_during_active_segment_restarts_from_first_rewritten
mock_server.MOCK_SEGMENT_BYTES = b"mock-fmp4-bytes"
mock_server.LATENCY_MS = 100
ws = _FakeWebSocket([
(
0.0,
{
"type": "session_init_v2",
"preset_id": "test_preset",
"curated_prompts": ["segment one", "segment two"],
"enhancement_enabled": True,
"auto_extension_enabled": False,
"loop_generation_enabled": False,
},
),
(
0.02,
{
"type": "rewrite_seed_prompts",
"rewrite_instruction": "restart from rewrite",
},
),
(0.40, {
"type": "leave"
}),
])
ws = _FakeWebSocket(
[
(
0.0,
{
"type": "session_init_v2",
"preset_id": "test_preset",
"curated_prompts": ["segment one", "segment two"],
"enhancement_enabled": True,
"auto_extension_enabled": False,
"loop_generation_enabled": False,
},
),
(
0.02,
{
"type": "rewrite_seed_prompts",
"rewrite_instruction": "restart from rewrite",
},
),
(0.40, {"type": "leave"}),
]
)
asyncio.run(mock_server.websocket_endpoint(ws))
segment_start_events = [payload for payload in ws.sent_json if payload["type"] == "ltx2_segment_start"]
segment_start_events = [
payload
for payload in ws.sent_json
if payload["type"] == "ltx2_segment_start"
]
assert [payload["prompt"] for payload in segment_start_events[:2]] == [
"segment one",
"segment one [restart from rewrite]",
]
assert all(payload["prompt"] != "segment two" for payload in segment_start_events[1:])
reset_events = [payload for payload in ws.sent_json if payload.get("type") == "seed_prompts_reset_applied"]
assert any(payload.get("reason") == "rewrite_during_generation" for payload in reset_events)
assert all(
payload["prompt"] != "segment two"
for payload in segment_start_events[1:]
)
reset_events = [
payload
for payload in ws.sent_json
if payload.get("type") == "seed_prompts_reset_applied"
]
assert any(
payload.get("reason") == "rewrite_during_generation"
for payload in reset_events
)
finally:
mock_server.MOCK_SEGMENT_BYTES = old_segment_bytes
mock_server.LATENCY_MS = old_latency_ms
@@ -219,24 +247,24 @@ def test_mock_server_supports_initial_custom_rollout_prompt():
mock_server.MOCK_SEGMENT_BYTES = b"mock-fmp4-bytes"
mock_server.LATENCY_MS = 1
ws = _FakeWebSocket([
(
0.0,
{
"type": "session_init_v2",
"preset_id": "custom_editable",
"preset_label": "Custom rollout",
"curated_prompts": [],
"initial_rollout_prompt": "A moonbase corridor thriller with flooding",
"enhancement_enabled": True,
"auto_extension_enabled": False,
"loop_generation_enabled": False,
},
),
(0.20, {
"type": "leave"
}),
])
ws = _FakeWebSocket(
[
(
0.0,
{
"type": "session_init_v2",
"preset_id": "custom_editable",
"preset_label": "Custom rollout",
"curated_prompts": [],
"initial_rollout_prompt": "A moonbase corridor thriller with flooding",
"enhancement_enabled": True,
"auto_extension_enabled": False,
"loop_generation_enabled": False,
},
),
(0.20, {"type": "leave"}),
]
)
asyncio.run(mock_server.websocket_endpoint(ws))
@@ -248,9 +276,161 @@ def test_mock_server_supports_initial_custom_rollout_prompt():
assert "ltx2_stream_start" in message_types
assert "prompt_sources_blocked" not in message_types
segment_start_events = [payload for payload in ws.sent_json if payload["type"] == "ltx2_segment_start"]
segment_start_events = [
payload
for payload in ws.sent_json
if payload["type"] == "ltx2_segment_start"
]
assert segment_start_events
assert segment_start_events[0]["prompt"] == ("A moonbase corridor thriller with flooding [segment 1]")
assert segment_start_events[0]["prompt"] == (
"A moonbase corridor thriller with flooding [segment 1]"
)
finally:
mock_server.MOCK_SEGMENT_BYTES = old_segment_bytes
mock_server.LATENCY_MS = old_latency_ms
def test_mock_server_manual_mode_streams_initial_prompt_without_rewrite_or_cap():
old_segment_bytes = mock_server.MOCK_SEGMENT_BYTES
old_latency_ms = mock_server.LATENCY_MS
old_generation_segment_cap = mock_server.GENERATION_SEGMENT_CAP
try:
mock_server.MOCK_SEGMENT_BYTES = b"mock-fmp4-bytes"
mock_server.LATENCY_MS = 1
mock_server.GENERATION_SEGMENT_CAP = 1
ws = _FakeWebSocket(
[
(
0.0,
{
"type": "session_init_v2",
"preset_id": "custom_editable",
"preset_label": "Custom rollout",
"curated_prompts": [],
"initial_rollout_prompt": "A drone skims a neon canyon",
"initial_rollout_prompt_id": "steer-prompt-1",
"manual_continuation_mode": True,
"enhancement_enabled": True,
"auto_extension_enabled": False,
"loop_generation_enabled": False,
},
),
(
0.08,
{
"type": "append_prompt",
"prompt": "The drone dives toward the river",
"prompt_id": "steer-prompt-2",
},
),
(0.30, {"type": "leave"}),
]
)
asyncio.run(mock_server.websocket_endpoint(ws))
message_types = [payload["type"] for payload in ws.sent_json]
assert "rewrite_seed_prompts_started" not in message_types
assert "rewrite_seed_prompts_complete" not in message_types
assert "ltx2_stream_start" in message_types
# cap=1 must not stop a manual-mode session after the first segment
assert "ltx2_stream_complete" not in message_types
prompt_ready_events = [
payload for payload in ws.sent_json if payload["type"] == "prompt_ready"
]
assert [payload["prompt_id"] for payload in prompt_ready_events] == [
"steer-prompt-1",
"steer-prompt-2",
]
segment_start_events = [
payload
for payload in ws.sent_json
if payload["type"] == "ltx2_segment_start"
]
assert [payload["prompt"] for payload in segment_start_events] == [
"A drone skims a neon canyon",
"The drone dives toward the river",
]
segment_source_events = [
payload
for payload in ws.sent_json
if payload["type"] == "segment_prompt_source"
]
assert [payload["prompt_id"] for payload in segment_source_events] == [
"steer-prompt-1",
"steer-prompt-2",
]
finally:
mock_server.MOCK_SEGMENT_BYTES = old_segment_bytes
mock_server.LATENCY_MS = old_latency_ms
mock_server.GENERATION_SEGMENT_CAP = old_generation_segment_cap
def test_mock_server_project_init_manual_mode_streams_initial_prompt():
old_segment_bytes = mock_server.MOCK_SEGMENT_BYTES
old_latency_ms = mock_server.LATENCY_MS
try:
mock_server.MOCK_SEGMENT_BYTES = b"mock-fmp4-bytes"
mock_server.LATENCY_MS = 1
ws = _FakeWebSocket(
[
(
0.0,
{
"type": "session_init_v2",
"preset_id": "test_preset",
"curated_prompts": ["segment one"],
"enhancement_enabled": True,
"auto_extension_enabled": False,
"loop_generation_enabled": False,
},
),
(0.05, {"type": "end_project_keep_session"}),
(
0.15,
{
"type": "project_init_v1",
"preset_id": "custom_editable",
"preset_label": "Custom rollout",
"curated_prompts": [],
"initial_rollout_prompt": "A drone skims a neon canyon",
"initial_rollout_prompt_id": "steer-prompt-1",
"manual_continuation_mode": True,
"enhancement_enabled": True,
"auto_extension_enabled": False,
"loop_generation_enabled": False,
},
),
(0.45, {"type": "leave"}),
]
)
asyncio.run(mock_server.websocket_endpoint(ws))
message_types = [payload["type"] for payload in ws.sent_json]
assert "project_idle" in message_types
project_idle_index = message_types.index("project_idle")
# manual-mode restart must not run the rewrite rollout
assert "rewrite_seed_prompts_started" not in message_types[project_idle_index:]
assert "ltx2_stream_start" in message_types[project_idle_index:]
prompt_ready_events = [
payload for payload in ws.sent_json if payload["type"] == "prompt_ready"
]
assert [payload["prompt_id"] for payload in prompt_ready_events] == [
"steer-prompt-1",
]
segment_start_events = [
payload
for payload in ws.sent_json
if payload["type"] == "ltx2_segment_start"
]
assert segment_start_events[-1]["prompt"] == "A drone skims a neon canyon"
finally:
mock_server.MOCK_SEGMENT_BYTES = old_segment_bytes
mock_server.LATENCY_MS = old_latency_ms
@@ -263,37 +443,35 @@ def test_mock_server_can_start_new_project_without_reconnecting():
mock_server.MOCK_SEGMENT_BYTES = b"mock-fmp4-bytes"
mock_server.LATENCY_MS = 40
ws = _FakeWebSocket([
(
0.0,
{
"type": "session_init_v2",
"preset_id": "test_preset",
"curated_prompts": ["segment one"],
"enhancement_enabled": True,
"auto_extension_enabled": False,
"loop_generation_enabled": False,
},
),
(0.02, {
"type": "end_project_keep_session"
}),
(
0.20,
{
"type": "project_init_v1",
"preset_id": "test_preset_2",
"preset_label": "Test Preset 2",
"curated_prompts": ["segment two"],
"enhancement_enabled": True,
"auto_extension_enabled": False,
"loop_generation_enabled": False,
},
),
(0.40, {
"type": "leave"
}),
])
ws = _FakeWebSocket(
[
(
0.0,
{
"type": "session_init_v2",
"preset_id": "test_preset",
"curated_prompts": ["segment one"],
"enhancement_enabled": True,
"auto_extension_enabled": False,
"loop_generation_enabled": False,
},
),
(0.02, {"type": "end_project_keep_session"}),
(
0.20,
{
"type": "project_init_v1",
"preset_id": "test_preset_2",
"preset_label": "Test Preset 2",
"curated_prompts": ["segment two"],
"enhancement_enabled": True,
"auto_extension_enabled": False,
"loop_generation_enabled": False,
},
),
(0.40, {"type": "leave"}),
]
)
asyncio.run(mock_server.websocket_endpoint(ws))
@@ -304,11 +482,16 @@ def test_mock_server_can_start_new_project_without_reconnecting():
project_idle_index = message_types.index("project_idle")
stream_start_indexes = [
index for index, message_type in enumerate(message_types) if message_type == "ltx2_stream_start"
index for index, message_type in enumerate(message_types)
if message_type == "ltx2_stream_start"
]
assert stream_start_indexes[0] < project_idle_index < stream_start_indexes[1]
segment_start_events = [payload for payload in ws.sent_json if payload["type"] == "ltx2_segment_start"]
segment_start_events = [
payload
for payload in ws.sent_json
if payload["type"] == "ltx2_segment_start"
]
assert [payload["prompt"] for payload in segment_start_events[:2]] == [
"segment one",
"segment two",
@@ -6,6 +6,8 @@ import os
import re
import time
import pytest
os.environ.setdefault("CEREBRAS_API_KEY", "dummy")
os.environ.setdefault("GROQ_API_KEY", "dummy")
@@ -21,7 +23,6 @@ from dreamverse.prompt_enhancer import (
class _FakeResponse:
def __init__(self, payload: dict):
self._payload = payload
@@ -30,7 +31,6 @@ class _FakeResponse:
class _FakeSyncCompletions:
def __init__(self, payload: dict):
self._payload = payload
@@ -39,7 +39,6 @@ class _FakeSyncCompletions:
class _FakeSyncClient:
def __init__(self, payload: dict):
self.chat = type(
"_FakeChat",
@@ -49,7 +48,6 @@ class _FakeSyncClient:
class _DelayedSyncCompletions:
def __init__(self, payload: dict, delay_s: float = 0.0, exc: Exception | None = None):
self._payload = payload
self._delay_s = delay_s
@@ -64,26 +62,29 @@ class _DelayedSyncCompletions:
class _DelayedSyncClient:
def __init__(self, payload: dict, delay_s: float = 0.0, exc: Exception | None = None):
self.chat = type(
"_FakeChat",
(),
{"completions": _DelayedSyncCompletions(
payload,
delay_s=delay_s,
exc=exc,
)},
{
"completions": _DelayedSyncCompletions(
payload,
delay_s=delay_s,
exc=exc,
)
},
)()
def _chat_payload_with_content(content: str) -> dict:
return {
"choices": [{
"message": {
"content": content,
"choices": [
{
"message": {
"content": content,
}
}
}]
]
}
@@ -172,7 +173,6 @@ def _build_staged_enhancer(
class _FakeOpenAIClient:
def __init__(self, **kwargs):
self.kwargs = kwargs
self.chat = type(
@@ -183,7 +183,6 @@ class _FakeOpenAIClient:
class _FakeCerebrasClient:
def __init__(self, **kwargs):
self.kwargs = kwargs
self.chat = type(
@@ -194,15 +193,72 @@ class _FakeCerebrasClient:
def test_parse_json_response_accepts_fenced_json_with_prose():
parsed = _parse_json_response("Here is the rewrite:\n```json\n{\"segment_prompts\":[\"A\",\"B\"]}\n```\nThanks.")
parsed = _parse_json_response(
"Here is the rewrite:\n```json\n{\"segment_prompts\":[\"A\",\"B\"]}\n```\nThanks."
)
assert parsed == {"segment_prompts": ["A", "B"]}
def test_parse_json_response_extracts_first_embedded_object():
parsed = _parse_json_response("Model output:\n{\"segment_prompts\":[\"A\",\"B\"]}\n(complete)")
parsed = _parse_json_response(
"Model output:\n{\"segment_prompts\":[\"A\",\"B\"]}\n(complete)"
)
assert parsed == {"segment_prompts": ["A", "B"]}
def test_parse_json_response_returns_outer_object_not_nested_value():
parsed = _parse_json_response(
"Final answer: {\"next_prompt\": \"a scene\", \"style\": {\"mood\": \"noir\"}} done."
)
assert parsed == {"next_prompt": "a scene", "style": {"mood": "noir"}}
def test_parse_json_response_returns_last_of_multiple_objects():
parsed = _parse_json_response(
"Draft: {\"next_prompt\": \"draft\"}\nRefined: {\"next_prompt\": \"final\"}"
)
assert parsed == {"next_prompt": "final"}
def test_parse_json_response_ignores_fragments_of_truncated_trailing_object():
# finish_reason=length cut the refined object short; the complete draft must
# win over a nested fragment of the truncated object.
parsed = _parse_json_response(
'{"next_prompt": "draft"} refined: {"next_prompt": "final", "style": {"mood": "noir"}'
)
assert parsed == {"next_prompt": "draft"}
def test_parse_json_response_ignores_fragments_of_mid_string_truncated_object():
# Unterminated-string truncation reports the error at the opening quote,
# not end-of-text; nested fragments still must not win over the draft.
parsed = _parse_json_response(
'{"next_prompt": "draft"} refined: {"style": {"mood": "noir"}, "next_prompt": "cut off'
)
assert parsed == {"next_prompt": "draft"}
def test_parse_json_response_ignores_fragments_of_malformed_object_with_trailing_prose():
parsed = _parse_json_response(
'{"next_prompt": "draft"} {"final": {"mood": "noir"}, "x": 1 and then some prose'
)
assert parsed == {"next_prompt": "draft"}
def test_parse_json_response_raises_when_only_object_is_truncated():
with pytest.raises(ValueError):
_parse_json_response('{"style": {"mood": "noir"}, "next_prompt": "cut off')
def test_parse_json_response_returns_outer_rollout_dict():
parsed = _parse_json_response(
"{\"rollout\": {\"segment_prompts\": [{\"prompt\": \"a\"}, {\"prompt\": \"b\"}]}}"
)
assert parsed == {
"rollout": {"segment_prompts": [{"prompt": "a"}, {"prompt": "b"}]}
}
def test_load_prompt_required_falls_back_to_default_path(tmp_path):
fallback_path = tmp_path / "next_segment_system_prompt.md"
fallback_path.write_text("fallback prompt\n", encoding="utf-8")
@@ -266,12 +322,16 @@ def test_build_client_supports_groq_provider(monkeypatch):
def test_rewrite_prompt_sequence_accepts_segment_prompts_output():
enhancer = _build_test_enhancer(
_chat_payload_with_content('{"id":"preset_a","label":"Preset A","segment_prompts":["A","B"]}'))
_chat_payload_with_content(
'{"id":"preset_a","label":"Preset A","segment_prompts":["A","B"]}'
)
)
result = asyncio.run(
enhancer.rewrite_prompt_sequence(
["prompt one", "prompt two"],
rewrite_instruction="make it cinematic",
))
)
)
assert result.fallback_used is False
assert result.error is None
assert result.rollout_id == "preset_a"
@@ -280,12 +340,15 @@ def test_rewrite_prompt_sequence_accepts_segment_prompts_output():
def test_rewrite_prompt_sequence_accepts_legacy_rewritten_prompts_output():
enhancer = _build_test_enhancer(_chat_payload_with_content('{"rewritten_prompts":["A","B"]}'))
enhancer = _build_test_enhancer(
_chat_payload_with_content('{"rewritten_prompts":["A","B"]}')
)
result = asyncio.run(
enhancer.rewrite_prompt_sequence(
["prompt one", "prompt two"],
rewrite_instruction="make it cinematic",
))
)
)
assert result.fallback_used is False
assert result.error is None
assert result.rollout_id == "current_rollout"
@@ -294,14 +357,19 @@ def test_rewrite_prompt_sequence_accepts_legacy_rewritten_prompts_output():
def test_rewrite_prompt_sequence_accepts_segment_dicts_without_top_level_rollout_metadata():
enhancer = _build_test_enhancer(_chat_payload_with_content('{"segments":[{"prompt":"A"},{"text":"B"}]}'))
enhancer = _build_test_enhancer(
_chat_payload_with_content(
'{"segments":[{"prompt":"A"},{"text":"B"}]}'
)
)
result = asyncio.run(
enhancer.rewrite_prompt_sequence(
["prompt one", "prompt two"],
preset_id="preset_a",
preset_label="Preset A",
rewrite_instruction="make it cinematic",
))
)
)
assert result.fallback_used is False
assert result.error is None
assert result.rollout_id == "preset_a"
@@ -315,12 +383,14 @@ def test_rewrite_prompt_sequence_accepts_numbered_prose_output():
"The user is asking for a cinematic rewrite.\n\n"
"1. A dog bounds across the moon's dusty surface, kicking up silver regolith as it chases a rabbit beneath the black sky.\n"
"2. The rabbit darts around a crater rim while the dog lunges after it, Earth glowing blue in the distance.\n"
))
)
)
result = asyncio.run(
enhancer.rewrite_prompt_sequence(
["prompt one", "prompt two"],
rewrite_instruction="make it cinematic",
))
)
)
assert result.fallback_used is False
assert result.error is None
assert result.rollout_id == "current_rollout"
@@ -339,10 +409,12 @@ def test_enhance_prompt_prefers_cerebras_before_groq_fallback():
groq_delay_s=0.01,
)
result = asyncio.run(enhancer.enhance_prompt(
"A rainy alley at night",
mode="single_clip",
))
result = asyncio.run(
enhancer.enhance_prompt(
"A rainy alley at night",
mode="single_clip",
)
)
assert result.fallback_used is False
assert result.error is None
@@ -363,10 +435,12 @@ def test_enhance_prompt_uses_groq_when_cerebras_fails():
groq_delay_s=0.01,
)
result = asyncio.run(enhancer.enhance_prompt(
"A rainy alley at night",
mode="single_clip",
))
result = asyncio.run(
enhancer.enhance_prompt(
"A rainy alley at night",
mode="single_clip",
)
)
assert result.fallback_used is False
assert result.error is None
@@ -388,11 +462,13 @@ def test_enhance_prompt_can_use_groq_when_cerebras_times_out():
enhancer.http_timeout_ms = 50
enhancer.default_timeout_ms = 50
result = asyncio.run(enhancer.enhance_prompt(
"A rainy alley at night",
mode="single_clip",
timeout_ms=50,
))
result = asyncio.run(
enhancer.enhance_prompt(
"A rainy alley at night",
mode="single_clip",
timeout_ms=50,
)
)
assert result.fallback_used is False
assert result.error is None
@@ -412,10 +488,12 @@ def test_enhance_prompt_can_use_cerebras_when_it_returns_first():
groq_delay_s=0.08,
)
result = asyncio.run(enhancer.enhance_prompt(
"A rainy alley at night",
mode="single_clip",
))
result = asyncio.run(
enhancer.enhance_prompt(
"A rainy alley at night",
mode="single_clip",
)
)
assert result.fallback_used is False
assert result.error is None
@@ -429,12 +507,15 @@ def test_enhance_prompt_can_use_cerebras_when_it_returns_first():
def test_rewrite_prompt_sequence_keeps_raw_output_on_parse_error():
enhancer = _build_test_enhancer(_chat_payload_with_content("I cannot comply with JSON right now."))
enhancer = _build_test_enhancer(
_chat_payload_with_content("I cannot comply with JSON right now.")
)
result = asyncio.run(
enhancer.rewrite_prompt_sequence(
["prompt one", "prompt two"],
rewrite_instruction="make it cinematic",
))
)
)
assert result.fallback_used is True
assert "No JSON object found in assistant response." in (result.error or "")
assert result.raw_response_text == "I cannot comply with JSON right now."
@@ -446,7 +527,9 @@ def test_rewrite_prompt_sequence_keeps_raw_output_on_parse_error():
def test_rewrite_prompt_sequence_uses_current_rollout_payload_shape():
enhancer = _build_test_enhancer(
_chat_payload_with_content(
'{"id":"rewritten_rollout","label":"Rewritten Rollout","segment_prompts":["A","B"]}'))
'{"id":"rewritten_rollout","label":"Rewritten Rollout","segment_prompts":["A","B"]}'
)
)
captured = {
"body": None,
"timeout_seconds": None,
@@ -457,7 +540,8 @@ def test_rewrite_prompt_sequence_uses_current_rollout_payload_shape():
captured["timeout_seconds"] = timeout_seconds
return (
_chat_payload_with_content(
'{"id":"rewritten_rollout","label":"Rewritten Rollout","segment_prompts":["A","B"]}'),
'{"id":"rewritten_rollout","label":"Rewritten Rollout","segment_prompts":["A","B"]}'
),
'{"id":"rewritten_rollout","label":"Rewritten Rollout","segment_prompts":["A","B"]}',
)
@@ -472,7 +556,8 @@ def test_rewrite_prompt_sequence_uses_current_rollout_payload_shape():
rewrite_model="gpt-test",
rewrite_temperature=0.2,
timeout_ms=800,
))
)
)
assert result.fallback_used is False
assert captured["body"]["messages"][0] == {
@@ -481,12 +566,12 @@ def test_rewrite_prompt_sequence_uses_current_rollout_payload_shape():
}
assert captured["body"]["messages"][1]["role"] == "user"
assert prompt_enhancer_module.json.loads(captured["body"]["messages"][1]["content"]) == {
"mode":
"edit_existing_rollout",
"request": ("Rewrite all segment prompts with improved continuity and cinematic detail. "
"Keep count and ordering identical."),
"user_instruction":
"make it cinematic",
"mode": "edit_existing_rollout",
"request": (
"Rewrite all segment prompts with improved continuity and cinematic detail. "
"Keep count and ordering identical."
),
"user_instruction": "make it cinematic",
"current_rollout": {
"id": "preset_a",
"label": "Preset A",
@@ -497,8 +582,11 @@ def test_rewrite_prompt_sequence_uses_current_rollout_payload_shape():
def test_rewrite_prompt_sequence_supports_new_rollout_mode():
enhancer = _build_test_enhancer(
_chat_payload_with_content('{"id":"custom_editable","label":"Custom rollout","segment_prompts":['
'"A","B","C","D","E","F"]}'))
_chat_payload_with_content(
'{"id":"custom_editable","label":"Custom rollout","segment_prompts":['
'"A","B","C","D","E","F"]}'
)
)
captured = {
"body": None,
}
@@ -507,8 +595,10 @@ def test_rewrite_prompt_sequence_supports_new_rollout_mode():
del timeout_seconds
captured["body"] = body
return (
_chat_payload_with_content('{"id":"custom_editable","label":"Custom rollout","segment_prompts":['
'"A","B","C","D","E","F"]}'),
_chat_payload_with_content(
'{"id":"custom_editable","label":"Custom rollout","segment_prompts":['
'"A","B","C","D","E","F"]}'
),
'{"id":"custom_editable","label":"Custom rollout","segment_prompts":['
'"A","B","C","D","E","F"]}',
)
@@ -524,29 +614,30 @@ def test_rewrite_prompt_sequence_supports_new_rollout_mode():
rewrite_model="gpt-test",
rewrite_temperature=0.2,
timeout_ms=800,
))
)
)
assert result.fallback_used is False
assert result.prompts == ["A", "B", "C", "D", "E", "F"]
assert prompt_enhancer_module.json.loads(captured["body"]["messages"][1]["content"]) == {
"mode":
"new_rollout",
"request": ("Rewrite all segment prompts with improved continuity and cinematic detail. "
"Keep count and ordering identical."),
"user_instruction":
"A moonbase corridor thriller with flooding and red alarms",
"desired_segment_count":
6,
"rollout_id_hint":
"custom_editable",
"rollout_label_hint":
"Custom rollout",
"mode": "new_rollout",
"request": (
"Rewrite all segment prompts with improved continuity and cinematic detail. "
"Keep count and ordering identical."
),
"user_instruction": "A moonbase corridor thriller with flooding and red alarms",
"desired_segment_count": 6,
"rollout_id_hint": "custom_editable",
"rollout_label_hint": "Custom rollout",
}
def test_rewrite_prompt_sequence_uses_session_override_system_prompt():
enhancer = _build_test_enhancer(
_chat_payload_with_content('{"id":"preset_a","label":"Preset A","segment_prompts":["A","B"]}'))
_chat_payload_with_content(
'{"id":"preset_a","label":"Preset A","segment_prompts":["A","B"]}'
)
)
enhancer.rewrite_all_system_prompt = "shared system prompt"
captured = {
"body": None,
@@ -556,7 +647,9 @@ def test_rewrite_prompt_sequence_uses_session_override_system_prompt():
del timeout_seconds
captured["body"] = body
return (
_chat_payload_with_content('{"id":"preset_a","label":"Preset A","segment_prompts":["A","B"]}'),
_chat_payload_with_content(
'{"id":"preset_a","label":"Preset A","segment_prompts":["A","B"]}'
),
'{"id":"preset_a","label":"Preset A","segment_prompts":["A","B"]}',
)
@@ -570,7 +663,8 @@ def test_rewrite_prompt_sequence_uses_session_override_system_prompt():
rewrite_instruction="make it cinematic",
rewrite_model="gpt-test",
system_prompt_override="session specific system prompt",
))
)
)
assert result.fallback_used is False
assert captured["body"]["messages"][0] == {
@@ -581,7 +675,10 @@ def test_rewrite_prompt_sequence_uses_session_override_system_prompt():
def test_resolve_rewrite_new_rollout_system_prompt_uses_dedicated_prompt():
enhancer = _build_test_enhancer(
_chat_payload_with_content('{"id":"preset_a","label":"Preset A","segment_prompts":["A","B"]}'))
_chat_payload_with_content(
'{"id":"preset_a","label":"Preset A","segment_prompts":["A","B"]}'
)
)
enhancer.rewrite_all_system_prompt = "shared rewrite system prompt"
enhancer.rewrite_user_system_prompt = "new rollout rewrite system prompt"
@@ -592,17 +689,24 @@ def test_resolve_rewrite_new_rollout_system_prompt_uses_dedicated_prompt():
def test_resolve_rewrite_new_rollout_system_prompt_prefers_override():
enhancer = _build_test_enhancer(
_chat_payload_with_content('{"id":"preset_a","label":"Preset A","segment_prompts":["A","B"]}'))
_chat_payload_with_content(
'{"id":"preset_a","label":"Preset A","segment_prompts":["A","B"]}'
)
)
enhancer.rewrite_all_system_prompt = "shared rewrite system prompt"
enhancer.rewrite_user_system_prompt = "new rollout rewrite system prompt"
resolved = enhancer.resolve_rewrite_new_rollout_system_prompt("session specific system prompt")
resolved = enhancer.resolve_rewrite_new_rollout_system_prompt(
"session specific system prompt"
)
assert resolved == "session specific system prompt"
def test_generate_auto_prompt_uses_selected_model():
enhancer = _build_test_enhancer(_chat_payload_with_content('{"next_prompt":"Auto next"}'))
enhancer = _build_test_enhancer(
_chat_payload_with_content('{"next_prompt":"Auto next"}')
)
enhancer.auto_system_prompt = "auto system prompt"
enhancer.rewrite_model_options = ["gpt-test", "gpt-alt"]
enhancer.rewrite_default_model = "gpt-test"
@@ -630,7 +734,8 @@ def test_generate_auto_prompt_uses_selected_model():
next_segment_idx=2,
model="gpt-alt",
timeout_ms=800,
))
)
)
assert result.fallback_used is False
assert result.error is None
assert result.prompt == "Auto next"
@@ -639,7 +744,9 @@ def test_generate_auto_prompt_uses_selected_model():
def test_enhance_prompt_uses_selected_model():
enhancer = _build_test_enhancer(_chat_payload_with_content('{"next_prompt":"Enhanced next"}'))
enhancer = _build_test_enhancer(
_chat_payload_with_content('{"next_prompt":"Enhanced next"}')
)
enhancer.enhance_system_prompt = "enhance system prompt"
enhancer.auto_system_prompt = "auto system prompt"
enhancer.rewrite_model_options = ["gpt-test", "gpt-alt"]
@@ -669,7 +776,8 @@ def test_enhance_prompt_uses_selected_model():
next_segment_idx=2,
model="gpt-alt",
timeout_ms=800,
))
)
)
assert result.fallback_used is False
assert result.error is None
assert result.prompt == "Enhanced next"
@@ -678,7 +786,9 @@ def test_enhance_prompt_uses_selected_model():
def test_enhance_prompt_single_clip_uses_auto_extension_prompt_and_prompt_field():
enhancer = _build_test_enhancer(_chat_payload_with_content('{"prompt":"Extended single clip"}'))
enhancer = _build_test_enhancer(
_chat_payload_with_content('{"prompt":"Extended single clip"}')
)
enhancer.enhance_system_prompt = "enhance system prompt"
enhancer.auto_system_prompt = "auto system prompt"
enhancer.rewrite_model_options = ["gpt-test", "gpt-alt"]
@@ -708,12 +818,14 @@ def test_enhance_prompt_single_clip_uses_auto_extension_prompt_and_prompt_field(
enhancer._request_content = _fake_request_content # type: ignore[attr-defined]
result = asyncio.run(enhancer.enhance_prompt(
"short 5s idea",
mode="single_clip",
model="gpt-alt",
timeout_ms=800,
))
result = asyncio.run(
enhancer.enhance_prompt(
"short 5s idea",
mode="single_clip",
model="gpt-alt",
timeout_ms=800,
)
)
assert result.fallback_used is False
assert result.error is None
assert result.prompt == "Extended single clip"
@@ -726,15 +838,17 @@ def test_enhance_prompt_single_clip_uses_auto_extension_prompt_and_prompt_field(
"single 5-second LTX-2.3 video clip. Respond with "
'valid JSON only as {"prompt": "..."}.' # noqa: E501
),
"user_prompt":
"short 5s idea",
"user_prompt": "short 5s idea",
}
def test_enhance_prompt_single_clip_rejects_plain_text_response():
enhancer = _build_test_enhancer(
_chat_payload_with_content("Medium shot of a woman by a rainy cafe window as she lifts her "
"phone, exhales softly, and the camera makes a slow push in."))
_chat_payload_with_content(
"Medium shot of a woman by a rainy cafe window as she lifts her "
"phone, exhales softly, and the camera makes a slow push in."
)
)
enhancer.auto_system_prompt = "auto system prompt"
result = asyncio.run(
@@ -743,14 +857,17 @@ def test_enhance_prompt_single_clip_rejects_plain_text_response():
mode="single_clip",
model="gpt-test",
timeout_ms=800,
))
)
)
assert result.fallback_used is True
assert "No JSON object found in assistant response." in result.error
assert result.prompt == ""
def test_enhance_prompt_single_clip_rejects_segment_prompts_json():
enhancer = _build_test_enhancer(_chat_payload_with_content('{"segment_prompts":["A","B"]}'))
enhancer = _build_test_enhancer(
_chat_payload_with_content('{"segment_prompts":["A","B"]}')
)
enhancer.auto_system_prompt = "auto system prompt"
enhancer.rewrite_model_options = ["gpt-test"]
enhancer.rewrite_default_model = "gpt-test"
@@ -761,14 +878,17 @@ def test_enhance_prompt_single_clip_rejects_segment_prompts_json():
mode="single_clip",
model="gpt-test",
timeout_ms=800,
))
)
)
assert result.fallback_used is True
assert result.prompt == ""
assert "Missing prompt string." in (result.error or "")
def test_enhance_prompt_requires_json_and_does_not_fallback_to_raw_text():
enhancer = _build_test_enhancer(_chat_payload_with_content("A cinematic continuation with slow dolly movement."))
enhancer = _build_test_enhancer(
_chat_payload_with_content("A cinematic continuation with slow dolly movement.")
)
enhancer.enhance_system_prompt = "enhance system prompt"
enhancer.rewrite_model_options = ["gpt-test"]
enhancer.rewrite_default_model = "gpt-test"
@@ -780,14 +900,17 @@ def test_enhance_prompt_requires_json_and_does_not_fallback_to_raw_text():
next_segment_idx=2,
model="gpt-test",
timeout_ms=800,
))
)
)
assert result.fallback_used is True
assert result.prompt == ""
assert "No JSON object found in assistant response." in (result.error or "")
def test_generate_auto_prompt_requires_json_and_does_not_fallback_to_raw_text():
enhancer = _build_test_enhancer(_chat_payload_with_content("A calm, grounded continuation with subtle motion."))
enhancer = _build_test_enhancer(
_chat_payload_with_content("A calm, grounded continuation with subtle motion.")
)
enhancer.auto_system_prompt = "auto system prompt"
enhancer.rewrite_model_options = ["gpt-test"]
enhancer.rewrite_default_model = "gpt-test"
@@ -798,30 +921,34 @@ def test_generate_auto_prompt_requires_json_and_does_not_fallback_to_raw_text():
next_segment_idx=2,
model="gpt-test",
timeout_ms=800,
))
)
)
assert result.fallback_used is True
assert result.prompt == ""
assert "No JSON object found in assistant response." in (result.error or "")
def test_rewrite_prompt_sequence_includes_raw_json_when_content_empty():
enhancer = _build_test_enhancer({
"choices": [{
"finish_reason": "length",
"message": {
"content": [],
"refusal": None,
},
}],
"usage": {
"completion_tokens": 0
},
})
enhancer = _build_test_enhancer(
{
"choices": [
{
"finish_reason": "length",
"message": {
"content": [],
"refusal": None,
},
}
],
"usage": {"completion_tokens": 0},
}
)
result = asyncio.run(
enhancer.rewrite_prompt_sequence(
["prompt one", "prompt two"],
rewrite_instruction="make it cinematic",
))
)
)
assert result.fallback_used is True
assert "No rewrite segment prompts found in assistant response." in (result.error or "")
assert isinstance(result.raw_response_text, str)
@@ -830,7 +957,10 @@ def test_rewrite_prompt_sequence_includes_raw_json_when_content_empty():
def test_get_rewrite_model_config_returns_fixed_defaults():
enhancer = _build_test_enhancer(
_chat_payload_with_content('{"id":"preset_a","label":"Preset A","segment_prompts":["A","B"]}'))
_chat_payload_with_content(
'{"id":"preset_a","label":"Preset A","segment_prompts":["A","B"]}'
)
)
enhancer.rewrite_default_model = "gpt-oss-120b"
enhancer.rewrite_model_options = ["gpt-oss-120b"]
@@ -842,7 +972,10 @@ def test_get_rewrite_model_config_returns_fixed_defaults():
def test_get_prompt_config_includes_auto_extension_prompt():
enhancer = _build_test_enhancer(
_chat_payload_with_content('{"id":"preset_a","label":"Preset A","segment_prompts":["A","B"]}'))
_chat_payload_with_content(
'{"id":"preset_a","label":"Preset A","segment_prompts":["A","B"]}'
)
)
enhancer.enhance_system_prompt_path = "/tmp/next.md"
enhancer.auto_system_prompt_path = "/tmp/auto.md"
enhancer.rewrite_all_system_prompt_path = "/tmp/rewrite.md"
@@ -869,14 +1002,19 @@ def test_get_prompt_config_includes_auto_extension_prompt():
def test_get_prompt_config_reports_loaded_fallback_prompt_path(tmp_path):
enhancer = _build_test_enhancer(
_chat_payload_with_content('{"id":"preset_a","label":"Preset A","segment_prompts":["A","B"]}'))
_chat_payload_with_content(
'{"id":"preset_a","label":"Preset A","segment_prompts":["A","B"]}'
)
)
rewrite_fallback_path = tmp_path / "rewrite_window_system_prompt.md"
rewrite_fallback_path.write_text("rewrite prompt\n", encoding="utf-8")
next_path = tmp_path / "next.md"
next_path.write_text("next prompt\n", encoding="utf-8")
auto_path = tmp_path / "auto.md"
auto_path.write_text("auto prompt\n", encoding="utf-8")
enhancer.rewrite_all_system_prompt_path = str(tmp_path / "prompts.local" / "rewrite_window_system_prompt.md")
enhancer.rewrite_all_system_prompt_path = str(
tmp_path / "prompts.local" / "rewrite_window_system_prompt.md"
)
enhancer.rewrite_all_system_prompt_fallback_path = str(rewrite_fallback_path)
enhancer.enhance_system_prompt_path = str(next_path)
enhancer.auto_system_prompt_path = str(auto_path)
@@ -889,9 +1027,14 @@ def test_get_prompt_config_reports_loaded_fallback_prompt_path(tmp_path):
assert config["rewrite_window_system_prompt_path"] == str(rewrite_fallback_path)
def test_reload_system_prompts_falls_back_to_rewrite_window_when_user_prompt_empty(tmp_path, ):
def test_reload_system_prompts_falls_back_to_rewrite_window_when_user_prompt_empty(
tmp_path,
):
enhancer = _build_test_enhancer(
_chat_payload_with_content('{"id":"preset_a","label":"Preset A","segment_prompts":["A","B"]}'))
_chat_payload_with_content(
'{"id":"preset_a","label":"Preset A","segment_prompts":["A","B"]}'
)
)
next_path = tmp_path / "next.md"
auto_path = tmp_path / "auto.md"
rewrite_path = tmp_path / "rewrite_window_system_prompt.md"
@@ -918,7 +1061,10 @@ def test_reload_system_prompts_falls_back_to_rewrite_window_when_user_prompt_emp
def test_save_prompt_config_updates_auto_extension_prompt(tmp_path):
enhancer = _build_test_enhancer(
_chat_payload_with_content('{"id":"preset_a","label":"Preset A","segment_prompts":["A","B"]}'))
_chat_payload_with_content(
'{"id":"preset_a","label":"Preset A","segment_prompts":["A","B"]}'
)
)
next_path = tmp_path / "next.md"
auto_path = tmp_path / "auto.md"
rewrite_path = tmp_path / "rewrite.md"
@@ -929,7 +1075,9 @@ def test_save_prompt_config_updates_auto_extension_prompt(tmp_path):
enhancer.auto_system_prompt_path = str(auto_path)
enhancer.rewrite_all_system_prompt_path = str(rewrite_path)
config = enhancer.save_prompt_config(auto_extension_system_prompt="auto updated", )
config = enhancer.save_prompt_config(
auto_extension_system_prompt="auto updated",
)
assert auto_path.read_text(encoding="utf-8").strip() == "auto updated"
assert config["auto_extension_system_prompt"] == "auto updated"
@@ -937,7 +1085,10 @@ def test_save_prompt_config_updates_auto_extension_prompt(tmp_path):
def test_save_prompt_config_updates_rewrite_user_prompt(tmp_path):
enhancer = _build_test_enhancer(
_chat_payload_with_content('{"id":"preset_a","label":"Preset A","segment_prompts":["A","B"]}'))
_chat_payload_with_content(
'{"id":"preset_a","label":"Preset A","segment_prompts":["A","B"]}'
)
)
next_path = tmp_path / "next.md"
auto_path = tmp_path / "auto.md"
rewrite_path = tmp_path / "rewrite.md"
@@ -955,7 +1106,9 @@ def test_save_prompt_config_updates_rewrite_user_prompt(tmp_path):
enhancer.rewrite_all_system_prompt_fallback_path = None
enhancer.rewrite_user_system_prompt_fallback_path = None
config = enhancer.save_prompt_config(rewrite_user_system_prompt="rewrite user updated", )
config = enhancer.save_prompt_config(
rewrite_user_system_prompt="rewrite user updated",
)
assert rewrite_user_path.read_text(encoding="utf-8").strip() == "rewrite user updated"
assert config["rewrite_user_system_prompt"] == "rewrite user updated"
@@ -963,7 +1116,10 @@ def test_save_prompt_config_updates_rewrite_user_prompt(tmp_path):
def test_save_prompt_config_updates_rewrite_model(tmp_path):
enhancer = _build_test_enhancer(
_chat_payload_with_content('{"id":"preset_a","label":"Preset A","segment_prompts":["A","B"]}'))
_chat_payload_with_content(
'{"id":"preset_a","label":"Preset A","segment_prompts":["A","B"]}'
)
)
next_path = tmp_path / "next.md"
auto_path = tmp_path / "auto.md"
rewrite_path = tmp_path / "rewrite.md"
@@ -979,7 +1135,9 @@ def test_save_prompt_config_updates_rewrite_model(tmp_path):
enhancer.rewrite_default_model = "gpt-test"
enhancer.rewrite_model_options = ["gpt-test", "gpt-alt"]
config = enhancer.save_prompt_config(rewrite_model="gpt-alt", )
config = enhancer.save_prompt_config(
rewrite_model="gpt-alt",
)
assert enhancer.rewrite_default_model == "gpt-alt"
assert config["rewrite_model"] == "gpt-alt"
@@ -988,7 +1146,10 @@ def test_save_prompt_config_updates_rewrite_model(tmp_path):
def test_save_prompt_config_updates_rewrite_temperature(tmp_path):
enhancer = _build_test_enhancer(
_chat_payload_with_content('{"id":"preset_a","label":"Preset A","segment_prompts":["A","B"]}'))
_chat_payload_with_content(
'{"id":"preset_a","label":"Preset A","segment_prompts":["A","B"]}'
)
)
next_path = tmp_path / "next.md"
auto_path = tmp_path / "auto.md"
rewrite_path = tmp_path / "rewrite.md"
@@ -1002,7 +1163,9 @@ def test_save_prompt_config_updates_rewrite_temperature(tmp_path):
enhancer.auto_system_prompt_fallback_path = None
enhancer.rewrite_all_system_prompt_fallback_path = None
config = enhancer.save_prompt_config(rewrite_temperature=1.3, )
config = enhancer.save_prompt_config(
rewrite_temperature=1.3,
)
assert enhancer.rewrite_default_temperature == 1.3
assert config["rewrite_temperature"] == 1.3
@@ -1010,7 +1173,10 @@ def test_save_prompt_config_updates_rewrite_temperature(tmp_path):
def test_save_prompt_config_creates_versioned_backup_for_existing_prompt(tmp_path):
enhancer = _build_test_enhancer(
_chat_payload_with_content('{"id":"preset_a","label":"Preset A","segment_prompts":["A","B"]}'))
_chat_payload_with_content(
'{"id":"preset_a","label":"Preset A","segment_prompts":["A","B"]}'
)
)
next_path = tmp_path / "next.md"
auto_path = tmp_path / "auto.md"
rewrite_path = tmp_path / "rewrite_window_system_prompt.md"
@@ -1024,9 +1190,13 @@ def test_save_prompt_config_creates_versioned_backup_for_existing_prompt(tmp_pat
enhancer.auto_system_prompt_fallback_path = None
enhancer.rewrite_all_system_prompt_fallback_path = None
enhancer.save_prompt_config(rewrite_window_system_prompt="rewrite updated", )
enhancer.save_prompt_config(
rewrite_window_system_prompt="rewrite updated",
)
backup_paths = sorted(tmp_path.glob("rewrite_window_system_prompt.*.bak.md"))
backup_paths = sorted(
tmp_path.glob("rewrite_window_system_prompt.*.bak.md")
)
assert rewrite_path.read_text(encoding="utf-8").strip() == "rewrite updated"
assert len(backup_paths) == 1
@@ -27,8 +27,13 @@ try:
except ModuleNotFoundError:
websockets = None # type: ignore[assignment]
DEFAULT_PRESET_FILE = (Path(__file__).resolve().parents[2] / "web" / "prompts" /
"selected_ltx2_continuation_story_presets.json")
DEFAULT_PRESET_FILE = (
Path(__file__).resolve().parents[2]
/ "web"
/ "prompts"
/ "selected_ltx2_continuation_story_presets.json"
)
def utc_now_iso() -> str:
@@ -60,7 +65,10 @@ def safe_percentile(values: list[float], percentile: float) -> float | None:
if lower == upper:
return sorted_values[lower]
fraction = rank - lower
return (sorted_values[lower] + (sorted_values[upper] - sorted_values[lower]) * fraction)
return (
sorted_values[lower]
+ (sorted_values[upper] - sorted_values[lower]) * fraction
)
def summarize_series(values: list[float]) -> dict[str, float | int | None]:
@@ -137,16 +145,24 @@ def load_curated_prompts(
selected_id = str(selected.get("id", "")).strip() or "unknown_preset"
raw_prompts = selected.get("segment_prompts", [])
if not isinstance(raw_prompts, list):
raise ValueError(f"Preset {selected_id} has invalid segment_prompts (must be list).")
raise ValueError(
f"Preset {selected_id} has invalid segment_prompts (must be list)."
)
prompts = [str(prompt).strip() for prompt in raw_prompts if isinstance(prompt, str) and str(prompt).strip()]
prompts = [
str(prompt).strip()
for prompt in raw_prompts
if isinstance(prompt, str) and str(prompt).strip()
]
if not prompts:
raise ValueError(f"Preset {selected_id} has no non-empty prompts.")
limited = prompts[:curated_limit]
if not limited:
raise ValueError(f"curated_limit={curated_limit} produced no prompts for preset "
f"{selected_id}.")
raise ValueError(
f"curated_limit={curated_limit} produced no prompts for preset "
f"{selected_id}."
)
return selected_id, limited, len(prompts)
@@ -208,11 +224,11 @@ async def run_single_session(
try:
async with websockets.connect(
url,
max_size=None,
ping_interval=None,
open_timeout=connect_timeout_s,
close_timeout=2.0,
url,
max_size=None,
ping_interval=None,
open_timeout=connect_timeout_s,
close_timeout=2.0,
) as ws:
connect_finish_monotonic = time.monotonic()
session_data["connect_finish_ts_utc"] = utc_now_iso()
@@ -233,7 +249,9 @@ async def run_single_session(
timeout_remaining = session_timeout_s - elapsed_s
if timeout_remaining <= 0:
session_data["status"] = "timeout"
session_data["error"] = (f"Session timed out after {session_timeout_s:.1f}s.")
session_data["error"] = (
f"Session timed out after {session_timeout_s:.1f}s."
)
break
recv_start_epoch = time.time()
@@ -247,7 +265,9 @@ async def run_single_session(
)
except asyncio.TimeoutError:
session_data["status"] = "timeout"
session_data["error"] = ("Timed out waiting for websocket message.")
session_data["error"] = (
"Timed out waiting for websocket message."
)
break
except Exception as exc:
session_data["status"] = "failed"
@@ -268,16 +288,20 @@ async def run_single_session(
chunk_gap_ms: float | None = None
if last_chunk_finish_monotonic is not None:
chunk_gap_ms = (recv_finish_monotonic - last_chunk_finish_monotonic) * 1000.0
chunk_gap_ms = (
recv_finish_monotonic - last_chunk_finish_monotonic
) * 1000.0
session_data["chunks"].append({
"segment_idx": current_segment_idx,
"chunk_idx": session_data["total_chunks"],
"size_bytes": len(message),
"chunk_start_ts_utc": recv_start_iso,
"chunk_finish_ts_utc": recv_finish_iso,
"chunk_gap_ms": chunk_gap_ms,
})
session_data["chunks"].append(
{
"segment_idx": current_segment_idx,
"chunk_idx": session_data["total_chunks"],
"size_bytes": len(message),
"chunk_start_ts_utc": recv_start_iso,
"chunk_finish_ts_utc": recv_finish_iso,
"chunk_gap_ms": chunk_gap_ms,
}
)
last_chunk_finish_monotonic = recv_finish_monotonic
last_chunk_finish_epoch = recv_finish_epoch
session_data["last_chunk_finish_ts_utc"] = recv_finish_iso
@@ -297,7 +321,9 @@ async def run_single_session(
if msg_type == "gpu_assigned":
session_data["gpu_assigned_ts_utc"] = recv_finish_iso
if connect_finish_monotonic is not None:
session_data["queue_wait_ms"] = (recv_finish_monotonic - connect_finish_monotonic) * 1000.0
session_data["queue_wait_ms"] = (
recv_finish_monotonic - connect_finish_monotonic
) * 1000.0
elif msg_type == "ltx2_stream_start":
if initial_total_segments is None:
parsed_total = parse_int(data.get("total_segments"))
@@ -312,13 +338,20 @@ async def run_single_session(
session_data["media_segments_completed"] += 1
if first_media_segment_complete_epoch is None:
first_media_segment_complete_epoch = recv_finish_epoch
session_data["first_media_segment_complete_ts_utc"] = recv_finish_iso
session_data[
"first_media_segment_complete_ts_utc"
] = recv_finish_iso
elif msg_type == "ltx2_segment_complete":
session_data["segments_completed"] += 1
seg_idx = parse_int(data.get("segment_idx"))
if (initial_total_segments is not None and seg_idx is not None
and seg_idx >= initial_total_segments):
session_data["target_segment_complete_ts_utc"] = recv_finish_iso
if (
initial_total_segments is not None
and seg_idx is not None
and seg_idx >= initial_total_segments
):
session_data[
"target_segment_complete_ts_utc"
] = recv_finish_iso
await asyncio.sleep(post_complete_wait_s)
session_data["leave_sent_ts_utc"] = utc_now_iso()
try:
@@ -329,11 +362,15 @@ async def run_single_session(
break
elif msg_type == "session_timeout":
session_data["status"] = "timeout"
session_data["error"] = str(data.get("message") or "Backend session timeout")
session_data["error"] = str(
data.get("message") or "Backend session timeout"
)
break
elif msg_type == "error":
session_data["status"] = "failed"
session_data["error"] = str(data.get("message") or "Backend error message")
session_data["error"] = str(
data.get("message") or "Backend error message"
)
break
if session_data["status"] == "failed" and session_data["error"] is None:
@@ -342,18 +379,29 @@ async def run_single_session(
session_data["status"] = "failed"
session_data["error"] = f"WebSocket connect/run failed: {exc}"
if (first_chunk_finish_epoch is not None and last_chunk_finish_epoch is not None
and session_data["total_chunk_bytes"] > 0):
if (
first_chunk_finish_epoch is not None
and last_chunk_finish_epoch is not None
and session_data["total_chunk_bytes"] > 0
):
duration_s = last_chunk_finish_epoch - first_chunk_finish_epoch
if duration_s > 0:
session_data["session_goodput_mbps"] = (session_data["total_chunk_bytes"] * 8.0 / duration_s / 1_000_000.0)
session_data["session_goodput_mbps"] = (
session_data["total_chunk_bytes"] * 8.0 / duration_s / 1_000_000.0
)
if (first_chunk_finish_epoch is not None and first_media_segment_complete_epoch is not None):
session_data["first_chunk_before_first_media_complete"] = (first_chunk_finish_epoch
< first_media_segment_complete_epoch)
if (
first_chunk_finish_epoch is not None
and first_media_segment_complete_epoch is not None
):
session_data["first_chunk_before_first_media_complete"] = (
first_chunk_finish_epoch < first_media_segment_complete_epoch
)
session_data["close_ts_utc"] = utc_now_iso()
session_data["duration_ms"] = (time.monotonic() - session_start_monotonic) * 1000.0
session_data["duration_ms"] = (
time.monotonic() - session_start_monotonic
) * 1000.0
return session_data
@@ -364,11 +412,14 @@ async def run_worker_sessions(
config: dict[str, Any],
) -> list[dict[str, Any]]:
tasks = [
asyncio.create_task(run_single_session(
worker_id=worker_id,
worker_session_idx=idx,
config=config,
)) for idx in range(session_count)
asyncio.create_task(
run_single_session(
worker_id=worker_id,
worker_session_idx=idx,
config=config,
)
)
for idx in range(session_count)
]
if not tasks:
return []
@@ -386,23 +437,29 @@ def worker_entry(
try:
ready_queue.put({"worker_id": worker_id, "status": "ready"})
start_event.wait()
sessions = asyncio.run(run_worker_sessions(
worker_id=worker_id,
session_count=session_count,
config=config,
))
result_queue.put({
"worker_id": worker_id,
"status": "ok",
"sessions": sessions,
})
sessions = asyncio.run(
run_worker_sessions(
worker_id=worker_id,
session_count=session_count,
config=config,
)
)
result_queue.put(
{
"worker_id": worker_id,
"status": "ok",
"sessions": sessions,
}
)
except Exception as exc:
result_queue.put({
"worker_id": worker_id,
"status": "error",
"error": str(exc),
"traceback": traceback.format_exc(),
})
result_queue.put(
{
"worker_id": worker_id,
"status": "error",
"error": str(exc),
"traceback": traceback.format_exc(),
}
)
def build_summary(
@@ -460,22 +517,33 @@ def build_summary(
if len(all_chunk_finish_epochs) >= 2 and total_chunk_bytes > 0:
duration_s = max(all_chunk_finish_epochs) - min(all_chunk_finish_epochs)
if duration_s > 0:
global_goodput_mbps = (total_chunk_bytes * 8.0 / duration_s / 1_000_000.0)
global_goodput_mbps = (
total_chunk_bytes * 8.0 / duration_s / 1_000_000.0
)
bucket_throughputs_mbps = [(bytes_count * 8.0) / 1_000_000.0 for _, bytes_count in sorted(bucket_bytes.items())]
bucket_throughputs_mbps = [
(bytes_count * 8.0) / 1_000_000.0
for _, bytes_count in sorted(bucket_bytes.items())
]
bucket_stats = summarize_series(bucket_throughputs_mbps)
chunk_gap_threshold_breaches = [value for value in chunk_gaps if value >= chunk_gap_threshold_ms]
chunk_gap_threshold_breaches = [
value for value in chunk_gaps if value >= chunk_gap_threshold_ms
]
non_success = len(sessions) - status_counts.get("success", 0)
fail_reasons: list[str] = []
if non_success > 0:
fail_reasons.append(f"{non_success} session(s) did not complete successfully.")
fail_reasons.append(
f"{non_success} session(s) did not complete successfully."
)
if not chunk_gaps:
fail_reasons.append("No chunk gap data collected.")
if chunk_gap_threshold_breaches:
fail_reasons.append(f"{len(chunk_gap_threshold_breaches)} chunk gap(s) were >= "
f"{chunk_gap_threshold_ms:.0f}ms.")
fail_reasons.append(
f"{len(chunk_gap_threshold_breaches)} chunk gap(s) were >= "
f"{chunk_gap_threshold_ms:.0f}ms."
)
passed = len(fail_reasons) == 0
progressive_ratio = None
@@ -486,18 +554,20 @@ def build_summary(
"passed": passed,
"fail_reasons": fail_reasons,
"sessions": {
"total":
len(sessions),
"success":
status_counts.get("success", 0),
"failed":
status_counts.get("failed", 0),
"timeout":
status_counts.get("timeout", 0),
"protocol_error":
status_counts.get("protocol_error", 0),
"other": (len(sessions) - (status_counts.get("success", 0) + status_counts.get("failed", 0) +
status_counts.get("timeout", 0) + status_counts.get("protocol_error", 0))),
"total": len(sessions),
"success": status_counts.get("success", 0),
"failed": status_counts.get("failed", 0),
"timeout": status_counts.get("timeout", 0),
"protocol_error": status_counts.get("protocol_error", 0),
"other": (
len(sessions)
- (
status_counts.get("success", 0)
+ status_counts.get("failed", 0)
+ status_counts.get("timeout", 0)
+ status_counts.get("protocol_error", 0)
)
),
},
"chunk_gap_ms": {
**chunk_gap_stats,
@@ -536,39 +606,51 @@ def print_summary(
bucket_bw = bandwidth["bucketed_1s"]
print("=== LTX2 Realtime Stress Test Summary ===")
print("Run: "
f"url={run_info['url']} clients={run_info['clients']} "
f"processes={run_info['processes']} "
f"preset={run_info['preset_id']} "
f"curated_limit={run_info['curated_limit']}")
print("Sessions: "
f"total={sessions['total']} success={sessions['success']} "
f"failed={sessions['failed']} timeout={sessions['timeout']} "
f"protocol_error={sessions['protocol_error']}")
print("Chunk gap ms: "
f"min={format_num(chunk_gap['min'])} "
f"p50={format_num(chunk_gap['p50'])} "
f"p95={format_num(chunk_gap['p95'])} "
f"p99={format_num(chunk_gap['p99'])} "
f"max={format_num(chunk_gap['max'])} "
f"threshold={format_num(chunk_gap['threshold_ms'])} "
f"breaches={chunk_gap['breach_count']}")
print("Queue wait ms: "
f"min={format_num(queue_wait['min'])} "
f"p50={format_num(queue_wait['p50'])} "
f"p95={format_num(queue_wait['p95'])} "
f"max={format_num(queue_wait['max'])}")
print(
"Run: "
f"url={run_info['url']} clients={run_info['clients']} "
f"processes={run_info['processes']} "
f"preset={run_info['preset_id']} "
f"curated_limit={run_info['curated_limit']}"
)
print(
"Sessions: "
f"total={sessions['total']} success={sessions['success']} "
f"failed={sessions['failed']} timeout={sessions['timeout']} "
f"protocol_error={sessions['protocol_error']}"
)
print(
"Chunk gap ms: "
f"min={format_num(chunk_gap['min'])} "
f"p50={format_num(chunk_gap['p50'])} "
f"p95={format_num(chunk_gap['p95'])} "
f"p99={format_num(chunk_gap['p99'])} "
f"max={format_num(chunk_gap['max'])} "
f"threshold={format_num(chunk_gap['threshold_ms'])} "
f"breaches={chunk_gap['breach_count']}"
)
print(
"Queue wait ms: "
f"min={format_num(queue_wait['min'])} "
f"p50={format_num(queue_wait['p50'])} "
f"p95={format_num(queue_wait['p95'])} "
f"max={format_num(queue_wait['max'])}"
)
ratio = progressive["ratio"]
ratio_text = "n/a" if ratio is None else f"{ratio * 100:.2f}%"
print("Progressive streaming: "
f"{progressive['success_sessions']}/"
f"{progressive['eligible_sessions']} ({ratio_text})")
print("Bandwidth Mbps: "
f"per_session_avg={format_num(per_session_bw['avg'])} "
f"per_session_p95={format_num(per_session_bw['p95'])} "
f"global={format_num(bandwidth['global_goodput_mbps'])} "
f"bucket_avg={format_num(bucket_bw['avg_mbps'])} "
f"bucket_peak={format_num(bucket_bw['peak_mbps'])}")
print(
"Progressive streaming: "
f"{progressive['success_sessions']}/"
f"{progressive['eligible_sessions']} ({ratio_text})"
)
print(
"Bandwidth Mbps: "
f"per_session_avg={format_num(per_session_bw['avg'])} "
f"per_session_p95={format_num(per_session_bw['p95'])} "
f"global={format_num(bandwidth['global_goodput_mbps'])} "
f"bucket_avg={format_num(bucket_bw['avg_mbps'])} "
f"bucket_peak={format_num(bucket_bw['peak_mbps'])}"
)
print(f"VERDICT: {'PASS' if summary['passed'] else 'FAIL'}")
if summary["fail_reasons"]:
print("Fail reasons:")
@@ -588,8 +670,10 @@ def distribute_sessions(total_clients: int, process_count: int) -> list[int]:
def run_stress(args: argparse.Namespace) -> tuple[dict[str, Any], int]:
if websockets is None:
raise RuntimeError("Missing dependency: websockets. Install it before running this "
"stress test.")
raise RuntimeError(
"Missing dependency: websockets. Install it before running this "
"stress test."
)
preset_file = Path(args.preset_file).expanduser().resolve()
selected_preset_id, curated_prompts, total_prompt_count = load_curated_prompts(
@@ -651,8 +735,13 @@ def run_stress(args: argparse.Namespace) -> tuple[dict[str, Any], int]:
start_event.set()
result_deadline = (time.monotonic() + args.connect_timeout_s + args.session_timeout_s +
args.post_complete_wait_s + 180.0)
result_deadline = (
time.monotonic()
+ args.connect_timeout_s
+ args.session_timeout_s
+ args.post_complete_wait_s
+ 180.0
)
worker_results: list[dict[str, Any]] = []
while len(worker_results) < len(processes):
timeout_s = max(0.1, result_deadline - time.monotonic())
@@ -676,20 +765,24 @@ def run_stress(args: argparse.Namespace) -> tuple[dict[str, Any], int]:
if result.get("status") == "ok":
sessions.extend(result.get("sessions", []))
else:
worker_errors.append({
"worker_id": result.get("worker_id"),
"error": result.get("error"),
"traceback": result.get("traceback"),
})
worker_errors.append(
{
"worker_id": result.get("worker_id"),
"error": result.get("error"),
"traceback": result.get("traceback"),
}
)
received_workers = {result.get("worker_id") for result in worker_results}
expected_workers = set(range(len(processes)))
missing_workers = sorted(expected_workers - received_workers)
for worker_id in missing_workers:
worker_errors.append({
"worker_id": worker_id,
"error": "No worker result received.",
})
worker_errors.append(
{
"worker_id": worker_id,
"error": "No worker result received.",
}
)
run_end_epoch = time.time()
run_end_iso = iso_from_epoch(run_end_epoch)
@@ -702,8 +795,9 @@ def run_stress(args: argparse.Namespace) -> tuple[dict[str, Any], int]:
if worker_errors:
summary["passed"] = False
summary["fail_reasons"] = list(
summary["fail_reasons"]) + [f"{len(worker_errors)} worker error(s) occurred."]
summary["fail_reasons"] = list(summary["fail_reasons"]) + [
f"{len(worker_errors)} worker error(s) occurred."
]
output_payload = {
"run_info": {
@@ -739,7 +833,9 @@ def run_stress(args: argparse.Namespace) -> tuple[dict[str, Any], int]:
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description="Multiprocess realtime stress test for LTX2 streaming.", )
parser = argparse.ArgumentParser(
description="Multiprocess realtime stress test for LTX2 streaming.",
)
parser.add_argument(
"-u",
"--url",
@@ -47,11 +47,13 @@ def test_persist_session_init_image_returns_none_when_missing_data():
def test_persist_session_init_image_rejects_unsupported_mime():
with pytest.raises(ValueError, match="PNG, JPEG, or WebP"):
persist_session_init_image({
"name": "frame.gif",
"mime_type": "image/gif",
"data_url": "data:image/gif;base64,R0lGODlhAQABAAAAACw=",
})
persist_session_init_image(
{
"name": "frame.gif",
"mime_type": "image/gif",
"data_url": "data:image/gif;base64,R0lGODlhAQABAAAAACw=",
}
)
def test_persist_session_init_image_rejects_large_payload(monkeypatch):
@@ -64,8 +66,10 @@ def test_persist_session_init_image_rejects_large_payload(monkeypatch):
monkeypatch.setattr(base64, "b64decode", fake_b64decode)
with pytest.raises(ValueError, match="15 MB or smaller"):
persist_session_init_image({
"name": "frame.png",
"mime_type": "image/png",
"data_url": data_url,
})
persist_session_init_image(
{
"name": "frame.png",
"mime_type": "image/png",
"data_url": data_url,
}
)
File diff suppressed because it is too large Load Diff
+3 -2
View File
@@ -15,5 +15,6 @@ def _utc_now_iso() -> str:
PROMPT_EXTENSION_FAILURE_USER_MESSAGE = ("Prompt extension failed for this request.")
def _resolve_generation_segment_cap(*, single_clip_mode: bool, cap: int) -> int:
return 0 if single_clip_mode else cap
def _resolve_generation_segment_cap(*, single_clip_mode: bool, cap: int, manual_continuation_mode: bool = False) -> int:
# Steering (manual continuation) lets the user keep going indefinitely, like single-clip mode.
return 0 if (single_clip_mode or manual_continuation_mode) else cap
@@ -489,7 +489,7 @@ class VideoGenerationWorker:
num_inference_steps=NUM_INFERENCE_STEPS,
guidance_scale=1.0,
seed=10,
ltx2_image_crf=0.0,
ltx2_image_crf=(33.0 if image_path and segment_idx == 1 else 0.0),
image_path=image_path if segment_idx == 1 else None,
return_continuation_state=False,
)
+3
View File
@@ -29,6 +29,9 @@ test = [
dreamverse-server = "dreamverse.server_entry:cli"
dreamverse-mock-server = "dreamverse.mock_server:cli"
[tool.setuptools.packages.find]
include = ["dreamverse*"]
[tool.uv]
package = false
+67
View File
@@ -0,0 +1,67 @@
#!/usr/bin/env bash
# launch-dreamverse.sh — launch dreamverse-server on a compute node.
#
# Usage (from repo root):
# bash apps/dreamverse/scripts/launch-dreamverse.sh # GPUs 0-3, SP_SIZE=4
# CUDA_VISIBLE_DEVICES=0 bash apps/dreamverse/scripts/launch-dreamverse.sh # single GPU, SP_SIZE=1
set -euo pipefail
CONDA_PREFIX="$HOME/miniconda3/envs/dreamverse"
CUDA_RT_DIR="$CONDA_PREFIX/lib/python3.11/site-packages/nvidia/cuda_runtime/lib"
GXX="$CONDA_PREFIX/bin/aarch64-conda-linux-gnu-g++"
export CUDA_HOME="$CONDA_PREFIX"
export FASTVIDEO_ENABLE_STARTUP_WARMUP=true
export FASTVIDEO_ENABLE_PROMPT_SAFETY=false
export DREAMVERSE_MAX_AUTOTUNE=true
export LTX2_USE_DISTILLED_SIGMAS=0
export LTX2_VIDEO_CONDITIONING_NUM_FRAMES=1
export AUDIO_CONDITIONING_NUM_FRAMES=41
export DREAMVERSE_SESSION_TIMEOUT_SECONDS="${DREAMVERSE_SESSION_TIMEOUT_SECONDS:-1800}"
# GB200 max-autotune warmup compiles can run for hours; keep the watchdog generous here
export FASTVIDEO_STARTUP_WARMUP_TIMEOUT_SECONDS="${FASTVIDEO_STARTUP_WARMUP_TIMEOUT_SECONDS:-24000}"
export CEREBRAS_API_KEY="${CEREBRAS_API_KEY:-}" # set this in your env or ~/.env
export FASTVIDEO_PROMPT_CEREBRAS_MODEL="gpt-oss-120b"
export TORCHINDUCTOR_CACHE_DIR="$HOME/.cache/torchinductor"
export TRITON_CACHE_DIR="$HOME/.triton/cache"
export TORCH_CUDA_ARCH_LIST="10.0a"
# Compiler env (needed for flashinfer JIT compilation at server startup)
export CXX="$CONDA_PREFIX/compiler_compat/g++"
export CC="$CONDA_PREFIX/compiler_compat/gcc"
export CUDAHOSTCXX="$GXX"
export NVCC_PREPEND_FLAGS="-ccbin $GXX -allow-unsupported-compiler"
# Link against libcudart.so.12 at JIT compile time; stubs for libcuda.so
# cuda-compat has libcudart.so -> libcudart.so.12 (linker needs unversioned name)
export LIBRARY_PATH="$CONDA_PREFIX/lib/cuda-compat:$CONDA_PREFIX/lib/stubs"
# Only libcudart.so.12 at runtime — prevents cuDNN from seeing .so.13
export LD_LIBRARY_PATH="$CUDA_RT_DIR"
CUDA_VISIBLE_DEVICES="${CUDA_VISIBLE_DEVICES:-0,1,2,3}"
export FASTVIDEO_GPU_COUNT="${FASTVIDEO_GPU_COUNT:-all}"
# Default SP size to the usable GPU count so single-GPU invocations work.
# A numeric FASTVIDEO_GPU_COUNT caps the pool below the visible count, and an
# SP size above the pool size fails GPUPool startup with "Not enough GPUs".
IFS=',' read -ra _VISIBLE_GPUS <<< "$CUDA_VISIBLE_DEVICES"
# Count only non-empty tokens, matching gpu_pool.get_available_gpus (e.g. ",0,1" is 2 GPUs).
_USABLE_GPU_COUNT=0
for _gpu in "${_VISIBLE_GPUS[@]}"; do
[[ -n "${_gpu//[[:space:]]/}" ]] && _USABLE_GPU_COUNT=$((_USABLE_GPU_COUNT + 1))
done
if [[ "$FASTVIDEO_GPU_COUNT" =~ ^[0-9]+$ ]] && (( FASTVIDEO_GPU_COUNT < _USABLE_GPU_COUNT )); then
_USABLE_GPU_COUNT="$FASTVIDEO_GPU_COUNT"
fi
export DREAMVERSE_SP_SIZE="${DREAMVERSE_SP_SIZE:-$_USABLE_GPU_COUNT}"
PORT="${DREAMVERSE_PORT:-8009}"
FFMPEG_ENV="$(dirname "$0")/ffmpeg-env.sh"
# shellcheck source=ffmpeg-env.sh
[[ -f "$FFMPEG_ENV" ]] && source "$FFMPEG_ENV"
echo "==> Launching dreamverse-server on GPU $CUDA_VISIBLE_DEVICES port $PORT"
CUDA_VISIBLE_DEVICES="$CUDA_VISIBLE_DEVICES" \
"$CONDA_PREFIX/bin/dreamverse-server" --host 0.0.0.0 --port "$PORT"
+30
View File
@@ -0,0 +1,30 @@
#!/usr/bin/env bash
# launch-frontend.sh — start the Dreamverse Next.js dev server and ngrok tunnel.
#
# Usage (from repo root):
# bash apps/dreamverse/scripts/launch-frontend.sh
#
# Override backend or ngrok URL via env:
# BACKEND_HOST=1.2.3.4 bash apps/dreamverse/scripts/launch-frontend.sh
set -euo pipefail
CONDA_PREFIX="$HOME/miniconda3/envs/dreamverse"
BACKEND_HOST="${BACKEND_HOST:-10.244.18.228}"
BACKEND_PORT="${BACKEND_PORT:-8009}"
NGROK_URL="${NGROK_URL:-ltx23.ngrok.app}"
WEB_DIR="$(git rev-parse --show-toplevel)/apps/dreamverse/web"
cleanup() {
echo "==> Shutting down..."
kill "$FRONTEND_PID" 2>/dev/null || true
}
trap cleanup EXIT
echo "==> Starting frontend (backend: $BACKEND_HOST:$BACKEND_PORT)"
BACKEND_HOST="$BACKEND_HOST" BACKEND_PORT="$BACKEND_PORT" \
npm run --prefix "$WEB_DIR" dev &
FRONTEND_PID=$!
echo "==> Starting ngrok tunnel -> $NGROK_URL"
"$CONDA_PREFIX/bin/ngrok" http --url="$NGROK_URL" 5299
@@ -39,17 +39,6 @@ export FASTVIDEO_GENERATION_SEGMENT_CAP="${FASTVIDEO_GENERATION_SEGMENT_CAP:-6}"
export FASTVIDEO_PROMPT_AUTO_SLEEP_MS="${FASTVIDEO_PROMPT_AUTO_SLEEP_MS:-120}"
export FASTVIDEO_PROMPT_AUTO_TIMEOUT_MS="${FASTVIDEO_PROMPT_AUTO_TIMEOUT_MS:-1800}"
if [[ "${ENABLE_TORCH_COMPILE}" == "1" ]]; then
# Persist Inductor, AOTAutograd, and Triton artifacts across launches.
export DREAMVERSE_TORCH_COMPILE_CACHE_ROOT="${DREAMVERSE_TORCH_COMPILE_CACHE_ROOT:-${HOME}/.cache/dreamverse/torch_compile}"
export TORCHINDUCTOR_CACHE_DIR="${TORCHINDUCTOR_CACHE_DIR:-${DREAMVERSE_TORCH_COMPILE_CACHE_ROOT}/inductor}"
export TRITON_CACHE_DIR="${TRITON_CACHE_DIR:-${DREAMVERSE_TORCH_COMPILE_CACHE_ROOT}/triton}"
export TORCHINDUCTOR_FX_GRAPH_CACHE="${TORCHINDUCTOR_FX_GRAPH_CACHE:-1}"
export TORCHINDUCTOR_AUTOGRAD_CACHE="${TORCHINDUCTOR_AUTOGRAD_CACHE:-1}"
mkdir -p "${TORCHINDUCTOR_CACHE_DIR}" "${TRITON_CACHE_DIR}"
echo "[launch-demo] torch.compile cache: ${DREAMVERSE_TORCH_COMPILE_CACHE_ROOT}"
fi
cd "${DREAMVERSE_ROOT}"
if ! command -v dreamverse-server >/dev/null 2>&1; then
+15 -9
View File
@@ -8,10 +8,12 @@ import modal
IMAGE = os.environ.get("DREAMVERSE_IMAGE")
if not IMAGE:
raise RuntimeError("DREAMVERSE_IMAGE is required. Set it to a published SHA-specific Dreamverse image, "
"for example a dreamverse-backend-cuda13.0.0-sha-* tag or a "
"dreamverse-ui-cuda13.0.0-sha-* tag if serving the static UI. "
"CUDA 12 / cu126 images use the corresponding cuda12.6.3 tag.")
raise RuntimeError(
"DREAMVERSE_IMAGE is required. Set it to a published SHA-specific Dreamverse image, "
"for example a dreamverse-backend-cuda13.0.0-sha-* tag or a "
"dreamverse-ui-cuda13.0.0-sha-* tag if serving the static UI. "
"CUDA 12 / cu126 images use the corresponding cuda12.6.3 tag."
)
# ``@modal.web_server`` invokes ``serve()`` directly and bypasses the image
# ENTRYPOINT (``docker/docker_entrypoint.sh``). That entrypoint normally
@@ -63,10 +65,14 @@ def serve():
# ``or ""`` collapses ``None`` (unset) into an empty string, ``.strip()``
# collapses whitespace-only values (e.g. ``" "``) — both should be
# treated as missing.
missing = [k for k in _REQUIRED_SECRET_KEYS if not (os.environ.get(k) or "").strip()]
missing = [
k for k in _REQUIRED_SECRET_KEYS
if not (os.environ.get(k) or "").strip()
]
if missing:
raise RuntimeError("dreamverse-api-keys secret is missing required entries: "
f"{', '.join(missing)}. Add them with `modal secret create "
"dreamverse-api-keys ... --force` and redeploy "
"(see apps/dreamverse/scripts/modal/README.md).")
raise RuntimeError(
"dreamverse-api-keys secret is missing required entries: "
f"{', '.join(missing)}. Add them with `modal secret create "
"dreamverse-api-keys ... --force` and redeploy "
"(see apps/dreamverse/scripts/modal/README.md).")
subprocess.Popen(["dreamverse-server", "--host", "0.0.0.0", "--port", "8009"])
+105
View File
@@ -0,0 +1,105 @@
#!/usr/bin/env bash
# setup-dreamverse-env.sh — create and configure the dreamverse conda env
# from scratch on this aarch64 NFS Slurm cluster.
#
# Run from the login node (from the repo root):
# bash apps/dreamverse/scripts/setup-dreamverse-env.sh
#
# After this script completes, use launch-dreamverse.sh on a compute node.
set -euo pipefail
REPO_ROOT="$(git rev-parse --show-toplevel)"
ENV_NAME="dreamverse"
LOCAL_DIR="/mnt/local/hal-kevin" # cache/pkgs — keep on local disk
CONDA_PREFIX="$HOME/miniconda3/envs/$ENV_NAME" # env — on shared NFS so it survives node changes
echo "==> Removing existing env if present"
conda env remove -p "$CONDA_PREFIX" -y 2>/dev/null || true
rm -rf "$CONDA_PREFIX" 2>/dev/null || true
echo "==> Creating conda env at $CONDA_PREFIX"
CONDA_PKGS_DIRS="$LOCAL_DIR/conda/pkgs" conda create -p "$CONDA_PREFIX" python=3.11 -y
GXX="$CONDA_PREFIX/bin/aarch64-conda-linux-gnu-g++"
GCC="$CONDA_PREFIX/bin/aarch64-conda-linux-gnu-gcc"
echo "==> Installing compiler"
CONDA_PKGS_DIRS="$LOCAL_DIR/conda/pkgs" conda install -p "$CONDA_PREFIX" gxx_linux-aarch64 -y
echo "==> Installing CUDA toolkit (nvcc + headers)"
CONDA_PKGS_DIRS="$LOCAL_DIR/conda/pkgs" conda install -p "$CONDA_PREFIX" -c nvidia cuda-toolkit -y
echo "==> Hiding conflicting libcudart.so.13 immediately"
mkdir -p "$CONDA_PREFIX/lib/hidden"
mv "$CONDA_PREFIX"/lib/libcudart.so* "$CONDA_PREFIX/lib/hidden/" 2>/dev/null || true
echo "==> Fixing compiler_compat symlinks"
mkdir -p "$CONDA_PREFIX/compiler_compat"
ln -sf "$GXX" "$CONDA_PREFIX/compiler_compat/g++"
ln -sf "$GCC" "$CONDA_PREFIX/compiler_compat/gcc"
echo "==> Symlinking CUDA headers to standard location"
for f in "$CONDA_PREFIX/targets/sbsa-linux/include/"*; do
ln -sf "$f" "$CONDA_PREFIX/include/$(basename "$f")" 2>/dev/null || true
done
echo "==> Installing ffmpeg (native build with x264 + NVENC)"
CUDA_PREFIX="$CONDA_PREFIX" bash "$REPO_ROOT/apps/dreamverse/scripts/install_native_ffmpeg.sh"
echo "==> Installing pip and uv"
CONDA_PKGS_DIRS="$LOCAL_DIR/conda/pkgs" conda install -p "$CONDA_PREFIX" pip -y
"$CONDA_PREFIX/bin/pip" install uv
echo "==> Setting compiler env vars"
export UV_CACHE_DIR="$LOCAL_DIR/cache"
export UV_LINK_MODE=copy
export CXX="$CONDA_PREFIX/compiler_compat/g++"
export CC="$CONDA_PREFIX/compiler_compat/gcc"
export CUDAHOSTCXX="$GXX"
export NVCC_PREPEND_FLAGS="-ccbin $GXX -allow-unsupported-compiler"
export CUDA_HOME="$CONDA_PREFIX"
# Only build for GB200 (sm_100a); CUDA 13 dropped support for older archs
export TORCH_CUDA_ARCH_LIST="10.0a"
echo "==> Installing torch with CUDA 12.8"
UV_CACHE_DIR="$LOCAL_DIR/cache" "$CONDA_PREFIX/bin/uv" pip install torch==2.11.0 torchvision \
--index-url https://download.pytorch.org/whl/cu128
echo "==> Hiding any newly introduced libcudart.so.13"
mv "$CONDA_PREFIX"/lib/libcudart.so* "$CONDA_PREFIX/lib/hidden/" 2>/dev/null || true
# Set paths now that torch (and its nvidia packages) are installed
CUDA_RT_DIR="$CONDA_PREFIX/lib/python3.11/site-packages/nvidia/cuda_runtime/lib"
CUDA_RT_SO="$(ls "$CUDA_RT_DIR"/libcudart.so.* 2>/dev/null | head -1)"
# The pip nvidia package only has libcudart.so.12 (versioned), not libcudart.so.
# The linker needs the unversioned name to satisfy -lcudart. Create a compat dir.
mkdir -p "$CONDA_PREFIX/lib/cuda-compat"
ln -sf "$CUDA_RT_SO" "$CONDA_PREFIX/lib/cuda-compat/libcudart.so"
export LIBRARY_PATH="$CONDA_PREFIX/lib/cuda-compat:$CONDA_PREFIX/lib/stubs"
export CMAKE_ARGS="-DCUDA_CUDART_LIBRARY=$CUDA_RT_SO -DCUDA_INCLUDE_DIRS=$CONDA_PREFIX/targets/sbsa-linux/include"
echo "==> Installing build tools"
"$CONDA_PREFIX/bin/pip" install scikit-build-core cmake ninja
echo "==> Initializing git submodules"
cd "$REPO_ROOT"
git submodule update --init fastvideo-kernel/include/cutlass fastvideo-kernel/include/tk
echo "==> Building fastvideo-kernel from local source"
UV_CACHE_DIR="$LOCAL_DIR/cache" "$CONDA_PREFIX/bin/uv" pip install \
-e "./fastvideo-kernel" --no-build-isolation
echo "==> Installing fastvideo + dreamverse extras"
UV_CACHE_DIR="$LOCAL_DIR/cache" "$CONDA_PREFIX/bin/uv" pip install \
-e ".[dreamverse]" --no-build-isolation
echo "==> Installing flashinfer-python (pinned, must be last)"
UV_CACHE_DIR="$LOCAL_DIR/cache" "$CONDA_PREFIX/bin/uv" pip install \
https://github.com/flashinfer-ai/flashinfer/releases/download/v0.6.11.post3/flashinfer_python-0.6.11.post3-py3-none-any.whl
echo ""
echo "Done. On a compute node run (GPUs 0-3 by default; set CUDA_VISIBLE_DEVICES to restrict):"
echo " bash apps/dreamverse/scripts/launch-dreamverse.sh"
@@ -110,7 +110,7 @@ default_request:
fps: 24 # internal: gpu_pool.py:85 TARGET_FPS
streaming:
# internal: config.py:33 SESSION_TIMEOUT_SECONDS = 300
# internal: config.py SESSION_TIMEOUT_SECONDS (env DREAMVERSE_SESSION_TIMEOUT_SECONDS, default 300)
session_timeout_seconds: 300
# internal: config.py:282-284 GENERATION_SEGMENT_CAP default 6
generation_segment_cap: 6
+125 -13
View File
@@ -9,7 +9,7 @@ import SessionTimeoutModal from "@/components/SessionTimeoutModal";
import Sidebar from "@/components/Sidebar";
import Header from "@/components/Header";
import VideoPlayer from "@/components/VideoPlayer";
import Workspace from "@/components/Workspace";
import Workspace, { SceneHistoryList } from "@/components/Workspace";
import { saveProject, saveProjectMetadata, listProjects, loadProjectClips, deleteProject, pruneOldProjects, type StoredProject, type StoredClip } from "@/lib/projectStorage";
import { isInfrastructureError } from "@/lib/ws/reducer";
import { useStore } from "@/hooks/useStore";
@@ -31,7 +31,7 @@ import { applyNormalizedSocketEvent } from "@/lib/ws/reducer";
import { createPromptWindowStore } from "@/stores/promptWindow";
import { createRewriteStore } from "@/stores/rewrite";
import { createSessionStore } from "@/stores/session";
import { createStreamStore } from "@/stores/stream";
import { createStreamStore, USER_PROMPT_SOURCES } from "@/stores/stream";
import { createUiStore } from "@/stores/ui";
import { Button } from "@/components/ui/button";
@@ -80,7 +80,7 @@ function yieldToEventLoop(): Promise<void> {
const HERO_WAVE_LIGHT = ["#2A4A98", "#4878E5", "#6FA0F2", "#B0BCC8", "#E8D99E", "#D8C844", "#C2A620"];
const HERO_WAVE_DARK = ["#143468", "#1E58B8", "#3892F0", "#80B8E8", "#B8D0EA", "#E2D498", "#DABB50"];
const HERO_TEXT = "Direct scenes in seconds";
const HERO_TEXT = "Direct scenes in seconds with";
function HeroTagline() {
const ref = useRef<HTMLHeadingElement>(null);
@@ -166,6 +166,8 @@ function HeroTagline() {
</span>
</Fragment>
))}
<span data-char className="transition-[color,filter] duration-150">{" "}</span>
<img src="/logo.svg" alt="FastVideo" className="inline-block h-[1.1em] w-auto align-middle" />
</h1>
);
}
@@ -237,6 +239,7 @@ export default function Page() {
enhancementEnabled,
promptExtensionError,
autoExtensionEnabled,
manualContinuationMode,
autoExtensionTimeoutHint,
loopGenerationEnabled,
generationPaused,
@@ -249,6 +252,8 @@ export default function Page() {
livePromptRewriteMode,
sessionExpired,
projectResetPending,
waitingForSegmentPrompt,
generatingNextScene,
} = sessionState;
const {
@@ -323,6 +328,11 @@ export default function Page() {
const [ttffValueMs, setTtffValueMs] = useState<number | null>(null);
const ttffIntervalRef = useRef<ReturnType<typeof setInterval> | null>(null);
const pendingInitialPromptRef = useRef("");
// Prompt id the opening scene is recorded under; sent as initial_rollout_prompt_id
// so the backend's pre-seeded opening PromptSubmission emits status updates
// (prompt_enhancing/prompt_ready/prompt_fallback_used) against the same id.
const pendingInitialPromptIdRef = useRef("");
const [initialImageDataUrl, setInitialImageDataUrl] = useState("");
const lastArchivedReplayKeyRef = useRef("");
const [sidebarOpen, setSidebarOpen] = useState(false);
const [currentThumbnail, setCurrentThumbnail] = useState<string | null>(null);
@@ -422,6 +432,11 @@ export default function Page() {
if (String(e?.source || "") === "user_rewrite" && typeof e?.text === "string" && e.text.trim()) {
return e.text.trim();
}
// Steering opening: the backend overwrites text/source with the enhanced
// prompt once ready, so fall back to the stable rawText record.
if (e?.steeringUserPrompt && typeof e?.rawText === "string" && e.rawText.trim()) {
return e.rawText.trim();
}
}
return "Untitled project";
}, [selectedPreset, promptEvents]);
@@ -432,11 +447,39 @@ export default function Page() {
const canDownloadVideo = useMemo(() => {
const currentActiveClip = activeClip as Record<string, any> | null;
if (currentActiveClip?.blob instanceof Blob) return true;
return (completedClips as Record<string, any>[]).some((clip) => clip?.blob instanceof Blob);
}, [activeClip, completedClips]);
if ((completedClips as Record<string, any>[]).some((clip) => clip?.blob instanceof Blob)) return true;
// Steering only: once playback has started the live AV pipeline holds playable segments, so
// the user can download the in-progress video at any time (handleDownloadVideo remuxes live
// segments). Auto mode keeps its original blob-gated behavior.
return Boolean(manualContinuationMode) && Boolean(avPlaybackStarted);
}, [activeClip, completedClips, avPlaybackStarted, manualContinuationMode]);
// Steering mode scene list (oldest first). Primary source is the user's own words, captured
// stably at submit time as `rawText` (the backend later overwrites text/source with the
// enhanced prompt, so we never read those). A segment with no user prompt — e.g. a preset's
// opening scene — falls back to promptHistory (the actual prompt that drove that segment).
const steeringScenes = useMemo(() => {
if (!manualContinuationMode) return [] as Record<string, any>[];
const userScenes = (promptEvents as Record<string, any>[])
.filter((e) => e?.steeringUserPrompt && !e?.steeringFailed && typeof e?.rawText === "string" && e.rawText.trim())
.slice()
.reverse() // oldest -> newest
.map((e) => ({ id: e.promptId, prompt: e.rawText as string }));
const scenes: Record<string, any>[] = [];
// Preset opening segments: curated seeds with no user prompt of their own.
const curatedHists = (promptHistory as Record<string, any>[])
.slice()
.reverse() // oldest first
.filter((h) => !USER_PROMPT_SOURCES.has(String(h?.source || "")) && typeof h?.prompt === "string" && (h.prompt as string).trim());
scenes.push(...curatedHists.map((h) => ({ id: h.id || "scene_open", prompt: h.prompt })));
scenes.push(...userScenes);
return scenes;
}, [manualContinuationMode, promptEvents, promptHistory]);
const hasEdits = useMemo(
() => Boolean(sessionStarted) && (promptEvents as Record<string, any>[]).some((e) => typeof e?.text === "string" && e.text.trim() && String(e?.source || "").trim() === "user_rewrite"),
() => Boolean(sessionStarted) && (
(promptEvents as Record<string, any>[]).some((e) => typeof e?.text === "string" && e.text.trim() && String(e?.source || "").trim() === "user_rewrite")
),
[sessionStarted, promptEvents],
);
@@ -1421,7 +1464,8 @@ export default function Page() {
if (!prompt) return;
lastSubmitTimeRef.current = now;
const rCAR = !uiStore.get().devtoolsMode && !uiStore.get().demoMode;
if (rCAR || (sessionStore.get().livePromptRewriteMode && !uiStore.get().demoMode)) {
const inManualMode = sessionStore.get().manualContinuationMode;
if (!inManualMode && (rCAR || (sessionStore.get().livePromptRewriteMode && !uiStore.get().demoMode))) {
if (rewriteStore.get().rewritingSeedPrompts) return;
const rewriteSourcePromptWindowPrompts = getActivePromptWindowPrompts();
const nextPendingClip = {
@@ -1462,6 +1506,11 @@ export default function Page() {
status: "submitted",
source: "user_raw",
text: prompt,
// Stable record of the user's own words for the steering scene list. The backend
// later overwrites `text`/`source` with the enhanced prompt via prompt/ready, but
// these two fields are never touched by trackPromptEvent.
steeringUserPrompt: true,
rawText: prompt,
});
ws.send(
JSON.stringify({
@@ -1475,7 +1524,14 @@ export default function Page() {
activeClipId: shouldUseArchivedPlaybackFallback() ? streamStore.get().activeClipId : "",
activePlaybackStartTime: shouldUseArchivedPlaybackFallback() ? streamStore.get().activePlaybackStartTime : 0,
});
sessionStore.patch({ livePromptDraft: "" });
sessionStore.patch({
livePromptDraft: "",
waitingForSegmentPrompt: false,
sessionNotice: "",
// Light the "Generating next scene" overlay immediately on a real submit;
// stream/media_init (or a fallback/error) clears it.
...(inManualMode ? { generatingNextScene: true } : {}),
});
}
function setLivePromptRewriteMode(enabled: boolean) {
@@ -1550,6 +1606,14 @@ export default function Page() {
);
}
// Steering (manual continuation) vs the automatic 6-segment rollout — a pre-session
// preference honored when the session starts.
function handleManualContinuationToggle(event: any) {
sessionStore.patch({
manualContinuationMode: Boolean(event.currentTarget.checked),
});
}
function handleLoopGenerationToggle(event: any) {
sessionStore.patch({
loopGenerationEnabled: Boolean(event.currentTarget.checked),
@@ -1725,6 +1789,9 @@ export default function Page() {
sessionNotice: preserveSessionNotice ? sessionStore.get().sessionNotice : "",
sessionExpired: preserveSessionNotice ? sessionStore.get().sessionExpired : false,
projectResetPending: false,
manualContinuationMode: true,
waitingForSegmentPrompt: false,
generatingNextScene: false,
});
rewriteStore.resetSessionState();
streamStore.resetSessionState();
@@ -1737,6 +1804,7 @@ export default function Page() {
function resetToProjectLobbyState() {
setVideoMuted(true);
pendingInitialPromptRef.current = "";
pendingInitialPromptIdRef.current = "";
sessionStore.patch({
sessionStarted: false,
livePromptDraft: "",
@@ -1750,6 +1818,9 @@ export default function Page() {
sessionNotice: "",
sessionExpired: false,
projectResetPending: false,
manualContinuationMode: true,
waitingForSegmentPrompt: false,
generatingNextScene: false,
});
rewriteStore.resetSessionState();
streamStore.resetSessionState();
@@ -1760,7 +1831,14 @@ export default function Page() {
}
function buildProjectInitPayload(type: "session_init_v2" | "project_init_v1") {
const segmentPrompts = getSessionInitPrompts();
const manualMode = Boolean(sessionStore.get().manualContinuationMode);
let segmentPrompts = getSessionInitPrompts();
// Steering mode: seed the first 2 segments from the preset so there's no
// gap between segment 1 and 2; the user drives every subsequent segment.
// Force auto/loop off so the backend waits after the seeded prompts run out.
if (manualMode) {
segmentPrompts = segmentPrompts.slice(0, 2);
}
setSeedPrompts(segmentPrompts);
return {
type,
@@ -1768,11 +1846,17 @@ export default function Page() {
preset_label: getInitialPresetLabel(),
curated_prompts: segmentPrompts,
initial_rollout_prompt: normalizeInitialPrompt(pendingInitialPromptRef.current),
initial_image: null,
// Ties the backend's pre-seeded opening PromptSubmission to the prompt event
// recorded in beginProjectLocally so its status updates land on that record.
initial_rollout_prompt_id: pendingInitialPromptIdRef.current,
initial_image: initialImageDataUrl
? { data_url: initialImageDataUrl, mime_type: initialImageDataUrl.split(";")[0].split(":")[1] || "image/png", name: "upload.png" }
: null,
single_clip_mode: false,
enhancement_enabled: sessionStore.get().enhancementEnabled,
auto_extension_enabled: sessionStore.get().autoExtensionEnabled,
loop_generation_enabled: sessionStore.get().loopGenerationEnabled,
auto_extension_enabled: manualMode ? false : sessionStore.get().autoExtensionEnabled,
loop_generation_enabled: manualMode ? false : sessionStore.get().loopGenerationEnabled,
manual_continuation_mode: manualMode,
};
}
@@ -1951,7 +2035,11 @@ export default function Page() {
setCurrentThumbnail(null);
const initialPrompt = normalizeInitialPrompt(sessionStore.get().livePromptDraft as string);
pendingInitialPromptRef.current = initialPrompt;
setInitialImageDataUrl("");
const rCAR = !uiStore.get().devtoolsMode && !uiStore.get().demoMode;
// The "Steering mode" toggle is authoritative: checked = manual per-segment steering,
// unchecked = automatic 6-segment rollout (default).
const nextManualContinuationMode = Boolean(sessionStore.get().manualContinuationMode);
sessionStore.patch({
sessionNotice: "",
sessionExpired: false,
@@ -1966,6 +2054,9 @@ export default function Page() {
autoExtensionTimeoutHint: "",
generationPaused: false,
projectResetPending: false,
manualContinuationMode: nextManualContinuationMode,
waitingForSegmentPrompt: false,
generatingNextScene: false,
});
resetPlaybackState();
streamStore.patch({
@@ -1979,12 +2070,15 @@ export default function Page() {
selectedHistoryId: "",
});
rewriteStore.resetSessionState();
pendingInitialPromptIdRef.current = initialPrompt ? makePromptId() : "";
if (initialPrompt) {
addPromptEvent({
promptId: makePromptId(),
promptId: pendingInitialPromptIdRef.current,
status: "rewrite_requested",
source: "user_rewrite",
text: initialPrompt,
// In steering mode the typed opening is the user's Scene 1 — record it stably.
...(nextManualContinuationMode ? { steeringUserPrompt: true, rawText: initialPrompt } : {}),
});
}
setSeedPrompts(getSessionInitPrompts());
@@ -2500,6 +2594,7 @@ export default function Page() {
selectedPresetId={selectedPresetId as string}
enhancementEnabled={enhancementEnabled as boolean}
autoExtensionEnabled={autoExtensionEnabled as boolean}
manualContinuationEnabled={manualContinuationMode as boolean}
loopGenerationEnabled={loopGenerationEnabled as boolean}
canJoinSession={canJoinSession as boolean}
canSubmitContinuation={canSubmitContinuation}
@@ -2512,6 +2607,7 @@ export default function Page() {
onEnhancementToggle={handleEnhancementToggle}
onCuratedPromptLimitChange={handleCuratedPromptLimitChange}
onAutoExtensionToggle={handleAutoExtensionToggle}
onManualContinuationToggle={handleManualContinuationToggle}
onLoopToggle={handleLoopGenerationToggle}
onJoin={joinSession}
onLeave={leaveSession}
@@ -2641,6 +2737,7 @@ export default function Page() {
/>
<Header timeLeft={headerTimeLeft} formatTime={formatTime} onToggleSidebar={() => setSidebarOpen((prev) => !prev)} />
<div className={cn("flex flex-1 min-h-0 flex-col", sessionStarted && "pb-16 sm:pb-28")}>
<div className="relative flex flex-1 min-h-0 flex-col justify-center px-4 pb-2 sm:px-6 sm:pb-12">
{isViewingMode && (
<>
@@ -2724,6 +2821,8 @@ export default function Page() {
showLivePlayback={showLivePlayback}
defaultMuted={videoMuted}
canDownload={canDownloadVideo}
waitingForSegmentPrompt={waitingForSegmentPrompt as boolean}
generatingNextScene={generatingNextScene as boolean}
onPlaying={markFirstFrameRendered}
onDownload={handleDownloadVideo}
/>
@@ -2734,6 +2833,7 @@ export default function Page() {
<section className={cn("mx-auto w-full max-w-2xl", hasEdits && "flex-1 min-h-0 overflow-y-auto")}>
<Workspace
promptEvents={promptEvents as any[]}
manualMode={manualContinuationMode as boolean}
currentThumbnail={currentThumbnail}
originalLabel={pendingInitialPromptRef.current || (selectedPreset as Record<string, any>)?.label || ""}
sessionStarted={sessionStarted as boolean}
@@ -2780,11 +2880,17 @@ export default function Page() {
isGenerating={loadingAnimation as boolean}
storyPresets={storyPresets as any[]}
continuationDraft={livePromptDraft as string}
manualContinuationEnabled={manualContinuationMode as boolean}
onModeChange={(manual) => sessionStore.patch({ manualContinuationMode: manual })}
initialImageDataUrl={initialImageDataUrl}
onImageUpload={(dataUrl) => setInitialImageDataUrl(dataUrl)}
onImageClear={() => setInitialImageDataUrl("")}
canJoinSession={canStartSession}
canSubmitContinuation={canSubmitContinuation}
sessionExpired={sessionExpired as boolean}
sessionNotice={sessionNotice as string}
projectResetPending={projectResetPending as boolean}
waitingForSegmentPrompt={waitingForSegmentPrompt as boolean}
onPresetGenerate={handlePresetGenerate}
onContinuationInput={handleLivePromptInput}
onContinuationKeydown={handleLivePromptKeydown}
@@ -2798,6 +2904,12 @@ export default function Page() {
</motion.div>
</div>
</div>
{manualContinuationMode && (
<div className="px-4 sm:px-6">
<SceneHistoryList sceneHistory={steeringScenes as any[]} />
</div>
)}
</div>
</main>
);
}
+96 -6
View File
@@ -2,13 +2,16 @@
import React, { useRef, useState, useCallback, useEffect } from "react";
import Image from "next/image";
import { Film, ArrowUp, X, Loader2, ArrowLeft } from "lucide-react";
import { Film, ArrowUp, X, Loader2, ArrowLeft, ImagePlus } from "lucide-react";
import { Button } from "@/components/ui/button";
import LeaveSessionModal, { shouldShowLeaveWarning } from "@/components/LeaveSessionModal";
import SpeechToTextButton from "@/components/SpeechToTextButton";
import { cn } from "@/lib/utils";
const PROMPT_MAX_LENGTH = 500;
// Must match backend session_init_image.py: MAX_SESSION_INIT_IMAGE_BYTES / SUPPORTED_SESSION_INIT_IMAGE_MIME_TYPES.
const IMAGE_MAX_BYTES = 15 * 1024 * 1024;
const IMAGE_ALLOWED_TYPES = ["image/png", "image/jpeg", "image/webp"];
interface Props {
sessionStarted?: boolean;
@@ -22,6 +25,12 @@ interface Props {
sessionNotice?: string;
projectResetPending?: boolean;
viewingReadOnly?: boolean;
waitingForSegmentPrompt?: boolean;
manualContinuationEnabled?: boolean;
onModeChange?: (manual: boolean) => void;
initialImageDataUrl?: string;
onImageUpload?: (dataUrl: string, mimeType: string, name: string) => void;
onImageClear?: () => void;
onPresetGenerate?: (presetId: string) => void;
onContinuationInput?: (e: React.ChangeEvent<HTMLTextAreaElement>) => void;
onContinuationKeydown?: (e: React.KeyboardEvent<HTMLTextAreaElement>) => void;
@@ -46,6 +55,12 @@ export default function ChatBar({
sessionNotice = "",
projectResetPending = false,
viewingReadOnly = false,
waitingForSegmentPrompt = false,
manualContinuationEnabled = false,
onModeChange = () => {},
initialImageDataUrl = "",
onImageUpload = () => {},
onImageClear = () => {},
onPresetGenerate = () => {},
onContinuationInput = () => {},
onContinuationKeydown = () => {},
@@ -59,15 +74,51 @@ export default function ChatBar({
}: Props) {
const [sttBusy, setSttBusy] = useState(false);
const [leaveModalOpen, setLeaveModalOpen] = useState(false);
const [imageError, setImageError] = useState("");
const fileInputRef = useRef<HTMLInputElement>(null);
const processImageFile = useCallback((file: File) => {
if (!IMAGE_ALLOWED_TYPES.includes(file.type)) {
setImageError("Unsupported image type. Use a PNG, JPEG, or WebP image.");
return;
}
if (file.size > IMAGE_MAX_BYTES) {
setImageError("Image is too large. The maximum size is 15MB.");
return;
}
const reader = new FileReader();
reader.onload = (e) => {
const dataUrl = e.target?.result as string;
if (dataUrl) {
setImageError("");
onImageUpload(dataUrl, file.type, file.name);
}
};
reader.onerror = () => {
setImageError("Could not read the image file. Please try again.");
};
reader.readAsDataURL(file);
}, [onImageUpload]);
const handleImagePaste = useCallback((e: React.ClipboardEvent) => {
if (sessionStarted) return;
const items = Array.from(e.clipboardData?.items ?? []);
const imageItem = items.find((item) => item.type.startsWith("image/"));
if (!imageItem) return;
const file = imageItem.getAsFile();
if (file) processImageFile(file);
}, [sessionStarted, processImageFile]);
const showSpinner = isGenerating || rewritingSeedPrompts;
const isBusy = isGenerating || rewritingSeedPrompts || projectResetPending;
const messagePlaceholder = projectResetPending
? "Starting new project\u2026"
: isBusy
? "Generating video\u2026"
: !sessionStarted
? "What video are you imagining?"
: "What do you want to edit?";
: waitingForSegmentPrompt
? "Describe the next scene\u2026"
: !sessionStarted
? "What video are you imagining?"
: "What do you want to edit?";
const actionLabel = !sessionStarted ? "Generate" : "Rewrite rollout";
const inputRef = useRef<HTMLTextAreaElement>(null);
@@ -267,9 +318,9 @@ export default function ChatBar({
<Button onClick={onStartNewProject} size="sm" className="rounded-full px-5">
New Project
</Button>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSe5zpO1iD8Ds-Ih-fOLm64qd7YZVvuvAyHuJaAfw1hkRHTe_A/viewform?usp=publish-editor" target="_blank" rel="noopener noreferrer">
<a href="https://haoailab.com/blogs/dreamverse/" target="_blank" rel="noopener noreferrer">
<Button variant="outline" size="sm" className="rounded-full px-5">
Join Waitlist
Blog
</Button>
</a>
</div>
@@ -346,6 +397,31 @@ export default function ChatBar({
</div>
)}
{!sessionStarted && imageError && (
<div className="rounded-xl border border-rose-500/20 bg-rose-500/10 px-4 py-2.5 text-center text-xs text-rose-700 dark:text-rose-300">
{imageError}
</div>
)}
{!sessionStarted && initialImageDataUrl && (
<div className="flex items-center gap-2 rounded-2xl border border-input bg-card/65 px-3 py-2">
<img src={initialImageDataUrl} alt="Initial frame" className="h-12 w-12 rounded-lg object-cover" />
<span className="flex-1 truncate text-xs text-muted-foreground">Starting image set</span>
<button type="button" onClick={() => { setImageError(""); onImageClear(); }} className="text-muted-foreground hover:text-foreground transition-colors">
<X className="size-4" />
</button>
</div>
)}
<input
ref={fileInputRef}
type="file"
accept={IMAGE_ALLOWED_TYPES.join(",")}
className="hidden"
onChange={(e) => { const f = e.target.files?.[0]; if (f) processImageFile(f); e.target.value = ""; }}
/>
<div
className={cn(
"flex min-w-0 items-center gap-1.5 rounded-4xl border py-2.5 pl-5 pr-2.5 shadow-md backdrop-blur-sm transition-all duration-200",
@@ -359,6 +435,7 @@ export default function ChatBar({
value={continuationDraft}
onChange={onContinuationInput}
onKeyDown={handleKeyDown}
onPaste={handleImagePaste}
placeholder={sttBusy ? "Listening\u2026" : messagePlaceholder}
maxLength={PROMPT_MAX_LENGTH}
disabled={isBusy || sttBusy}
@@ -369,6 +446,19 @@ export default function ChatBar({
)}
/>
{onSpeechTranscript && <SpeechToTextButton disabled={isBusy} onTranscript={onSpeechTranscript} onInterimChange={onSpeechInterimChange} onBusyChange={setSttBusy} />}
{!sessionStarted && (
<Button
type="button"
variant="ghost"
size="icon-sm"
title="Add starting image"
onClick={() => fileInputRef.current?.click()}
disabled={isBusy}
className="shrink-0 rounded-full text-muted-foreground hover:text-foreground"
>
<ImagePlus className="size-4" />
</Button>
)}
{!sessionStarted ? (
<Button
aria-label={actionLabel}
@@ -8,7 +8,6 @@ import { Badge } from "@/components/ui/badge";
import { Button } from "@/components/ui/button";
import { ThemeToggle } from "@/components/ui/theme-toggle";
const FASTVIDEO_REPO_URL = "https://haoailab.com/blogs/dreamverse/";
const FASTVIDEO_BLOG_URL = "https://haoailab.com/blogs/dreamverse/";
interface Props {
@@ -29,13 +28,13 @@ export default function Header({ timeLeft = null, formatTime = (seconds) => `${s
<SidePanelOpenFilled size={20} />
</Button>
)}
<a href={FASTVIDEO_REPO_URL} target="_blank" rel="noopener noreferrer" title="FastVideo on GitHub">
<a href="/" title="FastVideo home">
<Image src="/logo.svg" alt="FastVideo" width={32} height={32} className="h-8 w-auto sm:h-9 transition-opacity hover:opacity-70" />
</a>
<div className="hidden sm:flex items-center gap-3">
<a href="https://docs.google.com/forms/d/e/1FAIpQLSe5zpO1iD8Ds-Ih-fOLm64qd7YZVvuvAyHuJaAfw1hkRHTe_A/viewform?usp=publish-editor" target="_blank" rel="noopener noreferrer">
<a href={FASTVIDEO_BLOG_URL} target="_blank" rel="noopener noreferrer">
<Button variant="outline" size="sm" className="gap-1.5 rounded-full px-3 text-xs">
Join Waitlist
Blog
<ExternalLink className="size-3 opacity-60" />
</Button>
</a>
@@ -53,9 +52,9 @@ export default function Header({ timeLeft = null, formatTime = (seconds) => `${s
</div>
<div className="flex sm:hidden items-center gap-2 px-4 pb-3">
<a href="https://docs.google.com/forms/d/e/1FAIpQLSe5zpO1iD8Ds-Ih-fOLm64qd7YZVvuvAyHuJaAfw1hkRHTe_A/viewform?usp=publish-editor" target="_blank" rel="noopener noreferrer">
<a href={FASTVIDEO_BLOG_URL} target="_blank" rel="noopener noreferrer">
<Button variant="outline" size="sm" className="gap-1.5 rounded-full px-3 text-xs">
Join Waitlist
Blog
<ExternalLink className="size-3 opacity-60" />
</Button>
</a>
@@ -3,7 +3,7 @@
import React, { useState, useEffect, useRef, useCallback } from "react";
import { cn } from "@/lib/utils";
import { PlayFilledAlt } from "@carbon/icons-react";
import { Download, Loader2, Share } from "lucide-react";
import { Check, ChevronDown, Download, Loader2, Share } from "lucide-react";
import { Button } from "@/components/ui/button";
interface VideoPlayerProps {
videoRef?: React.RefCallback<HTMLVideoElement>;
@@ -21,6 +21,8 @@ interface VideoPlayerProps {
showLivePlayback?: boolean;
defaultMuted?: boolean;
rewritePending?: boolean;
waitingForSegmentPrompt?: boolean;
generatingNextScene?: boolean;
onPlaying?: () => void;
onDownload?: () => void;
}
@@ -57,6 +59,8 @@ export default function VideoPlayer({
showLivePlayback = true,
defaultMuted = true,
rewritePending = false,
waitingForSegmentPrompt = false,
generatingNextScene = false,
onPlaying = () => {},
onDownload,
}: VideoPlayerProps) {
@@ -88,6 +92,137 @@ export default function VideoPlayer({
setCanShare(typeof navigator.canShare === "function" && window.matchMedia("(pointer: coarse)").matches);
}, []);
// Steering mode: the backend may signal "waiting for next prompt" while the current
// segment is still PLAYING (it generates ahead). Only surface the "Segment complete"
// overlay once the playhead actually reaches the end of the buffered segment.
const [playbackReachedEnd, setPlaybackReachedEnd] = useState(false);
// Steering mode: when the user submits the next scene, the segment is generated
// (a few seconds of latency) before frames stream. Show a "Generating next scene…"
// indicator across that gap so the frozen frame isn't silent. Driven by the explicit
// generatingNextScene state (set on scene submit / prompt selection), never inferred
// from waitingForSegmentPrompt edges.
const [generatingNext, setGeneratingNext] = useState(false);
// End of the buffered timeline captured the moment generation starts; the freshly
// generated segment extends the buffer past this, which is how we know it landed.
const genBoundaryRef = useRef(0);
// Steering mode: the backend may signal "waiting for next prompt" while the current
// segment is still PLAYING (it generates ahead). Track whether the playhead has reached
// the end of the buffered segment so end-overlays only show there. Keep tracking through
// the generating phase too, so scrubbing back and replaying to the end re-shows them.
useEffect(() => {
if (!waitingForSegmentPrompt && !generatingNext) {
setPlaybackReachedEnd(false);
return;
}
const el = liveVideoEl.current;
if (!el) return;
const check = () => {
try {
const buffered = el.buffered;
if (buffered.length === 0) return;
const end = buffered.end(buffered.length - 1);
// Track proximity both ways: scrubbing back off the end hides the overlay,
// playing forward to the end re-shows it.
setPlaybackReachedEnd(el.ended || end - el.currentTime <= 0.2);
} catch {
/* buffered access can throw mid-append */
}
};
check();
el.addEventListener("timeupdate", check);
el.addEventListener("ended", check);
el.addEventListener("waiting", check);
el.addEventListener("stalled", check);
el.addEventListener("pause", check);
el.addEventListener("seeking", check);
el.addEventListener("seeked", check);
el.addEventListener("playing", check);
el.addEventListener("progress", check);
return () => {
el.removeEventListener("timeupdate", check);
el.removeEventListener("ended", check);
el.removeEventListener("waiting", check);
el.removeEventListener("stalled", check);
el.removeEventListener("pause", check);
el.removeEventListener("seeking", check);
el.removeEventListener("seeked", check);
el.removeEventListener("playing", check);
el.removeEventListener("progress", check);
};
}, [waitingForSegmentPrompt, generatingNext]);
useEffect(() => {
if (waitingForSegmentPrompt || !sessionStarted) {
// Back to waiting (or session over): nothing is generating.
setGeneratingNext(false);
return;
}
if (!generatingNextScene) return;
// Snapshot the current end of the buffered timeline; the generated segment will
// extend the buffer past this boundary.
const el = liveVideoEl.current;
let boundary = el?.currentTime ?? 0;
try {
const b = el?.buffered;
if (b && b.length) boundary = Math.max(boundary, b.end(b.length - 1));
} catch {
/* buffered access can throw mid-append */
}
genBoundaryRef.current = boundary;
setGeneratingNext(true);
}, [generatingNextScene, waitingForSegmentPrompt, sessionStarted]);
useEffect(() => {
if (!generatingNext) return;
if (!sessionStarted) {
setGeneratingNext(false);
return;
}
const el = liveVideoEl.current;
if (!el) return;
// Clear the instant the freshly generated segment lands: the buffer grows past the
// boundary captured at generation start (or the playhead advances into the new
// frames). Deliberately NOT a bare "playing" handler — scrubbing back and replaying
// the EXISTING segment must keep "Generating" up until the new frames actually arrive.
const check = () => {
try {
const b = el.buffered;
const end = b.length ? b.end(b.length - 1) : 0;
if (end > genBoundaryRef.current + 0.25 || el.currentTime > genBoundaryRef.current + 0.1) {
setGeneratingNext(false);
}
} catch {
/* buffered access can throw mid-append */
}
};
check();
el.addEventListener("progress", check);
el.addEventListener("timeupdate", check);
el.addEventListener("durationchange", check);
return () => {
el.removeEventListener("progress", check);
el.removeEventListener("timeupdate", check);
el.removeEventListener("durationchange", check);
};
}, [generatingNext, sessionStarted]);
// Drive a ~10.5s progress bar during generation so the wait has a visible ETA.
const GEN_DURATION_MS = 10500;
const [genProgress, setGenProgress] = useState(0);
useEffect(() => {
if (!generatingNext) {
setGenProgress(0);
return;
}
const start = performance.now();
setGenProgress(0);
const id = setInterval(() => {
setGenProgress(Math.min((performance.now() - start) / GEN_DURATION_MS, 1));
}, 50);
return () => clearInterval(id);
}, [generatingNext]);
return (
<div className="mx-auto w-full max-w-3xl mb-2 sm:mb-6">
<div className="rounded-2xl border border-border bg-card/50 p-2 shadow-lg backdrop-blur-md">
@@ -104,7 +239,7 @@ export default function VideoPlayer({
<PlayFilledAlt className="size-10 text-white/25" />
<p className="text-sm text-white/50">Your video will appear here</p>
</div>
) : !avPlaybackStarted && !mediaAppendError && !inQueue && loadingAnimation ? (
) : !avPlaybackStarted && !mediaAppendError && !inQueue && !waitingForSegmentPrompt && loadingAnimation ? (
<div className="absolute inset-0 flex flex-col items-center justify-center gap-4 bg-slate-900/60 p-4 backdrop-blur-[2px]">
<div className="pointer-events-none absolute inset-0 overflow-hidden">
<div className="absolute inset-0 -translate-x-full animate-[shimmer_3s_ease-in-out_infinite] bg-gradient-to-r from-transparent via-white/[0.04] to-transparent" />
@@ -114,6 +249,35 @@ export default function VideoPlayer({
</div>
) : null}
{/* Steering mode: this segment finished — wait gracefully for the user's next scene
instead of spinning. The last frame stays visible behind a soft bottom gradient. */}
{sessionStarted && waitingForSegmentPrompt && playbackReachedEnd && !mediaAppendError && !inQueue && (
<div className="pointer-events-none absolute inset-0 flex flex-col items-center justify-end gap-2 bg-gradient-to-t from-slate-950/85 via-slate-950/15 to-transparent p-5 pb-6 text-center">
<div className="flex size-9 items-center justify-center rounded-full border border-white/25 bg-white/10 shadow-lg backdrop-blur-md">
<Check className="size-4 text-white/90" />
</div>
<div className="space-y-0.5">
<p className="text-sm font-medium text-white/95">Segment complete</p>
<p className="text-xs text-white/65">Describe the next scene below to keep going</p>
</div>
<ChevronDown className="size-4 animate-bounce text-white/45" />
</div>
)}
{/* Steering mode: generating the next segment — show a ~10.5s progress bar so the wait has an ETA.
Gated on playbackReachedEnd like "Segment complete": scrubbing back hides it, playing to the end re-shows it. */}
{sessionStarted && generatingNext && playbackReachedEnd && !mediaAppendError && !inQueue && (
<div className="pointer-events-none absolute inset-0 flex flex-col items-center justify-end gap-3 bg-gradient-to-t from-slate-950/85 via-slate-950/15 to-transparent p-5 pb-7 text-center">
<p className="text-sm font-medium text-white/95">Generating next scene&hellip;</p>
<div className="h-1.5 w-48 overflow-hidden rounded-full bg-white/15 shadow-sm">
<div
className="h-full rounded-full bg-white/85 transition-[width] duration-100 ease-linear"
style={{ width: `${Math.round(genProgress * 100)}%` }}
/>
</div>
</div>
)}
{rewritePending && avPlaybackStarted && (
<div className="absolute inset-x-0 bottom-0 z-10 flex items-center justify-center gap-2 bg-gradient-to-t from-black/60 to-transparent px-4 pb-12 pt-8 pointer-events-none">
<Loader2 className="size-4 animate-spin text-white/90" />
@@ -3,11 +3,90 @@ import React, { useRef, useMemo, useEffect, useCallback, useState } from "react"
import { motion, useAnimationControls } from "framer-motion";
import { Badge } from "@/components/ui/badge";
import { cn } from "@/lib/utils";
import { Check, Lightbulb, Pencil } from "lucide-react";
import { Check, Clapperboard, Lightbulb, Pencil } from "lucide-react";
export const WORKSPACE_ORIGINAL_SELECTION_KEY = "original";
export const WORKSPACE_CURRENT_SELECTION_KEY = "current";
export function SceneHistoryList({ sceneHistory = [] }: { sceneHistory?: Record<string, any>[] }) {
const bottomSentinelRef = useRef<HTMLDivElement>(null);
const topSentinelRef = useRef<HTMLDivElement>(null);
const [showTopFade, setShowTopFade] = useState(false);
const scenes = useMemo(
() => (sceneHistory || []).filter((s) => normalizeText(s?.prompt)),
[sceneHistory],
);
const scrollToBottom = useCallback(() => {
setTimeout(() => {
bottomSentinelRef.current?.scrollIntoView({ block: "end", behavior: "smooth" });
}, 60);
}, []);
useEffect(() => {
if (scenes.length > 0) scrollToBottom();
}, [scenes.length, scrollToBottom]);
useEffect(() => {
const sentinel = bottomSentinelRef.current;
if (!sentinel || typeof ResizeObserver === "undefined") return;
let container: HTMLElement | null = sentinel.parentElement;
while (container) {
const oy = getComputedStyle(container).overflowY;
if (oy === "auto" || oy === "scroll") break;
container = container.parentElement;
}
if (!container) return;
const ro = new ResizeObserver(() => {
const nearBottom = container!.scrollHeight - container!.scrollTop - container!.clientHeight < 96;
if (nearBottom) scrollToBottom();
});
ro.observe(container);
return () => ro.disconnect();
}, [scenes.length, scrollToBottom]);
useEffect(() => {
const el = topSentinelRef.current;
if (!el) return;
const observer = new IntersectionObserver(([entry]) => setShowTopFade(!entry.isIntersecting), { threshold: 0.1 });
observer.observe(el);
return () => observer.disconnect();
}, [scenes.length]);
if (scenes.length === 0) return null;
return (
<section className="relative z-10 flex flex-col mx-auto w-full max-w-2xl max-h-32 overflow-y-auto">
<div
className={cn(
"pointer-events-none sticky top-0 z-20 -mb-12 h-12 bg-linear-to-b from-background to-transparent transition-opacity duration-200",
showTopFade ? "opacity-100" : "opacity-0",
)}
aria-hidden="true"
/>
<div ref={topSentinelRef} className="h-0 w-0" aria-hidden="true" />
<div className="flex flex-col gap-2 pt-12 pb-4">
{scenes.map((scene, index) => (
<div
key={scene.id || index}
className="flex items-start gap-3 rounded-xl p-3 transition-colors duration-200 hover:bg-slate-200/50 hover:dark:bg-slate-800/30"
>
<div className="flex min-w-0 flex-1 flex-col gap-2">
<Badge variant="secondary" className="horizontal gap-2 items-center w-fit">
<Clapperboard className="size-3 opacity-70" />
{`Scene ${index + 1}`}
</Badge>
<p className="line-clamp-2 text-sm leading-5 text-muted-foreground">{scene.prompt}</p>
</div>
</div>
))}
</div>
<div ref={bottomSentinelRef} className="h-0 w-0" aria-hidden="true" />
</section>
);
}
interface WorkspaceProps {
promptEvents?: Record<string, any>[];
currentThumbnail?: string | null;
@@ -19,6 +98,7 @@ interface WorkspaceProps {
selectedClipId?: string;
selectedEntryKey?: string;
originalClipId?: string;
manualMode?: boolean;
}
function normalizeText(value: any): string {
@@ -218,7 +298,7 @@ function ChromaGradient({ sessionStarted = false }: { sessionStarted?: boolean }
);
}
export default function Workspace({ promptEvents = [], currentThumbnail = null, originalLabel = "", sessionStarted = false, onSelectOriginal, onSelectEvent, onSelectCurrent, selectedClipId, selectedEntryKey: selectedEntryKeyProp, originalClipId = "" }: WorkspaceProps) {
export default function Workspace({ promptEvents = [], currentThumbnail = null, originalLabel = "", sessionStarted = false, onSelectOriginal, onSelectEvent, onSelectCurrent, selectedClipId, selectedEntryKey: selectedEntryKeyProp, originalClipId = "", manualMode = false }: WorkspaceProps) {
const bottomSentinelRef = useRef<HTMLDivElement>(null);
const topSentinelRef = useRef<HTMLDivElement>(null);
const [showTopFade, setShowTopFade] = useState(false);
@@ -259,6 +339,14 @@ export default function Workspace({ promptEvents = [], currentThumbnail = null,
return () => observer.disconnect();
}, [conversationEvents.length]);
if (manualMode) {
return (
<div className="mt-auto flex flex-col">
<ChromaGradient sessionStarted={sessionStarted} />
</div>
);
}
return (
<div className="mt-auto flex flex-col">
<ChromaGradient sessionStarted={sessionStarted} />
@@ -34,6 +34,7 @@ interface DevtoolsComposerProps {
demoMode?: boolean;
enhancementEnabled?: boolean;
autoExtensionEnabled?: boolean;
manualContinuationEnabled?: boolean;
loopGenerationEnabled?: boolean;
curatedPromptLimit?: number;
maxCuratedPromptCount?: number;
@@ -49,6 +50,7 @@ interface DevtoolsComposerProps {
onEnhancementToggle?: (e: React.ChangeEvent<HTMLInputElement>) => void;
onCuratedPromptLimitChange?: (e: React.ChangeEvent<HTMLInputElement>) => void;
onAutoExtensionToggle?: (e: React.ChangeEvent<HTMLInputElement>) => void;
onManualContinuationToggle?: (e: React.ChangeEvent<HTMLInputElement>) => void;
onLoopToggle?: (e: React.ChangeEvent<HTMLInputElement>) => void;
onLivePromptModeToggle?: (e: React.ChangeEvent<HTMLInputElement>) => void;
onSpeechTranscript?: (text: string) => void;
@@ -71,6 +73,7 @@ export default function DevtoolsComposer({
demoMode = false,
enhancementEnabled = true,
autoExtensionEnabled = false,
manualContinuationEnabled = false,
loopGenerationEnabled = false,
curatedPromptLimit = 0,
maxCuratedPromptCount = 0,
@@ -86,6 +89,7 @@ export default function DevtoolsComposer({
onEnhancementToggle = () => {},
onCuratedPromptLimitChange = () => {},
onAutoExtensionToggle = () => {},
onManualContinuationToggle = () => {},
onLoopToggle = () => {},
onLivePromptModeToggle = () => {},
onSpeechTranscript,
@@ -328,6 +332,28 @@ export default function DevtoolsComposer({
</div>
</div>
<div className="flex items-start gap-3">
<Checkbox
id="devtools-steering-mode"
checked={manualContinuationEnabled}
onCheckedChange={(checked) =>
onManualContinuationToggle({
target: { checked: Boolean(checked) },
currentTarget: { checked: Boolean(checked) },
} as React.ChangeEvent<HTMLInputElement>)
}
/>
<div className="space-y-1">
<Label htmlFor="devtools-steering-mode">
Steering mode
</Label>
<p className="text-sm text-muted-foreground">
Drive each segment manually — type the next scene to
continue (vs the automatic 6-segment rollout).
</p>
</div>
</div>
<div className="flex items-start gap-3">
<Checkbox
id="devtools-loop-generation"
@@ -21,6 +21,7 @@ interface DevtoolsShellProps {
selectedPresetId?: string;
enhancementEnabled?: boolean;
autoExtensionEnabled?: boolean;
manualContinuationEnabled?: boolean;
loopGenerationEnabled?: boolean;
canJoinSession?: boolean;
canSubmitContinuation?: boolean;
@@ -34,6 +35,7 @@ interface DevtoolsShellProps {
onEnhancementToggle?: (e: React.ChangeEvent<HTMLInputElement>) => void;
onCuratedPromptLimitChange?: (e: React.ChangeEvent<HTMLInputElement>) => void;
onAutoExtensionToggle?: (e: React.ChangeEvent<HTMLInputElement>) => void;
onManualContinuationToggle?: (e: React.ChangeEvent<HTMLInputElement>) => void;
onLoopToggle?: (e: React.ChangeEvent<HTMLInputElement>) => void;
onJoin?: () => void;
onLeave?: () => void;
@@ -128,6 +130,7 @@ export default function DevtoolsShell({
selectedPresetId = '',
enhancementEnabled = true,
autoExtensionEnabled = false,
manualContinuationEnabled = false,
loopGenerationEnabled = false,
canJoinSession = false,
canSubmitContinuation = false,
@@ -141,6 +144,7 @@ export default function DevtoolsShell({
onEnhancementToggle = () => {},
onCuratedPromptLimitChange = () => {},
onAutoExtensionToggle = () => {},
onManualContinuationToggle = () => {},
onLoopToggle = () => {},
onJoin = () => {},
onLeave = () => {},
@@ -284,6 +288,7 @@ export default function DevtoolsShell({
demoMode={demoMode}
enhancementEnabled={enhancementEnabled}
autoExtensionEnabled={autoExtensionEnabled}
manualContinuationEnabled={manualContinuationEnabled}
loopGenerationEnabled={loopGenerationEnabled}
curatedPromptLimit={curatedPromptLimit}
maxCuratedPromptCount={maxCuratedPromptCount}
@@ -299,6 +304,7 @@ export default function DevtoolsShell({
onEnhancementToggle={onEnhancementToggle}
onCuratedPromptLimitChange={onCuratedPromptLimitChange}
onAutoExtensionToggle={onAutoExtensionToggle}
onManualContinuationToggle={onManualContinuationToggle}
onLoopToggle={onLoopToggle}
onLivePromptModeToggle={onLivePromptModeToggle}
onSpeechTranscript={onSpeechTranscript}
@@ -57,4 +57,32 @@ describe('prependPromptEvent', () => {
expect(next[0].promptId).toBe('new');
expect(next.some((item: any) => item.promptId === 'p-23')).toBe(false);
});
it('never drops steering scene events when capping', () => {
// 30 scenes interleaved with 30 other events — well past the cap.
let events: Record<string, any>[] = [];
for (let i = 0; i < 30; i += 1) {
events = prependPromptEvent(events, {
promptId: `scene-${i}`,
status: 'submitted',
steeringUserPrompt: true,
rawText: `scene ${i}`,
});
events = prependPromptEvent(events, {
promptId: `other-${i}`,
status: 'submitted',
});
}
const scenes = events.filter((e) => e.steeringUserPrompt);
expect(scenes).toHaveLength(30);
// Oldest-first scene order (and therefore numbering) is stable and complete.
expect(scenes[scenes.length - 1].promptId).toBe('scene-0');
expect(scenes[0].promptId).toBe('scene-29');
// Non-scene events are still capped, oldest dropped first.
const others = events.filter((e) => !e.steeringUserPrompt);
expect(others.length).toBeLessThanOrEqual(24);
expect(others.some((e) => e.promptId === 'other-0')).toBe(false);
expect(others[0].promptId).toBe('other-29');
});
});
+15 -1
View File
@@ -16,5 +16,19 @@ export function prependPromptEvent(
events: Record<string, any>[],
event: Record<string, any>,
): Record<string, any>[] {
return [event, ...events].slice(0, MAX_PROMPT_EVENTS);
const next = [event, ...events];
if (next.length <= MAX_PROMPT_EVENTS) {
return next;
}
// Steering scene events (steeringUserPrompt) are exempt from the cap: the
// scene list is derived from them and must stay complete and stably numbered
// for long sessions. Only the oldest non-scene events are dropped.
let nonSceneKept = 0;
return next.filter((e) => {
if (e?.steeringUserPrompt) {
return true;
}
nonSceneKept += 1;
return nonSceneKept <= MAX_PROMPT_EVENTS;
});
}
+165 -1
View File
@@ -1,6 +1,11 @@
import { describe, expect, it } from 'vitest';
import { resolveSessionErrorMessage } from './reducer';
import { applyNormalizedSocketEvent, resolveSessionErrorMessage } from './reducer';
import { createSessionStore } from '../../stores/session';
import { createRewriteStore } from '../../stores/rewrite';
import { createStreamStore } from '../../stores/stream';
import { createUiStore } from '../../stores/ui';
import { createPromptWindowStore } from '../../stores/promptWindow';
describe('resolveSessionErrorMessage', () => {
it('returns a dedicated message for IP session limit errors', () => {
@@ -19,3 +24,162 @@ describe('resolveSessionErrorMessage', () => {
})).toBe('Backend replica unavailable. Rejoin session.');
});
});
function buildContext(overrides: Record<string, unknown> = {}) {
const sessionStore = createSessionStore();
const rewriteStore = createRewriteStore();
const streamStore = createStreamStore();
const uiStore = createUiStore();
const promptWindowStore = createPromptWindowStore();
const avPipeline = {
reset: () => {},
setStreamCompleted: () => {},
noteSegmentInit: () => {},
noteSegmentComplete: () => {},
maybeStartPlayback: () => {},
ensurePipeline: async () => {},
};
return {
sessionStore,
promptWindowStore,
rewriteStore,
streamStore,
uiStore,
avPipeline,
tick: async () => {},
defaultAvMime: 'video/mp4',
fixedRewriteModel: 'model',
parseLatencyMs: () => null,
formatPromptWindowEventText: () => '',
makePromptId: () => 'generated-id',
buildClipLabel: () => 'clip',
startSessionCountdown: () => {},
clearCountdownInterval: () => {},
resetTtffTimer: () => {},
startTtffTimer: () => {},
preserveArchivedPlaybackSelection: false,
finalizeStreamCompletion: async () => {},
...overrides,
};
}
describe('steering generatingNextScene flow', () => {
it('sets generatingNextScene on prompt/sources_resumed in manual mode', async () => {
const context = buildContext();
await applyNormalizedSocketEvent(
{ type: 'prompt/sources_resumed', payload: { segment_idx: 2 } },
context,
);
expect(context.sessionStore.get().generatingNextScene).toBe(true);
expect(context.sessionStore.get().waitingForSegmentPrompt).toBe(false);
});
it('does NOT set generatingNextScene on session/auto_extension_updated', async () => {
const context = buildContext();
await applyNormalizedSocketEvent(
{ type: 'session/auto_extension_updated', payload: { enabled: true } },
context,
);
expect(context.sessionStore.get().generatingNextScene).toBe(false);
expect(context.sessionStore.get().waitingForSegmentPrompt).toBe(false);
});
it('clears generatingNextScene when segment media arrives', async () => {
const context = buildContext();
context.sessionStore.patch({ generatingNextScene: true });
await applyNormalizedSocketEvent(
{
type: 'stream/media_init',
payload: { segment_idx: 2, stream_id: 's', mime: 'video/mp4' },
},
context,
);
expect(context.sessionStore.get().generatingNextScene).toBe(false);
});
it('clears generatingNextScene and returns to waiting on prompt/sources_blocked', async () => {
const context = buildContext();
context.sessionStore.patch({ generatingNextScene: true });
await applyNormalizedSocketEvent(
{ type: 'prompt/sources_blocked', payload: { segment_idx: 3 } },
context,
);
expect(context.sessionStore.get().generatingNextScene).toBe(false);
expect(context.sessionStore.get().waitingForSegmentPrompt).toBe(true);
});
it('clears generatingNextScene when the opening prompt falls back', async () => {
const context = buildContext();
context.sessionStore.patch({ generatingNextScene: true });
await applyNormalizedSocketEvent(
{
type: 'prompt/fallback_used',
payload: { prompt_id: 'p1', prompt: '', source: 'user_enhancement_failed' },
},
context,
);
expect(context.sessionStore.get().generatingNextScene).toBe(false);
expect(context.sessionStore.get().waitingForSegmentPrompt).toBe(true);
});
});
describe('opening prompt id tracking', () => {
it('routes prompt lifecycle updates to the frontend-recorded opening event', async () => {
// The frontend records the opening scene under its own prompt id and sends it
// as initial_rollout_prompt_id; the backend echoes it in status updates.
const context = buildContext();
context.rewriteStore.addPromptEvent({
promptId: 'opening-id',
status: 'rewrite_requested',
source: 'user_rewrite',
text: 'a castle at dawn',
steeringUserPrompt: true,
rawText: 'a castle at dawn',
});
await applyNormalizedSocketEvent(
{ type: 'prompt/enhancing', payload: { prompt_id: 'opening-id' } },
context,
);
let opening = (context.rewriteStore.get().promptEvents as Record<string, any>[])
.find((e) => e.promptId === 'opening-id');
expect(opening?.status).toBe('enhancing');
await applyNormalizedSocketEvent(
{
type: 'prompt/fallback_used',
payload: { prompt_id: 'opening-id', prompt: '', source: 'user_enhancement_failed' },
},
context,
);
opening = (context.rewriteStore.get().promptEvents as Record<string, any>[])
.find((e) => e.promptId === 'opening-id');
expect(opening?.status).toBe('ready_fallback');
// A failed opening is dropped from the steering scene list instead of
// lingering as a ghost "Scene 1".
expect(opening?.steeringFailed).toBe(true);
});
it('marks a prompt-scoped session/error (e.g. safety block) as steeringFailed', async () => {
const context = buildContext();
context.rewriteStore.addPromptEvent({
promptId: 'blocked-id',
status: 'queued',
source: 'user_raw',
text: 'a blocked prompt',
steeringUserPrompt: true,
rawText: 'a blocked prompt',
});
await applyNormalizedSocketEvent(
{
type: 'session/error',
payload: { message: 'Prompt blocked by safety filter.', prompt_id: 'blocked-id' },
},
context,
);
const blocked = (context.rewriteStore.get().promptEvents as Record<string, any>[])
.find((e) => e.promptId === 'blocked-id');
expect(blocked?.steeringFailed).toBe(true);
});
});
+50 -9
View File
@@ -99,7 +99,19 @@ export async function applyNormalizedSocketEvent(event: any, context: any): Prom
status: "ready_fallback",
source: payload.source || "user_raw",
text: payload.prompt,
// Steering: this prompt produced no segment — drop it from the scene list.
steeringFailed: true,
});
// Steering recovery: enhancement failed so the backend enqueued nothing AND won't
// re-emit prompt_sources_blocked (its drained flag is still set). Put the user back to
// "describe the next scene" ourselves so the generating overlay clears and they can retry.
if (!uiStore.get().simpleMode && sessionStore.get().manualContinuationMode) {
sessionStore.patch({
waitingForSegmentPrompt: true,
generatingNextScene: false,
sessionNotice: "Couldn't continue from that prompt — try rephrasing the next scene.",
});
}
console.warn("[PromptEnhanceFallback] Prompt extension failed for this request.");
return;
@@ -243,19 +255,32 @@ export async function applyNormalizedSocketEvent(event: any, context: any): Prom
}
case "prompt/sources_blocked":
sessionStore.patch({
autoExtensionTimeoutHint: uiStore.get().simpleMode ? "" : "blocked on user input, increase prompt count for smoother experience",
});
if (sessionStore.get().manualContinuationMode) {
sessionStore.patch({ waitingForSegmentPrompt: true, generatingNextScene: false, autoExtensionTimeoutHint: "" });
} else {
sessionStore.patch({
autoExtensionTimeoutHint: uiStore.get().simpleMode ? "" : "blocked on user input, increase prompt count for smoother experience",
});
}
return;
case "prompt/sources_resumed":
sessionStore.patch({
autoExtensionTimeoutHint: "",
waitingForSegmentPrompt: false,
// A real prompt was just selected for the next segment; media arriving
// (stream/media_init) clears this again.
...(sessionStore.get().manualContinuationMode ? { generatingNextScene: true } : {}),
});
return;
case "session/auto_extension_updated":
sessionStore.patch({ autoExtensionTimeoutHint: "" });
if (event.type === "session/auto_extension_updated") {
console.log("[AutoExtensionUpdated]", {
enabled: sessionStore.get().autoExtensionEnabled,
});
}
// Deliberately does NOT touch generatingNextScene: toggling auto extension
// starts no generation.
sessionStore.patch({ autoExtensionTimeoutHint: "", waitingForSegmentPrompt: false });
console.log("[AutoExtensionUpdated]", {
enabled: sessionStore.get().autoExtensionEnabled,
});
return;
case "segment/step_complete":
@@ -277,6 +302,7 @@ export async function applyNormalizedSocketEvent(event: any, context: any): Prom
projectResetPending: false,
sessionExpired: true,
sessionNotice: "",
generatingNextScene: false,
});
console.log("Session timed out");
clearCountdownInterval();
@@ -354,6 +380,8 @@ export async function applyNormalizedSocketEvent(event: any, context: any): Prom
return;
case "stream/media_init":
// Segment media is arriving — the "Generating next scene" phase is over.
sessionStore.patch({ generatingNextScene: false });
streamStore.patch({
mediaAppendError: null,
loadingAnimation: streamStore.get().avPlaybackStarted ? streamStore.get().loadingAnimation : true,
@@ -471,17 +499,30 @@ export async function applyNormalizedSocketEvent(event: any, context: any): Prom
sessionStore.patch({
generationCapReached: false,
sessionNotice: "",
generatingNextScene: false,
});
await finalizeStreamCompletion();
return;
case "session/error": {
const errorMessage = resolveSessionErrorMessage(payload);
if (payload?.prompt_id) {
// Prompt-scoped error (e.g. safety-blocked): the prompt produced no
// segment, so drop it from the steering scene list.
rewriteStore.trackPromptEvent(payload.prompt_id, {
steeringFailed: true,
});
}
sessionStore.patch({
generationCapReached: false,
preservePlaybackOnClose: false,
promptExtensionError: "",
sessionNotice: errorMessage,
// Steering: a blocked/failed prompt produced no segment and the backend won't re-emit
// prompt_sources_blocked, so recover the "describe the next scene" state ourselves.
...(!uiStore.get().simpleMode && sessionStore.get().manualContinuationMode
? { waitingForSegmentPrompt: true, generatingNextScene: false }
: {}),
});
rewriteStore.patch({
rewritingSeedPrompts: false,
@@ -24,6 +24,9 @@ export interface SessionState {
livePromptRewriteMode: boolean;
sessionExpired: boolean;
projectResetPending: boolean;
manualContinuationMode: boolean;
waitingForSegmentPrompt: boolean;
generatingNextScene: boolean;
}
const DEFAULT_SESSION_STATE: SessionState = {
@@ -49,6 +52,12 @@ const DEFAULT_SESSION_STATE: SessionState = {
livePromptRewriteMode: false,
sessionExpired: false,
projectResetPending: false,
// Steering-only product: every session drives scenes manually. There is no
// auto-rollout mode and no UI selector, so this stays true throughout.
manualContinuationMode: true,
waitingForSegmentPrompt: false,
// True from scene submit / prompt selection until the segment's media arrives.
generatingNextScene: false,
};
export type SessionStore = ManagedStore<SessionState> & {
+18 -1
View File
@@ -1,5 +1,18 @@
import { createManagedStore, type ManagedStore } from "./createManagedStore";
// Prompt-history sources driven by the user's own submissions. Curated (non-user)
// entries — e.g. a preset's opening scene — feed the steering scene list and are
// exempt from the history cap so Scene 1 survives long sessions.
export const USER_PROMPT_SOURCES = new Set([
"user_raw",
"user",
"user_enhanced",
"user_rewrite",
"user_enhancement_failed",
]);
const PROMPT_HISTORY_CAP = 120;
export interface StreamState {
playingSeedPromptIndex: number | null;
generatingSeedPromptIndex: number | null;
@@ -159,10 +172,14 @@ export function createStreamStore(initialState: Partial<StreamState> = {}): Stre
loopIteration: typeof loopIteration === "number" ? loopIteration : null,
};
const nextHistory = [entry, ...state.promptHistory];
return {
...state,
promptHistoryCounter: nextCounter,
promptHistory: [entry, ...state.promptHistory].slice(0, 120),
promptHistory: nextHistory.length > PROMPT_HISTORY_CAP
? nextHistory.filter((item, index) =>
index < PROMPT_HISTORY_CAP || !USER_PROMPT_SOURCES.has(String(item.source || "")))
: nextHistory,
selectedHistoryId: state.selectedHistoryId || (entry.id as string),
};
});
+1 -1
View File
@@ -1 +1 @@
NEXT_PUBLIC_API_BASE_URL=http://localhost:8189/api
PUBLIC_API_BASE_URL=http://localhost:8189/api
+1 -6
View File
@@ -13,12 +13,6 @@
# testing
/coverage
# playwright
/test-results/
/playwright-report/
/blob-report/
/.playwright/
# next.js
/.next/
/out/
@@ -45,3 +39,4 @@ yarn-error.log*
# typescript
*.tsbuildinfo
next-env.d.ts
.svelte-kit
+4
View File
@@ -0,0 +1,4 @@
{
"useTabs": true,
"tabWidth": 4
}
+5 -49
View File
@@ -11,63 +11,19 @@ The UI currently supports:
- Datasets
- Gallery View
## Project Structure
```
apps/fastvideo_studio/
├── server.py / job_runner.py / database.py # FastAPI backend + job lifecycle
├── mock_server.py # In-memory API mock for e2e tests
├── models/ # Pydantic request models (shared with the mock)
├── training_config.py # Studio workloads → fastvideo/train YAML configs
├── tests/ # Backend unit tests (pytest)
├── e2e/ # Playwright specs (run against the mock)
└── src/
├── app/ # Next.js App Router pages (thin routes)
├── components/
│ ├── shell/ # App chrome: header, sidebars, layout
│ ├── jobs/ # Job queue, cards, create-job modal, log sidebar
│ ├── datasets/ # Dataset cards, upload, captions
│ └── ui/ # shadcn-style primitives (shared theme)
├── stores/ # Framework-agnostic state + React bridge (hooks/)
├── lib/ # API client, types, option persistence
└── test/ # Vitest setup + factories
```
The visual theme (slate light/dark palettes, IBM Plex type, `#356cff` accent)
is shared with `apps/dreamverse`; the toggle in the header persists the choice
per browser.
## Testing
```bash
npm run typecheck # tsc
npm test # vitest unit tests
npm run e2e # Playwright against the in-memory mock backend
python -m pytest tests/ # backend unit tests (from apps/fastvideo_studio)
```
## Quick Start
For local development, install dependencies and start the Next.js dev server:
```bash
cd apps/fastvideo_studio
npm install
npm run dev
```
You can then access the app at [http://localhost:3000](http://localhost:3000).
Start the Python API server (default port 8189) in a separate terminal — see below.
For a production build of the web app together with the API server:
The easiest way to run the application is:
```bash
cd apps/fastvideo_studio
npm install
npm run build
npm run start:all
npm run start
```
You can then access the app at [http://localhost:3000](http://localhost:3000).
### Running API and Web Separately
The UI is composed of two separate components:
@@ -75,7 +31,7 @@ The UI is composed of two separate components:
- API Server
- Web Server
To run each component separately, you can use the commands `npm run start:web` (after `npm run build`) and `npm run start:api`.
To run each component separately, you can use the commands `npm run start:web` and `npm run start:api`.
You can also run the API server with this command from the `apps/` directory:
+28 -4
View File
@@ -251,10 +251,11 @@ class Database:
sp_size, negative_prompt,
data_path, max_train_steps, train_batch_size, learning_rate,
num_latent_t, validation_dataset_file, lora_rank,
ltx2_first_frame_conditioning_p,
dmd_use_vsa, dmd_vsa_sparsity, dmd_denoising_steps,
real_score_guidance_scale,
min_timestep_ratio, max_timestep_ratio, real_score_guidance_scale,
generator_update_interval, real_score_model_path, fake_score_model_path
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
""",
(
job["id"],
@@ -296,9 +297,12 @@ class Database:
job.get("num_latent_t", 20),
job.get("validation_dataset_file", ""),
job.get("lora_rank", 32),
job.get("ltx2_first_frame_conditioning_p"),
1 if job.get("dmd_use_vsa") else 0,
job.get("dmd_vsa_sparsity", 0.8),
job.get("dmd_denoising_steps", "1000,757,522"),
job.get("min_timestep_ratio", 0.02),
job.get("max_timestep_ratio", 0.98),
job.get("real_score_guidance_scale", 3.5),
job.get("generator_update_interval", 5),
job.get("real_score_model_path", ""),
@@ -397,6 +401,24 @@ class Database:
for file_name, caption in captions.items():
self.upsert_dataset_caption(dataset_id, file_name, caption)
def update_dataset(self, dataset_id: str, updates: dict[str, Any]) -> None:
"""Update dataset fields. Only provided keys are updated."""
if not updates:
return
allowed: set[str] = set()
cols = []
vals = []
for k, v in updates.items():
if k in allowed:
cols.append(f"{k} = ?")
vals.append(v)
if not cols:
return
vals.append(dataset_id)
sql = f"UPDATE datasets SET {', '.join(cols)} WHERE id = ?"
self._execute(sql, tuple(vals))
self._commit()
def delete_dataset(self, dataset_id: str) -> bool:
"""Delete a dataset. Returns True if a row was deleted."""
cur = self._execute("DELETE FROM datasets WHERE id = ?", (dataset_id, ))
@@ -421,7 +443,7 @@ class Database:
cur = self._execute("SELECT * FROM settings WHERE id = 1")
row = cur.fetchone()
if not row:
return default_settings_dict()
return _default_settings_dict()
t2v = ((row["default_model_id_t2v"] or row["default_model_id"] or "") if "default_model_id_t2v" in row else
(row["default_model_id"] or ""))
i2v = ((row["default_model_id_i2v"] or "") if "default_model_id_i2v" in row else "")
@@ -535,6 +557,8 @@ def _row_to_job(row: sqlite3.Row) -> dict[str, Any]:
}
float_defaults = {
"dmd_vsa_sparsity": 0.8,
"min_timestep_ratio": 0.02,
"max_timestep_ratio": 0.98,
"real_score_guidance_scale": 3.5,
}
result = {
@@ -601,7 +625,7 @@ def _row_to_job(row: sqlite3.Row) -> dict[str, Any]:
return result
def default_settings_dict() -> dict[str, Any]:
def _default_settings_dict() -> dict[str, Any]:
"""Return default settings as API-style dict."""
return {
"defaultModelId": DEFAULT_SETTINGS["default_model_id"],
@@ -1,42 +0,0 @@
import { expect, test } from '@playwright/test';
import { skipWithoutMock } from './helpers';
/**
* Create-job flow: open the Create Job modal on /inference, fill the prompt
* (the model auto-selects once the mock's /api/models loads), submit, and
* confirm the new job lands in the queue.
*/
test.describe('create inference job', () => {
skipWithoutMock();
test('creates a T2V job and shows it in the queue', async ({ page }) => {
await page.goto('/inference');
// The trigger opens a real menu on click, so this path works for touch,
// mouse, and keyboard users.
await page.getByRole('button', { name: /create job/i }).click();
const t2vItem = page.getByRole('menuitem', { name: /T2V/i });
await expect(t2vItem).toBeVisible();
await t2vItem.click();
const dialog = page.getByRole('dialog');
await expect(dialog).toBeVisible();
// Wait for the mock's model catalogue to populate the dropdown (more than
// just the disabled placeholder), then pick one explicitly — the app's
// auto-selection is racy.
const modelSelect = dialog.getByLabel('Model', { exact: true });
await expect(modelSelect.locator('option')).not.toHaveCount(1);
await modelSelect.selectOption({ index: 1 });
const prompt = `e2e raccoon in sunflowers ${Date.now()}`;
await dialog.getByLabel('Prompt', { exact: true }).fill(prompt);
await dialog.getByRole('button', { name: 'Create Job' }).click();
// Modal closes and the queue refreshes with the newly created job.
await expect(dialog).toBeHidden();
await expect(page.getByText(prompt)).toBeVisible();
});
});
@@ -1,37 +0,0 @@
import { expect, test } from '@playwright/test';
import { API_BASE, skipWithoutMock } from './helpers';
/**
* Datasets page: the seeded datasets render as cards, and the Create Dataset
* modal opens from the header action.
*/
test.describe('datasets', () => {
skipWithoutMock();
test('lists the seeded datasets', async ({ page, request }) => {
// Read the seeded names from the mock so the assertion tracks the fixture.
const res = await request.get(`${API_BASE}/datasets`);
const datasets = (await res.json()) as Array<{ name: string }>;
expect(datasets.length).toBeGreaterThan(0);
await page.goto('/datasets');
for (const ds of datasets) {
await expect(page.getByText(ds.name, { exact: true })).toBeVisible();
}
});
test('opens the Create Dataset modal', async ({ page }) => {
await page.goto('/datasets');
await page.getByRole('button', { name: 'Add Dataset' }).click();
const dialog = page.getByRole('dialog');
await expect(dialog).toBeVisible();
await expect(
dialog.getByRole('heading', { name: /Add Dataset/i }),
).toBeVisible();
await expect(dialog.getByLabel('Name', { exact: true })).toBeVisible();
});
});
-44
View File
@@ -1,44 +0,0 @@
import { expect, test } from '@playwright/test';
import { API_BASE, skipWithoutMock } from './helpers';
/**
* Gallery page: the seeded completed inference job surfaces as a media tile
* with playback controls or an explicit media-error fallback.
*/
test.describe('gallery', () => {
skipWithoutMock();
test('shows a media tile for the seeded completed job', async ({
page,
request,
}) => {
const res = await request.get(`${API_BASE}/jobs?job_type=inference`);
const jobs = (await res.json()) as Array<{
status: string;
output_path: string | null;
prompt: string;
}>;
const completed = jobs.find(
(j) => j.status === 'completed' && j.output_path,
);
expect(completed, 'mock should seed a completed inference job').toBeTruthy();
await page.goto('/gallery');
await expect(
page.getByRole('heading', { level: 1, name: 'Gallery' }),
).toBeVisible();
const tile = page.locator('article').filter({ hasText: completed!.prompt });
await expect(tile).toBeVisible();
await expect(
tile.locator('video').or(tile.getByText('Preview unavailable')),
).toBeVisible();
const video = tile.locator('video');
if (await video.isVisible()) {
await expect(video).toHaveAttribute('controls', '');
}
});
});
-30
View File
@@ -1,30 +0,0 @@
import { expect, test } from '@playwright/test';
import { skipWithoutMock } from './helpers';
/**
* GPU status page: the mock's fake GPUs render as cards with meters, and the
* page is reachable from the primary sidebar.
*/
test.describe('gpus', () => {
skipWithoutMock();
test('lists the mock GPUs with utilization meters', async ({ page }) => {
await page.goto('/gpus');
await expect(
page.getByRole('heading', { level: 1, name: 'GPUs' }),
).toBeVisible();
await expect(page.getByText('GPU 0')).toBeVisible();
await expect(page.getByText('GPU 1')).toBeVisible();
expect(await page.getByRole('meter').count()).toBe(4);
});
test('is reachable from the sidebar', async ({ page }) => {
await page.goto('/inference');
await page.getByRole('link', { name: 'GPUs' }).click();
await expect(page).toHaveURL(/\/gpus$/);
await expect(page.getByText('GPU 0')).toBeVisible();
});
});
-44
View File
@@ -1,44 +0,0 @@
import { test, type APIRequestContext } from '@playwright/test';
/**
* Port and base URL of the mock API — the single source of truth shared by
* playwright.config.ts (which boots the mock and points `npm run dev` at it
* via NEXT_PUBLIC_API_BASE_URL) and the specs (which probe/read mock state
* directly through the Playwright `request` fixture).
*
* The default port is deliberately NOT the real server's default (8189): the
* e2e specs create and (if autostart is on) launch jobs, so running them
* against a real backend would mutate the developer's database and fire real
* GPU runs. API_BASE is derived solely from MOCK_API_PORT (it does not inherit
* NEXT_PUBLIC_API_BASE_URL) so the specs always talk to the mock.
*/
export const MOCK_API_PORT = process.env.MOCK_API_PORT || '8190';
export const API_BASE = `http://127.0.0.1:${MOCK_API_PORT}/api`;
export const MOCK_SKIP_MESSAGE =
`Mock backend not reachable at ${API_BASE}/__mock__ ` +
`(start fastvideo_studio.mock_server on port ${MOCK_API_PORT}).`;
/**
* True only when the mock server answers on the configured port. Probes the
* mock-only /api/__mock__ sentinel (which the real server does not serve) so
* the suite refuses to run its mutating specs against a real backend.
*/
export async function mockIsUp(request: APIRequestContext): Promise<boolean> {
try {
const res = await request.get(`${API_BASE}/__mock__`);
if (!res.ok()) return false;
const body = await res.json();
return body?.mock === true;
} catch {
return false;
}
}
/** Self-skip every test in the enclosing describe when the mock is down. */
export function skipWithoutMock(): void {
test.beforeEach(async ({ request }) => {
test.skip(!(await mockIsUp(request)), MOCK_SKIP_MESSAGE);
});
}
-115
View File
@@ -1,115 +0,0 @@
import { expect, test } from '@playwright/test';
import { skipWithoutMock } from './helpers';
/**
* App-shell smoke: the Next.js frontend hydrates, renders the FastVideo logo
* and the primary-sidebar navigation, and routing between the top-level
* sections works. Each spec self-skips when the mock backend isn't reachable.
*/
test.describe('app shell', () => {
skipWithoutMock();
test('loads with the logo and primary-sidebar nav', async ({ page }) => {
await page.goto('/');
// The root route redirects to /inference.
await expect(page).toHaveURL(/\/inference$/);
await expect(page.getByRole('img', { name: /fastvideo/i })).toBeVisible();
for (const label of ['Inference', 'Datasets', 'Gallery', 'Settings']) {
await expect(page.getByRole('link', { name: label })).toBeVisible();
}
});
test('navigates between the primary sections', async ({ page }) => {
await page.goto('/inference');
await expect(
page.getByRole('heading', { level: 1, name: 'Jobs' }),
).toBeVisible();
const sections: Array<{ link: string; url: RegExp; title: string }> = [
{ link: 'Datasets', url: /\/datasets$/, title: 'Datasets' },
{ link: 'Gallery', url: /\/gallery$/, title: 'Gallery' },
{ link: 'Settings', url: /\/settings$/, title: 'Settings' },
{ link: 'Inference', url: /\/inference$/, title: 'Jobs' },
];
for (const section of sections) {
await page.getByRole('link', { name: section.link }).click();
await expect(page).toHaveURL(section.url);
// The header <h1> is the only level-1 heading and reflects the route.
await expect(
page.getByRole('heading', { level: 1, name: section.title }),
).toBeVisible();
await expect(page.getByRole('main')).toHaveCount(1);
}
});
test('keeps navigation and content usable at responsive breakpoints', async ({
page,
}) => {
for (const width of [320, 375, 414, 768]) {
await page.setViewportSize({ width, height: 800 });
await page.goto('/inference');
const main = page.getByRole('main');
await expect(main).toBeVisible();
await expect(
page.getByRole('button', { name: /Create Job/i }),
).toBeVisible();
const initialBox = await main.boundingBox();
expect(initialBox?.x).toBe(width < 768 ? 0 : 220);
expect(initialBox?.width).toBe(width < 768 ? width : width - 220);
const navigation = page.getByRole('navigation', {
name: 'Primary navigation',
});
if (width < 768) {
await expect(
page.getByRole('button', { name: 'Open navigation' }),
).toBeVisible();
await page.getByRole('button', { name: 'Open navigation' }).click();
}
await expect(navigation).toBeVisible();
await navigation.getByRole('link', { name: 'Datasets' }).click();
await expect(page).toHaveURL(/\/datasets$/);
expect(
await page.evaluate(
() => document.documentElement.scrollWidth <= window.innerWidth,
),
).toBe(true);
}
});
test('uses full-width detail drawers on mobile', async ({ page }) => {
await page.setViewportSize({ width: 320, height: 800 });
await page.goto('/inference');
await page
.locator('article button[aria-pressed="false"]')
.first()
.click();
const jobDrawer = page.getByRole('dialog', { name: 'Job details' });
await expect(jobDrawer).toBeVisible();
expect(await jobDrawer.boundingBox()).toMatchObject({ x: 0, width: 320 });
await jobDrawer.getByRole('button', { name: 'Close' }).click();
await page.goto('/datasets');
await page
.locator('article button[aria-pressed="false"]')
.first()
.click();
const datasetDrawer = page.getByRole('dialog', {
name: /dataset details$/,
});
await expect(datasetDrawer).toBeVisible();
expect(await datasetDrawer.boundingBox()).toMatchObject({
x: 0,
width: 320,
});
});
});
+2
View File
@@ -0,0 +1,2 @@
// Minimal ESLint config for SvelteKit (add typescript-eslint / eslint-plugin-svelte as needed)
export default [];
-74
View File
@@ -1,74 +0,0 @@
# SPDX-License-Identifier: Apache-2.0
"""GPU telemetry for the studio status page, via NVML (nvidia-ml-py)."""
from __future__ import annotations
import contextlib
import logging
from typing import Any
logger = logging.getLogger("fastvideo.studio.gpu")
_nvml_initialized = False
def _ensure_nvml() -> Any:
"""Import and initialize NVML once; raises on machines without it."""
global _nvml_initialized # noqa: PLW0603
import pynvml
if not _nvml_initialized:
pynvml.nvmlInit()
_nvml_initialized = True
return pynvml
def _device_snapshot(pynvml: Any, index: int) -> dict[str, Any]:
handle = pynvml.nvmlDeviceGetHandleByIndex(index)
name = pynvml.nvmlDeviceGetName(handle)
if isinstance(name, bytes):
name = name.decode()
mem = pynvml.nvmlDeviceGetMemoryInfo(handle)
util = pynvml.nvmlDeviceGetUtilizationRates(handle)
# Optional sensors: not every GPU/driver exposes them.
temperature: int | None = None
power_watts: float | None = None
power_limit_watts: float | None = None
with contextlib.suppress(pynvml.NVMLError):
temperature = int(pynvml.nvmlDeviceGetTemperature(handle, pynvml.NVML_TEMPERATURE_GPU))
with contextlib.suppress(pynvml.NVMLError):
power_watts = pynvml.nvmlDeviceGetPowerUsage(handle) / 1000.0
power_limit_watts = (pynvml.nvmlDeviceGetEnforcedPowerLimit(handle) / 1000.0)
return {
"index": index,
"name": name,
"utilization": int(util.gpu),
"memory_used_mib": int(mem.used / (1024 * 1024)),
"memory_total_mib": int(mem.total / (1024 * 1024)),
"temperature_c": temperature,
"power_watts": power_watts,
"power_limit_watts": power_limit_watts,
}
def get_gpu_snapshot() -> dict[str, Any]:
"""Return {"available", "gpus", "error"} — never raises.
``available: False`` covers both "no NVIDIA driver/library on this host"
and transient NVML failures; the frontend shows ``error`` as-is.
"""
try:
pynvml = _ensure_nvml()
count = pynvml.nvmlDeviceGetCount()
gpus = [_device_snapshot(pynvml, i) for i in range(count)]
return {"available": True, "gpus": gpus, "error": None}
except ImportError:
return {
"available": False,
"gpus": [],
"error": "nvidia-ml-py is not installed on the API server host.",
}
except Exception as exc: # NVMLError, driver issues, …
logger.warning("GPU snapshot failed: %s", exc)
return {"available": False, "gpus": [], "error": str(exc)}
+58 -33
View File
@@ -23,13 +23,12 @@ import time
from dataclasses import dataclass, field
from pathlib import Path
from typing import Any
import yaml
from fastvideo.utils import get_mp_context
from fastvideo_studio.database import Database
from fastvideo_studio.training_config import (
build_training_config,
build_training_args,
get_training_env,
get_training_module_info,
)
logger = logging.getLogger("fastvideo.studio.job_runner")
@@ -161,10 +160,13 @@ class Job:
num_latent_t: int = 20
validation_dataset_file: str = ""
lora_rank: int = 32
ltx2_first_frame_conditioning_p: float | None = None
# DMD options
dmd_use_vsa: bool = False
dmd_vsa_sparsity: float = 0.8
dmd_denoising_steps: str = "1000,757,522"
min_timestep_ratio: float = 0.02
max_timestep_ratio: float = 0.98
real_score_guidance_scale: float = 3.5
generator_update_interval: int = 5
real_score_model_path: str = ""
@@ -219,9 +221,12 @@ class Job:
"num_width": self.width,
"validation_dataset_file": self.validation_dataset_file,
"lora_rank": self.lora_rank,
"ltx2_first_frame_conditioning_p": self.ltx2_first_frame_conditioning_p,
"dmd_use_vsa": self.dmd_use_vsa,
"dmd_vsa_sparsity": self.dmd_vsa_sparsity,
"dmd_denoising_steps": self.dmd_denoising_steps,
"min_timestep_ratio": self.min_timestep_ratio,
"max_timestep_ratio": self.max_timestep_ratio,
"real_score_guidance_scale": self.real_score_guidance_scale,
"generator_update_interval": self.generator_update_interval,
"real_score_model_path": self.real_score_model_path or "",
@@ -325,9 +330,12 @@ class JobRunner:
num_latent_t=row.get("num_latent_t", 20),
validation_dataset_file=row.get("validation_dataset_file", "") or "",
lora_rank=row.get("lora_rank", 32),
ltx2_first_frame_conditioning_p=row.get("ltx2_first_frame_conditioning_p"),
dmd_use_vsa=row.get("dmd_use_vsa", False),
dmd_vsa_sparsity=float(row.get("dmd_vsa_sparsity", 0.8)),
dmd_denoising_steps=row.get("dmd_denoising_steps", "1000,757,522") or "1000,757,522",
min_timestep_ratio=float(row.get("min_timestep_ratio", 0.02)),
max_timestep_ratio=float(row.get("max_timestep_ratio", 0.98)),
real_score_guidance_scale=float(row.get("real_score_guidance_scale", 3.5)),
generator_update_interval=int(row.get("generator_update_interval", 5)),
real_score_model_path=row.get("real_score_model_path", "") or "",
@@ -404,9 +412,12 @@ class JobRunner:
num_latent_t: int = 20,
validation_dataset_file: str = "",
lora_rank: int = 32,
ltx2_first_frame_conditioning_p: float | None = None,
dmd_use_vsa: bool = False,
dmd_vsa_sparsity: float = 0.8,
dmd_denoising_steps: str = "1000,757,522",
min_timestep_ratio: float = 0.02,
max_timestep_ratio: float = 0.98,
real_score_guidance_scale: float = 3.5,
generator_update_interval: int = 5,
real_score_model_path: str = "",
@@ -446,9 +457,12 @@ class JobRunner:
num_latent_t=num_latent_t,
validation_dataset_file=validation_dataset_file or "",
lora_rank=lora_rank,
ltx2_first_frame_conditioning_p=ltx2_first_frame_conditioning_p,
dmd_use_vsa=dmd_use_vsa,
dmd_vsa_sparsity=dmd_vsa_sparsity,
dmd_denoising_steps=dmd_denoising_steps,
min_timestep_ratio=min_timestep_ratio,
max_timestep_ratio=max_timestep_ratio,
real_score_guidance_scale=real_score_guidance_scale,
generator_update_interval=generator_update_interval,
real_score_model_path=real_score_model_path or "",
@@ -719,23 +733,46 @@ class JobRunner:
self._save_job(job)
return
env = os.environ.copy()
env.update(get_training_env())
try:
# Job.to_dict() carries every key the config builder reads
# (extra keys are ignored by its .get() lookups).
train_config = build_training_config(job.to_dict(), job_output_dir)
except ValueError as exc:
module_info = get_training_module_info(job.workload_type, job.model_id)
if not module_info:
job.status = JobStatus.FAILED
job.error = str(exc)
job.error = f"Unknown workload type: {job.workload_type}"
job.finished_at = time.time()
self._save_job(job)
return
config_path = os.path.join(job_output_dir, "train_config.yaml")
with open(config_path, "w", encoding="utf-8") as f:
yaml.safe_dump(train_config, f, sort_keys=False)
module_path, _pipeline_workload, use_vsa, _is_lora = module_info
dmd_use_vsa = (job.workload_type.startswith("dmd_") and getattr(job, "dmd_use_vsa", False))
env = os.environ.copy()
env.update(get_training_env(use_vsa or dmd_use_vsa))
job_dict = {
"model_id": job.model_id,
"data_path": job.data_path,
"workload_type": job.workload_type,
"num_gpus": job.num_gpus,
"max_train_steps": job.max_train_steps,
"train_batch_size": job.train_batch_size,
"learning_rate": job.learning_rate,
"num_latent_t": job.num_latent_t,
"num_height": job.height,
"num_width": job.width,
"num_frames": job.num_frames,
"validation_dataset_file": job.validation_dataset_file,
"lora_rank": job.lora_rank,
"ltx2_first_frame_conditioning_p": job.ltx2_first_frame_conditioning_p,
}
if job.workload_type.startswith("dmd_") or job.workload_type.startswith("self_forcing_"):
job_dict["dmd_use_vsa"] = getattr(job, "dmd_use_vsa", False)
job_dict["dmd_vsa_sparsity"] = getattr(job, "dmd_vsa_sparsity", 0.8)
job_dict["dmd_denoising_steps"] = getattr(job, "dmd_denoising_steps", "1000,757,522")
job_dict["min_timestep_ratio"] = getattr(job, "min_timestep_ratio", 0.02)
job_dict["max_timestep_ratio"] = getattr(job, "max_timestep_ratio", 0.98)
job_dict["real_score_guidance_scale"] = getattr(job, "real_score_guidance_scale", 3.5)
job_dict["generator_update_interval"] = getattr(job, "generator_update_interval", 5)
job_dict["real_score_model_path"] = (getattr(job, "real_score_model_path", "") or job.model_id)
job_dict["fake_score_model_path"] = (getattr(job, "fake_score_model_path", "") or job.model_id)
train_args = build_training_args(job_dict, job_output_dir)
repo_root = Path(__file__).resolve().parent.parent
torchrun_cmd = [
@@ -746,12 +783,9 @@ class JobRunner:
str(job.num_gpus),
"--nnodes",
"1",
"-m",
"fastvideo.train.entrypoint.train",
"--config",
config_path,
]
buf.write(f"Starting training: {' '.join(torchrun_cmd)}")
str(repo_root / module_path),
] + train_args
buf.write(f"Starting training: {' '.join(torchrun_cmd[:12])}...")
buf.phase = "starting"
try:
@@ -791,21 +825,12 @@ class JobRunner:
job._process.wait()
exit_code = job._process.returncode or 0
if job._stop_event.is_set():
# Terminated by stop_job() without the reader loop observing the
# flag (e.g. the process died between log lines).
job.status = JobStatus.STOPPED
buf.phase = "stopped"
elif exit_code == 0:
if exit_code == 0:
job.status = JobStatus.COMPLETED
buf.progress = 100.0
buf.phase = "done"
# Training outputs checkpoints, not video. Sort by step number,
# not lexically ("checkpoint-1000" < "checkpoint-500" as strings).
ckpt_dirs = sorted(
Path(job_output_dir).glob("checkpoint-*"),
key=lambda p: int(m.group(1)) if (m := re.fullmatch(r"checkpoint-(\d+)", p.name)) else -1,
)
# Training outputs checkpoints, not video
ckpt_dirs = sorted(Path(job_output_dir).glob("checkpoint-*"))
if ckpt_dirs:
job.output_path = str(ckpt_dirs[-1])
else:
-660
View File
@@ -1,660 +0,0 @@
# SPDX-License-Identifier: Apache-2.0
"""
In-memory mock of the FastVideo Studio API (``server.py``).
This mock implements the same ``/api`` routes and response shapes as the real
FastAPI server but keeps everything in memory and never touches the FastVideo
library, a GPU, or a database. It is meant to back the Playwright e2e suite so
the Next.js frontend can be exercised end-to-end without a real backend.
Job lifecycle is simulated by recording a start timestamp and *computing* the
status on read: a started job reports ``running`` for a few seconds and then
flips to ``completed`` with an ``output_path``, so polling the job list / logs
shows progression. Generated media is a tiny 1s ``testsrc`` MP4 built lazily
with ffmpeg and cached.
Usage (from the ``apps/`` directory)::
PYTHONPATH=.. python -m fastvideo_studio.mock_server --port 8190
"""
from __future__ import annotations
import contextlib
import os
import random
import shutil
import subprocess
import tempfile
import threading
import time
import uuid
from typing import Annotated, Any
from fastapi import FastAPI, File, HTTPException, UploadFile
from fastapi.middleware.cors import CORSMiddleware
from fastapi.responses import FileResponse, PlainTextResponse
from fastvideo_studio.database import default_settings_dict
from fastvideo_studio.models import (CreateDatasetRequest, CreateJobRequest, SettingsUpdate, UpdateCaptionRequest,
model_label)
# --- Config -----------------------------------------------------------------
# How long a started job stays "running" before it flips to "completed".
COMPLETE_AFTER_SECONDS = 3.0
FFMPEG_BIN = shutil.which(os.getenv("FASTVIDEO_FFMPEG_BIN", "ffmpeg"))
# A small catalogue of fake models keyed by workload type. Mirrors the real
# server's {id, label} shape (label derived from the HF-style path).
_MODELS_BY_WORKLOAD: dict[str, list[str]] = {
"t2v": [
"Wan-AI/Wan2.1-T2V-1.3B-Diffusers",
"FastVideo/FastHunyuan-diffusers",
],
"i2v": [
"Wan-AI/Wan2.1-I2V-14B-480P-Diffusers",
],
"t2i": [
"black-forest-labs/FLUX.1-schnell",
],
}
def _models_for(workload_type: str | None) -> list[dict[str, str]]:
if workload_type:
paths = _MODELS_BY_WORKLOAD.get(workload_type, [])
else:
seen: dict[str, None] = {}
for paths_for_workload in _MODELS_BY_WORKLOAD.values():
for path in paths_for_workload:
seen.setdefault(path, None)
paths = list(seen)
return [{"id": path, "label": model_label(path)} for path in paths]
# The real settings catalogue lives in database.py; the mock only pre-fills
# the default model ids so the Create Job modal auto-selects one.
_DEFAULT_SETTINGS: dict[str, Any] = {
**default_settings_dict(),
"defaultModelId": "Wan-AI/Wan2.1-T2V-1.3B-Diffusers",
"defaultModelIdT2v": "Wan-AI/Wan2.1-T2V-1.3B-Diffusers",
"defaultModelIdI2v": "Wan-AI/Wan2.1-I2V-14B-480P-Diffusers",
"defaultModelIdT2i": "black-forest-labs/FLUX.1-schnell",
}
# --- In-memory state --------------------------------------------------------
_state_lock = threading.Lock()
_settings: dict[str, Any] = dict(_DEFAULT_SETTINGS)
_jobs: dict[str, dict[str, Any]] = {}
_datasets: dict[str, dict[str, Any]] = {}
# dataset_id -> {"file_names": [...], "captions": {file_name: caption}}
_dataset_files: dict[str, dict[str, Any]] = {}
# Lazily-built, cached media clips (shared by every job/dataset media response),
# keyed by extension: "mp4" (1s testsrc video) or "png" (single testsrc frame).
_mock_media_cache: dict[str, str] = {}
_mock_media_lock = threading.Lock()
def _build_mock_media(kind: str) -> str:
"""Build (once) a tiny testsrc clip of the given ``kind`` with ffmpeg; cache the path."""
with _mock_media_lock:
cached = _mock_media_cache.get(kind)
if cached and os.path.isfile(cached):
return cached
if not FFMPEG_BIN:
raise HTTPException(status_code=500, detail="ffmpeg is required to build mock media")
fd, path = tempfile.mkstemp(prefix="fvstudio_mock_", suffix=f".{kind}")
os.close(fd)
if kind == "png":
command = [
FFMPEG_BIN,
"-hide_banner",
"-loglevel",
"error",
"-y",
"-f",
"lavfi",
"-i",
"testsrc=size=320x240:rate=1",
"-frames:v",
"1",
"-f",
"image2",
path,
]
else:
command = [
FFMPEG_BIN,
"-hide_banner",
"-loglevel",
"error",
"-y",
"-f",
"lavfi",
"-i",
"testsrc=size=320x240:rate=24",
"-t",
"1",
"-c:v",
"libx264",
"-preset",
"ultrafast",
"-pix_fmt",
"yuv420p",
"-movflags",
"+faststart",
"-f",
"mp4",
path,
]
try:
subprocess.run(command, check=True, capture_output=True)
except (subprocess.CalledProcessError, OSError) as exc:
with contextlib.suppress(OSError):
os.remove(path)
raise HTTPException(status_code=500, detail=f"ffmpeg failed to build mock {kind}: {exc}") from exc
if not os.path.isfile(path) or os.path.getsize(path) == 0:
raise HTTPException(status_code=500, detail=f"ffmpeg produced no mock {kind} bytes")
_mock_media_cache[kind] = path
return path
# --- Job helpers ------------------------------------------------------------
def _new_job_dict(req: CreateJobRequest) -> dict[str, Any]:
"""Build a job dict (mirrors job_runner.Job.to_dict()) in the pending state."""
job_id = str(uuid.uuid4())
return {
"id": job_id,
"model_id": req.model_id,
"prompt": req.prompt,
"workload_type": req.workload_type or "t2v",
"job_type": req.job_type or "inference",
"image_path": req.image_path or "",
"status": "pending",
"created_at": time.time(),
"started_at": None,
"finished_at": None,
"error": None,
"output_path": None,
"log_file_path": None,
"num_inference_steps": req.num_inference_steps,
"num_frames": req.num_frames,
"height": req.height,
"width": req.width,
"guidance_scale": req.guidance_scale,
"guidance_rescale": req.guidance_rescale,
"fps": req.fps,
"seed": req.seed,
"negative_prompt": req.negative_prompt or "",
"num_gpus": req.num_gpus,
"data_path": req.data_path or "",
"progress": 0.0,
"progress_msg": "",
"phase": "pending",
}
def _advance_job(job: dict[str, Any]) -> None:
"""Flip a running job to completed once enough wall-clock time has passed.
Status is *computed on read* from the recorded start timestamp, so polling
the job list / logs naturally shows pending -> running -> completed.
"""
if job["status"] != "running" or not job.get("started_at"):
return
elapsed = time.time() - job["started_at"]
if elapsed >= COMPLETE_AFTER_SECONDS:
job["status"] = "completed"
job["finished_at"] = job["started_at"] + COMPLETE_AFTER_SECONDS
job["progress"] = 100.0
job["progress_msg"] = "50/50 steps"
job["phase"] = "done"
ext = "png" if job.get("workload_type") == "t2i" else "mp4"
job["output_path"] = f"/mock/outputs/{job['id']}/output.{ext}"
def _public_job(job: dict[str, Any]) -> dict[str, Any]:
"""Advance + return a copy safe to serialize."""
_advance_job(job)
return dict(job)
_LOG_TAIL = [
"Loading model...",
"Model loaded.",
"Starting generation...",
"Denoising step 10/50",
"Denoising step 20/50",
"Denoising step 30/50",
"Denoising step 40/50",
"Denoising step 50/50",
"Generation complete. Saving output...",
"Saved output file.",
"Job completed successfully.",
]
def _log_sequence(job: dict[str, Any]) -> list[str]:
return [
f"Job {job['id']} started",
f"Model: {job['model_id']}",
f"Prompt: {job['prompt']}",
*_LOG_TAIL,
]
def _compute_logs(job: dict[str, Any]) -> dict[str, Any]:
"""Return JobLogs-shaped data, growing the visible lines as time passes."""
seq = _log_sequence(job)
status = job["status"]
if status == "pending":
return {"lines": [], "progress": 0.0, "progress_msg": "", "phase": "pending"}
if status == "completed":
return {"lines": seq, "progress": 100.0, "progress_msg": "50/50 steps", "phase": "done"}
if status in ("failed", "stopped"):
# Keep the total line count monotonic across running -> terminal (the
# `after` cursor relies on it): show every non-final line plus a terminal
# notice, which is always >= any prefix a running job revealed.
lines = seq[:-1] + [f"Job {status}."]
return {"lines": lines, "progress": job.get("progress", 0.0), "progress_msg": "", "phase": status}
# running: reveal a prefix proportional to elapsed time, but never the final
# "completed" line — that appears only once the job is actually completed.
elapsed = time.time() - (job.get("started_at") or time.time())
frac = max(0.0, min(elapsed / COMPLETE_AFTER_SECONDS, 0.99))
reveal = min(max(4, round(frac * len(seq))), len(seq) - 1)
lines = seq[:reveal]
if frac < 0.2:
phase = "loading model"
elif frac < 0.85:
phase = "denoising"
else:
phase = "saving"
return {
"lines": lines,
"progress": round(frac * 100.0, 1),
"progress_msg": f"{int(frac * 50)}/50 steps",
"phase": phase,
}
# --- Dataset helpers --------------------------------------------------------
def _dataset_stats(dataset_id: str) -> tuple[int, int]:
files = _dataset_files.get(dataset_id, {}).get("file_names", [])
count = len(files)
# Fake but stable per-file size so the UI shows a non-zero footprint.
return count, count * 1_048_576
def _public_dataset(dataset: dict[str, Any]) -> dict[str, Any]:
count, size = _dataset_stats(dataset["id"])
return {**dataset, "file_count": count, "size_bytes": size}
def _seed() -> None:
"""Seed a couple of datasets and one completed inference job."""
now = time.time()
seeds = [
("Sunset Clips", ["sunset_01.mp4", "sunset_02.mp4"], {
"sunset_01.mp4": "A sunset over the ocean",
"sunset_02.mp4": "A sunset over the mountains"
}),
("City Timelapse", ["city_01.mp4"], {
"city_01.mp4": "A busy city intersection at night"
}),
]
for idx, (name, file_names, captions) in enumerate(seeds):
dataset_id = str(uuid.uuid4())
_datasets[dataset_id] = {"id": dataset_id, "name": name, "created_at": now - 600 + idx}
_dataset_files[dataset_id] = {"file_names": list(file_names), "captions": dict(captions)}
job_id = str(uuid.uuid4())
_jobs[job_id] = {
"id": job_id,
"model_id": "Wan-AI/Wan2.1-T2V-1.3B-Diffusers",
"prompt": "A curious raccoon peers through a field of yellow sunflowers",
"workload_type": "t2v",
"job_type": "inference",
"image_path": "",
"status": "completed",
"created_at": now - 300,
"started_at": now - 280,
"finished_at": now - 250,
"error": None,
"output_path": f"/mock/outputs/{job_id}/output.mp4",
"log_file_path": f"/mock/logs/{job_id}.log",
"num_inference_steps": 50,
"num_frames": 81,
"height": 480,
"width": 832,
"guidance_scale": 5.0,
"guidance_rescale": 0.0,
"fps": 24,
"seed": 1024,
"negative_prompt": "",
"num_gpus": 1,
"data_path": "",
"progress": 100.0,
"progress_msg": "50/50 steps",
"phase": "done",
}
# --- App --------------------------------------------------------------------
app = FastAPI(title="FastVideo Studio Mock API", version="0.1.0")
app.add_middleware(
CORSMiddleware,
allow_origins=["*"],
allow_methods=["*"],
allow_headers=["*"],
)
_seed()
@app.get("/api/__mock__")
def mock_sentinel() -> dict[str, bool]:
"""Mock-only marker so the e2e suite can prove it isn't hitting a real
backend before running its mutating specs (the real server has no such
route)."""
return {"mock": True}
# --- Settings ---------------------------------------------------------------
@app.get("/api/settings")
def get_settings() -> dict[str, Any]:
with _state_lock:
return dict(_settings)
@app.put("/api/settings")
def update_settings(settings: SettingsUpdate) -> dict[str, Any]:
updates = settings.model_dump(exclude_unset=True)
with _state_lock:
_settings.update({k: v for k, v in updates.items() if v is not None})
return dict(_settings)
# --- Models -----------------------------------------------------------------
@app.get("/api/models")
def list_models(workload_type: str | None = None) -> list[dict[str, Any]]:
return _models_for(workload_type)
# --- GPUs -------------------------------------------------------------------
@app.get("/api/gpus")
def list_gpus() -> dict[str, Any]:
"""Two fake GPUs with slight per-request jitter so the page looks live."""
gpus = []
for index, (base_util, used_mib) in enumerate([(62, 61_440), (7, 4_096)]):
gpus.append({
"index": index,
"name": "NVIDIA Mock GPU 80GB",
"utilization": max(0, min(100, base_util + random.randint(-5, 5))),
"memory_used_mib": used_mib + random.randint(-256, 256),
"memory_total_mib": 81_920,
"temperature_c": 55 + random.randint(-3, 3),
"power_watts": 310.0 + random.randint(-20, 20),
"power_limit_watts": 700.0,
})
return {"available": True, "gpus": gpus, "error": None}
# --- Uploads ----------------------------------------------------------------
@app.post("/api/upload-image")
async def upload_image(file: Annotated[UploadFile, File()]) -> dict[str, str]:
name = file.filename or "image.png"
return {"path": f"/mock/uploads/{uuid.uuid4().hex}_{os.path.basename(name)}"}
@app.post("/api/upload-raw-dataset")
async def upload_raw_dataset(files: Annotated[list[UploadFile], File()]) -> dict[str, Any]:
video_exts = {".mp4", ".webm", ".avi", ".mov", ".mkv"}
file_names: list[str] = []
for uf in files:
name = os.path.basename(uf.filename or f"{uuid.uuid4().hex}.mp4")
if os.path.splitext(name)[1].lower() in video_exts:
file_names.append(name)
if not file_names:
raise HTTPException(status_code=400, detail="No video files found.")
upload_id = uuid.uuid4().hex
path = f"/mock/uploads/{upload_id}"
return {"path": path, "upload_id": upload_id, "file_names": file_names}
# --- Jobs -------------------------------------------------------------------
@app.get("/api/jobs")
def list_jobs(job_type: str | None = None) -> list[dict[str, Any]]:
# _public_job mutates the shared job dict via _advance_job, so it must run
# under the lock (matching every other job route) to avoid racing start/stop.
with _state_lock:
jobs = list(_jobs.values())
if job_type:
jobs = [j for j in jobs if j.get("job_type") == job_type]
jobs.sort(key=lambda j: j["created_at"], reverse=True)
return [_public_job(j) for j in jobs]
@app.get("/api/jobs/{job_id}")
def get_job(job_id: str) -> dict[str, Any]:
with _state_lock:
job = _jobs.get(job_id)
if job is None:
raise HTTPException(status_code=404, detail="Job not found")
return _public_job(job)
@app.post("/api/jobs", status_code=201)
def create_job(req: CreateJobRequest) -> dict[str, Any]:
job = _new_job_dict(req)
with _state_lock:
_jobs[job["id"]] = job
if _settings.get("autoStartJob"):
_start(job)
return _public_job(job)
def _start(job: dict[str, Any]) -> None:
job["status"] = "running"
job["started_at"] = time.time()
job["finished_at"] = None
job["error"] = None
job["output_path"] = None
# The real server assigns the log path once the job starts running; mirror
# that so the Job Details "Download Log" button (gated on log_file_path) works.
job["log_file_path"] = f"/mock/logs/{job['id']}.log"
job["progress"] = 0.0
job["progress_msg"] = ""
job["phase"] = "starting"
@app.post("/api/jobs/{job_id}/start")
def start_job(job_id: str) -> dict[str, Any]:
with _state_lock:
job = _jobs.get(job_id)
if job is None:
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
_advance_job(job)
if job["status"] == "running":
raise HTTPException(status_code=409, detail="Job is already running")
if job["status"] == "completed":
raise HTTPException(status_code=409, detail="Job already completed. Delete and re-create to run again.")
_start(job)
return _public_job(job)
@app.post("/api/jobs/{job_id}/stop")
def stop_job(job_id: str) -> dict[str, Any]:
with _state_lock:
job = _jobs.get(job_id)
if job is None:
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
_advance_job(job)
if job["status"] != "running":
raise HTTPException(status_code=409, detail=f"Job is not running (status={job['status']})")
job["status"] = "stopped"
job["finished_at"] = time.time()
job["phase"] = "stopped"
return dict(job)
@app.delete("/api/jobs/{job_id}")
def delete_job(job_id: str) -> dict[str, str]:
with _state_lock:
if _jobs.pop(job_id, None) is None:
raise HTTPException(status_code=404, detail="Job not found")
return {"detail": f"Job {job_id} deleted"}
@app.get("/api/jobs/{job_id}/logs")
def get_job_logs(job_id: str, after: int = 0) -> dict[str, Any]:
with _state_lock:
job = _jobs.get(job_id)
if job is None:
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
_advance_job(job)
result = _compute_logs(job)
all_lines = result["lines"]
return {
"lines": all_lines[after:],
"total": len(all_lines),
"progress": result["progress"],
"progress_msg": result["progress_msg"],
"phase": result["phase"],
}
@app.get("/api/jobs/{job_id}/video")
def get_video(job_id: str) -> FileResponse:
with _state_lock:
job = _jobs.get(job_id)
if job is None:
raise HTTPException(status_code=404, detail="Job not found")
_advance_job(job)
output_path = job.get("output_path")
if job["status"] != "completed" or not output_path:
raise HTTPException(status_code=404, detail="No output available for this job")
# Mirror the real server: image workloads (t2i) output a .png served as an
# image; everything else is a video.
if output_path.endswith(".png"):
return FileResponse(_build_mock_media("png"), media_type="image/png", filename=f"job_{job_id}.png")
return FileResponse(_build_mock_media("mp4"), media_type="video/mp4", filename=f"job_{job_id}.mp4")
@app.get("/api/jobs/{job_id}/download_log")
def download_log(job_id: str) -> PlainTextResponse:
with _state_lock:
job = _jobs.get(job_id)
if job is None:
raise HTTPException(status_code=404, detail="Job not found")
_advance_job(job)
lines = _compute_logs(job)["lines"]
return PlainTextResponse("\n".join(lines) + "\n", media_type="text/plain")
# --- Datasets ---------------------------------------------------------------
@app.get("/api/datasets")
def list_datasets() -> list[dict[str, Any]]:
with _state_lock:
datasets = sorted(_datasets.values(), key=lambda d: d["created_at"], reverse=True)
return [_public_dataset(d) for d in datasets]
@app.get("/api/datasets/{dataset_id}")
def get_dataset(dataset_id: str) -> dict[str, Any]:
with _state_lock:
dataset = _datasets.get(dataset_id)
if dataset is None:
raise HTTPException(status_code=404, detail="Dataset not found")
return _public_dataset(dataset)
@app.post("/api/datasets", status_code=201)
def create_dataset(req: CreateDatasetRequest) -> dict[str, Any]:
if not req.upload_path:
raise HTTPException(status_code=400, detail="upload_path is required. Upload media files first.")
if not req.file_names:
raise HTTPException(status_code=400, detail="No media files found.")
dataset_id = str(uuid.uuid4())
dataset = {"id": dataset_id, "name": req.name, "created_at": time.time()}
captions = {fn: (req.captions.get(fn, "") if req.captions else "") for fn in req.file_names}
with _state_lock:
_datasets[dataset_id] = dataset
_dataset_files[dataset_id] = {"file_names": list(req.file_names), "captions": captions}
return _public_dataset(dataset)
@app.get("/api/datasets/{dataset_id}/files")
def get_dataset_files(dataset_id: str) -> dict[str, Any]:
with _state_lock:
if dataset_id not in _datasets:
raise HTTPException(status_code=404, detail="Dataset not found")
files = _dataset_files.get(dataset_id, {"file_names": [], "captions": {}})
return {"file_names": list(files["file_names"]), "captions": dict(files["captions"])}
@app.put("/api/datasets/{dataset_id}/captions")
def update_dataset_caption(dataset_id: str, req: UpdateCaptionRequest) -> dict[str, str]:
with _state_lock:
if dataset_id not in _datasets:
raise HTTPException(status_code=404, detail="Dataset not found")
files = _dataset_files.setdefault(dataset_id, {"file_names": [], "captions": {}})
files["captions"][req.file_name] = req.caption
return {"detail": "Caption updated"}
@app.get("/api/datasets/{dataset_id}/media/{file_name:path}")
def serve_dataset_media(dataset_id: str, file_name: str) -> FileResponse:
with _state_lock:
if dataset_id not in _datasets:
raise HTTPException(status_code=404, detail="Dataset not found")
if file_name not in _dataset_files.get(dataset_id, {}).get("file_names", []):
raise HTTPException(status_code=404, detail="File not found")
return FileResponse(_build_mock_media("mp4"), media_type="video/mp4")
@app.delete("/api/datasets/{dataset_id}")
def delete_dataset(dataset_id: str) -> dict[str, str]:
with _state_lock:
if _datasets.pop(dataset_id, None) is None:
raise HTTPException(status_code=404, detail="Dataset not found")
_dataset_files.pop(dataset_id, None)
return {"detail": f"Dataset {dataset_id} deleted"}
def main() -> None:
import argparse
import uvicorn
parser = argparse.ArgumentParser(description="FastVideo Studio mock API server")
parser.add_argument("--host", default="127.0.0.1", help="Bind address (default: 127.0.0.1)")
# Default off the real server's 8189 so the mock never shadows a real API.
parser.add_argument("--port", type=int, default=8190, help="Port number (default: 8190)")
args = parser.parse_args()
uvicorn.run(app, host=args.host, port=args.port, log_level="info")
if __name__ == "__main__":
main()
+1 -9
View File
@@ -1,22 +1,14 @@
# SPDX-License-Identifier: Apache-2.0
"""Pydantic request/response models for the API, plus tiny shared helpers
usable by both the real server and the dependency-light mock server."""
"""Pydantic request/response models for the API."""
from fastvideo_studio.models.create_job_request import CreateJobRequest
from fastvideo_studio.models.settings_update import SettingsUpdate
from fastvideo_studio.models.create_dataset_request import CreateDatasetRequest
from fastvideo_studio.models.update_caption_request import UpdateCaptionRequest
def model_label(model_path: str) -> str:
"""Derive a readable label from an HF-style model path."""
return model_path.split("/")[-1].replace("-", " ").replace("_", " ")
__all__ = [
"CreateJobRequest",
"SettingsUpdate",
"CreateDatasetRequest",
"UpdateCaptionRequest",
"model_label",
]
@@ -17,6 +17,7 @@ class CreateJobRequest(BaseModel):
num_latent_t: int = 20
validation_dataset_file: str = ""
lora_rank: int = 32
ltx2_first_frame_conditioning_p: float | None = None
negative_prompt: str = ""
num_inference_steps: int = 50
num_frames: int = 81
@@ -40,6 +41,8 @@ class CreateJobRequest(BaseModel):
dmd_use_vsa: bool = False
dmd_vsa_sparsity: float = 0.8
dmd_denoising_steps: str = "1000,757,522"
min_timestep_ratio: float = 0.02
max_timestep_ratio: float = 0.98
real_score_guidance_scale: float = 3.5
generator_update_interval: int = 5
real_score_model_path: str = ""
-13
View File
@@ -1,13 +0,0 @@
import path from 'node:path';
import { fileURLToPath } from 'node:url';
import type { NextConfig } from 'next';
const configDir = path.dirname(fileURLToPath(import.meta.url));
const nextConfig: NextConfig = {
// Point tracing at the monorepo root (apps/fastvideo_studio -> apps -> repo
// root) so Next stops warning about multiple lockfiles in the workspace.
outputFileTracingRoot: path.join(configDir, '..', '..'),
};
export default nextConfig;
+1082 -5343
View File
File diff suppressed because it is too large Load Diff
+22 -44
View File
@@ -4,55 +4,33 @@
"private": true,
"type": "module",
"scripts": {
"dev": "next dev --port 3000",
"build": "next build",
"start": "next start --port 3000",
"typecheck": "tsc --noEmit",
"test": "vitest run",
"test:watch": "vitest",
"test:coverage": "vitest run --coverage",
"e2e": "playwright test",
"dev": "vite dev",
"build": "vite build",
"preview": "vite preview",
"start": "concurrently --kill-others-on-fail \"npm:start:api\" \"npm:start:web\"",
"start:api": "cd .. && python -m fastvideo_studio.server",
"start:web": "next start --port 3000",
"start:all": "concurrently --kill-others-on-fail \"npm:start:api\" \"npm:start:web\""
"start:web": "node build",
"lint": "eslint ."
},
"dependencies": {
"@radix-ui/react-dialog": "^1.1.0",
"@radix-ui/react-dropdown-menu": "^2.1.24",
"@radix-ui/react-label": "^2.1.8",
"@radix-ui/react-scroll-area": "^1.2.10",
"@radix-ui/react-select": "^2.2.6",
"@radix-ui/react-separator": "^1.1.8",
"@radix-ui/react-slider": "^1.2.0",
"@radix-ui/react-slot": "^1.2.4",
"@radix-ui/react-switch": "^1.1.0",
"@radix-ui/react-tabs": "^1.1.0",
"class-variance-authority": "^0.7.1",
"clsx": "^2.1.1",
"lucide-react": "^0.577.0",
"next": "15.5.18",
"react": "^19.1.0",
"react-dom": "^19.1.0",
"sonner": "^2.0.7",
"tailwind-merge": "^3.5.0"
"@sveltejs/kit": "2.57.1",
"svelte": "5.55.7"
},
"devDependencies": {
"@playwright/test": "^1.59.1",
"@tailwindcss/postcss": "^4.2.1",
"@testing-library/dom": "^10.4.1",
"@testing-library/jest-dom": "^6.9.1",
"@testing-library/react": "^16.3.0",
"@testing-library/user-event": "^14.6.1",
"@types/node": "^22",
"@types/react": "^19.1.0",
"@types/react-dom": "^19.1.0",
"@vitejs/plugin-react": "^4.5.2",
"@vitest/coverage-v8": "^3.2.4",
"@sveltejs/adapter-node": "^5.5.4",
"@sveltejs/vite-plugin-svelte": "^5.0.0",
"@types/node": "^20",
"concurrently": "^9",
"jsdom": "^26.1.0",
"postcss": "8.5.10",
"tailwindcss": "^4.2.1",
"typescript": "^5.8.3",
"vitest": "^3.2.4"
"eslint": "^9",
"typescript": "^5",
"vite": "6.4.2"
},
"overrides": {
"cookie": "0.7.0",
"devalue": "5.8.1",
"flatted": "3.4.2",
"minimatch": "3.1.4",
"picomatch": "4.0.4",
"postcss": "8.5.10"
}
}
@@ -1,54 +0,0 @@
import { defineConfig, devices } from '@playwright/test';
import { API_BASE, MOCK_API_PORT } from './e2e/helpers';
/**
* Playwright config for FastVideo Studio end-to-end tests.
*
* The Next.js frontend runs on port 3000 and talks to the in-memory mock API
* (fastvideo_studio.mock_server) on port 8189. The webServer block boots both
* the mock backend and `npm run dev` (pointed at the mock via
* NEXT_PUBLIC_API_BASE_URL) if nothing is already listening, so the suite works
* both locally and in CI. Set PLAYWRIGHT_SKIP_WEBSERVER to reuse externally
* managed servers.
*/
export default defineConfig({
testDir: './e2e',
fullyParallel: false,
workers: 1,
forbidOnly: !!process.env.CI,
retries: process.env.CI ? 2 : 0,
reporter: process.env.CI ? 'github' : 'list',
timeout: 120_000,
expect: { timeout: 30_000 },
use: {
baseURL: process.env.PLAYWRIGHT_BASE_URL ?? 'http://127.0.0.1:3000',
headless: true,
viewport: { width: 1280, height: 720 },
screenshot: 'only-on-failure',
trace: 'retain-on-failure',
},
projects: [{ name: 'chromium', use: { ...devices['Desktop Chrome'] } }],
webServer: process.env.PLAYWRIGHT_SKIP_WEBSERVER
? undefined
: [
{
command: `PYTHONPATH=.. python -m fastvideo_studio.mock_server --port ${MOCK_API_PORT}`,
// Probe the mock-only sentinel, not /api/models (which a real server
// also serves), so a real backend on this port is never mistaken for
// the mock. Always boot a fresh mock in CI.
url: `http://127.0.0.1:${MOCK_API_PORT}/api/__mock__`,
reuseExistingServer: !process.env.CI,
timeout: 120_000,
},
{
command: 'npm run dev',
url: 'http://127.0.0.1:3000',
reuseExistingServer: true,
timeout: 120_000,
env: {
NEXT_PUBLIC_API_BASE_URL: API_BASE,
},
},
],
});
-5
View File
@@ -1,5 +0,0 @@
export default {
plugins: {
'@tailwindcss/postcss': {},
},
};
+142
View File
@@ -0,0 +1,142 @@
# SPDX-License-Identifier: Apache-2.0
"""
Runs preprocessing subprocess for datasets.
"""
from __future__ import annotations
import logging
import os
import subprocess
import sys
import threading
from pathlib import Path
from collections.abc import Callable
logger = logging.getLogger("fastvideo.studio.preprocess_runner")
def build_preprocess_args(
dataset_id: str,
raw_path: str,
output_dir: str,
workload_type: str,
model_path: str,
dataset_type: str = "merged",
num_gpus: int = 1,
) -> list[str]:
"""Build CLI args for v1_preprocessing_new."""
return [
"--model-path",
model_path,
"--mode",
"preprocess",
"--workload-type",
workload_type,
"--preprocess.video_loader_type",
"torchvision",
"--preprocess.dataset_type",
dataset_type,
"--preprocess.dataset_path",
raw_path,
"--preprocess.dataset_output_dir",
output_dir,
"--preprocess.preprocess_video_batch_size",
"2",
"--preprocess.dataloader_num_workers",
"0",
"--preprocess.max_height",
"480",
"--preprocess.max_width",
"832",
"--preprocess.num_frames",
"77",
"--preprocess.train_fps",
"16",
"--preprocess.samples_per_file",
"8",
"--preprocess.flush_frequency",
"8",
"--preprocess.video_length_tolerance_range",
"5",
]
def run_preprocess(
dataset_id: str,
raw_path: str,
output_dir: str,
workload_type: str,
model_path: str,
dataset_type: str,
num_gpus: int,
log_file_path: str,
on_status_change: Callable[[str, str | None], None],
stop_event: threading.Event,
) -> None:
"""Run preprocessing subprocess. Calls on_status_change(status, error)."""
repo_root = Path(__file__).resolve().parent.parent
preprocess_module = "fastvideo.pipelines.preprocess.v1_preprocessing_new"
args = build_preprocess_args(
dataset_id=dataset_id,
raw_path=raw_path,
output_dir=output_dir,
workload_type=workload_type,
model_path=model_path,
dataset_type=dataset_type,
num_gpus=num_gpus,
)
cmd = [
sys.executable,
"-m",
"torch.distributed.run",
"--nproc_per_node",
str(num_gpus),
"--nnodes",
"1",
"-m",
preprocess_module,
] + args
env = os.environ.copy()
env["TOKENIZERS_PARALLELISM"] = "false"
try:
on_status_change("preprocessing", None)
with open(log_file_path, "w", encoding="utf-8") as log_file:
proc = subprocess.Popen(
cmd,
cwd=str(repo_root),
env=env,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
text=True,
bufsize=1,
)
assert proc.stdout is not None
for line in iter(proc.stdout.readline, ""):
if stop_event.is_set():
proc.terminate()
try:
proc.wait(timeout=30)
except subprocess.TimeoutExpired:
proc.kill()
on_status_change("stopped", "Preprocessing stopped by user")
return
line = line.rstrip()
if line:
log_file.write(line + "\n")
log_file.flush()
proc.wait()
exit_code = proc.returncode or 0
if exit_code == 0:
on_status_change("ready", None)
else:
on_status_change("failed", f"Preprocessing exited with code {exit_code}")
except Exception as exc:
on_status_change("failed", f"{type(exc).__name__}: {exc}")
logger.exception("Preprocessing failed for dataset %s", dataset_id)
+20 -54
View File
@@ -32,10 +32,8 @@ from fastapi.responses import FileResponse
from fastvideo.registry import (get_registered_model_paths, get_registered_models_with_workloads)
from fastvideo_studio.database import Database, _get_db_path
from fastvideo_studio.gpu import get_gpu_snapshot
from fastvideo_studio.job_runner import JobRunner, JobStatus
from fastvideo_studio.models import (CreateDatasetRequest, CreateJobRequest, SettingsUpdate, UpdateCaptionRequest,
model_label)
from fastvideo_studio.models import (CreateDatasetRequest, CreateJobRequest, SettingsUpdate, UpdateCaptionRequest)
logging.basicConfig(
level=logging.INFO,
@@ -45,9 +43,15 @@ logger = logging.getLogger("fastvideo.studio.api")
DEFAULT_OUTPUT_DIR = os.path.join(os.path.dirname(__file__), "..", "outputs", "ui_jobs")
def _get_model_label(model_path: str) -> str:
"""Derive a readable label from a HF model path."""
return model_path.split("/")[-1].replace("-", " ").replace("_", " ")
_available_models: list[dict[str, str]] = [{
"id": path,
"label": model_label(path)
"label": _get_model_label(path)
} for path in get_registered_model_paths()]
job_runner: JobRunner
@@ -96,12 +100,6 @@ def update_settings(settings: SettingsUpdate) -> dict[str, Any]:
return database.get_settings()
@app.get("/api/gpus")
def list_gpus() -> dict[str, Any]:
"""Return an NVML snapshot of every GPU on the API server host."""
return get_gpu_snapshot()
@app.get("/api/models")
def list_models(workload_type: str | None = None) -> list[dict[str, Any]]:
"""Return the catalogue of available video-generation models.
@@ -150,7 +148,7 @@ async def upload_image(file: Annotated[UploadFile, File()], ) -> dict[str, str]:
return {"path": os.path.abspath(dest_path)}
ALLOWED_VIDEO_EXTENSIONS = {".mp4", ".webm", ".avi", ".mov", ".mkv"}
ALLOWED_VIDEO_EXTENSIONS = {".mp4", ".webm", ".avi", ".mov"}
def _filter_video_files(files: list[UploadFile]) -> list[UploadFile]:
@@ -158,23 +156,6 @@ def _filter_video_files(files: list[UploadFile]) -> list[UploadFile]:
return [f for f in files if Path(f.filename or "").suffix.lower() in ALLOWED_VIDEO_EXTENSIONS]
def _path_is_within(child: str, parent: str) -> bool:
"""True if ``child`` resolves to a location inside ``parent``."""
try:
parent_real = os.path.realpath(parent)
return os.path.commonpath([os.path.realpath(child), parent_real]) == parent_real
except (ValueError, OSError):
return False
def _staging_base_path() -> str:
"""Root under which raw uploads are staged (settings override or default)."""
settings = database.get_settings() if database is not None else {}
raw_path = (settings.get("datasetUploadPath") or settings.get("dataset_upload_path") or "")
base_path = (raw_path.strip() if raw_path and isinstance(raw_path, str) else "")
return datasets_upload_dir if not base_path else os.path.abspath(base_path)
@app.post("/api/upload-raw-dataset")
async def upload_raw_dataset(files: Annotated[list[UploadFile], File()], ) -> dict[str, Any]:
"""
@@ -186,7 +167,10 @@ async def upload_raw_dataset(files: Annotated[list[UploadFile], File()], ) -> di
status_code=503,
detail="Database not initialized",
)
base_path = _staging_base_path()
settings = database.get_settings()
raw_path = (settings.get("datasetUploadPath") or settings.get("dataset_upload_path") or "")
base_path = (raw_path.strip() if raw_path and isinstance(raw_path, str) else "")
base_path = (datasets_upload_dir if not base_path else os.path.abspath(base_path))
if not base_path:
raise HTTPException(
status_code=503,
@@ -266,19 +250,6 @@ def create_job(req: CreateJobRequest) -> dict[str, Any]:
f"Valid options: {sorted(valid_ids)}"),
)
# Training jobs reference a dataset by id; resolve it to the on-disk media
# directory the trainer reads (the UI has no free-text path field). Falls
# through unchanged for inference and for anything already a real path.
def _resolve_dataset_path(value: str) -> str:
media_dir = _dataset_media_dir(value) if value else ""
return media_dir if media_dir and os.path.isdir(media_dir) else value
data_path = req.data_path or ""
validation_dataset_file = req.validation_dataset_file or ""
if job_type != "inference":
data_path = _resolve_dataset_path(data_path)
validation_dataset_file = _resolve_dataset_path(validation_dataset_file)
job = job_runner.create_job(
job_id=str(uuid.uuid4()),
model_id=req.model_id,
@@ -286,13 +257,14 @@ def create_job(req: CreateJobRequest) -> dict[str, Any]:
workload_type=req.workload_type or "t2v",
job_type=job_type,
image_path=req.image_path or "",
data_path=data_path,
data_path=req.data_path or "",
max_train_steps=req.max_train_steps,
train_batch_size=req.train_batch_size,
learning_rate=req.learning_rate,
num_latent_t=req.num_latent_t,
validation_dataset_file=validation_dataset_file,
validation_dataset_file=req.validation_dataset_file or "",
lora_rank=req.lora_rank,
ltx2_first_frame_conditioning_p=req.ltx2_first_frame_conditioning_p,
negative_prompt=req.negative_prompt,
num_inference_steps=req.num_inference_steps,
num_frames=req.num_frames,
@@ -315,6 +287,8 @@ def create_job(req: CreateJobRequest) -> dict[str, Any]:
dmd_use_vsa=req.dmd_use_vsa,
dmd_vsa_sparsity=req.dmd_vsa_sparsity,
dmd_denoising_steps=req.dmd_denoising_steps or "1000,757,522",
min_timestep_ratio=req.min_timestep_ratio,
max_timestep_ratio=req.max_timestep_ratio,
real_score_guidance_scale=req.real_score_guidance_scale,
generator_update_interval=req.generator_update_interval,
real_score_model_path=req.real_score_model_path or "",
@@ -444,13 +418,6 @@ def create_dataset(req: CreateDatasetRequest) -> dict[str, Any]:
status_code=400,
detail="No media files found. Ensure at least one image or video.",
)
# upload_path must be a staging dir produced by /api/upload-raw-dataset,
# not an arbitrary client path — the code below copies then rmtrees it.
if not _path_is_within(req.upload_path, _staging_base_path()):
raise HTTPException(
status_code=400,
detail="upload_path must be a staged upload directory.",
)
dataset_id = str(uuid.uuid4())
created_at = time.time()
dest_dir = os.path.join(datasets_upload_dir, dataset_id)
@@ -524,9 +491,8 @@ def serve_dataset_media(dataset_id: str, file_name: str) -> FileResponse:
ds = database.get_dataset(dataset_id)
if ds is None:
raise HTTPException(status_code=404, detail="Dataset not found")
media_dir = _dataset_media_dir(dataset_id)
media_path = os.path.realpath(os.path.join(media_dir, file_name))
if not (_path_is_within(media_path, media_dir) and os.path.isfile(media_path)):
media_path = os.path.join(_dataset_media_dir(dataset_id), file_name)
if not os.path.isfile(media_path):
raise HTTPException(status_code=404, detail="File not found")
import mimetypes
mime, _ = mimetypes.guess_type(media_path)
+66
View File
@@ -0,0 +1,66 @@
/* CSS Variables */
:root {
--header-height: 74px;
--bg: #0f1117;
--surface: #1a1d27;
--border: #2a2d3a;
--text: #e4e4e7;
--text-dim: #9ca3af;
--accent: #356cff;
--accent-h: #818cf8;
--green: #22c55e;
--red: #ef4444;
--yellow: #eab308;
--blue: #3b82f6;
--radius: 8px;
--shadow: 0 2px 8px rgba(0, 0, 0, 0.35);
}
/* Reset */
*,
*::before,
*::after {
box-sizing: border-box;
margin: 0;
padding: 0;
}
/* Base body styles */
body {
font-family:
"Inter",
system-ui,
-apple-system,
sans-serif;
background: var(--bg);
color: var(--text);
line-height: 1.6;
min-height: 100vh;
display: flex;
flex-direction: column;
}
/* Base form input styles (used globally) */
select,
input,
textarea {
background: var(--bg);
color: var(--text);
border: 1px solid var(--border);
border-radius: var(--radius);
padding: 0.55rem 0.75rem;
font-size: 0.9rem;
font-family: inherit;
outline: none;
transition: border-color 0.15s;
}
select:focus,
input:focus,
textarea:focus {
border-color: var(--accent);
}
textarea {
resize: vertical;
}
+13
View File
@@ -0,0 +1,13 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<link rel="icon" href="%sveltekit.assets%/fastvideo.ico" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>FastVideo</title>
%sveltekit.head%
</head>
<body class="antialiased" data-sveltekit-preload-data="hover">
<div style="display: contents">%sveltekit.body%</div>
</body>
</html>
@@ -1,64 +0,0 @@
import { act, fireEvent, render, screen } from '@testing-library/react';
import { describe, expect, it, vi } from 'vitest';
import { HeaderActionsProvider } from '@/components/shell/HeaderActionsContext';
import { getDatasets, type Dataset } from '@/lib/api';
import DatasetsPage from './page';
vi.mock('@/lib/api', () => ({
getDatasets: vi.fn(),
}));
vi.mock('@/components/datasets/AddDatasetButton', () => ({
default: () => null,
}));
vi.mock('@/components/datasets/CreateDatasetModal', () => ({
default: () => null,
}));
vi.mock('@/components/datasets/DatasetCard', () => ({
default: ({ dataset }: { dataset: Dataset }) => <div>{dataset.name}</div>,
}));
function renderPage() {
return render(
<HeaderActionsProvider>
<DatasetsPage />
</HeaderActionsProvider>,
);
}
describe('DatasetsPage', () => {
it('shows loading content before the initial request settles', async () => {
let resolveDatasets: (datasets: Dataset[]) => void = () => {};
vi.mocked(getDatasets).mockReturnValue(
new Promise<Dataset[]>((resolve) => {
resolveDatasets = resolve;
}),
);
renderPage();
expect(screen.getByLabelText('Loading datasets')).toBeInTheDocument();
expect(screen.queryByText('No datasets yet.')).not.toBeInTheDocument();
act(() => resolveDatasets([]));
expect(await screen.findByText('No datasets yet.')).toBeInTheDocument();
});
it('shows API failures separately from an empty list and retries', async () => {
vi.spyOn(console, 'error').mockImplementation(() => {});
vi.mocked(getDatasets).mockRejectedValueOnce(new Error('network down'));
renderPage();
expect(
await screen.findByText(/Could not load datasets from the Studio API/),
).toBeInTheDocument();
expect(screen.queryByText('No datasets yet.')).not.toBeInTheDocument();
vi.mocked(getDatasets).mockResolvedValueOnce([]);
fireEvent.click(screen.getByRole('button', { name: 'Try Again' }));
expect(await screen.findByText('No datasets yet.')).toBeInTheDocument();
});
});
@@ -1,136 +0,0 @@
'use client';
import * as React from 'react';
import { AlertTriangle } from 'lucide-react';
import AddDatasetButton from '@/components/datasets/AddDatasetButton';
import CreateDatasetModal from '@/components/datasets/CreateDatasetModal';
import DatasetCard from '@/components/datasets/DatasetCard';
import { HeaderActions } from '@/components/shell/HeaderActionsContext';
import { Card } from '@/components/ui/card';
import { Button } from '@/components/ui/button';
import { useStore } from '@/hooks/useStore';
import { getDatasets } from '@/lib/api';
import type { Dataset } from '@/lib/api';
import {
setActiveDataset,
setActiveDatasetId,
} from '@/stores/activeDataset';
import {
createDatasetModalStore,
setCreateDatasetModalOpen,
} from '@/stores/createDatasetModalOpen';
export default function DatasetsPage() {
const [datasets, setDatasets] = React.useState<Dataset[]>([]);
const [isInitialLoading, setIsInitialLoading] = React.useState(true);
const [error, setError] = React.useState<string | null>(null);
const { open } = useStore(createDatasetModalStore);
const fetchSequence = React.useRef(0);
const fetchDatasets = React.useCallback(async () => {
const sequence = ++fetchSequence.current;
try {
const next = await getDatasets();
if (sequence === fetchSequence.current) {
setDatasets(next);
setError(null);
}
} catch (err) {
console.error('Failed to fetch datasets:', err);
if (sequence === fetchSequence.current) {
setError(
'Could not load datasets from the Studio API. Check the server and try again.',
);
}
} finally {
if (sequence === fetchSequence.current) setIsInitialLoading(false);
}
}, []);
React.useEffect(() => {
fetchDatasets();
}, [fetchDatasets]);
function handleSelectDataset(ds: Dataset) {
setActiveDataset(ds);
setActiveDatasetId(ds.id);
}
return (
<>
<HeaderActions>
<AddDatasetButton />
</HeaderActions>
<div className="mx-auto flex w-full max-w-[850px] flex-col gap-6 px-4 pb-12">
<Card className="p-6">
<div aria-busy={isInitialLoading}>
{isInitialLoading ? (
<div
aria-label="Loading datasets"
className="flex flex-col gap-3 py-2"
>
{[0, 1, 2].map((item) => (
<div
key={item}
className="h-24 animate-pulse rounded-lg border border-border bg-muted/50"
/>
))}
</div>
) : error && datasets.length === 0 ? (
<div
role="alert"
className="flex flex-col items-center gap-3 py-8 text-center"
>
<AlertTriangle
className="size-6 text-destructive"
aria-hidden
/>
<p className="max-w-md text-sm text-muted-foreground">
{error}
</p>
<Button type="button" variant="outline" onClick={fetchDatasets}>
Try Again
</Button>
</div>
) : (
<>
{error && (
<p
role="status"
className="mb-3 rounded-lg border border-amber-500/50 bg-amber-500/10 px-3 py-2 text-sm text-foreground"
>
Dataset updates are temporarily unavailable. Showing the
most recent results.
</p>
)}
{datasets.length === 0 ? (
<p className="py-8 text-center text-muted-foreground">
No datasets yet.
</p>
) : (
datasets.map((ds) => (
<DatasetCard
key={ds.id}
dataset={ds}
onUpdated={fetchDatasets}
onSelect={() => handleSelectDataset(ds)}
/>
))
)}
</>
)}
</div>
</Card>
</div>
<CreateDatasetModal
isOpen={open}
onClose={() => setCreateDatasetModalOpen(false)}
onSuccess={() => {
fetchDatasets();
setCreateDatasetModalOpen(false);
}}
/>
</>
);
}
@@ -1,16 +0,0 @@
'use client';
import CreateJobButton from '@/components/jobs/CreateJobButton';
import { HeaderActions } from '@/components/shell/HeaderActionsContext';
import JobQueue from '@/components/jobs/JobQueue';
export default function DistillationPage() {
return (
<>
<HeaderActions>
<CreateJobButton jobType="distillation" />
</HeaderActions>
<JobQueue jobType="distillation" />
</>
);
}
@@ -1,21 +0,0 @@
'use client';
import CreateJobButton from '@/components/jobs/CreateJobButton';
import { HeaderActions } from '@/components/shell/HeaderActionsContext';
import JobQueue from '@/components/jobs/JobQueue';
import type { JobType } from '@/lib/types';
// 'lora' is a backend job_type (LoRA finetunes) that isn't part of the
// JobType union; the finetuning queue lists both alongside full finetunes.
const FINETUNING_LIST = ['finetuning', 'lora'] as JobType[];
export default function FinetuningPage() {
return (
<>
<HeaderActions>
<CreateJobButton jobType="finetuning" />
</HeaderActions>
<JobQueue jobType="finetuning" jobTypesForList={FINETUNING_LIST} />
</>
);
}
@@ -1,76 +0,0 @@
import { fireEvent, render, screen } from '@testing-library/react';
import { describe, expect, it, vi } from 'vitest';
import GalleryPage from './page';
import { HeaderActionsProvider } from '@/components/shell/HeaderActionsContext';
import { getJobsList } from '@/lib/api';
import type { Job } from '@/lib/types';
import { makeJob as makeBaseJob } from '@/test/factories';
vi.mock('@/lib/api', () => ({
getJobsList: vi.fn(),
getJobVideoUrl: (id: string) => `http://test.local/api/jobs/${id}/video`,
}));
const makeJob = (overrides: Partial<Job> = {}): Job =>
makeBaseJob({
model_id: 'wan',
prompt: 'a cat surfing a wave',
status: 'completed',
created_at: 1,
finished_at: 2,
output_path: '/out/clip.mp4',
...overrides,
});
function renderGallery() {
return render(
<HeaderActionsProvider>
<GalleryPage />
</HeaderActionsProvider>,
);
}
describe('GalleryPage', () => {
it('renders a grid item for a completed inference job', async () => {
vi.mocked(getJobsList).mockResolvedValue([
makeJob({ prompt: 'a cat surfing a wave' }),
]);
renderGallery();
expect(await screen.findByText('a cat surfing a wave')).toBeInTheDocument();
expect(getJobsList).toHaveBeenCalledWith('inference');
});
it('provides video controls and a visible fallback when media fails', async () => {
vi.mocked(getJobsList).mockResolvedValue([makeJob()]);
renderGallery();
const video = await screen.findByLabelText(
'Generated video: a cat surfing a wave',
);
expect(video).toHaveAttribute('controls');
fireEvent.error(video);
expect(screen.getByText('Preview unavailable')).toBeInTheDocument();
expect(
screen.getByText('The generated file could not be loaded.'),
).toBeInTheDocument();
});
it('shows the empty state when no completed videos exist', async () => {
vi.mocked(getJobsList).mockResolvedValue([
makeJob({ status: 'running', output_path: null }),
]);
renderGallery();
expect(
await screen.findByText('No completed videos yet'),
).toBeInTheDocument();
expect(
screen.queryByText('a cat surfing a wave'),
).not.toBeInTheDocument();
});
});
@@ -1,161 +0,0 @@
'use client';
import { AlertTriangle, ImageOff, Loader2 } from 'lucide-react';
import { useEffect, useState } from 'react';
import { Button } from '@/components/ui/button';
import { Card } from '@/components/ui/card';
import { getJobVideoUrl, getJobsList } from '@/lib/api';
import type { Job } from '@/lib/types';
function isImage(job: Job): boolean {
return job.output_path?.toLowerCase().endsWith('.png') ?? false;
}
function GalleryMedia({ job }: { job: Job }) {
const [failed, setFailed] = useState(false);
if (failed) {
return (
<div
role="status"
className="flex h-full flex-col items-center justify-center gap-2 px-4 text-center text-muted-foreground"
>
<ImageOff className="size-7" aria-hidden />
<span className="text-sm font-medium">Preview unavailable</span>
<span className="text-xs">
The generated file could not be loaded.
</span>
</div>
);
}
if (isImage(job)) {
return (
// eslint-disable-next-line @next/next/no-img-element
<img
src={getJobVideoUrl(job.id)}
alt={job.prompt}
className="block h-full w-full object-contain"
loading="lazy"
onError={() => setFailed(true)}
/>
);
}
return (
<video
src={getJobVideoUrl(job.id)}
aria-label={
job.prompt ? `Generated video: ${job.prompt}` : 'Generated video'
}
className="block h-full w-full object-contain"
controls
muted
loop
playsInline
preload="metadata"
onError={() => setFailed(true)}
/>
);
}
export default function GalleryPage() {
const [jobs, setJobs] = useState<Job[]>([]);
const [isLoading, setIsLoading] = useState(true);
const [error, setError] = useState<string | null>(null);
const [reloadKey, setReloadKey] = useState(0);
useEffect(() => {
let cancelled = false;
async function load() {
try {
const list = await getJobsList('inference');
if (cancelled) return;
const sorted = [...list].sort(
(a, b) =>
(b.finished_at ?? b.created_at ?? 0) -
(a.finished_at ?? a.created_at ?? 0),
);
setJobs(sorted);
} catch (e) {
if (!cancelled) {
setError(e instanceof Error ? e.message : 'Failed to load jobs');
}
} finally {
if (!cancelled) setIsLoading(false);
}
}
void load();
return () => {
cancelled = true;
};
}, [reloadKey]);
function retry() {
setError(null);
setIsLoading(true);
setReloadKey((k) => k + 1);
}
const galleryJobs = jobs.filter(
(j) =>
j.status === 'completed' &&
j.output_path &&
(j.job_type === 'inference' || !j.job_type),
);
return (
<div className="mx-auto w-full max-w-[1200px] px-4 pb-12 pt-4">
<Card className="p-6">
<h2 className="mb-1 text-2xl font-semibold text-foreground">Gallery</h2>
<p className="mb-6 text-sm text-muted-foreground">
Generated videos from completed inference jobs. Captions show the
prompt used for each generation.
</p>
{isLoading ? (
<div className="flex items-center gap-3 p-8 text-muted-foreground">
<Loader2 className="h-6 w-6 animate-spin text-primary" />
<span>Loading gallery…</span>
</div>
) : error ? (
<div
role="alert"
className="flex flex-col items-center gap-3 py-8 text-center"
>
<AlertTriangle className="size-6 text-destructive" aria-hidden />
<p className="max-w-md text-sm text-muted-foreground">{error}</p>
<Button type="button" variant="outline" onClick={retry}>
Try Again
</Button>
</div>
) : galleryJobs.length === 0 ? (
<p className="py-8 text-center text-muted-foreground">
No completed videos yet
</p>
) : (
<div className="grid grid-cols-[repeat(auto-fill,minmax(280px,1fr))] gap-5">
{galleryJobs.map((job) => (
<article
key={job.id}
className="flex flex-col overflow-hidden rounded-lg border border-border bg-background"
>
<div className="relative aspect-video overflow-hidden bg-muted">
<GalleryMedia job={job} />
</div>
<p
className="line-clamp-3 border-t border-border px-4 py-3 text-sm text-muted-foreground"
title={job.prompt}
>
{job.prompt || '—'}
</p>
</article>
))}
</div>
)}
</Card>
</div>
);
}
-238
View File
@@ -1,238 +0,0 @@
@import "tailwindcss";
@custom-variant dark (&:is(.dark *));
/*
* FastVideo Studio theme — shared with the Dreamverse design system:
* slate light/dark palettes, #356cff accent blue, IBM Plex type. The studio
* defaults to dark (see the theme init script in layout.tsx); the toggle in
* the header persists the choice to localStorage.
*/
:root {
color-scheme: light;
--header-height: 74px;
--accent-blue: #356cff;
--background: #f5f4f4;
--foreground: #0f172a;
--card: #ffffff;
--card-foreground: #0f172a;
--popover: #ffffff;
--popover-foreground: #0f172a;
--primary: #0f172a;
--primary-foreground: #f8fafc;
--secondary: #f1f5f9;
--secondary-foreground: #1e293b;
--muted: #f1f5f9;
--muted-foreground: #64748b;
--accent: #e2e8f0;
--accent-foreground: #0f172a;
--destructive: #ef4444;
--destructive-foreground: #ffffff;
--border: #e2e8f0;
--input: #cbd5e1;
--ring: #1d4ed8;
--radius: 0.5rem;
}
.dark {
color-scheme: dark;
--accent-blue: #356cff;
--background: #0f172a;
--foreground: #e2e8f0;
--card: #0f172a;
--card-foreground: #f1f5f9;
--popover: #0f172a;
--popover-foreground: #f1f5f9;
--primary: #f1f5f9;
--primary-foreground: #0f172a;
--secondary: #1e293b;
--secondary-foreground: #f1f5f9;
--muted: #1e293b;
--muted-foreground: #94a3b8;
--accent: #1e293b;
--accent-foreground: #f1f5f9;
--destructive: #991b1b;
--destructive-foreground: #fecaca;
--border: #334155;
--input: #334155;
--ring: #7dd3fc;
}
@theme inline {
--font-sans: var(--font-plex-sans), ui-sans-serif, system-ui, sans-serif;
--font-mono: var(--font-plex-mono), ui-monospace, monospace;
--color-accent-blue: var(--accent-blue);
--color-background: var(--background);
--color-foreground: var(--foreground);
--color-card: var(--card);
--color-card-foreground: var(--card-foreground);
--color-popover: var(--popover);
--color-popover-foreground: var(--popover-foreground);
--color-primary: var(--primary);
--color-primary-foreground: var(--primary-foreground);
--color-secondary: var(--secondary);
--color-secondary-foreground: var(--secondary-foreground);
--color-muted: var(--muted);
--color-muted-foreground: var(--muted-foreground);
--color-accent: var(--accent);
--color-accent-foreground: var(--accent-foreground);
--color-destructive: var(--destructive);
--color-destructive-foreground: var(--destructive-foreground);
--color-border: var(--border);
--color-input: var(--input);
--color-ring: var(--ring);
--radius-sm: calc(var(--radius) - 4px);
--radius-md: calc(var(--radius) - 2px);
--radius-lg: var(--radius);
--radius-xl: calc(var(--radius) + 4px);
}
/* ——— Resets ——— */
html,
body {
min-height: 100%;
overflow-x: clip;
}
html {
background: var(--background);
}
body {
margin: 0;
line-height: 1.6;
color: var(--foreground);
font-family: var(--font-plex-sans), ui-sans-serif, system-ui, sans-serif;
background: var(--background);
}
/* Dreamverse's dark-mode backdrop: subtle radial glows over a deep fade. */
body::after {
content: "";
position: fixed;
inset: 0;
z-index: -1;
opacity: 0;
pointer-events: none;
transition: opacity 300ms ease;
background:
radial-gradient(circle at top, rgba(56, 189, 248, 0.14), transparent 36%),
radial-gradient(circle at right top, rgba(129, 140, 248, 0.12), transparent 28%),
linear-gradient(180deg, #020617 0%, #000000 100%);
background-attachment: fixed;
}
.dark body::after {
opacity: 1;
}
a {
color: inherit;
}
:where(
a,
button,
input,
textarea,
select,
summary,
[role="button"],
[role="menuitem"],
[role="slider"],
[tabindex]
):focus-visible {
outline: 3px solid var(--ring) !important;
outline-offset: 2px !important;
}
summary {
list-style: none;
}
summary::-webkit-details-marker {
display: none;
}
::selection {
background: rgba(56, 189, 248, 0.25);
}
.dark ::selection {
background: rgba(56, 189, 248, 0.35);
color: #ffffff;
}
/* Smooth the light/dark switch (class applied briefly by the theme toggle). */
html.theme-transition,
html.theme-transition *,
html.theme-transition *::before,
html.theme-transition *::after {
transition:
background-color 300ms ease,
color 300ms ease,
border-color 300ms ease,
box-shadow 300ms ease,
fill 300ms ease,
stroke 300ms ease !important;
}
/* ——— Base layer ——— */
@layer base {
button,
input,
textarea,
select {
font: inherit;
}
img,
svg,
video,
canvas {
display: block;
max-width: 100%;
}
* {
@apply border-border;
}
body {
@apply bg-background text-foreground;
}
}
@@ -1,55 +0,0 @@
import { readFileSync } from 'node:fs';
import { join } from 'node:path';
import { describe, expect, it } from 'vitest';
const css = readFileSync(join(process.cwd(), 'src/app/globals.css'), 'utf8');
function token(block: string, name: string): string {
const match = block.match(new RegExp(`--${name}:\\s*(#[0-9a-fA-F]{6})`));
if (!match) throw new Error(`Missing --${name} token`);
return match[1];
}
function luminance(hex: string): number {
const channels = hex
.slice(1)
.match(/.{2}/g)!
.map((channel) => parseInt(channel, 16) / 255)
.map((channel) =>
channel <= 0.04045
? channel / 12.92
: ((channel + 0.055) / 1.055) ** 2.4,
);
return (
0.2126 * channels[0] + 0.7152 * channels[1] + 0.0722 * channels[2]
);
}
function contrast(first: string, second: string): number {
const firstLuminance = luminance(first);
const secondLuminance = luminance(second);
return (
(Math.max(firstLuminance, secondLuminance) + 0.05) /
(Math.min(firstLuminance, secondLuminance) + 0.05)
);
}
describe('global focus styles', () => {
it('keeps focus tokens above 3:1 against both page themes', () => {
const light = css.match(/:root\s*{([\s\S]*?)\n}/)?.[1] ?? '';
const dark = css.match(/\.dark\s*{([\s\S]*?)\n}/)?.[1] ?? '';
expect(contrast(token(light, 'ring'), token(light, 'background'))).toBeGreaterThanOrEqual(
3,
);
expect(contrast(token(dark, 'ring'), token(dark, 'background'))).toBeGreaterThanOrEqual(
3,
);
});
it('applies a non-animated three-pixel outline to focus-visible controls', () => {
expect(css).toContain('):focus-visible {');
expect(css).toContain('outline: 3px solid var(--ring) !important;');
expect(css).toContain('outline-offset: 2px !important;');
});
});
@@ -1,11 +0,0 @@
'use client';
import GpuGrid from '@/components/system/GpuGrid';
export default function GpusPage() {
return (
<div className="mx-auto flex w-full max-w-[1100px] flex-col gap-6 px-4 pb-12 pt-6">
<GpuGrid />
</div>
);
}
@@ -1,16 +0,0 @@
'use client';
import CreateJobButton from '@/components/jobs/CreateJobButton';
import { HeaderActions } from '@/components/shell/HeaderActionsContext';
import JobQueue from '@/components/jobs/JobQueue';
export default function InferencePage() {
return (
<>
<HeaderActions>
<CreateJobButton jobType="inference" />
</HeaderActions>
<JobQueue jobType="inference" />
</>
);
}
-52
View File
@@ -1,52 +0,0 @@
import type { Metadata } from 'next';
import { IBM_Plex_Mono, IBM_Plex_Sans } from 'next/font/google';
import { AppShell } from '@/components/shell/AppShell';
import './globals.css';
const plexSans = IBM_Plex_Sans({
subsets: ['latin'],
weight: ['400', '500', '600', '700'],
variable: '--font-plex-sans',
display: 'swap',
});
const plexMono = IBM_Plex_Mono({
subsets: ['latin'],
weight: ['400', '500', '600'],
variable: '--font-plex-mono',
display: 'swap',
});
export const metadata: Metadata = {
title: 'FastVideo Studio',
icons: { icon: '/fastvideo.ico' },
};
export default function RootLayout({
children,
}: {
children: React.ReactNode;
}) {
return (
<html
lang="en"
suppressHydrationWarning
className={`${plexSans.variable} ${plexMono.variable}`}
>
<head>
{/* The studio defaults to dark; apply the stored choice before paint. */}
<script
id="theme-init-script"
suppressHydrationWarning
dangerouslySetInnerHTML={{
__html: `(function(){var dark=true;try{dark=localStorage.getItem('theme')!=='light'}catch(e){}if(dark)document.documentElement.classList.add('dark')})()`,
}}
/>
</head>
<body className="antialiased">
<AppShell>{children}</AppShell>
</body>
</html>
);
}
@@ -1,5 +0,0 @@
import { redirect } from 'next/navigation';
export default function Page() {
redirect('/finetuning');
}
-5
View File
@@ -1,5 +0,0 @@
import { redirect } from 'next/navigation';
export default function Page() {
redirect('/inference');
}
@@ -1,74 +0,0 @@
import { fireEvent, render, screen } from '@testing-library/react';
import { beforeEach, describe, expect, it, vi } from 'vitest';
import { HeaderActionsProvider } from '@/components/shell/HeaderActionsContext';
import { resetToDefaults, updateOption } from '@/stores/defaultOptions';
import SettingsPage from './page';
vi.mock('@/lib/api', () => ({
getModels: vi.fn().mockResolvedValue([]),
getSettings: vi.fn().mockResolvedValue({}),
updateSettings: vi.fn().mockResolvedValue({}),
}));
// Keep the real store (so `useStore` resolves) but spy on the mutators.
vi.mock('@/stores/defaultOptions', async (importOriginal) => {
const actual =
await importOriginal<typeof import('@/stores/defaultOptions')>();
return {
...actual,
updateOption: vi.fn(),
resetToDefaults: vi.fn(),
};
});
function renderPage() {
return render(
<HeaderActionsProvider>
<SettingsPage />
</HeaderActionsProvider>,
);
}
describe('Settings page', () => {
beforeEach(() => {
vi.clearAllMocks();
});
it('calls updateOption when a toggle changes', () => {
renderPage();
fireEvent.click(
screen.getByRole('switch', { name: 'Auto Start Job on Create' }),
);
expect(updateOption).toHaveBeenCalledWith('autoStartJob', true);
});
it('calls updateOption when a slider changes', () => {
renderPage();
// First slider in DOM order is "Frames".
const slider = screen.getAllByRole('slider')[0];
fireEvent.keyDown(slider, { key: 'ArrowRight' });
expect(updateOption).toHaveBeenCalledWith('numFrames', expect.any(Number));
});
it('gives every slider an accessible name', () => {
renderPage();
const sliders = screen.getAllByRole('slider');
expect(sliders).toHaveLength(11);
for (const slider of sliders) {
expect(slider).toHaveAccessibleName();
}
expect(screen.getByRole('slider', { name: 'Frames' })).toBeInTheDocument();
expect(
screen.getByRole('slider', { name: 'Guidance Scale' }),
).toBeInTheDocument();
});
it('calls resetToDefaults when Reset to Defaults is clicked', () => {
renderPage();
fireEvent.click(screen.getByRole('button', { name: 'Reset to Defaults' }));
expect(resetToDefaults).toHaveBeenCalledTimes(1);
});
});
@@ -1,358 +0,0 @@
'use client';
import * as React from 'react';
import {
FieldRow,
NumberRow,
SliderRow,
ToggleRow,
} from '@/components/form-rows';
import { Button } from '@/components/ui/button';
import { Card, CardContent } from '@/components/ui/card';
import { Input } from '@/components/ui/input';
import { Label } from '@/components/ui/label';
import { NativeSelect } from '@/components/ui/native-select';
import { Separator } from '@/components/ui/separator';
import { Switch } from '@/components/ui/switch';
import { useStore } from '@/hooks/useStore';
import { getModels, type Model } from '@/lib/api';
import {
defaultOptionsStore,
resetToDefaults,
updateOption,
} from '@/stores/defaultOptions';
function TextSettingRow({
id,
label,
value,
placeholder,
onCommit,
}: {
id: string;
label: string;
value: string;
placeholder?: string;
onCommit: (v: string) => void;
}) {
// Buffer keystrokes locally and persist on blur/Enter so each character
// doesn't fire a settings PUT (or a localStorage write).
const [draft, setDraft] = React.useState<string | null>(null);
return (
<FieldRow htmlFor={id} label={label}>
<Input
id={id}
type="text"
className="font-mono text-sm"
value={draft ?? value}
onChange={(e) => setDraft(e.target.value)}
onBlur={() => {
if (draft !== null && draft !== value) onCommit(draft);
setDraft(null);
}}
onKeyDown={(e) => {
if (e.key === 'Enter') e.currentTarget.blur();
}}
placeholder={placeholder}
/>
</FieldRow>
);
}
function SelectRow({
id,
label,
value,
onChange,
children,
}: {
id: string;
label: string;
value: string;
onChange: (v: string) => void;
children: React.ReactNode;
}) {
return (
<FieldRow htmlFor={id} label={label}>
<NativeSelect
id={id}
value={value}
onChange={(e) => onChange(e.target.value)}
>
{children}
</NativeSelect>
</FieldRow>
);
}
export default function SettingsPage() {
const { options } = useStore(defaultOptionsStore);
const [models, setModels] = React.useState<
Record<'t2v' | 'i2v' | 't2i', Model[]>
>({ t2v: [], i2v: [], t2i: [] });
React.useEffect(() => {
Promise.all([getModels('t2v'), getModels('i2v'), getModels('t2i')])
.then(([t2v, i2v, t2i]) => setModels({ t2v, i2v, t2i }))
.catch((e) => console.error('Failed to load models:', e));
}, []);
return (
<div className="mx-auto flex w-full max-w-[850px] flex-col gap-6 px-4 pb-12 pt-6">
<Card>
<CardContent className="space-y-4 p-6">
<h2 className="text-lg font-semibold">Behavior</h2>
<div className="flex items-center justify-between gap-4">
<Label
htmlFor="settings-auto-start-job"
className="pl-0.5 text-xs font-normal tracking-wide text-muted-foreground"
>
Auto Start Job on Create
</Label>
<Switch
id="settings-auto-start-job"
checked={options.autoStartJob}
onCheckedChange={(v) => updateOption('autoStartJob', v)}
/>
</div>
<Separator />
<h2 className="text-lg font-semibold">Paths</h2>
<div className="space-y-4">
<TextSettingRow
id="settings-api-server-base-url"
label="API Server Base URL"
value={options.apiServerBaseUrl ?? ''}
onCommit={(v) => updateOption('apiServerBaseUrl', v)}
placeholder="http://localhost:8189/api"
/>
<TextSettingRow
id="settings-dataset-upload-path"
label="Dataset Upload Path"
value={options.datasetUploadPath ?? ''}
onCommit={(v) => updateOption('datasetUploadPath', v)}
placeholder="outputs/ui_data/uploads/datasets"
/>
</div>
<Separator />
<div className="flex items-center justify-between">
<h2 className="text-lg font-semibold">Default Options</h2>
<Button
type="button"
variant="outline"
size="sm"
onClick={resetToDefaults}
>
Reset to Defaults
</Button>
</div>
<p className="text-sm text-muted-foreground">
These values are used as defaults when creating new jobs.
</p>
<div className="grid gap-x-3 gap-y-2 [grid-template-columns:repeat(auto-fill,minmax(160px,1fr))]">
<SelectRow
id="settings-default-model-t2v"
label="Default Model (T2V)"
value={options.defaultModelIdT2v}
onChange={(v) => updateOption('defaultModelIdT2v', v)}
>
<option value="">None (select when creating job)</option>
{models.t2v.map((model) => (
<option key={model.id} value={model.id}>
{model.label} ({model.id})
</option>
))}
</SelectRow>
<SelectRow
id="settings-default-model-i2v"
label="Default Model (I2V)"
value={options.defaultModelIdI2v}
onChange={(v) => updateOption('defaultModelIdI2v', v)}
>
<option value="">None</option>
{models.i2v.map((model) => (
<option key={model.id} value={model.id}>
{model.label} ({model.id})
</option>
))}
</SelectRow>
<SelectRow
id="settings-default-model-t2i"
label="Default Model (T2I)"
value={options.defaultModelIdT2i}
onChange={(v) => updateOption('defaultModelIdT2i', v)}
>
<option value="">None</option>
{models.t2i.map((model) => (
<option key={model.id} value={model.id}>
{model.label} ({model.id})
</option>
))}
</SelectRow>
<SliderRow
id="settings-num-frames"
label="Frames"
min={1}
max={500}
step={1}
value={options.numFrames}
onChange={(v) => updateOption('numFrames', v)}
/>
<SliderRow
id="settings-height"
label="Height"
min={64}
max={1080}
step={16}
value={options.height}
onChange={(v) => updateOption('height', v)}
/>
<SliderRow
id="settings-width"
label="Width"
min={64}
max={1920}
step={16}
value={options.width}
onChange={(v) => updateOption('width', v)}
/>
<SliderRow
id="settings-num-steps"
label="Inference Steps"
min={1}
max={200}
step={1}
value={options.numInferenceSteps}
onChange={(v) => updateOption('numInferenceSteps', v)}
/>
<SliderRow
id="settings-vsa-sparsity"
label="VSA Sparsity"
title="VSA sparsity (0–1)"
min={0}
max={1}
step={0.05}
value={options.vsaSparsity}
onChange={(v) => updateOption('vsaSparsity', v)}
format={(v) => v.toFixed(2)}
/>
<SliderRow
id="settings-guidance"
label="Guidance Scale"
min={0}
max={20}
step={0.1}
value={options.guidanceScale}
onChange={(v) => updateOption('guidanceScale', v)}
format={(v) => v.toFixed(1)}
/>
<SliderRow
id="settings-guidance-rescale"
label="Guidance Rescale"
title="0 = disabled"
min={0}
max={1}
step={0.05}
value={options.guidanceRescale ?? 0}
onChange={(v) => updateOption('guidanceRescale', v)}
format={(v) => v.toFixed(2)}
/>
<SliderRow
id="settings-tp-size"
label="TP Size"
title="-1 = auto"
min={-1}
max={8}
step={1}
value={options.tpSize}
onChange={(v) => updateOption('tpSize', v)}
format={(v) => (v === -1 ? 'Auto' : String(v))}
/>
<SliderRow
id="settings-sp-size"
label="SP Size"
title="-1 = auto"
min={-1}
max={8}
step={1}
value={options.spSize}
onChange={(v) => updateOption('spSize', v)}
format={(v) => (v === -1 ? 'Auto' : String(v))}
/>
<SliderRow
id="settings-fps"
label="FPS"
min={1}
max={60}
step={1}
value={options.fps ?? 24}
onChange={(v) => updateOption('fps', v)}
/>
<ToggleRow
id="settings-dit-cpu-offload"
label="DiT CPU Offload"
checked={options.ditCpuOffload}
onChange={(v) => updateOption('ditCpuOffload', v)}
/>
<ToggleRow
id="settings-text-encoder-cpu-offload"
label="Text Encoder CPU Offload"
checked={options.textEncoderCpuOffload}
onChange={(v) => updateOption('textEncoderCpuOffload', v)}
/>
<ToggleRow
id="settings-use-fsdp-inference"
label="Use FSDP Inference"
checked={options.useFsdpInference}
onChange={(v) => updateOption('useFsdpInference', v)}
/>
<ToggleRow
id="settings-vae-cpu-offload"
label="VAE CPU Offload"
checked={options.vaeCpuOffload}
onChange={(v) => updateOption('vaeCpuOffload', v)}
/>
<ToggleRow
id="settings-image-encoder-cpu-offload"
label="Image Encoder CPU Offload"
checked={options.imageEncoderCpuOffload}
onChange={(v) => updateOption('imageEncoderCpuOffload', v)}
/>
<ToggleRow
id="settings-enable-torch-compile"
label="Torch Compile"
checked={options.enableTorchCompile}
onChange={(v) => updateOption('enableTorchCompile', v)}
/>
<SliderRow
id="settings-num-gpus"
label="GPUs"
min={1}
max={8}
step={1}
value={options.numGpus}
onChange={(v) => updateOption('numGpus', v)}
/>
<NumberRow
id="settings-seed"
label="Seed"
min={0}
value={options.seed}
onChange={(v) => updateOption('seed', v)}
/>
</div>
</CardContent>
</Card>
</div>
);
}
@@ -1,12 +0,0 @@
'use client';
import { Button } from '@/components/ui/button';
import { setCreateDatasetModalOpen } from '@/stores/createDatasetModalOpen';
export default function AddDatasetButton() {
return (
<Button type="button" onClick={() => setCreateDatasetModalOpen(true)}>
Add Dataset
</Button>
);
}
@@ -1,43 +0,0 @@
import { describe, expect, it, vi } from 'vitest';
import { fireEvent, render, screen } from '@testing-library/react';
import CreateDatasetModal from '@/components/datasets/CreateDatasetModal';
import { createDataset } from '@/lib/api';
vi.mock('@/lib/api', () => ({
createDataset: vi.fn(),
uploadRawDataset: vi.fn(),
}));
const mockedCreateDataset = vi.mocked(createDataset);
describe('CreateDatasetModal', () => {
it('renders nothing when closed', () => {
render(
<CreateDatasetModal isOpen={false} onClose={() => {}} onSuccess={() => {}} />,
);
expect(screen.queryByText('Add Dataset — Raw')).not.toBeInTheDocument();
});
it('renders the form and the default JSON caption upload when open', () => {
render(
<CreateDatasetModal isOpen onClose={() => {}} onSuccess={() => {}} />,
);
expect(screen.getByText('Add Dataset — Raw')).toBeInTheDocument();
expect(screen.getByText('Upload video files')).toBeInTheDocument();
expect(screen.getByText('Upload videos2caption.json')).toBeInTheDocument();
expect(
screen.getByRole('button', { name: 'Create Dataset' }),
).toBeInTheDocument();
});
it('does not create a dataset when the name is empty', () => {
const onSuccess = vi.fn();
render(
<CreateDatasetModal isOpen onClose={() => {}} onSuccess={onSuccess} />,
);
fireEvent.click(screen.getByRole('button', { name: 'Create Dataset' }));
expect(mockedCreateDataset).not.toHaveBeenCalled();
expect(onSuccess).not.toHaveBeenCalled();
});
});
@@ -1,465 +0,0 @@
'use client';
import * as React from 'react';
import UploadZone from '@/components/datasets/UploadZone';
import { Button } from '@/components/ui/button';
import {
Dialog,
DialogContent,
DialogHeader,
DialogTitle,
} from '@/components/ui/dialog';
import { Input } from '@/components/ui/input';
import { Label } from '@/components/ui/label';
import { Tabs, TabsContent, TabsList, TabsTrigger } from '@/components/ui/tabs';
import { createDataset, uploadRawDataset } from '@/lib/api';
import {
parseCaptionCsv,
parseVideos2Caption,
parseVideosCaptionsTxt,
} from '@/lib/captionParsing';
const ALLOWED_VIDEO_EXT = '.mp4,.webm,.avi,.mov,.mkv';
type CaptionFormat = 'json' | 'txt' | 'csv';
export interface CreateDatasetModalProps {
isOpen: boolean;
onClose: () => void;
onSuccess: () => void;
}
export default function CreateDatasetModal({
isOpen,
onClose,
onSuccess,
}: CreateDatasetModalProps) {
const [name, setName] = React.useState('');
const [isSubmitting, setIsSubmitting] = React.useState(false);
const [rawPath, setRawPath] = React.useState('');
const [fileNames, setFileNames] = React.useState<string[]>([]);
const [isUploading, setIsUploading] = React.useState(false);
const [validationError, setValidationError] = React.useState<string | null>(
null,
);
const [captionFormat, setCaptionFormat] = React.useState<CaptionFormat>('json');
const [captionMap, setCaptionMap] = React.useState<Record<
string,
string
> | null>(null);
const [captionFileName, setCaptionFileName] = React.useState<string | null>(
null,
);
const [videosTxtLines, setVideosTxtLines] = React.useState<string[] | null>(
null,
);
const [videosTxtFileName, setVideosTxtFileName] = React.useState<
string | null
>(null);
const [captionsTxtLines, setCaptionsTxtLines] = React.useState<
string[] | null
>(null);
const [captionsTxtFileName, setCaptionsTxtFileName] = React.useState<
string | null
>(null);
// Bumped on every new media selection and on reset/clear; an in-flight
// upload whose generation no longer matches is discarded, so a superseded or
// abandoned upload can't repopulate/overwrite state.
const uploadGeneration = React.useRef(0);
const txtCaptionMap = React.useMemo(() => {
if (!captionsTxtLines || fileNames.length === 0) return null;
const { captions, error } = parseVideosCaptionsTxt(
videosTxtLines ?? null,
captionsTxtLines,
fileNames,
);
if (error) return null;
return Object.keys(captions).length > 0 ? captions : null;
}, [captionsTxtLines, fileNames, videosTxtLines]);
const effectiveCaptionMap =
captionFormat === 'txt' ? txtCaptionMap : captionMap;
// Keep the validation message in sync with the TXT inputs.
React.useEffect(() => {
if (captionFormat === 'txt' && captionsTxtLines && fileNames.length > 0) {
const hasVideosTxt =
!!videosTxtLines &&
videosTxtLines.length > 0 &&
videosTxtLines.some((s) => s.trim());
if (hasVideosTxt && videosTxtLines) {
const { error } = parseVideosCaptionsTxt(
videosTxtLines,
captionsTxtLines,
fileNames,
);
setValidationError(error);
} else {
setValidationError(null);
}
}
}, [captionFormat, captionsTxtLines, fileNames, videosTxtLines]);
function resetState() {
uploadGeneration.current += 1;
setName('');
setRawPath('');
setFileNames([]);
setValidationError(null);
setCaptionFormat('json');
setCaptionMap(null);
setCaptionFileName(null);
setVideosTxtLines(null);
setVideosTxtFileName(null);
setCaptionsTxtLines(null);
setCaptionsTxtFileName(null);
}
function handleClose() {
if (isSubmitting) return;
resetState();
onClose();
}
async function handleMediaChange(files: File[]) {
const gen = (uploadGeneration.current += 1);
setValidationError(null);
if (files.length === 0) {
setRawPath('');
setFileNames([]);
return;
}
setIsUploading(true);
try {
const res = await uploadRawDataset(files);
if (gen !== uploadGeneration.current) return; // superseded or abandoned
setRawPath(res.path);
setFileNames(res.file_names);
if (res.file_names.length === 0) {
setValidationError(`No video files found. Allowed: ${ALLOWED_VIDEO_EXT}`);
}
} catch (err) {
if (gen !== uploadGeneration.current) return;
setRawPath('');
setFileNames([]);
setValidationError(err instanceof Error ? err.message : 'Upload failed');
} finally {
if (gen === uploadGeneration.current) setIsUploading(false);
}
}
async function handleCaptionJsonChange(files: File[]) {
setValidationError(null);
setCaptionMap(null);
setCaptionFileName(null);
if (files.length === 0) return;
const file = files[0];
try {
const text = await file.text();
const { captions, error } = parseVideos2Caption(text, fileNames);
if (error) {
setValidationError(error);
return;
}
setCaptionMap(captions);
setCaptionFileName(file.name);
} catch {
setValidationError('Could not read the file.');
}
}
async function handleCaptionCsvChange(files: File[]) {
setValidationError(null);
setCaptionMap(null);
setCaptionFileName(null);
if (files.length === 0) return;
const file = files[0];
try {
const text = await file.text();
const { captions, error } = parseCaptionCsv(text, fileNames);
if (error) {
setValidationError(error);
return;
}
setCaptionMap(captions);
setCaptionFileName(file.name);
} catch {
setValidationError('Could not read the file.');
}
}
async function handleVideosTxtChange(files: File[]) {
setValidationError(null);
setVideosTxtLines(null);
setVideosTxtFileName(null);
if (files.length === 0) return;
try {
const text = await files[0].text();
setVideosTxtLines(text.split(/\r?\n/).map((s) => s.trim()));
setVideosTxtFileName(files[0].name);
} catch {
setValidationError('Could not read videos.txt.');
}
}
async function handleCaptionsTxtChange(files: File[]) {
setValidationError(null);
setCaptionsTxtLines(null);
setCaptionsTxtFileName(null);
if (files.length === 0) return;
try {
const text = await files[0].text();
setCaptionsTxtLines(text.split(/\r?\n/).map((s) => s.trim()));
setCaptionsTxtFileName(files[0].name);
} catch {
setValidationError('Could not read captions.txt.');
}
}
function handleCaptionFormatChange(format: CaptionFormat) {
setCaptionFormat(format);
setValidationError(null);
setCaptionMap(null);
setCaptionFileName(null);
setVideosTxtLines(null);
setVideosTxtFileName(null);
setCaptionsTxtLines(null);
setCaptionsTxtFileName(null);
}
async function handleSubmit(e: React.FormEvent) {
e.preventDefault();
setValidationError(null);
if (!name.trim()) return;
if (!rawPath || fileNames.length === 0) {
setValidationError('No data was found. Upload at least one video.');
return;
}
if (captionFormat === 'json' || captionFormat === 'csv') {
if (captionFileName && !captionMap) {
setValidationError(
'Caption file has errors. Fix or remove it before creating the dataset.',
);
return;
}
} else if (captionFormat === 'txt') {
if (videosTxtFileName || captionsTxtFileName) {
if (!captionsTxtFileName) {
setValidationError('Upload captions.txt to use TXT captions.');
return;
}
// Re-validate against current state rather than the stale render-time
// `validationError`, so a videos.txt mismatch both blocks submission
// and keeps its message visible.
const hasVideosTxt =
!!videosTxtLines &&
videosTxtLines.length > 0 &&
videosTxtLines.some((s) => s.trim());
if (hasVideosTxt && videosTxtLines) {
const { error } = parseVideosCaptionsTxt(
videosTxtLines,
captionsTxtLines ?? [],
fileNames,
);
if (error) {
setValidationError(error);
return;
}
}
}
}
const finalCaptionMap =
effectiveCaptionMap && Object.keys(effectiveCaptionMap).length > 0
? effectiveCaptionMap
: null;
if (finalCaptionMap) {
const missing = fileNames.filter((fn) => !(fn in finalCaptionMap));
if (missing.length > 0) {
const list =
missing.length <= 5
? missing.join(', ')
: `${missing.slice(0, 5).join(', ')} and ${missing.length - 5} more`;
const ok = window.confirm(
`The caption file does not include captions for ${missing.length} video(s): ${list}. They will get empty captions. Continue?`,
);
if (!ok) return;
}
}
setIsSubmitting(true);
try {
await createDataset({
name: name.trim(),
upload_path: rawPath,
file_names: fileNames,
...(finalCaptionMap ? { captions: finalCaptionMap } : {}),
});
onSuccess();
resetState();
onClose();
} catch (err) {
setValidationError(
err instanceof Error ? err.message : 'Failed to create dataset',
);
} finally {
setIsSubmitting(false);
}
}
return (
<Dialog
open={isOpen}
onOpenChange={(open) => {
if (!open) handleClose();
}}
>
<DialogContent
aria-describedby={undefined}
className="max-h-[90vh] w-[90vw] max-w-[850px] overflow-y-auto"
>
<DialogHeader>
<DialogTitle>Add Dataset — Raw</DialogTitle>
</DialogHeader>
<form onSubmit={handleSubmit} autoComplete="off">
<div className="mb-3.5 flex flex-col gap-1.5">
<Label htmlFor="add-dataset-name">Name</Label>
<Input
id="add-dataset-name"
type="text"
value={name}
onChange={(e) => setName(e.target.value)}
placeholder="My dataset"
required
disabled={isSubmitting}
/>
</div>
<div className="mb-3.5 flex flex-col gap-1.5">
<Label>Videos</Label>
<UploadZone
label="Upload video files"
hint="Select files or a folder (.mp4, .webm, .avi, .mov, .mkv)"
accept={ALLOWED_VIDEO_EXT}
multiple
directory
allowBothFileAndDirectory
value={rawPath}
fileName={
fileNames.length > 0 ? `${fileNames.length} file(s)` : undefined
}
onFiles={handleMediaChange}
onClear={() => {
uploadGeneration.current += 1;
setRawPath('');
setFileNames([]);
setCaptionMap(null);
setCaptionFileName(null);
setValidationError(null);
}}
disabled={isSubmitting}
uploading={isUploading}
/>
</div>
<div className="mb-3.5 flex flex-col gap-1.5">
<Tabs
value={captionFormat}
onValueChange={(value) =>
handleCaptionFormatChange(value as CaptionFormat)
}
>
<div className="mb-1.5 flex flex-wrap items-center gap-4">
<Label>Captions (optional)</Label>
<TabsList>
<TabsTrigger value="json" disabled={isSubmitting}>
JSON
</TabsTrigger>
<TabsTrigger value="txt" disabled={isSubmitting}>
TXT
</TabsTrigger>
<TabsTrigger value="csv" disabled={isSubmitting}>
CSV
</TabsTrigger>
</TabsList>
</div>
<TabsContent value="json">
<UploadZone
label="Upload videos2caption.json"
hint="Array of { path, cap } or object mapping file names to captions"
accept=".json,application/json"
value={captionFileName ? '1' : ''}
fileName={captionFileName ?? undefined}
onFiles={handleCaptionJsonChange}
onClear={() => {
setCaptionMap(null);
setCaptionFileName(null);
setValidationError(null);
}}
disabled={isSubmitting}
/>
</TabsContent>
<TabsContent value="txt">
<div className="flex flex-wrap gap-[15px] [&>*]:min-w-[200px] [&>*]:flex-1">
<UploadZone
label="Upload videos.txt (optional)"
hint="One video path per line, or leave empty to match captions to videos in alphabetical order"
accept=".txt,text/plain"
value={videosTxtFileName ? '1' : ''}
fileName={videosTxtFileName ?? undefined}
onFiles={handleVideosTxtChange}
onClear={() => {
setVideosTxtLines(null);
setVideosTxtFileName(null);
setValidationError(null);
}}
disabled={isSubmitting}
/>
<UploadZone
label="Upload captions.txt"
hint="One caption per line (same order as videos.txt or alphabetical)"
accept=".txt,text/plain"
value={captionsTxtFileName ? '1' : ''}
fileName={captionsTxtFileName ?? undefined}
onFiles={handleCaptionsTxtChange}
onClear={() => {
setCaptionsTxtLines(null);
setCaptionsTxtFileName(null);
setValidationError(null);
}}
disabled={isSubmitting}
/>
</div>
</TabsContent>
<TabsContent value="csv">
<UploadZone
label="Upload captions CSV"
hint="Header: video_name, caption"
accept=".csv,text/csv"
value={captionFileName ? '1' : ''}
fileName={captionFileName ?? undefined}
onFiles={handleCaptionCsvChange}
onClear={() => {
setCaptionMap(null);
setCaptionFileName(null);
setValidationError(null);
}}
disabled={isSubmitting}
/>
</TabsContent>
</Tabs>
</div>
{validationError && (
<p className="mb-2 text-sm text-destructive">{validationError}</p>
)}
<Button type="submit" disabled={isSubmitting}>
{isSubmitting ? 'Creating…' : 'Create Dataset'}
</Button>
</form>
</DialogContent>
</Dialog>
);
}

Some files were not shown because too many files have changed in this diff Show More