Compare commits

...
Author SHA1 Message Date
kevin314 c59cf79ea4 Add clip segmenting 2026-09-29 11:11:29 +00:00
kevin314 fab9b3159c Add last-clip, cutting, color grading 2026-09-28 11:47:24 +00:00
kevin314 d377f463f1 Add job queueing 2026-09-27 09:48:21 +00:00
kevin314 01050277ea Add dialogue/shot editor 2026-09-25 11:05:41 +00:00
83 changed files with 15781 additions and 108 deletions
+99
View File
@@ -16,6 +16,9 @@ The UI currently supports:
```
apps/fastvideo_studio/
├── server.py / job_runner.py / database.py # FastAPI backend + job lifecycle
├── job_queue.py # Queue scheduling rules + last-frame references (GPU-free)
├── merge.py # Join a scene's clips into one video (ffmpeg stream copy, GPU-free)
├── trim.py # Cut a clip to an in/out range, non-destructively (GPU-free)
├── mock_server.py # In-memory API mock for e2e tests
├── models/ # Pydantic request models (shared with the mock)
├── training_config.py # Studio workloads → fastvideo/train YAML configs
@@ -88,6 +91,93 @@ Running it this way allows you to pass command line parameters.
The API server defaults to port 8189. To configure this, you can edit `.env.local` file.
Refer to `.env.example` for reference.
### Job queue
`POST /api/jobs/queue` puts jobs in a queue; the server runs them oldest-first,
`--max-concurrent-jobs` at a time (default 1). Start on a job while that many
are already running is refused; queue it instead. Queued jobs survive a server
restart and carry on.
Inference jobs run on that many long-lived worker threads. The model's GPU
processes are killed when the thread that started them exits, so a worker that
never exits is what keeps the model loaded from one job to the next (it loads
once, not per job). A worker holds one model at a time: a job with a different
model or settings replaces it, and a failed or stopped job unloads it so the next
one starts clean. Each extra worker loads its own copy of the model.
A job can start from how another job's video ended, two ways:
* **Last frame** -- a reference whose source is `job-last-frame:<job id>`. A
hard, exact opening frame: the prompt is rewritten to say the shot begins
from it.
* **Last clip** -- a reference whose source is `job-last-clip:<job id>`, and
`media_type: "video"`. The trailing ~1s of the source job's video instead of
a single still, so the model has real motion to continue rather than a pose
to guess it from. Costs one of H3's 3 video-reference slots instead of one
of its 9 image slots.
Either is extracted when the job runs, so the reference can be added before
that job has finished, and the queue holds the job until it has. If that job
fails, is stopped, or is deleted, the waiting job fails with a message saying
so (and so does anything waiting on *it*); fix the earlier job and queue them
again. Starting such a job directly while its source is unfinished is refused
with a 409.
### Merging a scene
**Merge scene** on the Scenes page joins a scene's clips, in the order shown (the
take chosen for each), into one video under `<output-dir>/merged/`, then plays it
and offers a download. It is enabled once every clip has finished, and refuses
otherwise, so a scene missing a clip can't pass for the whole thing.
The clips are copied, not re-encoded (about a second for 15 clips, no quality
loss). That needs them to share size, frame rate and audio, which clips from one
scene do; if one differs, the merge says which. Each clip is cut at the end of its
last video frame, because H3's audio runs a few milliseconds longer and would
otherwise leave a small gap at every join. It uses `ffmpeg` from `PATH`,
`$FASTVIDEO_FFMPEG_BIN`, or the copy bundled with `imageio-ffmpeg`.
### Editing a clip's video
**Edit video** on a finished clip in Scenes cuts it to a chosen in/out range and
adjusts its brightness, contrast and saturation, in place -- other jobs'
last-frame/last-clip references, and a later merge, all just read the job's
`output_path`, so nothing else needs to know a clip was edited. Frame-accurate
(re-encoded, not stream-copied), so the cut isn't limited to the source's
keyframes. Audio is cut to the same range and carried over too (re-encoded to
AAC regardless of the source's codec), unless the source has none.
Color grading is per **section**, not the whole clip at once, like a razor/blade
tool: **Split at current time** cuts the kept range wherever the preview is
paused, and each resulting section gets its own brightness, contrast and
saturation sliders, next to a before/after preview of a still frame
(**Preview** on a section seeks to its start and captures that frame; **Use
current frame** captures wherever the preview is paused instead). That preview
is computed entirely in the browser (`src/lib/grading.ts` mirrors `trim.py`'s
`_apply_grade` pixel math) as the sliders move, so it never waits on the
server -- only **Apply** does that, to actually re-render the file. **Previous
frame**/**Next frame** step the preview by exactly one frame (at the clip's
own fps), for lining up a cut precisely -- the native scrub bar alone isn't
frame-accurate. **Merge with next** undoes a cut, keeping the earlier of the
two sections' grade.
Sections are always contiguous and cover the whole kept range -- there's no
reordering or gaps, only where to cut and what to grade each side; a plain
range trim with no cuts is just the one-section case. Audio is one continuous
track regardless of how many sections there are.
The untouched original is always kept alongside it the first time a clip is
edited, and **every edit always re-renders from that original** with the full
set of values shown in the dialog -- range and sections together, in one pass.
That's what lets adjusting either one leave the other in place instead of each
one silently discarding the other. The job itself records the range/sections
of its last edit (`edit_start_seconds`/`edit_end_seconds`/`edit_segments`,
alongside `output_path`), so reopening the dialog -- even after closing it or
reloading the page -- shows exactly what produced the current video, not a
guess: a never-edited clip shows one neutral section covering its full length;
an edited one shows every section that made it, cuts and start offset
included. **Restore original** undoes every edit in one step and resets those
recorded values back to one neutral section too.
### API Endpoints
| Method | Path | Description |
@@ -98,6 +188,15 @@ Refer to `.env.example` for reference.
| `POST` | `/api/jobs` | Create a new job |
| `POST` | `/api/jobs/{id}/start` | Start a pending/stopped/failed job |
| `POST` | `/api/jobs/{id}/stop` | Request a running job to stop |
| `POST` | `/api/jobs/queue` | Queue jobs (`{"job_ids": [...]}`), in order |
| `POST` | `/api/jobs/{id}/queue` | Queue one job |
| `POST` | `/api/jobs/{id}/dequeue` | Take a queued job back out of the queue |
| `POST` | `/api/jobs/{id}/last-frame` | Save the job's last frame, for another job to start from |
| `POST` | `/api/jobs/{id}/last-clip` | Save the job's trailing ~1s, for another job to continue from |
| `POST` | `/api/scenes/merge` | Join finished jobs' videos (`{"job_ids": [...], "name": ""}`) in order |
| `POST` | `/api/jobs/{id}/trim` | Cut a job's video to a range and/or grade it section by section (`{"start_seconds": 0, "end_seconds": null, "segments": [{"end_seconds": null, "brightness": 0, "contrast": 1, "saturation": 1}]}`), in place |
| `POST` | `/api/jobs/{id}/restore-video`| Undo every edit, back to the original video |
| `GET` | `/api/merged/{filename}` | Stream a merged scene video |
| `DELETE` | `/api/jobs/{id}` | Delete a job |
| `GET` | `/api/jobs/{id}/video` | Stream the generated video/image |
| `GET` | `/api/jobs/{id}/logs` | Get job logs (polling, supports `?after=`) |
+79 -1
View File
@@ -110,6 +110,15 @@ def _migrate_db(conn: sqlite3.Connection) -> None:
_add_column_if_missing(conn, "jobs", "generator_update_interval", "INTEGER", "5")
_add_column_if_missing(conn, "jobs", "real_score_model_path", "TEXT", "''")
_add_column_if_missing(conn, "jobs", "fake_score_model_path", "TEXT", "''")
_add_column_if_missing(conn, "jobs", "queued_at", "REAL", "NULL")
_add_column_if_missing(conn, "jobs", "edit_start_seconds", "REAL", "0.0")
_add_column_if_missing(conn, "jobs", "edit_end_seconds", "REAL", "NULL")
# Superseded by edit_segments_json (a list of razor-cut sections, each with
# its own grade); left in place, unused, rather than risking a DROP COLUMN.
_add_column_if_missing(conn, "jobs", "edit_brightness", "REAL", "0.0")
_add_column_if_missing(conn, "jobs", "edit_contrast", "REAL", "1.0")
_add_column_if_missing(conn, "jobs", "edit_saturation", "REAL", "1.0")
_add_column_if_missing(conn, "jobs", "edit_segments_json", "TEXT", "'[]'")
# Settings table
_add_column_if_missing(conn, "settings", "vae_cpu_offload", "INTEGER", "0")
_add_column_if_missing(conn, "settings", "image_encoder_cpu_offload", "INTEGER", "0")
@@ -326,6 +335,7 @@ class Database:
return
allowed = {
"status",
"queued_at",
"started_at",
"finished_at",
"error",
@@ -345,6 +355,43 @@ class Database:
self._execute(sql, tuple(vals))
self._commit()
#: Job fields stored under a different column name, and how they are encoded.
_CONFIG_COLUMNS = {"references": "references_json", "edit_segments": "edit_segments_json"}
_JSON_LIST_CONFIG_FIELDS = {"references", "edit_segments"}
_BOOL_CONFIG_FIELDS = {
"dit_cpu_offload",
"text_encoder_cpu_offload",
"vae_cpu_offload",
"image_encoder_cpu_offload",
"use_fsdp_inference",
"enable_torch_compile",
"dmd_use_vsa",
}
def update_job_config(self, job_id: str, updates: dict[str, Any]) -> None:
"""Persist edits to a job's configuration (prompt, references, sizes...).
Fields the table has no column for are skipped, as they are on insert.
"""
columns = _get_table_columns(self._conn(), "jobs")
cols = []
vals: list[Any] = []
for key, value in updates.items():
column = self._CONFIG_COLUMNS.get(key, key)
if column not in columns or column == "id":
continue
if key in self._JSON_LIST_CONFIG_FIELDS:
value = json.dumps(value or [])
elif key in self._BOOL_CONFIG_FIELDS:
value = 1 if value else 0
cols.append(f"{column} = ?")
vals.append(value)
if not cols:
return
vals.append(job_id)
self._execute(f"UPDATE jobs SET {', '.join(cols)} WHERE id = ?", tuple(vals))
self._commit()
def delete_job(self, job_id: str) -> bool:
"""Delete a job. Returns True if a row was deleted."""
cur = self._execute("DELETE FROM jobs WHERE id = ?", (job_id, ))
@@ -537,8 +584,31 @@ def _row_to_dataset(row: sqlite3.Row) -> dict[str, Any]:
}
def _row_to_job(row: sqlite3.Row) -> dict[str, Any]:
class _ColumnRow:
"""A sqlite3.Row whose ``in`` tests column names.
sqlite3.Row's own ``in`` tests *values*, so ``"fps" in row`` is always False
and a job restored from the database silently lost fps, negative_prompt and
the other optional columns.
"""
def __init__(self, row: sqlite3.Row):
self._row = row
self._columns = set(row.keys())
def __getitem__(self, key: str) -> Any:
return self._row[key]
def __contains__(self, key: object) -> bool:
return key in self._columns
def keys(self) -> list[str]:
return self._row.keys()
def _row_to_job(sqlite_row: sqlite3.Row) -> dict[str, Any]:
"""Convert a DB row to job dict (snake_case, matching Job.to_dict)."""
row = _ColumnRow(sqlite_row)
int_defaults = {
"max_train_steps": 1000,
"train_batch_size": 1,
@@ -562,6 +632,7 @@ def _row_to_job(row: sqlite3.Row) -> dict[str, Any]:
"job_type": _sqlite_row_get(row, "job_type", "inference"),
"status": row["status"],
"created_at": row["created_at"],
"queued_at": row["queued_at"] if "queued_at" in row else None,
"started_at": row["started_at"],
"finished_at": row["finished_at"],
"error": row["error"],
@@ -614,6 +685,13 @@ def _row_to_job(row: sqlite3.Row) -> dict[str, Any]:
for col in ("dmd_denoising_steps", "real_score_model_path", "fake_score_model_path"):
if col in row:
result[col] = (row[col] or "") or ""
if "edit_start_seconds" in row:
start_val: Any = row["edit_start_seconds"]
result["edit_start_seconds"] = float(start_val) if start_val is not None else 0.0
if "edit_end_seconds" in row:
end_val: Any = row["edit_end_seconds"]
result["edit_end_seconds"] = float(end_val) if end_val is not None else None
result["edit_segments"] = _sqlite_row_get(row, "edit_segments_json", "") or ""
return result
+146
View File
@@ -0,0 +1,146 @@
# SPDX-License-Identifier: Apache-2.0
"""Save the end of a finished clip, so the next clip can continue from it.
Two forms: the final frame alone (a still, for a hard "opens on this exact
picture" continuation), or a short trailing clip (a few hundred milliseconds of
video, attached as an H3 video reference so the model has actual motion to
continue rather than guessing it from one still). Kept apart from ``server.py``
because it needs nothing from FastVideo -- only PyAV -- and so can be exercised
without a GPU.
"""
from __future__ import annotations
import os
from fractions import Fraction
from pathlib import Path
from typing import Any
VIDEO_EXTENSIONS = {".mp4", ".webm", ".avi", ".mov", ".mkv"}
class FrameError(Exception):
"""A request that can't be satisfied, carrying the HTTP status to report."""
def __init__(self, status_code: int, detail: str):
super().__init__(detail)
self.status_code = status_code
self.detail = detail
def save_last_frame(video_path: str, out_path: str) -> None:
"""Decode ``video_path`` and write its final frame to ``out_path`` as a PNG.
Clips are a few hundred frames, so decoding through to the end is simpler
and more reliable than seeking (which lands on keyframes, not the last frame).
"""
import av
tmp_path = f"{out_path}.tmp.png"
with av.open(video_path) as container:
last = None
for frame in container.decode(container.streams.video[0]):
last = frame
if last is None:
raise ValueError("The video has no frames.")
last.to_image().save(tmp_path)
os.replace(tmp_path, out_path) # atomic, so a reader never sees a half-written file
def last_frame_for_job(job: Any, upload_dir: str) -> str:
"""Path of a PNG of the last frame of ``job``'s video, extracting it if needed.
Raises :class:`FrameError` with the status the API should return.
"""
status = getattr(job.status, "value", job.status)
if status != "completed" or not job.output_path:
raise FrameError(404, "No output available for this job")
if not os.path.isfile(job.output_path):
raise FrameError(404, "Output file not found on disk")
if Path(job.output_path).suffix.lower() not in VIDEO_EXTENSIONS:
raise FrameError(400, "This job's output is not a video")
if not upload_dir:
raise FrameError(503, "Upload directory not configured")
out_dir = os.path.join(upload_dir, "last_frames")
os.makedirs(out_dir, exist_ok=True)
out_path = os.path.abspath(os.path.join(out_dir, f"last_frame_{job.id}.png"))
# Reuse an earlier extraction unless the video has been rewritten since.
if not (os.path.isfile(out_path) and os.path.getmtime(out_path) >= os.path.getmtime(job.output_path)):
try:
save_last_frame(job.output_path, out_path)
except Exception as e:
raise FrameError(500, f"Could not read the last frame: {e}") from e
return out_path
DEFAULT_LAST_CLIP_SECONDS = 1.0
def save_last_clip(video_path: str, out_path: str, seconds: float = DEFAULT_LAST_CLIP_SECONDS) -> None:
"""Decode the final ``seconds`` of ``video_path`` and re-encode them to ``out_path``.
Re-encoded rather than stream-copied: the source's keyframes rarely align
with an arbitrary cut ``seconds`` from the end, and a copy can only start on
one. Silent -- the trailing video is for motion continuity; voice continuity
already comes from each job's own audio reference.
"""
import av
if seconds <= 0:
raise ValueError(f"seconds must be positive, got {seconds}.")
tmp_path = f"{out_path}.tmp.mp4"
with av.open(video_path) as src:
video_stream = src.streams.video[0]
fps = float(video_stream.average_rate or video_stream.guessed_rate or 24)
frames = list(src.decode(video_stream))
if not frames:
raise ValueError("The video has no frames.")
keep = max(1, round(seconds * fps))
tail = frames[-keep:]
with av.open(tmp_path, mode="w") as out:
out_stream = out.add_stream("h264", rate=round(fps))
out_stream.width, out_stream.height = tail[0].width, tail[0].height
out_stream.pix_fmt = "yuv420p"
# Each frame needs its own increasing pts at a known time base, or PyAV
# stamps every frame identically (pts=0), which corrupts the muxed
# container's average-frame-rate metadata -- H3's reference pipeline
# trusts that metadata to resample the clip, so a wrong rate there
# silently collapses a short clip down to as little as one frame.
frame_time_base = Fraction(1, round(fps))
for i, frame in enumerate(tail):
frame.pts = i
frame.time_base = frame_time_base
for packet in out_stream.encode(frame):
out.mux(packet)
for packet in out_stream.encode():
out.mux(packet)
os.replace(tmp_path, out_path) # atomic, so a reader never sees a half-written file
def last_clip_for_job(job: Any, upload_dir: str, seconds: float = DEFAULT_LAST_CLIP_SECONDS) -> str:
"""Path of a short video of the end of ``job``'s video, extracting it if needed.
Raises :class:`FrameError` with the status the API should return.
"""
status = getattr(job.status, "value", job.status)
if status != "completed" or not job.output_path:
raise FrameError(404, "No output available for this job")
if not os.path.isfile(job.output_path):
raise FrameError(404, "Output file not found on disk")
if Path(job.output_path).suffix.lower() not in VIDEO_EXTENSIONS:
raise FrameError(400, "This job's output is not a video")
if not upload_dir:
raise FrameError(503, "Upload directory not configured")
out_dir = os.path.join(upload_dir, "last_clips")
os.makedirs(out_dir, exist_ok=True)
out_path = os.path.abspath(os.path.join(out_dir, f"last_clip_{job.id}.mp4"))
if not (os.path.isfile(out_path) and os.path.getmtime(out_path) >= os.path.getmtime(job.output_path)):
try:
save_last_clip(job.output_path, out_path, seconds)
except Exception as e:
raise FrameError(500, f"Could not read the end of the clip: {e}") from e
return out_path
+208
View File
@@ -0,0 +1,208 @@
# SPDX-License-Identifier: Apache-2.0
"""Scheduling rules for queued jobs, and references to another job's ending.
Kept apart from ``job_runner.py`` because it is pure -- it works on plain
snapshots and needs neither FastVideo nor a GPU -- so it can be tested anywhere.
A job can start from how another job's video ended -- either its last frame (a
still, for a hard "opens on this exact picture" continuation) or its last
second or so (a short video, so the model has actual motion to continue, not
just a pose to guess motion from). Neither exists until the other job
finishes, so the reference is *deferred*: its source is ``job-last-frame:<job
id>`` or ``job-last-clip:<job id>``, resolved to a real file only when the job
runs. That link is also the queue's dependency graph: a queued job waits until
every job it takes an ending from has completed.
"""
from __future__ import annotations
from collections.abc import Callable, Iterable, Mapping
from dataclasses import dataclass, field
from typing import Any, Protocol
DEFERRED_LAST_FRAME_PREFIX = "job-last-frame:"
DEFERRED_LAST_CLIP_PREFIX = "job-last-clip:"
_DEFERRED_PREFIXES = (DEFERRED_LAST_FRAME_PREFIX, DEFERRED_LAST_CLIP_PREFIX)
QUEUED = "queued"
RUNNING = "running"
COMPLETED = "completed"
PENDING = "pending"
FAILED = "failed"
STOPPED = "stopped"
def deferred_last_frame_source(job_id: str) -> str:
return f"{DEFERRED_LAST_FRAME_PREFIX}{job_id}"
def deferred_last_clip_source(job_id: str) -> str:
return f"{DEFERRED_LAST_CLIP_PREFIX}{job_id}"
def dependency_of(reference: Mapping[str, Any] | None) -> str | None:
"""The job a deferred reference takes its ending from, or None for any other."""
source = (reference or {}).get("source")
if not isinstance(source, str):
return None
for prefix in _DEFERRED_PREFIXES:
if source.startswith(prefix):
return source[len(prefix):] or None
return None
def dependencies(references: Iterable[Mapping[str, Any]] | None) -> list[str]:
"""Job ids a job needs finished first, in reference order, without repeats."""
found: list[str] = []
for ref in references or []:
dep = dependency_of(ref)
if dep and dep not in found:
found.append(dep)
return found
class JobView(Protocol):
"""The parts of a job the scheduler looks at."""
id: str
references: Any
queued_at: float | None
@property
def status(self) -> Any: # a JobStatus (str enum) or plain str
...
def _status(job: Any) -> str:
return str(getattr(job.status, "value", job.status))
def circular_dependency(jobs: Mapping[str, JobView], job_id: str) -> list[str] | None:
"""The chain of job ids if ``job_id`` (transitively) waits on itself, else None."""
def walk(current: str, path: list[str]) -> list[str] | None:
job = jobs.get(current)
if job is None:
return None
for dep in dependencies(job.references):
if dep == job_id:
return [*path, current, dep]
if dep in path or dep == current:
continue
found = walk(dep, [*path, current])
if found:
return found
return None
return walk(job_id, [])
def _label(job: Any) -> str:
name = getattr(job, "name", "") or ""
return f"'{name}'" if name else job.id
@dataclass
class Plan:
"""What to do now: jobs to launch, and jobs that can never run."""
start: list[str] = field(default_factory=list)
fail: dict[str, str] = field(default_factory=dict) # job id -> reason
def plan_dispatch(jobs: Mapping[str, JobView], max_concurrent: int) -> Plan:
"""Decide which queued jobs launch now and which have become impossible.
* Queued jobs run oldest-queued first, at most ``max_concurrent`` at a time
(counting jobs already running).
* A job whose frame source hasn't completed *waits*, and doesn't hold up
jobs behind it that don't need it.
* A job whose frame source failed, was stopped, or was deleted can never
run, so it fails -- and so does anything waiting on *it*.
* A job whose frame source is idle (neither queued nor running) keeps
waiting: the user may still queue it.
"""
plan = Plan()
status = {jid: _status(job) for jid, job in jobs.items()}
queued = sorted(
(job for job in jobs.values() if status[job.id] == QUEUED),
key=lambda job: (job.queued_at or 0.0, job.id),
)
# Failures cascade down a chain, so repeat until nothing new fails.
changed = True
while changed:
changed = False
for job in queued:
if job.id in plan.fail:
continue
for dep_id in dependencies(job.references):
dep = jobs.get(dep_id)
if dep is None:
plan.fail[job.id] = "Waiting on the ending of a job that was deleted."
elif dep_id in plan.fail or status[dep_id] in (FAILED, STOPPED):
reason = "failed" if dep_id in plan.fail or status[dep_id] == FAILED else "was stopped"
plan.fail[job.id] = f"Needs the ending of {_label(dep)}, which {reason}."
else:
continue
status[job.id] = FAILED
changed = True
break
slots = max(max_concurrent, 1) - sum(1 for s in status.values() if s == RUNNING)
for job in queued:
if slots <= 0:
break
if job.id in plan.fail:
continue
if all(status.get(dep) == COMPLETED for dep in dependencies(job.references)):
plan.start.append(job.id)
slots -= 1
return plan
class UnresolvedReference(ValueError):
"""A deferred reference can't be turned into a real file."""
def resolve_references(
references: list[dict[str, Any]] | None,
get_job: Callable[[str], Any],
last_frame: Callable[[Any], str],
last_clip: Callable[[Any], str] | None = None,
) -> list[dict[str, Any]]:
"""Copy of ``references`` with each deferred reference made real.
``last_frame(job)`` and ``last_clip(job)`` each return the path of an
extracted file and may raise (see ``frames.FrameError``); the message is
passed on. ``last_clip`` is optional so callers that never attach a
trailing-clip reference (e.g. the mock server) don't need to supply one --
it is only required if a ``job-last-clip:`` reference is actually present.
Other references are returned unchanged, and the stored list is never
modified, so a re-run resolves again against the source job's current
output.
"""
resolved: list[dict[str, Any]] = []
for ref in references or []:
source = (ref or {}).get("source")
prefix = next((p for p in _DEFERRED_PREFIXES if isinstance(source, str) and source.startswith(p)), None)
if prefix is None:
resolved.append(ref)
continue
dep_id = source[len(prefix):] or None
extract = last_frame if prefix == DEFERRED_LAST_FRAME_PREFIX else last_clip
what = "the last frame of" if prefix == DEFERRED_LAST_FRAME_PREFIX else "the ending of"
if extract is None:
raise UnresolvedReference("This server can't resolve a trailing-clip reference yet.")
dep = get_job(dep_id) if dep_id else None
if dep is None:
raise UnresolvedReference(f"A reference needs {what} a job that no longer exists.")
if _status(dep) != COMPLETED:
raise UnresolvedReference(f"A reference needs {what} {_label(dep)}, which hasn't finished "
f"(it is {_status(dep)}).")
try:
path = extract(dep)
except Exception as exc: # FrameError carries a readable .detail
raise UnresolvedReference(str(getattr(exc, "detail", None) or exc)) from exc
resolved.append({**ref, "source": path})
return resolved
+404 -46
View File
@@ -17,6 +17,7 @@ import logging
import logging.handlers
import multiprocessing as mp
import os
import queue
import re
import subprocess
import sys
@@ -29,6 +30,15 @@ import yaml
from fastvideo.utils import get_mp_context
from fastvideo_studio.database import Database
from fastvideo_studio.frames import last_clip_for_job, last_frame_for_job
from fastvideo_studio.trim import restore_original, trim_video
from fastvideo_studio.job_queue import (
UnresolvedReference,
circular_dependency,
dependencies,
plan_dispatch,
resolve_references,
)
from fastvideo_studio.training_config import (
build_training_config,
get_training_env,
@@ -46,6 +56,7 @@ _MAX_LOG_LINES = 2000 # ring-buffer cap per job
class JobStatus(str, enum.Enum):
PENDING = "pending"
QUEUED = "queued"
RUNNING = "running"
COMPLETED = "completed"
FAILED = "failed"
@@ -138,6 +149,16 @@ class Job:
finished_at: float | None = None
error: str | None = None
output_path: str | None = None
queued_at: float | None = None
# The range/sections last applied via trim_job(), so the Edit video dialog
# can reopen showing what actually produced the current output_path instead
# of resetting to neutral. Reset to these defaults by restore_job_video().
edit_start_seconds: float = 0.0
edit_end_seconds: float | None = None
#: Ordered, contiguous razor-cut sections, each a {end_seconds, brightness,
#: contrast, saturation} dict; see trim.py's DEFAULT_SEGMENTS. Empty means
#: never edited (or restored) -- one neutral section covering everything.
edit_segments: list[dict[str, Any]] = field(default_factory=list)
log_file_path: str | None = None # Path to the job's log file
num_inference_steps: int = 50
num_frames: int = 81
@@ -195,11 +216,15 @@ class Job:
"references": self.references,
"status": self.status.value,
"created_at": self.created_at,
"queued_at": self.queued_at,
"started_at": self.started_at,
"finished_at": self.finished_at,
"error": self.error,
"output_path": self.output_path,
"log_file_path": self.log_file_path,
"edit_start_seconds": self.edit_start_seconds,
"edit_end_seconds": self.edit_end_seconds,
"edit_segments": self.edit_segments,
"num_inference_steps": self.num_inference_steps,
"num_frames": self.num_frames,
"height": self.height,
@@ -278,8 +303,8 @@ def _job_log_path(output_dir: str, job_id: str) -> str:
return os.path.join(output_dir, job_id, JOB_LOG_FILENAME)
def _decode_references(value: Any) -> list[dict[str, Any]]:
"""Reference lists round-trip through the DB as JSON text."""
def _decode_json_list(value: Any) -> list[dict[str, Any]]:
"""References and edit_segments both round-trip through the DB as JSON text."""
if not value:
return []
if isinstance(value, list):
@@ -287,7 +312,7 @@ def _decode_references(value: Any) -> list[dict[str, Any]]:
try:
decoded = json.loads(value)
except (TypeError, ValueError):
logger.warning("Could not decode stored references: %r", value)
logger.warning("Could not decode stored JSON list: %r", value)
return []
return list(decoded) if isinstance(decoded, list) else []
@@ -320,6 +345,8 @@ class JobRunner:
log_dir: str,
database: Database,
verbose: bool = False,
max_concurrent_jobs: int = 1,
upload_dir: str = "",
):
"""Initialize the job runner.
@@ -328,19 +355,38 @@ class JobRunner:
log_dir: Directory where job log files are saved
verbose: Whether to print full tracebacks in error messages
database: Optional SQLite database for job persistence
max_concurrent_jobs: How many *queued* jobs may run at once. Jobs
started directly with start_job() run immediately regardless
(but count toward the limit, so the queue waits for them).
upload_dir: Where extracted last frames are saved.
"""
self.output_dir = output_dir
self.log_dir = log_dir
self.verbose = verbose
self._db = database
self.max_concurrent_jobs = max(max_concurrent_jobs, 1)
self.upload_dir = upload_dir or os.path.join(output_dir, "uploads")
self._jobs: dict[str, Job] = {}
self._jobs_lock = threading.Lock()
# Serializes queue decisions (enqueue / dequeue / dispatch) so two
# finishing jobs can't both take the same free slot.
self._queue_lock = threading.RLock()
self._last_queued_at = 0.0
self._load_jobs()
# Cache of loaded generators keyed by model config so that we only pay
# the model-loading cost once per model configuration.
self._generators: dict[tuple, Any] = {}
# Cached generators are keyed by (owning thread, config). The model's GPU
# worker processes are killed (PR_SET_PDEATHSIG) when the *thread* that
# started them exits, so a generator is only safe for that thread to reuse
# -- which is why inference jobs run on long-lived worker threads (below)
# rather than a new thread each.
self._generator_threads: dict[tuple, threading.Thread] = {}
# Inference jobs are handed to these threads through one shared queue.
self._work: queue.Queue[Job | None] = queue.Queue()
self._workers: list[threading.Thread] = []
self._generators_lock = threading.Lock()
# Shared Manager for log queues (avoids spawning a new process per job)
@@ -351,6 +397,14 @@ class JobRunner:
os.makedirs(self.output_dir, exist_ok=True)
os.makedirs(self.log_dir, exist_ok=True)
for i in range(self.max_concurrent_jobs):
worker = threading.Thread(target=self._worker_loop, name=f"inference-worker-{i}", daemon=True)
worker.start()
self._workers.append(worker)
# Jobs that were queued when the server last stopped carry on.
self._dispatch()
def _load_logs(self, job: Job) -> None:
"""Populate job's log buffer from its log file if it exists."""
path = job.log_file_path
@@ -399,7 +453,7 @@ class JobRunner:
job_type=row.get("job_type", "inference"),
image_path=row.get("image_path", "") or "",
last_image_path=row.get("last_image_path", "") or "",
references=_decode_references(row.get("references")),
references=_decode_json_list(row.get("references")),
data_path=row.get("data_path", "") or "",
max_train_steps=row.get("max_train_steps", 1000),
train_batch_size=row.get("train_batch_size", 1),
@@ -416,11 +470,16 @@ class JobRunner:
fake_score_model_path=row.get("fake_score_model_path", "") or "",
status=JobStatus(status),
created_at=row["created_at"],
queued_at=row.get("queued_at"),
started_at=row.get("started_at"),
finished_at=row.get("finished_at"),
error=row.get("error"),
output_path=row.get("output_path"),
log_file_path=row.get("log_file_path"),
# `or <default>` would be wrong here: 0.0 is a real, intentional value, not just "unset".
edit_start_seconds=(float(row["edit_start_seconds"]) if row.get("edit_start_seconds") is not None else 0.0),
edit_end_seconds=(float(row["edit_end_seconds"]) if row.get("edit_end_seconds") is not None else None),
edit_segments=_decode_json_list(row.get("edit_segments")),
num_inference_steps=row.get("num_inference_steps", 50),
num_frames=row.get("num_frames", 81),
height=row.get("height", 480),
@@ -456,6 +515,7 @@ class JobRunner:
self._db.update_job(
job.id, {
"status": job.status.value,
"queued_at": job.queued_at,
"started_at": job.started_at,
"finished_at": job.finished_at,
"error": job.error,
@@ -466,7 +526,9 @@ class JobRunner:
logger.warning("Failed to persist job %s: %s", job.id, exc)
def _shutdown(self) -> None:
"""Shutdown the shared multiprocessing manager on exit."""
"""Stop the inference workers and the shared multiprocessing manager on exit."""
for _ in self._workers:
self._work.put(None)
try:
self._mp_manager.shutdown()
except Exception as exc:
@@ -610,6 +672,9 @@ class JobRunner:
except Exception as exc:
logger.warning("Failed to delete job %s from database: %s", job_id, exc)
logger.info("Deleted job %s", job.id)
# Anything queued behind this job's last frame can no longer run, and a
# freed queue slot may let the next job start.
self._dispatch()
return True
CONFIG_FIELDS: tuple[str, ...] = (
@@ -688,24 +753,30 @@ class JobRunner:
for field_name, value in updates.items():
setattr(job, field_name, value)
self._save_job(job)
try:
self._db.update_job_config(job.id, updates)
except Exception as exc:
logger.warning("Failed to persist edits to job %s: %s", job.id, exc)
return job
def start_job(self, job_id: str) -> Job:
"""Start (or restart) a pending / stopped / failed job.
Raises:
ValueError: If job not found or cannot be started
"""
#: Statuses a job can be started or queued from.
STARTABLE_STATUSES = (JobStatus.PENDING, JobStatus.FAILED, JobStatus.STOPPED)
def _startable_job(self, job_id: str) -> Job:
with self._jobs_lock:
job = self._jobs.get(job_id)
if job is None:
raise ValueError(f"Job {job_id} not found")
if job.status == JobStatus.RUNNING:
raise ValueError("Job is already running")
if job.status == JobStatus.QUEUED:
raise ValueError("Job is already queued")
if job.status == JobStatus.COMPLETED:
raise ValueError("Job already completed. Delete and re-create to run again.")
# Reset state for re-run
job.status = JobStatus.PENDING
return job
def _reset_for_run(self, job: Job) -> None:
"""Clear the previous attempt's state so the job starts clean."""
job.error = None
job.output_path = None
job.log_file_path = None # Will be set when job starts
@@ -716,26 +787,189 @@ class JobRunner:
job.log_file_handler = None # Reset file handler
job._process = None # Reset subprocess handle
# Wrap _run_job in an additional safety layer to catch any exceptions
# that might escape (though they shouldn't with our comprehensive handling)
def safe_run_job(job: Job):
"""Wrapper to ensure _run_job never raises an unhandled exception."""
try:
self._run_job(job)
except BaseException as exc:
# This should never happen, but if it does, we catch it here
logger.critical("Unhandled exception escaped from _run_job for job %s: %s", job.id, exc, exc_info=True)
with contextlib.suppress(Exception):
job.status = JobStatus.FAILED
job.error = (f"Unhandled exception: {type(exc).__name__}: {str(exc)}")
job.finished_at = time.time()
def _launch(self, job: Job) -> None:
"""Start ``job``; when it ends, look for more queued work.
thread = threading.Thread(target=safe_run_job, args=(job, ), daemon=True)
Inference jobs go to a long-lived worker thread so the loaded model
survives from one job to the next. Training jobs only babysit a
subprocess, so they get a thread of their own.
"""
# Marked running now, not when the job gets going, so the queue counts it
# immediately and can't start one job too many.
job.status = JobStatus.RUNNING
job.started_at = time.time()
self._save_job(job)
if job.job_type == "inference":
self._work.put(job)
logger.info("Sent job %s to an inference worker", job.id)
return
def run_training_job(job: Job):
self._run_job_safely(job)
self._dispatch_after_exit(threading.current_thread())
thread = threading.Thread(target=run_training_job, args=(job, ), daemon=True)
job._thread = thread
thread.start()
logger.info("Started job %s", job.id)
def _run_job_safely(self, job: Job) -> None:
"""Run a job, turning anything that escapes into a failed job."""
try:
self._run_job(job)
except BaseException as exc:
# This should never happen, but if it does, we catch it here
logger.critical("Unhandled exception escaped from _run_job for job %s: %s", job.id, exc, exc_info=True)
with contextlib.suppress(Exception):
job.status = JobStatus.FAILED
job.error = (f"Unhandled exception: {type(exc).__name__}: {str(exc)}")
job.finished_at = time.time()
self._save_job(job)
def _worker_loop(self) -> None:
"""Run inference jobs one after another on this thread, for the life of the server.
The thread never exits between jobs. That matters: it started the model's
GPU processes, which are killed when it exits, so staying alive is what
lets the next job find the model already loaded.
"""
while True:
job = self._work.get()
if job is None:
return
self._run_job_safely(job)
# The slot is free; dependents may now be runnable (or impossible).
with contextlib.suppress(Exception):
self._dispatch()
def start_job(self, job_id: str) -> Job:
"""Start (or restart) a pending / stopped / failed job right now.
Refused if every inference worker is already busy; use enqueue_job() to
wait for a free one. A job that takes another job's last frame can only
start once that job has completed.
Raises:
ValueError: If job not found or cannot be started
"""
with self._queue_lock:
job = self._startable_job(job_id)
# Fail here, before touching the job, rather than as a failed run.
self._check_dependencies_finished(job)
if job.job_type == "inference":
busy = self._running_inference_jobs()
if busy >= self.max_concurrent_jobs:
raise ValueError(f"{busy} inference job(s) already running and the model workers are busy. "
"Queue this job to run next.")
self._reset_for_run(job)
job.queued_at = None
self._launch(job)
return job
def _running_inference_jobs(self) -> int:
with self._jobs_lock:
return sum(1 for j in self._jobs.values() if j.job_type == "inference" and j.status == JobStatus.RUNNING)
def _check_dependencies_finished(self, job: Job) -> None:
for dep_id in dependencies(job.references):
dep = self.get_job(dep_id)
if dep is None:
raise ValueError("This job starts from the ending of a job that no longer exists.")
if dep.status != JobStatus.COMPLETED:
label = f"'{dep.name}'" if dep.name else dep.id
raise ValueError(f"This job starts from the ending of {label}, which hasn't finished "
f"(it is {dep.status.value}). Queue it to run after that job.")
# -- queue ---------------------------------------------------------------
def _next_queued_at(self) -> float:
"""A strictly increasing timestamp, so jobs queued together keep their order."""
self._last_queued_at = max(time.time(), self._last_queued_at + 1e-6)
return self._last_queued_at
def enqueue_jobs(self, job_ids: list[str]) -> list[Job]:
"""Queue jobs, in the given order, to run as slots free up.
All-or-nothing: if any job can't be queued, none are.
Raises:
ValueError: If a job is missing, not startable, or waits on itself.
"""
if len(set(job_ids)) != len(job_ids):
raise ValueError("A job was listed more than once.")
with self._queue_lock:
jobs = [self._startable_job(job_id) for job_id in job_ids]
with self._jobs_lock:
snapshot = dict(self._jobs)
for job in jobs:
loop = circular_dependency(snapshot, job.id)
if loop:
raise ValueError("Jobs would wait on each other's last frames in a loop: " + " -> ".join(loop))
for job in jobs:
self._reset_for_run(job)
job.status = JobStatus.QUEUED
job.queued_at = self._next_queued_at()
self._save_job(job)
logger.info("Queued job %s", job.id)
self._dispatch()
return jobs
def enqueue_job(self, job_id: str) -> Job:
return self.enqueue_jobs([job_id])[0]
def dequeue_job(self, job_id: str) -> Job:
"""Take a queued job out of the queue, back to pending.
Raises:
ValueError: If the job is missing or isn't queued.
"""
with self._queue_lock:
job = self.get_job(job_id)
if job is None:
raise ValueError(f"Job {job_id} not found")
if job.status != JobStatus.QUEUED:
raise ValueError(f"Job is not queued (status={job.status.value})")
job.status = JobStatus.PENDING
job.queued_at = None
self._save_job(job)
logger.info("Dequeued job %s", job.id)
# Anything queued behind this one's frame keeps waiting; a slot it held
# in line may now go to another job.
self._dispatch()
return job
def _dispatch_after_exit(self, finished: threading.Thread) -> None:
"""Wait for a job's thread to end, then start whatever is next in the queue."""
finished.join()
with contextlib.suppress(Exception):
self._dispatch()
def _dispatch(self) -> None:
"""Launch queued jobs that are ready, and fail those that can never be."""
with self._queue_lock:
with self._jobs_lock:
snapshot = dict(self._jobs)
plan = plan_dispatch(snapshot, self.max_concurrent_jobs)
for job_id, reason in plan.fail.items():
job = snapshot[job_id]
job.status = JobStatus.FAILED
job.error = reason
job.finished_at = time.time()
self._save_job(job)
logger.warning("Queued job %s can't run: %s", job_id, reason)
for job_id in plan.start:
job = snapshot[job_id]
job.queued_at = None
try:
self._launch(job)
except Exception as exc: # e.g. the OS refused a thread
logger.exception("Could not launch queued job %s", job_id)
job.status = JobStatus.FAILED
job.error = f"Could not start: {exc}"
job.finished_at = time.time()
self._save_job(job)
def stop_job(self, job_id: str) -> Job:
"""Request a running job to stop.
@@ -749,6 +983,8 @@ class JobRunner:
job = self._jobs.get(job_id)
if job is None:
raise ValueError(f"Job {job_id} not found")
if job.status == JobStatus.QUEUED:
return self.dequeue_job(job_id)
if job.status != JobStatus.RUNNING:
raise ValueError(f"Job is not running (status={job.status.value})")
@@ -759,6 +995,91 @@ class JobRunner:
logger.info("Stop requested for job %s", job.id)
return job
# -- trimming --------------------------------------------------------------
def _set_output(
self,
job: Job,
output_path: str,
num_frames: int,
*,
edit_start_seconds: float = 0.0,
edit_end_seconds: float | None = None,
edit_segments: list[dict[str, Any]] | None = None,
) -> None:
"""Point a completed job at a different output file, e.g. after trimming.
The edit_* keyword args record what produced that file (left at their
neutral/full-range defaults by restore_job_video()), so the Edit video
dialog can reopen showing the edit actually in effect instead of always
resetting to neutral -- nothing else needs these; other jobs' last-frame/
last-clip references and a merge only ever read output_path.
Bypasses update_job_config()'s pending/failed/stopped gate on purpose:
a completed job's *generation* config stays fixed, but which cut of its
own video is showing is not a generation setting.
"""
edit_segments = edit_segments or []
job.output_path = output_path
job.num_frames = num_frames
job.edit_start_seconds = edit_start_seconds
job.edit_end_seconds = edit_end_seconds
job.edit_segments = edit_segments
self._save_job(job)
try:
self._db.update_job_config(
job.id, {
"num_frames": num_frames,
"edit_start_seconds": edit_start_seconds,
"edit_end_seconds": edit_end_seconds,
"edit_segments": edit_segments,
})
except Exception as exc:
logger.warning("Failed to persist edit state for job %s: %s", job.id, exc)
def trim_job(
self,
job_id: str,
start_seconds: float = 0.0,
end_seconds: float | None = None,
segments: list[dict[str, Any]] | None = None,
) -> Job:
"""Cut a completed job's video to [start_seconds, end_seconds) and grade it section by section, in place.
The untouched original is kept, so this can be called again with a
different range and/or sections -- it always renders from that original
with the full set of parameters given, never compounding quality loss
or discarding one edit by changing the other -- or undone with
restore_job_video().
"""
with self._jobs_lock:
job = self._jobs.get(job_id)
if job is None:
raise ValueError(f"Job {job_id} not found")
path, kept = trim_video(job, start_seconds, end_seconds, segments)
self._set_output(
job,
path,
kept,
edit_start_seconds=start_seconds,
edit_end_seconds=end_seconds,
edit_segments=segments,
)
logger.info("Edited job %s: %.2f-%s s, %d section(s) (%d frames)", job.id, start_seconds,
f"{end_seconds:.2f}" if end_seconds is not None else "end", len(segments or [1]), kept)
return job
def restore_job_video(self, job_id: str) -> Job:
"""Undo every trim on a job's video, back to what it originally generated."""
with self._jobs_lock:
job = self._jobs.get(job_id)
if job is None:
raise ValueError(f"Job {job_id} not found")
path, kept = restore_original(job)
self._set_output(job, path, kept)
logger.info("Restored job %s to its original video", job.id)
return job
def get_job_logs(self, job_id: str, after: int = 0) -> dict[str, Any]:
"""Return log lines for a job.
@@ -786,6 +1107,22 @@ class JobRunner:
"phase": job._log_buf.phase,
}
def _discard_generators(self) -> None:
"""Unload this thread's models so the next job on it starts from a clean load."""
with self._generators_lock:
for key in [k for k in self._generators if k[0] == threading.current_thread().ident]:
self._drop_generator(key)
def _drop_generator(self, key: tuple) -> None:
"""Forget a cached generator and shut it down. Call with ``_generators_lock`` held."""
gen = self._generators.pop(key, None)
self._generator_threads.pop(key, None)
if gen is not None:
try:
gen.shutdown()
except Exception:
logger.debug("Shutdown of a cached generator failed", exc_info=True)
def _get_or_create_generator(
self,
model_id: str,
@@ -823,23 +1160,27 @@ class JobRunner:
sp_size,
)
# Generators are cached by model_id and configuration parameters
# Generators are cached per thread and configuration. Only the thread that
# built one may reuse it: its GPU processes die with that thread.
thread = threading.current_thread()
entry_key = (thread.ident, cache_key)
with self._generators_lock:
cached = self._generators.get(cache_key)
# Generators whose thread has exited have dead workers already.
for key, owner in list(self._generator_threads.items()):
if not owner.is_alive():
self._drop_generator(key)
cached = self._generators.get(entry_key)
if cached is not None:
if _generator_is_alive(cached):
return cached
# Workers can exit while a generator sits idle in the cache;
# reusing it fails every later job with the same config.
logger.warning(
"Cached generator for %s has dead workers; reloading.",
model_id,
)
self._generators.pop(cache_key, None)
try:
cached.shutdown()
except Exception:
logger.debug("Shutdown of the dead generator failed", exc_info=True)
# Workers can exit while a generator sits idle in the cache.
logger.warning("Cached generator for %s has dead workers; reloading.", model_id)
self._drop_generator(entry_key)
# A worker keeps one model resident. A different configuration replaces
# it, freeing its GPU memory first, rather than loading beside it.
for key in [k for k in self._generators if k[0] == thread.ident]:
logger.info("Unloading the previous model to load %s.", model_id)
self._drop_generator(key)
# Import lazily so starting the server is fast even without a GPU.
from fastvideo import VideoGenerator
@@ -883,11 +1224,8 @@ class JobRunner:
)
with self._generators_lock:
if cache_key not in self._generators:
self._generators[cache_key] = gen
else: # Another thread may have created it while we were loading.
gen.shutdown()
gen = self._generators[cache_key]
self._generators[entry_key] = gen
self._generator_threads[entry_key] = thread
return gen
def _run_job(self, job: Job):
@@ -1064,6 +1402,15 @@ class JobRunner:
logger.warning("Job stopped before execution started")
return
# Frames and trailing clips borrowed from other jobs are extracted now,
# when they exist. Done before loading the model so a missing one fails fast.
resolved_references = resolve_references(
job.references,
self.get_job,
last_frame=lambda source: last_frame_for_job(source, self.upload_dir),
last_clip=lambda source: last_clip_for_job(source, self.upload_dir),
)
buf.phase = "loading model"
logger.info("Loading model...")
@@ -1121,7 +1468,7 @@ class JobRunner:
if job.image_path:
gen_kwargs["image_path"] = job.image_path
if job.references:
gen_kwargs["references"] = _build_h3_references(job.references)
gen_kwargs["references"] = _build_h3_references(resolved_references)
if job.last_image_path:
# _prepare_fl2va requires a PIL image, not a path.
from PIL import Image as _PILImage
@@ -1148,6 +1495,7 @@ class JobRunner:
if job._stop_event.is_set():
job.status = JobStatus.STOPPED
logger.warning("Job was stopped during execution")
self._discard_generators()
else:
job.status = JobStatus.COMPLETED
buf.progress = 100.0
@@ -1156,9 +1504,19 @@ class JobRunner:
self._save_job(job)
buf.phase = "done"
except UnresolvedReference as exception:
logger.error("Job %s can't get its reference frame: %s", job.id, exception)
job.status = JobStatus.FAILED
job.error = str(exception)
job.finished_at = time.time()
self._save_job(job)
buf.phase = "failed"
except Exception as exception:
error_msg = str(exception)
logger.exception("Critical error in job thread: %s", error_msg)
# The model may be mid-request. Reload rather than trust it for the next job.
self._discard_generators()
job.status = JobStatus.FAILED
job.error = f"Critical error ({type(exception).__name__}): {error_msg}"
job.finished_at = time.time()
+223
View File
@@ -0,0 +1,223 @@
# SPDX-License-Identifier: Apache-2.0
"""Join a scene's clips, in order, into one video.
Kept apart from ``server.py`` because it needs nothing from FastVideo -- only
PyAV (to read each clip's format) and an ``ffmpeg`` binary -- so it can be
exercised without a GPU.
Clips from one scene share codec, size and frame rate, so they are joined by
copying the streams rather than re-encoding: it takes about a second, loses no
quality, and keeps picture and sound in step. Each clip is cut off at the end of
its last video frame, because the audio track runs a few milliseconds past it and
would otherwise open a gap between clips.
"""
from __future__ import annotations
import os
import re
import shutil
import subprocess
import tempfile
import time
from dataclasses import dataclass
from pathlib import Path
from typing import Any
VIDEO_EXTENSIONS = {".mp4", ".mov", ".mkv", ".webm", ".avi"}
MERGED_DIRNAME = "merged"
_FILENAME_RE = re.compile(r"^[\w.-]+\.mp4$")
class MergeError(Exception):
"""A merge that can't be done, carrying the HTTP status to report."""
def __init__(self, status_code: int, detail: str):
super().__init__(detail)
self.status_code = status_code
self.detail = detail
def find_ffmpeg() -> str | None:
"""An ``ffmpeg`` to run: $FASTVIDEO_FFMPEG_BIN, then PATH, then imageio-ffmpeg's bundled one."""
configured = os.getenv("FASTVIDEO_FFMPEG_BIN")
if configured and (shutil.which(configured) or os.path.isfile(configured)):
return shutil.which(configured) or configured
on_path = shutil.which("ffmpeg")
if on_path:
return on_path
try:
import imageio_ffmpeg
return imageio_ffmpeg.get_ffmpeg_exe()
except Exception:
return None
@dataclass(frozen=True)
class ClipInfo:
path: str
video_codec: str
width: int
height: int
fps: float
pix_fmt: str
video_seconds: float
audio: tuple[str, int, int] | None # codec, sample rate, channels
def describe(self) -> str:
audio = f"{self.audio[0]} {self.audio[1]} Hz x{self.audio[2]}" if self.audio else "no audio"
return f"{self.width}x{self.height} {self.fps:.2f} fps {self.video_codec}/{self.pix_fmt}, {audio}"
def signature(self) -> tuple[Any, ...]:
return (self.video_codec, self.width, self.height, round(self.fps, 3), self.pix_fmt, self.audio)
def probe_clip(path: str) -> ClipInfo:
"""Read what ``check_compatible`` needs from a video file."""
import av
try:
with av.open(path) as container:
video = next((s for s in container.streams if s.type == "video"), None)
if video is None:
raise MergeError(409, f"{os.path.basename(path)} has no video stream.")
audio = next((s for s in container.streams if s.type == "audio"), None)
fps = float(video.average_rate or video.guessed_rate or 0)
if video.duration is not None:
seconds = float(video.duration * video.time_base)
elif container.duration is not None:
seconds = container.duration / 1_000_000
else:
seconds = (video.frames or 0) / fps if fps else 0.0
if seconds <= 0:
raise MergeError(409, f"{os.path.basename(path)} has no playable video.")
return ClipInfo(
path=path,
video_codec=video.codec_context.name,
width=video.codec_context.width,
height=video.codec_context.height,
fps=fps,
pix_fmt=video.codec_context.pix_fmt or "",
video_seconds=seconds,
audio=((audio.codec_context.name, audio.codec_context.sample_rate, audio.codec_context.channels)
if audio else None),
)
except MergeError:
raise
except Exception as exc: # av.error.* on a corrupt or unreadable file
raise MergeError(409, f"Could not read {os.path.basename(path)}: {exc}") from exc
def check_compatible(infos: list[ClipInfo], labels: list[str]) -> None:
"""Refuse clips that can't be joined by copying, naming the first that differs."""
first = infos[0]
for info, label in zip(infos[1:], labels[1:], strict=True):
if info.signature() != first.signature():
raise MergeError(
409,
f"{label} ({info.describe()}) doesn't match {labels[0]} ({first.describe()}). "
"Clips need the same size, frame rate and audio to be joined.",
)
def _concat_list(infos: list[ClipInfo]) -> str:
lines = ["ffconcat version 1.0"]
for info in infos:
escaped = info.path.replace("'", "'\\''")
lines += [f"file '{escaped}'", f"outpoint {info.video_seconds:.6f}"]
return "\n".join(lines) + "\n"
def merge_clips(paths: list[str], out_path: str, labels: list[str] | None = None, ffmpeg: str | None = None) -> float:
"""Join ``paths`` in order into ``out_path``. Returns the merged length in seconds."""
if not paths:
raise MergeError(400, "No clips to merge.")
labels = labels or [os.path.basename(p) for p in paths]
ffmpeg = ffmpeg or find_ffmpeg()
if not ffmpeg:
raise MergeError(503, "ffmpeg isn't available on the server. Install it or set FASTVIDEO_FFMPEG_BIN.")
infos = [probe_clip(p) for p in paths]
check_compatible(infos, labels)
os.makedirs(os.path.dirname(out_path) or ".", exist_ok=True)
tmp_out = f"{out_path}.tmp.mp4"
list_fd, list_path = tempfile.mkstemp(suffix=".ffconcat", text=True)
try:
with os.fdopen(list_fd, "w", encoding="utf-8") as f:
f.write(_concat_list(infos))
result = subprocess.run(
[ffmpeg, "-y", "-loglevel", "error", "-f", "concat", "-safe", "0", "-i", list_path, "-c", "copy",
"-movflags", "+faststart", tmp_out],
capture_output=True,
text=True,
timeout=600,
check=False,
)
if result.returncode != 0:
tail = " ".join(result.stderr.strip().splitlines()[-3:])
raise MergeError(500, f"ffmpeg failed: {tail or 'unknown error'}")
os.replace(tmp_out, out_path) # atomic, so a reader never sees a half-written file
except subprocess.TimeoutExpired as exc:
raise MergeError(500, "ffmpeg took too long.") from exc
finally:
for leftover in (list_path, tmp_out):
try:
os.remove(leftover)
except OSError:
pass
return sum(i.video_seconds for i in infos)
@dataclass(frozen=True)
class MergeResult:
filename: str
path: str
clips: int
seconds: float
def _safe_stem(name: str) -> str:
return re.sub(r"[^\w.-]+", "-", name).strip("-.")[:60] or "scene"
def _label(job: Any) -> str:
name = (getattr(job, "name", "") or "").strip()
return f"'{name}'" if name else str(job.id)
def merge_jobs(jobs: list[Any], out_dir: str, name: str = "") -> MergeResult:
"""Join the finished videos of ``jobs`` (in the order given) into ``<out_dir>/merged``.
Every job must be completed with a video on disk; otherwise nothing is made,
so a scene missing a clip can't be mistaken for the whole thing.
"""
if not jobs:
raise MergeError(400, "No clips to merge.")
paths: list[str] = []
for job in jobs:
status = getattr(job.status, "value", job.status)
if status != "completed" or not job.output_path:
raise MergeError(409, f"{_label(job)} hasn't finished ({status}), so the scene can't be merged yet.")
if not os.path.isfile(job.output_path):
raise MergeError(409, f"{_label(job)}'s video is missing from disk.")
if Path(job.output_path).suffix.lower() not in VIDEO_EXTENSIONS:
raise MergeError(409, f"{_label(job)}'s output isn't a video.")
paths.append(job.output_path)
filename = f"{_safe_stem(name)}-{time.strftime('%Y%m%d-%H%M%S')}.mp4"
merged_dir = os.path.join(out_dir, MERGED_DIRNAME)
out_path = os.path.join(merged_dir, filename)
seconds = merge_clips(paths, out_path, labels=[_label(j) for j in jobs])
return MergeResult(filename=filename, path=out_path, clips=len(paths), seconds=seconds)
def merged_path(out_dir: str, filename: str) -> str:
"""Path of a merged video by filename, refusing anything that isn't one."""
if not _FILENAME_RE.match(filename):
raise MergeError(404, "Merged video not found.")
path = os.path.join(out_dir, MERGED_DIRNAME, filename)
if not os.path.isfile(path):
raise MergeError(404, "Merged video not found.")
return path
+584 -36
View File
@@ -4,8 +4,8 @@ In-memory mock of the FastVideo Studio API (``server.py``).
This mock implements the same ``/api`` routes and response shapes as the real
FastAPI server but keeps everything in memory and never touches the FastVideo
library, a GPU, or a database. It is meant to back the Playwright e2e suite so
the Next.js frontend can be exercised end-to-end without a real backend.
library or a GPU. It is meant to back the Playwright e2e suite so the Next.js
frontend can be exercised end-to-end without a real backend.
Job lifecycle is simulated by recording a start timestamp and *computing* the
status on read: a started job reports ``running`` for a few seconds and then
@@ -13,14 +13,24 @@ flips to ``completed`` with an ``output_path``, so polling the job list / logs
shows progression. Generated media is a tiny 1s ``testsrc`` MP4 built lazily
with ffmpeg and cached.
Pass ``--data-dir`` to seed jobs/datasets/settings from a real server's
persisted SQLite database instead of the built-in fake demo data, so you can
browse real job history through the mock without a GPU. This is a read-only
snapshot taken once at startup: nothing created or changed while the mock
runs is ever written back to that database, so it's safe to point at the
same ``--data-dir`` a real ``server.py`` is using.
Usage (from the ``apps/`` directory)::
PYTHONPATH=.. python -m fastvideo_studio.mock_server --port 8190
PYTHONPATH=.. python -m fastvideo_studio.mock_server --port 8190 --data-dir ../outputs/ui_data
"""
from __future__ import annotations
import contextlib
import json
import logging
import os
import random
import shutil
@@ -29,22 +39,36 @@ import tempfile
import threading
import time
import uuid
from pathlib import Path
from types import SimpleNamespace
from typing import Annotated, Any
from fastapi import FastAPI, File, HTTPException, UploadFile
from fastapi.middleware.cors import CORSMiddleware
from fastapi.responses import FileResponse, PlainTextResponse
from fastvideo_studio.database import default_settings_dict
from fastvideo_studio.models import (CreateDatasetRequest, CreateJobRequest, SettingsUpdate, UpdateCaptionRequest,
from fastvideo_studio.database import Database, default_settings_dict
from fastvideo_studio.job_queue import circular_dependency, dependencies, plan_dispatch
from fastvideo_studio.merge import MergeError, merge_jobs, merged_path
from fastvideo_studio.trim import BRIGHTNESS_RANGE, CONTRAST_RANGE, SATURATION_RANGE
from fastvideo_studio.models import (CreateDatasetRequest, CreateJobRequest, GradeSegment, MergeSceneRequest,
QueueJobsRequest, SettingsUpdate, TrimRequest, UpdateCaptionRequest,
model_label)
logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(name)s: %(message)s")
logger = logging.getLogger("fastvideo.studio.mock")
# --- Config -----------------------------------------------------------------
# How long a started job stays "running" before it flips to "completed".
COMPLETE_AFTER_SECONDS = 3.0
FFMPEG_BIN = shutil.which(os.getenv("FASTVIDEO_FFMPEG_BIN", "ffmpeg"))
# Marks a job dict as a frozen row loaded from a real --data-dir snapshot
# (see _load_snapshot_from_db) rather than something this process is
# simulating. Stripped from API responses by _public_job.
_FROM_SNAPSHOT_KEY = "_from_db_snapshot"
# A small catalogue of fake models keyed by workload type. Mirrors the real
# server's {id, label} shape (label derived from the HF-style path).
_MODELS_BY_WORKLOAD: dict[str, list[str]] = {
@@ -54,6 +78,10 @@ _MODELS_BY_WORKLOAD: dict[str, list[str]] = {
],
"i2v": [
"Wan-AI/Wan2.1-I2V-14B-480P-Diffusers",
# Real registered id (fastvideo/registry.py) -- needed so the frontend's
# `modelId.toLowerCase().includes('minimax-h3')` gate (speech-tag
# console, references, last-frame upload) actually lights up in the mock.
"MiniMaxAI/MiniMax-H3",
],
"t2i": [
"black-forest-labs/FLUX.1-schnell",
@@ -98,16 +126,76 @@ _mock_media_cache: dict[str, str] = {}
_mock_media_lock = threading.Lock()
def _testsrc_frame(width: int, height: int, index: int):
"""A cheap, dependency-light stand-in for ffmpeg's `testsrc` pattern:
a gradient that shifts over `index` so an encoded clip visibly animates."""
import numpy as np
frame = np.zeros((height, width, 3), dtype=np.uint8)
frame[:, :, 0] = np.linspace(0, 255, width, dtype=np.uint8)[None, :]
frame[:, :, 1] = np.linspace(0, 255, height, dtype=np.uint8)[:, None]
frame[:, :, 2] = (index * 10) % 256
return frame
def _build_mock_media_with_pyav(kind: str, path: str) -> None:
"""ffmpeg isn't installed on every dev box; PyAV (a real FastVideo
dependency already, used for reference-video decoding) can encode the
same tiny clip directly via its bundled libav, no external binary."""
from PIL import Image
if kind == "png":
Image.fromarray(_testsrc_frame(320, 240, 0)).save(path)
return
import av as pyav
# h264 is what every browser's <video> element actually plays; mpeg4
# (MPEG-4 Part 2) is a PyAV-always-has-it fallback that no mainstream
# browser decodes, so gallery/job previews would just fail to load.
try:
pyav.codec.Codec("h264", "w")
codec_name = "h264"
except Exception:
logger.warning("PyAV has no h264 encoder available; falling back to mpeg4, which most "
"browsers cannot play in <video>. Install ffmpeg for real preview playback.")
codec_name = "mpeg4"
container = pyav.open(path, mode="w")
try:
stream = container.add_stream(codec_name, rate=24)
stream.width = 320
stream.height = 240
stream.pix_fmt = "yuv420p"
for i in range(24): # 1s at 24fps, matching the ffmpeg path's clip length
frame = pyav.VideoFrame.from_ndarray(_testsrc_frame(320, 240, i), format="rgb24")
for packet in stream.encode(frame):
container.mux(packet)
for packet in stream.encode():
container.mux(packet)
finally:
container.close()
def _build_mock_media(kind: str) -> str:
"""Build (once) a tiny testsrc clip of the given ``kind`` with ffmpeg; cache the path."""
"""Build (once) a tiny testsrc-like clip of the given ``kind``; cache the path."""
with _mock_media_lock:
cached = _mock_media_cache.get(kind)
if cached and os.path.isfile(cached):
return cached
if not FFMPEG_BIN:
raise HTTPException(status_code=500, detail="ffmpeg is required to build mock media")
fd, path = tempfile.mkstemp(prefix="fvstudio_mock_", suffix=f".{kind}")
os.close(fd)
if not FFMPEG_BIN:
try:
_build_mock_media_with_pyav(kind, path)
except Exception as exc:
with contextlib.suppress(OSError):
os.remove(path)
raise HTTPException(status_code=500, detail=f"PyAV fallback failed to build mock {kind}: {exc}") from exc
if not os.path.isfile(path) or os.path.getsize(path) == 0:
raise HTTPException(status_code=500, detail=f"PyAV fallback produced no mock {kind} bytes")
_mock_media_cache[kind] = path
return path
if kind == "png":
command = [
FFMPEG_BIN,
@@ -166,33 +254,34 @@ def _build_mock_media(kind: str) -> str:
def _new_job_dict(req: CreateJobRequest) -> dict[str, Any]:
"""Build a job dict (mirrors job_runner.Job.to_dict()) in the pending state."""
"""Build a job dict (mirrors job_runner.Job.to_dict()) in the pending state.
Starts from req.model_dump() rather than re-listing CreateJobRequest's
fields by hand -- the previous version silently dropped name,
last_image_path, references, and every offload/tuning flag because it
was never updated as CreateJobRequest grew those fields. Spreading the
request means new fields flow through automatically.
"""
job_id = str(uuid.uuid4())
return {
**req.model_dump(),
"id": job_id,
"model_id": req.model_id,
"prompt": req.prompt,
"workload_type": req.workload_type or "t2v",
"job_type": req.job_type or "inference",
"image_path": req.image_path or "",
"negative_prompt": req.negative_prompt or "",
"data_path": req.data_path or "",
"status": "pending",
"created_at": time.time(),
"queued_at": None,
"started_at": None,
"finished_at": None,
"error": None,
"output_path": None,
"log_file_path": None,
"num_inference_steps": req.num_inference_steps,
"num_frames": req.num_frames,
"height": req.height,
"width": req.width,
"guidance_scale": req.guidance_scale,
"guidance_rescale": req.guidance_rescale,
"fps": req.fps,
"seed": req.seed,
"negative_prompt": req.negative_prompt or "",
"num_gpus": req.num_gpus,
"data_path": req.data_path or "",
"edit_start_seconds": 0.0,
"edit_end_seconds": None,
"edit_segments": [],
"progress": 0.0,
"progress_msg": "",
"phase": "pending",
@@ -204,8 +293,13 @@ def _advance_job(job: dict[str, Any]) -> None:
Status is *computed on read* from the recorded start timestamp, so polling
the job list / logs naturally shows pending -> running -> completed.
Jobs loaded from a persisted DB snapshot (``_FROM_SNAPSHOT_KEY``) are a
frozen historical record, not something this process is simulating — skip
them so a real "pending" or "running" row doesn't silently start ticking
toward a fake "completed" just because someone polled it.
"""
if job["status"] != "running" or not job.get("started_at"):
if job.get(_FROM_SNAPSHOT_KEY) or job["status"] != "running" or not job.get("started_at"):
return
elapsed = time.time() - job["started_at"]
if elapsed >= COMPLETE_AFTER_SECONDS:
@@ -219,9 +313,9 @@ def _advance_job(job: dict[str, Any]) -> None:
def _public_job(job: dict[str, Any]) -> dict[str, Any]:
"""Advance + return a copy safe to serialize."""
"""Advance + return a copy safe to serialize (internal-only keys stripped)."""
_advance_job(job)
return dict(job)
return {k: v for k, v in job.items() if not k.startswith("_")}
_LOG_TAIL = [
@@ -252,8 +346,8 @@ def _compute_logs(job: dict[str, Any]) -> dict[str, Any]:
"""Return JobLogs-shaped data, growing the visible lines as time passes."""
seq = _log_sequence(job)
status = job["status"]
if status == "pending":
return {"lines": [], "progress": 0.0, "progress_msg": "", "phase": "pending"}
if status in ("pending", "queued"):
return {"lines": [], "progress": 0.0, "progress_msg": "", "phase": status}
if status == "completed":
return {"lines": seq, "progress": 100.0, "progress_msg": "50/50 steps", "phase": "done"}
if status in ("failed", "stopped"):
@@ -346,6 +440,89 @@ def _seed() -> None:
}
# --- DB snapshot (--data-dir) ------------------------------------------------
def _decode_references(value: Any) -> list[dict[str, Any]]:
"""Reference lists round-trip through the DB as JSON text."""
if not value:
return []
if isinstance(value, list):
return list(value)
try:
decoded = json.loads(value)
except (TypeError, ValueError):
return []
return list(decoded) if isinstance(decoded, list) else []
def _job_from_snapshot(row: dict[str, Any]) -> dict[str, Any]:
"""Adapt a persisted job row into the mock's job dict shape.
Frozen as a read-only historical record (see _FROM_SNAPSHOT_KEY): the mock
never actually ran this job, so it doesn't get to simulate progress for
one. A row still marked "running" is orphaned -- the real JobRunner
reclassifies those as failed on its own restart, so mirror that here.
"""
job = dict(row)
status = job["status"]
if status == "running":
status = "failed"
job["error"] = job.get("error") or "Server restarted (job was running)"
job["finished_at"] = job.get("finished_at") or time.time()
elif status == "queued":
# The real server resumes its queue on restart; the mock never runs
# snapshot jobs, so leave it as something the user can queue again.
status = "pending"
job["queued_at"] = None
job["status"] = status
job["references"] = _decode_references(row.get("references"))
job["progress"] = 100.0 if status == "completed" else 0.0
job["progress_msg"] = "50/50 steps" if status == "completed" else ""
job["phase"] = "done" if status == "completed" else status
job[_FROM_SNAPSHOT_KEY] = True
return job
def _load_snapshot_from_db(data_dir: Path) -> bool:
"""Seed jobs/datasets/settings from a real server's persisted SQLite db.
Read-only: opens the db (Database.__init__ runs the same idempotent
schema migrations a real server startup would) and copies rows into the
mock's in-memory state; nothing the mock does afterward is written back.
Returns False (leaving the caller to fall back to _seed()) if no db file
is found at that path, rather than silently creating an empty one.
"""
db_path = data_dir / "fastvideo_ui.db"
if not db_path.is_file():
logger.warning("--data-dir %s has no fastvideo_ui.db; falling back to fake demo data.", data_dir)
return False
db = Database(db_path)
settings = db.get_settings()
jobs = [_job_from_snapshot(row) for row in db.get_all_jobs()]
datasets = db.get_all_datasets()
with _state_lock:
_settings.update({k: v for k, v in settings.items() if v is not None})
_jobs.clear()
_jobs.update({j["id"]: j for j in jobs})
_datasets.clear()
_datasets.update({d["id"]: d for d in datasets})
_dataset_files.clear()
for d in datasets:
captions = db.get_dataset_captions(d["id"])
_dataset_files[d["id"]] = {"file_names": list(captions.keys()), "captions": captions}
logger.info(
"Loaded read-only snapshot from %s: %d job(s), %d dataset(s).",
db_path,
len(jobs),
len(datasets),
)
return True
# --- App --------------------------------------------------------------------
app = FastAPI(title="FastVideo Studio Mock API", version="0.1.0")
@@ -357,8 +534,6 @@ app.add_middleware(
allow_headers=["*"],
)
_seed()
@app.get("/api/__mock__")
def mock_sentinel() -> dict[str, bool]:
@@ -423,6 +598,32 @@ async def upload_image(file: Annotated[UploadFile, File()]) -> dict[str, str]:
return {"path": f"/mock/uploads/{uuid.uuid4().hex}_{os.path.basename(name)}"}
ALLOWED_REFERENCE_VIDEO_EXTENSIONS = {".mp4", ".mov", ".mkv", ".webm", ".avi"}
ALLOWED_REFERENCE_AUDIO_EXTENSIONS = {".wav", ".mp3", ".flac", ".m4a", ".ogg"}
ALLOWED_REFERENCE_IMAGE_EXTENSIONS = {".png", ".jpg", ".jpeg", ".webp", ".bmp"}
@app.post("/api/upload-media")
async def upload_media(file: Annotated[UploadFile, File()]) -> dict[str, str]:
"""Ref2VA reference upload (image/video/audio) -- the real server's
counterpart to upload_image, which only ever handles images."""
name = file.filename or "reference"
ext = os.path.splitext(name)[1].lower()
allowed = ALLOWED_REFERENCE_IMAGE_EXTENSIONS | ALLOWED_REFERENCE_VIDEO_EXTENSIONS | ALLOWED_REFERENCE_AUDIO_EXTENSIONS
if ext not in allowed:
raise HTTPException(status_code=400, detail=f"Invalid file type. Allowed: {', '.join(sorted(allowed))}")
if ext in ALLOWED_REFERENCE_VIDEO_EXTENSIONS:
media_type = "video"
elif ext in ALLOWED_REFERENCE_AUDIO_EXTENSIONS:
media_type = "audio"
else:
media_type = "image"
return {
"path": f"/mock/uploads/{uuid.uuid4().hex}_{os.path.basename(name)}",
"media_type": media_type,
}
@app.post("/api/upload-raw-dataset")
async def upload_raw_dataset(files: Annotated[list[UploadFile], File()]) -> dict[str, Any]:
video_exts = {".mp4", ".webm", ".avi", ".mov", ".mkv"}
@@ -446,6 +647,7 @@ def list_jobs(job_type: str | None = None) -> list[dict[str, Any]]:
# _public_job mutates the shared job dict via _advance_job, so it must run
# under the lock (matching every other job route) to avoid racing start/stop.
with _state_lock:
_drain_queue()
jobs = list(_jobs.values())
if job_type:
jobs = [j for j in jobs if j.get("job_type") == job_type]
@@ -456,6 +658,7 @@ def list_jobs(job_type: str | None = None) -> list[dict[str, Any]]:
@app.get("/api/jobs/{job_id}")
def get_job(job_id: str) -> dict[str, Any]:
with _state_lock:
_drain_queue()
job = _jobs.get(job_id)
if job is None:
raise HTTPException(status_code=404, detail="Job not found")
@@ -466,15 +669,20 @@ def get_job(job_id: str) -> dict[str, Any]:
def create_job(req: CreateJobRequest) -> dict[str, Any]:
job = _new_job_dict(req)
with _state_lock:
_drain_queue()
_jobs[job["id"]] = job
if _settings.get("autoStartJob"):
_start(job)
return _public_job(job)
def _start(job: dict[str, Any]) -> None:
def _start(job: dict[str, Any], at: float | None = None) -> None:
# Once the mock is actually driving this job, it's no longer a frozen
# historical record -- let _advance_job simulate it like any other.
job.pop(_FROM_SNAPSHOT_KEY, None)
job["status"] = "running"
job["started_at"] = time.time()
job["queued_at"] = None
job["started_at"] = at if at is not None else time.time()
job["finished_at"] = None
job["error"] = None
job["output_path"] = None
@@ -486,17 +694,281 @@ def _start(job: dict[str, Any]) -> None:
job["phase"] = "starting"
@app.post("/api/jobs/{job_id}/start")
def start_job(job_id: str) -> dict[str, Any]:
class _QueueView:
"""The scheduler's view of a mock job dict (it reads attributes, not keys)."""
def __init__(self, job: dict[str, Any]):
self.id = job["id"]
self.name = job.get("name", "")
self.status = job["status"]
self.references = job.get("references")
self.queued_at = job.get("queued_at")
def _drain_queue() -> None:
"""Run the queue up to now. Call with ``_state_lock`` held.
Like everything else in the mock this is computed on read: each pass
finishes jobs whose time is up, then starts whatever the queue allows.
A job takes over the slot at the moment the previous job *finished* (not at
the moment someone happened to poll), so a queue that has been sitting for
a minute shows every clip already done rather than advancing one per poll.
Same rules as the real runner: one at a time, oldest first, and a clip that
takes another's last frame waits for it -- see job_queue.plan_dispatch.
"""
while True:
for job in _jobs.values():
_advance_job(job)
plan = plan_dispatch({jid: _QueueView(j) for jid, j in _jobs.items()}, max_concurrent=1)
now = time.time()
for job_id, reason in plan.fail.items():
job = _jobs[job_id]
job["status"] = "failed"
job["error"] = reason
job["finished_at"] = now
job["queued_at"] = None
job["phase"] = "failed"
if not plan.start:
if plan.fail:
continue # a failure can free a slot or fail more jobs
return
for job_id in plan.start:
job = _jobs[job_id]
finished = [j["finished_at"] for j in _jobs.values() if j.get("finished_at")]
# The slot came free when the last job finished, if that was after the job was queued.
slot_free = max([job.get("queued_at") or 0.0, *[f for f in finished if f <= now]])
_start(job, at=min(now, slot_free))
def _check_dependencies_finished(job: dict[str, Any]) -> None:
for dep_id in dependencies(job.get("references")):
dep = _jobs.get(dep_id)
if dep is None:
raise HTTPException(status_code=409,
detail="This job starts from the ending of a job that no longer exists.")
if dep["status"] != "completed":
label = f"'{dep['name']}'" if dep.get("name") else dep["id"]
raise HTTPException(
status_code=409,
detail=(f"This job starts from the ending of {label}, which hasn't finished "
f"(it is {dep['status']}). Queue it to run after that job."),
)
# Mirrors JobRunner.EDITABLE_STATUSES: exactly the jobs that can still be started.
_EDITABLE_STATUSES = ("pending", "failed", "stopped")
@app.patch("/api/jobs/{job_id}")
def update_job(job_id: str, updates: dict[str, Any]) -> dict[str, Any]:
"""Edit a job's configuration; only jobs that have not produced a result."""
with _state_lock:
_drain_queue()
job = _jobs.get(job_id)
if job is None:
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
_advance_job(job)
if job["status"] == "running":
raise HTTPException(status_code=409, detail="Job is already running")
if job["status"] == "completed":
raise HTTPException(status_code=409, detail="Job already completed. Delete and re-create to run again.")
if job["status"] not in _EDITABLE_STATUSES:
raise HTTPException(
status_code=400,
detail=(f"Job is {job['status']}; only {', '.join(_EDITABLE_STATUSES)} jobs can be edited. "
"Duplicate it instead."),
)
unknown = set(updates) - set(CreateJobRequest.model_fields)
if unknown:
raise HTTPException(status_code=400, detail=f"Not editable: {', '.join(sorted(unknown))}")
job.update(updates)
return _public_job(job)
@app.post("/api/jobs/{job_id}/last-frame")
def extract_last_frame(job_id: str) -> dict[str, str]:
"""Stand-in for the real route: there is no video to read, so hand back a made-up path."""
with _state_lock:
_drain_queue()
job = _jobs.get(job_id)
if job is None:
raise HTTPException(status_code=404, detail="Job not found")
_advance_job(job)
if job["status"] != "completed" or not job.get("output_path"):
raise HTTPException(status_code=404, detail="No output available for this job")
return {"path": f"/mock/last_frames/last_frame_{job_id}.png", "media_type": "image"}
@app.post("/api/jobs/{job_id}/last-clip")
def extract_last_clip(job_id: str) -> dict[str, str]:
"""Stand-in for the real route: there is no video to read, so hand back a made-up path."""
with _state_lock:
_drain_queue()
job = _jobs.get(job_id)
if job is None:
raise HTTPException(status_code=404, detail="Job not found")
_advance_job(job)
if job["status"] != "completed" or not job.get("output_path"):
raise HTTPException(status_code=404, detail="No output available for this job")
return {"path": f"/mock/last_clips/last_clip_{job_id}.mp4", "media_type": "video"}
#: Key holding a trimmed job's frame count before any trim, so it can be restored.
_ORIGINAL_NUM_FRAMES_KEY = "_original_num_frames"
def _trimmable_job(job_id: str) -> dict[str, Any]:
"""Look up a completed job with an output, or raise. Call with _state_lock held."""
_drain_queue()
job = _jobs.get(job_id)
if job is None:
raise HTTPException(status_code=404, detail="Job not found")
_advance_job(job)
if job["status"] != "completed" or not job.get("output_path"):
raise HTTPException(status_code=404, detail="No output available for this job")
return job
def _check_grade_range(name: str, value: float, bounds: tuple[float, float]) -> None:
low, high = bounds
if not (low <= value <= high):
raise HTTPException(status_code=400, detail=f"{name} must be between {low:g} and {high:g}, got {value:g}.")
def _check_segments(segments: list[GradeSegment]) -> None:
"""Structural checks only (ordering, open-endedness, color bounds) -- the mock has no
frames to count, so it doesn't reproduce trim.py's frame-rounding/coverage checks.
An empty list isn't an error here either, matching trim.py: it defaults to
one neutral section covering everything (Pydantic's own default_factory
already supplies that when the field is omitted, but an explicit "segments":
[] reaches here as a real empty list).
"""
prev_end = 0.0
for i, s in enumerate(segments):
_check_grade_range("brightness", s.brightness, BRIGHTNESS_RANGE)
_check_grade_range("contrast", s.contrast, CONTRAST_RANGE)
_check_grade_range("saturation", s.saturation, SATURATION_RANGE)
if s.end_seconds is None:
if i != len(segments) - 1:
raise HTTPException(status_code=400, detail="Only the last section can be open-ended.")
else:
if s.end_seconds <= prev_end:
raise HTTPException(status_code=400, detail="Sections must be in increasing order.")
prev_end = s.end_seconds
@app.post("/api/jobs/{job_id}/trim")
def trim_job(job_id: str, req: TrimRequest) -> dict[str, Any]:
"""Stand-in for the real route: no video to cut or grade, so just resize num_frames to match."""
with _state_lock:
job = _trimmable_job(job_id)
if req.start_seconds < 0:
raise HTTPException(status_code=400, detail="The start of the range can't be negative.")
_check_segments(req.segments)
fps = job.get("fps") or 24
original_frames = job.setdefault(_ORIGINAL_NUM_FRAMES_KEY, job["num_frames"])
duration = original_frames / fps
end = duration if req.end_seconds is None else min(req.end_seconds, duration)
if end <= req.start_seconds:
raise HTTPException(status_code=400, detail="The end of the range must come after the start.")
job["num_frames"] = max(1, round((end - req.start_seconds) * fps))
# Recorded so the dialog can reopen showing what's actually applied,
# like the real server's edit_* job fields.
job["edit_start_seconds"] = req.start_seconds
job["edit_end_seconds"] = req.end_seconds
job["edit_segments"] = [s.model_dump() for s in req.segments]
return _public_job(job)
@app.post("/api/jobs/{job_id}/restore-video")
def restore_job_video(job_id: str) -> dict[str, Any]:
"""Stand-in for the real route: undo the simulated edit by restoring num_frames."""
with _state_lock:
job = _trimmable_job(job_id)
if _ORIGINAL_NUM_FRAMES_KEY not in job:
raise HTTPException(status_code=404, detail="This job has not been edited.")
job["num_frames"] = job.pop(_ORIGINAL_NUM_FRAMES_KEY)
job["edit_start_seconds"] = 0.0
job["edit_end_seconds"] = None
job["edit_segments"] = []
return _public_job(job)
def _check_startable(job: dict[str, Any]) -> None:
if job["status"] == "running":
raise HTTPException(status_code=409, detail="Job is already running")
if job["status"] == "queued":
raise HTTPException(status_code=409, detail="Job is already queued")
if job["status"] == "completed":
raise HTTPException(status_code=409, detail="Job already completed. Delete and re-create to run again.")
def _enqueue(job_ids: list[str]) -> list[dict[str, Any]]:
"""Queue jobs in order, all-or-nothing (mirrors JobRunner.enqueue_jobs). Lock held."""
if len(set(job_ids)) != len(job_ids):
raise HTTPException(status_code=409, detail="A job was listed more than once.")
jobs = []
for job_id in job_ids:
job = _jobs.get(job_id)
if job is None:
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
_check_startable(job)
jobs.append(job)
views = {jid: _QueueView(j) for jid, j in _jobs.items()}
for job in jobs:
loop = circular_dependency(views, job["id"])
if loop:
raise HTTPException(
status_code=409,
detail="Jobs would wait on each other's last frames in a loop: " + " -> ".join(loop),
)
for job in jobs:
job.pop(_FROM_SNAPSHOT_KEY, None)
job.update(status="queued", queued_at=time.time(), error=None, output_path=None, log_file_path=None,
started_at=None, finished_at=None, progress=0.0, progress_msg="", phase="queued")
# Strictly increasing, so jobs queued together keep their order.
others = [j["queued_at"] for j in _jobs.values() if j.get("queued_at") and j is not job]
job["queued_at"] = max([job["queued_at"], *[o + 1e-6 for o in others]])
_drain_queue()
return [_public_job(j) for j in jobs]
@app.post("/api/jobs/queue")
def queue_jobs(req: QueueJobsRequest) -> list[dict[str, Any]]:
with _state_lock:
_drain_queue()
return _enqueue(req.job_ids)
@app.post("/api/jobs/{job_id}/queue")
def queue_job(job_id: str) -> dict[str, Any]:
with _state_lock:
_drain_queue()
return _enqueue([job_id])[0]
@app.post("/api/jobs/{job_id}/dequeue")
def dequeue_job(job_id: str) -> dict[str, Any]:
with _state_lock:
_drain_queue()
job = _jobs.get(job_id)
if job is None:
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
if job["status"] != "queued":
raise HTTPException(status_code=409, detail=f"Job is not queued (status={job['status']})")
job["status"] = "pending"
job["queued_at"] = None
_drain_queue()
return _public_job(job)
@app.post("/api/jobs/{job_id}/start")
def start_job(job_id: str) -> dict[str, Any]:
with _state_lock:
_drain_queue()
job = _jobs.get(job_id)
if job is None:
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
_advance_job(job)
_check_startable(job)
_check_dependencies_finished(job)
_start(job)
return _public_job(job)
@@ -504,10 +976,16 @@ def start_job(job_id: str) -> dict[str, Any]:
@app.post("/api/jobs/{job_id}/stop")
def stop_job(job_id: str) -> dict[str, Any]:
with _state_lock:
_drain_queue()
job = _jobs.get(job_id)
if job is None:
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
_advance_job(job)
if job["status"] == "queued":
job["status"] = "pending"
job["queued_at"] = None
_drain_queue()
return _public_job(job)
if job["status"] != "running":
raise HTTPException(status_code=409, detail=f"Job is not running (status={job['status']})")
job["status"] = "stopped"
@@ -519,14 +997,17 @@ def stop_job(job_id: str) -> dict[str, Any]:
@app.delete("/api/jobs/{job_id}")
def delete_job(job_id: str) -> dict[str, str]:
with _state_lock:
_drain_queue()
if _jobs.pop(job_id, None) is None:
raise HTTPException(status_code=404, detail="Job not found")
_drain_queue() # clips waiting on this one's last frame can no longer run
return {"detail": f"Job {job_id} deleted"}
@app.get("/api/jobs/{job_id}/logs")
def get_job_logs(job_id: str, after: int = 0) -> dict[str, Any]:
with _state_lock:
_drain_queue()
job = _jobs.get(job_id)
if job is None:
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
@@ -545,6 +1026,7 @@ def get_job_logs(job_id: str, after: int = 0) -> dict[str, Any]:
@app.get("/api/jobs/{job_id}/video")
def get_video(job_id: str) -> FileResponse:
with _state_lock:
_drain_queue()
job = _jobs.get(job_id)
if job is None:
raise HTTPException(status_code=404, detail="Job not found")
@@ -552,6 +1034,12 @@ def get_video(job_id: str) -> FileResponse:
output_path = job.get("output_path")
if job["status"] != "completed" or not output_path:
raise HTTPException(status_code=404, detail="No output available for this job")
# A job loaded from a --data-dir snapshot may point at a real file this
# process never wrote, only reads -- serve it if it's actually there,
# otherwise fall back to the same synthetic clip every other job gets.
if os.path.isfile(output_path):
media_type = "video/mp4" if output_path.endswith(".mp4") else "image/png"
return FileResponse(output_path, media_type=media_type, filename=os.path.basename(output_path))
# Mirror the real server: image workloads (t2i) output a .png served as an
# image; everything else is a video.
if output_path.endswith(".png"):
@@ -559,9 +1047,59 @@ def get_video(job_id: str) -> FileResponse:
return FileResponse(_build_mock_media("mp4"), media_type="video/mp4", filename=f"job_{job_id}.mp4")
_MERGED_DIR = tempfile.mkdtemp(prefix="fvstudio_merged_")
@app.post("/api/scenes/merge")
def merge_scene(req: MergeSceneRequest) -> dict[str, Any]:
"""Real merge (same code as the real server) of whatever videos the jobs have.
Jobs whose video only exists in the mock get the stand-in clip, so the
flow can be tried end to end without generating anything.
"""
with _state_lock:
_drain_queue()
picked = []
for job_id in req.job_ids:
job = _jobs.get(job_id)
if job is None:
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
picked.append(dict(job))
all_real = all(j.get("output_path") and os.path.isfile(j["output_path"]) for j in picked)
stand_in = None if all_real else _build_mock_media("mp4")
jobs = [
SimpleNamespace(
id=j["id"],
name=j.get("name", ""),
status=j["status"],
output_path=(j["output_path"] if all_real else (stand_in if j["status"] == "completed" else None)),
) for j in picked
]
try:
result = merge_jobs(jobs, _MERGED_DIR, req.name)
except MergeError as e:
raise HTTPException(status_code=e.status_code, detail=e.detail) from e
return {
"filename": result.filename,
"url": f"/api/merged/{result.filename}",
"clips": result.clips,
"seconds": result.seconds,
}
@app.get("/api/merged/{filename}")
def get_merged_video(filename: str) -> FileResponse:
try:
path = merged_path(_MERGED_DIR, filename)
except MergeError as e:
raise HTTPException(status_code=e.status_code, detail=e.detail) from e
return FileResponse(path, media_type="video/mp4", filename=filename)
@app.get("/api/jobs/{job_id}/download_log")
def download_log(job_id: str) -> PlainTextResponse:
with _state_lock:
_drain_queue()
job = _jobs.get(job_id)
if job is None:
raise HTTPException(status_code=404, detail="Job not found")
@@ -651,8 +1189,18 @@ def main() -> None:
parser.add_argument("--host", default="127.0.0.1", help="Bind address (default: 127.0.0.1)")
# Default off the real server's 8189 so the mock never shadows a real API.
parser.add_argument("--port", type=int, default=8190, help="Port number (default: 8190)")
parser.add_argument(
"--data-dir",
default=None,
help=("Read-only: seed jobs/datasets/settings from a real server's "
"fastvideo_ui.db under this directory (its --data-dir) instead "
"of fake demo data. Nothing the mock does is written back."),
)
args = parser.parse_args()
if args.data_dir is None or not _load_snapshot_from_db(Path(args.data_dir).resolve()):
_seed()
uvicorn.run(app, host=args.host, port=args.port, log_level="info")
+7
View File
@@ -6,6 +6,9 @@ from fastvideo_studio.models.create_job_request import CreateJobRequest
from fastvideo_studio.models.settings_update import SettingsUpdate
from fastvideo_studio.models.create_dataset_request import CreateDatasetRequest
from fastvideo_studio.models.update_caption_request import UpdateCaptionRequest
from fastvideo_studio.models.queue_jobs_request import QueueJobsRequest
from fastvideo_studio.models.merge_scene_request import MergeSceneRequest
from fastvideo_studio.models.trim_request import GradeSegment, TrimRequest
def model_label(model_path: str) -> str:
@@ -18,5 +21,9 @@ __all__ = [
"SettingsUpdate",
"CreateDatasetRequest",
"UpdateCaptionRequest",
"QueueJobsRequest",
"MergeSceneRequest",
"GradeSegment",
"TrimRequest",
"model_label",
]
@@ -0,0 +1,11 @@
# SPDX-License-Identifier: Apache-2.0
"""Request model for merging a scene's clips into one video."""
from pydantic import BaseModel
class MergeSceneRequest(BaseModel):
#: Completed jobs, in the order their videos should play.
job_ids: list[str]
#: Used to name the merged file.
name: str = ""
@@ -0,0 +1,8 @@
# SPDX-License-Identifier: Apache-2.0
"""Request model for queueing jobs."""
from pydantic import BaseModel
class QueueJobsRequest(BaseModel):
job_ids: list[str]
@@ -0,0 +1,30 @@
# SPDX-License-Identifier: Apache-2.0
"""Request model for cutting a job's video and color-grading it section by section."""
from pydantic import BaseModel, Field
class GradeSegment(BaseModel):
"""One razor-cut section of the kept range, and its own color grade.
``end_seconds`` is seconds into the *kept* (already start/end-trimmed)
range -- the same timeline the preview video plays -- not the original
file's. Only the last section in a list may leave it ``None``, meaning
"to the end"; sections must be given in increasing order and cover the
whole kept range, with no gaps or reordering.
"""
end_seconds: float | None = None
#: Added directly to 0-255 pixel values. 0 = unchanged.
brightness: float = 0.0
#: 1.0 = unchanged, 0 = flat gray, >1 = more contrast.
contrast: float = 1.0
#: 1.0 = unchanged, 0 = grayscale, >1 = more saturated.
saturation: float = 1.0
class TrimRequest(BaseModel):
start_seconds: float = 0.0
#: None keeps to the end of the video.
end_seconds: float | None = None
#: Defaults to one neutral section covering the whole kept range, i.e. a plain trim with no grading.
segments: list[GradeSegment] = Field(default_factory=lambda: [GradeSegment()])
+156 -2
View File
@@ -33,10 +33,12 @@ from fastapi.responses import FileResponse
from fastvideo.registry import (get_registered_model_paths, get_registered_models_with_workloads)
from fastvideo_studio.database import Database, _get_db_path
from fastvideo_studio.frames import FrameError, last_clip_for_job, last_frame_for_job
from fastvideo_studio.gpu import get_gpu_snapshot
from fastvideo_studio.job_runner import JobRunner, JobStatus
from fastvideo_studio.models import (CreateDatasetRequest, CreateJobRequest, SettingsUpdate, UpdateCaptionRequest,
model_label)
from fastvideo_studio.merge import MergeError, merge_jobs, merged_path
from fastvideo_studio.models import (CreateDatasetRequest, CreateJobRequest, MergeSceneRequest, QueueJobsRequest,
SettingsUpdate, TrimRequest, UpdateCaptionRequest, model_label)
logging.basicConfig(
level=logging.INFO,
@@ -439,6 +441,41 @@ def start_job(job_id: str) -> dict[str, Any]:
) from e
def _queue_error(e: ValueError) -> HTTPException:
return HTTPException(status_code=404 if "not found" in str(e) else 409, detail=str(e))
@app.post("/api/jobs/queue")
def queue_jobs(req: QueueJobsRequest) -> list[dict[str, Any]]:
"""Queue jobs, in the order given, to run as GPU slots free up.
A job that starts from another job's last frame waits for that job to
complete. All-or-nothing: if any job can't be queued, none are.
"""
try:
return [j.to_dict() for j in job_runner.enqueue_jobs(req.job_ids)]
except ValueError as e:
raise _queue_error(e) from e
@app.post("/api/jobs/{job_id}/queue")
def queue_job(job_id: str) -> dict[str, Any]:
"""Queue one job to run when a slot is free."""
try:
return job_runner.enqueue_job(job_id).to_dict()
except ValueError as e:
raise _queue_error(e) from e
@app.post("/api/jobs/{job_id}/dequeue")
def dequeue_job(job_id: str) -> dict[str, Any]:
"""Take a queued job out of the queue (back to pending)."""
try:
return job_runner.dequeue_job(job_id).to_dict()
except ValueError as e:
raise _queue_error(e) from e
@app.post("/api/jobs/{job_id}/stop")
def stop_job(job_id: str) -> dict[str, Any]:
"""Request a running job to stop.
@@ -665,6 +702,113 @@ def get_video(job_id: str) -> FileResponse:
return FileResponse(job.output_path, media_type=media_type)
@app.post("/api/jobs/{job_id}/last-frame")
def extract_last_frame(job_id: str) -> dict[str, str]:
"""Save the last frame of a completed job's video and return its path.
The path is meant to be attached to another job as an image reference, so
the next clip of a scene can start from where this one ended.
"""
job = job_runner.get_job(job_id)
if job is None:
raise HTTPException(status_code=404, detail="Job not found")
try:
path = last_frame_for_job(job, upload_dir)
except FrameError as e:
if e.status_code >= 500:
logger.warning("Last-frame extraction failed for job %s: %s", job_id, e.detail)
raise HTTPException(status_code=e.status_code, detail=e.detail) from e
return {"path": path, "media_type": "image"}
@app.post("/api/jobs/{job_id}/last-clip")
def extract_last_clip(job_id: str) -> dict[str, str]:
"""Save the trailing second of a completed job's video and return its path.
The path is meant to be attached to another job as a video reference, so
the next clip continues with real motion instead of guessing it from a
single still frame.
"""
job = job_runner.get_job(job_id)
if job is None:
raise HTTPException(status_code=404, detail="Job not found")
try:
path = last_clip_for_job(job, upload_dir)
except FrameError as e:
if e.status_code >= 500:
logger.warning("Last-clip extraction failed for job %s: %s", job_id, e.detail)
raise HTTPException(status_code=e.status_code, detail=e.detail) from e
return {"path": path, "media_type": "video"}
def _trim_error(e: Exception) -> HTTPException:
if isinstance(e, FrameError):
return HTTPException(status_code=e.status_code, detail=e.detail)
return HTTPException(status_code=404 if "not found" in str(e) else 400, detail=str(e))
@app.post("/api/jobs/{job_id}/trim")
def trim_job(job_id: str, req: TrimRequest) -> dict[str, Any]:
"""Cut a completed job's video to [start_seconds, end_seconds) and grade it section by section, in place.
The untouched original is kept, so this can be called again with a
different range and/or sections, or undone with POST .../restore-video.
"""
try:
segments = [s.model_dump() for s in req.segments]
job = job_runner.trim_job(job_id, req.start_seconds, req.end_seconds, segments)
except (ValueError, FrameError) as e:
raise _trim_error(e) from e
return job.to_dict()
@app.post("/api/jobs/{job_id}/restore-video")
def restore_job_video(job_id: str) -> dict[str, Any]:
"""Undo every trim on a job's video, back to what it originally generated."""
try:
job = job_runner.restore_job_video(job_id)
except (ValueError, FrameError) as e:
raise _trim_error(e) from e
return job.to_dict()
@app.post("/api/scenes/merge")
def merge_scene(req: MergeSceneRequest) -> dict[str, Any]:
"""Join the finished clips of a scene, in the order given, into one video.
The clips are copied, not re-encoded, so this takes about a second.
"""
jobs = []
for job_id in req.job_ids:
job = job_runner.get_job(job_id)
if job is None:
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
jobs.append(job)
try:
result = merge_jobs(jobs, job_runner.output_dir, req.name)
except MergeError as e:
if e.status_code >= 500:
logger.warning("Scene merge failed: %s", e.detail)
raise HTTPException(status_code=e.status_code, detail=e.detail) from e
logger.info("Merged %d clips into %s", result.clips, result.path)
return {
"filename": result.filename,
"url": f"/api/merged/{result.filename}",
"clips": result.clips,
"seconds": result.seconds,
}
@app.get("/api/merged/{filename}")
def get_merged_video(filename: str) -> FileResponse:
"""Stream a merged scene video."""
try:
path = merged_path(job_runner.output_dir, filename)
except MergeError as e:
raise HTTPException(status_code=e.status_code, detail=e.detail) from e
return FileResponse(path, media_type="video/mp4", filename=filename)
@app.get("/api/jobs/{job_id}/download_log")
def get_job_log_file(job_id: str) -> FileResponse:
"""Download the log file for a job."""
@@ -750,6 +894,14 @@ def main() -> None:
help=("Directory for SQLite database (jobs + settings persistence) "
f"(default: {default_data_dir})"),
)
parser.add_argument(
"--max-concurrent-jobs",
type=int,
default=1,
help=("How many inference jobs may run at once (default: 1). Each gets its own "
"long-lived worker that keeps its model loaded between jobs, so raising "
"this loads one copy of the model per worker."),
)
parser.add_argument(
"--verbose",
action="store_true",
@@ -775,6 +927,8 @@ def main() -> None:
log_dir=log_dir,
verbose=args.verbose,
database=database,
max_concurrent_jobs=args.max_concurrent_jobs,
upload_dir=upload_dir,
)
logger.info("Output directory: %s", output_dir)
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,276 @@
'use client';
import { AlertTriangle, Loader2, RefreshCw } from 'lucide-react';
import * as React from 'react';
import CreateJobModal from '@/components/jobs/CreateJobModal';
import { EDITABLE_STATUSES, SceneBoard } from '@/components/scenes/SceneBoard';
import { MergedSceneDialog } from '@/components/scenes/MergedSceneDialog';
import { SceneQueueDialog } from '@/components/scenes/SceneQueueDialog';
import { TrimClipDialog } from '@/components/scenes/TrimClipDialog';
import { Button } from '@/components/ui/button';
import { Card } from '@/components/ui/card';
import { NativeSelect } from '@/components/ui/native-select';
import { toast } from 'sonner';
import { dequeueJob, getJobsList, mergeScene, queueJobs, updateJob, type MergedScene } from '@/lib/api';
import { attachedLastFrames, attachLastFrame, deferredLastFrameSource, detachLastFrame } from '@/lib/continuation';
import type { SceneQueuePlan } from '@/lib/sceneQueue';
import { buildScenes, type Scene, type SceneClip } from '@/lib/scenes';
import type { Job } from '@/lib/types';
const POLL_MS = 3000;
export default function ScenesPage() {
const [jobs, setJobs] = React.useState<Job[]>([]);
const [isLoading, setIsLoading] = React.useState(true);
const [error, setError] = React.useState<string | null>(null);
const [reloadKey, setReloadKey] = React.useState(0);
const [selectedKey, setSelectedKey] = React.useState<string | null>(null);
const [openJob, setOpenJob] = React.useState<Job | null>(null);
// which re-run of a clip number the user picked, by clip
const [choices, setChoices] = React.useState<Record<string, string>>({});
const [queueTarget, setQueueTarget] = React.useState<Scene | null>(null);
const [merged, setMerged] = React.useState<(MergedScene & { title: string }) | null>(null);
const [trimTarget, setTrimTarget] = React.useState<Job | null>(null);
// Bumped per job id whenever its video is edited in place, to cache-bust every
// <video> for that job (the dialog's own preview and the scene board's). Owned
// here, not by the dialog, so it survives the dialog re-rendering with a fresh
// job object after each edit -- a per-dialog counter got reset by that same
// re-render before the browser ever saw the new URL.
const [videoVersions, setVideoVersions] = React.useState<Record<string, number>>({});
const bumpVideoVersion = (jobId: string) =>
setVideoVersions((v) => ({ ...v, [jobId]: (v[jobId] ?? 0) + 1 }));
React.useEffect(() => {
let cancelled = false;
async function load() {
try {
const list = await getJobsList('inference');
if (cancelled) return;
setJobs(list);
setError(null);
} catch (e) {
if (!cancelled) setError(e instanceof Error ? e.message : 'Failed to load jobs');
} finally {
if (!cancelled) setIsLoading(false);
}
}
void load();
return () => {
cancelled = true;
};
}, [reloadKey]);
function reload() {
setIsLoading(true);
setReloadKey((k) => k + 1);
}
// While anything is queued or running, keep the board current without the
// loading state (which would blank or flicker it).
const hasActive = jobs.some((j) => j.status === 'queued' || j.status === 'running');
React.useEffect(() => {
if (!hasActive) return;
const interval = setInterval(() => {
getJobsList('inference')
.then(setJobs)
.catch(() => {});
}, POLL_MS);
return () => clearInterval(interval);
}, [hasActive]);
// Attach the previous clip's last frame to `clip` and rewrite its prompt to open on
// it. The frame is a reference the server resolves when the clip runs, so the
// previous clip doesn't have to be finished yet. Returns the picture label.
async function attachPreviousFrame(clip: SceneClip, previous: SceneClip): Promise<string> {
const result = attachLastFrame(
clip.job.prompt,
clip.job.references ?? [],
deferredLastFrameSource(previous.job.id),
);
if (!result.ok) throw new Error(result.error);
await updateJob(clip.job.id, { references: result.references, prompt: result.prompt });
return result.label;
}
async function useLastFrame(clip: SceneClip, previous: SceneClip) {
try {
const label = await attachPreviousFrame(clip, previous);
toast.success(`Clip ${clip.position} now starts from the last frame of clip ${previous.position} (${label}).`);
reload();
} catch (e) {
toast.error(e instanceof Error ? e.message : "Couldn't use the last frame.");
}
}
// Undo the above: take every attached last frame back off the clip.
async function stopUsingLastFrame(clip: SceneClip) {
try {
let prompt = clip.job.prompt;
let references = clip.job.references ?? [];
// Newest first: only the most recently added image can be removed cleanly.
for (const { source } of attachedLastFrames(references).reverse()) {
const result = detachLastFrame(prompt, references, source);
if (!result.ok) throw new Error(result.error);
prompt = result.prompt;
references = result.references;
}
await updateJob(clip.job.id, { references, prompt });
toast.success(`Clip ${clip.position} no longer starts from a previous clip's last frame.`);
reload();
} catch (e) {
toast.error(e instanceof Error ? e.message : "Couldn't remove the last frame.");
}
}
// Join the scene's clips, in the order shown (the take chosen for each), into one video.
async function mergeSceneClips(scene: Scene) {
try {
const result = await mergeScene(scene.clips.map((c) => c.job.id), scene.title);
setMerged({ ...result, title: scene.title });
} catch (e) {
toast.error(e instanceof Error ? e.message : "Couldn't merge the scene.");
}
}
async function queueClip(clip: SceneClip) {
try {
await queueJobs([clip.job.id]);
toast.success(`Queued clip ${clip.position}.`);
} catch (e) {
toast.error(e instanceof Error ? e.message : "Couldn't queue the clip.");
}
reload();
}
async function dequeueClip(clip: SceneClip) {
try {
await dequeueJob(clip.job.id);
} catch (e) {
toast.error(e instanceof Error ? e.message : "Couldn't remove the clip from the queue.");
}
reload();
}
// Link clips to the frames before them (if asked), then queue them all in scene order.
async function queueScene(plan: SceneQueuePlan) {
try {
for (const { clip, previous } of plan.chain) {
await attachPreviousFrame(clip, previous).catch((e) => {
throw new Error(`Clip ${clip.position}: ${e instanceof Error ? e.message : e}. Nothing was queued.`);
});
}
await queueJobs(plan.queue.map((c) => c.job.id));
const n = plan.queue.length;
toast.success(`Queued ${n} clip${n === 1 ? '' : 's'}.`);
setQueueTarget(null);
} catch (e) {
toast.error(e instanceof Error ? e.message : "Couldn't queue the scene.");
}
reload();
}
const scenes = React.useMemo(() => buildScenes(jobs, choices), [jobs, choices]);
const scene = scenes.find((s) => s.key === selectedKey) ?? scenes[0];
return (
<div className="mx-auto w-full max-w-[1200px] px-4 pb-12 pt-4">
<Card className="p-6">
<div className="mb-1 flex flex-wrap items-center gap-3">
<h2 className="text-2xl font-semibold text-foreground">Scenes</h2>
<Button
type="button"
variant="outline"
size="sm"
className="ml-auto gap-1.5"
onClick={reload}
disabled={isLoading}
>
<RefreshCw className={isLoading ? 'size-3.5 animate-spin' : 'size-3.5'} aria-hidden />
Refresh
</Button>
</div>
<p className="mb-6 text-sm text-muted-foreground">
Every shot across your jobs, in order. Jobs named with a shared stem and a number, like{' '}
<code className="font-mono text-xs">wolf-lunch-clip-01</code>,{' '}
<code className="font-mono text-xs">wolf-lunch-clip-02</code>, are grouped into one scene.
</p>
{isLoading && jobs.length === 0 ? (
<div className="flex items-center gap-3 p-8 text-muted-foreground">
<Loader2 className="h-6 w-6 animate-spin text-primary" />
<span>Loading scenes…</span>
</div>
) : error && jobs.length === 0 ? (
<div role="alert" className="flex flex-col items-center gap-3 py-8 text-center">
<AlertTriangle className="size-6 text-destructive" aria-hidden />
<p className="max-w-md text-sm text-muted-foreground">{error}</p>
<Button type="button" variant="outline" onClick={reload}>
Try Again
</Button>
</div>
) : !scene ? (
<p className="py-8 text-center text-muted-foreground">
No inference jobs yet. Create some and they&apos;ll show up here.
</p>
) : (
<div className="flex flex-col gap-6">
{scenes.length > 1 && (
<label className="flex max-w-md flex-col gap-1 text-xs text-muted-foreground">
Scene
<NativeSelect value={scene.key} onChange={(e) => setSelectedKey(e.target.value)}>
{scenes.map((s) => (
<option key={s.key} value={s.key}>
{s.title} ({s.clips.length} clip{s.clips.length === 1 ? '' : 's'})
</option>
))}
</NativeSelect>
</label>
)}
<SceneBoard
scene={scene}
onOpenJob={setOpenJob}
onUseLastFrame={useLastFrame}
onDetachLastFrame={stopUsingLastFrame}
onQueueScene={setQueueTarget}
onMergeScene={mergeSceneClips}
onQueueClip={queueClip}
onDequeueClip={dequeueClip}
onTrimClip={(clip) => setTrimTarget(clip.job)}
onChooseTake={(takeKey, jobId) => setChoices((c) => ({ ...c, [takeKey]: jobId }))}
videoVersionFor={(jobId) => videoVersions[jobId] ?? 0}
/>
</div>
)}
</Card>
<MergedSceneDialog merged={merged} onClose={() => setMerged(null)} />
<SceneQueueDialog scene={queueTarget} onConfirm={queueScene} onClose={() => setQueueTarget(null)} />
<TrimClipDialog
clip={trimTarget}
videoVersion={trimTarget ? (videoVersions[trimTarget.id] ?? 0) : 0}
onChanged={(job) => {
setTrimTarget(job);
bumpVideoVersion(job.id);
getJobsList('inference').then(setJobs).catch(() => {});
}}
onClose={() => setTrimTarget(null)}
/>
{openJob && (
<CreateJobModal
isOpen
readOnly={!EDITABLE_STATUSES.includes(openJob.status)}
editingJob={openJob}
jobType={(openJob.job_type ?? 'inference') as never}
workloadType={openJob.workload_type ?? 't2v'}
onClose={() => setOpenJob(null)}
onSuccess={() => {
setOpenJob(null);
reload();
}}
/>
)}
</div>
);
}
@@ -1,19 +1,29 @@
import { render, screen } from '@testing-library/react';
import userEvent from '@testing-library/user-event';
import { toast } from 'sonner';
import { describe, expect, it, vi } from 'vitest';
import CreateJobButton from './CreateJobButton';
vi.mock('sonner', () => ({ toast: { error: vi.fn(), warning: vi.fn() } }));
vi.mock('./CreateJobModal', () => ({
default: ({
isOpen,
workloadType,
prefillJob,
}: {
isOpen: boolean;
workloadType: string;
prefillJob?: { name?: string; prompt: string } | null;
}) =>
isOpen ? (
<div role="dialog" data-workload-type={workloadType}>
<div
role="dialog"
data-workload-type={workloadType}
data-prefill-name={prefillJob?.name ?? ''}
data-prefill-prompt={prefillJob?.prompt ?? ''}
>
Create job form
</div>
) : null,
@@ -27,7 +37,11 @@ describe('CreateJobButton', () => {
await user.click(screen.getByRole('button', { name: 'Create Job' }));
await user.click(screen.getByRole('menuitem', { name: /I2V/i }));
expect(screen.getByRole('dialog')).toHaveAttribute(
// The dialog opens a frame after the menu selection (see CreateJobButton's
// openModal: deferred via requestAnimationFrame to dodge a Radix
// DropdownMenu + Dialog pointer-events race), so wait for it rather than
// asserting synchronously.
expect(await screen.findByRole('dialog')).toHaveAttribute(
'data-workload-type',
'i2v',
);
@@ -45,9 +59,75 @@ describe('CreateJobButton', () => {
expect(firstItem).toHaveFocus();
await user.keyboard('{Enter}');
expect(screen.getByRole('dialog')).toHaveAttribute(
expect(await screen.findByRole('dialog')).toHaveAttribute(
'data-workload-type',
't2v',
);
});
describe('importing a job file', () => {
const file = (contents: string, name = 'job.json') =>
new File([contents], name, { type: 'application/json' });
it('opens the form for the file\'s workload with the job pre-filled', async () => {
const user = userEvent.setup();
render(<CreateJobButton jobType="inference" />);
await user.click(screen.getByRole('button', { name: 'Create Job' }));
await user.click(screen.getByRole('menuitem', { name: /Import job JSON/ }));
await user.upload(
screen.getByLabelText('Import job JSON file'),
file(
JSON.stringify({
model_id: 'MiniMaxAI/MiniMax-H3',
name: 'from-file',
workload_type: 'i2v',
prompt: ['summary:', 'Hi.'],
}),
),
);
const dialog = await screen.findByRole('dialog');
expect(dialog).toHaveAttribute('data-workload-type', 'i2v');
expect(dialog).toHaveAttribute('data-prefill-name', 'from-file');
expect(dialog).toHaveAttribute('data-prefill-prompt', 'summary:\nHi.');
});
it('reports an unreadable file instead of opening the form', async () => {
const user = userEvent.setup();
render(<CreateJobButton jobType="inference" />);
await user.click(screen.getByRole('button', { name: 'Create Job' }));
await user.click(screen.getByRole('menuitem', { name: /Import job JSON/ }));
await user.upload(screen.getByLabelText('Import job JSON file'), file('not json', 'bad.json'));
await vi.waitFor(() => expect(toast.error).toHaveBeenCalled());
expect(vi.mocked(toast.error).mock.calls.at(-1)?.[0]).toContain("bad.json");
expect(screen.queryByRole('dialog')).not.toBeInTheDocument();
});
it('a normal open after an import starts clean', async () => {
const user = userEvent.setup();
render(<CreateJobButton jobType="inference" />);
await user.click(screen.getByRole('button', { name: 'Create Job' }));
await user.click(screen.getByRole('menuitem', { name: /Import job JSON/ }));
await user.upload(
screen.getByLabelText('Import job JSON file'),
file(JSON.stringify({ model_id: 'm', name: 'from-file', prompt: 'p' })),
);
expect(await screen.findByRole('dialog')).toHaveAttribute('data-prefill-name', 'from-file');
await user.click(screen.getByRole('button', { name: 'Create Job' }));
await user.click(await screen.findByRole('menuitem', { name: /T2V/i }));
await vi.waitFor(() => expect(screen.getByRole('dialog')).toHaveAttribute('data-prefill-name', ''));
});
it('is not offered on the training pages', async () => {
const user = userEvent.setup();
render(<CreateJobButton jobType="finetuning" />);
await user.click(screen.getByRole('button', { name: 'Create Job' }));
expect(screen.queryByRole('menuitem', { name: /Import job JSON/ })).not.toBeInTheDocument();
});
});
});
@@ -6,7 +6,10 @@ import * as DropdownMenu from '@radix-ui/react-dropdown-menu';
import CreateJobModal from '@/components/jobs/CreateJobModal';
import { Button } from '@/components/ui/button';
import { toast } from 'sonner';
import { WORKLOAD_OPTIONS } from '@/lib/jobConfig';
import { parseJobJson } from '@/lib/jobJson';
import type { JobLike } from '@/lib/jobToFields';
import type { JobType } from '@/lib/types';
import { triggerRefresh } from '@/stores/jobsRefresh';
@@ -22,9 +25,32 @@ export default function CreateJobButton({ jobType }: CreateJobButtonProps) {
options[0]?.type ?? 't2v',
);
function openModal(type: string) {
// An imported job file pre-fills the form; a normal open starts clean.
const [prefillJob, setPrefillJob] = React.useState<JobLike | null>(null);
const fileInputRef = React.useRef<HTMLInputElement>(null);
async function handleImportFile(e: React.ChangeEvent<HTMLInputElement>) {
const file = e.target.files?.[0];
e.target.value = ''; // allow re-picking the same file
if (!file) return;
const result = parseJobJson(await file.text());
if (!result.ok) {
toast.error(`Couldn't import ${file.name}: ${result.error}`);
return;
}
openModal(result.workloadType, result.job);
}
function openModal(type: string, prefill: JobLike | null = null) {
setWorkloadType(type);
setModalOpen(true);
setPrefillJob(prefill);
// Opening the Dialog synchronously inside the DropdownMenu.Item's
// onSelect races the two Radix layers' pointer-events body lock/unlock:
// the dropdown's own close cleanup can lose that race, leaving <body>
// stuck with `pointer-events: none` after this dialog later closes --
// silently blocking every click, including this same button. Deferring
// to the next frame lets the dropdown finish closing first.
requestAnimationFrame(() => setModalOpen(true));
}
function handleSuccess() {
@@ -60,15 +86,39 @@ export default function CreateJobButton({ jobType }: CreateJobButtonProps) {
</span>
</DropdownMenu.Item>
))}
{jobType === 'inference' && (
<>
<DropdownMenu.Separator className="my-1 h-px bg-border" />
<DropdownMenu.Item
onSelect={() => fileInputRef.current?.click()}
className="flex min-h-11 cursor-pointer select-none flex-col justify-center px-4 py-2 text-left text-sm font-medium outline-none data-[highlighted]:bg-secondary"
>
Import job JSON…
<span className="mt-0.5 block text-xs font-normal text-muted-foreground">
Fill the form from a job file
</span>
</DropdownMenu.Item>
</>
)}
</DropdownMenu.Content>
</DropdownMenu.Portal>
</DropdownMenu.Root>
<input
ref={fileInputRef}
type="file"
accept=".json,application/json"
aria-label="Import job JSON file"
tabIndex={-1}
className="sr-only"
onChange={handleImportFile}
/>
<CreateJobModal
isOpen={modalOpen}
onClose={() => setModalOpen(false)}
onSuccess={handleSuccess}
jobType={jobType}
workloadType={workloadType}
prefillJob={prefillJob}
/>
</>
);
@@ -4,10 +4,13 @@ import userEvent from '@testing-library/user-event';
import { beforeEach, describe, expect, it, vi } from 'vitest';
import CreateJobModal from './CreateJobModal';
import { toast } from 'sonner';
import { createJob, getDatasets, getModels, uploadImage } from '@/lib/api';
import { defaultOptionsStore } from '@/stores/defaultOptions';
import { DEFAULT_OPTIONS } from '@/lib/defaultOptions';
vi.mock('sonner', () => ({ toast: { warning: vi.fn(), error: vi.fn(), success: vi.fn() } }));
vi.mock('@/lib/api', () => ({
createJob: vi.fn(),
getModels: vi.fn(),
@@ -206,4 +209,179 @@ describe('CreateJobModal', () => {
await waitFor(() => expect(onSuccess).toHaveBeenCalledTimes(1));
});
it('shows the H3 speech-tag console only for an H3 model and inserts at the prompt cursor', async () => {
vi.mocked(getModels).mockResolvedValue([
...MODELS,
{ id: 'minimax-h3/i2v', label: 'MiniMax H3', type: 'i2v' },
]);
const user = userEvent.setup();
renderModal({ workloadType: 'i2v' });
await screen.findByRole('option', { name: 'Wan T2V (wan/t2v-1.3b)' });
expect(screen.queryByText(/Speech tags/)).not.toBeInTheDocument();
await user.selectOptions(
screen.getByLabelText('Model'),
'minimax-h3/i2v',
);
expect(await screen.findByText(/Speech tags/)).toBeInTheDocument();
const promptField = screen.getByLabelText('Prompt') as HTMLTextAreaElement;
await user.type(promptField, 'Hello there.');
promptField.setSelectionRange(5, 5);
await user.click(screen.getByRole('button', { name: '<pause>' }));
expect(promptField.value).toBe('Hello <pause> there.');
// Focus and cursor return to the prompt field so tags can be chained.
await waitFor(() => expect(promptField).toHaveFocus());
});
});
const H3 = { id: 'minimax-h3/i2v', label: 'MiniMax H3', type: 'i2v' };
const BASE_PROMPT = [
'For the target video, <Picture 1> is fully referenced.',
'',
'integrated_multimodal_description: [Shot 1] A close-up. He (S1) says: <d>[English] Sorry.</d> <cutoff>',
'',
'overall_soundscape: Room tone.',
'',
'non_diegetic_music: N/A',
].join('\n');
describe('CreateJobModal with a prefilled job', () => {
beforeEach(() => {
vi.mocked(getModels).mockResolvedValue([...MODELS, H3]);
vi.mocked(toast.warning).mockClear();
});
it('fills the form from the job and creates a new job on submit', async () => {
const user = userEvent.setup();
const { onSuccess } = renderModal({
workloadType: 'i2v',
prefillJob: {
id: 'imported-1',
model_id: H3.id,
name: 'from-file',
prompt: BASE_PROMPT,
workload_type: 'i2v',
references: [{ source: '/data/inputs/face.png', media_type: 'image' }],
num_frames: 243,
seed: 7,
},
});
await screen.findByRole('option', { name: /MiniMax H3/ });
await waitFor(() => expect(screen.getByLabelText('Model')).toHaveValue(H3.id));
expect(screen.getByLabelText('Name (optional)')).toHaveValue('from-file');
expect(screen.getByLabelText('Prompt')).toHaveValue(BASE_PROMPT);
expect(screen.getByText('face.png')).toBeInTheDocument();
expect(screen.getByText('New Inference Job (I2V)')).toBeInTheDocument();
await user.click(screen.getByRole('button', { name: 'Create Job' }));
await waitFor(() => expect(onSuccess).toHaveBeenCalledTimes(1));
const payload = vi.mocked(createJob).mock.calls[0][0] as unknown as Record<string, unknown>;
expect(payload).toMatchObject({
model_id: H3.id,
name: 'from-file',
prompt: BASE_PROMPT,
num_frames: 243,
seed: 7,
references: [{ source: '/data/inputs/face.png', media_type: 'image' }],
});
});
it("warns when the file's model isn't offered for the workload", async () => {
renderModal({
workloadType: 't2v',
prefillJob: { id: 'imported-2', model_id: 'not/a-real-model', prompt: 'p' },
});
await waitFor(() => expect(toast.warning).toHaveBeenCalled());
expect(vi.mocked(toast.warning).mock.calls[0][0]).toContain('not/a-real-model');
expect(screen.getByLabelText('Model')).toHaveValue(MODELS[0].id);
});
});
describe('CreateJobModal shot list on a raw prompt', () => {
beforeEach(() => {
vi.mocked(getModels).mockResolvedValue([...MODELS, H3]);
});
async function openShotList(user: ReturnType<typeof userEvent.setup>) {
await user.click(screen.getByRole('button', { name: 'Open shot list' }));
return screen.findByRole('dialog', { name: 'Shot list' });
}
it('edits dialogue inside integrated_multimodal_description and leaves the rest of the prompt alone', async () => {
const user = userEvent.setup();
renderModal({
workloadType: 'i2v',
prefillJob: {
id: 'imported-3',
model_id: H3.id,
prompt: BASE_PROMPT,
references: [{ source: '/data/face.png', media_type: 'image' }],
},
});
await waitFor(() => expect(screen.getByLabelText('Model')).toHaveValue(H3.id));
const dialog = await openShotList(user);
expect(within(dialog).getAllByText(/integrated_multimodal_description/).length).toBeGreaterThan(0);
expect(within(dialog).queryByText('Retention analysis')).not.toBeInTheDocument();
const line = within(dialog).getByPlaceholderText('What the speaker says') as HTMLTextAreaElement;
expect(line.value).toBe('Sorry.');
await user.type(line, ' Really.');
await user.click(within(dialog).getByRole('button', { name: 'Apply' }));
await waitFor(() =>
expect(screen.getByLabelText('Prompt')).toHaveValue(BASE_PROMPT.replace('Sorry.', 'Sorry. Really.')),
);
});
it('treats a plain prompt as the shot text and writes shot markers back over it', async () => {
const user = userEvent.setup();
renderModal({ workloadType: 't2v' });
await screen.findByRole('option', { name: /MiniMax H3/ });
await user.selectOptions(screen.getByLabelText('Model'), H3.id);
await user.type(screen.getByLabelText('Prompt'), 'A dock at dawn.');
const dialog = await openShotList(user);
await user.click(within(dialog).getByRole('button', { name: 'Apply' }));
await waitFor(() => expect(screen.getByLabelText('Prompt')).toHaveValue('[Shot 1] A dock at dawn.'));
});
it('keeps retention analysis in step when the raw prompt has a detailed_description and retention_analysis', async () => {
const user = userEvent.setup();
const sections = [
'subject_definitions:',
'<Subject 1> is the courier.',
'',
'retention_analysis:',
'<Subject 1> (appears in [Shot 1]): fully_preserved - jacket.',
'',
'detailed_description:',
'[Shot 1] Wide shot. A rooftop with <Subject 1>.',
].join('\n');
renderModal({ workloadType: 't2v', prefillJob: { id: 'imported-4', model_id: H3.id, prompt: sections } });
await waitFor(() => expect(screen.getByLabelText('Model')).toHaveValue(H3.id));
const dialog = await openShotList(user);
expect(within(dialog).getByText('Retention analysis')).toBeInTheDocument();
await user.click(within(dialog).getAllByRole('button', { name: /Add shot/ })[0]);
await user.click(within(dialog).getAllByRole('button', { name: '<Subject 1>' })[1]);
await user.click(within(dialog).getByRole('button', { name: 'Apply' }));
const out = (screen.getByLabelText('Prompt') as HTMLTextAreaElement).value;
expect(out).toContain('<Subject 1> (appears in [Shot 1], [Shot 2]): fully_preserved - jacket.');
expect(out).toContain('subject_definitions:\n<Subject 1> is the courier.');
});
it('is not offered for a non-H3 model', async () => {
renderModal({ workloadType: 't2v' });
await screen.findByRole('option', { name: 'Wan T2V (wan/t2v-1.3b)' });
expect(screen.queryByRole('button', { name: 'Open shot list' })).not.toBeInTheDocument();
});
});
@@ -49,8 +49,18 @@ import {
parseH3Prompt,
serializeH3Prompt,
type H3PromptFields,
type H3PromptSection,
} from '@/lib/h3Prompt';
import { applyH3Tag, type H3Tag } from '@/lib/h3Tags';
import { H3TagConsole } from '@/components/jobs/H3TagConsole';
import { H3DialogueEditor } from '@/components/jobs/H3DialogueEditor';
import { H3ShotListEditor } from '@/components/jobs/H3ShotListEditor';
import { jobToFormFields, type JobLike } from '@/lib/jobToFields';
import { readField, shotTarget, writeField, writeShotTarget } from '@/lib/h3PromptFields';
import { toast } from 'sonner';
/** Prompt fields the speech-tag console can insert into. */
type TagTarget = 'prompt' | H3PromptSection;
export interface CreateJobModalProps {
isOpen: boolean;
@@ -60,6 +70,8 @@ export interface CreateJobModalProps {
workloadType: string;
/** When set, the modal edits this pending job instead of creating a new one. */
editingJob?: JobLike | null;
/** Seed the form from this job (an imported job file) but still create a new job on submit. */
prefillJob?: JobLike | null;
/** Show the configuration without allowing changes (started/finished jobs). */
readOnly?: boolean;
}
@@ -71,6 +83,7 @@ export default function CreateJobModal({
jobType,
workloadType,
editingJob,
prefillJob,
readOnly = false,
}: CreateJobModalProps) {
const { options } = useStore(defaultOptionsStore);
@@ -121,11 +134,85 @@ export default function CreateJobModal({
// H3 is the only registered model with an end frame or references.
const supportsLastImage = modelId.toLowerCase().includes('minimax-h3');
const usingReferences = supportsLastImage && references.length > 0;
// Same check, named for where it gates H3-only prompt features (speech tags)
// rather than the end-frame/reference upload UI.
const isH3Model = supportsLastImage;
// Which prompt field the speech-tag console inserts into: whichever one is
// actually visible, tracked via each field's onFocus below.
const [activeTagField, setActiveTagField] = React.useState<TagTarget>('prompt');
const tagFieldRefs = React.useRef<Partial<Record<TagTarget, HTMLTextAreaElement | null>>>({});
const showGuidedFields = usingReferences && useGuidedPrompt;
React.useEffect(() => {
setActiveTagField(showGuidedFields ? 'detailed_description' : 'prompt');
}, [showGuidedFields]);
function getTagFieldValue(field: TagTarget): string {
return field === 'prompt' ? prompt : promptFields[field];
}
function setTagFieldValue(field: TagTarget, value: string) {
if (field === 'prompt') setPrompt(value);
else setPromptField(field, value);
}
function insertH3Tag(tag: H3Tag) {
const field = activeTagField;
const el = tagFieldRefs.current[field] ?? null;
const currentValue = getTagFieldValue(field);
const start = el?.selectionStart ?? currentValue.length;
const end = el?.selectionEnd ?? currentValue.length;
const result = applyH3Tag(currentValue, start, end, tag);
setTagFieldValue(field, result.value);
// The textarea re-renders with the new value before this runs; restore
// focus and the cursor position so tags can be chained without reaching
// for the mouse each time.
requestAnimationFrame(() => {
const node = tagFieldRefs.current[field];
if (node) {
node.focus();
node.setSelectionRange(result.start, result.end);
}
});
}
// Full-screen dialogue editor: opens with a copy of whichever field the
// speech-tag console would currently insert into, and on Apply overwrites
// that same field wholesale (not just an insertion at the cursor).
const [dialogueEditorOpen, setDialogueEditorOpen] = React.useState(false);
const dialogueEditorField = activeTagField;
// Shot list: writes into detailed_description AND retention_analysis
// together, so unlike the speech-tag console/dialogue editor it isn't
// scoped to "whichever field is focused" -- it's specific to those two.
const [shotListOpen, setShotListOpen] = React.useState(false);
// 'sections' edits the guided Detailed description; 'prompt' edits the shots
// inside the raw prompt (base format, or a plain prompt).
const [shotListMode, setShotListMode] = React.useState<'sections' | 'prompt'>('sections');
function applyShotList(detailedDescription: string, retentionAnalysis: string) {
setPromptFields((prev) => ({
...prev,
detailed_description: detailedDescription,
retention_analysis: retentionAnalysis,
}));
}
const promptShots = React.useMemo(() => shotTarget(prompt), [prompt]);
const promptSubjects = React.useMemo(() => readField(prompt, 'subject_definitions') ?? '', [prompt]);
const promptRetention = React.useMemo(() => readField(prompt, 'retention_analysis'), [prompt]);
const promptSyncsRetention = promptShots.field === 'detailed_description' && promptRetention !== null;
function applyShotListToPrompt(shotText: string, retentionAnalysis: string) {
let next = writeShotTarget(prompt, promptShots, shotText);
if (promptSyncsRetention) next = writeField(next, 'retention_analysis', retentionAnalysis);
setPrompt(next);
}
// JobCard re-renders on every job-list poll, so `editingJob` is a fresh
// object each time. Effects must depend on these, never on the object.
const editingJobId = editingJob?.id ?? null;
const editingJobModelId = editingJob?.model_id ?? null;
const seedJob = editingJob ?? prefillJob ?? null;
const seedJobId = seedJob?.id ?? null;
const seedJobModelId = seedJob?.model_id ?? null;
const isPrefill = !editingJob && !!prefillJob;
// Layerwise offload and FSDP compete for the DiT weights and FastVideoArgs
// silently picks a winner (fastvideo_args.py:859); resolve it visibly here.
@@ -204,10 +291,10 @@ export default function CreateJobModal({
const justOpened = isOpen && !justOpenedRef.current;
justOpenedRef.current = isOpen;
if (!justOpened) return;
if (editingJob) {
if (seedJob) {
// Must not fall through to the defaults below: a partially-seeded
// form silently edits values the user never saw.
const f = jobToFormFields(editingJob);
const f = jobToFormFields(seedJob);
setModelId(f.modelId);
setName(f.name);
setPrompt(f.prompt);
@@ -296,7 +383,7 @@ export default function CreateJobModal({
setRealScoreModelPath('');
setFakeScoreModelPath('');
}
}, [isOpen, workloadType, inferenceWorkload, options, editingJobId]);
}, [isOpen, workloadType, inferenceWorkload, options, seedJobId]);
// Load the models available for this workload.
React.useEffect(() => {
@@ -319,7 +406,7 @@ export default function CreateJobModal({
// When editing, the job's own model wins over the workload default --
// this resolves after the seeding effect, so choosing a default here
// would silently swap the model out from under the user.
const editedId = editingJobModelId;
const editedId = seedJobModelId;
const chosen =
editedId && ids.includes(editedId)
? editedId
@@ -327,6 +414,11 @@ export default function CreateJobModal({
? defaultId
: (list[0]?.id ?? '');
setModelId(chosen);
if (isPrefill && editedId && chosen !== editedId) {
toast.warning(
`${editedId} isn't available for ${inferenceWorkload.toUpperCase()}${chosen ? `, so ${chosen} is selected instead` : ''}.`,
);
}
if (workloadType === 'dmd_t2v') {
setRealScoreModelPath(chosen);
setFakeScoreModelPath(chosen);
@@ -347,7 +439,7 @@ export default function CreateJobModal({
return () => {
stale = true;
};
}, [isOpen, inferenceWorkload, workloadType, editingJobModelId]);
}, [isOpen, inferenceWorkload, workloadType, seedJobModelId, isPrefill]);
// Training jobs need a dataset; load the ready datasets when relevant.
React.useEffect(() => {
@@ -857,6 +949,22 @@ export default function CreateJobModal({
</FieldRow>
)}
{isH3Model && isInference && workloadType !== 't2i' && (
<div className="flex flex-col gap-2">
<H3TagConsole onInsert={insertH3Tag} disabled={isSubmitting} />
<Button
type="button"
variant="outline"
size="sm"
className="self-start"
onClick={() => setDialogueEditorOpen(true)}
disabled={isSubmitting}
>
Open dialogue editor
</Button>
</div>
)}
{usingReferences && useGuidedPrompt ? (
/* Six-section format from the model's reference prompt guide. */
<>
@@ -868,12 +976,31 @@ export default function CreateJobModal({
>
<Textarea
id={`modal-prompt-${section}`}
ref={(el) => {
tagFieldRefs.current[section] = el;
}}
value={promptFields[section]}
onChange={(e) => setPromptField(section, e.target.value)}
onFocus={() => setActiveTagField(section)}
rows={section === 'detailed_description' ? 5 : 2}
placeholder={H3_SECTION_HINTS[section]}
disabled={isSubmitting}
/>
{section === 'detailed_description' && (
<Button
type="button"
variant="outline"
size="sm"
className="self-start"
onClick={() => {
setShotListMode('sections');
setShotListOpen(true);
}}
disabled={isSubmitting}
>
Open shot list
</Button>
)}
</FieldRow>
))}
<button
@@ -893,8 +1020,12 @@ export default function CreateJobModal({
>
<Textarea
id="modal-prompt"
ref={(el) => {
tagFieldRefs.current.prompt = el;
}}
value={prompt}
onChange={(e) => setPrompt(e.target.value)}
onFocus={() => setActiveTagField('prompt')}
rows={isInference ? 3 : 2}
placeholder={
isInference
@@ -904,6 +1035,21 @@ export default function CreateJobModal({
required={!(usingReferences && useGuidedPrompt)}
disabled={isSubmitting}
/>
{isH3Model && isInference && workloadType !== 't2i' && (
<Button
type="button"
variant="outline"
size="sm"
className="self-start"
onClick={() => {
setShotListMode('prompt');
setShotListOpen(true);
}}
disabled={isSubmitting}
>
Open shot list
</Button>
)}
</FieldRow>
{usingReferences && (
<button
@@ -1379,6 +1525,38 @@ export default function CreateJobModal({
</Button>
</div>
</form>
{/* Nested in the React tree (still portaled to <body>) so Radix treats
clicks and focus inside these as inside this dialog. As siblings
they counted as "outside", and closing one could close this too. */}
<H3DialogueEditor
open={dialogueEditorOpen}
initialValue={getTagFieldValue(dialogueEditorField)}
onApply={(value) => setTagFieldValue(dialogueEditorField, value)}
onClose={() => setDialogueEditorOpen(false)}
/>
<H3ShotListEditor
open={shotListOpen}
subjectDefinitions={shotListMode === 'sections' ? promptFields.subject_definitions : promptSubjects}
initialDetailedDescription={
shotListMode === 'sections' ? promptFields.detailed_description : promptShots.body
}
initialRetentionAnalysis={
shotListMode === 'sections' ? promptFields.retention_analysis : (promptRetention ?? '')
}
targetLabel={
shotListMode === 'sections'
? undefined
: promptShots.field === 'integrated_multimodal_description'
? 'integrated_multimodal_description'
: promptShots.field === 'detailed_description'
? 'Detailed description'
: 'the prompt'
}
withRetention={shotListMode === 'sections' || promptSyncsRetention}
onApply={shotListMode === 'sections' ? applyShotList : applyShotListToPrompt}
onClose={() => setShotListOpen(false)}
/>
</DialogContent>
</Dialog>
);
@@ -0,0 +1,346 @@
'use client';
import * as React from 'react';
import { Button } from '@/components/ui/button';
import {
Dialog,
DialogContent,
DialogDescription,
DialogFooter,
DialogHeader,
DialogTitle,
} from '@/components/ui/dialog';
import { Textarea } from '@/components/ui/textarea';
import { H3DialoguePreview } from '@/components/jobs/H3DialoguePreview';
import { H3TagPalette } from '@/components/jobs/H3TagPalette';
import { useUndoableText } from '@/hooks/useUndoableText';
import { applyH3Preset, applyH3Tag, type H3Preset, type H3Tag } from '@/lib/h3Tags';
import {
SPEECH_RATES,
dialogueTree,
estimateDuration,
type DialogueNode,
type SpeechRate,
} from '@/lib/h3DialogueEstimate';
import { cn } from '@/lib/utils';
export interface H3DialogueEditorProps {
open: boolean;
/** Seeded from whichever prompt field the editor was opened for. */
initialValue: string;
onApply: (value: string) => void;
onClose: () => void;
}
/** Every `<d>...</d>` block's inner text -- what's actually spoken, ignoring surrounding shot/camera prose. */
function extractDialogueBlocks(text: string): string[] {
const blocks: string[] = [];
const re = /<d>([\s\S]*?)<\/d>/g;
let m: RegExpExecArray | null;
while ((m = re.exec(text))) blocks.push(m[1].replace(/^\s*\[[^\]]*\]\s*/, '')); // drop a leading [Language] marker
return blocks;
}
/**
* Full-screen dialogue editor for MiniMax-H3 speech tags: the same tag
* registry as `H3TagConsole`, plus a duration estimate (scoped to `<d>`
* blocks so scene-description prose doesn't inflate it), a styled
* performance preview, and emotion presets. Opens with a copy of the target
* field's text; `onApply` hands the edited text back, `onClose` discards it.
* The shot list editor reuses the same palette, preview, and undo history.
*/
export function H3DialogueEditor({ open, initialValue, onApply, onClose }: H3DialogueEditorProps) {
const { text, type, commit, undo, redo, reset, canUndo, canRedo } = useUndoableText(initialValue);
const [pace, setPace] = React.useState<SpeechRate>('Natural');
const [calib, setCalib] = React.useState(1);
const [actualSeconds, setActualSeconds] = React.useState('');
const taRef = React.useRef<HTMLTextAreaElement>(null);
const pending = React.useRef<[number, number] | null>(null);
function restoreCaretAtEnd(restored: string | null) {
if (restored != null) pending.current = [restored.length, restored.length];
}
// Re-seed from the field's current value each time the editor opens, the
// same "seed on the open transition only" guard CreateJobModal itself
// uses -- otherwise an in-progress edit here could be clobbered by a
// stray parent re-render while open. History resets with it: undo
// shouldn't reach back into a previous time this editor was opened.
const wasOpenRef = React.useRef(false);
React.useEffect(() => {
if (open && !wasOpenRef.current) {
reset(initialValue);
setActualSeconds('');
setCalib(1);
}
wasOpenRef.current = open;
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [open]);
React.useEffect(() => {
if (pending.current && taRef.current) {
const [s, e] = pending.current;
taRef.current.focus();
taRef.current.setSelectionRange(s, e);
pending.current = null;
}
});
function handleKeyDown(e: React.KeyboardEvent<HTMLTextAreaElement>) {
if (!(e.metaKey || e.ctrlKey) || e.key.toLowerCase() !== 'z') return;
e.preventDefault();
restoreCaretAtEnd(e.shiftKey ? redo() : undo());
}
function insertTag(tag: H3Tag) {
const ta = taRef.current;
const result = applyH3Tag(text, ta?.selectionStart ?? text.length, ta?.selectionEnd ?? text.length, tag);
commit(result.value);
pending.current = [result.start, result.end];
}
function applyPreset(p: H3Preset) {
const ta = taRef.current;
const result = applyH3Preset(text, ta?.selectionStart ?? text.length, ta?.selectionEnd ?? text.length, p);
commit(result.value);
pending.current = [result.start, result.end];
}
const dialogueBlocks = React.useMemo(() => extractDialogueBlocks(text), [text]);
// Falls back to the whole field only when it contains no <d> blocks at
// all yet -- e.g. a fresh field the user is about to write dialogue into.
const segments = dialogueBlocks.length > 0 ? dialogueBlocks : [text];
const estimates = React.useMemo(
() => segments.map((s) => estimateDuration(s, SPEECH_RATES[pace], calib)),
// eslint-disable-next-line react-hooks/exhaustive-deps
[segments.join('\n'), pace, calib],
);
const totalSeconds = estimates.reduce((sum, e) => sum + e.total, 0);
function calibrate() {
const actual = parseFloat(actualSeconds);
if (!actual || actual <= 0 || segments.length === 0) return;
const base = segments.reduce((sum, s) => sum + estimateDuration(s, SPEECH_RATES[pace], 1).total, 0);
if (base > 0) setCalib(actual / base);
}
/* ---- rough browser-voice preview, for rhythm/pacing only ---- */
const synth = typeof window !== 'undefined' ? window.speechSynthesis : null;
const [playing, setPlaying] = React.useState(false);
const playTokenRef = React.useRef({ cancelled: true });
function stopPreview() {
playTokenRef.current.cancelled = true;
synth?.cancel();
setPlaying(false);
}
function playPreview() {
if (!synth) return;
stopPreview();
const token = { cancelled: false };
playTokenRef.current = token;
setPlaying(true);
const VOX: Record<string, string> = {
laughs: 'ha ha ha', chuckle: 'heh', sighs: 'haah', gasp: 'ah', uh: 'uh', mhm: 'mhm', phew: 'phew',
};
const atoms: { say?: string; gap?: number }[] = [];
for (const [i, seg] of segments.entries()) {
if (i > 0) atoms.push({ gap: 0.6 });
const walk = (nodes: DialogueNode[]) => {
for (const n of nodes) {
if (n.t === 'text') {
const t = n.v.replace(/\s+/g, ' ').trim();
if (t) atoms.push({ say: t });
} else if (n.t === 'point') {
atoms.push(VOX[n.name] ? { say: VOX[n.name] } : { gap: 0.25 });
} else {
if (n.name === 'humming') atoms.push({ say: 'hmm hmm' });
else walk(n.children);
}
}
};
walk(dialogueTree(seg));
}
let i = 0;
const step = () => {
if (token.cancelled) return;
if (i >= atoms.length) {
setPlaying(false);
return;
}
const a = atoms[i++];
if (a.gap != null) {
setTimeout(() => { if (!token.cancelled) step(); }, a.gap * 1000);
return;
}
const u = new SpeechSynthesisUtterance(a.say);
u.rate = 0.95;
u.onend = () => { if (!token.cancelled) step(); };
u.onerror = () => { if (!token.cancelled) step(); };
synth.speak(u);
};
step();
}
React.useEffect(() => () => stopPreview(), []); // eslint-disable-line react-hooks/exhaustive-deps
function handleApply() {
onApply(text);
onClose();
}
return (
<Dialog open={open} onOpenChange={(o) => { if (!o) onClose(); }}>
<DialogContent className="flex h-[92vh] max-h-[920px] w-[96vw] max-w-[1180px] flex-col gap-0 overflow-hidden p-0">
<DialogHeader className="border-b border-border px-5 py-4">
<DialogTitle>Dialogue editor</DialogTitle>
<DialogDescription>
Click a tag to insert it at the cursor, or select text first to wrap it. Applying writes the edited text
back into the field you opened this from.
</DialogDescription>
</DialogHeader>
<div className="flex min-h-0 flex-1">
{/* palette */}
<H3TagPalette
onInsertTag={insertTag}
onApplyPreset={applyPreset}
className="w-64 shrink-0 overflow-y-auto border-r border-border p-3"
/>
{/* editor + preview */}
<div className="flex min-w-0 flex-1 flex-col gap-4 overflow-y-auto p-4">
{/* duration estimate */}
<div className="rounded-xl border border-border bg-secondary/30 p-4">
<div className="flex flex-wrap items-baseline justify-between gap-3">
<div className="flex items-baseline gap-3">
<span className="text-2xl font-semibold tracking-tight text-foreground">
{totalSeconds.toFixed(1)}<span className="ml-1 text-sm font-normal text-muted-foreground">s spoken</span>
</span>
<span className="font-mono text-xs text-muted-foreground">
{dialogueBlocks.length > 0
? `across ${dialogueBlocks.length} <d> block${dialogueBlocks.length === 1 ? '' : 's'}`
: 'no <d> blocks yet — estimating the whole field'}
</span>
</div>
<div className="flex items-center gap-1 rounded-lg border border-border bg-card p-1">
{(Object.keys(SPEECH_RATES) as SpeechRate[]).map((p) => (
<button
key={p}
type="button"
onClick={() => setPace(p)}
className={cn(
'rounded-md px-2.5 py-1 text-xs transition-colors',
pace === p ? 'bg-secondary text-foreground' : 'text-muted-foreground hover:bg-accent',
)}
>
{p}
</button>
))}
</div>
</div>
{segments.length > 1 && (
<div className="mt-3 flex flex-wrap gap-1.5">
{estimates.map((e, i) => (
<span key={i} className="rounded-md border border-border bg-card px-2 py-0.5 font-mono text-[11px] text-muted-foreground">
block {i + 1}: {e.total.toFixed(1)}s
</span>
))}
</div>
)}
<div className="mt-3 flex flex-wrap items-center gap-2 text-xs text-muted-foreground">
<span title="Enter the real length of a clip you already generated, to tune the estimate to your setup">
calibrate
</span>
<input
value={actualSeconds}
onChange={(e) => setActualSeconds(e.target.value)}
placeholder="actual s"
inputMode="decimal"
className="w-16 rounded-md border border-input bg-card px-2 py-1 text-foreground"
/>
<Button type="button" variant="outline" size="sm" onClick={calibrate}>Set</Button>
{Math.abs(calib - 1) > 0.001 && (
<button
type="button"
onClick={() => { setCalib(1); setActualSeconds(''); }}
className="font-mono text-muted-foreground/80 hover:text-foreground"
title="Reset calibration"
>
×{calib.toFixed(2)}
</button>
)}
</div>
</div>
{/* audio preview */}
{synth && (
<div className="flex items-center gap-3 rounded-xl border border-border bg-secondary/30 p-3">
<Button type="button" size="sm" onClick={playing ? stopPreview : playPreview}>
{playing ? '■ Stop' : '▶ Play preview'}
</Button>
<span className="text-xs text-muted-foreground">
A stand-in browser voice for rhythm only — it won&apos;t match MiniMax&apos;s voice.
</span>
</div>
)}
{/* the field itself */}
<div className="flex items-center justify-between">
<span className="text-xs font-semibold text-foreground">Text</span>
<div className="flex items-center gap-1">
<Button
type="button"
variant="outline"
size="sm"
onClick={() => restoreCaretAtEnd(undo())}
disabled={!canUndo}
title="Undo (⌘/Ctrl+Z)"
>
↶ Undo
</Button>
<Button
type="button"
variant="outline"
size="sm"
onClick={() => restoreCaretAtEnd(redo())}
disabled={!canRedo}
title="Redo (⌘/Ctrl+Shift+Z)"
>
↷ Redo
</Button>
</div>
</div>
<Textarea
ref={taRef}
value={text}
onChange={(e) => type(e.target.value)}
onKeyDown={handleKeyDown}
spellCheck={false}
rows={8}
className="flex-1 font-mono text-sm"
/>
{/* performance preview */}
<div>
<div className="mb-2 text-xs font-semibold text-foreground">Performance preview</div>
<H3DialoguePreview
text={text}
className="rounded-xl border border-border bg-card p-4 text-sm leading-7 text-foreground"
/>
</div>
</div>
</div>
<DialogFooter className="border-t border-border px-5 py-3">
<Button type="button" variant="outline" onClick={onClose}>Cancel</Button>
<Button type="button" onClick={handleApply}>Apply to field</Button>
</DialogFooter>
</DialogContent>
</Dialog>
);
}
@@ -0,0 +1,44 @@
'use client';
import * as React from 'react';
import { CATEGORY_STYLE } from '@/components/jobs/H3TagPalette';
import { dialogueTree, type DialogueNode } from '@/lib/h3DialogueEstimate';
import { H3_TAG_CATEGORIES } from '@/lib/h3Tags';
import { cn } from '@/lib/utils';
const CATEGORY_BY_TAG_NAME: Record<string, string> = Object.fromEntries(
H3_TAG_CATEGORIES.flatMap((c) => c.tags.map((t) => [t.open.replace(/[<>]/g, ''), c.id])),
);
function renderPreviewNodes(nodes: DialogueNode[], keyPrefix = ''): React.ReactNode[] {
return nodes.map((n, i) => {
const key = `${keyPrefix}${i}`;
if (n.t === 'text') return <span key={key}>{n.v}</span>;
if (n.t === 'point') {
const cat = CATEGORY_BY_TAG_NAME[n.name] ?? 'body';
const style = CATEGORY_STYLE[cat] ?? CATEGORY_STYLE.body;
return (
<span
key={key}
className={cn(
'mx-0.5 inline-block rounded border px-1.5 py-0.5 font-mono text-[11px] leading-none',
style.chip,
)}
>
{n.name}
</span>
);
}
const inner = renderPreviewNodes(n.children, `${key}-`);
if (n.name === 'i') return <strong key={key} className="italic">{inner}</strong>;
if (n.name === 'whisper') return <span key={key} className="text-[0.92em] tracking-wide opacity-60">{inner}</span>;
if (n.name === 'humming') return <em key={key} className="underline decoration-wavy decoration-emerald-500 underline-offset-4">{inner}</em>;
return <span key={key}>{inner}</span>;
});
}
/** Styled "performance" rendering of speech tags: chips for point tags, styling for wrapping ones. */
export function H3DialoguePreview({ text, className }: { text: string; className?: string }) {
return <div className={className}>{renderPreviewNodes(dialogueTree(text))}</div>;
}
@@ -0,0 +1,184 @@
import { render, screen, within } from '@testing-library/react';
import userEvent from '@testing-library/user-event';
import { describe, expect, it, vi } from 'vitest';
import { H3DialogueEditor } from './H3DialogueEditor';
import { H3ShotListEditor } from './H3ShotListEditor';
const SUBJECTS = '<Subject 1> is the courier.\n<Subject 2> is the rival.';
const DETAILED = [
'[Shot 1] Wide shot. A rooftop at night.',
'[Shot 2] Close-up shot, handheld. He turns. (S2) <d>[English] I had no choice.</d>',
].join('\n\n');
describe('H3DialogueEditor (shared palette + undo)', () => {
it('inserts a tag at the cursor, undoes it, and applies the edited text', async () => {
const user = userEvent.setup();
const onApply = vi.fn();
render(<H3DialogueEditor open initialValue="Hello there" onApply={onApply} onClose={() => {}} />);
const textarea = screen.getByDisplayValue('Hello there') as HTMLTextAreaElement;
textarea.focus();
textarea.setSelectionRange(5, 5);
await user.click(screen.getByRole('button', { name: /<pause>/ }));
expect(textarea.value).toBe('Hello <pause> there');
await user.click(screen.getByRole('button', { name: /Undo/ }));
expect(textarea.value).toBe('Hello there');
await user.click(screen.getByRole('button', { name: 'Nervous' }));
expect(textarea.value).toContain('<uh> <stutter>');
await user.click(screen.getByRole('button', { name: 'Apply to field' }));
expect(onApply).toHaveBeenCalledWith(expect.stringContaining('<uh> <stutter>'));
});
});
describe('H3ShotListEditor dialogue tooling', () => {
function renderEditor(onApply = vi.fn()) {
render(
<H3ShotListEditor
open
subjectDefinitions={SUBJECTS}
initialDetailedDescription={DETAILED}
initialRetentionAnalysis=""
onApply={onApply}
onClose={() => {}}
/>,
);
return onApply;
}
it('disables the palette until a shot dialogue is focused', () => {
renderEditor();
expect(screen.getByText(/Click into a shot's dialogue/)).toBeInTheDocument();
expect(screen.getByRole('button', { name: /<pause>/ })).toBeDisabled();
});
it('inserts a palette tag into the focused shot dialogue and writes it on apply', async () => {
const user = userEvent.setup();
const onApply = renderEditor();
const dialogueBoxes = screen.getAllByPlaceholderText('What the speaker says') as HTMLTextAreaElement[];
expect(dialogueBoxes).toHaveLength(1);
expect(dialogueBoxes[0].value).toBe('I had no choice.');
await user.click(dialogueBoxes[0]);
dialogueBoxes[0].setSelectionRange(1, 1);
expect(screen.getByText('Tags insert into Shot 2 dialogue')).toBeInTheDocument();
await user.click(screen.getByRole('button', { name: /<pause>/ }));
expect(dialogueBoxes[0].value).toBe('I <pause> had no choice.');
await user.click(screen.getByRole('button', { name: 'Apply' }));
expect(onApply).toHaveBeenCalledTimes(1);
expect(onApply.mock.calls[0][0]).toContain('<d>[English] I <pause> had no choice.</d>');
});
it('adds and removes dialogue lines, writing every line on apply', async () => {
const user = userEvent.setup();
const onApply = renderEditor();
await user.click(screen.getAllByRole('button', { name: '+ Add line' })[1]);
const boxes = screen.getAllByPlaceholderText('What the speaker says') as HTMLTextAreaElement[];
await user.type(boxes[1], 'Then go.');
await user.click(screen.getByRole('button', { name: 'Apply' }));
expect(onApply.mock.calls[0][0]).toContain(
'(S2) <d>[English] I had no choice.</d> (S2) <d>[English] Then go.</d>',
);
});
it('shows the styled preview once a shot dialogue contains tags', async () => {
const user = userEvent.setup();
renderEditor();
const boxes = screen.getAllByPlaceholderText('What the speaker says') as HTMLTextAreaElement[];
await user.click(boxes[0]);
await user.click(screen.getByRole('button', { name: /<long pause>/ }));
const shot2 = boxes[0].closest('[id^="shot-card-"]') as HTMLElement;
expect(within(shot2).getByText('long pause')).toBeInTheDocument();
});
});
describe('H3ShotListEditor ordered body', () => {
function renderWith(detailed: string) {
const onApply = vi.fn();
render(
<H3ShotListEditor
open
subjectDefinitions={SUBJECTS}
initialDetailedDescription={detailed}
initialRetentionAnalysis=""
onApply={onApply}
onClose={() => {}}
/>,
);
return onApply;
}
it('shows hand-written dialogue as an editable line between its prose and applies the text unchanged', async () => {
const user = userEvent.setup();
const text = "[Shot 1] He (S1) says: <d>[English] I think you're sorry.</d> <cutoff>";
const onApply = renderWith(text);
expect(screen.getByText(/keeps its original wording/)).toBeInTheDocument();
const lines = screen.getAllByPlaceholderText('What the speaker says') as HTMLTextAreaElement[];
expect(lines).toHaveLength(1);
expect(lines[0].value).toBe("I think you're sorry.");
const prose = screen.getAllByPlaceholderText("What's happening in this shot") as HTMLTextAreaElement[];
expect(prose.map((p) => p.value)).toEqual(['He (S1) says: ', ' <cutoff>']);
await user.click(screen.getByRole('button', { name: 'Apply' }));
expect(onApply.mock.calls[0][0]).toBe(text);
});
it('editing a hand-written line keeps the tag after it where it was', async () => {
const user = userEvent.setup();
const onApply = renderWith("[Shot 1] He (S1) says: <d>[English] Sorry.</d> <cutoff>");
const [line] = screen.getAllByPlaceholderText('What the speaker says') as HTMLTextAreaElement[];
await user.type(line, ' Really.');
await user.click(screen.getByRole('button', { name: 'Apply' }));
expect(onApply.mock.calls[0][0]).toBe("[Shot 1] He (S1) says: <d>[English] Sorry. Really.</d> <cutoff>");
});
it('reorders dialogue lines with the move buttons', async () => {
const user = userEvent.setup();
const onApply = renderWith('[Shot 1] Wide shot. (S1) <d>[English] One.</d> (S2) <d>[English] Two.</d>');
await user.click(screen.getByRole('button', { name: 'Move line 2 up' }));
await user.click(screen.getByRole('button', { name: 'Apply' }));
expect(onApply.mock.calls[0][0]).toBe(
'[Shot 1] Wide shot. (S2) <d>[English] Two.</d> (S1) <d>[English] One.</d>',
);
});
it('adds a text block after the existing content', async () => {
const user = userEvent.setup();
const onApply = renderWith('[Shot 1] Wide shot. A dock.');
await user.click(screen.getByRole('button', { name: '+ Add text' }));
const prose = screen.getAllByPlaceholderText("What's happening in this shot") as HTMLTextAreaElement[];
expect(prose).toHaveLength(2);
await user.type(prose[1], 'Rain falls.');
await user.click(screen.getByRole('button', { name: 'Apply' }));
expect(onApply.mock.calls[0][0]).toBe('[Shot 1] Wide shot. A dock. Rain falls.');
});
it('removes a text block', async () => {
const user = userEvent.setup();
const onApply = renderWith('[Shot 1] Wide shot. A dock. (S1) <d>[English] Hi.</d>');
await user.click(screen.getByRole('button', { name: 'Remove text 1' }));
await user.click(screen.getByRole('button', { name: 'Apply' }));
expect(onApply.mock.calls[0][0]).toBe('[Shot 1] Wide shot. (S1) <d>[English] Hi.</d>');
});
it('adds a new line at the end of the shot, after any closing prose', async () => {
const user = userEvent.setup();
const onApply = renderWith('[Shot 1] Wide shot. He speaks. (S1) <d>[English] First.</d> He pauses.');
await user.click(screen.getByRole('button', { name: '+ Add line' }));
const lines = screen.getAllByPlaceholderText('What the speaker says') as HTMLTextAreaElement[];
await user.type(lines[1], 'Later.');
await user.click(screen.getByRole('button', { name: 'Apply' }));
expect(onApply.mock.calls[0][0]).toBe(
'[Shot 1] Wide shot. He speaks. (S1) <d>[English] First.</d> He pauses. (S1) <d>[English] Later.</d>',
);
});
});
@@ -0,0 +1,123 @@
import { render, screen, within } from '@testing-library/react';
import userEvent from '@testing-library/user-event';
import { describe, expect, it, vi } from 'vitest';
import { H3ShotListEditor } from './H3ShotListEditor';
// Original test material -- not from any real screenplay.
const SCRIPT = [
'CAPTAIN: Sit down. (EXHALES) We need to talk about the ledger.',
'',
'CLERK: Yes, sir.',
'',
'CAPTAIN: Three crates are missing and nobody on this dock saw a thing.',
].join('\n');
const SUBJECTS = '<Subject 1> is the captain.\n<Subject 2> is the clerk.';
function renderEditor(detailed = '', onApply = vi.fn()) {
render(
<H3ShotListEditor
open
subjectDefinitions={SUBJECTS}
initialDetailedDescription={detailed}
initialRetentionAnalysis=""
onApply={onApply}
onClose={() => {}}
/>,
);
return onApply;
}
async function importScript(user: ReturnType<typeof userEvent.setup>) {
await user.click(screen.getByRole('button', { name: 'Import script…' }));
const dialog = await screen.findByRole('dialog', { name: 'Import script' });
await user.click(within(dialog).getByLabelText('Script'));
await user.paste(SCRIPT);
return dialog;
}
describe('script import', () => {
it('maps speakers to subjects by default and previews the clip', async () => {
const user = userEvent.setup();
renderEditor();
const dialog = await importScript(user);
expect(within(dialog).getByLabelText('CAPTAIN subject')).toHaveValue('<Subject 1>');
expect(within(dialog).getByLabelText('CLERK subject')).toHaveValue('<Subject 2>');
expect(within(dialog).getByLabelText('CAPTAIN speaker tag')).toHaveValue('S1');
expect(within(dialog).getByLabelText('CLERK speaker tag')).toHaveValue('S2');
expect(within(dialog).getByRole('option', { name: /Clip 1 of \d+/ })).toBeInTheDocument();
expect(within(dialog).getByText('Establishing')).toBeInTheDocument();
});
it('loads the chosen clip into the shot list and writes it on apply', async () => {
const user = userEvent.setup();
const onApply = renderEditor();
const dialog = await importScript(user);
await user.click(within(dialog).getByRole('button', { name: 'Load clip' }));
const boxes = (await screen.findAllByPlaceholderText('What the speaker says')) as HTMLTextAreaElement[];
expect(boxes[0].value).toBe('Sit down. <exhale> We need to talk about the ledger.');
expect(boxes[1].value).toBe('Yes, sir.');
await user.click(screen.getByRole('button', { name: 'Apply' }));
const [detailed, retention] = onApply.mock.calls[0];
expect(detailed).toContain('[Shot 1] Establishing shot.');
expect(detailed).toContain('(S1) <d>[English] Sit down. <exhale> We need to talk about the ledger.</d> (S2) <d>[English] Yes, sir.</d>');
expect(retention).toContain('<Subject 1> (appears in');
expect(retention).toContain('<Subject 2> (appears in');
});
it('warns that loading replaces the shots already in the list', async () => {
const user = userEvent.setup();
renderEditor('[Shot 1] Wide shot. A dock.\n\n[Shot 2] Close-up shot. A face.');
const dialog = await importScript(user);
expect(within(dialog).getByText('Replaces the 2 shots currently in the list.')).toBeInTheDocument();
});
it('reloading the same clip after edits restores the imported text', async () => {
const user = userEvent.setup();
renderEditor();
let dialog = await importScript(user);
await user.click(within(dialog).getByRole('button', { name: 'Load clip' }));
let boxes = (await screen.findAllByPlaceholderText('What the speaker says')) as HTMLTextAreaElement[];
await user.type(boxes[1], ' EDITED');
expect(boxes[1].value).toBe('Yes, sir. EDITED');
await user.click(screen.getByRole('button', { name: 'Import script…' }));
dialog = await screen.findByRole('dialog', { name: 'Import script' });
// Loading a clip advances the picker to the next one; choose clip 1 again.
await user.selectOptions(within(dialog).getByLabelText('Clip to load'), '0');
await user.click(within(dialog).getByRole('button', { name: 'Load clip' }));
boxes = (await screen.findAllByPlaceholderText('What the speaker says')) as HTMLTextAreaElement[];
expect(boxes[1].value).toBe('Yes, sir.');
});
it('advances to the next clip after loading one', async () => {
const user = userEvent.setup();
renderEditor();
let dialog = await importScript(user);
const picker = within(dialog).getByLabelText('Clip to load') as HTMLSelectElement;
expect(picker.options.length).toBeGreaterThan(1);
await user.click(within(dialog).getByRole('button', { name: 'Load clip' }));
await user.click(screen.getByRole('button', { name: 'Import script…' }));
dialog = await screen.findByRole('dialog', { name: 'Import script' });
expect((within(dialog).getByLabelText('Clip to load') as HTMLSelectElement).value).toBe('1');
});
it('explains when the text has no speaker turns', async () => {
const user = userEvent.setup();
renderEditor();
await user.click(screen.getByRole('button', { name: 'Import script…' }));
const dialog = await screen.findByRole('dialog', { name: 'Import script' });
await user.click(within(dialog).getByLabelText('Script'));
await user.paste('just some prose');
expect(within(dialog).getByRole('alert')).toHaveTextContent(/No "NAME: line" turns found/);
expect(within(dialog).getByRole('button', { name: 'Load clip' })).toBeDisabled();
});
});
@@ -0,0 +1,247 @@
'use client';
import * as React from 'react';
import { Button } from '@/components/ui/button';
import {
Dialog,
DialogContent,
DialogDescription,
DialogFooter,
DialogHeader,
DialogTitle,
} from '@/components/ui/dialog';
import { Input } from '@/components/ui/input';
import { NativeSelect } from '@/components/ui/native-select';
import { Textarea } from '@/components/ui/textarea';
import { buildClips, parseScript, scriptSpeakers, type ImportSpeaker } from '@/lib/h3ScriptImport';
import { SHOT_TYPES, shotLines, shotProse, type Shot } from '@/lib/h3Shots';
const TYPE_LABEL = Object.fromEntries(SHOT_TYPES.map((t) => [t.id, t.label]));
const DEFAULT_CLIP_SECONDS = 11;
export interface H3ScriptImportDialogProps {
open: boolean;
subjectLabels: string[];
/** shots currently in the list -- loading a clip replaces them */
existingShotCount: number;
onLoad: (shots: Shot[]) => void;
onClose: () => void;
}
function preview(shot: Shot): string {
const line = shotLines(shot).find((l) => l.text.trim());
const text = line ? `${line.speaker ? `${line.speaker}: ` : ''}“${line.text.trim()}”` : shotProse(shot);
return text.length > 90 ? `${text.slice(0, 89)}…` : text;
}
/**
* Paste (or load) a "NAME: line" script and pick one clip-sized chunk of it
* to load into the shot list. The script and speaker mapping persist between
* openings, so working through a long scene is: load clip, apply, next clip.
*/
export function H3ScriptImportDialog({
open,
subjectLabels,
existingShotCount,
onLoad,
onClose,
}: H3ScriptImportDialogProps) {
const [script, setScript] = React.useState('');
const [clipSecondsInput, setClipSecondsInput] = React.useState(String(DEFAULT_CLIP_SECONDS));
const [overrides, setOverrides] = React.useState<Record<string, Partial<ImportSpeaker>>>({});
const [clipIndex, setClipIndex] = React.useState(0);
const turns = React.useMemo(() => parseScript(script), [script]);
const names = React.useMemo(() => scriptSpeakers(turns), [turns]);
const speakers = React.useMemo(() => {
const map: Record<string, ImportSpeaker> = {};
names.forEach((name, i) => {
const o = overrides[name] ?? {};
map[name] = {
subject: 'subject' in o ? (o.subject ?? null) : (subjectLabels[i] ?? null),
tag: o.tag ?? `S${i + 1}`,
};
});
return map;
}, [names, overrides, subjectLabels]);
const parsedSeconds = Number(clipSecondsInput);
const targetSeconds = Number.isFinite(parsedSeconds) ? Math.min(Math.max(parsedSeconds, 4), 12) : DEFAULT_CLIP_SECONDS;
const clips = React.useMemo(
() => (turns.length ? buildClips(turns, { speakers, targetSeconds }) : []),
[turns, speakers, targetSeconds],
);
const selected = Math.min(clipIndex, Math.max(clips.length - 1, 0));
const clip = clips[selected];
function setOverride(name: string, patch: Partial<ImportSpeaker>) {
setOverrides((prev) => ({ ...prev, [name]: { ...prev[name], ...patch } }));
}
async function handleFile(e: React.ChangeEvent<HTMLInputElement>) {
const file = e.target.files?.[0];
e.target.value = '';
if (!file) return;
setScript(await file.text());
setClipIndex(0);
}
function handleLoad() {
if (!clip) return;
onLoad(clip.shots);
setClipIndex(Math.min(selected + 1, clips.length - 1));
onClose();
}
return (
<Dialog open={open} onOpenChange={(o) => { if (!o) onClose(); }}>
<DialogContent className="flex max-h-[90vh] w-[94vw] max-w-[760px] flex-col gap-0 overflow-hidden p-0">
<DialogHeader className="border-b border-border px-5 py-4">
<DialogTitle>Import script</DialogTitle>
<DialogDescription>
Paste a script of NAME: line turns. It&apos;s split into clip-sized chunks; load one at a time into the
shot list, then set the camera and descriptions.
</DialogDescription>
</DialogHeader>
<div className="flex min-h-0 flex-1 flex-col gap-4 overflow-y-auto p-5">
<div className="flex flex-col gap-2">
<label htmlFor="script-import-text" className="text-xs font-semibold text-foreground">
Script
</label>
<Textarea
id="script-import-text"
value={script}
onChange={(e) => {
setScript(e.target.value);
setClipIndex(0);
}}
rows={8}
spellCheck={false}
placeholder={'MARK: First line...\n\nJORDAN: Reply...'}
className="font-mono text-xs"
/>
<Input
type="file"
accept=".md,.txt,.text"
aria-label="Load script file"
onChange={handleFile}
className="h-auto py-2 file:mr-3 file:cursor-pointer file:rounded-md file:border-0 file:bg-secondary file:px-2 file:py-1 file:text-sm file:text-secondary-foreground"
/>
</div>
{script.trim() && turns.length === 0 && (
<p role="alert" className="text-sm text-destructive">
No &quot;NAME: line&quot; turns found. Each turn should start with a speaker name and a colon.
</p>
)}
{names.length > 0 && (
<div className="flex flex-col gap-2">
<span className="text-xs font-semibold text-foreground">Speakers</span>
{names.map((name) => (
<div key={name} className="grid grid-cols-[1fr_1fr_5rem] items-center gap-2">
<span className="truncate text-sm text-foreground">{name}</span>
<NativeSelect
aria-label={`${name} subject`}
value={speakers[name].subject ?? ''}
onChange={(e) => setOverride(name, { subject: e.target.value || null })}
>
<option value="">Not a tracked subject</option>
{subjectLabels.map((label) => (
<option key={label} value={label}>{label}</option>
))}
</NativeSelect>
<Input
aria-label={`${name} speaker tag`}
value={speakers[name].tag}
onChange={(e) => setOverride(name, { tag: e.target.value })}
className="font-mono"
/>
</div>
))}
{subjectLabels.length === 0 && (
<p className="text-xs text-muted-foreground">
No subjects are defined yet, so speakers can&apos;t be linked. Add them under Subject definitions
to track who appears in each shot.
</p>
)}
</div>
)}
{turns.length > 0 && (
<>
<label className="flex flex-col gap-1 text-xs text-muted-foreground">
Clip length target (seconds)
<Input
type="number"
min={4}
max={12}
step={1}
value={clipSecondsInput}
onChange={(e) => {
setClipSecondsInput(e.target.value);
setClipIndex(0);
}}
className="w-28"
/>
<span>One generation runs about 11 s (277 frames at 24 fps), so each clip becomes its own job.</span>
</label>
<div className="flex flex-col gap-2">
<label htmlFor="script-import-clip" className="text-xs font-semibold text-foreground">
Clip to load
</label>
<NativeSelect
id="script-import-clip"
value={selected}
onChange={(e) => setClipIndex(Number(e.target.value))}
>
{clips.map((c, i) => (
<option key={i} value={i}>
Clip {i + 1} of {clips.length} · {c.shots.length} shot{c.shots.length === 1 ? '' : 's'}
{c.continuation ? ' · continues previous shot' : ''}
</option>
))}
</NativeSelect>
{clip?.continuation && (
<p className="rounded-lg border border-border bg-secondary/30 px-3 py-2 text-xs text-muted-foreground">
This clip continues the previous clip&apos;s last shot, cut short because that speech didn&apos;t
fit one generation. Generate it from the previous clip&apos;s last frame (as the start Image)
with the same references.
</p>
)}
{clip && (
<ol className="flex flex-col gap-1 rounded-xl border border-border bg-secondary/20 p-3">
{clip.shots.map((shot, i) => (
<li key={shot.id} className="flex gap-2 text-xs">
<span className="w-6 shrink-0 font-mono text-muted-foreground">{i + 1}</span>
<span className="w-24 shrink-0 font-medium text-foreground">{TYPE_LABEL[shot.type]}</span>
<span className="min-w-0 text-muted-foreground">{preview(shot)}</span>
</li>
))}
</ol>
)}
</div>
</>
)}
</div>
<DialogFooter className="items-center border-t border-border px-5 py-3">
{existingShotCount > 0 && clip && (
<span className="mr-auto text-xs text-muted-foreground">
Replaces the {existingShotCount} shot{existingShotCount === 1 ? '' : 's'} currently in the list.
</span>
)}
<Button type="button" variant="outline" onClick={onClose}>Cancel</Button>
<Button type="button" onClick={handleLoad} disabled={!clip}>Load clip</Button>
</DialogFooter>
</DialogContent>
</Dialog>
);
}
@@ -0,0 +1,92 @@
'use client';
import * as React from 'react';
import { Textarea } from '@/components/ui/textarea';
import { H3DialoguePreview } from '@/components/jobs/H3DialoguePreview';
import { applyH3Preset, applyH3Tag, type H3Preset, type H3Tag } from '@/lib/h3Tags';
/** What the shared tag palette drives: the last-focused shot dialogue field. */
export interface H3ShotDialogueHandle {
insertTag: (tag: H3Tag) => void;
applyPreset: (preset: H3Preset) => void;
}
export interface H3ShotDialogueFieldProps {
/** The line's text. The parent owns it (and the undo history), so this is fully controlled. */
value: string;
/** Typing. */
onChange: (value: string) => void;
/** A tag or preset inserted from the palette -- a discrete edit, unlike typing. */
onInsert: (value: string) => void;
onFocus: () => void;
active?: boolean;
label?: string;
}
/**
* One spoken line: a textarea plus, once it contains tags, a styled preview.
* The shared palette inserts at the cursor through the ref. Undo/redo isn't
* here -- the shot list editor keeps one history for everything in it.
*/
export const H3ShotDialogueField = React.forwardRef<H3ShotDialogueHandle, H3ShotDialogueFieldProps>(
function H3ShotDialogueField({ value, onChange, onInsert, onFocus, active, label = 'Dialogue' }, ref) {
const taRef = React.useRef<HTMLTextAreaElement>(null);
// Selection to restore once the inserted text has rendered.
const pending = React.useRef<[number, number] | null>(null);
React.useEffect(() => {
if (pending.current && taRef.current) {
const [s, e] = pending.current;
taRef.current.focus();
taRef.current.setSelectionRange(s, e);
pending.current = null;
}
});
function selection(): [number, number] {
const ta = taRef.current;
return [ta?.selectionStart ?? value.length, ta?.selectionEnd ?? value.length];
}
React.useImperativeHandle(ref, () => ({
insertTag(tag) {
const [s, e] = selection();
const result = applyH3Tag(value, s, e, tag);
pending.current = [result.start, result.end];
onInsert(result.value);
},
applyPreset(preset) {
const [s, e] = selection();
const result = applyH3Preset(value, s, e, preset);
pending.current = [result.start, result.end];
onInsert(result.value);
},
}));
return (
<div className="flex flex-col gap-1">
<div className="text-xs text-muted-foreground">
{label}
{active && <span className="ml-2 text-accent-blue">tags insert here</span>}
</div>
<Textarea
ref={taRef}
value={value}
onChange={(e) => onChange(e.target.value)}
onFocus={onFocus}
rows={2}
spellCheck={false}
placeholder="What the speaker says"
className="font-mono text-sm"
/>
{value.includes('<') && (
<H3DialoguePreview
text={value}
className="rounded-lg border border-border bg-card px-3 py-2 text-sm leading-7 text-foreground"
/>
)}
</div>
);
},
);
@@ -0,0 +1,690 @@
'use client';
import * as React from 'react';
import { Button } from '@/components/ui/button';
import {
Dialog,
DialogContent,
DialogDescription,
DialogFooter,
DialogHeader,
DialogTitle,
} from '@/components/ui/dialog';
import { Input } from '@/components/ui/input';
import { NativeSelect } from '@/components/ui/native-select';
import { Textarea } from '@/components/ui/textarea';
import { H3ShotDialogueField, type H3ShotDialogueHandle } from '@/components/jobs/H3ShotDialogueField';
import { H3ScriptImportDialog } from '@/components/jobs/H3ScriptImportDialog';
import { H3ShotStoryboard } from '@/components/jobs/H3ShotStoryboard';
import { H3TagPalette } from '@/components/jobs/H3TagPalette';
import type { H3Preset, H3Tag } from '@/lib/h3Tags';
import {
CAMERA_MOVEMENTS,
SHOT_TYPES,
detailedDescriptionFromShots,
formatCutTime,
linePart,
newLine,
newShot,
newTextPart,
parseCutTime,
parseDetailedDescription,
parseSubjectLabels,
resolveShotStarts,
retentionAnalysisFromShots,
shotLines,
withFreshIds,
type CameraMovement,
type DialogueLine,
type LinePart,
type ShotPart,
type Shot,
type ShotType,
} from '@/lib/h3Shots';
import { useUndoableState } from '@/hooks/useUndoableState';
import { cn } from '@/lib/utils';
export interface H3ShotListEditorProps {
open: boolean;
/** Raw subject_definitions text -- parsed for <Subject N> labels to track per shot. */
subjectDefinitions: string;
initialDetailedDescription: string;
initialRetentionAnalysis: string;
onApply: (detailedDescription: string, retentionAnalysis: string) => void;
onClose: () => void;
/** Where the shots get written, for the description text. */
targetLabel?: string;
/** False when there is no retention analysis to keep in sync (base-format and plain prompts). */
withRetention?: boolean;
}
/**
* Structured camera-direction editor for MiniMax-H3: an ordered shot list
* (type, movement, which subjects appear, action, optional dialogue) that
* writes into detailed_description's [Shot N] blocks and keeps
* retention_analysis's "appears in [...]" lines in sync automatically,
* instead of hand-maintaining both fields' free text in agreement.
*/
/** Cut-in time field: blank means "automatic" (shown as the placeholder); Enter or blur commits. */
function CutTimeInput({
value,
auto,
disabled,
onCommit,
}: {
value: number | null;
auto: number;
disabled?: boolean;
onCommit: (seconds: number | null) => void;
}) {
const show = (v: number | null) => (v == null ? '' : formatCutTime(v));
const [text, setText] = React.useState(show(value));
React.useEffect(() => setText(show(value)), [value]);
function commit() {
const trimmed = text.trim();
if (!trimmed) return onCommit(null);
const parsed = parseCutTime(trimmed);
if (parsed == null) return setText(show(value));
onCommit(parsed);
}
return (
<Input
value={text}
onChange={(e) => setText(e.target.value)}
onBlur={commit}
onKeyDown={(e) => {
if (e.key === 'Enter') {
e.preventDefault();
commit();
}
}}
placeholder={disabled ? 'Opening shot' : formatCutTime(auto)}
disabled={disabled}
inputMode="decimal"
className="font-mono"
/>
);
}
interface EditorState {
shots: Shot[];
preamble: string;
}
export function H3ShotListEditor({
open,
subjectDefinitions,
initialDetailedDescription,
initialRetentionAnalysis,
onApply,
onClose,
targetLabel = 'Detailed description',
withRetention = true,
}: H3ShotListEditorProps) {
// One history covers everything editable here: the shots (structure, fields and
// every line of text) and the scene notes. Typing carries a key so a burst
// merges into one step; everything else is a step of its own.
const history = useUndoableState<EditorState>({ shots: [], preamble: '' });
const { shots, preamble } = history.state;
// Returning the same list (or text) hands the history back the same state object,
// which it treats as "nothing changed" and does not record.
const editShots = (fn: (shots: Shot[]) => Shot[], key: string | null = null) =>
history.set((s) => {
const next = fn(s.shots);
return next === s.shots ? s : { ...s, shots: next };
}, key);
const setPreamble = (value: string) =>
history.set((s) => (s.preamble === value ? s : { ...s, preamble: value }), 'preamble');
const [selectedId, setSelectedId] = React.useState<string | null>(null);
// The shared tag palette inserts into whichever shot's dialogue was last focused.
const [dialogueTargetId, setDialogueTargetId] = React.useState<string | null>(null);
const dialogueFields = React.useRef(new Map<string, H3ShotDialogueHandle>());
const [importOpen, setImportOpen] = React.useState(false);
const subjectLabels = React.useMemo(() => parseSubjectLabels(subjectDefinitions), [subjectDefinitions]);
// Re-seed from the field's current text each time the editor opens, the
// same "seed on the open transition only" guard H3DialogueEditor uses --
// otherwise a stray parent re-render while open could clobber in-progress edits.
const wasOpenRef = React.useRef(false);
React.useEffect(() => {
if (open && !wasOpenRef.current) {
const { preamble: seededPreamble, shots: seeded } = parseDetailedDescription(
initialDetailedDescription,
subjectLabels,
);
history.reset({ preamble: seededPreamble, shots: seeded });
setSelectedId(seeded[0]?.id ?? null);
setDialogueTargetId(null);
}
wasOpenRef.current = open;
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [open]);
function addShot() {
const shot = newShot();
editShots((prev) => [...prev, shot]);
selectShot(shot.id);
}
// Storyboard panel -> jump to that shot's editor card.
function selectShot(id: string) {
setSelectedId(id);
requestAnimationFrame(() =>
document.getElementById(`shot-card-${id}`)?.scrollIntoView({ behavior: 'smooth', block: 'nearest' }),
);
}
function removeShot(id: string) {
editShots((prev) => prev.filter((s) => s.id !== id));
setDialogueTargetId((cur) => (cur === id ? null : cur));
}
function insertTag(tag: H3Tag) {
if (dialogueTargetId) dialogueFields.current.get(dialogueTargetId)?.insertTag(tag);
}
function applyPreset(preset: H3Preset) {
if (dialogueTargetId) dialogueFields.current.get(dialogueTargetId)?.applyPreset(preset);
}
const partKey = (p: ShotPart) => (p.kind === 'text' ? p.id : p.line.id);
// An edit that changes nothing returns the same list and so records no undo step
// (a blur that re-commits the same value, for instance).
function updateBody(shotId: string, fn: (body: ShotPart[]) => ShotPart[], editKey: string | null = null) {
editShots((prev) => {
let changed = false;
const next = prev.map((s) => {
if (s.id !== shotId) return s;
const body = fn(s.body);
if (body === s.body) return s;
changed = true;
return { ...s, body };
});
return changed ? next : prev;
}, editKey);
}
function addLine(shotId: string) {
updateBody(shotId, (body) => {
const lastSpeaker = [...body].reverse().find((p): p is LinePart => p.kind === 'line')?.line.speaker;
return [...body, linePart(newLine({ speaker: lastSpeaker ?? 'S1' }))];
});
}
function addText(shotId: string) {
updateBody(shotId, (body) => [...body, newTextPart()]);
}
function removePart(shotId: string, key: string) {
updateBody(shotId, (body) => body.filter((p) => partKey(p) !== key));
setDialogueTargetId((cur) => (cur === key ? null : cur));
}
function movePart(shotId: string, key: string, dir: -1 | 1) {
updateBody(shotId, (body) => {
const i = body.findIndex((p) => partKey(p) === key);
const j = i + dir;
if (i < 0 || j < 0 || j >= body.length) return body;
const next = [...body];
[next[i], next[j]] = [next[j], next[i]];
return next;
});
}
function updateText(shotId: string, key: string, text: string) {
updateBody(
shotId,
(body) => {
const part = body.find((p) => p.kind === 'text' && p.id === key);
if (!part || (part.kind === 'text' && part.text === text)) return body;
return body.map((p) => (p === part ? { ...part, text } : p));
},
`text:${key}`,
);
}
/** `editKey` groups typing into one undo step; omit it for a discrete edit like a tag insertion. */
function updateLine(shotId: string, lineId: string, patch: Partial<DialogueLine>, editKey?: string) {
updateBody(
shotId,
(body) => {
const part = body.find((p) => p.kind === 'line' && p.line.id === lineId);
if (!part || part.kind !== 'line') return body;
const unchanged = (Object.keys(patch) as (keyof DialogueLine)[]).every((k) => part.line[k] === patch[k]);
return unchanged ? body : body.map((p) => (p === part ? { ...part, line: { ...part.line, ...patch } } : p));
},
editKey ?? null,
);
}
const undo = () => void history.undo();
const redo = () => void history.redo();
// Undo/redo for the whole list, from anywhere inside it -- including from a textbox,
// where the browser's own undo would only know about that one box. Not while the
// script import (nested inside this dialog) is open: its box keeps native undo.
function handleKeyDown(e: React.KeyboardEvent) {
if (importOpen || e.altKey || !(e.metaKey || e.ctrlKey)) return;
const k = e.key.toLowerCase();
if (k === 'z') {
e.preventDefault();
if (e.shiftKey) redo();
else undo();
} else if (k === 'y' && !e.shiftKey) {
e.preventDefault();
redo();
}
}
// An imported clip replaces the list wholesale, as one undoable step.
function loadImportedShots(imported: Shot[]) {
const next = withFreshIds(imported);
editShots(() => next);
setSelectedId(next[0]?.id ?? null);
setDialogueTargetId(null);
}
const dialogueTarget = (() => {
for (let i = 0; i < shots.length; i += 1) {
const lines = shotLines(shots[i]);
const li = lines.findIndex((l) => l.id === dialogueTargetId);
if (li >= 0) return { shot: i + 1, line: li + 1, multi: lines.length > 1 };
}
return null;
})();
function moveShot(id: string, dir: -1 | 1) {
editShots((prev) => {
const i = prev.findIndex((s) => s.id === id);
const j = i + dir;
if (i < 0 || j < 0 || j >= prev.length) return prev;
const next = [...prev];
[next[i], next[j]] = [next[j], next[i]];
return next;
});
}
function updateShot(id: string, patch: Partial<Shot>) {
editShots((prev) => {
const shot = prev.find((s) => s.id === id);
const unchanged = shot && (Object.keys(patch) as (keyof Shot)[]).every((k) => shot[k] === patch[k]);
return !shot || unchanged ? prev : prev.map((s) => (s === shot ? { ...s, ...patch } : s));
});
}
function toggleSubject(id: string, label: string) {
editShots((prev) =>
prev.map((s) =>
s.id === id
? {
...s,
subjectIds: s.subjectIds.includes(label)
? s.subjectIds.filter((l) => l !== label)
: [...s.subjectIds, label],
}
: s,
),
);
}
const starts = React.useMemo(() => resolveShotStarts(shots), [shots]);
const previewDetailedDescription = React.useMemo(
() => detailedDescriptionFromShots(shots, preamble),
[shots, preamble],
);
const previewRetentionAnalysis = React.useMemo(
() => (withRetention ? retentionAnalysisFromShots(shots, subjectLabels, initialRetentionAnalysis) : ''),
[withRetention, shots, subjectLabels, initialRetentionAnalysis],
);
function handleApply() {
onApply(previewDetailedDescription, previewRetentionAnalysis);
onClose();
}
return (
<Dialog open={open} onOpenChange={(o) => { if (!o) onClose(); }}>
<DialogContent
onKeyDown={handleKeyDown}
className="flex h-[92vh] max-h-[920px] w-[96vw] max-w-[1100px] flex-col gap-0 overflow-hidden p-0"
>
<DialogHeader className="border-b border-border px-5 py-4">
<DialogTitle>Shot list</DialogTitle>
<DialogDescription>
Build the scene shot by shot. Applying writes [Shot N] blocks into {targetLabel}
{withRetention ? ' and keeps Retention analysis\u2019s \u201cappears in\u201d lines in sync' : ''}.
</DialogDescription>
</DialogHeader>
<div className="flex shrink-0 flex-col gap-2 border-b border-border bg-secondary/10 px-4 py-3">
<div className="flex items-center justify-between">
<span className="text-xs font-semibold text-foreground">Storyboard</span>
<span className="flex items-center gap-1.5">
<Button
type="button"
variant="outline"
size="sm"
onClick={undo}
disabled={!history.canUndo}
title="Undo (Ctrl/Cmd+Z)"
>
↶ Undo
</Button>
<Button
type="button"
variant="outline"
size="sm"
onClick={redo}
disabled={!history.canRedo}
title="Redo (Ctrl/Cmd+Shift+Z)"
>
↷ Redo
</Button>
<Button type="button" variant="outline" size="sm" onClick={() => setImportOpen(true)}>
Import script…
</Button>
</span>
</div>
<H3ShotStoryboard shots={shots} selectedId={selectedId} onSelect={selectShot} onAdd={addShot} />
</div>
<div className="flex min-h-0 flex-1 flex-col md:flex-row">
<div className="flex min-h-0 min-w-0 flex-1 flex-col gap-4 overflow-y-auto p-4">
<label className="flex flex-col gap-1 text-xs text-muted-foreground">
Scene notes (optional, written before Shot 1 -- style, lighting, anything true of the whole clip)
<Textarea
value={preamble}
onChange={(e) => setPreamble(e.target.value)}
rows={2}
placeholder="e.g. Live-action, cinematic, shallow depth of field."
/>
</label>
{shots.length === 0 && (
<p className="rounded-xl border border-dashed border-border p-4 text-sm text-muted-foreground">
No shots yet.
</p>
)}
{shots.map((shot, i) => (
<div
key={shot.id}
id={`shot-card-${shot.id}`}
onFocusCapture={() => setSelectedId(shot.id)}
className={cn(
'flex flex-col gap-3 rounded-xl border bg-secondary/20 p-3',
shot.id === selectedId ? 'border-accent-blue' : 'border-border',
)}
>
<div className="flex items-center justify-between">
<span className="text-sm font-semibold text-foreground">
Shot {i + 1}
{i > 0 && (
<span className="ml-2 font-mono text-xs font-normal text-muted-foreground">
cuts in at {formatCutTime(starts[i])}
</span>
)}
</span>
<div className="flex items-center gap-1">
<Button
type="button"
variant="outline"
size="sm"
onClick={() => moveShot(shot.id, -1)}
disabled={i === 0}
title="Move up"
>
↑
</Button>
<Button
type="button"
variant="outline"
size="sm"
onClick={() => moveShot(shot.id, 1)}
disabled={i === shots.length - 1}
title="Move down"
>
↓
</Button>
<Button type="button" variant="outline" size="sm" onClick={() => removeShot(shot.id)}>
Remove
</Button>
</div>
</div>
<div className="grid grid-cols-1 gap-2 sm:grid-cols-3">
<label className="flex flex-col gap-1 text-xs text-muted-foreground">
Shot type
<NativeSelect
value={shot.type}
onChange={(e) => updateShot(shot.id, { type: e.target.value as ShotType, freeLead: false })}
>
{SHOT_TYPES.map((t) => (
<option key={t.id} value={t.id}>{t.label}</option>
))}
</NativeSelect>
</label>
<label className="flex flex-col gap-1 text-xs text-muted-foreground">
Camera movement
<NativeSelect
value={shot.movement}
onChange={(e) => updateShot(shot.id, { movement: e.target.value as CameraMovement, freeLead: false })}
>
{CAMERA_MOVEMENTS.map((m) => (
<option key={m.id} value={m.id}>{m.label}</option>
))}
</NativeSelect>
</label>
<label className="flex flex-col gap-1 text-xs text-muted-foreground">
Cut time (blank = automatic)
<CutTimeInput
value={shot.startSeconds}
auto={starts[i]}
disabled={i === 0}
onCommit={(startSeconds) => updateShot(shot.id, { startSeconds })}
/>
</label>
</div>
{shot.freeLead && (
<p className="text-xs text-muted-foreground">
This shot keeps its original wording, so type and movement aren&apos;t written. Changing either
adds a generated lead-in in front of the text below.
</p>
)}
{subjectLabels.length > 0 && (
<div className="flex flex-col gap-1">
<span className="text-xs text-muted-foreground">Subjects in this shot</span>
<div className="flex flex-wrap gap-1.5">
{subjectLabels.map((label) => {
const active = shot.subjectIds.includes(label);
return (
<button
key={label}
type="button"
aria-pressed={active}
onClick={() => toggleSubject(shot.id, label)}
className={cn(
'rounded-full border px-2 py-0.5 font-mono text-[11px] transition-colors',
active
? 'border-accent-blue bg-accent-blue/15 text-accent-blue'
: 'border-input bg-card text-muted-foreground hover:bg-accent',
)}
>
{label}
</button>
);
})}
</div>
</div>
)}
<div className="flex flex-col gap-2">
<span className="text-xs text-muted-foreground">
Description and dialogue, in the order they play. Dialogue is wrapped in &lt;d&gt; on apply.
</span>
{(() => {
let lineNo = 0;
let textNo = 0;
return shot.body.map((part, bi) => {
const key = partKey(part);
const noun = part.kind === 'line' ? `line ${(lineNo += 1)}` : `text ${(textNo += 1)}`;
const controls = (
<div className="mt-6 flex shrink-0 gap-1">
<Button
type="button"
variant="outline"
size="sm"
onClick={() => movePart(shot.id, key, -1)}
disabled={bi === 0}
aria-label={`Move ${noun} up`}
title="Move up"
>
↑
</Button>
<Button
type="button"
variant="outline"
size="sm"
onClick={() => movePart(shot.id, key, 1)}
disabled={bi === shot.body.length - 1}
aria-label={`Move ${noun} down`}
title="Move down"
>
↓
</Button>
<Button
type="button"
variant="outline"
size="sm"
onClick={() => removePart(shot.id, key)}
aria-label={`Remove ${noun}`}
title="Remove"
>
×
</Button>
</div>
);
if (part.kind === 'text') {
return (
<div key={key} className="flex items-start gap-2">
<label className="flex min-w-0 flex-1 flex-col gap-1 text-xs text-muted-foreground">
{textNo === 1 ? 'Description (composition, action, environment)' : `Text ${textNo}`}
<Textarea
value={part.text}
onChange={(e) => updateText(shot.id, key, e.target.value)}
rows={2}
placeholder="What's happening in this shot"
/>
</label>
{controls}
</div>
);
}
const line = part.line;
return (
<div key={key} className="flex items-start gap-2">
<Input
value={line.speaker}
onChange={(e) => updateLine(shot.id, line.id, { speaker: e.target.value }, `speaker:${line.id}`)}
placeholder="S1"
aria-label={`Line ${lineNo} speaker`}
className="mt-7 w-16 shrink-0 font-mono"
/>
<div className="min-w-0 flex-1">
<H3ShotDialogueField
ref={(handle) => {
if (handle) dialogueFields.current.set(line.id, handle);
else dialogueFields.current.delete(line.id);
}}
value={line.text}
onChange={(text) => updateLine(shot.id, line.id, { text }, `line:${line.id}`)}
onInsert={(text) => updateLine(shot.id, line.id, { text })}
onFocus={() => setDialogueTargetId(line.id)}
active={line.id === dialogueTargetId}
label={`Line ${lineNo}`}
/>
</div>
{controls}
</div>
);
});
})()}
<div className="flex gap-2">
<Button type="button" variant="outline" size="sm" onClick={() => addText(shot.id)}>
+ Add text
</Button>
<Button type="button" variant="outline" size="sm" onClick={() => addLine(shot.id)}>
+ Add line
</Button>
</div>
</div>
</div>
))}
<Button type="button" variant="outline" onClick={addShot} className="self-start">
+ Add shot
</Button>
<div className="flex flex-col gap-2 rounded-xl border border-border bg-card p-3">
<span className="text-xs font-semibold text-foreground">Preview</span>
<div className="flex flex-col gap-1">
<span className="text-[11px] font-medium uppercase tracking-wide text-muted-foreground">
{targetLabel}
</span>
<pre className="whitespace-pre-wrap rounded-lg bg-secondary/30 p-2 font-mono text-[11px] text-foreground">
{previewDetailedDescription || '—'}
</pre>
</div>
{withRetention && (
<div className="flex flex-col gap-1">
<span className="text-[11px] font-medium uppercase tracking-wide text-muted-foreground">
Retention analysis
</span>
<pre className="whitespace-pre-wrap rounded-lg bg-secondary/30 p-2 font-mono text-[11px] text-foreground">
{previewRetentionAnalysis || '—'}
</pre>
</div>
)}
</div>
</div>
<aside className="flex max-h-56 shrink-0 flex-col border-t border-border md:max-h-none md:w-64 md:border-l md:border-t-0">
<p className="border-b border-border px-3 py-2 text-xs text-muted-foreground">
{dialogueTarget
? `Tags insert into Shot ${dialogueTarget.shot}${dialogueTarget.multi ? `, line ${dialogueTarget.line}` : ''} dialogue`
: 'Click into a shot\'s dialogue to insert tags'}
</p>
<H3TagPalette
onInsertTag={insertTag}
onApplyPreset={applyPreset}
disabled={!dialogueTarget}
className="overflow-y-auto p-3"
/>
</aside>
</div>
<DialogFooter className="border-t border-border px-5 py-3">
<Button type="button" variant="outline" onClick={onClose}>Cancel</Button>
<Button type="button" onClick={handleApply}>Apply</Button>
</DialogFooter>
{/* Nested so Radix treats it as part of this dialog (see CreateJobModal). */}
<H3ScriptImportDialog
open={importOpen}
subjectLabels={subjectLabels}
existingShotCount={shots.length}
onLoad={loadImportedShots}
onClose={() => setImportOpen(false)}
/>
</DialogContent>
</Dialog>
);
}
@@ -0,0 +1,214 @@
'use client';
import * as React from 'react';
import {
CAMERA_MOVEMENTS,
SHOT_TYPES,
shotLines,
shotProse,
type CameraMovement,
type Shot,
type ShotType,
} from '@/lib/h3Shots';
const TYPE_LABEL = Object.fromEntries(SHOT_TYPES.map((t) => [t.id, t.label])) as Record<ShotType, string>;
const MOVEMENT_LABEL = Object.fromEntries(CAMERA_MOVEMENTS.map((m) => [m.id, m.label])) as Record<
CameraMovement,
string
>;
// Schematic framing per shot type: `scale` sizes the figures and `baseY` is
// where their shoulders bottom out in the 160x90 frame, so a wide shot reads
// as small distant figures and a close-up as a head filling the frame.
const FRAMING: Partial<Record<ShotType, { scale: number; baseY: number }>> = {
establishing: { scale: 0.45, baseY: 72 },
wide: { scale: 0.8, baseY: 78 },
medium: { scale: 1.5, baseY: 92 },
'close-up': { scale: 2.6, baseY: 118 },
};
function figureXs(count: number): number[] {
if (count <= 1) return [80];
if (count === 2) return [55, 105];
return [38, 80, 122];
}
function Figure({ x, baseY, scale, opacity }: { x: number; baseY: number; scale: number; opacity: number }) {
return (
<g fill="currentColor" opacity={opacity}>
<circle cx={x} cy={baseY - 30 * scale} r={8 * scale} />
<ellipse cx={x} cy={baseY - 12 * scale} rx={14 * scale} ry={12 * scale} />
</g>
);
}
export function ShotFrame({ shot }: { shot: Shot }) {
const count = Math.min(Math.max(shot.subjectIds.length, 1), 3);
const empty = shot.subjectIds.length === 0;
const opacity = empty ? 0.25 : 0.6;
const framing = FRAMING[shot.type];
let scene: React.ReactNode;
if (shot.type === 'extreme-close-up') {
scene = (
<g fill="currentColor" opacity={opacity}>
<circle cx={80} cy={52} r={62} />
<ellipse cx={60} cy={42} rx={7} ry={4} className="fill-background" />
<ellipse cx={100} cy={42} rx={7} ry={4} className="fill-background" />
</g>
);
} else if (shot.type === 'over-the-shoulder') {
scene = (
<>
<Figure x={22} baseY={135} scale={3} opacity={0.85} />
<Figure x={116} baseY={80} scale={1} opacity={opacity} />
</>
);
} else if (shot.type === 'pov') {
scene = (
<>
<Figure x={80} baseY={72} scale={0.9} opacity={opacity} />
<path
d="M0 0H160V90H0Z M80 45m-52 0a52 40 0 1 0 104 0a52 40 0 1 0 -104 0"
fill="currentColor"
fillRule="evenodd"
opacity={0.3}
/>
<path d="M74 45H86M80 39V51" stroke="currentColor" strokeWidth={1.5} opacity={0.6} />
</>
);
} else if (shot.type === 'insert') {
scene = (
<g opacity={0.6} fill="none" stroke="currentColor" strokeWidth={2.5}>
<rect x={52} y={16} width={56} height={58} rx={8} />
<path d="M64 34H96M64 46H96M64 58H84" strokeLinecap="round" />
</g>
);
} else if (framing) {
scene = (
<>
{shot.type === 'establishing' && (
<path d="M0 66H160" stroke="currentColor" strokeWidth={1} opacity={0.35} />
)}
{figureXs(count).map((x, i) => (
<Figure
key={x}
x={shot.type === 'close-up' ? 80 : x}
baseY={framing.baseY}
scale={framing.scale}
opacity={opacity * (i === 0 ? 1 : 0.75)}
/>
))}
</>
);
}
return (
<svg viewBox="0 0 160 90" className="block h-full w-full text-foreground" aria-hidden>
{scene}
<MovementOverlay movement={shot.movement} />
</svg>
);
}
function MovementOverlay({ movement }: { movement: CameraMovement }) {
const g = (children: React.ReactNode) => (
<g
className="text-accent-blue"
stroke="currentColor"
fill="none"
strokeWidth={2.5}
strokeLinecap="round"
strokeLinejoin="round"
>
{children}
</g>
);
switch (movement) {
case 'pan':
return g(<path d="M28 80H132M28 80l6-4M28 80l6 4M132 80l-6-4M132 80l-6 4" />);
case 'tilt':
return g(<path d="M146 18V72M146 18l-4 6M146 18l4 6M146 72l-4-6M146 72l4-6" />);
case 'dolly':
return g(<path d="M10 86L54 56M150 86L106 56M10 86H150" strokeDasharray="4 4" />);
case 'zoom':
return g(<path d="M16 26V14H28M132 14H144V26M16 64V76H28M132 76H144V64" />);
case 'handheld':
return g(
<rect x={7} y={7} width={146} height={76} rx={4} strokeDasharray="3 4" transform="rotate(-1.6 80 45)" />,
);
case 'tracking':
return g(
<>
<path d="M22 82H134M134 82l-6-4M134 82l-6 4" />
<circle cx={30} cy={82} r={3} fill="currentColor" />
</>,
);
case 'crane':
return g(<path d="M24 80L134 20M134 20h-9M134 20l-3 8" />);
default:
return null;
}
}
function subjectShortLabel(label: string): string {
const m = label.match(/(\d+)/);
return m ? `Subj ${m[1]}` : label;
}
export interface H3ShotPanelProps {
shot: Shot;
/** the number shown in the badge */
number: number | string;
/** time printed on the frame, e.g. "00:03.500" */
timeLabel: string;
}
/**
* The contents of one storyboard panel: badge, shot type and camera move, a
* schematic frame, who is in it, what is said, and the shot's prose. Callers
* supply the surrounding card (a button in the editor, plain in the scene view).
*/
export function H3ShotPanel({ shot, number, timeLabel }: H3ShotPanelProps) {
const spoken = shotLines(shot).filter((l) => l.text.trim());
return (
<>
<div className="flex items-center gap-1.5">
<span className="flex size-5 shrink-0 items-center justify-center rounded-full bg-secondary text-[11px] font-semibold text-foreground">
{number}
</span>
<span className="truncate text-xs font-medium text-foreground">{TYPE_LABEL[shot.type]}</span>
<span className="ml-auto shrink-0 text-[10px] text-muted-foreground">{MOVEMENT_LABEL[shot.movement]}</span>
</div>
<div className="relative aspect-video overflow-hidden rounded-md border border-border bg-secondary/40">
<ShotFrame shot={shot} />
<span className="absolute bottom-1 left-1 rounded bg-background/80 px-1 font-mono text-[9px] text-muted-foreground">
{timeLabel}
</span>
</div>
{shot.subjectIds.length > 0 && (
<div className="flex flex-wrap gap-1">
{shot.subjectIds.map((label) => (
<span
key={label}
className="rounded-full border border-input bg-secondary/50 px-1.5 py-px font-mono text-[10px] text-muted-foreground"
>
{subjectShortLabel(label)}
</span>
))}
</div>
)}
{spoken.length > 0 && (
<p className="line-clamp-3 rounded-md bg-accent-blue/10 px-1.5 py-1 text-[11px] italic text-foreground">
{spoken.map((l) => `${l.speaker.trim() ? `${l.speaker.trim()}: ` : ''}“${l.text.trim()}”`).join(' ')}
</p>
)}
<p className="line-clamp-2 text-[11px] text-muted-foreground">{shotProse(shot) || 'No description yet'}</p>
</>
);
}
@@ -0,0 +1,51 @@
import { render, screen } from '@testing-library/react';
import userEvent from '@testing-library/user-event';
import { describe, expect, it, vi } from 'vitest';
import { H3ShotStoryboard } from './H3ShotStoryboard';
import { CAMERA_MOVEMENTS, SHOT_TYPES, makeShot, newLine, type Shot, type ShotInit } from '@/lib/h3Shots';
function shot(init: ShotInit): Shot {
return makeShot(init);
}
describe('H3ShotStoryboard', () => {
it('lays shots out in order with type, movement, subjects, and dialogue', () => {
const shots = [
shot({ type: 'establishing', description: 'Wide rooftop.', subjectIds: ['<Subject 1>', '<Subject 2>'] }),
shot({ type: 'close-up', movement: 'handheld', lines: [newLine({ speaker: 'S2', text: 'I had no choice.' }), newLine({ speaker: 'S1', text: 'Then go.' })], subjectIds: ['<Subject 2>'] }),
];
render(<H3ShotStoryboard shots={shots} selectedId={shots[1].id} onSelect={() => {}} onAdd={() => {}} />);
expect(screen.getByText('Establishing')).toBeInTheDocument();
expect(screen.getByText('Close-up')).toBeInTheDocument();
expect(screen.getByText('Handheld')).toBeInTheDocument();
expect(screen.getByText(/S2: “I had no choice.” S1: “Then go.”/)).toBeInTheDocument();
expect(screen.getAllByText('Subj 2')).toHaveLength(2);
const panels = screen.getAllByRole('button').filter((b) => b.hasAttribute('aria-current') || b.textContent?.match(/^\d/));
expect(panels[0]).toHaveTextContent(/^1Establishing/);
expect(screen.getByRole('button', { name: /Close-up/ })).toHaveAttribute('aria-current', 'true');
});
it('selects a shot on click and adds via the trailing tile', async () => {
const user = userEvent.setup();
const onSelect = vi.fn();
const onAdd = vi.fn();
const shots = [shot({ description: 'First.' }), shot({ description: 'Second.' })];
render(<H3ShotStoryboard shots={shots} selectedId={null} onSelect={onSelect} onAdd={onAdd} />);
await user.click(screen.getByRole('button', { name: /Second\./ }));
expect(onSelect).toHaveBeenCalledWith(shots[1].id);
await user.click(screen.getByRole('button', { name: /Add shot/ }));
expect(onAdd).toHaveBeenCalled();
});
it('renders every shot type and camera movement without crashing', () => {
const shots = SHOT_TYPES.flatMap((t) =>
CAMERA_MOVEMENTS.map((m) => shot({ type: t.id, movement: m.id, subjectIds: ['<Subject 1>'] })),
);
const { container } = render(<H3ShotStoryboard shots={shots} selectedId={null} onSelect={() => {}} onAdd={() => {}} />);
expect(container.querySelectorAll('svg')).toHaveLength(SHOT_TYPES.length * CAMERA_MOVEMENTS.length);
});
});
@@ -0,0 +1,61 @@
'use client';
import * as React from 'react';
import { H3ShotPanel } from '@/components/jobs/H3ShotPanel';
import { formatCutTime, resolveShotStarts, type Shot } from '@/lib/h3Shots';
import { cn } from '@/lib/utils';
export interface H3ShotStoryboardProps {
shots: Shot[];
selectedId: string | null;
onSelect: (id: string) => void;
onAdd: () => void;
}
/**
* The shot list as a storyboard: one panel per shot, left to right in
* playback order. Panels are schematic (no generated imagery exists yet) --
* subject size reflects shot type and an overlay marks camera movement --
* so the sequence, who's in frame, and where the dialogue falls read at a
* glance without scrolling the per-shot editors.
*/
export function H3ShotStoryboard({ shots, selectedId, onSelect, onAdd }: H3ShotStoryboardProps) {
const starts = React.useMemo(() => resolveShotStarts(shots), [shots]);
return (
<div className="flex items-stretch gap-2 overflow-x-auto pb-1" aria-label="Storyboard">
{shots.map((shot, i) => {
const selected = shot.id === selectedId;
return (
<React.Fragment key={shot.id}>
{i > 0 && (
<span className="flex shrink-0 items-center text-muted-foreground/60" aria-hidden>
›
</span>
)}
<button
type="button"
onClick={() => onSelect(shot.id)}
aria-current={selected ? 'true' : undefined}
className={cn(
'flex w-52 shrink-0 flex-col gap-1.5 rounded-xl border bg-card p-2 text-left transition-colors hover:bg-accent/40',
selected ? 'border-accent-blue ring-2 ring-accent-blue/50' : 'border-border',
)}
>
<H3ShotPanel shot={shot} number={i + 1} timeLabel={formatCutTime(starts[i])} />
</button>
</React.Fragment>
);
})}
<button
type="button"
onClick={onAdd}
className="flex min-h-32 w-28 shrink-0 flex-col items-center justify-center gap-1 rounded-xl border border-dashed border-border text-xs text-muted-foreground transition-colors hover:bg-accent/40 hover:text-foreground"
>
<span className="text-lg leading-none">+</span>
Add shot
</button>
</div>
);
}
@@ -0,0 +1,57 @@
'use client';
import * as React from 'react';
import { H3_TAG_CATEGORIES, type H3Tag } from '@/lib/h3Tags';
import { cn } from '@/lib/utils';
export interface H3TagConsoleProps {
/** Called with the tag to insert into whichever field is currently focused. */
onInsert: (tag: H3Tag) => void;
disabled?: boolean;
className?: string;
}
/**
* A microexpression "console" for MiniMax-H3 speech tags: click a tag to
* insert it at the cursor of whichever prompt field was last focused. Kept
* as plain buttons (not a menu) so every tag stays one click away.
*/
export function H3TagConsole({ onInsert, disabled, className }: H3TagConsoleProps) {
return (
<div
className={cn(
'flex flex-col gap-1.5 rounded-xl border border-input bg-secondary/40 p-2',
className,
)}
>
<span className="px-0.5 text-[11px] font-medium uppercase tracking-wide text-muted-foreground">
Speech tags — click to insert at the cursor
</span>
<div className="flex flex-col gap-1.5">
{H3_TAG_CATEGORIES.map((category) => (
<div key={category.id} className="flex flex-wrap items-center gap-1">
<span className="mr-0.5 w-full shrink-0 text-[10px] text-muted-foreground/80 sm:w-auto">
{category.label}
</span>
{category.tags.map((tag) => (
<button
key={tag.id}
type="button"
title={`${tag.description} · e.g. "${tag.example}"`}
// Keep the target textarea focused (and its selection intact)
// so the click lands where the user was typing, not nowhere.
onMouseDown={(e) => e.preventDefault()}
onClick={() => onInsert(tag)}
disabled={disabled}
className="rounded-full border border-input bg-card px-2 py-0.5 font-mono text-[11px] text-foreground transition-colors hover:bg-accent disabled:cursor-not-allowed disabled:opacity-50"
>
{tag.label}
</button>
))}
</div>
))}
</div>
</div>
);
}
@@ -0,0 +1,74 @@
'use client';
import * as React from 'react';
import { H3_PRESETS, H3_TAG_CATEGORIES, type H3Preset, type H3Tag } from '@/lib/h3Tags';
import { cn } from '@/lib/utils';
// Reuses the app's existing status hues as a purely visual category key --
// no status meaning implied, just four distinguishable accents that already
// work in both themes.
export const CATEGORY_STYLE: Record<string, { chip: string; dot: string }> = {
pace: { chip: 'border-blue-400/35 bg-blue-500/10 text-blue-900 dark:text-blue-100', dot: 'bg-blue-500' },
delivery: { chip: 'border-amber-400/35 bg-amber-500/10 text-amber-900 dark:text-amber-100', dot: 'bg-amber-500' },
vocal: { chip: 'border-emerald-400/35 bg-emerald-500/10 text-emerald-900 dark:text-emerald-100', dot: 'bg-emerald-500' },
body: { chip: 'border-rose-400/35 bg-rose-500/10 text-rose-900 dark:text-rose-100', dot: 'bg-rose-500' },
};
export interface H3TagPaletteProps {
onInsertTag: (tag: H3Tag) => void;
onApplyPreset: (preset: H3Preset) => void;
disabled?: boolean;
className?: string;
}
/** Full speech-tag palette + emotion presets. Buttons keep the target textarea focused so the click lands at its cursor. */
export function H3TagPalette({ onInsertTag, onApplyPreset, disabled, className }: H3TagPaletteProps) {
return (
<div className={className}>
{H3_TAG_CATEGORIES.map((cat) => {
const style = CATEGORY_STYLE[cat.id] ?? CATEGORY_STYLE.body;
return (
<div key={cat.id} className="mb-4">
<div className="mb-1.5 flex items-center gap-2 px-0.5">
<span className={cn('size-2 rounded-full', style.dot)} />
<span className="text-xs font-semibold text-foreground">{cat.label}</span>
</div>
<div className="flex flex-col gap-1">
{cat.tags.map((tag) => (
<button
key={tag.id}
type="button"
title={tag.example}
disabled={disabled}
onMouseDown={(e) => e.preventDefault()}
onClick={() => onInsertTag(tag)}
className="flex items-baseline justify-between gap-2 rounded-lg border border-input bg-card px-2.5 py-1.5 text-left transition-colors hover:bg-accent disabled:cursor-not-allowed disabled:opacity-50"
>
<span className="font-mono text-[11.5px] text-foreground">{tag.label}</span>
<span className="truncate text-[10.5px] text-muted-foreground">{tag.description}</span>
</button>
))}
</div>
</div>
);
})}
<div className="mb-1.5 mt-5 px-0.5 text-xs font-semibold text-foreground">Emotion presets</div>
<div className="flex flex-wrap gap-1.5">
{H3_PRESETS.map((p) => (
<button
key={p.label}
type="button"
disabled={disabled}
onMouseDown={(e) => e.preventDefault()}
onClick={() => onApplyPreset(p)}
className="rounded-full border border-input bg-card px-2.5 py-1 text-[11.5px] text-foreground transition-colors hover:bg-accent disabled:cursor-not-allowed disabled:opacity-50"
>
{p.label}
</button>
))}
</div>
</div>
);
}
@@ -5,7 +5,9 @@ import { beforeEach, describe, expect, it, vi } from 'vitest';
import JobCard from '@/components/jobs/JobCard';
import {
deleteJob,
dequeueJob,
downloadJobVideo,
queueJobs,
startJob,
stopJob,
} from '@/lib/api';
@@ -18,6 +20,8 @@ vi.mock('@/lib/api', () => ({
stopJob: vi.fn(),
deleteJob: vi.fn(),
downloadJobVideo: vi.fn(),
queueJobs: vi.fn(),
dequeueJob: vi.fn(),
}));
vi.mock('@/lib/utils', async (importOriginal) => {
@@ -47,6 +51,8 @@ beforeEach(() => {
setActiveJobId(null);
vi.mocked(startJob).mockResolvedValue({} as Job);
vi.mocked(stopJob).mockResolvedValue({} as Job);
vi.mocked(queueJobs).mockResolvedValue([]);
vi.mocked(dequeueJob).mockResolvedValue({} as Job);
vi.mocked(deleteJob).mockResolvedValue(undefined);
vi.mocked(downloadJobVideo).mockResolvedValue(new Blob());
vi.spyOn(window, 'confirm').mockReturnValue(true);
@@ -91,6 +97,46 @@ describe('JobCard', () => {
expect(onJobUpdated).toHaveBeenCalled();
});
it.each(['pending', 'failed', 'stopped'])('queues a %s job and notifies the parent', async (status) => {
const onJobUpdated = vi.fn();
render(<JobCard job={makeJob({ status })} onJobUpdated={onJobUpdated} />);
await userEvent.click(screen.getByRole('button', { name: 'Queue' }));
await waitFor(() => expect(queueJobs).toHaveBeenCalledWith(['job-1']));
expect(onJobUpdated).toHaveBeenCalled();
expect(startJob).not.toHaveBeenCalled();
});
it('reports a job that cannot be queued', async () => {
vi.mocked(queueJobs).mockRejectedValue(new Error('Jobs would wait on each other'));
render(<JobCard job={makeJob({ status: 'pending' })} />);
await userEvent.click(screen.getByRole('button', { name: 'Queue' }));
await waitFor(() => expect(window.alert).toHaveBeenCalledWith('Jobs would wait on each other'));
});
it('shows a queued job as queued, and takes it back out of the queue', async () => {
const onJobUpdated = vi.fn();
render(<JobCard job={makeJob({ status: 'queued' })} onJobUpdated={onJobUpdated} />);
expect(screen.getByText('queued')).toBeInTheDocument();
// nothing to start or queue again; it can't be edited either
for (const name of ['Start', 'Restart', 'Queue', 'Stop', 'Edit']) {
expect(screen.queryByRole('button', { name })).not.toBeInTheDocument();
}
expect(screen.getByRole('button', { name: 'View' })).toBeInTheDocument();
await userEvent.click(screen.getByRole('button', { name: 'Remove from queue' }));
await waitFor(() => expect(dequeueJob).toHaveBeenCalledWith('job-1'));
expect(onJobUpdated).toHaveBeenCalled();
});
it('does not offer to queue a running or completed job', () => {
for (const status of ['running', 'completed']) {
const { unmount } = render(<JobCard job={makeJob({ status, started_at: Date.now() })} />);
expect(screen.queryByRole('button', { name: 'Queue' })).not.toBeInTheDocument();
unmount();
}
});
it('stops a running job', async () => {
render(<JobCard job={makeJob({ status: 'running', started_at: Date.now() })} />);
await userEvent.click(screen.getByRole('button', { name: 'Stop' }));
@@ -10,7 +10,9 @@ import { useStore } from '@/hooks/useStore';
import {
deleteJob,
duplicateJob,
dequeueJob,
downloadJobVideo,
queueJobs,
startJob,
stopJob,
} from '@/lib/api';
@@ -51,6 +53,7 @@ function computeElapsed(job: Job, currentTime: number): string | null {
const BADGE_VARIANTS: Record<string, BadgeProps['variant']> = {
pending: 'secondary',
queued: 'default',
running: 'warning',
completed: 'success',
ready: 'success',
@@ -92,6 +95,36 @@ export default function JobCard({ job, onJobUpdated }: JobCardProps) {
}
}
async function handleQueue(e: React.MouseEvent) {
e.preventDefault();
e.stopPropagation();
if (isLoading) return;
setIsLoading(true);
try {
await queueJobs([job.id]);
onJobUpdated?.();
} catch (err) {
alert(err instanceof Error ? err.message : 'Failed to queue job');
} finally {
setIsLoading(false);
}
}
async function handleDequeue(e: React.MouseEvent) {
e.preventDefault();
e.stopPropagation();
if (isLoading) return;
setIsLoading(true);
try {
await dequeueJob(job.id);
onJobUpdated?.();
} catch (err) {
alert(err instanceof Error ? err.message : 'Failed to remove job from the queue');
} finally {
setIsLoading(false);
}
}
async function handleStop(e: React.MouseEvent) {
e.preventDefault();
e.stopPropagation();
@@ -238,7 +271,30 @@ export default function JobCard({ job, onJobUpdated }: JobCardProps) {
>
Start
</Button>
) : job.status === 'queued' ? (
<Button
size="sm"
variant="outline"
onClick={handleDequeue}
disabled={isLoading}
title="Take this job out of the queue"
>
Remove from queue
</Button>
) : null}
{(job.status === 'pending' ||
job.status === 'failed' ||
job.status === 'stopped') && (
<Button
size="sm"
variant="outline"
onClick={handleQueue}
disabled={isLoading}
title="Run this job when the ones queued before it have finished"
>
Queue
</Button>
)}
{job.status === 'completed' &&
job.output_path &&
job.job_type === 'inference' && (
@@ -77,7 +77,7 @@ export default function JobDetailsSidebar({
const isRestarting =
previousJobId.current === job.id &&
wasTerminal &&
(job.status === 'pending' || job.status === 'running');
(job.status === 'pending' || job.status === 'queued' || job.status === 'running');
if (previousJobId.current !== job.id || isRestarting) {
stateRef.current.text = '';
@@ -89,7 +89,10 @@ export default function JobDetailsSidebar({
}, [job.id, job.status]);
React.useEffect(() => {
const shouldPoll = job.status === 'running' || job.status === 'pending';
const shouldPoll =
job.status === 'running' ||
job.status === 'pending' ||
job.status === 'queued';
let pollInterval: ReturnType<typeof setInterval> | null = null;
let mounted = true;
// Effect-local lock so this job's first fetch is never blocked by a
@@ -108,7 +108,8 @@ export default function JobQueue({ jobType, jobTypesForList }: JobQueueProps) {
// Poll every second while any job is running/pending; stop otherwise.
const hasActive = jobs.some(
(j) => j.status === 'running' || j.status === 'pending',
(j) =>
j.status === 'running' || j.status === 'pending' || j.status === 'queued',
);
React.useEffect(() => {
if (!hasActive) return;
@@ -0,0 +1,158 @@
import * as React from 'react';
import { flushSync } from 'react-dom';
import { render, screen, waitFor, within } from '@testing-library/react';
import userEvent from '@testing-library/user-event';
import { beforeEach, describe, expect, it, vi } from 'vitest';
import CreateJobModal from './CreateJobModal';
import { createJob, getDatasets, getModels } from '@/lib/api';
import { defaultOptionsStore } from '@/stores/defaultOptions';
import { DEFAULT_OPTIONS } from '@/lib/defaultOptions';
vi.mock('sonner', () => ({ toast: { warning: vi.fn(), error: vi.fn(), success: vi.fn() } }));
vi.mock('@/lib/api', () => ({
createJob: vi.fn(),
getModels: vi.fn(),
getDatasets: vi.fn(),
uploadImage: vi.fn(),
getSettings: vi.fn(),
updateSettings: vi.fn(),
}));
const H3 = { id: 'minimax-h3/i2v', label: 'MiniMax H3', type: 'i2v' };
// A browser runs a microtask checkpoint -- so React flushes -- between the
// listeners of a single native event; jsdom's synchronous dispatch never does.
// Radix decides whether a parent dialog should dismiss from the layer stack at
// the moment its listener runs, so without this the bug can't show up here.
const wrapped = new WeakMap<object, EventListener>();
beforeEach(() => {
const realAdd = Document.prototype.addEventListener;
const realRemove = Document.prototype.removeEventListener;
vi.spyOn(document, 'addEventListener').mockImplementation(function (
this: Document,
type: string,
listener: unknown,
options?: unknown,
) {
if (typeof listener === 'function' && ['keydown', 'pointerdown', 'click'].includes(type)) {
const flushing: EventListener = (e) => {
(listener as EventListener)(e);
flushSync(() => {});
};
wrapped.set(listener, flushing);
return realAdd.call(this, type, flushing, options as AddEventListenerOptions);
}
return realAdd.call(this, type, listener as EventListener, options as AddEventListenerOptions);
} as typeof document.addEventListener);
vi.spyOn(document, 'removeEventListener').mockImplementation(function (
this: Document,
type: string,
listener: unknown,
options?: unknown,
) {
return realRemove.call(
this,
type,
(wrapped.get(listener as object) ?? listener) as EventListener,
options as EventListenerOptions,
);
} as typeof document.removeEventListener);
defaultOptionsStore.set({ options: DEFAULT_OPTIONS });
vi.mocked(getModels).mockResolvedValue([H3]);
vi.mocked(getDatasets).mockResolvedValue([]);
vi.mocked(createJob).mockResolvedValue({ id: 'j' } as never);
});
function Harness() {
const [open, setOpen] = React.useState(true);
return (
<>
<div data-testid="state">{open ? 'open' : 'closed'}</div>
<CreateJobModal
isOpen={open}
onClose={() => setOpen(false)}
onSuccess={() => {}}
jobType="inference"
workloadType="i2v"
/>
</>
);
}
type User = ReturnType<typeof userEvent.setup>;
type Closer = (user: User, dialog: HTMLElement) => Promise<void>;
async function renderH3Modal(user: User) {
render(<Harness />);
await screen.findByRole('option', { name: /MiniMax H3/ });
await user.selectOptions(screen.getByLabelText('Model'), H3.id);
}
// Radix marks every dialog under the topmost one aria-hidden, which also hides its
// accessible name, so the job dialog is identified by its title text instead.
const expectParentStillOpen = () => {
expect(screen.getByTestId('state')).toHaveTextContent('open');
expect(screen.getByText(/New Inference Job/)).toBeInTheDocument();
};
// [0] is the job dialog's overlay; the last is the topmost dialog's.
const topOverlay = () => [...document.querySelectorAll('.bg-black\\/70')].at(-1) as Element;
const CLOSERS: [string, Closer][] = [
['Cancel', (u, d) => u.click(within(d).getByRole('button', { name: 'Cancel' }))],
['the X button', (u, d) => u.click(within(d).getByRole('button', { name: 'Close' }))],
['Escape', (u) => u.keyboard('{Escape}')],
['a click on the overlay', (u) => u.click(topOverlay())],
];
describe('closing a dialog opened from the job form', () => {
describe.each([
['shot list', 'Open shot list', 'Shot list', [...CLOSERS, ['Apply', (u: User, d: HTMLElement) => u.click(within(d).getByRole('button', { name: 'Apply' }))] as [string, Closer]]],
['dialogue editor', 'Open dialogue editor', 'Dialogue editor', [...CLOSERS, ['Apply to field', (u: User, d: HTMLElement) => u.click(within(d).getByRole('button', { name: 'Apply to field' }))] as [string, Closer]]],
] as const)('%s', (_label, openButton, title, closers) => {
it.each(closers)('closing with %s leaves the job form open', async (_name, close) => {
const user = userEvent.setup();
await renderH3Modal(user);
await user.click(await screen.findByRole('button', { name: openButton }));
const child = await screen.findByRole('dialog', { name: title });
await close(user, child);
await waitFor(() => expect(screen.queryByRole('dialog', { name: title })).not.toBeInTheDocument());
expectParentStillOpen();
});
});
it('the job form is still usable afterwards', async () => {
const user = userEvent.setup();
await renderH3Modal(user);
await user.click(await screen.findByRole('button', { name: 'Open shot list' }));
const child = await screen.findByRole('dialog', { name: 'Shot list' });
await user.click(within(child).getByRole('button', { name: 'Cancel' }));
await waitFor(() => expect(screen.queryByRole('dialog', { name: 'Shot list' })).not.toBeInTheDocument());
await user.type(screen.getByLabelText('Name (optional)'), 'still here');
expect(screen.getByLabelText('Name (optional)')).toHaveValue('still here');
// ...and the dialog can be opened again
await user.click(screen.getByRole('button', { name: 'Open shot list' }));
expect(await screen.findByRole('dialog', { name: 'Shot list' })).toBeInTheDocument();
});
describe('the script import inside the shot list', () => {
it.each(CLOSERS.slice(0, 4))('closing it with %s leaves the shot list and the job form open', async (_name, close) => {
const user = userEvent.setup();
await renderH3Modal(user);
await user.click(await screen.findByRole('button', { name: 'Open shot list' }));
const shotList = await screen.findByRole('dialog', { name: 'Shot list' });
await user.click(within(shotList).getByRole('button', { name: 'Import script…' }));
const importDialog = await screen.findByRole('dialog', { name: 'Import script' });
await close(user, importDialog);
await waitFor(() => expect(screen.queryByRole('dialog', { name: 'Import script' })).not.toBeInTheDocument());
expect(screen.getByRole('dialog', { name: 'Shot list' })).toBeInTheDocument();
expectParentStillOpen();
});
});
});
@@ -0,0 +1,253 @@
import { render, screen, within } from '@testing-library/react';
import userEvent from '@testing-library/user-event';
import { describe, expect, it, vi } from 'vitest';
import { H3ShotListEditor } from './H3ShotListEditor';
const TWO_SHOTS = '[Shot 1] Wide shot. A dock.\n[Shot 2] At 00:02.000, the shot cuts to a close-up shot. A face.';
const WITH_LINE = '[Shot 1] Wide shot. (S1) <d>[English] Hi.</d>';
type User = ReturnType<typeof userEvent.setup>;
function setup(detailed: string, extra: { open?: boolean } = {}) {
const onApply = vi.fn();
const ui = (open: boolean) => (
<H3ShotListEditor
open={open}
subjectDefinitions="<Subject 1> is a man."
initialDetailedDescription={detailed}
initialRetentionAnalysis=""
onApply={onApply}
onClose={() => {}}
/>
);
const view = render(ui(extra.open ?? true));
return { onApply, view, ui };
}
const undoButton = () => screen.getByRole('button', { name: /Undo/ });
const redoButton = () => screen.getByRole('button', { name: /Redo/ });
const applied = (onApply: ReturnType<typeof vi.fn>) => onApply.mock.calls.at(-1)?.[0] as string;
const apply = (user: User) => user.click(screen.getByRole('button', { name: 'Apply' }));
const lineBox = () => screen.getByPlaceholderText('What the speaker says') as HTMLTextAreaElement;
const ctrl = (key: string) => `{Control>}${key}{/Control}`;
describe('shot list undo/redo', () => {
it('starts with nothing to undo or redo', () => {
setup(TWO_SHOTS);
expect(undoButton()).toBeDisabled();
expect(redoButton()).toBeDisabled();
});
it('undoes and redoes moving a shot', async () => {
const user = userEvent.setup();
const { onApply } = setup(TWO_SHOTS);
await user.click(screen.getAllByRole('button', { name: '↑' })[1]);
expect(undoButton()).toBeEnabled();
await user.click(undoButton());
expect(redoButton()).toBeEnabled();
await apply(user);
expect(applied(onApply)).toBe(TWO_SHOTS);
});
it('redo reapplies what was undone', async () => {
const user = userEvent.setup();
const { onApply } = setup(TWO_SHOTS);
await user.click(screen.getAllByRole('button', { name: '↑' })[1]);
await user.click(undoButton());
await user.click(redoButton());
await apply(user);
expect(applied(onApply).startsWith('[Shot 1] Close-up shot. A face.')).toBe(true);
});
it('undoes removing a shot, dialogue and all', async () => {
const user = userEvent.setup();
const { onApply } = setup(`${WITH_LINE}\n[Shot 2] At 00:02.500, the shot cuts to a wide shot. B.`);
await user.click(screen.getAllByRole('button', { name: 'Remove' })[0]);
expect(screen.queryByPlaceholderText('What the speaker says')).not.toBeInTheDocument();
await user.click(undoButton());
expect(lineBox().value).toBe('Hi.');
await apply(user);
expect(applied(onApply)).toContain('(S1) <d>[English] Hi.</d>');
});
it('undoes an add, a type change, a subject toggle, and a cut time in turn', async () => {
const user = userEvent.setup();
const { onApply } = setup(TWO_SHOTS);
await user.click(screen.getAllByRole('button', { name: /Add shot/ })[0]);
await user.selectOptions(screen.getAllByLabelText('Shot type')[0], 'pov');
await user.click(screen.getAllByRole('button', { name: '<Subject 1>' })[0]);
const cut = screen.getAllByPlaceholderText(/\d\d:\d\d\.\d{3}/)[0] as HTMLInputElement;
await user.type(cut, '5{Enter}');
// four discrete steps, undone one at a time
for (let i = 0; i < 4; i += 1) await user.click(undoButton());
expect(undoButton()).toBeDisabled();
await apply(user);
expect(applied(onApply)).toBe(TWO_SHOTS);
});
it('committing a cut time that has not changed is not a step', async () => {
const user = userEvent.setup();
setup(TWO_SHOTS);
const cut = screen.getAllByPlaceholderText(/\d\d:\d\d\.\d{3}/)[0] as HTMLInputElement;
await user.type(cut, '5{Enter}');
await user.click(undoButton()); // the click blurs the box, which re-commits 00:05.000
expect(undoButton()).toBeDisabled();
});
it('a burst of typing is one step', async () => {
const user = userEvent.setup();
const { onApply } = setup(WITH_LINE);
await user.type(lineBox(), ' there my friend');
expect(lineBox().value).toBe('Hi. there my friend');
await user.click(undoButton());
expect(lineBox().value).toBe('Hi.');
expect(undoButton()).toBeDisabled();
await apply(user);
expect(applied(onApply)).toContain('<d>[English] Hi.</d>');
});
it('a tag inserted from the palette is its own step, separate from typing', async () => {
const user = userEvent.setup();
setup(WITH_LINE);
await user.type(lineBox(), ' there');
lineBox().setSelectionRange(lineBox().value.length, lineBox().value.length);
await user.click(screen.getByRole('button', { name: /<pause>/ }));
expect(lineBox().value).toBe('Hi. there <pause>');
await user.click(undoButton());
expect(lineBox().value).toBe('Hi. there');
await user.click(undoButton());
expect(lineBox().value).toBe('Hi.');
});
it('undoes edits to the description and the scene notes', async () => {
const user = userEvent.setup();
const { onApply } = setup('[Shot 1] Wide shot. A dock.');
await user.type(screen.getByPlaceholderText(/Live-action, cinematic/), 'Noir.');
await user.type(screen.getByPlaceholderText("What's happening in this shot"), ' At dawn.');
await user.click(undoButton());
await user.click(undoButton());
expect(undoButton()).toBeDisabled();
await apply(user);
expect(applied(onApply)).toBe('[Shot 1] Wide shot. A dock.');
});
it('typing in different boxes makes separate steps', async () => {
const user = userEvent.setup();
setup('[Shot 1] Wide shot. A dock.');
await user.type(screen.getByPlaceholderText(/Live-action, cinematic/), 'Noir.');
await user.type(screen.getByPlaceholderText("What's happening in this shot"), ' At dawn.');
await user.click(undoButton());
expect((screen.getByPlaceholderText("What's happening in this shot") as HTMLTextAreaElement).value).toBe('A dock.');
expect((screen.getByPlaceholderText(/Live-action, cinematic/) as HTMLTextAreaElement).value).toBe('Noir.');
});
it('a new edit clears what could be redone', async () => {
const user = userEvent.setup();
setup(TWO_SHOTS);
await user.click(screen.getAllByRole('button', { name: '↑' })[1]);
await user.click(undoButton());
expect(redoButton()).toBeEnabled();
await user.click(screen.getAllByRole('button', { name: /Add shot/ })[0]);
expect(redoButton()).toBeDisabled();
});
});
describe('shot list undo/redo shortcuts', () => {
it('Ctrl+Z from inside a textbox undoes the whole list, not just that box', async () => {
const user = userEvent.setup();
const { onApply } = setup(TWO_SHOTS);
await user.click(screen.getAllByRole('button', { name: '↑' })[1]); // move a shot
// focus a different textbox, then undo from there
const box = screen.getAllByPlaceholderText("What's happening in this shot")[0] as HTMLTextAreaElement;
await user.click(box);
await user.keyboard(ctrl('z'));
expect(undoButton()).toBeDisabled();
await apply(user);
expect(applied(onApply)).toBe(TWO_SHOTS);
});
it('redoes with Ctrl+Shift+Z and with Ctrl+Y', async () => {
const user = userEvent.setup();
setup(TWO_SHOTS);
await user.click(screen.getAllByRole('button', { name: '↑' })[1]);
const box = screen.getAllByPlaceholderText("What's happening in this shot")[0];
await user.click(box);
await user.keyboard(ctrl('z'));
expect(redoButton()).toBeEnabled();
await user.keyboard('{Control>}{Shift>}z{/Shift}{/Control}');
expect(redoButton()).toBeDisabled();
await user.keyboard(ctrl('z'));
await user.keyboard(ctrl('y'));
expect(redoButton()).toBeDisabled();
expect(undoButton()).toBeEnabled();
});
it('Cmd+Z works too', async () => {
const user = userEvent.setup();
setup(TWO_SHOTS);
await user.click(screen.getAllByRole('button', { name: '↑' })[1]);
await user.click(screen.getAllByPlaceholderText("What's happening in this shot")[0]);
await user.keyboard('{Meta>}z{/Meta}');
expect(undoButton()).toBeDisabled();
});
it('leaves the script import box to its own undo', async () => {
const user = userEvent.setup();
const { onApply } = setup(TWO_SHOTS);
await user.click(screen.getAllByRole('button', { name: '↑' })[1]);
expect(undoButton()).toBeEnabled();
await user.click(screen.getByRole('button', { name: 'Import script…' }));
const dialog = await screen.findByRole('dialog', { name: 'Import script' });
const box = within(dialog).getByLabelText('Script');
await user.click(box);
await user.keyboard(ctrl('z'));
await user.click(within(dialog).getByRole('button', { name: 'Cancel' }));
// the shot list's history was untouched: the move is still there to undo
await user.click(await screen.findByRole('button', { name: /Undo/ }));
await apply(user);
expect(applied(onApply)).toBe(TWO_SHOTS);
});
});
describe('shot list history across imports and reopening', () => {
it('loading a script clip is one undoable step', async () => {
const user = userEvent.setup();
const { onApply } = setup(TWO_SHOTS);
await user.click(screen.getByRole('button', { name: 'Import script…' }));
const dialog = await screen.findByRole('dialog', { name: 'Import script' });
await user.click(within(dialog).getByLabelText('Script'));
await user.paste('CAPTAIN: Sit down and hear me out.\n\nCLERK: Yes, sir.');
await user.click(within(dialog).getByRole('button', { name: 'Load clip' }));
await screen.findAllByPlaceholderText('What the speaker says');
await user.click(undoButton());
expect(screen.queryByPlaceholderText('What the speaker says')).not.toBeInTheDocument();
await apply(user);
expect(applied(onApply)).toBe(TWO_SHOTS);
});
it('opening the editor again starts a fresh history', async () => {
const user = userEvent.setup();
const { view, ui } = setup(TWO_SHOTS);
await user.click(screen.getAllByRole('button', { name: '↑' })[1]);
expect(undoButton()).toBeEnabled();
view.rerender(ui(false));
view.rerender(ui(true));
expect(await screen.findByRole('button', { name: /Undo/ })).toBeDisabled();
});
});
@@ -0,0 +1,111 @@
import { fireEvent, render, screen } from '@testing-library/react';
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import { BeforeAfterGrade } from './BeforeAfterGrade';
// jsdom has no real video decoder, canvas 2D context, or `ImageData` constructor,
// so all three are stubbed: the video always "has" an 8x6 frame ready, the canvas
// records what was drawn/read/written instead of actually rasterizing anything,
// and ImageData is a plain data holder (its real job -- backing a canvas -- is
// already covered by the getContext stub).
if (typeof globalThis.ImageData === 'undefined') {
class FakeImageData {
data: Uint8ClampedArray;
width: number;
height: number;
constructor(data: Uint8ClampedArray, width: number, height: number) {
this.data = data;
this.width = width;
this.height = height;
}
}
// @ts-expect-error -- test-only polyfill for a browser global jsdom doesn't provide
globalThis.ImageData = FakeImageData;
}
function stubVideoFrame() {
Object.defineProperty(HTMLMediaElement.prototype, 'readyState', { configurable: true, get: () => 4 });
Object.defineProperty(HTMLVideoElement.prototype, 'videoWidth', { configurable: true, get: () => 8 });
Object.defineProperty(HTMLVideoElement.prototype, 'videoHeight', { configurable: true, get: () => 6 });
}
function stubCanvasContext() {
const data = new Uint8ClampedArray(8 * 6 * 4).fill(100);
const putImageData = vi.fn();
const drawImage = vi.fn();
const getImageData = vi.fn(() => ({ width: 8, height: 6, data }));
vi.spyOn(HTMLCanvasElement.prototype, 'getContext').mockReturnValue({
drawImage,
getImageData,
putImageData,
} as unknown as CanvasRenderingContext2D);
return { drawImage, getImageData, putImageData, data };
}
describe('BeforeAfterGrade', () => {
beforeEach(() => {
stubVideoFrame();
});
afterEach(() => {
vi.restoreAllMocks();
});
it('shows the "load a frame" hint until a video frame is captured', () => {
const ref = { current: null };
render(<BeforeAfterGrade videoRef={ref} brightness={0} contrast={1} saturation={1} />);
expect(screen.getByText(/Play or seek the preview above/)).toBeInTheDocument();
});
it('captures a frame once the video has data, and draws it into both canvases', () => {
const { getImageData, putImageData } = stubCanvasContext();
const video = document.createElement('video');
document.body.appendChild(video);
const ref = { current: video };
render(<BeforeAfterGrade videoRef={ref} brightness={0} contrast={1} saturation={1} />);
fireEvent.loadedData(video);
expect(getImageData).toHaveBeenCalled();
expect(putImageData).toHaveBeenCalled();
expect(screen.queryByText(/Play or seek the preview above/)).not.toBeInTheDocument();
document.body.removeChild(video);
});
it('re-grades the "after" canvas when the color values change, without touching the video again', () => {
const { putImageData } = stubCanvasContext();
const video = document.createElement('video');
document.body.appendChild(video);
const ref = { current: video };
const { rerender } = render(<BeforeAfterGrade videoRef={ref} brightness={0} contrast={1} saturation={1} />);
fireEvent.loadedData(video);
const callsAfterCapture = putImageData.mock.calls.length;
rerender(<BeforeAfterGrade videoRef={ref} brightness={50} contrast={1} saturation={1} />);
expect(putImageData.mock.calls.length).toBeGreaterThan(callsAfterCapture);
// Neutral values were drawn as-is; the brightened redraw's pixel buffer must differ.
const neutralData = putImageData.mock.calls[0][0].data as Uint8ClampedArray;
const brightenedData = putImageData.mock.calls[putImageData.mock.calls.length - 1][0].data as Uint8ClampedArray;
expect(brightenedData[0]).toBeGreaterThan(neutralData[0]);
document.body.removeChild(video);
});
it('"Use current frame" re-captures the video at its current position', () => {
const { drawImage } = stubCanvasContext();
const video = document.createElement('video');
document.body.appendChild(video);
const ref = { current: video };
render(<BeforeAfterGrade videoRef={ref} brightness={0} contrast={1} saturation={1} />);
fireEvent.loadedData(video);
const callsAfterAutoCapture = drawImage.mock.calls.length;
fireEvent.click(screen.getByRole('button', { name: 'Use current frame' }));
expect(drawImage.mock.calls.length).toBeGreaterThan(callsAfterAutoCapture);
document.body.removeChild(video);
});
});
@@ -0,0 +1,113 @@
'use client';
import * as React from 'react';
import { applyGradeInPlace, isNeutralGrade } from '@/lib/grading';
export interface BeforeAfterGradeProps {
/** the video the snapshot is captured from -- must be `crossOrigin="anonymous"` for pixel access to work */
videoRef: React.RefObject<HTMLVideoElement | null>;
brightness: number;
contrast: number;
saturation: number;
/** called with the video's currentTime whenever a frame is (re)captured, so a caller with multiple
* sections can tell which one the previewed frame falls into and keep its own selection in sync */
onFrameCaptured?: (timeSeconds: number) => void;
}
// Downscaled so the per-pixel pass stays instant on every slider tick.
const PREVIEW_WIDTH = 320;
/**
* A still frame from `videoRef`, shown twice: untouched, and with the current
* color values applied via `applyGradeInPlace` -- entirely in the browser, so
* dragging a slider never waits on the server. Captures a frame automatically
* once the video has one to show; "Use current frame" re-captures whatever the
* video is currently paused/seeked to, for comparing a specific moment.
*/
export function BeforeAfterGrade({ videoRef, brightness, contrast, saturation, onFrameCaptured }: BeforeAfterGradeProps) {
const beforeRef = React.useRef<HTMLCanvasElement>(null);
const afterRef = React.useRef<HTMLCanvasElement>(null);
const sourceRef = React.useRef<ImageData | null>(null);
const [ready, setReady] = React.useState(false);
// Kept in a ref, not a `capture` dependency, so an inline arrow function prop
// doesn't force the video-listener effect below to tear down and re-add itself every render.
const onFrameCapturedRef = React.useRef(onFrameCaptured);
onFrameCapturedRef.current = onFrameCaptured;
const capture = React.useCallback(() => {
const video = videoRef.current;
const before = beforeRef.current;
if (!video || !before || video.readyState < 2 || !video.videoWidth) return;
const scale = PREVIEW_WIDTH / video.videoWidth;
const width = PREVIEW_WIDTH;
const height = Math.round(video.videoHeight * scale);
before.width = width;
before.height = height;
const ctx = before.getContext('2d');
if (!ctx) return;
ctx.drawImage(video, 0, 0, width, height);
try {
sourceRef.current = ctx.getImageData(0, 0, width, height);
setReady(true);
onFrameCapturedRef.current?.(video.currentTime);
} catch {
// A cross-origin frame the browser wouldn't expose pixels for -- comparison stays unavailable.
sourceRef.current = null;
setReady(false);
}
}, [videoRef]);
React.useEffect(() => {
const video = videoRef.current;
if (!video) return;
if (video.readyState >= 2) capture();
video.addEventListener('loadeddata', capture);
video.addEventListener('seeked', capture);
return () => {
video.removeEventListener('loadeddata', capture);
video.removeEventListener('seeked', capture);
};
}, [videoRef, capture]);
React.useEffect(() => {
const source = sourceRef.current;
const after = afterRef.current;
if (!source || !after) return;
after.width = source.width;
after.height = source.height;
const ctx = after.getContext('2d');
if (!ctx) return;
const copy = new ImageData(new Uint8ClampedArray(source.data), source.width, source.height);
if (!isNeutralGrade(brightness, contrast, saturation)) {
applyGradeInPlace(copy.data, brightness, contrast, saturation);
}
ctx.putImageData(copy, 0, 0);
// eslint-disable-next-line react-hooks/exhaustive-deps -- redraws whenever a new frame is captured (`ready`) too
}, [brightness, contrast, saturation, ready]);
return (
<div className="flex flex-col gap-2">
<div className="flex items-center justify-between">
<p className="text-xs font-medium uppercase tracking-wide text-muted-foreground">Before / after</p>
<button type="button" onClick={capture} className="text-xs text-accent-blue hover:underline">
Use current frame
</button>
</div>
<div className="grid grid-cols-2 gap-2">
<div className="flex flex-col gap-1">
<canvas ref={beforeRef} role="img" aria-label="Before color grading" className="w-full rounded-md border border-border bg-black" />
<span className="text-center text-[11px] text-muted-foreground">Before</span>
</div>
<div className="flex flex-col gap-1">
<canvas ref={afterRef} role="img" aria-label="After color grading" className="w-full rounded-md border border-border bg-black" />
<span className="text-center text-[11px] text-muted-foreground">After</span>
</div>
</div>
{!ready && (
<p className="text-xs text-muted-foreground">Play or seek the preview above to load a frame to compare.</p>
)}
</div>
);
}
@@ -0,0 +1,72 @@
'use client';
import * as React from 'react';
import { toast } from 'sonner';
import { Button } from '@/components/ui/button';
import {
Dialog,
DialogContent,
DialogDescription,
DialogFooter,
DialogHeader,
DialogTitle,
} from '@/components/ui/dialog';
import { downloadMergedVideo, getMergedVideoUrl, type MergedScene } from '@/lib/api';
import { formatClock } from '@/lib/scenes';
import { downloadBlob } from '@/lib/utils';
export interface MergedSceneDialogProps {
/** the merged video to show, or null when closed */
merged: (MergedScene & { title: string }) | null;
onClose: () => void;
}
/** The finished, merged scene: play it, and save it. */
export function MergedSceneDialog({ merged, onClose }: MergedSceneDialogProps) {
const [downloading, setDownloading] = React.useState(false);
async function download() {
if (!merged) return;
setDownloading(true);
try {
downloadBlob(await downloadMergedVideo(merged.filename), merged.filename);
} catch (e) {
toast.error(e instanceof Error ? e.message : "Couldn't download the video.");
} finally {
setDownloading(false);
}
}
return (
<Dialog open={!!merged} onOpenChange={(open) => !open && onClose()}>
<DialogContent className="max-w-3xl">
<DialogHeader>
<DialogTitle>{merged?.title} merged</DialogTitle>
<DialogDescription>
{merged &&
`${merged.clips} clip${merged.clips === 1 ? '' : 's'} joined in order · ${formatClock(merged.seconds)}`}
</DialogDescription>
</DialogHeader>
{merged && (
<video
src={getMergedVideoUrl(merged.filename)}
aria-label={`Merged video of ${merged.title}`}
className="max-h-[60vh] w-full rounded-md border border-border bg-black"
controls
playsInline
preload="metadata"
/>
)}
<DialogFooter>
<Button type="button" variant="outline" onClick={onClose}>
Close
</Button>
<Button type="button" onClick={download} disabled={downloading}>
{downloading ? 'Downloading…' : 'Download'}
</Button>
</DialogFooter>
</DialogContent>
</Dialog>
);
}
@@ -0,0 +1,453 @@
'use client';
import * as React from 'react';
import { H3ShotPanel } from '@/components/jobs/H3ShotPanel';
import { Badge, type BadgeProps } from '@/components/ui/badge';
import { Button } from '@/components/ui/button';
import { NativeSelect } from '@/components/ui/native-select';
import { getJobVideoUrl } from '@/lib/api';
import { attachedLastFrames, whyCannotAttach } from '@/lib/continuation';
import { formatCutTime } from '@/lib/h3Shots';
import { QUEUEABLE_STATUSES } from '@/lib/sceneQueue';
import { formatClock, type Scene, type SceneClip } from '@/lib/scenes';
import type { Job } from '@/lib/types';
import { cn } from '@/lib/utils';
const BADGE_VARIANTS: Record<string, BadgeProps['variant']> = {
pending: 'secondary',
queued: 'default',
running: 'warning',
completed: 'success',
failed: 'destructive',
stopped: 'secondary',
};
const SEGMENT_TONE: Record<string, string> = {
completed: 'bg-emerald-500/70',
running: 'bg-amber-500/70',
queued: 'bg-sky-500/60',
failed: 'bg-rose-500/70',
};
/** Statuses a job can still be edited in (mirrors the Jobs page). */
export const EDITABLE_STATUSES = ['pending', 'failed', 'stopped'];
function clipLabel(clip: SceneClip): string {
return clip.job.name?.trim() || `Job ${clip.job.id.slice(0, 8)}`;
}
function takeLabel(job: Job, index: number, total: number): string {
const ms = job.created_at < 1e12 ? job.created_at * 1000 : job.created_at;
const when = new Date(ms).toLocaleString(undefined, {
month: 'short',
day: 'numeric',
hour: '2-digit',
minute: '2-digit',
});
return `Take ${index + 1} of ${total} · ${job.status} · ${when}`;
}
function scrollToClip(id: string) {
document.getElementById(`clip-${id}`)?.scrollIntoView({ behavior: 'smooth', block: 'start' });
}
function Timeline({ scene }: { scene: Scene }) {
return (
<div className="flex h-4 w-full gap-px overflow-hidden rounded-md" role="group" aria-label="Scene timeline">
{scene.clips.map((clip) => (
<button
key={clip.job.id}
type="button"
onClick={() => scrollToClip(clip.job.id)}
aria-label={`Go to clip ${clip.position}: ${clipLabel(clip)}`}
title={`Clip ${clip.position} · ${clipLabel(clip)} · ${clip.job.status} · ${formatClock(clip.startSeconds)}–${formatClock(clip.startSeconds + clip.durationSeconds)}`}
style={{ flexGrow: Math.max(clip.durationSeconds, 0.1), flexBasis: 0 }}
className={cn(
'min-w-[3px] transition-opacity hover:opacity-70',
SEGMENT_TONE[clip.job.status] ?? 'bg-muted-foreground/40',
)}
/>
))}
</div>
);
}
function ClipPreview({ job, videoVersion }: { job: Job; videoVersion: number }) {
const isImage = job.output_path?.toLowerCase().endsWith('.png') ?? false;
const src = getJobVideoUrl(job.id, videoVersion);
// Sets the *initial* muted state only, imperatively, so it doesn't fight the
// user's own unmute via the native controls: `muted` as a plain JSX prop is
// controlled, and React re-applies it on every re-render of this component
// (e.g. the scene board's poll while a job is queued/running), silently
// re-muting a video the user just unmuted.
const videoRef = React.useRef<HTMLVideoElement | null>(null);
React.useEffect(() => {
if (videoRef.current) videoRef.current.muted = true;
}, []);
return isImage ? (
// eslint-disable-next-line @next/next/no-img-element
<img src={src} alt={job.name ?? 'Generated output'} className="max-h-56 rounded-md border border-border" />
) : (
<video
ref={videoRef}
src={src}
aria-label={`Generated video for ${job.name ?? job.id}`}
className="max-h-56 rounded-md border border-border bg-black"
controls
loop
playsInline
preload="metadata"
/>
);
}
function ClipRow({
clip,
previous,
firstShotNumber,
onOpenJob,
onChooseTake,
onUseLastFrame,
onDetachLastFrame,
onQueueClip,
onDequeueClip,
onTrimClip,
videoVersionFor,
}: {
clip: SceneClip;
/** the clip before this one in the scene, if any */
previous: SceneClip | undefined;
firstShotNumber: number;
onOpenJob: (job: Job) => void;
onChooseTake: (takeKey: string, jobId: string) => void;
onUseLastFrame: (clip: SceneClip, previous: SceneClip) => Promise<void>;
onDetachLastFrame: (clip: SceneClip) => Promise<void>;
onQueueClip: (clip: SceneClip) => Promise<void>;
onDequeueClip: (clip: SceneClip) => Promise<void>;
onTrimClip: (clip: SceneClip) => void;
videoVersionFor: (jobId: string) => number;
}) {
const [previewing, setPreviewing] = React.useState(false);
const [attaching, setAttaching] = React.useState(false);
const [queueing, setQueueing] = React.useState(false);
const { job } = clip;
const canPreview = job.status === 'completed' && !!job.output_path;
const editable = EDITABLE_STATUSES.includes(job.status);
const queued = job.status === 'queued';
const canQueue = QUEUEABLE_STATUSES.includes(job.status);
// Starting from the previous clip's last frame: offered on clips that can still be
// edited. The frame is taken when this clip runs, so the previous clip needn't be done yet.
const frames = attachedLastFrames(job.references);
const fromPrevious = previous && frames.some((f) => f.jobId === previous.job.id);
const fromAnotherTake = frames.some((f) => f.jobId !== previous?.job.id);
const previousDone = !!previous && previous.job.status === 'completed';
const cannotUseFrame = previous ? whyCannotAttach(job.prompt, job.references) : null;
const usingLastFrame = frames.length > 0;
const lastFrameBlocked = attaching || (!usingLastFrame && !!cannotUseFrame);
async function toggleLastFrame() {
if (!previous) return;
setAttaching(true);
try {
await (usingLastFrame ? onDetachLastFrame(clip) : onUseLastFrame(clip, previous));
} finally {
setAttaching(false);
}
}
async function queueOrDequeue(action: (clip: SceneClip) => Promise<void>) {
setQueueing(true);
try {
await action(clip);
} finally {
setQueueing(false);
}
}
return (
<li id={`clip-${job.id}`} className="scroll-mt-32 rounded-xl border border-border bg-card p-3">
<div className="mb-2 flex flex-wrap items-center gap-2">
<span className="flex size-6 shrink-0 items-center justify-center rounded-md bg-secondary text-xs font-semibold text-foreground">
{clip.position}
</span>
<h3 className="min-w-0 truncate text-sm font-semibold text-foreground">{clipLabel(clip)}</h3>
<Badge variant={BADGE_VARIANTS[job.status] ?? 'secondary'}>{job.status}</Badge>
<span className="font-mono text-xs text-muted-foreground">
{formatClock(clip.startSeconds)}–{formatClock(clip.startSeconds + clip.durationSeconds)}
</span>
{clip.continuation && (
<Badge variant="outline" className="normal-case tracking-normal">
continues previous shot
</Badge>
)}
<span className="ml-auto flex flex-wrap items-center justify-end gap-2">
{previous && queued && fromPrevious && (
<Badge variant="outline" className="normal-case tracking-normal">
starts from clip {previous.position}&apos;s last frame
</Badge>
)}
{previous && queued && fromPrevious && !previousDone && (
<Badge variant="outline" className="normal-case tracking-normal">
waiting for clip {previous.position}
</Badge>
)}
{previous && editable && (
<label
className={cn(
'flex items-center gap-1.5 text-xs text-foreground',
lastFrameBlocked && 'text-muted-foreground',
)}
title={
usingLastFrame
? "Stop starting this clip from the previous clip's last frame"
: (cannotUseFrame ??
`Attach the last frame of clip ${previous.position} as this clip's opening frame${
previousDone ? '' : ` (taken when this clip runs, after clip ${previous.position} has finished)`
}`)
}
>
<input
type="checkbox"
checked={usingLastFrame}
onChange={toggleLastFrame}
disabled={lastFrameBlocked}
className="size-4"
/>
{attaching
? usingLastFrame
? 'Removing…'
: 'Attaching…'
: fromPrevious || !usingLastFrame
? `Use last frame of clip ${previous.position}`
: "Starts from another take's last frame"}
</label>
)}
{clip.takeKey && clip.takes.length > 1 && (
<NativeSelect
aria-label={`Take for clip ${clip.position}`}
value={job.id}
onChange={(e) => onChooseTake(clip.takeKey as string, e.target.value)}
className="h-8 w-auto py-0 text-xs"
>
{clip.takes.map((take, i) => (
<option key={take.id} value={take.id}>
{takeLabel(take, i, clip.takes.length)}
</option>
))}
</NativeSelect>
)}
{canPreview && (
<Button type="button" size="sm" variant="outline" onClick={() => setPreviewing((p) => !p)}>
{previewing ? 'Hide video' : 'Preview'}
</Button>
)}
{canPreview && (
<Button type="button" size="sm" variant="outline" onClick={() => onTrimClip(clip)}>
Edit video
</Button>
)}
{canQueue && (
<Button
type="button"
size="sm"
variant="outline"
onClick={() => queueOrDequeue(onQueueClip)}
disabled={queueing}
title="Run this clip when the ones queued before it have finished"
>
Queue
</Button>
)}
{queued && (
<Button
type="button"
size="sm"
variant="outline"
onClick={() => queueOrDequeue(onDequeueClip)}
disabled={queueing}
>
Remove from queue
</Button>
)}
<Button type="button" size="sm" variant="outline" onClick={() => onOpenJob(job)}>
{editable ? 'Edit' : 'View'}
</Button>
</span>
</div>
{previewing && canPreview && (
<div className="mb-2">
<ClipPreview job={job} videoVersion={videoVersionFor(job.id)} />
</div>
)}
{clip.shots.length === 0 ? (
<p className="rounded-lg border border-dashed border-border p-3 text-sm text-muted-foreground">
This job&apos;s prompt has no shots.
</p>
) : (
<div className="flex items-stretch gap-2 overflow-x-auto pb-1" aria-label={`Shots in ${clipLabel(clip)}`}>
{clip.shots.map((shot, i) => (
<div
key={shot.id}
className="flex w-52 shrink-0 flex-col gap-1.5 rounded-xl border border-border bg-background p-2"
>
<H3ShotPanel
shot={shot}
number={firstShotNumber + i}
timeLabel={formatCutTime(clip.shotStarts[i])}
/>
</div>
))}
</div>
)}
</li>
);
}
export interface SceneBoardProps {
scene: Scene;
onOpenJob: (job: Job) => void;
/** start a clip from the last frame of the one before it; resolves when done (errors are reported by the caller) */
onUseLastFrame: (clip: SceneClip, previous: SceneClip) => Promise<void>;
/** pick which re-run of a clip number to show */
onChooseTake: (takeKey: string, jobId: string) => void;
/** open the confirmation for queueing every runnable clip of the scene */
onQueueScene: (scene: Scene) => void;
/** join the scene's finished clips into one video; resolves when done (errors are reported by the caller) */
onMergeScene: (scene: Scene) => Promise<void>;
/** stop starting a clip from a previous clip's last frame; resolves when done */
onDetachLastFrame: (clip: SceneClip) => Promise<void>;
/** queue / unqueue one clip; resolve when done (errors are reported by the caller) */
onQueueClip: (clip: SceneClip) => Promise<void>;
onDequeueClip: (clip: SceneClip) => Promise<void>;
/** open the trim dialog for one clip's video */
onTrimClip: (clip: SceneClip) => void;
/** cache-busting version for a job's video preview, bumped after each edit */
videoVersionFor: (jobId: string) => number;
}
/** One scene: a timeline of its clips, then every clip's shots in order, numbered straight through. */
export function SceneBoard({
scene,
onOpenJob,
onChooseTake,
onUseLastFrame,
onDetachLastFrame,
onQueueScene,
onMergeScene,
onQueueClip,
onDequeueClip,
onTrimClip,
videoVersionFor,
}: SceneBoardProps) {
let shotNumber = 1;
const statuses = Object.entries(scene.statusCounts);
const anythingToQueue = scene.clips.some((c) => QUEUEABLE_STATUSES.includes(c.job.status));
// Merging needs every clip's video, so a scene missing one can't pass for the whole thing.
const unfinished = scene.clips.filter((c) => c.job.status !== 'completed' || !c.job.output_path).length;
const [merging, setMerging] = React.useState(false);
async function merge() {
setMerging(true);
try {
await onMergeScene(scene);
} finally {
setMerging(false);
}
}
// The header stays where it is until it reaches the top of the scrolling area,
// then sticks there. A 1px sentinel sitting just above it tells us which: once
// the sentinel has scrolled out of view, the header is stuck.
const sentinelRef = React.useRef<HTMLDivElement>(null);
const [stuck, setStuck] = React.useState(false);
React.useEffect(() => {
const el = sentinelRef.current;
if (!el || typeof IntersectionObserver === 'undefined') return;
const observer = new IntersectionObserver(([entry]) => setStuck(!entry.isIntersecting));
observer.observe(el);
return () => observer.disconnect();
}, []);
return (
<section aria-label={scene.title} className="flex flex-col gap-4">
<div ref={sentinelRef} aria-hidden className="-mb-4 h-px" />
<div
data-stuck={stuck}
className={cn(
'sticky top-0 z-30 -mx-3 flex flex-col gap-2 bg-card px-3 pb-3 pt-2 transition-shadow',
stuck && 'border-b border-border shadow-md',
)}
>
<div className="flex flex-wrap items-baseline gap-x-4 gap-y-1">
<h2 className="text-xl font-semibold text-foreground">{scene.title}</h2>
<p className="text-sm text-muted-foreground">
{scene.clips.length} clip{scene.clips.length === 1 ? '' : 's'} · {scene.shotCount} shot
{scene.shotCount === 1 ? '' : 's'} · about {formatClock(scene.totalSeconds)}
</p>
<span className="flex flex-wrap gap-1.5">
{statuses.map(([status, count]) => (
<Badge key={status} variant={BADGE_VARIANTS[status] ?? 'secondary'}>
{count} {status}
</Badge>
))}
</span>
<span className="ml-auto flex flex-wrap gap-2">
<Button
type="button"
size="sm"
variant="outline"
onClick={merge}
disabled={unfinished > 0 || merging}
title={
unfinished > 0
? `${unfinished} clip${unfinished === 1 ? " hasn't" : "s haven't"} finished yet`
: 'Join all the clips, in order, into one video'
}
>
{merging ? 'Merging…' : 'Merge scene'}
</Button>
<Button
type="button"
size="sm"
onClick={() => onQueueScene(scene)}
disabled={!anythingToQueue}
title={anythingToQueue ? 'Queue every clip that has not run yet' : 'Every clip has already run or is queued'}
>
Queue scene
</Button>
</span>
</div>
<Timeline scene={scene} />
</div>
<ol className="flex list-none flex-col gap-3 p-0">
{scene.clips.map((clip, index) => {
const first = shotNumber;
shotNumber += clip.shots.length;
return (
<ClipRow
key={clip.takeKey ?? clip.job.id}
clip={clip}
previous={scene.clips[index - 1]}
firstShotNumber={first}
onOpenJob={onOpenJob}
onChooseTake={onChooseTake}
onUseLastFrame={onUseLastFrame}
onDetachLastFrame={onDetachLastFrame}
onQueueClip={onQueueClip}
onDequeueClip={onDequeueClip}
onTrimClip={onTrimClip}
videoVersionFor={videoVersionFor}
/>
);
})}
</ol>
</section>
);
}
@@ -0,0 +1,132 @@
'use client';
import * as React from 'react';
import { Button } from '@/components/ui/button';
import {
Dialog,
DialogContent,
DialogDescription,
DialogFooter,
DialogHeader,
DialogTitle,
} from '@/components/ui/dialog';
import { attachedLastFrames } from '@/lib/continuation';
import { planSceneQueue, QUEUEABLE_STATUSES, type SceneQueuePlan } from '@/lib/sceneQueue';
import type { Scene, SceneClip } from '@/lib/scenes';
const plural = (n: number, word: string) => `${n} ${word}${n === 1 ? '' : 's'}`;
const clipNumbers = (clips: SceneClip[]) => clips.map((c) => c.position).join(', ');
export interface SceneQueueDialogProps {
scene: Scene | null;
onConfirm: (plan: SceneQueuePlan) => Promise<void>;
onClose: () => void;
}
/**
* Confirm queueing a whole scene: which clips are queued, and whether each
* starts from the previous clip's last frame. Nothing is sent until "Queue".
*/
export function SceneQueueDialog({ scene, onConfirm, onClose }: SceneQueueDialogProps) {
// Off by default: the per-clip "Use last frame" boxes are the source of truth, and
// this would add links to every clip that doesn't have one.
const [chain, setChain] = React.useState(false);
const [busy, setBusy] = React.useState(false);
// Each opening starts from the default choice.
React.useEffect(() => {
if (scene) setChain(false);
}, [scene]);
const plan = React.useMemo(() => (scene ? planSceneQueue(scene.clips, { chain }) : null), [scene, chain]);
const alreadyLinked = React.useMemo(
() =>
scene
? scene.clips.filter(
(c) => QUEUEABLE_STATUSES.includes(c.job.status) && attachedLastFrames(c.job.references).length > 0,
)
: [],
[scene],
);
// Whether the option would do anything is independent of the checkbox.
const linkable = React.useMemo(
() => (scene ? planSceneQueue(scene.clips, { chain: true }) : null),
[scene],
);
async function confirm() {
if (!plan) return;
setBusy(true);
try {
await onConfirm(plan);
} finally {
setBusy(false);
}
}
return (
<Dialog open={!!scene} onOpenChange={(open) => !open && !busy && onClose()}>
<DialogContent>
<DialogHeader>
<DialogTitle>Queue {scene?.title}</DialogTitle>
<DialogDescription>
{plan && plan.queue.length > 0
? `${plural(plan.queue.length, 'clip')} will be queued and run one at a time, in order.`
: 'There is nothing in this scene to queue.'}
</DialogDescription>
</DialogHeader>
{plan && linkable && (
<div className="flex flex-col gap-3 text-sm">
{alreadyLinked.length > 0 && (
<p className="text-xs text-muted-foreground">
Clip {clipNumbers(alreadyLinked)} already start{alreadyLinked.length === 1 ? 's' : ''} from a previous
frame (as you set on the clip{alreadyLinked.length === 1 ? '' : 's'}). Other clips are queued as they are.
</p>
)}
{linkable.chain.length > 0 && (
<label className="flex items-start gap-2">
<input
type="checkbox"
checked={chain}
onChange={(e) => setChain(e.target.checked)}
className="mt-1 size-4"
/>
<span>
Start each clip from the previous clip&apos;s last frame
<span className="block text-xs text-muted-foreground">
Also links {plural(linkable.chain.length, 'clip')} that don&apos;t have it (
{clipNumbers(linkable.chain.map((c) => c.clip))}), rewriting their prompts. Each waits for the clip
before it to finish.
</span>
</span>
</label>
)}
{chain && plan.cannotChain.length > 0 && (
<p className="text-xs text-muted-foreground">
Clip {clipNumbers(plan.cannotChain.map((c) => c.clip))} can&apos;t be linked automatically (its prompt
isn&apos;t in the six-section format) and will start without the previous frame.
</p>
)}
{plan.skipped.length > 0 && (
<p className="text-xs text-muted-foreground">
Skipping{' '}
{plan.skipped.map((s) => `clip ${s.clip.position} (${s.reason})`).join(', ')}.
</p>
)}
</div>
)}
<DialogFooter>
<Button type="button" variant="outline" onClick={onClose} disabled={busy}>
Cancel
</Button>
<Button type="button" onClick={confirm} disabled={busy || !plan || plan.queue.length === 0}>
{busy ? 'Queueing…' : plan ? `Queue ${plural(plan.queue.length, 'clip')}` : 'Queue'}
</Button>
</DialogFooter>
</DialogContent>
</Dialog>
);
}
@@ -0,0 +1,500 @@
'use client';
import * as React from 'react';
import { toast } from 'sonner';
import { BeforeAfterGrade } from '@/components/scenes/BeforeAfterGrade';
import { Button } from '@/components/ui/button';
import {
Dialog,
DialogContent,
DialogDescription,
DialogFooter,
DialogHeader,
DialogTitle,
} from '@/components/ui/dialog';
import { Input } from '@/components/ui/input';
import { Label } from '@/components/ui/label';
import { Slider } from '@/components/ui/slider';
import { getJobVideoUrl, restoreJobVideo, trimJob } from '@/lib/api';
import { BRIGHTNESS_BOUNDS, CONTRAST_BOUNDS, NEUTRAL_GRADE, SATURATION_BOUNDS } from '@/lib/grading';
import { clipSeconds, DEFAULT_FPS, formatClock } from '@/lib/scenes';
import type { GradeSegment, Job } from '@/lib/types';
import { cn } from '@/lib/utils';
export interface TrimClipDialogProps {
/** the clip to edit, or null when closed */
clip: Job | null;
/**
* Cache-busts the preview after an edit (the video keeps the same URL, only
* the file changes). Owned by the caller, not this component -- a version
* counter kept here gets reset by the very re-render `onChanged` causes
* (a fresh `clip` object comes back from the edit), before the browser
* ever sees the new URL.
*/
videoVersion: number;
onClose: () => void;
/** called with the job as the edit/restore left it, so the caller can refresh */
onChanged: (job: Job) => void;
}
/** A section as edited locally -- same shape as GradeSegment plus a stable React key. */
interface EditableSection extends GradeSegment {
key: string;
}
let sectionSeq = 0;
function makeSectionKey(): string {
return `section-${Date.now()}-${sectionSeq++}`;
}
const NEUTRAL_SECTION: Omit<EditableSection, 'key'> = { end_seconds: null, ...NEUTRAL_GRADE };
/** The clip's last-applied sections, or one neutral section covering everything if it was never edited. */
function sectionsFromJob(job: Job): EditableSection[] {
const segments = job.edit_segments;
if (segments && segments.length > 0) {
return segments.map((s) => ({ ...s, key: makeSectionKey() }));
}
return [{ ...NEUTRAL_SECTION, key: makeSectionKey() }];
}
// A split closer than this to an existing edge would leave a sliver too thin
// to be a meaningful section (and trim.py would likely refuse it outright).
const MIN_SPLIT_MARGIN_SECONDS = 0.1;
/**
* Cut a finished clip to a chosen in/out range and grade it section by
* section -- like a razor tool: "Split at current time" cuts the clip where
* the preview is paused, and each resulting section gets its own brightness,
* contrast and saturation. Previous/Next frame step the preview by exactly
* one frame (at the clip's own fps), for lining a cut up precisely -- the
* native scrub bar alone isn't frame-accurate. The server always keeps the
* untouched original and re-renders from it with the full set of values
* shown here, so range and sections are applied together and neither is
* lost by changing the other --
* as long as both are set the way you want them before you click Apply. The
* fields keep whatever you last applied while this dialog stays open for the
* same clip, so a second, smaller tweak doesn't also reset the range. Opening
* it fresh for a clip (or reopening it later) shows exactly what's currently
* applied -- the server records the range/sections of the last edit on the
* job itself (`edit_*`), so this isn't guessing from the clip's current
* length. A never-edited clip shows one neutral section covering its full
* length. Restoring clears the fields back to that same neutral state, since
* that's what it just did to the video.
*/
export function TrimClipDialog({ clip, videoVersion, onClose, onChanged }: TrimClipDialogProps) {
const duration = clip ? clipSeconds(clip) : 0;
const fps = clip?.fps && clip.fps > 0 ? clip.fps : DEFAULT_FPS;
const [start, setStart] = React.useState('0');
const [end, setEnd] = React.useState('');
const [sections, setSections] = React.useState<EditableSection[]>([]);
const [previewIndex, setPreviewIndex] = React.useState(0);
const [busy, setBusy] = React.useState<'apply' | 'restore' | null>(null);
const [displayTime, setDisplayTime] = React.useState(0);
// Reset the fields only when a *different* clip opens -- onChanged() re-renders
// this dialog with a fresh job object for the same edit, which must not reset
// what the user is about to apply next.
const openClipId = React.useRef<string | null>(null);
React.useEffect(() => {
if (clip && clip.id !== openClipId.current) {
openClipId.current = clip.id;
setStart(String(clip.edit_start_seconds ?? 0));
setEnd(clip.edit_end_seconds != null ? clip.edit_end_seconds.toFixed(2) : clipSeconds(clip).toFixed(2));
setSections(sectionsFromJob(clip));
setPreviewIndex(0);
} else if (!clip) {
openClipId.current = null;
}
}, [clip]);
// This dialog is always mounted by the page (it just renders closed when
// `clip` is null), so a `useEffect(fn, [])` here would only ever fire once,
// the very first time -- before `clip` is ever set and before the <video>
// below exists at all -- and never again on later opens. A callback ref
// fires on every real mount/unmount of the DOM node itself, however many
// times that happens across this component's one long lifetime, which is
// what both of these actually need:
// - setting the *initial* muted state imperatively, so it doesn't fight the
// user's own unmute via the native controls (`muted` as a plain JSX prop
// is controlled, and React would re-apply it on every re-render instead)
// - keeping the frame-step readout in sync with wherever the video is,
// whether that's a step button, a Preview click, or the native scrub bar
const videoRef = React.useRef<HTMLVideoElement | null>(null);
const setVideoNode = React.useCallback((node: HTMLVideoElement | null) => {
videoRef.current = node;
if (!node) return;
node.muted = true;
const onTimeChange = () => setDisplayTime(node.currentTime);
node.addEventListener('timeupdate', onTimeChange);
node.addEventListener('seeked', onTimeChange);
return () => {
node.removeEventListener('timeupdate', onTimeChange);
node.removeEventListener('seeked', onTimeChange);
};
}, []);
if (!clip) {
return <Dialog open={false} onOpenChange={() => {}} />;
}
const startSeconds = Number(start);
const endSeconds = end.trim() === '' ? undefined : Number(end);
const rangeError =
!Number.isFinite(startSeconds) || startSeconds < 0
? 'The start must be 0 or later.'
: endSeconds !== undefined && (!Number.isFinite(endSeconds) || endSeconds <= startSeconds)
? 'The end must come after the start.'
: null;
// Color has no equivalent error: the sliders can't produce a value outside their bounds,
// and sections are only ever built by split/merge below, which keep them in valid order.
const invalid = !!rangeError;
/** The [start, end) of section `i`, in the kept range's own seconds -- the same timeline the preview plays. */
function sectionBounds(i: number): [number, number] {
const start = i === 0 ? 0 : (sections[i - 1].end_seconds ?? duration);
const end = sections[i].end_seconds ?? duration;
return [start, end];
}
function updateSection(i: number, patch: Partial<Omit<EditableSection, 'key' | 'end_seconds'>>) {
setSections((prev) => prev.map((s, idx) => (idx === i ? { ...s, ...patch } : s)));
}
/**
* Seeks the preview to `target`, clamped to the clip's length. A seek
* requested before the video has loaded metadata (readyState is still
* HAVE_NOTHING) is unreliable -- browsers commonly drop or silently reset it
* once metadata actually arrives -- so this defers to 'loadedmetadata' when
* that's the case, instead of jumping to the wrong spot (often 0).
*/
function seekTo(target: number) {
const video = videoRef.current;
if (!video) return;
const clamped = Math.max(0, Math.min(duration, target));
const apply = () => {
video.currentTime = clamped;
};
if (video.readyState >= 1) {
apply();
} else {
video.addEventListener('loadedmetadata', apply, { once: true });
}
}
/** Steps the preview by exactly one frame (at the clip's own fps), pausing playback first. */
function stepFrame(direction: 1 | -1) {
const video = videoRef.current;
if (!video) return;
video.pause();
seekTo(video.currentTime + direction / fps);
}
function splitAtCurrentTime() {
const video = videoRef.current;
if (!video) return;
const t = video.currentTime;
for (let i = 0; i < sections.length; i++) {
const [sectionStart, sectionEnd] = sectionBounds(i);
if (t <= sectionStart + MIN_SPLIT_MARGIN_SECONDS || t >= sectionEnd - MIN_SPLIT_MARGIN_SECONDS) continue;
setSections((prev) => {
const next = [...prev];
next.splice(
i,
1,
{ ...prev[i], end_seconds: t },
{ ...NEUTRAL_SECTION, end_seconds: prev[i].end_seconds, key: makeSectionKey() },
);
return next;
});
setPreviewIndex(i + 1);
return;
}
toast.error('Pause the preview somewhere inside a section (away from its edges) to split it there.');
}
function mergeWithNext(i: number) {
setSections((prev) => {
const next = [...prev];
next[i] = { ...next[i], end_seconds: next[i + 1].end_seconds };
next.splice(i + 1, 1);
return next;
});
setPreviewIndex((p) => Math.min(p, i));
}
/** Seeks the preview to this section's start and previews it; BeforeAfterGrade re-captures on 'seeked'. */
function previewSection(i: number) {
const [sectionStart] = sectionBounds(i);
seekTo(sectionStart);
setPreviewIndex(i);
}
/** Keeps the preview panel's selected section in sync with wherever the video actually is. */
function handleFrameCaptured(t: number) {
for (let i = 0; i < sections.length; i++) {
const [, sectionEnd] = sectionBounds(i);
if (t < sectionEnd || i === sections.length - 1) {
setPreviewIndex(i);
return;
}
}
}
async function apply() {
if (!clip || invalid) return;
setBusy('apply');
try {
const updated = await trimJob(clip.id, {
startSeconds,
endSeconds,
segments: sections.map(({ key: _key, ...segment }) => segment),
});
toast.success(`Applied: ${formatClock(startSeconds)}–${endSeconds === undefined ? 'end' : formatClock(endSeconds)}.`);
onChanged(updated);
} catch (e) {
toast.error(e instanceof Error ? e.message : "Couldn't edit the clip.");
} finally {
setBusy(null);
}
}
async function restore() {
if (!clip) return;
setBusy('restore');
try {
const updated = await restoreJobVideo(clip.id);
// Restoring doesn't open a *different* clip, so the reset effect above
// won't fire on its own -- but the fields have to reflect it too, since
// that's exactly what just happened to the video.
setStart('0');
setEnd(clipSeconds(updated).toFixed(2));
setSections([{ ...NEUTRAL_SECTION, key: makeSectionKey() }]);
setPreviewIndex(0);
toast.success('Restored the original video.');
onChanged(updated);
} catch (e) {
toast.error(e instanceof Error ? e.message : "Couldn't restore the original video.");
} finally {
setBusy(null);
}
}
const preview = sections[previewIndex] ?? sections[0];
return (
<Dialog open onOpenChange={(open) => !open && !busy && onClose()}>
<DialogContent className="max-h-[90vh] max-w-lg overflow-y-auto">
<DialogHeader>
<DialogTitle>Edit {clip.name?.trim() || clip.id.slice(0, 8)}</DialogTitle>
<DialogDescription>
Currently {formatClock(duration)}. Editing keeps the untouched original, so this can be adjusted or undone.
</DialogDescription>
</DialogHeader>
<video
ref={setVideoNode}
src={getJobVideoUrl(clip.id, videoVersion)}
aria-label={`Preview of ${clip.name ?? clip.id}`}
className="max-h-64 w-full rounded-md border border-border bg-black"
controls
playsInline
preload="metadata"
crossOrigin="anonymous"
/>
<div className="flex items-center justify-between gap-2">
<div className="flex gap-2">
<Button type="button" variant="outline" size="sm" onClick={() => stepFrame(-1)} disabled={!!busy}>
◂ Previous frame
</Button>
<Button type="button" variant="outline" size="sm" onClick={() => stepFrame(1)} disabled={!!busy}>
Next frame ▸
</Button>
</div>
<span className="text-xs tabular-nums text-muted-foreground">
Frame {Math.round(displayTime * fps)} · {displayTime.toFixed(2)}s
</span>
</div>
<div className="flex flex-col gap-2">
<p className="text-xs font-medium uppercase tracking-wide text-muted-foreground">Range</p>
<div className="flex items-end gap-3">
<div className="flex flex-1 flex-col gap-1 text-sm">
<Label htmlFor="trim-start">Start (seconds)</Label>
<Input
id="trim-start"
type="number"
min={0}
step={0.1}
value={start}
onChange={(e) => setStart(e.target.value)}
disabled={!!busy}
/>
</div>
<div className="flex flex-1 flex-col gap-1 text-sm">
<Label htmlFor="trim-end">End (seconds, optional)</Label>
<Input
id="trim-end"
type="number"
min={0}
step={0.1}
placeholder="end of clip"
value={end}
onChange={(e) => setEnd(e.target.value)}
disabled={!!busy}
/>
</div>
</div>
{rangeError && <p className="text-sm text-destructive">{rangeError}</p>}
</div>
<div className="flex flex-col gap-3">
<div className="flex items-center justify-between">
<p className="text-xs font-medium uppercase tracking-wide text-muted-foreground">Sections</p>
<Button type="button" variant="outline" size="sm" onClick={splitAtCurrentTime} disabled={!!busy}>
Split at current time
</Button>
</div>
{sections.map((section, i) => {
const [sectionStart, sectionEnd] = sectionBounds(i);
return (
<div
key={section.key}
className={cn(
'flex flex-col gap-2 rounded-md border p-3',
i === previewIndex ? 'border-accent-blue' : 'border-border',
)}
>
<div className="flex items-center justify-between">
<span className="text-xs font-medium text-muted-foreground">
Section {i + 1} · {formatClock(sectionStart)}–{section.end_seconds == null ? 'end' : formatClock(sectionEnd)}
</span>
<div className="flex gap-3">
<button
type="button"
onClick={() => previewSection(i)}
disabled={!!busy}
className="text-xs text-accent-blue underline-offset-2 hover:underline disabled:cursor-not-allowed disabled:opacity-50"
>
Preview
</button>
{i < sections.length - 1 && (
<button
type="button"
onClick={() => mergeWithNext(i)}
disabled={!!busy}
className="text-xs text-accent-blue underline-offset-2 hover:underline disabled:cursor-not-allowed disabled:opacity-50"
>
Merge with next
</button>
)}
</div>
</div>
<GradeSlider
id={`grade-brightness-${section.key}`}
label="Brightness"
ariaLabel={`Section ${i + 1} Brightness`}
bounds={BRIGHTNESS_BOUNDS}
step={5}
value={section.brightness}
onChange={(v) => updateSection(i, { brightness: v })}
format={(v) => String(v)}
disabled={!!busy}
/>
<GradeSlider
id={`grade-contrast-${section.key}`}
label="Contrast"
ariaLabel={`Section ${i + 1} Contrast`}
bounds={CONTRAST_BOUNDS}
step={0.1}
value={section.contrast}
onChange={(v) => updateSection(i, { contrast: v })}
format={(v) => `${v.toFixed(1)}x`}
disabled={!!busy}
/>
<GradeSlider
id={`grade-saturation-${section.key}`}
label="Saturation"
ariaLabel={`Section ${i + 1} Saturation`}
bounds={SATURATION_BOUNDS}
step={0.1}
value={section.saturation}
onChange={(v) => updateSection(i, { saturation: v })}
format={(v) => `${v.toFixed(1)}x`}
disabled={!!busy}
/>
</div>
);
})}
<BeforeAfterGrade
videoRef={videoRef}
brightness={preview?.brightness ?? NEUTRAL_GRADE.brightness}
contrast={preview?.contrast ?? NEUTRAL_GRADE.contrast}
saturation={preview?.saturation ?? NEUTRAL_GRADE.saturation}
onFrameCaptured={handleFrameCaptured}
/>
</div>
<DialogFooter className="justify-between sm:justify-between">
<Button type="button" variant="outline" onClick={restore} disabled={!!busy}>
{busy === 'restore' ? 'Restoring…' : 'Restore original'}
</Button>
<div className="flex gap-2">
<Button type="button" variant="outline" onClick={onClose} disabled={!!busy}>
Close
</Button>
<Button type="button" onClick={apply} disabled={!!busy || invalid}>
{busy === 'apply' ? 'Applying…' : 'Apply'}
</Button>
</div>
</DialogFooter>
</DialogContent>
</Dialog>
);
}
function GradeSlider({
id,
label,
ariaLabel,
bounds,
step,
value,
onChange,
format,
disabled,
}: {
id: string;
label: string;
ariaLabel: string;
bounds: readonly [number, number];
step: number;
value: number;
onChange: (v: number) => void;
format: (v: number) => string;
disabled?: boolean;
}) {
return (
<div className="flex flex-col gap-1 text-sm">
<div className="flex items-center justify-between">
<Label htmlFor={id}>{label}</Label>
<span className="text-xs tabular-nums text-muted-foreground">{format(value)}</span>
</div>
<Slider
id={id}
min={bounds[0]}
max={bounds[1]}
step={step}
value={[value]}
onValueChange={(v) => onChange(v[0])}
disabled={disabled}
aria-label={ariaLabel}
/>
</div>
);
}
@@ -13,6 +13,7 @@ const TAB_TITLES: Record<string, string> = {
'/distillation': 'Jobs',
'/datasets': 'Datasets',
'/gallery': 'Gallery',
'/scenes': 'Scenes',
'/gpus': 'GPUs',
'/settings': 'Settings',
};
@@ -150,6 +150,14 @@ export default function PrimarySidebar({
>
Datasets
</Link>
<Link
href="/scenes"
aria-current={pathname === '/scenes' ? 'page' : undefined}
onClick={onMobileClose}
className={cn(TAB_BASE, pathname === '/scenes' && TAB_ACTIVE)}
>
Scenes
</Link>
<Link
href="/gallery"
aria-current={pathname === '/gallery' ? 'page' : undefined}
@@ -0,0 +1,30 @@
import { render, screen } from '@testing-library/react';
import { describe, expect, it } from 'vitest';
import { Dialog, DialogContent, DialogDescription, DialogTitle } from '@/components/ui/dialog';
// The app header is `fixed` at z-[100] (components/shell/Header.tsx).
const HEADER_Z = 100;
function zOf(el: Element | null): number {
const m = el?.className.match(/\bz-\[(\d+)\]/);
return m ? Number(m[1]) : 0;
}
describe('Dialog stacking', () => {
it('renders its overlay and content above the fixed app header', () => {
render(
<Dialog open>
<DialogContent>
<DialogTitle>Tall dialog</DialogTitle>
<DialogDescription>Body</DialogDescription>
</DialogContent>
</Dialog>,
);
const content = screen.getByRole('dialog');
const overlay = document.querySelector('[data-state="open"].bg-black\\/70');
expect(zOf(content)).toBeGreaterThan(HEADER_Z);
expect(zOf(overlay)).toBeGreaterThan(HEADER_Z);
});
});
@@ -11,6 +11,8 @@ const DialogTrigger = DialogPrimitive.Trigger;
const DialogPortal = DialogPrimitive.Portal;
const DialogClose = DialogPrimitive.Close;
// Dialogs sit above the app header (z-[100], fixed). Below it, a tall dialog's
// top edge slides under the header and, being fixed itself, can't be scrolled clear.
const DialogOverlay = React.forwardRef<
React.ElementRef<typeof DialogPrimitive.Overlay>,
React.ComponentPropsWithoutRef<typeof DialogPrimitive.Overlay>
@@ -18,7 +20,7 @@ const DialogOverlay = React.forwardRef<
<DialogPrimitive.Overlay
ref={ref}
className={cn(
'fixed inset-0 z-50 bg-black/70 backdrop-blur-sm data-[state=open]:animate-in data-[state=closed]:animate-out data-[state=closed]:fade-out-0 data-[state=open]:fade-in-0',
'fixed inset-0 z-[110] bg-black/70 backdrop-blur-sm data-[state=open]:animate-in data-[state=closed]:animate-out data-[state=closed]:fade-out-0 data-[state=open]:fade-in-0',
className,
)}
{...props}
@@ -36,7 +38,7 @@ const DialogContent = React.forwardRef<
ref={ref}
data-modal=""
className={cn(
'fixed left-1/2 top-1/2 z-50 grid w-full max-w-lg -translate-x-1/2 -translate-y-1/2 gap-4 rounded-2xl border border-border bg-card p-6 text-card-foreground shadow-lg duration-200 data-[state=open]:animate-in data-[state=closed]:animate-out data-[state=closed]:fade-out-0 data-[state=open]:fade-in-0',
'fixed left-1/2 top-1/2 z-[110] grid w-full max-w-lg -translate-x-1/2 -translate-y-1/2 gap-4 rounded-2xl border border-border bg-card p-6 text-card-foreground shadow-lg duration-200 data-[state=open]:animate-in data-[state=closed]:animate-out data-[state=closed]:fade-out-0 data-[state=open]:fade-in-0',
className,
)}
{...props}
@@ -0,0 +1,140 @@
import { act, renderHook } from '@testing-library/react';
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import { COALESCE_MS, useUndoableState } from './useUndoableState';
describe('useUndoableState', () => {
let now = 1_000_000;
beforeEach(() => {
now = 1_000_000;
vi.spyOn(Date, 'now').mockImplementation(() => now);
});
afterEach(() => vi.restoreAllMocks());
const tick = (ms: number) => {
now += ms;
};
it('makes each unkeyed change its own step', () => {
const { result } = renderHook(() => useUndoableState({ n: 0 }));
act(() => result.current.set({ n: 1 }));
act(() => result.current.set({ n: 2 }));
expect(result.current.state.n).toBe(2);
act(() => void result.current.undo());
expect(result.current.state.n).toBe(1);
act(() => void result.current.undo());
expect(result.current.state.n).toBe(0);
expect(result.current.canUndo).toBe(false);
});
it('merges same-key edits made close together into one step', () => {
const { result } = renderHook(() => useUndoableState('a'));
act(() => result.current.set('ab', 'text'));
tick(100);
act(() => result.current.set('abc', 'text'));
tick(100);
act(() => result.current.set('abcd', 'text'));
act(() => void result.current.undo());
expect(result.current.state).toBe('a');
expect(result.current.canUndo).toBe(false);
});
it('starts a new step after a pause, or when the key changes', () => {
const { result } = renderHook(() => useUndoableState('a'));
act(() => result.current.set('ab', 'text'));
tick(COALESCE_MS + 1);
act(() => result.current.set('abc', 'text'));
act(() => result.current.set('abcX', 'other'));
act(() => void result.current.undo());
expect(result.current.state).toBe('abc');
act(() => void result.current.undo());
expect(result.current.state).toBe('ab');
act(() => void result.current.undo());
expect(result.current.state).toBe('a');
});
it('never merges an unkeyed change into typing, or typing into it', () => {
const { result } = renderHook(() => useUndoableState('a'));
act(() => result.current.set('ab', 'text'));
act(() => result.current.set('AB'));
act(() => result.current.set('ABc', 'text'));
act(() => void result.current.undo());
expect(result.current.state).toBe('AB');
act(() => void result.current.undo());
expect(result.current.state).toBe('ab');
});
it('typing after an undo begins a fresh step instead of merging into the undone one', () => {
const { result } = renderHook(() => useUndoableState('a'));
act(() => result.current.set('ab', 'text'));
act(() => void result.current.undo());
act(() => result.current.set('ax', 'text'));
expect(result.current.canUndo).toBe(true);
act(() => void result.current.undo());
expect(result.current.state).toBe('a');
});
it('redoes, and a new change clears what could be redone', () => {
const { result } = renderHook(() => useUndoableState(0));
act(() => result.current.set(1));
act(() => result.current.set(2));
act(() => void result.current.undo());
act(() => void result.current.undo());
expect(result.current.canRedo).toBe(true);
act(() => void result.current.redo());
expect(result.current.state).toBe(1);
act(() => result.current.set(9));
expect(result.current.canRedo).toBe(false);
expect(result.current.redo()).toBe(false);
});
it('returns false when there is nothing to undo or redo', () => {
const { result } = renderHook(() => useUndoableState(0));
expect(result.current.undo()).toBe(false);
expect(result.current.redo()).toBe(false);
});
it('chains functional updates made in the same event', () => {
const { result } = renderHook(() => useUndoableState(0));
act(() => {
result.current.set((n) => n + 1);
result.current.set((n) => n + 1);
});
expect(result.current.state).toBe(2);
act(() => void result.current.undo());
expect(result.current.state).toBe(1);
});
it('ignores a set that produces the same value', () => {
const { result } = renderHook(() => useUndoableState('x'));
act(() => result.current.set('x'));
expect(result.current.canUndo).toBe(false);
});
it('reset replaces the value and drops all history', () => {
const { result } = renderHook(() => useUndoableState(0));
act(() => result.current.set(1));
act(() => void result.current.undo());
act(() => result.current.reset(7));
expect(result.current.state).toBe(7);
expect(result.current.canUndo).toBe(false);
expect(result.current.canRedo).toBe(false);
});
it('keeps at most 100 steps', () => {
const { result } = renderHook(() => useUndoableState(0));
for (let i = 1; i <= 130; i += 1) act(() => result.current.set(i));
let undone = 0;
while (result.current.canUndo) {
act(() => void result.current.undo());
undone += 1;
}
expect(undone).toBe(100);
expect(result.current.state).toBe(30);
});
});
@@ -0,0 +1,86 @@
'use client';
import * as React from 'react';
const MAX_HISTORY = 100;
// Edits made under the same key, this close together, merge into one undo step.
export const COALESCE_MS = 500;
interface Snapshot<T> {
present: T;
past: T[];
future: T[];
}
/**
* Undo/redo over a whole state value. Every `set` is one undo step, except
* that consecutive sets with the same `key` inside COALESCE_MS merge -- pass a
* key for typing so a burst is one step, and none for discrete changes (moves,
* adds, selects) so each is its own. Values are treated as immutable, so
* snapshots share structure and are cheap to keep.
*/
export function useUndoableState<T>(initial: T) {
const [snapshot, setSnapshot] = React.useState<Snapshot<T>>({ present: initial, past: [], future: [] });
// Mirrors the snapshot synchronously so several sets in one event chain correctly.
const ref = React.useRef(snapshot);
const last = React.useRef<{ key: string | null; at: number }>({ key: null, at: 0 });
const apply = React.useCallback((next: Snapshot<T>) => {
ref.current = next;
setSnapshot(next);
}, []);
const set = React.useCallback(
(update: T | ((prev: T) => T), key: string | null = null) => {
const cur = ref.current;
const next = typeof update === 'function' ? (update as (prev: T) => T)(cur.present) : update;
if (Object.is(next, cur.present)) return;
const now = Date.now();
const merges = key !== null && last.current.key === key && now - last.current.at < COALESCE_MS;
last.current = { key, at: now };
if (merges) {
apply({ ...cur, present: next });
} else {
apply({ present: next, past: [...cur.past.slice(-(MAX_HISTORY - 1)), cur.present], future: [] });
}
},
[apply],
);
const undo = React.useCallback(() => {
const cur = ref.current;
if (cur.past.length === 0) return false;
last.current = { key: null, at: 0 };
apply({ present: cur.past[cur.past.length - 1], past: cur.past.slice(0, -1), future: [cur.present, ...cur.future] });
return true;
}, [apply]);
const redo = React.useCallback(() => {
const cur = ref.current;
if (cur.future.length === 0) return false;
last.current = { key: null, at: 0 };
apply({ present: cur.future[0], past: [...cur.past, cur.present], future: cur.future.slice(1) });
return true;
}, [apply]);
/** Replace the value and forget all history (e.g. when an editor reopens). */
const reset = React.useCallback(
(value: T) => {
last.current = { key: null, at: 0 };
apply({ present: value, past: [], future: [] });
},
[apply],
);
return {
state: snapshot.present,
set,
undo,
redo,
reset,
canUndo: snapshot.past.length > 0,
canRedo: snapshot.future.length > 0,
};
}
@@ -0,0 +1,84 @@
import { act, renderHook } from '@testing-library/react';
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import { useUndoableText } from './useUndoableText';
describe('useUndoableText', () => {
beforeEach(() => vi.useFakeTimers());
afterEach(() => vi.useRealTimers());
it('coalesces a typing burst into one undo step', () => {
const { result } = renderHook(() => useUndoableText('a'));
act(() => {
result.current.type('ab');
result.current.type('abc');
});
act(() => {
vi.advanceTimersByTime(600);
});
expect(result.current.canUndo).toBe(true);
let restored: string | null = null;
act(() => {
restored = result.current.undo();
});
expect(restored).toBe('a');
expect(result.current.text).toBe('a');
expect(result.current.canUndo).toBe(false);
});
it('gives a committed edit its own step and supports redo', () => {
const { result } = renderHook(() => useUndoableText('hi'));
act(() => result.current.commit('hi <pause>'));
expect(result.current.text).toBe('hi <pause>');
act(() => {
result.current.undo();
});
expect(result.current.text).toBe('hi');
expect(result.current.canRedo).toBe(true);
let redone: string | null = null;
act(() => {
redone = result.current.redo();
});
expect(redone).toBe('hi <pause>');
expect(result.current.text).toBe('hi <pause>');
});
it('flushes pending typing before a commit so undo steps stay separate', () => {
const { result } = renderHook(() => useUndoableText(''));
act(() => result.current.type('hello'));
act(() => result.current.commit('hello <pause>'));
act(() => {
result.current.undo();
});
expect(result.current.text).toBe('hello');
act(() => {
result.current.undo();
});
expect(result.current.text).toBe('');
});
it('returns null when there is nothing to undo or redo', () => {
const { result } = renderHook(() => useUndoableText('x'));
let undone: string | null = 'sentinel';
let redone: string | null = 'sentinel';
act(() => {
undone = result.current.undo();
redone = result.current.redo();
});
expect(undone).toBeNull();
expect(redone).toBeNull();
});
it('reset replaces the text and drops history', () => {
const { result } = renderHook(() => useUndoableText('a'));
act(() => result.current.commit('b'));
act(() => result.current.reset('fresh'));
expect(result.current.text).toBe('fresh');
expect(result.current.canUndo).toBe(false);
expect(result.current.canRedo).toBe(false);
});
});
@@ -0,0 +1,109 @@
'use client';
import * as React from 'react';
const MAX_HISTORY = 100;
// How long a pause in typing has to be before it becomes its own undo step.
// Keeps undo from removing text one keystroke at a time while still giving
// each tag/preset insertion (the actual ask) a guaranteed, immediate step.
const TYPING_COALESCE_MS = 500;
/**
* Text state with undo/redo. Typing is coalesced into bursts; discrete edits
* (`commit`, e.g. a tag insertion) always get their own step. `textRef`
* mirrors `text` synchronously because history bookkeeping needs the true
* current value the instant an edit happens, not whatever the last render
* closed over.
*/
export function useUndoableText(initial: string) {
const [text, setTextState] = React.useState(initial);
const textRef = React.useRef(initial);
const [past, setPast] = React.useState<string[]>([]);
const [future, setFuture] = React.useState<string[]>([]);
// Start of the in-progress typing burst that hasn't become a history step
// yet; flushed into `past` once typing pauses or a discrete edit happens.
const typingBaseRef = React.useRef<string | null>(null);
const typingTimerRef = React.useRef<ReturnType<typeof setTimeout> | null>(null);
const setText = React.useCallback((next: string) => {
textRef.current = next;
setTextState(next);
}, []);
const pushHistory = React.useCallback((previous: string) => {
setPast((p) => (p.length >= MAX_HISTORY ? [...p.slice(1), previous] : [...p, previous]));
setFuture([]);
}, []);
const clearTypingTimer = React.useCallback(() => {
if (typingTimerRef.current) {
clearTimeout(typingTimerRef.current);
typingTimerRef.current = null;
}
}, []);
const flushTypingCheckpoint = React.useCallback(() => {
clearTypingTimer();
if (typingBaseRef.current != null && typingBaseRef.current !== textRef.current) {
pushHistory(typingBaseRef.current);
}
typingBaseRef.current = null;
}, [clearTypingTimer, pushHistory]);
/** Free typing: coalesced into one undo step per burst. */
const type = React.useCallback(
(next: string) => {
if (typingBaseRef.current == null) typingBaseRef.current = textRef.current;
setText(next);
clearTypingTimer();
typingTimerRef.current = setTimeout(flushTypingCheckpoint, TYPING_COALESCE_MS);
},
[setText, clearTypingTimer, flushTypingCheckpoint],
);
/** Discrete edit (tag/preset insertion): always its own undo step. */
const commit = React.useCallback(
(next: string) => {
flushTypingCheckpoint();
pushHistory(textRef.current);
setText(next);
},
[flushTypingCheckpoint, pushHistory, setText],
);
/** Returns the restored text (so the caller can place the caret), or null if nothing to undo. */
function undo(): string | null {
flushTypingCheckpoint();
if (past.length === 0) return null;
const prev = past[past.length - 1];
setPast(past.slice(0, -1));
setFuture([...future, textRef.current]);
setText(prev);
return prev;
}
function redo(): string | null {
if (future.length === 0) return null;
const next = future[future.length - 1];
setFuture(future.slice(0, -1));
setPast([...past, textRef.current]);
setText(next);
return next;
}
/** Replace the text and drop all history (e.g. when an editor is reopened). */
const reset = React.useCallback(
(value: string) => {
setText(value);
setPast([]);
setFuture([]);
typingBaseRef.current = null;
clearTypingTimer();
},
[setText, clearTypingTimer],
);
React.useEffect(() => clearTypingTimer, [clearTypingTimer]);
return { text, type, commit, undo, redo, reset, canUndo: past.length > 0, canRedo: future.length > 0 };
}
+162 -3
View File
@@ -5,7 +5,7 @@ import {
loadDefaultOptions,
type DefaultOptions,
} from "./defaultOptions";
import type { Job, JobType } from "./types";
import type { GradeSegment, Job, JobType } from "./types";
const DEFAULT_API_BASE_URL = "http://localhost:8189/api";
@@ -50,8 +50,15 @@ function stripTrailingSlash(url: string): string {
}
/** Full URL for streaming a job's video/image output (for &lt;video&gt; or &lt;img&gt; src). */
export function getJobVideoUrl(jobId: string): string {
return `${getApiBaseUrl()}/jobs/${jobId}/video`;
/**
* `version` busts the browser's cache after editing a job's video in place
* (Trim/Edit video): the URL is otherwise identical before and after an edit,
* even though the file on disk changed, so without this the same cached
* response keeps showing. Omit it where staleness doesn't matter.
*/
export function getJobVideoUrl(jobId: string, version?: number): string {
const base = `${getApiBaseUrl()}/jobs/${jobId}/video`;
return version ? `${base}?v=${version}` : base;
}
export interface CreateJobRequest {
@@ -194,6 +201,84 @@ export async function updateJob(
return response.json();
}
/**
* Ask the server to save the last frame of a finished clip and return its path,
* ready to attach to another job as an image reference.
*/
export async function extractLastFrame(jobId: string): Promise<{ path: string; media_type: MediaType }> {
const baseApiUrl = getApiBaseUrl();
const response = await fetch(`${baseApiUrl}/jobs/${jobId}/last-frame`, { method: "POST" });
if (!response.ok) {
const err = await response.json().catch(() => ({ detail: "Could not get the last frame" }));
throw new Error(err.detail || "Could not get the last frame");
}
return response.json();
}
export interface EditVideoParams {
startSeconds: number;
/** omitted (or undefined) keeps to the end of the video */
endSeconds?: number;
/**
* Ordered, contiguous razor-cut sections covering the whole kept range --
* each section's `end_seconds` is relative to that kept range, not the
* original file. Omitted (or empty) means one neutral section, i.e. a
* plain range trim with no grading.
*/
segments?: GradeSegment[];
}
/**
* Cut a completed job's video to a range and/or grade it section by section,
* in place. The untouched original is kept server-side, so this always
* renders fresh from it with the *given* parameters -- pass every value you
* want kept, not just the one you changed, or the rest reset to their neutral
* defaults. Can be undone with restoreJobVideo.
*/
export async function trimJob(jobId: string, params: EditVideoParams): Promise<Job> {
const baseApiUrl = getApiBaseUrl();
const response = await fetch(`${baseApiUrl}/jobs/${jobId}/trim`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
start_seconds: params.startSeconds,
end_seconds: params.endSeconds ?? null,
segments: params.segments ?? [],
}),
});
if (!response.ok) {
const err = await response.json().catch(() => ({ detail: "Could not edit the clip" }));
throw new Error(err.detail || "Could not edit the clip");
}
return response.json();
}
/** Undo every trim and color adjustment on a job's video, back to what it originally generated. */
export async function restoreJobVideo(jobId: string): Promise<Job> {
const baseApiUrl = getApiBaseUrl();
const response = await fetch(`${baseApiUrl}/jobs/${jobId}/restore-video`, { method: "POST" });
if (!response.ok) {
const err = await response.json().catch(() => ({ detail: "Could not restore the original video" }));
throw new Error(err.detail || "Could not restore the original video");
}
return response.json();
}
/**
* Ask the server to save the last second or so of a finished clip and return its
* path, ready to attach to another job as a video reference -- so the next
* clip continues with real motion instead of guessing it from a single frame.
*/
export async function extractLastClip(jobId: string): Promise<{ path: string; media_type: MediaType }> {
const baseApiUrl = getApiBaseUrl();
const response = await fetch(`${baseApiUrl}/jobs/${jobId}/last-clip`, { method: "POST" });
if (!response.ok) {
const err = await response.json().catch(() => ({ detail: "Could not get the end of the clip" }));
throw new Error(err.detail || "Could not get the end of the clip");
}
return response.json();
}
export async function uploadMedia(
file: File,
): Promise<{ path: string; media_type: MediaType }> {
@@ -313,6 +398,80 @@ export async function startJob(id: string): Promise<Job> {
return response.json();
}
/**
* Queue jobs, in the order given, to run one after another as the server has room.
* A job that starts from another job's last frame waits for that job to finish.
* All-or-nothing: if any job can't be queued, none are.
*/
export async function queueJobs(ids: string[]): Promise<Job[]> {
const baseApiUrl = getApiBaseUrl();
const response = await fetch(`${baseApiUrl}/jobs/queue`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ job_ids: ids }),
});
if (!response.ok) {
const error = await response
.json()
.catch(() => ({ detail: "Failed to queue jobs" }));
throw new Error(error.detail || "Failed to queue jobs");
}
return response.json();
}
/** Take a queued job out of the queue; it goes back to pending. */
export async function dequeueJob(id: string): Promise<Job> {
const baseApiUrl = getApiBaseUrl();
const response = await fetch(`${baseApiUrl}/jobs/${id}/dequeue`, {
method: "POST",
});
if (!response.ok) {
const error = await response
.json()
.catch(() => ({ detail: "Failed to remove job from the queue" }));
throw new Error(error.detail || "Failed to remove job from the queue");
}
return response.json();
}
export interface MergedScene {
/** file name on the server; pass to getMergedVideoUrl / downloadMergedVideo */
filename: string;
clips: number;
/** length of the merged video */
seconds: number;
}
/**
* Join the finished clips of a scene, in the order given, into one video. Fails,
* making nothing, if any clip hasn't finished.
*/
export async function mergeScene(jobIds: string[], name: string): Promise<MergedScene> {
const baseApiUrl = getApiBaseUrl();
const response = await fetch(`${baseApiUrl}/scenes/merge`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ job_ids: jobIds, name }),
});
if (!response.ok) {
const error = await response.json().catch(() => ({ detail: "Failed to merge the scene" }));
throw new Error(error.detail || "Failed to merge the scene");
}
return response.json();
}
export function getMergedVideoUrl(filename: string): string {
return `${getApiBaseUrl()}/merged/${encodeURIComponent(filename)}`;
}
export async function downloadMergedVideo(filename: string): Promise<Blob> {
const response = await fetch(getMergedVideoUrl(filename));
if (!response.ok) {
throw new Error("Failed to download the merged video");
}
return response.blob();
}
export async function stopJob(id: string): Promise<Job> {
const baseApiUrl = getApiBaseUrl();
const response = await fetch(`${baseApiUrl}/jobs/${id}/stop`, {
@@ -0,0 +1,478 @@
import { describe, expect, it } from "vitest";
import {
attachLastClip,
attachLastFrame,
attachedLastClips,
attachedLastFrames,
deferredLastClipSource,
deferredLastFrameSource,
detachLastClip,
detachLastFrame,
lastClipSourceJobId,
lastFrameSourceJobId,
whyCannotAttach,
whyCannotAttachClip,
} from "@/lib/continuation";
import { parseH3Prompt } from "@/lib/h3Prompt";
import { readField } from "@/lib/h3PromptFields";
const FRAME = "/data/uploads/last_frames/last_frame_job-5.png";
const REFS = [
{ source: "/refs/veteran.png", media_type: "image" },
{ source: "/refs/young.png", media_type: "image" },
];
const CLIP = [
"subject_definitions:",
"<Subject 1> is the veteran broker from <Picture 1>: a confident man.",
"<Subject 2> is the young broker from <Picture 2>: eager.",
"",
"summary:",
"[reference generation] The target video shows <Subject 1> and <Subject 2> at a lunch.",
"",
"retention_analysis:",
"<Subject 1> (appears in [Shot 1]): fully_preserved - his suit.",
"<Subject 2> (appears in [Shot 1]): fully_preserved - his suit.",
"",
"detailed_description:",
"[Shot 1] Close-up shot, dolly. Continuing the same shot on <Subject 1>, still speaking. (S1) <d>[English] And then, well.</d>",
"[Shot 2] At 00:04.000, the shot cuts to a wide shot. Both men at the table.",
"",
"overall_soundscape:",
"Muted restaurant ambience.",
"",
"non_diegetic_music:",
"N/A",
].join("\n");
function attach(prompt = CLIP, refs = REFS, frame = FRAME) {
const r = attachLastFrame(prompt, refs, frame);
if (!r.ok) throw new Error(r.error);
return r;
}
describe("attachLastFrame", () => {
it("adds the frame as the next image reference and names it by its picture number", () => {
const r = attach();
expect(r.label).toBe("<Picture 3>");
expect(r.references).toEqual([...REFS, { source: FRAME, media_type: "image" }]);
});
it("defines the frame as the first frame of Shot 1, after the existing definitions", () => {
const defs = readField(attach().prompt, "subject_definitions") as string;
expect(defs.split("\n")).toEqual([
"<Subject 1> is the veteran broker from <Picture 1>: a confident man.",
"<Subject 2> is the young broker from <Picture 2>: eager.",
"<Picture 3> is the first frame of [Shot 1]: the last frame of the previous clip, showing the scene, positions, and lighting exactly as that clip ended.",
]);
});
it("tags the summary as keyframe completion alongside its existing task type, and names the frame", () => {
const summary = readField(attach().prompt, "summary") as string;
expect(summary.startsWith("[reference generation + keyframe completion] The target video shows")).toBe(true);
expect(summary.endsWith("The target video starts from <Picture 3>, the last frame of the previous clip.")).toBe(true);
});
it("adds a first-frame line to the retention analysis", () => {
const lines = (readField(attach().prompt, "retention_analysis") as string).split("\n");
expect(lines).toHaveLength(3);
expect(lines[2]).toBe(
"<Picture 3> ([Shot 1] first frame): fully_preserved - the scene, positions, and lighting of the previous clip's last frame are retained.",
);
});
it("starts the first shot from the frame and leaves the other shots alone", () => {
const shots = (readField(attach().prompt, "detailed_description") as string).split("\n");
expect(shots[0]).toBe(
"[Shot 1] Close-up shot, dolly. The shot begins from <Picture 3>. Continuing the same shot on <Subject 1>, still speaking. (S1) <d>[English] And then, well.</d>",
);
expect(shots[1]).toBe("[Shot 2] At 00:04.000, the shot cuts to a wide shot. Both men at the table.");
});
it("leaves every other section untouched", () => {
const out = attach().prompt;
expect(readField(out, "overall_soundscape")).toBe("Muted restaurant ambience.");
expect(readField(out, "non_diegetic_music")).toBe("N/A");
});
it("still reads as a six-section prompt", () => {
expect(parseH3Prompt(attach().prompt)).not.toBeNull();
});
it("counts only images when numbering, so a video reference doesn't shift the label", () => {
const refs = [REFS[0], { source: "/refs/clip.mp4", media_type: "video" }, REFS[1]];
expect(attach(CLIP, refs).label).toBe("<Picture 3>");
expect(attach(CLIP, [{ source: "/refs/clip.mp4", media_type: "video" }]).label).toBe("<Picture 1>");
});
it("does not mutate what it was given", () => {
const refs = [...REFS];
attach(CLIP, refs);
expect(refs).toEqual(REFS);
});
});
describe("attachLastFrame: summary wording", () => {
const summaryOf = (summary: string) =>
readField(attach(CLIP.replace("[reference generation] The target video shows <Subject 1> and <Subject 2> at a lunch.", summary)).prompt, "summary") as string;
it("adds the tag when the summary has none", () => {
expect(summaryOf("Two men at a lunch.").startsWith("[keyframe completion] Two men at a lunch.")).toBe(true);
});
it("extends a combined tag", () => {
expect(summaryOf("[video editing + reference generation] Edit.").startsWith("[video editing + reference generation + keyframe completion] Edit.")).toBe(true);
});
it("does not repeat a tag that is already there", () => {
const s = summaryOf("[keyframe completion + reference generation] Both.");
expect(s.startsWith("[keyframe completion + reference generation] Both.")).toBe(true);
expect(s.match(/keyframe completion/g)).toHaveLength(1);
});
});
describe("attachLastFrame: other prompt shapes", () => {
it("replaces an N/A subject_definitions instead of appending under it", () => {
const out = attach(CLIP.replace(/subject_definitions:\n[\s\S]*?\n\nsummary:/, "subject_definitions:\nN/A\n\nsummary:")).prompt;
expect(readField(out, "subject_definitions")).toBe(
"<Picture 3> is the first frame of [Shot 1]: the last frame of the previous clip, showing the scene, positions, and lighting exactly as that clip ended.",
);
});
it("keeps hand-written shots verbatim, only prefixing the first", () => {
const hand = CLIP.replace(
/\[Shot 1\][^\n]*\n\[Shot 2\][^\n]*/,
"[Shot 1] A handheld medium shot holds <Subject 1> at frame left.\n[Shot 2] At 00:03.800, the shot cuts to a low tracking shot behind him.",
);
const shots = (readField(attach(hand).prompt, "detailed_description") as string).split("\n");
expect(shots).toEqual([
"[Shot 1] The shot begins from <Picture 3>. A handheld medium shot holds <Subject 1> at frame left.",
"[Shot 2] At 00:03.800, the shot cuts to a low tracking shot behind him.",
]);
});
it("adds the sentence to a first shot that is only dialogue", () => {
const dialogueOnly = CLIP.replace(
"[Shot 1] Close-up shot, dolly. Continuing the same shot on <Subject 1>, still speaking. (S1) <d>[English] And then, well.</d>",
"[Shot 1] Wide shot. (S1) <d>[English] Hi.</d>",
);
const first = (readField(attach(dialogueOnly).prompt, "detailed_description") as string).split("\n")[0];
expect(first).toBe("[Shot 1] Wide shot. The shot begins from <Picture 3>. (S1) <d>[English] Hi.</d>");
});
});
describe("attachLastFrame: refusals", () => {
it("won't attach the same frame twice", () => {
const r = attachLastFrame(CLIP, [...REFS, { source: FRAME, media_type: "image" }], FRAME);
expect(r).toEqual({ ok: false, error: "That frame is already attached to this clip." });
});
it("won't rewrite a prompt that isn't in the six-section format", () => {
const r = attachLastFrame("A toy car drives into a plush dog.", REFS, FRAME);
expect(r.ok).toBe(false);
expect(!r.ok && r.error).toMatch(/six-section/);
});
it("won't rewrite a base-format prompt either", () => {
const base = "For the target video, <Picture 1> is used.\n\nintegrated_multimodal_description: [Shot 1] A.\n\noverall_soundscape: Quiet.";
expect(whyCannotAttach(base, REFS)).toMatch(/six-section/);
});
it("stops at nine image references", () => {
const nine = Array.from({ length: 9 }, (_, i) => ({ source: `/r/${i}.png`, media_type: "image" }));
const r = attachLastFrame(CLIP, nine, FRAME);
expect(!r.ok && r.error).toMatch(/at most 9 image/);
});
it("reports a prompt with no shots", () => {
const empty = CLIP.replace(/detailed_description:\n[\s\S]*?\n\noverall_soundscape:/, "detailed_description:\n\n\noverall_soundscape:");
const r = attachLastFrame(empty, REFS, FRAME);
expect(r.ok).toBe(false);
});
});
describe("recognizing an attached last frame", () => {
it("reads the job id out of the saved path", () => {
expect(lastFrameSourceJobId(FRAME)).toBe("job-5");
expect(lastFrameSourceJobId("last_frame_abc-123.png")).toBe("abc-123");
expect(lastFrameSourceJobId("/refs/veteran.png")).toBeNull();
expect(lastFrameSourceJobId("/x/not_last_frame_job-5.png")).toBeNull();
});
it("finds last-frame references among a job's references", () => {
expect(attachedLastFrames([...REFS, { source: FRAME, media_type: "image" }])).toEqual([{ source: FRAME, jobId: "job-5" }]);
expect(attachedLastFrames(REFS)).toEqual([]);
expect(attachedLastFrames(null)).toEqual([]);
});
it("reads the job id out of a reference the server resolves when the job runs", () => {
expect(deferredLastFrameSource("job-5")).toBe("job-last-frame:job-5");
expect(lastFrameSourceJobId(deferredLastFrameSource("job-5"))).toBe("job-5");
expect(lastFrameSourceJobId("job-last-frame:")).toBeNull();
expect(attachedLastFrames([...REFS, { source: deferredLastFrameSource("job-5"), media_type: "image" }])).toEqual([
{ source: "job-last-frame:job-5", jobId: "job-5" },
]);
});
it("attaches a deferred frame exactly like a saved one", () => {
const result = attachLastFrame(CLIP, REFS, deferredLastFrameSource("job-5"));
expect(result.ok).toBe(true);
if (result.ok) {
expect(result.references.at(-1)).toEqual({ source: "job-last-frame:job-5", media_type: "image" });
expect(result.label).toBe("<Picture 3>");
}
});
});
describe("detachLastFrame", () => {
const roundTrip = (prompt: string, frame = FRAME) => {
const attached = attach(prompt, REFS, frame);
const r = detachLastFrame(attached.prompt, attached.references, frame);
if (!r.ok) throw new Error(r.error);
return r;
};
it("takes back exactly what attaching added", () => {
const r = roundTrip(CLIP);
expect(r.references).toEqual(REFS);
expect(r.prompt).toBe(CLIP);
});
it("works for a frame the server resolves when the job runs", () => {
expect(roundTrip(CLIP, deferredLastFrameSource("job-5")).prompt).toBe(CLIP);
});
it.each([
["a summary with no tag", "Two men at a lunch."],
["a combined tag", "[video editing + reference generation] Edit."],
])("restores %s", (_name, summary) => {
const prompt = CLIP.replace("[reference generation] The target video shows <Subject 1> and <Subject 2> at a lunch.", summary);
expect(roundTrip(prompt).prompt).toBe(prompt);
});
it("restores hand-written shots and an N/A section", () => {
const hand = CLIP.replace(/\[Shot 1\][^\n]*\n\[Shot 2\][^\n]*/, "[Shot 1] A handheld medium shot.\n[Shot 2] At 00:03.800, the shot cuts to a low tracking shot.").replace(
/subject_definitions:\n[\s\S]*?\n\nsummary:/,
"subject_definitions:\nN/A\n\nsummary:",
);
expect(roundTrip(hand).prompt).toBe(hand);
});
it("removes the sentence from a first shot that is only dialogue", () => {
const dialogueOnly = CLIP.replace(
"[Shot 1] Close-up shot, dolly. Continuing the same shot on <Subject 1>, still speaking. (S1) <d>[English] And then, well.</d>",
"[Shot 1] Wide shot. (S1) <d>[English] Hi.</d>",
);
expect(roundTrip(dialogueOnly).prompt).toBe(dialogueOnly);
});
it("only removes the reference from a prompt that was never rewritten", () => {
const r = detachLastFrame("A plain prompt.", [...REFS, { source: FRAME, media_type: "image" }], FRAME);
expect(r).toEqual({ ok: true, prompt: "A plain prompt.", references: REFS });
});
it("refuses when other images were added after the frame", () => {
const attached = attach();
const r = detachLastFrame(attached.prompt, [...attached.references, { source: "/refs/extra.png", media_type: "image" }], FRAME);
expect(r.ok).toBe(false);
});
it("refuses a frame that isn't attached", () => {
expect(detachLastFrame(CLIP, REFS, FRAME)).toEqual({ ok: false, error: "That frame isn't attached to this clip." });
});
});
const CLIP_VIDEO = "/data/uploads/last_clips/last_clip_job-5.mp4";
function attachClip(prompt = CLIP, refs = REFS, clip = CLIP_VIDEO) {
const r = attachLastClip(prompt, refs, clip);
if (!r.ok) throw new Error(r.error);
return r;
}
describe("attachLastClip", () => {
it("adds the clip as the next video reference and names it by its video number", () => {
const r = attachClip();
expect(r.label).toBe("<Video 1>");
expect(r.references).toEqual([...REFS, { source: CLIP_VIDEO, media_type: "video" }]);
});
it("numbers a second video reference after the first", () => {
const withOne = attachClip();
const r = attachLastClip(withOne.prompt, withOne.references, "/refs/another.mp4");
if (!r.ok) throw new Error(r.error);
expect(r.label).toBe("<Video 2>");
});
it("declares the video standalone, not bound to a <Subject> the way pictures and audio are", () => {
const defs = readField(attachClip().prompt, "subject_definitions") as string;
expect(defs).toContain("<Video 1> is the source video for [Shot 1]");
expect(defs).not.toMatch(/<Subject \d+> is .*<Video 1>/);
});
it("adds a retention line with the fully_preserved status", () => {
const line = (readField(attachClip().prompt, "retention_analysis") as string).split("\n").at(-1);
expect(line).toBe(
"<Video 1> (continuation source): fully_preserved - the motion, scene, positions, and lighting of the previous clip's ending are continued exactly.",
);
});
it("prefixes the first shot with a continuation lead, keeping hand-written shots verbatim otherwise", () => {
const hand = CLIP.replace(
/\[Shot 1\][^\n]*\n\[Shot 2\][^\n]*/,
"[Shot 1] A handheld medium shot holds <Subject 1> at frame left.\n[Shot 2] At 00:03.800, the shot cuts to a low tracking shot.",
);
const shots = (readField(attachClip(hand).prompt, "detailed_description") as string).split("\n");
expect(shots).toEqual([
"[Shot 1] The shot continues from <Video 1>. A handheld medium shot holds <Subject 1> at frame left.",
"[Shot 2] At 00:03.800, the shot cuts to a low tracking shot.",
]);
});
describe("summary wording", () => {
const summaryOf = (summary: string) =>
readField(attachClip(CLIP.replace("[reference generation] The target video shows <Subject 1> and <Subject 2> at a lunch.", summary)).prompt, "summary") as string;
it("adds the tag when the summary has none", () => {
expect(summaryOf("Two men at a lunch.")).toBe("[video continuation] Two men at a lunch.");
});
it("replaces reference generation, not adds to it", () => {
expect(summaryOf("[reference generation] Two men.")).toBe("[video continuation] Two men.");
});
it("keeps a combined audio reference tag, per the guide's documented combination", () => {
expect(summaryOf("[reference generation + audio reference] Two men.")).toBe("[video continuation + audio reference] Two men.");
});
it("drops keyframe completion when switching from a picture-based continuation", () => {
expect(summaryOf("[reference generation + keyframe completion] Two men.")).toBe("[video continuation] Two men.");
});
});
describe("refusals", () => {
it("won't attach the same clip twice", () => {
const r = attachLastClip(CLIP, [...REFS, { source: CLIP_VIDEO, media_type: "video" }], CLIP_VIDEO);
expect(r).toEqual({ ok: false, error: "That clip is already attached to this job." });
});
it("refuses a prompt that is not in the six-section format", () => {
const r = attachLastClip("A plain prompt.", REFS, CLIP_VIDEO);
expect(r).toEqual({ ok: false, error: "This prompt isn't in the six-section format, so it can't be rewritten automatically. Attach the clip in Edit instead." });
});
it("refuses past H3's video reference limit", () => {
const many = Array.from({ length: 3 }, (_, i) => ({ source: `/v${i}.mp4`, media_type: "video" }));
const r = attachLastClip(CLIP, [...REFS, ...many], CLIP_VIDEO);
expect(r).toEqual({ ok: false, error: "A job can have at most 3 video references." });
});
it("refuses a prompt with no shots", () => {
const noShots = CLIP.replace(/detailed_description:\n[\s\S]*?\n\noverall_soundscape:/, "detailed_description:\n\noverall_soundscape:");
const r = attachLastClip(noShots, REFS, CLIP_VIDEO);
expect(r).toEqual({ ok: false, error: "This prompt has no shots to continue from the clip." });
});
});
});
describe("recognizing an attached last clip", () => {
it("reads the job id out of the saved path", () => {
expect(lastClipSourceJobId(CLIP_VIDEO)).toBe("job-5");
expect(lastClipSourceJobId("last_clip_abc-123.mp4")).toBe("abc-123");
expect(lastClipSourceJobId("/refs/veteran.png")).toBeNull();
});
it("reads the job id out of a reference the server resolves when the job runs", () => {
expect(deferredLastClipSource("job-5")).toBe("job-last-clip:job-5");
expect(lastClipSourceJobId(deferredLastClipSource("job-5"))).toBe("job-5");
});
it("finds last-clip references among a job's references, separately from last-frame ones", () => {
const frame = { source: "/data/uploads/last_frames/last_frame_job-2.png", media_type: "image" };
expect(attachedLastClips([...REFS, frame, { source: CLIP_VIDEO, media_type: "video" }])).toEqual([
{ source: CLIP_VIDEO, jobId: "job-5" },
]);
expect(attachedLastClips([frame])).toEqual([]);
expect(attachedLastClips(null)).toEqual([]);
});
});
describe("whyCannotAttachClip", () => {
it("blocks a prompt not in the six-section format", () => {
expect(whyCannotAttachClip("A plain prompt.", REFS)).toMatch(/six-section/);
});
it("allows a normal prompt under the video limit", () => {
expect(whyCannotAttachClip(CLIP, REFS)).toBeNull();
});
});
describe("detachLastClip", () => {
const roundTrip = (prompt: string, clip = CLIP_VIDEO) => {
const attached = attachClip(prompt, REFS, clip);
const r = detachLastClip(attached.prompt, attached.references, clip);
if (!r.ok) throw new Error(r.error);
return r;
};
it("takes back exactly what attaching added", () => {
const r = roundTrip(CLIP);
expect(r.references).toEqual(REFS);
expect(r.prompt).toBe(CLIP);
});
it("works for a clip the server resolves when the job runs", () => {
expect(roundTrip(CLIP, deferredLastClipSource("job-5")).prompt).toBe(CLIP);
});
it("round-trips a starting tag combined with audio reference", () => {
// A realistic starting point: audio references already attached (see continuation.test.ts's
// own audio-reference wiring), before any video continuation is attached.
const prompt = CLIP.replace(
"[reference generation] The target video shows <Subject 1> and <Subject 2> at a lunch.",
"[reference generation + audio reference] Two men at a lunch.",
);
expect(roundTrip(prompt).prompt).toBe(prompt);
});
// Unlike attachLastFrame's "keyframe completion" (only ever added alongside the existing base
// tag), "video continuation" *replaces* the base tag -- so once attached, nothing remembers
// what was there before, and detaching lands on the default "reference generation", not back
// on "no tag".
it("restores a summary with no tag to the default reference-generation tag, not back to no tag", () => {
const prompt = CLIP.replace("[reference generation] The target video shows <Subject 1> and <Subject 2> at a lunch.", "Two men at a lunch.");
expect(roundTrip(prompt).prompt).toBe(CLIP.replace("The target video shows <Subject 1> and <Subject 2> at a lunch.", "Two men at a lunch."));
});
it("restores hand-written shots and an N/A section", () => {
const hand = CLIP.replace(/\[Shot 1\][^\n]*\n\[Shot 2\][^\n]*/, "[Shot 1] A handheld medium shot.\n[Shot 2] At 00:03.800, the shot cuts to a low tracking shot.").replace(
/subject_definitions:\n[\s\S]*?\n\nsummary:/,
"subject_definitions:\nN/A\n\nsummary:",
);
expect(roundTrip(hand).prompt).toBe(hand);
});
it("only removes the reference from a prompt that was never rewritten", () => {
const r = detachLastClip("A plain prompt.", [...REFS, { source: CLIP_VIDEO, media_type: "video" }], CLIP_VIDEO);
expect(r).toEqual({ ok: true, prompt: "A plain prompt.", references: REFS });
});
it("refuses when other videos were added after the clip", () => {
const attached = attachClip();
const r = detachLastClip(attached.prompt, [...attached.references, { source: "/refs/extra.mp4", media_type: "video" }], CLIP_VIDEO);
expect(r.ok).toBe(false);
});
it("refuses a clip that isn't attached", () => {
expect(detachLastClip(CLIP, REFS, CLIP_VIDEO)).toEqual({ ok: false, error: "That clip isn't attached to this job." });
});
it("does not remove a last-frame attachment, and vice versa", () => {
const withFrame = attach(CLIP, REFS, FRAME);
const withBoth = attachClip(withFrame.prompt, withFrame.references, CLIP_VIDEO);
const r = detachLastClip(withBoth.prompt, withBoth.references, CLIP_VIDEO);
if (!r.ok) throw new Error(r.error);
expect(attachedLastFrames(r.references)).toEqual([{ source: FRAME, jobId: "job-5" }]);
expect(attachedLastClips(r.references)).toEqual([]);
});
});
@@ -0,0 +1,334 @@
/**
* Start a clip from how the one before it ended -- either its last frame (a
* still) or its last second or so (a short video).
*
* A scene is generated as many short clips. To make one pick up where the
* previous ended, the previous clip's ending is attached as a reference, and
* the prompt is rewritten the way the prompt guide describes for a picture
* that anchors the opening frame (`attachLastFrame`) or a video that the shot
* continues from (`attachLastClip`). The wording is kept together here so it
* is easy to change if the guide is read differently.
*
* The two approaches trade off differently: a still is a hard, exact opening
* frame, but the model has only a pose to guess the continuing motion from. A
* short video carries real motion, at the cost of one more reference slot and
* (per H3's own limits) at most 3 per job instead of 9.
*/
import { readField, writeField } from "@/lib/h3PromptFields";
import { H3_REFERENCE_LIMITS } from "@/lib/h3References";
import {
detailedDescriptionFromShots,
newTextPart,
parseDetailedDescription,
} from "@/lib/h3Shots";
export interface Reference {
source: string;
media_type: string;
}
/**
* A reference that means "the last frame of that job's video", resolved by the
* server when this job runs -- so it can be attached before that job has
* finished, and the server's queue knows this job must wait for it.
*/
export const DEFERRED_LAST_FRAME_PREFIX = "job-last-frame:";
/** As above, but for the previous job's trailing video instead of its last frame. */
export const DEFERRED_LAST_CLIP_PREFIX = "job-last-clip:";
export const deferredLastFrameSource = (jobId: string) => `${DEFERRED_LAST_FRAME_PREFIX}${jobId}`;
export const deferredLastClipSource = (jobId: string) => `${DEFERRED_LAST_CLIP_PREFIX}${jobId}`;
/** Saved by the server as `.../last_frames/last_frame_<job id>.png`. */
const LAST_FRAME_RE = /(?:^|\/)last_frame_([^/]+)\.png$/;
/** Saved by the server as `.../last_clips/last_clip_<job id>.mp4`. */
const LAST_CLIP_RE = /(?:^|\/)last_clip_([^/]+)\.mp4$/;
const REQUIRED_SECTIONS = [
"subject_definitions",
"summary",
"retention_analysis",
"detailed_description",
] as const;
/** The job a last-frame reference came from, or null if `source` isn't one. */
export function lastFrameSourceJobId(source: string): string | null {
if (source.startsWith(DEFERRED_LAST_FRAME_PREFIX)) return source.slice(DEFERRED_LAST_FRAME_PREFIX.length) || null;
return source.match(LAST_FRAME_RE)?.[1] ?? null;
}
/** Last-frame references already on a job, with the job each came from. */
export function attachedLastFrames(references: Reference[] | null | undefined): { source: string; jobId: string }[] {
return (references ?? []).flatMap((r) => {
const jobId = lastFrameSourceJobId(r.source);
return jobId ? [{ source: r.source, jobId }] : [];
});
}
/** The job a last-clip reference came from, or null if `source` isn't one. */
export function lastClipSourceJobId(source: string): string | null {
if (source.startsWith(DEFERRED_LAST_CLIP_PREFIX)) return source.slice(DEFERRED_LAST_CLIP_PREFIX.length) || null;
return source.match(LAST_CLIP_RE)?.[1] ?? null;
}
/** Last-clip references already on a job, with the job each came from. */
export function attachedLastClips(references: Reference[] | null | undefined): { source: string; jobId: string }[] {
return (references ?? []).flatMap((r) => {
const jobId = lastClipSourceJobId(r.source);
return jobId ? [{ source: r.source, jobId }] : [];
});
}
/** Why a last frame can't be attached to this job, or null if it can. */
export function whyCannotAttach(prompt: string, references: Reference[] | null | undefined): string | null {
const missing = REQUIRED_SECTIONS.filter((s) => readField(prompt, s) === null);
if (missing.length > 0) {
return "This prompt isn't in the six-section format, so it can't be rewritten automatically. Attach the frame in Edit instead.";
}
const images = (references ?? []).filter((r) => r.media_type === "image").length;
if (images >= H3_REFERENCE_LIMITS.image) {
return `A job can have at most ${H3_REFERENCE_LIMITS.image} image references.`;
}
return null;
}
/** "[reference generation]" -> "[reference generation + keyframe completion]"; no tag -> "[keyframe completion] ...". */
function withKeyframeCompletion(summary: string): string {
const tag = summary.match(/^\[([^\]]*)\]\s*/);
if (!tag) return `[keyframe completion] ${summary}`.trim();
const types = tag[1].split("+").map((t) => t.trim()).filter(Boolean);
if (types.includes("keyframe completion")) return summary;
return `[${[...types, "keyframe completion"].join(" + ")}] ${summary.slice(tag[0].length)}`.trim();
}
const isBlank = (text: string) => !text.trim() || text.trim().toUpperCase() === "N/A";
const appendLine = (body: string, line: string) => (isBlank(body) ? line : `${body}\n${line}`);
export type AttachResult =
| { ok: true; prompt: string; references: Reference[]; label: string }
| { ok: false; error: string };
/**
* Add `framePath` as the next image reference and rewrite the prompt to use it
* as the opening frame of Shot 1. Fails, changing nothing, if the frame is
* already attached or the prompt isn't in a form that can be rewritten safely.
*/
export function attachLastFrame(prompt: string, references: Reference[], framePath: string): AttachResult {
if (references.some((r) => r.source === framePath)) {
return { ok: false, error: "That frame is already attached to this clip." };
}
const blocked = whyCannotAttach(prompt, references);
if (blocked) return { ok: false, error: blocked };
// Labels count per media type in reference order, so the new image is the next <Picture N>.
const label = `<Picture ${references.filter((r) => r.media_type === "image").length + 1}>`;
const { preamble, shots } = parseDetailedDescription(readField(prompt, "detailed_description") ?? "", []);
if (shots.length === 0) return { ok: false, error: "This prompt has no shots to start from the frame." };
const first = shots[0];
const firstText = first.body.find((p) => p.kind === "text");
if (firstText && firstText.kind === "text") firstText.text = `The shot begins from ${label}. ${firstText.text}`;
else first.body.unshift(newTextPart(`The shot begins from ${label}.`));
let next = prompt;
next = writeField(
next,
"subject_definitions",
appendLine(
readField(next, "subject_definitions") ?? "",
`${label} is the first frame of [Shot 1]: the last frame of the previous clip, showing the scene, positions, and lighting exactly as that clip ended.`,
),
);
next = writeField(
next,
"summary",
`${withKeyframeCompletion(readField(next, "summary") ?? "")} The target video starts from ${label}, the last frame of the previous clip.`,
);
next = writeField(
next,
"retention_analysis",
appendLine(
readField(next, "retention_analysis") ?? "",
`${label} ([Shot 1] first frame): fully_preserved - the scene, positions, and lighting of the previous clip's last frame are retained.`,
),
);
next = writeField(next, "detailed_description", detailedDescriptionFromShots(shots, preamble));
return { ok: true, prompt: next, references: [...references, { source: framePath, media_type: "image" }], label };
}
/** Drop the lines `attachLastFrame` added, leaving "N/A" if nothing else was there. */
function removeLine(body: string, startsWith: string): string {
const kept = body.split("\n").filter((l) => !l.trim().startsWith(startsWith));
const text = kept.join("\n").trim();
return text || "N/A";
}
export type DetachResult = { ok: true; prompt: string; references: Reference[] } | { ok: false; error: string };
/**
* Undo `attachLastFrame`: remove the reference at `source` and take back what was
* added to the prompt for it. Only the most recently added image can be removed --
* removing an earlier one would renumber the pictures after it.
*/
export function detachLastFrame(prompt: string, references: Reference[], source: string): DetachResult {
const images = references.filter((r) => r.media_type === "image");
const index = images.findIndex((r) => r.source === source);
if (index < 0) return { ok: false, error: "That frame isn't attached to this clip." };
if (index !== images.length - 1) {
return { ok: false, error: "Other images were added after that frame. Remove it in Edit instead." };
}
const remaining = references.filter((r) => r.source !== source);
// A prompt that isn't in the six-section form was never rewritten; only the reference goes.
if (REQUIRED_SECTIONS.some((s) => readField(prompt, s) === null)) {
return { ok: true, prompt, references: remaining };
}
const label = `<Picture ${index + 1}>`;
let next = prompt;
next = writeField(next, "subject_definitions", removeLine(readField(next, "subject_definitions") ?? "", `${label} is the first frame of [Shot 1]`));
next = writeField(next, "retention_analysis", removeLine(readField(next, "retention_analysis") ?? "", `${label} ([Shot 1] first frame)`));
const summary = readField(next, "summary") ?? "";
const sentence = ` The target video starts from ${label}, the last frame of the previous clip.`;
if (summary.includes(sentence)) {
let restored = summary.replace(sentence, "");
const tag = restored.match(/^\[([^\]]*)\]\s*/);
if (tag) {
const types = tag[1].split("+").map((t) => t.trim()).filter((t) => t && t !== "keyframe completion");
restored = types.length > 0 ? `[${types.join(" + ")}] ${restored.slice(tag[0].length)}` : restored.slice(tag[0].length);
}
next = writeField(next, "summary", restored.trim());
}
const { preamble, shots } = parseDetailedDescription(readField(next, "detailed_description") ?? "", []);
const lead = `The shot begins from ${label}.`;
const first = shots[0];
const part = first?.body.find((p) => p.kind === "text" && p.text.startsWith(lead));
if (first && part && part.kind === "text") {
part.text = part.text.slice(lead.length).trimStart();
if (!part.text) first.body = first.body.filter((p) => p !== part);
next = writeField(next, "detailed_description", detailedDescriptionFromShots(shots, preamble));
}
return { ok: true, prompt: next, references: remaining };
}
/** Why a trailing clip can't be attached to this job, or null if it can. */
export function whyCannotAttachClip(prompt: string, references: Reference[] | null | undefined): string | null {
const missing = REQUIRED_SECTIONS.filter((s) => readField(prompt, s) === null);
if (missing.length > 0) {
return "This prompt isn't in the six-section format, so it can't be rewritten automatically. Attach the clip in Edit instead.";
}
const videos = (references ?? []).filter((r) => r.media_type === "video").length;
if (videos >= H3_REFERENCE_LIMITS.video) {
return `A job can have at most ${H3_REFERENCE_LIMITS.video} video references.`;
}
return null;
}
/** "[reference generation + audio reference]" -> "[video continuation + audio reference]"; drops "keyframe completion". */
function withVideoContinuation(summary: string): string {
const tag = summary.match(/^\[([^\]]*)\]\s*/);
const rest = tag ? summary.slice(tag[0].length) : summary;
const kept = tag
? tag[1].split("+").map((t) => t.trim()).filter((t) => t && t !== "reference generation" && t !== "keyframe completion")
: [];
if (!kept.includes("video continuation")) kept.unshift("video continuation");
return `[${kept.join(" + ")}] ${rest}`.trim();
}
/**
* Add `clipPath` as the next video reference and rewrite the prompt so Shot 1
* continues from it, per the guide's video-reference form: a standalone
* `<Video N>` declaration (videos aren't bound to a `<Subject>` the way
* pictures and audio are), a `[video continuation]` task tag, a retention
* line, and "the shot continues from <Video N>" in the first shot.
*/
export function attachLastClip(prompt: string, references: Reference[], clipPath: string): AttachResult {
if (references.some((r) => r.source === clipPath)) {
return { ok: false, error: "That clip is already attached to this job." };
}
const blocked = whyCannotAttachClip(prompt, references);
if (blocked) return { ok: false, error: blocked };
const label = `<Video ${references.filter((r) => r.media_type === "video").length + 1}>`;
const { preamble, shots } = parseDetailedDescription(readField(prompt, "detailed_description") ?? "", []);
if (shots.length === 0) return { ok: false, error: "This prompt has no shots to continue from the clip." };
const first = shots[0];
const lead = `The shot continues from ${label}.`;
const firstText = first.body.find((p) => p.kind === "text");
if (firstText && firstText.kind === "text") firstText.text = `${lead} ${firstText.text}`;
else first.body.unshift(newTextPart(lead));
let next = prompt;
next = writeField(
next,
"subject_definitions",
appendLine(
readField(next, "subject_definitions") ?? "",
`${label} is the source video for [Shot 1]: the last second of the previous clip, providing the continuation starting point and its motion.`,
),
);
next = writeField(next, "summary", withVideoContinuation(readField(next, "summary") ?? ""));
next = writeField(
next,
"retention_analysis",
appendLine(
readField(next, "retention_analysis") ?? "",
`${label} (continuation source): fully_preserved - the motion, scene, positions, and lighting of the previous clip's ending are continued exactly.`,
),
);
next = writeField(next, "detailed_description", detailedDescriptionFromShots(shots, preamble));
return { ok: true, prompt: next, references: [...references, { source: clipPath, media_type: "video" }], label };
}
/**
* Undo `attachLastClip`: remove the reference at `source` and take back what
* was added to the prompt for it. Only the most recently added video can be
* removed -- removing an earlier one would renumber the videos after it.
*/
export function detachLastClip(prompt: string, references: Reference[], source: string): DetachResult {
const videos = references.filter((r) => r.media_type === "video");
const index = videos.findIndex((r) => r.source === source);
if (index < 0) return { ok: false, error: "That clip isn't attached to this job." };
if (index !== videos.length - 1) {
return { ok: false, error: "Other videos were added after that clip. Remove it in Edit instead." };
}
const remaining = references.filter((r) => r.source !== source);
if (REQUIRED_SECTIONS.some((s) => readField(prompt, s) === null)) {
return { ok: true, prompt, references: remaining };
}
const label = `<Video ${index + 1}>`;
let next = prompt;
next = writeField(next, "subject_definitions", removeLine(readField(next, "subject_definitions") ?? "", `${label} is the source video for [Shot 1]`));
next = writeField(next, "retention_analysis", removeLine(readField(next, "retention_analysis") ?? "", `${label} (continuation source)`));
// "video continuation" replaced "reference generation" as the base tag (attachLastClip drops
// it outright, unlike "keyframe completion" which is only ever added alongside it) -- so
// undoing it means putting "reference generation" back, not just deleting the tag.
const summaryTag = (readField(next, "summary") ?? "").match(/^\[([^\]]*)\]\s*/);
if (summaryTag && summaryTag[1].split("+").map((t) => t.trim()).includes("video continuation")) {
const rest = (readField(next, "summary") ?? "").slice(summaryTag[0].length);
const types = summaryTag[1]
.split("+")
.map((t) => t.trim())
.map((t) => (t === "video continuation" ? "reference generation" : t));
next = writeField(next, "summary", `[${types.join(" + ")}] ${rest}`.trim());
}
const { preamble, shots } = parseDetailedDescription(readField(next, "detailed_description") ?? "", []);
const lead = `The shot continues from ${label}.`;
const first = shots[0];
const part = first?.body.find((p) => p.kind === "text" && p.text.startsWith(lead));
if (first && part && part.kind === "text") {
part.text = part.text.slice(lead.length).trimStart();
if (!part.text) first.body = first.body.filter((p) => p !== part);
next = writeField(next, "detailed_description", detailedDescriptionFromShots(shots, preamble));
}
return { ok: true, prompt: next, references: remaining };
}
@@ -0,0 +1,110 @@
import { describe, expect, it } from 'vitest';
import { applyGradeInPlace, isNeutralGrade, NEUTRAL_GRADE } from './grading';
function solidRgba(rgb: [number, number, number], count = 4, alpha = 255): Uint8ClampedArray {
const data = new Uint8ClampedArray(count * 4);
for (let i = 0; i < count; i++) {
data[i * 4] = rgb[0];
data[i * 4 + 1] = rgb[1];
data[i * 4 + 2] = rgb[2];
data[i * 4 + 3] = alpha;
}
return data;
}
function meanRgb(data: Uint8ClampedArray): [number, number, number] {
let r = 0, g = 0, b = 0;
const n = data.length / 4;
for (let i = 0; i < data.length; i += 4) {
r += data[i];
g += data[i + 1];
b += data[i + 2];
}
return [r / n, g / n, b / n];
}
describe('isNeutralGrade', () => {
it('is true only for brightness 0, contrast 1, saturation 1', () => {
expect(isNeutralGrade(0, 1, 1)).toBe(true);
expect(isNeutralGrade(NEUTRAL_GRADE.brightness, NEUTRAL_GRADE.contrast, NEUTRAL_GRADE.saturation)).toBe(true);
expect(isNeutralGrade(1, 1, 1)).toBe(false);
expect(isNeutralGrade(0, 1.1, 1)).toBe(false);
expect(isNeutralGrade(0, 1, 0.9)).toBe(false);
});
});
describe('applyGradeInPlace', () => {
it('leaves neutral values as a no-op', () => {
const data = solidRgba([200, 100, 50]);
const before = Array.from(data);
applyGradeInPlace(data, 0, 1, 1);
expect(Array.from(data)).toEqual(before);
});
it('leaves alpha untouched', () => {
const data = solidRgba([200, 100, 50], 1, 137);
applyGradeInPlace(data, 50, 1.5, 0.5);
expect(data[3]).toBe(137);
});
it('positive brightness lightens every channel', () => {
const data = solidRgba([200, 100, 50]);
const [r0, g0, b0] = meanRgb(data);
applyGradeInPlace(data, 50, 1, 1);
const [r1, g1, b1] = meanRgb(data);
expect(r1).toBeGreaterThan(r0);
expect(g1).toBeGreaterThan(g0);
expect(b1).toBeGreaterThan(b0);
});
it('negative brightness darkens', () => {
const data = solidRgba([200, 100, 50]);
const [r0, g0, b0] = meanRgb(data);
applyGradeInPlace(data, -50, 1, 1);
const [r1, g1, b1] = meanRgb(data);
expect(r1).toBeLessThan(r0);
expect(g1).toBeLessThan(g0);
expect(b1).toBeLessThan(b0);
});
it('clamps brightness at the top of the range instead of wrapping', () => {
const data = solidRgba([240, 240, 240]);
applyGradeInPlace(data, 100, 1, 1);
expect(Array.from(data.subarray(0, 3))).toEqual([255, 255, 255]);
});
it('clamps brightness at the bottom of the range instead of wrapping', () => {
const data = solidRgba([10, 10, 10]);
applyGradeInPlace(data, -100, 1, 1);
expect(Array.from(data.subarray(0, 3))).toEqual([0, 0, 0]);
});
it('zero saturation makes every channel converge on the luma', () => {
const data = solidRgba([200, 100, 50]);
applyGradeInPlace(data, 0, 1, 0);
const [r, g, b] = meanRgb(data);
expect(Math.abs(r - g)).toBeLessThan(2);
expect(Math.abs(g - b)).toBeLessThan(2);
});
it('saturation above one widens the channel spread', () => {
const base = solidRgba([200, 100, 50]);
const [baseR, , baseB] = meanRgb(base);
const data = solidRgba([200, 100, 50]);
applyGradeInPlace(data, 0, 1, 2);
const [r, , b] = meanRgb(data);
expect(r - b).toBeGreaterThan(baseR - baseB);
});
it('high contrast pushes values away from mid-gray, in both directions', () => {
const base = solidRgba([200, 100, 50]);
const [baseR, baseG, baseB] = meanRgb(base);
const data = solidRgba([200, 100, 50]);
applyGradeInPlace(data, 0, 2, 1);
const [r, g, b] = meanRgb(data);
expect(r).toBeGreaterThan(baseR); // 200 sits above 128, pushed further up
expect(g).toBeLessThan(baseG); // 100 sits below 128, pushed further down
expect(b).toBeLessThan(baseB); // 50 sits below 128, pushed further down
});
});
+61
View File
@@ -0,0 +1,61 @@
/**
* Client-side mirror of trim.py's `_apply_grade`, so the "before/after" preview in
* the Edit video dialog can show the effect of a color grade instantly, without a
* server round trip. Same bounds, same order of operations (saturation, then
* contrast, then brightness). Not pixel-exact with the server's numpy path --
* `Uint8ClampedArray` rounds to the nearest integer on assignment where numpy
* truncates -- but the difference is well under a visible unit per channel, and
* this is only ever a preview; the server's render (via `POST .../trim`) is what
* actually gets saved.
*/
export const BRIGHTNESS_BOUNDS = [-100, 100] as const;
export const CONTRAST_BOUNDS = [0, 3] as const;
export const SATURATION_BOUNDS = [0, 3] as const;
export const NEUTRAL_GRADE = { brightness: 0, contrast: 1, saturation: 1 } as const;
// Perceptual luma weights, matching trim.py's _LUMA_WEIGHTS.
const LUMA_R = 0.2126;
const LUMA_G = 0.7152;
const LUMA_B = 0.0722;
/** True for the values that leave every pixel unchanged -- mirrors trim.py's `graded` flag. */
export function isNeutralGrade(brightness: number, contrast: number, saturation: number): boolean {
return brightness === NEUTRAL_GRADE.brightness && contrast === NEUTRAL_GRADE.contrast && saturation === NEUTRAL_GRADE.saturation;
}
/**
* Adjusts an RGBA buffer (as in `ImageData.data`) in place; alpha is left alone.
* Skipping the call entirely when `isNeutralGrade` is true (as callers should)
* avoids the per-pixel work for the common no-op case.
*/
export function applyGradeInPlace(data: Uint8ClampedArray, brightness: number, contrast: number, saturation: number): void {
for (let i = 0; i < data.length; i += 4) {
let r = data[i];
let g = data[i + 1];
let b = data[i + 2];
if (saturation !== 1) {
const luma = r * LUMA_R + g * LUMA_G + b * LUMA_B;
r = luma + (r - luma) * saturation;
g = luma + (g - luma) * saturation;
b = luma + (b - luma) * saturation;
}
if (contrast !== 1) {
r = (r - 128) * contrast + 128;
g = (g - 128) * contrast + 128;
b = (b - 128) * contrast + 128;
}
if (brightness !== 0) {
r += brightness;
g += brightness;
b += brightness;
}
// Assigning into a Uint8ClampedArray clamps to [0, 255] and rounds on its own.
data[i] = r;
data[i + 1] = g;
data[i + 2] = b;
}
}
@@ -0,0 +1,171 @@
/**
* Ballpark spoken-duration estimate for MiniMax-H3 dialogue text, plus the
* lexer/tree used to render a styled "performance preview" of the tags.
*
* Numbers here are heuristic (syllable counting + fixed per-tag seconds from
* `H3_TAGS[].seconds`), not measured against the model -- good enough to
* flag "this clip is clearly too long for one generation," calibrated
* against a real render via `calibrationFactor`.
*/
import { H3_TAGS, type H3Tag } from "@/lib/h3Tags";
/** Bare tag name, e.g. "<long pause>" -> "long pause". */
function bareName(bracketed: string): string {
return bracketed.replace(/[<>/]/g, "").trim();
}
const WRAP_TAG_NAMES = new Set(H3_TAGS.filter((t) => t.kind === "wrap").map((t) => bareName(t.open)));
const TAG_SECONDS: Record<string, number> = Object.fromEntries(
H3_TAGS.map((t) => [bareName(t.open), t.seconds ?? 0]),
);
/** Bare tag name -> its registry entry, so callers (the preview renderer) can look up category/description. */
export const TAG_BY_NAME: Record<string, H3Tag> = Object.fromEntries(H3_TAGS.map((t) => [bareName(t.open), t]));
/* -------------------------------------------------------------------- */
/* Lexer / tree — mirrors the tag registry so preview & estimate always */
/* agree with what the console/editor can actually insert. */
/* -------------------------------------------------------------------- */
export type LexToken =
| { t: "text"; v: string }
| { t: "point"; name: string }
| { t: "open"; name: string }
| { t: "close"; name: string };
const TAG_RE = /<(\/?)([a-zA-Z ]+?)>/g;
export function lexDialogue(text: string): LexToken[] {
const tokens: LexToken[] = [];
let last = 0;
let m: RegExpExecArray | null;
TAG_RE.lastIndex = 0;
while ((m = TAG_RE.exec(text))) {
if (m.index > last) tokens.push({ t: "text", v: text.slice(last, m.index) });
const closing = m[1] === "/";
const name = m[2].trim();
if (WRAP_TAG_NAMES.has(name)) tokens.push({ t: closing ? "close" : "open", name });
else if (!closing) tokens.push({ t: "point", name }); // stray/unknown closers are ignored
last = TAG_RE.lastIndex;
}
if (last < text.length) tokens.push({ t: "text", v: text.slice(last) });
return tokens;
}
export interface DialogueTextNode {
t: "text";
v: string;
}
export interface DialoguePointNode {
t: "point";
name: string;
}
export interface DialogueSpanNode {
t: "span";
name: string;
children: DialogueNode[];
}
export type DialogueNode = DialogueTextNode | DialoguePointNode | DialogueSpanNode;
export function dialogueTree(text: string): DialogueNode[] {
const root: { children: DialogueNode[] } = { children: [] };
const stack: { children: DialogueNode[] }[] = [root];
for (const tk of lexDialogue(text)) {
const top = stack[stack.length - 1];
if (tk.t === "text") top.children.push({ t: "text", v: tk.v });
else if (tk.t === "point") top.children.push({ t: "point", name: tk.name });
else if (tk.t === "open") {
const node: DialogueSpanNode = { t: "span", name: tk.name, children: [] };
top.children.push(node);
stack.push(node);
} else if (tk.t === "close" && stack.length > 1) stack.pop();
}
return root.children;
}
/* -------------------------------------------------------------------- */
/* Duration estimate */
/* -------------------------------------------------------------------- */
export const SPEECH_RATES = { Measured: 3.2, Natural: 4.0, Quick: 4.8 } as const; // syllables/sec
export type SpeechRate = keyof typeof SPEECH_RATES;
export function syllables(word: string): number {
let w = word.toLowerCase().replace(/[^a-z]/g, "");
if (!w) return 0;
if (w.length <= 3) return 1;
w = w.replace(/(?:[^laeiouy]es|ed|[^laeiouy]e)$/, "");
w = w.replace(/^y/, "");
const m = w.match(/[aeiouy]{1,2}/g);
return m ? m.length : 1;
}
export interface DurationEstimate {
total: number;
speech: number;
punct: number;
tags: number;
syl: number;
words: number;
low: number;
high: number;
}
export function estimateDuration(text: string, rate: number, calib: number): DurationEstimate {
const spoken = text.replace(/<\/?[a-zA-Z ]+?>/g, " ");
const words = spoken.split(/\s+/).filter(Boolean);
const syl = words.reduce((n, w) => n + syllables(w), 0);
const speech = syl / rate;
const sentence = (spoken.match(/[.!?]/g) || []).length * 0.35;
const commas = (spoken.match(/[,;:]/g) || []).length * 0.18;
const ellipsis = (spoken.match(/\.\.\.|…/g) || []).length * 0.5;
const dashes = (spoken.match(/—/g) || []).length * 0.2;
const punct = sentence + commas + ellipsis + dashes;
let tags = 0;
const re = /<(\/?)([a-zA-Z ]+?)>/g;
let m: RegExpExecArray | null;
while ((m = re.exec(text))) {
if (m[1] === "/") continue;
tags += TAG_SECONDS[m[2].trim()] ?? 0;
}
const total = (speech + punct + tags) * calib;
return {
total,
speech: speech * calib,
punct: punct * calib,
tags: tags * calib,
syl,
words: words.length,
low: total * 0.86,
high: total * 1.16,
};
}
/** Find the best sentence/pause boundary at which to split text that runs past `ceiling` seconds. */
export function suggestSplit(text: string, rate: number, calib: number, ceiling: number): number {
const re = /(<\/?[a-zA-Z ]+?>)|([.!?]+)|([,;:]+)|(\s+)|([^\s<.!?,;:]+)/g;
let cum = 0;
let boundary = 0;
let m: RegExpExecArray | null;
const budget = ceiling - 1.5;
while ((m = re.exec(text))) {
if (m[1]) {
const closing = m[1].startsWith("</");
const name = bareName(m[1]);
if (!closing) cum += (TAG_SECONDS[name] ?? 0) * calib;
if (name === "pause" || name === "long pause") boundary = re.lastIndex;
} else if (m[2]) {
cum += 0.35 * calib;
boundary = re.lastIndex;
} else if (m[3]) {
cum += 0.18 * calib;
} else if (m[5]) {
cum += (syllables(m[5]) / rate) * calib;
}
if (cum > budget && boundary > 0) return boundary;
}
return boundary > 0 ? boundary : -1;
}
@@ -0,0 +1,87 @@
import { describe, expect, it } from "vitest";
import { readField, shotTarget, writeField, writeShotTarget } from "@/lib/h3PromptFields";
const BASE = [
"For the target video, <Picture 1> is fully referenced.",
"",
"integrated_multimodal_description: [Shot 1] A close-up. He (S1) says: <d>[English] Hi.</d> <cutoff>",
"",
"overall_soundscape: Room tone.",
"",
"non_diegetic_music: N/A",
].join("\n");
const SECTIONS = [
"subject_definitions:",
"<Subject 1> is the courier.",
"",
"retention_analysis:",
"<Subject 1> (appears in [Shot 1]): fully_preserved - x.",
"",
"detailed_description:",
"[Shot 1] Wide shot. A rooftop.",
"[Shot 2] At 00:02.000, the shot cuts to a close-up shot. A face.",
"",
"overall_soundscape:",
"Rain.",
].join("\n");
describe("readField / writeField", () => {
it("reads a field whose text follows the heading on the same line", () => {
expect(readField(BASE, "integrated_multimodal_description")).toBe(
"[Shot 1] A close-up. He (S1) says: <d>[English] Hi.</d> <cutoff>",
);
});
it("reads a field whose text starts on the line after the heading", () => {
expect(readField(SECTIONS, "detailed_description")).toBe(
"[Shot 1] Wide shot. A rooftop.\n[Shot 2] At 00:02.000, the shot cuts to a close-up shot. A face.",
);
expect(readField(SECTIONS, "subject_definitions")).toBe("<Subject 1> is the courier.");
});
it("returns null for a field that isn't there", () => {
expect(readField(BASE, "detailed_description")).toBeNull();
});
it("rewrites only that field, leaving the rest byte-identical (inline style)", () => {
const out = writeField(BASE, "integrated_multimodal_description", "[Shot 1] New.");
expect(out).toBe(BASE.replace("[Shot 1] A close-up. He (S1) says: <d>[English] Hi.</d> <cutoff>", "[Shot 1] New."));
});
it("rewrites only that field (own-line style)", () => {
const out = writeField(SECTIONS, "detailed_description", "[Shot 1] Only.");
expect(out).toContain("detailed_description:\n[Shot 1] Only.\n\noverall_soundscape:\nRain.");
expect(out.startsWith("subject_definitions:\n<Subject 1> is the courier.")).toBe(true);
});
it("writing a field's own text back is a no-op", () => {
for (const [prompt, field] of [
[BASE, "integrated_multimodal_description"],
[SECTIONS, "detailed_description"],
[SECTIONS, "retention_analysis"],
] as const) {
expect(writeField(prompt, field, readField(prompt, field) as string)).toBe(prompt);
}
});
it("ignores a heading that isn't at the start of a line", () => {
expect(readField("see detailed_description: not a header", "detailed_description")).toBeNull();
});
});
describe("shotTarget", () => {
it("prefers integrated_multimodal_description", () => {
expect(shotTarget(BASE).field).toBe("integrated_multimodal_description");
});
it("falls back to detailed_description", () => {
expect(shotTarget(SECTIONS).field).toBe("detailed_description");
});
it("treats a plain prompt as the shot text itself", () => {
const t = shotTarget("A toy car drives into a plush dog.");
expect(t).toEqual({ field: null, body: "A toy car drives into a plush dog." });
expect(writeShotTarget("A toy car drives into a plush dog.", t, "[Shot 1] X.")).toBe("[Shot 1] X.");
});
});
@@ -0,0 +1,77 @@
/**
* Locate and rewrite a single named field inside a raw MiniMax-H3 prompt,
* leaving every other character of the prompt untouched. Handles both header
* styles seen in real prompts: the heading on its own line (six-section
* reference format) and the heading followed by its text on the same line
* (`integrated_multimodal_description: [Shot 1] ...`, the base format).
*/
const FIELD_NAMES = [
"subject_definitions",
"summary",
"retention_analysis",
"detailed_description",
"integrated_multimodal_description",
"overall_soundscape",
"non_diegetic_music",
] as const;
export type H3PromptFieldName = (typeof FIELD_NAMES)[number];
const HEADER_RE = new RegExp(`^(${FIELD_NAMES.join("|")}):[ \\t]*`, "gm");
interface Span {
start: number;
end: number;
}
function locate(prompt: string, field: H3PromptFieldName): Span | null {
const headers = [...prompt.matchAll(HEADER_RE)];
const i = headers.findIndex((h) => h[1] === field);
if (i < 0) return null;
const header = headers[i];
let start = (header.index ?? 0) + header[0].length;
// Own-line style: nothing follows the colon, so the body starts on the next line.
const newline = prompt.slice(start).match(/^\r?\n/);
if (newline) start += newline[0].length;
const nextHeader = i + 1 < headers.length ? (headers[i + 1].index ?? prompt.length) : prompt.length;
const end = start + Math.max(0, prompt.slice(start, nextHeader).trimEnd().length);
return { start, end: Math.max(start, end) };
}
/** The field's text exactly as written, or null if the prompt has no such field. */
export function readField(prompt: string, field: H3PromptFieldName): string | null {
const span = locate(prompt, field);
return span ? prompt.slice(span.start, span.end) : null;
}
/** Replace only the field's text. Returns the prompt unchanged if the field isn't there. */
export function writeField(prompt: string, field: H3PromptFieldName, body: string): string {
const span = locate(prompt, field);
return span ? prompt.slice(0, span.start) + body + prompt.slice(span.end) : prompt;
}
export interface ShotTarget {
/** the field the [Shot N] blocks live in, or null when the whole prompt is the shot text */
field: "integrated_multimodal_description" | "detailed_description" | null;
body: string;
}
/**
* Where a prompt's shots live: the base format's integrated_multimodal_description,
* else the reference format's detailed_description, else the prompt as a whole.
*/
export function shotTarget(prompt: string): ShotTarget {
for (const field of ["integrated_multimodal_description", "detailed_description"] as const) {
const body = readField(prompt, field);
if (body !== null) return { field, body };
}
return { field: null, body: prompt };
}
/** Write shot text back to wherever shotTarget found it. */
export function writeShotTarget(prompt: string, target: ShotTarget, body: string): string {
return target.field ? writeField(prompt, target.field, body) : body;
}
@@ -0,0 +1,253 @@
import { describe, expect, it } from "vitest";
import {
buildClips,
convertDirections,
parseScript,
scriptSpeakers,
type ImportSpeaker,
} from "@/lib/h3ScriptImport";
import { detailedDescriptionFromShots, shotLines, shotProse } from "@/lib/h3Shots";
// Original test material -- not from any real screenplay.
const SCRIPT = [
"INT. HARBOR OFFICE - DAY",
"",
"CAPTAIN: Sit down. (EXHALES) We need to talk about the ledger.",
"",
"CLERK: Yes, sir.",
"",
"CAPTAIN: Three crates are missing, and nobody on this dock saw a thing.",
"Not one of you. That does not happen by accident.",
"",
"CLERK: I counted twice.",
"",
"CAPTAIN: (WHISPERING) Then count a third time.",
"",
"CLERK: (TAPPING TABLE) Right away.",
].join("\n");
const SPEAKERS: Record<string, ImportSpeaker> = {
CAPTAIN: { subject: "<Subject 1>", tag: "S1" },
CLERK: { subject: "<Subject 2>", tag: "S2" },
};
describe("parseScript", () => {
it("reads NAME: line turns and ignores headings between turns", () => {
const turns = parseScript(SCRIPT);
expect(turns.map((t) => t.speaker)).toEqual(["CAPTAIN", "CLERK", "CAPTAIN", "CLERK", "CAPTAIN", "CLERK"]);
expect(turns.some((t) => t.text.includes("HARBOR"))).toBe(false);
});
it("joins continuation lines directly under a turn", () => {
const turns = parseScript(SCRIPT);
expect(turns[2].text).toBe(
"Three crates are missing, and nobody on this dock saw a thing. Not one of you. That does not happen by accident.",
);
});
it("tolerates markdown bold speaker names", () => {
expect(parseScript("**MARK:** hello there")).toEqual([{ speaker: "MARK", text: "hello there" }]);
});
it("lists speakers in order of first appearance", () => {
expect(scriptSpeakers(parseScript(SCRIPT))).toEqual(["CAPTAIN", "CLERK"]);
});
it("returns nothing for text with no speaker turns", () => {
expect(parseScript("just some prose without turns")).toEqual([]);
});
});
describe("convertDirections", () => {
it("turns known parentheticals into speech tags in place", () => {
expect(convertDirections("Sit down. (EXHALES) We talk.").text).toBe("Sit down. <exhale> We talk.");
expect(convertDirections("Well (BEAT) fine.").text).toBe("Well <pause> fine.");
});
it("wraps the line for a whispering direction", () => {
expect(convertDirections("(WHISPERING) Keep it down.").text).toBe("<whisper>Keep it down.</whisper>");
});
it("keeps unmapped directions out of the dialogue but reports them", () => {
const r = convertDirections("(TAPPING TABLE) Right away.");
expect(r.text).toBe("Right away.");
expect(r.directions).toEqual(["tapping table"]);
});
});
describe("buildClips", () => {
it("opens the first clip with an establishing shot", () => {
const [clip] = buildClips(parseScript(SCRIPT), { speakers: SPEAKERS, targetSeconds: 60 });
expect(clip.shots[0].type).toBe("establishing");
expect(shotLines(clip.shots[0])).toHaveLength(0);
expect(clip.shots[0].subjectIds).toEqual(["<Subject 1>", "<Subject 2>"]);
});
it("folds a short reaction into the shot it reacts to", () => {
const [clip] = buildClips(parseScript(SCRIPT), { speakers: SPEAKERS, targetSeconds: 60 });
const first = clip.shots[1];
expect(shotLines(first).map((l) => [l.speaker, l.text])).toEqual([
["S1", "Sit down. <exhale> We need to talk about the ledger."],
["S2", "Yes, sir."],
]);
expect(first.type).toBe("medium");
expect(first.subjectIds).toEqual(["<Subject 1>", "<Subject 2>"]);
});
it("never adds camera movement, however long the shot runs", () => {
const long = "We keep the clients on the wheel and the wheel keeps turning, every day of the week. ".repeat(4).trim();
const clips = buildClips(
[
{ speaker: "CAPTAIN", text: long },
{ speaker: "CLERK", text: long },
{ speaker: "CAPTAIN", text: "Yes." },
],
{ speakers: SPEAKERS, targetSeconds: 60 },
);
const shots = clips.flatMap((c) => c.shots);
expect(shots.length).toBeGreaterThan(2);
expect(shots.every((s) => s.movement === "static")).toBe(true);
});
it("uses a close-up when only one person speaks in a shot", () => {
const [clip] = buildClips([{ speaker: "CAPTAIN", text: "Nobody leaves until the count is done." }], {
speakers: SPEAKERS,
targetSeconds: 60,
});
const shot = clip.shots[1];
expect(shot.type).toBe("close-up");
expect(shot.subjectIds).toEqual(["<Subject 1>"]);
expect(shotProse(shot)).toBe("Close on <Subject 1>, speaking.");
});
it("records unmapped stage directions on the shot they belong to", () => {
const [clip] = buildClips(parseScript(SCRIPT), { speakers: SPEAKERS, targetSeconds: 60 });
const last = clip.shots[clip.shots.length - 1];
expect(shotLines(last).map((l) => l.text)).toEqual(["<whisper>Then count a third time.</whisper>", "Right away."]);
expect(shotProse(last)).toContain("Stage direction: tapping table.");
});
it("splits a long scene into several clips, each with dialogue", () => {
const clips = buildClips(parseScript(SCRIPT), { speakers: SPEAKERS, targetSeconds: 5 });
expect(clips.length).toBeGreaterThan(1);
for (const c of clips) expect(c.shots.some((s) => shotLines(s).length > 0)).toBe(true);
expect(clips[1].shots[0].type).not.toBe("establishing");
});
it("keeps every spoken word across clips", () => {
const clips = buildClips(parseScript(SCRIPT), { speakers: SPEAKERS, targetSeconds: 5 });
const all = clips.flatMap((c) => c.shots.flatMap((s) => shotLines(s).map((l) => l.text))).join(" ");
for (const word of ["ledger", "crates", "accident", "counted", "third", "Right away"]) {
expect(all).toContain(word);
}
});
it("serializes to valid [Shot N] text with speaker-tagged dialogue", () => {
const [clip] = buildClips(parseScript(SCRIPT), { speakers: SPEAKERS, targetSeconds: 60 });
const text = detailedDescriptionFromShots(clip.shots);
expect(text).toContain("[Shot 1] Establishing shot.");
expect(text).toContain("(S1) <d>[English] Sit down. <exhale> We need to talk about the ledger.</d>");
expect(text).toContain("(S2) <d>[English] Yes, sir.</d>");
});
describe("speaker tags per clip", () => {
it("numbers speakers by who speaks first in each clip, not across the whole script", () => {
const turns = [
{ speaker: "CAPTAIN", text: "Every one of these lines is long enough to stand as its own turn today." },
{ speaker: "CLERK", text: "And every reply here is long enough to stand as its own turn as well." },
{ speaker: "CLERK", text: "The clerk speaks again first when the next clip begins, with a full sentence." },
{ speaker: "CAPTAIN", text: "And then the captain answers the clerk with another complete sentence here." },
];
const clips = buildClips(turns, { speakers: SPEAKERS, targetSeconds: 8 });
expect(clips.length).toBeGreaterThan(1);
for (const c of clips) {
const first = c.shots.flatMap((s) => shotLines(s))[0];
expect(first.speaker).toBe("S1");
}
// a clip that opens on the clerk maps the clerk to S1 and the captain to S2
const clerkFirst = clips.find((c) => shotLines(c.shots.find((s) => shotLines(s).length)!)[0].text.startsWith("The clerk"));
const speakers = clerkFirst!.shots.flatMap((s) => shotLines(s).map((l) => l.speaker));
expect(new Set(speakers).size).toBeLessThanOrEqual(2);
});
it("leaves custom speaker tags alone", () => {
const clips = buildClips(
[{ speaker: "CLERK", text: "A line for the clerk that is long enough to stand alone here." }],
{ speakers: { CLERK: { subject: "<Subject 2>", tag: "CLERK" }, CAPTAIN: SPEAKERS.CAPTAIN }, targetSeconds: 11 },
);
expect(shotLines(clips[0].shots[1])[0].speaker).toBe("CLERK");
});
});
describe("clip boundaries", () => {
const longSpeech = Array.from({ length: 12 }, (_, i) => `This is sentence number ${i + 1} of a long speech.`).join(" ");
it("keeps a long turn as one continuous shot when a clip can hold it", () => {
const clips = buildClips([{ speaker: "CAPTAIN", text: longSpeech }], { speakers: SPEAKERS, targetSeconds: 60 });
expect(clips).toHaveLength(1);
const dialogueShots = clips[0].shots.filter((s) => shotLines(s).length > 0);
expect(dialogueShots).toHaveLength(1);
expect(shotLines(dialogueShots[0])[0].text).toBe(longSpeech);
});
it("only splits inside a shot when the turn is longer than a whole clip, and marks the follow-on clips", () => {
const clips = buildClips([{ speaker: "CAPTAIN", text: longSpeech }], { speakers: SPEAKERS, targetSeconds: 11 });
expect(clips.length).toBeGreaterThan(1);
expect(clips[0].continuation).toBe(false);
for (const c of clips.slice(1)) {
expect(c.continuation).toBe(true);
expect(shotProse(c.shots[0])).toContain("Continuing the same shot");
expect(c.shots[0].type).toBe("close-up");
}
const spoken = clips.flatMap((c) => c.shots.flatMap((s) => shotLines(s).map((l) => l.text))).join(" ");
expect(spoken).toBe(longSpeech);
});
it("marks speech that runs past the end of a clip with <cutoff>, and only there", () => {
const clips = buildClips([{ speaker: "CAPTAIN", text: longSpeech }], { speakers: SPEAKERS, targetSeconds: 11 });
const endings = clips.map((c) => detailedDescriptionFromShots(c.shots));
for (const text of endings.slice(0, -1)) expect(text).toMatch(/<\/d> <cutoff>$/);
expect(endings[endings.length - 1]).not.toContain("<cutoff>");
// the marker is outside the spoken text, so it never leaks into the dialogue itself
const spoken = clips.flatMap((c) => c.shots.flatMap((s) => shotLines(s).map((l) => l.text))).join(" ");
expect(spoken).not.toContain("cutoff");
});
it("breaks between separate turns at a cut, not as a continuation", () => {
const turns = Array.from({ length: 6 }, () => ({
speaker: "CAPTAIN",
text: "Every one of these turns is a complete thought that takes a while to say aloud.",
}));
const clips = buildClips(turns, { speakers: SPEAKERS, targetSeconds: 11 });
expect(clips.length).toBeGreaterThan(1);
expect(clips.every((c) => !c.continuation)).toBe(true);
});
it("re-wraps a whispered line on every piece so no tag is left unmatched", () => {
const clips = buildClips([{ speaker: "CAPTAIN", text: `(WHISPERING) ${longSpeech}` }], {
speakers: SPEAKERS,
targetSeconds: 11,
});
const texts = clips.flatMap((c) => c.shots.flatMap((s) => shotLines(s).map((l) => l.text)));
expect(texts.length).toBeGreaterThan(1);
for (const t of texts) {
expect(t.startsWith("<whisper>")).toBe(true);
expect(t.endsWith("</whisper>")).toBe(true);
}
});
it("puts a trailing reaction with the last piece of a split speech", () => {
const clips = buildClips(
[
{ speaker: "CAPTAIN", text: longSpeech },
{ speaker: "CLERK", text: "Understood." },
],
{ speakers: SPEAKERS, targetSeconds: 11 },
);
const lastShot = clips[clips.length - 1].shots.at(-1)!;
expect(shotLines(lastShot).map((l) => l.speaker)).toEqual(["S1", "S2"]);
const earlier = clips.slice(0, -1).flatMap((c) => c.shots.flatMap((s) => shotLines(s).map((l) => l.speaker)));
expect(earlier.every((sp) => sp === "S1")).toBe(true);
});
});
});
@@ -0,0 +1,328 @@
/**
* Turns a plain dialogue script ("NAME: line" turns) into clip-sized shot
* lists for the shot editor. Mechanical work only -- speakers, dialogue,
* stage-direction-to-tag mapping, grouping turns into shots, and splitting
* the whole scene into clips that fit one generation. Camera choices are
* defaults for the user to refine; a script says nothing about them.
*/
import { SPEECH_RATES, estimateDuration } from "@/lib/h3DialogueEstimate";
import { estimateSpokenSeconds, makeShot, newLine, newTextPart, shotLines, type Shot } from "@/lib/h3Shots";
import { H3_TAGS } from "@/lib/h3Tags";
export interface ScriptTurn {
speaker: string;
text: string;
}
// "MARK: ..." -- 1-3 capitalized words, so prose like "note: ..." or scene
// headings without a colon are never mistaken for a speaker.
const SPEAKER_LINE_RE = /^([A-Z][A-Za-z0-9.'_-]*(?: [A-Z][A-Za-z0-9.'_-]*){0,2}):\s*(.*)$/;
function stripMarkdown(line: string): string {
return line
.replace(/^\s*(?:[>*-]\s+)+/, "")
.replace(/\*\*|__/g, "")
.trim();
}
/**
* A speaker line starts a turn; non-blank lines directly after it continue
* that turn. A blank line ends the turn, so scene headings and action lines
* between turns are ignored rather than glued onto someone's dialogue.
*/
export function parseScript(script: string): ScriptTurn[] {
const turns: ScriptTurn[] = [];
let continuing = false;
for (const raw of script.split(/\r?\n/)) {
const line = stripMarkdown(raw);
if (!line) {
continuing = false;
continue;
}
const m = line.match(SPEAKER_LINE_RE);
if (m) {
turns.push({ speaker: m[1], text: m[2].trim() });
continuing = true;
} else if (continuing && turns.length > 0) {
const last = turns[turns.length - 1];
last.text = `${last.text} ${line}`.trim();
}
}
return turns.filter((t) => t.text);
}
/** Speaker names in order of first appearance. */
export function scriptSpeakers(turns: ScriptTurn[]): string[] {
const seen: string[] = [];
for (const t of turns) if (!seen.includes(t.speaker)) seen.push(t.speaker);
return seen;
}
/* ---------------------------------------------------------------- */
/* Stage directions -> speech tags */
/* ---------------------------------------------------------------- */
const TAG_BY_ID = Object.fromEntries(H3_TAGS.map((t) => [t.id, t]));
// Order matters: the more specific phrase must be tried before its substring.
const DIRECTION_TAGS: [RegExp, string][] = [
[/\blong pause\b/, "long-pause"],
[/\bcatch(?:es)? (?:his |her |their )?breath\b/, "catches-breath"],
[/\bexhal/, "exhale"],
[/\binhal/, "inhale"],
[/\bsigh/, "sighs"],
[/\blaugh/, "laughs"],
[/\bchuckl/, "chuckle"],
[/\bgasp/, "gasp"],
[/\bcough/, "coughs"],
[/\bclears? (?:his |her |their )?throat\b/, "clears-throat"],
[/\bsniff/, "sniff"],
[/\b(?:pause|beat)\b/, "pause"],
[/\bstutter|\bstammer/, "stutter"],
[/\bbreath/, "breath"],
];
export interface ConvertedLine {
text: string;
/** parentheticals with no matching tag, kept as stage directions for the shot description */
directions: string[];
}
/** Rewrites "(EXHALES)"-style parentheticals as speech tags; "(WHISPERING)" wraps the whole line. */
export function convertDirections(text: string): ConvertedLine {
const directions: string[] = [];
let wrapWhisper = false;
let out = text.replace(/\(([^)]*)\)/g, (_m, inner: string) => {
const key = inner.trim().toLowerCase();
if (/\bwhisper/.test(key)) {
wrapWhisper = true;
return " ";
}
for (const [re, id] of DIRECTION_TAGS) {
const tag = TAG_BY_ID[id];
if (tag && tag.kind === "point" && re.test(key)) return ` ${tag.open} `;
}
if (key) directions.push(key);
return " ";
});
out = out
.replace(/\s+/g, " ")
.replace(/\s+([.,!?…])/g, "$1")
.trim();
const whisper = TAG_BY_ID.whisper;
if (wrapWhisper && whisper?.close && out) out = `${whisper.open}${out}${whisper.close}`;
return { text: out, directions };
}
/* ---------------------------------------------------------------- */
/* Shots and clips */
/* ---------------------------------------------------------------- */
export interface ImportSpeaker {
/** <Subject N> label this speaker is, or null when they aren't a tracked subject */
subject: string | null;
/** speaker tag written as "(S1)" */
tag: string;
}
export interface ImportOptions {
speakers: Record<string, ImportSpeaker>;
/** rough length one clip should fit, in seconds */
targetSeconds: number;
}
export interface ImportClip {
shots: Shot[];
/**
* True when this clip picks up in the middle of the previous clip's last
* shot (a speech too long for one generation). Its first shot continues
* that one instead of cutting to something new, so it should be generated
* from the previous clip's last frame. False means the boundary is a cut.
*/
continuation: boolean;
}
const ESTABLISHING_SECONDS = 1.5;
const REACTION_MAX_WORDS = 4;
// Space left in a clip is only worth starting a forced mid-shot split in if
// it can hold at least this much of the speech.
const MIN_PART_SECONDS = 3;
function spokenSeconds(text: string): number {
return estimateDuration(text, SPEECH_RATES.Natural, 1).total;
}
function wordCount(text: string): number {
return (text.replace(/<[^>]*>/g, " ").match(/[\p{L}\p{N}'’-]+/gu) ?? []).length;
}
/**
* Split one over-long line at sentence boundaries into parts that fit the
* given budgets (first part, then every later part). An outer wrapping tag
* like <whisper>...</whisper> is re-applied to each part so none is left
* with an unmatched tag.
*/
function splitToBudgets(text: string, firstBudget: number, laterBudget: number): string[] {
const wrap = text.match(/^<([a-z ]+)>([\s\S]*)<\/\1>$/);
const inner = wrap ? wrap[2] : text;
const sentences = inner.match(/[^.!?…]+(?:[.!?…]+|$)\s*/g) ?? [inner];
const parts: string[] = [];
let current = "";
let budget = firstBudget;
for (const sentence of sentences) {
const candidate = `${current}${sentence}`;
if (current && spokenSeconds(candidate) > budget) {
parts.push(current.trim());
current = sentence;
budget = laterBudget;
} else {
current = candidate;
}
}
if (current.trim()) parts.push(current.trim());
return wrap ? parts.map((p) => `<${wrap[1]}>${p}</${wrap[1]}>`) : parts;
}
interface Beat {
/** the first line is the turn the shot is about; later lines are short reactions */
lines: { speaker: string; text: string }[];
directions: string[];
}
function beatSeconds(beat: Beat): number {
return estimateSpokenSeconds(beat.lines.map((l) => l.text));
}
function titleCase(name: string): string {
return name.toLowerCase().replace(/(^|[\s'-])(\p{L})/gu, (_m, sep: string, ch: string) => sep + ch.toUpperCase());
}
function joinNatural(items: string[]): string {
return items.length <= 2 ? items.join(" and ") : `${items.slice(0, -1).join(", ")}, and ${items[items.length - 1]}`;
}
function shotFromBeat(beat: Beat, opts: ImportOptions, continued = false, truncated = false): Shot {
const speakers = [...new Set(beat.lines.map((l) => l.speaker))];
const subjectIds = [
...new Set(speakers.map((s) => opts.speakers[s]?.subject).filter((s): s is string => Boolean(s))),
];
const named = speakers.map((s) => opts.speakers[s]?.subject ?? titleCase(s));
let description: string;
if (continued) {
description = `Continuing the same shot on ${joinNatural(named)}, still speaking.`;
} else {
description =
speakers.length > 1
? `${joinNatural(named)} in frame, exchanging lines.`
: `Close on ${named[0]}, speaking.`;
}
if (beat.directions.length) description += ` Stage direction: ${beat.directions.join("; ")}.`;
const shot = makeShot({
type: speakers.length > 1 ? "medium" : "close-up",
// A static camera: added movement (a dolly on every long shot) showed up as
// unwanted panning. Movement can still be set per shot in the shot editor.
movement: "static",
subjectIds,
description,
lines: beat.lines.map((l) => newLine({ speaker: opts.speakers[l.speaker]?.tag ?? "", text: l.text })),
});
// The guide's marker for speech truncated by the video ending: this clip
// stops mid-speech and the next one picks it up.
if (truncated) shot.body.push(newTextPart("<cutoff>"));
return shot;
}
/** One shot per speaking turn; a short reaction ("Yeah.") rides along in the shot it reacts to. */
function groupBeats(turns: ScriptTurn[]): Beat[] {
const beats: Beat[] = [];
for (const turn of turns) {
const { text, directions } = convertDirections(turn.text);
if (!text) continue;
const last = beats[beats.length - 1];
if (last && wordCount(text) <= REACTION_MAX_WORDS) {
last.lines.push({ speaker: turn.speaker, text });
last.directions.push(...directions);
} else {
beats.push({ lines: [{ speaker: turn.speaker, text }], directions: [...directions] });
}
}
return beats;
}
/**
* The guide assigns speaker IDs by who first speaks in *each video*, so a
* clip that opens on the second character must call them S1. Renumber a
* clip's default S1/S2/... tags in order of first appearance; custom tags
* (anything not of the form S<n>) are left alone.
*/
function renumberSpeakerTags(clip: ImportClip): void {
const lines = clip.shots.flatMap((s) => shotLines(s));
const seen: string[] = [];
for (const l of lines) if (l.speaker && !seen.includes(l.speaker)) seen.push(l.speaker);
if (seen.length === 0 || !seen.every((t) => /^S\d+$/.test(t))) return;
const renamed = new Map(seen.map((t, i) => [t, `S${i + 1}`]));
for (const l of lines) if (l.speaker) l.speaker = renamed.get(l.speaker) ?? l.speaker;
}
/**
* Pack shots into clips that fit `targetSeconds`. Clips break only at cuts
* (a cut should bring new information), so a turn is never divided just to
* balance lengths. The one exception is a single turn longer than a whole
* clip: it is split at sentence breaks, and each later piece starts a
* `continuation` clip. The first clip opens with an establishing shot.
*/
export function buildClips(turns: ScriptTurn[], opts: ImportOptions): ImportClip[] {
const beats = groupBeats(turns);
const allSubjects = [
...new Set(Object.values(opts.speakers).map((s) => s.subject).filter((s): s is string => Boolean(s))),
];
const establishing = makeShot({
type: "establishing",
subjectIds: allSubjects,
description: `Establishing view of the setting${allSubjects.length ? ` with ${joinNatural(allSubjects)} in frame` : ""}.`,
});
const clips: ImportClip[] = [];
let current: ImportClip = { shots: [establishing], continuation: false };
let seconds = ESTABLISHING_SECONDS;
const hasDialogue = () => current.shots.some((s) => shotLines(s).length > 0);
const close = (continuation: boolean) => {
clips.push(current);
current = { shots: [], continuation };
seconds = 0;
};
for (const beat of beats) {
const total = beatSeconds(beat);
if (total <= opts.targetSeconds) {
if (hasDialogue() && seconds + total > opts.targetSeconds) close(false);
current.shots.push(shotFromBeat(beat, opts));
seconds += total;
continue;
}
// Longer than any clip can hold: the split has to fall inside this shot.
if (current.shots.length > 0 && opts.targetSeconds - seconds < MIN_PART_SECONDS) close(false);
const [main, ...reactions] = beat.lines;
const parts = splitToBudgets(main.text, opts.targetSeconds - seconds, opts.targetSeconds);
parts.forEach((part, k) => {
if (k > 0) close(true);
const lines = [{ ...main, text: part }, ...(k === parts.length - 1 ? reactions : [])];
const pieceBeat: Beat = { lines, directions: k === 0 ? beat.directions : [] };
current.shots.push(shotFromBeat(pieceBeat, opts, k > 0, k < parts.length - 1));
seconds += beatSeconds(pieceBeat);
});
}
if (current.shots.length > 0 && (clips.length === 0 || hasDialogue())) clips.push(current);
for (const clip of clips) renumberSpeakerTags(clip);
return clips;
}
@@ -0,0 +1,336 @@
import { describe, expect, it } from "vitest";
import {
detailedDescriptionFromShots,
formatCutTime,
makeShot,
newLine,
parseCutTime,
parseDetailedDescription,
parseSubjectLabels,
resolveShotStarts,
retentionAnalysisFromShots,
shotLines,
shotProse,
shotsFromDetailedDescription,
type Shot,
type ShotInit,
} from "@/lib/h3Shots";
function shot(init: ShotInit): Shot {
return makeShot(init);
}
describe("parseSubjectLabels", () => {
it("returns ordered, de-duplicated <Subject N> labels", () => {
const text = "<Subject 1> is the dog.\n<Subject 2> is the cat. <Subject 1> again.";
expect(parseSubjectLabels(text)).toEqual(["<Subject 1>", "<Subject 2>"]);
});
it("returns an empty list when there are none", () => {
expect(parseSubjectLabels("no subjects here")).toEqual([]);
});
});
describe("detailedDescriptionFromShots", () => {
it("serializes shot type, movement, description, and dialogue", () => {
const shots = [
shot({
type: "close-up",
movement: "handheld",
description: "She looks up.",
lines: [newLine({ speaker: "S1", text: "I found it." })],
}),
];
expect(detailedDescriptionFromShots(shots)).toBe(
"[Shot 1] Close-up shot, handheld. She looks up. (S1) <d>[English] I found it.</d>",
);
});
it("omits the movement clause for a static camera", () => {
const shots = [shot({ type: "wide", movement: "static", description: "An empty street." })];
expect(detailedDescriptionFromShots(shots)).toBe("[Shot 1] Wide shot. An empty street.");
});
it("numbers shots in order, one per line, and marks later shots with a cut time", () => {
const shots = [shot({ description: "First." }), shot({ description: "Second." })];
const out = detailedDescriptionFromShots(shots);
expect(out).toBe(
"[Shot 1] Medium shot. First.\n[Shot 2] At 00:02.000, the shot cuts to a medium shot. Second.",
);
});
it("uses the right article and movement in the cut wording", () => {
const shots = [
shot({ type: "wide" }),
shot({ type: "establishing", startSeconds: 3 }),
shot({ type: "extreme-close-up", movement: "dolly", startSeconds: 5.25 }),
shot({ type: "pov", startSeconds: 7 }),
];
const out = detailedDescriptionFromShots(shots).split("\n");
expect(out[1]).toBe("[Shot 2] At 00:03.000, the shot cuts to an establishing shot.");
expect(out[2]).toBe("[Shot 3] At 00:05.250, the shot cuts to an extreme close-up shot, dolly.");
expect(out[3]).toBe("[Shot 4] At 00:07.000, the shot cuts to a POV shot.");
});
it("writes the preamble before the first shot", () => {
const out = detailedDescriptionFromShots([shot({ description: "A." })], "Keep the film-noir look.");
expect(out).toBe("Keep the film-noir look.\n[Shot 1] Medium shot. A.");
});
});
describe("cut times", () => {
it("formats and parses MM:SS.mmm", () => {
expect(formatCutTime(3.5)).toBe("00:03.500");
expect(formatCutTime(75.04)).toBe("01:15.040");
expect(parseCutTime("00:03.500")).toBe(3.5);
expect(parseCutTime("0:03.5")).toBe(3.5);
expect(parseCutTime("3.5")).toBe(3.5);
expect(parseCutTime("1:15")).toBe(75);
expect(parseCutTime("soon")).toBeNull();
});
it("starts the opening shot at 0 and follows estimated shot lengths", () => {
const shots = [
shot({ lines: [newLine({ text: "A short line." })] }),
shot({}),
shot({}),
];
const starts = resolveShotStarts(shots);
expect(starts[0]).toBe(0);
expect(starts[1]).toBeGreaterThanOrEqual(2);
expect(starts[2]).toBeCloseTo(starts[1] + 2, 3);
});
it("honors an explicit time and keeps times strictly increasing", () => {
const shots = [shot({}), shot({ startSeconds: 4 }), shot({ startSeconds: 1 })];
const starts = resolveShotStarts(shots);
expect(starts[1]).toBe(4);
expect(starts[2]).toBeGreaterThan(starts[1]);
});
});
describe("shotsFromDetailedDescription", () => {
it("round-trips type, movement, dialogue, speaker, and subjects", () => {
const original = [
shot({
type: "extreme-close-up",
movement: "dolly",
description: "Her hand trembles.",
lines: [newLine({ speaker: "S1", text: "Wait." })],
subjectIds: ["<Subject 1>"],
}),
];
const serialized = detailedDescriptionFromShots(original);
const withSubject = serialized.replace("Her hand trembles.", "Her hand trembles, <Subject 1> hesitates.");
const parsed = shotsFromDetailedDescription(withSubject, ["<Subject 1>", "<Subject 2>"]);
expect(parsed).toHaveLength(1);
expect(parsed[0].type).toBe("extreme-close-up");
expect(parsed[0].movement).toBe("dolly");
expect(shotLines(parsed[0])).toHaveLength(1);
expect(shotLines(parsed[0])[0].speaker).toBe("S1");
expect(shotLines(parsed[0])[0].text).toBe("Wait.");
expect(parsed[0].subjectIds).toEqual(["<Subject 1>"]);
});
it("keeps every dialogue line of an exchange, in order", () => {
const original = [
shot({
description: "They talk.",
lines: [
newLine({ speaker: "S1", text: "Ready?" }),
newLine({ speaker: "S2", text: "<pause> Always." }),
newLine({ speaker: "S1", text: "Then go." }),
],
}),
];
const serialized = detailedDescriptionFromShots(original);
expect(serialized).toBe(
"[Shot 1] Medium shot. They talk. (S1) <d>[English] Ready?</d> (S2) <d>[English] <pause> Always.</d> (S1) <d>[English] Then go.</d>",
);
const parsed = shotsFromDetailedDescription(serialized, []);
expect(shotProse(parsed[0])).toBe("They talk.");
expect(shotLines(parsed[0]).map((l) => [l.speaker, l.text])).toEqual([
["S1", "Ready?"],
["S2", "<pause> Always."],
["S1", "Then go."],
]);
});
it("round-trips a non-English language marker instead of relabelling it English", () => {
const original = [shot({ lines: [newLine({ speaker: "S1", text: "你好。", language: "Chinese" })] })];
const serialized = detailedDescriptionFromShots(original);
expect(serialized).toContain("<d>[Chinese] 你好。</d>");
expect(shotLines(shotsFromDetailedDescription(serialized, [])[0])[0].language).toBe("Chinese");
});
it("round-trips a multi-shot list without pinning the automatic cut times", () => {
const original = [
shot({ type: "establishing", description: "A rooftop." }),
shot({
type: "close-up",
movement: "handheld",
description: "He turns.",
lines: [newLine({ speaker: "S2", text: "I had no choice." })],
}),
shot({ type: "wide", movement: "dolly", description: "They stand apart." }),
];
const serialized = detailedDescriptionFromShots(original);
const parsed = shotsFromDetailedDescription(serialized, []);
expect(parsed.map((p) => [p.type, p.movement, p.freeLead])).toEqual([
["establishing", "static", false],
["close-up", "handheld", false],
["wide", "dolly", false],
]);
expect(parsed.every((p) => p.startSeconds === null)).toBe(true);
expect(detailedDescriptionFromShots(parsed)).toBe(serialized);
});
it("pins a cut time that differs from the automatic one", () => {
const parsed = shotsFromDetailedDescription(
"[Shot 1] Wide shot. A dock.\n[Shot 2] At 00:07.400, the shot cuts to a close-up shot. A face.",
[],
);
expect(parsed[1].startSeconds).toBe(7.4);
});
it("keeps hand-written shots verbatim, with the preamble and cut times", () => {
const handWritten = [
"The target video keeps the live-action style of <Video 1> unchanged.",
"[Shot 1] A handheld medium shot holds <Subject 1> at frame left, half-occluded by plastic sheeting.",
"[Shot 2] At 00:03.800, the shot cuts to a low tracking shot behind the boots of <Subject 1> as he sprints.",
"[Shot 3] At 00:07.400, the camera cuts to a close front-on tracking shot. He (S1) says: <d>[English] Go.</d> <cutoff>",
].join("\n");
const { preamble, shots } = parseDetailedDescription(handWritten, ["<Subject 1>"]);
expect(preamble).toBe("The target video keeps the live-action style of <Video 1> unchanged.");
expect(shots).toHaveLength(3);
expect(shots.every((s) => s.freeLead)).toBe(true);
expect(shots.map((s) => s.startSeconds)).toEqual([null, 3.8, 7.4]);
expect(shots[0].subjectIds).toEqual(["<Subject 1>"]);
// Dialogue in a hand-written shot is pulled out into an editable line...
expect(shotLines(shots[2]).map((l) => [l.speaker, l.text])).toEqual([["", "Go."]]);
expect(detailedDescriptionFromShots(shots, preamble)).toBe(handWritten);
});
it("keeps a trailing tag after the dialogue it modifies", () => {
const text =
"[Shot 1] He (S1) says: <d>[English] <breath> I think you're... <i>sorry</i>.</d> <cutoff>";
const [shot1] = shotsFromDetailedDescription(text, []);
expect(shot1.freeLead).toBe(true);
expect(shot1.body.map((p) => p.kind)).toEqual(["text", "line", "text"]);
expect(shotLines(shot1)[0].text).toBe("<breath> I think you're... <i>sorry</i>.");
expect(detailedDescriptionFromShots([shot1])).toBe(text);
});
it("keeps interleaved dialogue in place, whitespace and all, in hand-written shots", () => {
const text = "[Shot 1] She waits.\n (S1) <d>[English] Ready?</d>\nHe nods. (S2) <d>[English] Ready.</d> The door opens.";
const [s1] = shotsFromDetailedDescription(text, []);
expect(s1.body.map((p) => p.kind)).toEqual(["text", "line", "text", "line", "text"]);
expect(detailedDescriptionFromShots([s1])).toBe(text);
});
it("keeps dialogue between prose in shots this editor wrote too", () => {
const text = "[Shot 1] Wide shot. He looks up. (S1) <d>[English] Hello.</d> Then he leaves.";
const [s1] = shotsFromDetailedDescription(text, []);
expect(s1.freeLead).toBe(false);
expect(s1.body.map((p) => p.kind)).toEqual(["text", "line", "text"]);
expect(detailedDescriptionFromShots([s1])).toBe(text);
});
it("edits a line without moving the text or lines around it", () => {
const text = "[Shot 1] Wide shot. A. (S1) <d>[English] One.</d> B. (S2) <d>[English] Two.</d> C.";
const [s1] = shotsFromDetailedDescription(text, []);
const second = shotLines(s1)[1];
second.text = "Changed.";
expect(detailedDescriptionFromShots([s1])).toBe(
"[Shot 1] Wide shot. A. (S1) <d>[English] One.</d> B. (S2) <d>[English] Changed.</d> C.",
);
});
it("puts a newly added line at the end and separates it from the prose", () => {
const text = "[Shot 1] Wide shot. He speaks. (S1) <d>[English] First.</d> He pauses.";
const [s1] = shotsFromDetailedDescription(text, []);
s1.body.push({ kind: "line", line: newLine({ speaker: "S1", text: "Later." }) });
expect(detailedDescriptionFromShots([s1])).toBe(
"[Shot 1] Wide shot. He speaks. (S1) <d>[English] First.</d> He pauses. (S1) <d>[English] Later.</d>",
);
});
it("reordering blocks changes the written order", () => {
const [s1] = shotsFromDetailedDescription(
"[Shot 1] Wide shot. (S1) <d>[English] One.</d> (S2) <d>[English] Two.</d>",
[],
);
const lineBlocks = s1.body.filter((p) => p.kind === "line");
s1.body = [...s1.body.filter((p) => p.kind !== "line"), lineBlocks[1], lineBlocks[0]];
expect(detailedDescriptionFromShots([s1])).toBe(
"[Shot 1] Wide shot. (S2) <d>[English] Two.</d> (S1) <d>[English] One.</d>",
);
});
it("recovers type and movement from the guide's cut wording", () => {
const parsed = shotsFromDetailedDescription(
"[Shot 1] Wide shot.\n[Shot 2] At 00:04.000, the camera cuts to an over-the-shoulder shot, handheld. Two people talk.",
[],
);
expect(parsed[1].type).toBe("over-the-shoulder");
expect(parsed[1].movement).toBe("handheld");
expect(shotProse(parsed[1])).toBe("Two people talk.");
expect(parsed[1].freeLead).toBe(false);
});
it("falls back to defaults for arbitrary prose, keeping it as the description", () => {
const parsed = shotsFromDetailedDescription("A toy car drives into a plush dog.", []);
expect(parsed).toHaveLength(1);
expect(parsed[0].type).toBe("medium");
expect(parsed[0].movement).toBe("static");
expect(shotProse(parsed[0])).toBe("A toy car drives into a plush dog.");
});
it("returns an empty list for blank text", () => {
expect(shotsFromDetailedDescription("", [])).toEqual([]);
});
});
describe("retentionAnalysisFromShots", () => {
it("generates an appears-in line per subject with shots", () => {
const shots = [
shot({ subjectIds: ["<Subject 1>"] }),
shot({ subjectIds: ["<Subject 1>", "<Subject 2>"] }),
];
const out = retentionAnalysisFromShots(shots, ["<Subject 1>", "<Subject 2>"], "");
expect(out).toBe(
[
"<Subject 1> (appears in [Shot 1], [Shot 2]): fully_preserved - what is retained.",
"<Subject 2> (appears in [Shot 2]): fully_preserved - what is retained.",
].join("\n"),
);
});
it("preserves an existing retention note instead of overwriting it", () => {
const shots = [shot({ subjectIds: ["<Subject 1>"] })];
const previous = "<Subject 1> (appears in [Shot 1]): fully_preserved - fur and collar retained.";
const out = retentionAnalysisFromShots(shots, ["<Subject 1>"], previous);
expect(out).toBe("<Subject 1> (appears in [Shot 1]): fully_preserved - fur and collar retained.");
});
it("passes through lines it doesn't own, like audio references", () => {
const shots = [shot({ subjectIds: ["<Subject 1>"] })];
const previous = [
"<Subject 1> (appears in [Shot 3]): fully_preserved - old note.",
"<Audio 1>: reference - how it guides the audio.",
].join("\n");
const out = retentionAnalysisFromShots(shots, ["<Subject 1>"], previous);
expect(out).toContain("<Subject 1> (appears in [Shot 1]): fully_preserved - old note.");
expect(out).toContain("<Audio 1>: reference - how it guides the audio.");
});
it("omits a subject with no shots rather than emitting an empty appears-in", () => {
const shots = [shot({ subjectIds: [] })];
const out = retentionAnalysisFromShots(shots, ["<Subject 1>"], "");
expect(out).toBe("");
});
});
+478
View File
@@ -0,0 +1,478 @@
/**
* MiniMax-H3 shot list: a structured camera-direction editor that serializes
* into `detailed_description`'s `[Shot N]` blocks and keeps
* `retention_analysis`'s "appears in [...]" lines in sync, per the model's
* reference prompt guide (see h3Prompt.ts / h3References.ts for the six
* guided sections this plugs into).
*/
import { SPEECH_RATES, estimateDuration } from "@/lib/h3DialogueEstimate";
export type ShotType =
| "establishing"
| "wide"
| "medium"
| "close-up"
| "extreme-close-up"
| "over-the-shoulder"
| "pov"
| "insert";
export type CameraMovement =
| "static"
| "pan"
| "tilt"
| "dolly"
| "zoom"
| "handheld"
| "tracking"
| "crane";
export interface DialogueLine {
/** stable key for React lists -- not part of the serialized prompt */
id: string;
/** speaker tag without parens, e.g. "S1" -- blank omits the "(S1)" prefix */
speaker: string;
/** spoken text only (speech tags allowed); wrapped in <d>[language] ...</d> on serialize */
text: string;
/** language marker written inside <d>; recovered on parse so non-English dialogue round-trips */
language: string;
}
/** A run of prose inside a shot (composition, action, environment, delivery notes). */
export interface TextPart {
kind: "text";
/** stable key for React lists -- not part of the serialized prompt */
id: string;
text: string;
}
/** One spoken line, written as `(S1) <d>[English] ...</d>`. Its position among the parts is where it plays. */
export interface LinePart {
kind: "line";
line: DialogueLine;
}
export type ShotPart = TextPart | LinePart;
export interface Shot {
/** stable key for React lists -- not part of the serialized prompt */
id: string;
type: ShotType;
movement: CameraMovement;
/** <Subject N> labels (from subject_definitions) that appear in this shot */
subjectIds: string[];
/**
* The shot's content in the order it reads: prose and spoken lines
* interleaved. This one list is the only record of where each line falls,
* so a `<d>` block stays mid-sentence if that is where it was written.
*/
body: ShotPart[];
/** cut-in time in seconds; null = computed from the preceding shots. Ignored for the opening shot. */
startSeconds: number | null;
/**
* The description carries its own lead-in (text opened from a hand-written
* prompt) so type/movement are not written and the wording is kept verbatim.
*/
freeLead: boolean;
}
export const SHOT_TYPES: { id: ShotType; label: string }[] = [
{ id: "establishing", label: "Establishing" },
{ id: "wide", label: "Wide" },
{ id: "medium", label: "Medium" },
{ id: "close-up", label: "Close-up" },
{ id: "extreme-close-up", label: "Extreme close-up" },
{ id: "over-the-shoulder", label: "Over-the-shoulder" },
{ id: "pov", label: "POV" },
{ id: "insert", label: "Insert" },
];
export const CAMERA_MOVEMENTS: { id: CameraMovement; label: string }[] = [
{ id: "static", label: "Static" },
{ id: "pan", label: "Pan" },
{ id: "tilt", label: "Tilt" },
{ id: "dolly", label: "Dolly" },
{ id: "zoom", label: "Zoom" },
{ id: "handheld", label: "Handheld" },
{ id: "tracking", label: "Tracking" },
{ id: "crane", label: "Crane" },
];
const SHOT_TYPE_LABEL = Object.fromEntries(SHOT_TYPES.map((t) => [t.id, t.label])) as Record<ShotType, string>;
const MOVEMENT_LABEL = Object.fromEntries(CAMERA_MOVEMENTS.map((m) => [m.id, m.label])) as Record<
CameraMovement,
string
>;
const SHOT_TYPE_BY_LABEL = Object.fromEntries(SHOT_TYPES.map((t) => [t.label.toLowerCase(), t.id])) as Record<
string,
ShotType
>;
const MOVEMENT_BY_LABEL = Object.fromEntries(
CAMERA_MOVEMENTS.map((m) => [m.label.toLowerCase(), m.id]),
) as Record<string, CameraMovement>;
let shotSeq = 0;
export function newShot(): Shot {
shotSeq += 1;
return {
id: `shot-${Date.now()}-${shotSeq}`,
type: "medium",
movement: "static",
subjectIds: [],
body: [newTextPart()],
startSeconds: null,
freeLead: false,
};
}
let partSeq = 0;
export function newTextPart(text = ""): TextPart {
partSeq += 1;
return { kind: "text", id: `text-${Date.now()}-${partSeq}`, text };
}
export function linePart(line: DialogueLine): LinePart {
return { kind: "line", line };
}
/** Every spoken line of the shot, in playback order. */
export function shotLines(shot: Shot): DialogueLine[] {
return shot.body.flatMap((p) => (p.kind === "line" ? [p.line] : []));
}
/** The shot's prose with the dialogue left out. */
export function shotProse(shot: Shot): string {
return shot.body
.flatMap((p) => (p.kind === "text" ? [p.text.trim()] : []))
.filter(Boolean)
.join(" ");
}
/** A body of one prose part followed by the given lines -- the common "description, then dialogue" shape. */
export function makeBody(description = "", lines: DialogueLine[] = []): ShotPart[] {
return [newTextPart(description), ...lines.map(linePart)];
}
export interface ShotInit extends Partial<Omit<Shot, "id" | "body">> {
description?: string;
lines?: DialogueLine[];
body?: ShotPart[];
}
/** Build a shot from a description and lines (or an explicit body). */
export function makeShot(init: ShotInit = {}): Shot {
const { description, lines, body, ...rest } = init;
return { ...newShot(), ...rest, body: body ?? makeBody(description ?? "", lines ?? []) };
}
let lineSeq = 0;
export function newLine(patch: Partial<Omit<DialogueLine, "id">> = {}): DialogueLine {
lineSeq += 1;
return { id: `line-${Date.now()}-${lineSeq}`, speaker: "", text: "", language: "English", ...patch };
}
/** Copy shots with brand-new ids so components keyed by id (which own their own text state) remount instead of showing stale edits. */
export function withFreshIds(shots: Shot[]): Shot[] {
return shots.map((s) => ({
...s,
id: newShot().id,
subjectIds: [...s.subjectIds],
body: s.body.map((p) =>
p.kind === "text" ? { ...p, id: newTextPart().id } : { kind: "line" as const, line: { ...p.line, id: newLine().id } },
),
}));
}
/** Ordered, de-duplicated <Subject N> labels mentioned in subject_definitions. */
export function parseSubjectLabels(subjectDefinitions: string): string[] {
const seen: string[] = [];
const re = /<Subject\s+\d+>/g;
let m: RegExpExecArray | null;
while ((m = re.exec(subjectDefinitions))) {
if (!seen.includes(m[0])) seen.push(m[0]);
}
return seen;
}
/* ---------------------------------------------------------------- */
/* Cut times */
/* ---------------------------------------------------------------- */
const MIN_SHOT_SECONDS = 2;
const LINE_GAP_SECONDS = 0.5;
/** Ballpark seconds a shot with these spoken lines runs; drives auto cut times. */
export function estimateSpokenSeconds(texts: string[]): number {
const spoken = texts.filter((t) => t.trim());
const speech = spoken.reduce((n, t) => n + estimateDuration(t, SPEECH_RATES.Natural, 1).total, 0);
return Math.max(MIN_SHOT_SECONDS, speech + (spoken.length ? LINE_GAP_SECONDS : 0));
}
export function estimateShotSeconds(shot: Shot): number {
return estimateSpokenSeconds(shotLines(shot).map((l) => l.text));
}
/**
* Cut-in time of every shot: the opening shot is 0, later ones use their
* explicit time or follow the previous shot's estimated length. Times are
* kept strictly increasing, as the guide requires.
*/
export function resolveShotStarts(shots: Shot[]): number[] {
const starts: number[] = [];
shots.forEach((shot, i) => {
if (i === 0) {
starts.push(0);
return;
}
const prev = starts[i - 1];
let start = shot.startSeconds ?? prev + estimateShotSeconds(shots[i - 1]);
if (start <= prev) start = prev + 0.1;
starts.push(Math.round(start * 1000) / 1000);
});
return starts;
}
/** MM:SS.mmm, the format the guide uses for cut times ("At 00:03.500, ..."). */
export function formatCutTime(seconds: number): string {
const totalMs = Math.max(0, Math.round(seconds * 1000));
const pad = (n: number, w: number) => String(n).padStart(w, "0");
const totalSec = Math.floor(totalMs / 1000);
return `${pad(Math.floor(totalSec / 60), 2)}:${pad(totalSec % 60, 2)}.${pad(totalMs % 1000, 3)}`;
}
/** Accepts "3.5", "0:03.5", or "00:03.500"; returns seconds, or null if unreadable. */
export function parseCutTime(text: string): number | null {
const m = text.trim().match(/^(?:(\d+):)?(\d+(?:\.\d+)?)$/);
if (!m) return null;
return (m[1] ? Number(m[1]) * 60 : 0) + Number(m[2]);
}
/* ---------------------------------------------------------------- */
/* Serialize */
/* ---------------------------------------------------------------- */
/** `raw` keeps a line's text exactly as stored (hand-written shots); otherwise it is trimmed. */
function partText(part: ShotPart, raw = false): string {
if (part.kind === "text") return part.text;
const { speaker, language, text } = part.line;
if (!text.trim()) return "";
return `${speaker.trim() ? `(${speaker.trim()}) ` : ""}<d>[${language.trim() || "English"}] ${raw ? text : text.trim()}</d>`;
}
/** Concatenate pieces, adding one space only where neither side already has whitespace. */
function joinPieces(pieces: string[]): string {
let out = "";
for (const piece of pieces) {
if (!piece) continue;
if (out && !/\s$/.test(out) && !/^\s/.test(piece)) out += " ";
out += piece;
}
return out;
}
function serializeShot(shot: Shot, n: number, start: number): string {
// [Shot 1] has no timestamp; later shots mark their cut time.
const cut = n > 1 ? `At ${formatCutTime(start)}, ` : "";
if (shot.freeLead) {
// Text parts keep their exact whitespace, so hand-written wording round-trips byte for byte.
return `[Shot ${n}] ${cut}${joinPieces(shot.body.map((p) => partText(p, true)))}`.trim();
}
const body = shot.body
.map((p) => partText(p).trim())
.filter(Boolean)
.join(" ");
const label = SHOT_TYPE_LABEL[shot.type];
const movementPart = shot.movement === "static" ? "" : `, ${MOVEMENT_LABEL[shot.movement].toLowerCase()}`;
const lead =
n > 1
? `the shot cuts to ${/^[aeiou]/i.test(label) ? "an" : "a"} ${label === "POV" ? label : label.toLowerCase()} shot${movementPart}.`
: `${label} shot${movementPart}.`;
return [`[Shot ${n}] ${cut}${lead}`, body].filter(Boolean).join(" ");
}
/**
* Serialize the shot list into detailed_description's [Shot N] format:
* `[Shot 1] ...` then `[Shot N] At MM:SS.mmm, the shot cuts to ...`. Optional
* `preamble` (style notes that come before the first shot) is written first.
*/
export function detailedDescriptionFromShots(shots: Shot[], preamble = ""): string {
const starts = resolveShotStarts(shots);
return [preamble.trim(), ...shots.map((s, i) => serializeShot(s, i + 1, starts[i]))]
.filter(Boolean)
.join("\n");
}
/* ---------------------------------------------------------------- */
/* Parse */
/* ---------------------------------------------------------------- */
const CUT_TIME_RE = /^At (\d+):(\d{2})(?:\.(\d{1,3}))?,\s*/i;
// "[the shot cuts to] [a] <type> shot[, <movement>]." -- the lead-in this
// editor writes, recovered on parse so type/movement survive a reopen.
const LEAD_RE =
/^(?:(?:the (?:shot|camera) (?:cuts|transitions|changes|switches) to |cuts to )?(?:an? )?)([a-zA-Z -]+?) shot(?:,\s*([a-zA-Z]+))?\.\s*/i;
const DIALOGUE_RE = /(?:\(([^)]+)\)\s*)?<d>(?:\[([^\]]*)\]\s*)?([\s\S]*?)<\/d>/g;
/**
* Split shot text at its `<d>` blocks into an ordered body. `exact` keeps the
* surrounding text untouched (whitespace included) so it can be rebuilt
* byte for byte; otherwise prose is trimmed and empty runs dropped.
*/
function splitBody(text: string, exact: boolean): ShotPart[] {
const parts: ShotPart[] = [];
const pushText = (raw: string) => {
const t = exact ? raw : raw.trim();
if (t) parts.push(newTextPart(t));
};
let last = 0;
for (const m of text.matchAll(DIALOGUE_RE)) {
const start = m.index ?? 0;
pushText(text.slice(last, start));
parts.push(
linePart(
newLine({
speaker: m[1]?.trim() ?? "",
language: m[2]?.trim() || "English",
text: exact ? (m[3] ?? "") : (m[3] ?? "").trim(),
}),
),
);
last = start + m[0].length;
}
pushText(text.slice(last));
return parts;
}
function parseShotBlock(block: string, subjectLabels: string[]): { shot: Shot; time: number | null } {
const shot = newShot();
shot.subjectIds = subjectLabels.filter((label) => block.includes(label));
let rest = block;
let time: number | null = null;
const t = rest.match(CUT_TIME_RE);
if (t) {
time = Number(t[1]) * 60 + Number(t[2]) + (t[3] ? Number(`0.${t[3]}`) : 0);
rest = rest.slice(t[0].length);
}
const lead = rest.match(LEAD_RE);
const typeId = lead ? SHOT_TYPE_BY_LABEL[lead[1].trim().toLowerCase()] : undefined;
if (lead && typeId) {
shot.type = typeId;
const moveLabel = lead[2]?.trim().toLowerCase();
if (moveLabel && MOVEMENT_BY_LABEL[moveLabel]) shot.movement = MOVEMENT_BY_LABEL[moveLabel];
rest = rest.slice(lead[0].length);
shot.body = splitBody(rest, false);
} else {
// Hand-written wording: keep it exactly as written. Dialogue is still
// pulled out into editable lines, but each stays where it was written.
shot.freeLead = true;
shot.body = splitBody(rest, true);
}
if (shot.body.length === 0) shot.body = [newTextPart()];
return { shot, time };
}
/**
* Split an existing detailed_description back into shots so opening the
* builder on a job someone already wrote doesn't discard it. Shots are found
* by their line-start [Shot N] markers; anything before the first is kept as
* the preamble. Only wording this editor generated is decomposed into
* type/movement/dialogue -- anything else is kept verbatim.
*/
export function parseDetailedDescription(
text: string,
subjectLabels: string[],
): { preamble: string; shots: Shot[] } {
const markers = [...text.matchAll(/^\[Shot\s+\d+\][ \t]*/gm)];
if (markers.length === 0) {
const body = text.trim();
if (!body) return { preamble: "", shots: [] };
const { shot } = parseShotBlock(body, subjectLabels);
shot.freeLead = true;
shot.body = splitBody(body, true);
return { preamble: "", shots: [shot] };
}
const preamble = text.slice(0, markers[0].index).trim();
const parsed = markers.map((m, i) => {
const from = (m.index ?? 0) + m[0].length;
const to = i + 1 < markers.length ? (markers[i + 1].index ?? text.length) : text.length;
return parseShotBlock(text.slice(from, to).trim(), subjectLabels);
});
const shots = parsed.map((p) => p.shot);
// Only pin a cut time when it differs from what the editor would compute,
// so text this editor wrote keeps flowing as earlier shots change.
const starts: number[] = [0];
shots.forEach((shot, i) => {
if (i === 0) return;
const auto = starts[i - 1] + estimateShotSeconds(shots[i - 1]);
const time = parsed[i].time;
if (time == null) {
starts.push(auto);
return;
}
starts.push(time);
if (Math.abs(time - auto) > 0.001) shot.startSeconds = time;
});
return { preamble, shots };
}
export function shotsFromDetailedDescription(text: string, subjectLabels: string[]): Shot[] {
return parseDetailedDescription(text, subjectLabels).shots;
}
const RETENTION_LINE_RE = /^(<[^>]+>)\s*\(appears in [^)]*\):\s*(.*)$/gm;
const DEFAULT_RETENTION_NOTE = "fully_preserved - what is retained.";
/** label -> free-text retention note (everything after the colon), for lines this parser recognizes. */
function parseRetentionNotes(retentionAnalysis: string): Map<string, string> {
const notes = new Map<string, string>();
let m: RegExpExecArray | null;
RETENTION_LINE_RE.lastIndex = 0;
while ((m = RETENTION_LINE_RE.exec(retentionAnalysis))) notes.set(m[1], m[2].trim());
return notes;
}
/**
* Rebuild retention_analysis's shot-tracked lines from the shot list. Each
* subject's existing retention note (the free text after the colon) is kept
* untouched -- only the "(appears in [...])" shot list is regenerated, using
* a sensible default note for a subject that's newly shot-tracked. Anything
* this shot list doesn't own (untracked subjects, audio-reference lines,
* free text) passes through unchanged rather than being discarded.
*/
export function retentionAnalysisFromShots(
shots: Shot[],
subjectLabels: string[],
previousRetentionAnalysis: string,
): string {
const existingNotes = parseRetentionNotes(previousRetentionAnalysis);
const regenerated = new Set<string>();
const shotLines: string[] = [];
for (const label of subjectLabels) {
const shotNums = shots
.map((s, i) => (s.subjectIds.includes(label) ? i + 1 : null))
.filter((n): n is number => n !== null);
if (shotNums.length === 0) continue;
regenerated.add(label);
const note = existingNotes.get(label) ?? DEFAULT_RETENTION_NOTE;
shotLines.push(`${label} (appears in ${shotNums.map((n) => `[Shot ${n}]`).join(", ")}): ${note}`);
}
const passthroughLines = previousRetentionAnalysis
.split("\n")
.map((l) => l.trim())
.filter((line) => {
if (!line) return false;
const m = line.match(/^(<[^>]+>)\s*\(appears in/);
return !(m && regenerated.has(m[1]));
});
return [...shotLines, ...passthroughLines].join("\n");
}
@@ -0,0 +1,117 @@
import { describe, expect, it } from "vitest";
import {
applyH3Preset,
applyH3Tag,
H3_PRESETS,
H3_TAG_CATEGORIES,
H3_TAGS,
type H3Preset,
type H3Tag,
} from "@/lib/h3Tags";
const pointTag = H3_TAGS.find((t) => t.id === "pause") as H3Tag;
const wrapTag = H3_TAGS.find((t) => t.id === "emphasis") as H3Tag;
const emptyWrapTag = H3_TAGS.find((t) => t.id === "smacks-lips") as H3Tag;
describe("H3 tag catalogue", () => {
it("gives every tag a unique id", () => {
const ids = H3_TAGS.map((t) => t.id);
expect(new Set(ids).size).toBe(ids.length);
});
it("gives wrap tags a close tag and point tags none", () => {
for (const tag of H3_TAGS) {
if (tag.kind === "wrap") expect(tag.close).toBeTruthy();
else expect(tag.close).toBeUndefined();
}
});
it("flattens to the same tags the categories declare", () => {
const flattened = H3_TAG_CATEGORIES.flatMap((c) => c.tags);
expect(H3_TAGS).toEqual(flattened);
});
});
describe("applyH3Tag: point tags", () => {
it("inserts at the cursor with padding on both sides", () => {
const result = applyH3Tag("Okay, so. This is me.", 9, 9, pointTag);
expect(result.value).toBe("Okay, so. <pause> This is me.");
// cursor collapses to just past the inserted token
expect(result.value.slice(result.start, result.end)).toBe("");
expect(result.value.slice(0, result.start)).toBe("Okay, so. <pause> ".slice(0, result.start));
});
it("does not double up whitespace already present", () => {
const result = applyH3Tag("Okay, so. This is me.", 10, 10, pointTag);
expect(result.value).toBe("Okay, so. <pause> This is me.");
});
it("inserts cleanly at the very start and end of the text", () => {
expect(applyH3Tag("Hello.", 0, 0, pointTag).value).toBe("<pause> Hello.");
expect(applyH3Tag("Hello.", 6, 6, pointTag).value).toBe("Hello. <pause>");
});
it("inserts before a selection, keeping the selected text", () => {
const result = applyH3Tag("Hello there.", 6, 11, pointTag);
expect(result.value).toBe("Hello <pause> there.");
expect(result.value.slice(result.start, result.end)).toBe("there");
});
});
describe("applyH3Tag: wrap tags", () => {
it("wraps a selection and collapses the cursor after the close tag", () => {
const result = applyH3Tag("I was not expecting that.", 6, 9, wrapTag);
expect(result.value).toBe("I was <i>not</i> expecting that.");
expect(result.start).toBe(result.end);
expect(result.value.slice(0, result.start)).toBe("I was <i>not</i>");
});
it("inserts an empty open+close pair with the cursor between them when nothing is selected", () => {
const result = applyH3Tag("Don't tell anyone.", 0, 0, wrapTag);
expect(result.value).toBe("<i></i>Don't tell anyone.");
expect(result.start).toBe("<i>".length);
expect(result.end).toBe(result.start);
});
it("supports an intentionally empty pair, as used for a sentence-final effect", () => {
const result = applyH3Tag("Okay.", 0, 0, emptyWrapTag);
expect(result.value).toBe("<smacks lips></smacks lips>Okay.");
});
});
describe("applyH3Tag: out-of-range selection", () => {
it("clamps a selection beyond the text length", () => {
const result = applyH3Tag("Hi.", 10, 20, pointTag);
expect(result.value).toBe("Hi. <pause>");
});
});
describe("applyH3Preset", () => {
const insertPreset = H3_PRESETS.find((p) => p.label === "Nervous") as H3Preset;
const wrapPreset = H3_PRESETS.find((p) => p.label === "Intimate") as H3Preset;
it("inserts with a leading space when the cursor follows text", () => {
const r = applyH3Preset("Well", 4, 4, insertPreset);
expect(r.value).toBe("Well <uh> <stutter> ");
expect(r.start).toBe(r.value.length);
expect(r.end).toBe(r.value.length);
});
it("adds no leading space at the start or after whitespace", () => {
expect(applyH3Preset("", 0, 0, insertPreset).value).toBe("<uh> <stutter> ");
expect(applyH3Preset("Well ", 5, 5, insertPreset).value).toBe("Well <uh> <stutter> ");
});
it("wraps a selection and keeps it selected", () => {
const r = applyH3Preset("say it softly now", 4, 13, wrapPreset);
expect(r.value).toBe("say <whisper>it softly</whisper> now");
expect(r.value.slice(r.start, r.end)).toBe("it softly");
});
it("wraps an empty selection with the caret between the tags", () => {
const r = applyH3Preset("hi ", 3, 3, wrapPreset);
expect(r.value).toBe("hi <whisper></whisper>");
expect(r.start).toBe(r.end);
expect(r.value.slice(0, r.start)).toBe("hi <whisper>");
});
});
+409
View File
@@ -0,0 +1,409 @@
/**
* MiniMax-H3 speech tags: the inline markers the model recognizes inside
* `<d>[English] dialogue</d>` blocks (and elsewhere in a prompt) to shape
* delivery -- pauses, breath, emphasis, and non-verbal sounds. `applyH3Tag`
* is the pure insertion logic; the tag/category tables drive the console UI.
*/
export type H3TagKind = "point" | "wrap";
export interface H3Tag {
/** stable key for React lists and lookups */
id: string;
/** short button label, usually the bare tag name */
label: string;
description: string;
/** short usage example shown as a tooltip */
example: string;
kind: H3TagKind;
/** point: the literal token inserted, e.g. "<pause>". wrap: the opening tag. */
open: string;
/** wrap only: the closing tag, e.g. "</i>". */
close?: string;
/**
* Rough seconds this tag adds to spoken duration, used only for the
* dialogue editor's length estimate -- ballpark figures, not measured.
* Delivery-only tags (emphasis, whisper, softer) add ~0.
*/
seconds?: number;
}
export interface H3TagCategory {
id: string;
label: string;
tags: H3Tag[];
}
export const H3_TAG_CATEGORIES: H3TagCategory[] = [
{
id: "pace",
label: "Pace & breath",
tags: [
{
id: "pause",
label: "<pause>",
description: "Short pause",
example: "Okay, so. <pause> This is just me talking.",
kind: "point",
open: "<pause>",
seconds: 0.45,
},
{
id: "long-pause",
label: "<long pause>",
description: "Longer pause",
example: "I mean... <long pause> I don't even know.",
kind: "point",
open: "<long pause>",
seconds: 1.2,
},
{
id: "breath",
label: "<breath>",
description: "Breathing sound",
example: "And then... <breath> it just happened.",
kind: "point",
open: "<breath>",
seconds: 0.55,
},
{
id: "inhale",
label: "<inhale>",
description: "In-breath",
example: "<inhale> Alright, let's do this.",
kind: "point",
open: "<inhale>",
seconds: 0.5,
},
{
id: "exhale",
label: "<exhale>",
description: "Out-breath",
example: "<exhale> Okay then.",
kind: "point",
open: "<exhale>",
seconds: 0.6,
},
{
id: "catches-breath",
label: "<catches breath>",
description: "Out of breath",
example: "Wait... <catches breath> hold on a sec.",
kind: "point",
open: "<catches breath>",
seconds: 0.8,
},
{
id: "deep-breath",
label: "<deep breath>",
description: "Calming down",
example: "<deep breath> Okay. I can do this.",
kind: "point",
open: "<deep breath>",
seconds: 1.2,
},
{
id: "pant",
label: "<pant>",
description: "Panting (single)",
example: "Run... <pant> run now!",
kind: "point",
open: "<pant>",
seconds: 0.7,
},
{
id: "pants",
label: "<pants>",
description: "Panting (continued)",
example: "Run... <pants> run now!",
kind: "point",
open: "<pants>",
seconds: 0.7,
},
],
},
{
id: "delivery",
label: "Emphasis & delivery",
tags: [
{
id: "emphasis",
label: "<i>…</i>",
description: "Emphasize the wrapped words (1-4 words)",
example: "I was <i>not</i> expecting that.",
kind: "wrap",
open: "<i>",
close: "</i>",
seconds: 0,
},
{
id: "whisper",
label: "<whisper>…</whisper>",
description: "Whisper delivery",
example: "<whisper>Don't tell anyone this.</whisper>",
kind: "wrap",
open: "<whisper>",
close: "</whisper>",
seconds: 0,
},
{
id: "softer",
label: "<softer>",
description: "Quieter delivery",
example: "<softer> I don't think I can say it.",
kind: "point",
open: "<softer>",
seconds: 0,
},
{
id: "humming",
label: "<humming>…</humming>",
description: "Humming a tune",
example: "<humming>da-da-da-beautiful-day.</humming>",
kind: "wrap",
open: "<humming>",
close: "</humming>",
seconds: 2,
},
],
},
{
id: "vocal",
label: "Vocalizations",
tags: [
{
id: "laughs",
label: "<laughs>",
description: "Laughing",
example: "That's... <laughs> that's actually funny.",
kind: "point",
open: "<laughs>",
seconds: 1.1,
},
{
id: "chuckle",
label: "<chuckle>",
description: "Laughing (light)",
example: "That's... <chuckle> that's actually funny.",
kind: "point",
open: "<chuckle>",
seconds: 0.7,
},
{
id: "sighs",
label: "<sighs>",
description: "Sigh",
example: "<sighs> I really tried.",
kind: "point",
open: "<sighs>",
seconds: 0.85,
},
{
id: "gasp",
label: "<gasp>",
description: "Sharp intake",
example: "<gasp> Oh my God.",
kind: "point",
open: "<gasp>",
seconds: 0.5,
},
{
id: "phew",
label: "<phew>",
description: "Relief",
example: "<phew> That was close.",
kind: "point",
open: "<phew>",
seconds: 0.6,
},
{
id: "mhm",
label: "<mhm>",
description: "Agreement sound",
example: "Yeah, <mhm> exactly.",
kind: "point",
open: "<mhm>",
seconds: 0.4,
},
{
id: "uh",
label: "<uh>",
description: "Filler / hesitation",
example: "So, like... <uh> what was I saying?",
kind: "point",
open: "<uh>",
seconds: 0.35,
},
],
},
{
id: "body",
label: "Disfluency & body",
tags: [
{
id: "stutter",
label: "<stutter>",
description: "Stutters the words that follow",
example: "<stutter> I ca can't believe that.",
kind: "point",
open: "<stutter>",
seconds: 0.45,
},
{
id: "coughs",
label: "<coughs>",
description: "Cough",
example: "<coughs> Sorry, one sec.",
kind: "point",
open: "<coughs>",
seconds: 0.6,
},
{
id: "clears-throat",
label: "<clears throat>",
description: "Throat clear",
example: "<clears throat> So anyway...",
kind: "point",
open: "<clears throat>",
seconds: 0.8,
},
{
id: "sniff",
label: "<sniff>",
description: "Sniffing",
example: "<sniff> It's just... really sad.",
kind: "point",
open: "<sniff>",
seconds: 0.45,
},
{
id: "smacks-lips",
label: "<smacks lips>",
description:
"Lip smack; leave empty to trigger at the end of the sentence, or wrap words to place it there",
example: "<smacks lips></smacks lips> Okay.",
kind: "wrap",
open: "<smacks lips>",
close: "</smacks lips>",
seconds: 0.3,
},
],
},
];
export const H3_TAGS: H3Tag[] = H3_TAG_CATEGORIES.flatMap((c) => c.tags);
export interface H3TagApplyResult {
value: string;
/** selection to restore focus to after the insertion (collapsed unless text was wrapped). */
start: number;
end: number;
}
function needsLeadingSpace(before: string): boolean {
return before.length > 0 && !/\s$/.test(before);
}
function needsTrailingSpace(after: string): boolean {
return after.length > 0 && !/^\s/.test(after);
}
/**
* Insert `tag` into `value` at the given selection.
*
* Point tags are inserted at the selection start, padded with a space on
* either side where the surrounding text doesn't already have one, and the
* cursor collapses to just past the inserted token.
*
* Wrap tags wrap the current selection (cursor lands just past the closing
* tag), or insert an empty `open+close` pair with the cursor left between
* them when nothing is selected, ready to type into.
*/
export function applyH3Tag(
value: string,
selectionStart: number,
selectionEnd: number,
tag: H3Tag,
): H3TagApplyResult {
const start = Math.max(0, Math.min(selectionStart, value.length));
const end = Math.max(start, Math.min(selectionEnd, value.length));
const before = value.slice(0, start);
const selected = value.slice(start, end);
const after = value.slice(end);
if (tag.kind === "point") {
const lead = needsLeadingSpace(before) ? " " : "";
const trailingContext = selected.length > 0 ? selected : after;
const trail = needsTrailingSpace(trailingContext) ? " " : "";
const token = `${lead}${tag.open}${trail}`;
const cursor = before.length + token.length;
return {
value: before + token + selected + after,
start: cursor,
end: cursor + selected.length,
};
}
const close = tag.close ?? "";
if (selected.length > 0) {
const cursor = before.length + tag.open.length + selected.length + close.length;
return {
value: before + tag.open + selected + close + after,
start: cursor,
end: cursor,
};
}
const cursor = before.length + tag.open.length;
return {
value: before + tag.open + close + after,
start: cursor,
end: cursor,
};
}
/** One-click bundles of tags for common delivery beats (dialogue editor + shot list). */
export interface H3Preset {
label: string;
/** literal text inserted at the cursor (a leading space is added when needed). */
insert?: string;
/** wrap the selection between these two tokens instead of inserting. */
wrap?: [string, string];
}
export const H3_PRESETS: H3Preset[] = [
{ label: "Cracking", insert: "<catches breath> <stutter> " },
{ label: "Nervous", insert: "<uh> <stutter> " },
{ label: "Relief", insert: "<phew> <exhale> " },
{ label: "Cold turn", insert: "<long pause> <softer> " },
{ label: "Intimate", wrap: ["<whisper>", "</whisper>"] },
{ label: "Amused", insert: "<chuckle> " },
];
/** Pure counterpart of `applyH3Tag` for presets: same {value, selection} contract. */
export function applyH3Preset(
value: string,
selectionStart: number,
selectionEnd: number,
preset: H3Preset,
): H3TagApplyResult {
const start = Math.max(0, Math.min(selectionStart, value.length));
const end = Math.max(start, Math.min(selectionEnd, value.length));
const before = value.slice(0, start);
const mid = value.slice(start, end);
const after = value.slice(end);
if (preset.wrap) {
const [open, close] = preset.wrap;
const innerStart = before.length + open.length;
return {
value: before + open + mid + close + after,
start: innerStart,
end: innerStart + mid.length,
};
}
const lead = before.length && !/\s$/.test(before) ? " " : "";
const chunk = lead + (preset.insert ?? "");
const caret = before.length + chunk.length;
return { value: before + chunk + after, start: caret, end: caret };
}
@@ -0,0 +1,78 @@
import { describe, expect, it } from "vitest";
import { parseJobJson } from "@/lib/jobJson";
const ok = (text: string) => {
const r = parseJobJson(text);
if (!r.ok) throw new Error(r.error);
return r;
};
describe("parseJobJson", () => {
it("joins a prompt given as lines and carries the settings over", () => {
const r = ok(
JSON.stringify({
model_id: "MiniMaxAI/MiniMax-H3",
name: "demo",
workload_type: "i2v",
prompt: ["summary:", "Hello.", "", "detailed_description:", "[Shot 1] A."],
references: [{ source: "/data/a.png", media_type: "image" }],
guidance_scale: 1,
num_frames: 243,
fps: 24,
num_gpus: 4,
use_fsdp_inference: true,
}),
);
expect(r.workloadType).toBe("i2v");
expect(r.job.prompt).toBe("summary:\nHello.\n\ndetailed_description:\n[Shot 1] A.");
expect(r.job).toMatchObject({
model_id: "MiniMaxAI/MiniMax-H3",
name: "demo",
guidance_scale: 1,
num_frames: 243,
num_gpus: 4,
use_fsdp_inference: true,
references: [{ source: "/data/a.png", media_type: "image" }],
});
});
it("accepts a plain string prompt", () => {
expect(ok(JSON.stringify({ model_id: "m", prompt: "one line" })).job.prompt).toBe("one line");
});
it("infers the workload when it isn't declared", () => {
expect(ok(JSON.stringify({ model_id: "m", prompt: "p" })).workloadType).toBe("t2v");
expect(
ok(JSON.stringify({ model_id: "m", prompt: "p", references: [{ source: "/a/b.mp4" }] })).workloadType,
).toBe("i2v");
expect(ok(JSON.stringify({ model_id: "m", prompt: "p", image_path: "/a.png" })).workloadType).toBe("i2v");
});
it("infers a reference's media type from its extension", () => {
const r = ok(
JSON.stringify({
model_id: "m",
prompt: "p",
references: [{ source: "/a/b.mp4" }, { source: "/a/c.wav" }, { source: "/a/d.png" }],
}),
);
expect(r.job.references?.map((x) => x.media_type)).toEqual(["video", "audio", "image"]);
});
it("ignores settings of the wrong type instead of passing them through", () => {
const r = ok(JSON.stringify({ model_id: "m", prompt: "p", num_frames: "many", use_fsdp_inference: "yes", extra: 1 }));
expect(r.job.num_frames).toBeUndefined();
expect(r.job.use_fsdp_inference).toBeUndefined();
expect(r.job).not.toHaveProperty("extra");
});
it.each([
["not json", "That file isn't valid JSON."],
["[1,2]", "Expected a JSON object describing one job."],
[JSON.stringify({ prompt: "p" }), "The job file has no model_id."],
[JSON.stringify({ model_id: "m", prompt: 5 }), "prompt must be text or a list of text lines."],
[JSON.stringify({ model_id: "m", prompt: "p", references: [{}] }), "references[0] needs a source path."],
])("rejects %s", (text, error) => {
expect(parseJobJson(text)).toEqual({ ok: false, error });
});
});
+102
View File
@@ -0,0 +1,102 @@
/**
* Read a job file (the `ref2va_*.json` shape) into the create-job form's
* job shape. Keys match the API's CreateJobRequest; `prompt` may be a string
* or a list of lines (joined with newlines), since job files keep it as lines
* for readability.
*/
import type { JobLike } from "@/lib/jobToFields";
export type JobJsonResult =
| { ok: true; job: JobLike; workloadType: "t2v" | "i2v" | "t2i" }
| { ok: false; error: string };
const WORKLOADS = ["t2v", "i2v", "t2i"] as const;
const MEDIA_TYPES = ["image", "video", "audio"] as const;
const VIDEO_EXT = /\.(mp4|mov|mkv|webm|avi)$/i;
const AUDIO_EXT = /\.(wav|mp3|flac|m4a|ogg)$/i;
const STRING_KEYS = ["name", "image_path", "last_image_path", "negative_prompt"] as const;
const NUMBER_KEYS = [
"num_inference_steps",
"num_frames",
"height",
"width",
"guidance_scale",
"guidance_rescale",
"fps",
"seed",
"num_gpus",
"vsa_sparsity",
"tp_size",
"sp_size",
] as const;
const BOOLEAN_KEYS = [
"dit_cpu_offload",
"dit_layerwise_offload",
"text_encoder_cpu_offload",
"vae_cpu_offload",
"image_encoder_cpu_offload",
"use_fsdp_inference",
"enable_torch_compile",
] as const;
function inferMediaType(source: string): (typeof MEDIA_TYPES)[number] {
if (VIDEO_EXT.test(source)) return "video";
if (AUDIO_EXT.test(source)) return "audio";
return "image";
}
export function parseJobJson(text: string): JobJsonResult {
let raw: unknown;
try {
raw = JSON.parse(text);
} catch {
return { ok: false, error: "That file isn't valid JSON." };
}
if (typeof raw !== "object" || raw === null || Array.isArray(raw)) {
return { ok: false, error: "Expected a JSON object describing one job." };
}
const data = raw as Record<string, unknown>;
if (typeof data.model_id !== "string" || !data.model_id.trim()) {
return { ok: false, error: "The job file has no model_id." };
}
let prompt: string;
if (typeof data.prompt === "string") prompt = data.prompt;
else if (Array.isArray(data.prompt) && data.prompt.every((l) => typeof l === "string")) {
prompt = (data.prompt as string[]).join("\n");
} else {
return { ok: false, error: "prompt must be text or a list of text lines." };
}
const references: { source: string; media_type: string }[] = [];
if (data.references !== undefined && data.references !== null) {
if (!Array.isArray(data.references)) return { ok: false, error: "references must be a list." };
for (const [i, ref] of data.references.entries()) {
const r = ref as Record<string, unknown> | null;
if (!r || typeof r.source !== "string" || !r.source) {
return { ok: false, error: `references[${i}] needs a source path.` };
}
const mediaType =
typeof r.media_type === "string" && (MEDIA_TYPES as readonly string[]).includes(r.media_type)
? r.media_type
: inferMediaType(r.source);
references.push({ source: r.source, media_type: mediaType });
}
}
const job: JobLike = { id: `imported-${Date.now()}`, model_id: data.model_id.trim(), prompt, references };
const target = job as unknown as Record<string, unknown>;
for (const key of STRING_KEYS) if (typeof data[key] === "string") target[key] = data[key];
for (const key of NUMBER_KEYS) if (typeof data[key] === "number" && Number.isFinite(data[key])) target[key] = data[key];
for (const key of BOOLEAN_KEYS) if (typeof data[key] === "boolean") target[key] = data[key];
const declared = (WORKLOADS as readonly string[]).includes(data.workload_type as string)
? (data.workload_type as (typeof WORKLOADS)[number])
: null;
const workloadType = declared ?? (references.length > 0 || job.image_path ? "i2v" : "t2v");
job.workload_type = workloadType;
return { ok: true, job, workloadType };
}
@@ -71,4 +71,8 @@ describe("referenceFileName", () => {
expect(referenceFileName("/a/b/c.mp4")).toBe("c.mp4");
expect(referenceFileName("c.mp4")).toBe("c.mp4");
});
it("names a last-frame reference by the job it comes from, not its internal id", () => {
expect(referenceFileName("job-last-frame:0a1b2c3d-1111-2222-3333-444455556666")).toBe("Last frame of job 0a1b2c3d");
});
});
@@ -72,6 +72,7 @@ export interface JobFormFields {
/** Uploads keep their original basename, so this is the display name. */
export function referenceFileName(source: string): string {
if (source.startsWith("job-last-frame:")) return `Last frame of job ${source.slice(15, 23)}`;
return source.split("/").filter(Boolean).pop() ?? source;
}
@@ -0,0 +1,107 @@
import { describe, expect, it } from 'vitest';
import { deferredLastFrameSource } from '@/lib/continuation';
import { planSceneQueue } from '@/lib/sceneQueue';
import { buildScenes, type SceneClip } from '@/lib/scenes';
import { makeJob } from '@/test/factories';
const SIX_SECTION = [
'subject_definitions:',
'<Subject 1> is a man.',
'',
'summary:',
'[reference generation] A man.',
'',
'retention_analysis:',
'<Subject 1> (appears in [Shot 1]): fully_preserved - suit.',
'',
'detailed_description:',
'[Shot 1] Wide shot. A man.',
'',
'overall_soundscape:',
'Quiet.',
'',
'non_diegetic_music:',
'N/A',
].join('\n');
const clips = (...overrides: Parameters<typeof makeJob>[0][]): SceneClip[] => {
const jobs = overrides.map((over, i) =>
makeJob({
id: `c${i + 1}`,
name: `s-clip-${i + 1}`,
created_at: i + 1,
status: 'pending',
prompt: SIX_SECTION,
...over,
}),
);
return buildScenes(jobs)[0].clips;
};
const ids = (list: { job: { id: string } }[]) => list.map((c) => c.job.id);
describe('planSceneQueue', () => {
it('queues every clip that can run, in scene order', () => {
const plan = planSceneQueue(clips({}, {}, {}), { chain: false });
expect(ids(plan.queue)).toEqual(['c1', 'c2', 'c3']);
expect(plan.skipped).toEqual([]);
});
it.each([
['completed', 'already finished'],
['running', 'already running'],
['queued', 'already queued'],
])('leaves %s clips alone', (status, reason) => {
const plan = planSceneQueue(clips({}, { status }, {}), { chain: false });
expect(ids(plan.queue)).toEqual(['c1', 'c3']);
expect(plan.skipped.map((s) => [s.clip.job.id, s.reason])).toEqual([['c2', reason]]);
});
it('requeues failed and stopped clips', () => {
const plan = planSceneQueue(clips({ status: 'failed' }, { status: 'stopped' }), { chain: false });
expect(ids(plan.queue)).toEqual(['c1', 'c2']);
});
describe('chaining clips through last frames', () => {
it('links each clip after the first to the one before it', () => {
const plan = planSceneQueue(clips({}, {}, {}), { chain: true });
expect(plan.chain.map((c) => [c.clip.job.id, c.previous.job.id])).toEqual([
['c2', 'c1'],
['c3', 'c2'],
]);
});
it('links nothing when not asked to', () => {
expect(planSceneQueue(clips({}, {}), { chain: false }).chain).toEqual([]);
});
it('links a clip to a previous clip that has already finished', () => {
const plan = planSceneQueue(clips({ status: 'completed' }, {}), { chain: true });
expect(plan.chain.map((c) => c.previous.job.id)).toEqual(['c1']);
expect(ids(plan.queue)).toEqual(['c2']);
});
it('does not link clips that are not being queued', () => {
const plan = planSceneQueue(clips({}, { status: 'completed' }, {}), { chain: true });
expect(plan.chain.map((c) => c.clip.job.id)).toEqual(['c3']);
});
it('keeps a frame the clip already opens on', () => {
const own = [{ source: deferredLastFrameSource('c1'), media_type: 'image' }];
const stale = [{ source: '/uploads/last_frames/last_frame_old-take.png', media_type: 'image' }];
const plan = planSceneQueue(clips({}, { references: own }, { references: stale }), { chain: true });
expect(plan.chain).toEqual([]);
expect(plan.cannotChain).toEqual([]);
});
it("reports clips whose prompt can't be rewritten instead of linking them", () => {
const plan = planSceneQueue(clips({}, { prompt: 'A plain prompt.' }), { chain: true });
expect(plan.chain).toEqual([]);
expect(plan.cannotChain).toHaveLength(1);
expect(plan.cannotChain[0].clip.job.id).toBe('c2');
expect(plan.cannotChain[0].reason).toMatch(/six-section/);
expect(ids(plan.queue)).toEqual(['c1', 'c2']); // still queued, just not linked
});
});
});
@@ -0,0 +1,52 @@
/**
* Work out what "queue this scene" would do, so the user can confirm it first.
*
* Clips are queued in scene order. The server runs them one after another and
* holds a clip that starts from another clip's last frame until that clip has
* finished -- so a whole scene can be queued at once and each clip still opens
* on the frame the one before it ended on. This decides which clips are queued,
* and (optionally) which get that link attached first.
*/
import { attachedLastFrames, whyCannotAttach } from "@/lib/continuation";
import type { SceneClip } from "@/lib/scenes";
/** Statuses a job can be queued from (the same ones it can be started or edited in). */
export const QUEUEABLE_STATUSES = ["pending", "failed", "stopped"];
export interface SceneQueuePlan {
/** clips to queue, in order */
queue: SceneClip[];
/** clips that will get the previous clip's last frame attached first */
chain: { clip: SceneClip; previous: SceneClip }[];
/** clips that would need the link but can't have it attached automatically */
cannotChain: { clip: SceneClip; reason: string }[];
/** clips left alone, and why */
skipped: { clip: SceneClip; reason: string }[];
}
function skipReason(status: string): string {
if (status === "completed") return "already finished";
if (status === "running") return "already running";
if (status === "queued") return "already queued";
return `is ${status}`;
}
export function planSceneQueue(clips: SceneClip[], { chain: wantChain }: { chain: boolean }): SceneQueuePlan {
const plan: SceneQueuePlan = { queue: [], chain: [], cannotChain: [], skipped: [] };
clips.forEach((clip, i) => {
if (!QUEUEABLE_STATUSES.includes(clip.job.status)) {
plan.skipped.push({ clip, reason: skipReason(clip.job.status) });
return;
}
plan.queue.push(clip);
const previous = clips[i - 1];
if (!wantChain || !previous) return;
// A clip that already opens on a frame (of this or another take) keeps it.
if (attachedLastFrames(clip.job.references).length > 0) return;
const blocked = whyCannotAttach(clip.job.prompt, clip.job.references);
if (blocked) plan.cannotChain.push({ clip, reason: blocked });
else plan.chain.push({ clip, previous });
});
return plan;
}
@@ -0,0 +1,250 @@
import { describe, expect, it } from "vitest";
import { UNNAMED_SCENE, buildScenes, clipSeconds, formatClock, parseSceneName, shotsOfJob } from "@/lib/scenes";
import { shotLines, shotProse } from "@/lib/h3Shots";
import type { Job } from "@/lib/types";
function job(over: Partial<Job> & { id: string }): Job {
return {
model_id: "m",
prompt: "",
status: "pending",
created_at: 0,
started_at: null,
finished_at: null,
error: null,
output_path: null,
log_file_path: null,
num_inference_steps: 50,
num_frames: 240,
fps: 24,
height: 768,
width: 1344,
guidance_scale: 1,
seed: 0,
num_gpus: 1,
progress: 0,
progress_msg: "",
phase: "",
...over,
};
}
const SECTIONS = (detailed: string, subjects = "<Subject 1> is a man.") =>
["subject_definitions:", subjects, "", "detailed_description:", detailed].join("\n");
describe("parseSceneName", () => {
it.each([
["wolf-lunch-clip-01", "wolf-lunch", 1],
["wolf-lunch-clip-15", "wolf-lunch", 15],
["closeup-fictional-actor-monologue-shot2", "closeup-fictional-actor-monologue", 2],
["Rooftop Standoff part 3", "Rooftop Standoff", 3],
["scene_a_07", "scene_a", 7],
])("%s -> stem %s, clip %i", (name, title, order) => {
expect(parseSceneName(name)).toEqual({ key: title.toLowerCase(), title, order });
});
it("treats a name with no trailing number as its own scene", () => {
expect(parseSceneName("Rooftop Standoff")).toEqual({ key: "rooftop standoff", title: "Rooftop Standoff", order: null });
});
it("keeps the word scene as part of the name", () => {
expect(parseSceneName("rooftop-scene-2")).toEqual({ key: "rooftop-scene", title: "rooftop-scene", order: 2 });
});
it("does not treat a bare number or number-only suffix as a stem", () => {
expect(parseSceneName("42").order).toBeNull();
expect(parseSceneName("clip-3").order).toBeNull();
});
it("puts blank or missing names in the unnamed group", () => {
expect(parseSceneName(undefined).key).toBe(UNNAMED_SCENE);
expect(parseSceneName(" ").key).toBe(UNNAMED_SCENE);
});
it("groups names that differ only in case", () => {
expect(parseSceneName("Wolf-Lunch-Clip-1").key).toBe(parseSceneName("wolf-lunch-clip-2").key);
});
});
describe("shotsOfJob", () => {
it("reads shots and dialogue out of a six-section prompt", () => {
const shots = shotsOfJob({
prompt: SECTIONS(
"[Shot 1] Wide shot. A dock.\n[Shot 2] At 00:02.000, the shot cuts to a close-up shot. (S1) <d>[English] Hi.</d>",
),
});
expect(shots).toHaveLength(2);
expect(shots[1].type).toBe("close-up");
expect(shotLines(shots[1])[0].text).toBe("Hi.");
expect(shots[0].subjectIds).toEqual([]);
});
it("reads shots out of a base-format integrated_multimodal_description", () => {
const shots = shotsOfJob({
prompt: "For the target video, <Picture 1> is used.\n\nintegrated_multimodal_description: [Shot 1] He (S1) says: <d>[English] Sorry.</d> <cutoff>\n\noverall_soundscape: Quiet.",
});
expect(shots).toHaveLength(1);
expect(shotLines(shots[0]).map((l) => l.text)).toEqual(["Sorry."]);
});
it("shows a plain prompt as a single shot", () => {
const shots = shotsOfJob({ prompt: "A toy car drives into a plush dog." });
expect(shots).toHaveLength(1);
expect(shotProse(shots[0])).toBe("A toy car drives into a plush dog.");
});
it("returns no shots for an empty prompt", () => {
expect(shotsOfJob({ prompt: "" })).toEqual([]);
});
});
describe("buildScenes", () => {
const three = [
job({ id: "c", name: "wolf-lunch-clip-03", created_at: 3, status: "pending", prompt: "[Shot 1] Wide shot. C." }),
job({ id: "a", name: "wolf-lunch-clip-01", created_at: 1, status: "completed", num_frames: 240, prompt: "[Shot 1] Wide shot. A.\n[Shot 2] At 00:02.000, the shot cuts to a close-up shot. A2." }),
job({ id: "b", name: "wolf-lunch-clip-02", created_at: 2, status: "completed", num_frames: 120, fps: 24, prompt: "[Shot 1] Close-up shot. B." }),
];
it("groups by stem and orders clips by their number, not creation order", () => {
const [scene] = buildScenes(three);
expect(scene.title).toBe("wolf-lunch");
expect(scene.clips.map((c) => c.job.id)).toEqual(["a", "b", "c"]);
expect(scene.clips.map((c) => c.position)).toEqual([1, 2, 3]);
});
it("lays every shot on one timeline using each clip's frames and fps", () => {
const [scene] = buildScenes(three);
expect(scene.clips.map((c) => c.startSeconds)).toEqual([0, 10, 15]);
expect(scene.totalSeconds).toBe(25); // 240/24 + 120/24 + 240/24 = 10 + 5 + 10
});
it("counts shots and statuses across the scene", () => {
const [scene] = buildScenes(three);
expect(scene.shotCount).toBe(4);
expect(scene.statusCounts).toEqual({ completed: 2, pending: 1 });
});
it("offsets each shot's cut time by the clip's start", () => {
const [scene] = buildScenes(three);
const second = scene.clips[1];
expect(second.shotStarts).toEqual([10]);
const first = scene.clips[0];
expect(first.shotStarts[0]).toBe(0);
expect(first.shotStarts[1]).toBe(2);
});
it("keeps separate scenes apart, newest first, with unnamed jobs last", () => {
const scenes = buildScenes([
job({ id: "1", name: "old-scene-1", created_at: 5 }),
job({ id: "2", name: "new-scene-1", created_at: 50 }),
job({ id: "3", name: "", created_at: 100 }),
]);
expect(scenes.map((s) => s.title)).toEqual(["new-scene", "old-scene", "Unnamed jobs"]);
});
it("puts clips without a number after numbered ones, oldest first", () => {
const [scene] = buildScenes([
job({ id: "x", name: "wolf-1", created_at: 9 }),
job({ id: "y", name: "wolf", created_at: 1 }),
job({ id: "z", name: "wolf-2", created_at: 2 }),
]);
// "wolf" shares the stem with wolf-1 / wolf-2, so it joins them -- last, having no number
expect(scene.clips.map((c) => c.job.id)).toEqual(["x", "z", "y"]);
});
it("flags a clip that continues the previous shot", () => {
const [scene] = buildScenes([
job({ id: "a", name: "s-1", prompt: "[Shot 1] Close-up shot. A speech. <cutoff>" }),
job({ id: "b", name: "s-2", prompt: "[Shot 1] Close-up shot, dolly. Continuing the same shot on <Subject 1>, still speaking." }),
job({ id: "c", name: "s-3", prompt: "summary:\n[reference generation] It continues the previous shot without a cut.\n\ndetailed_description:\n[Shot 1] Wide shot. X." }),
job({ id: "d", name: "s-4", prompt: "[Shot 1] Wide shot. Fresh." }),
]);
expect(scene.clips.map((c) => c.continuation)).toEqual([false, true, true, false]);
});
it("falls back to 24 fps when a job has none", () => {
expect(clipSeconds({ num_frames: 240, fps: undefined })).toBe(10);
expect(clipSeconds({ num_frames: 240, fps: 0 })).toBe(10);
expect(clipSeconds({ num_frames: 240, fps: 30 })).toBe(8);
});
it("returns no scenes for no jobs", () => {
expect(buildScenes([])).toEqual([]);
});
});
describe("formatClock", () => {
it("formats m:ss and h:mm:ss", () => {
expect(formatClock(0)).toBe("0:00");
expect(formatClock(9.6)).toBe("0:10");
expect(formatClock(65)).toBe("1:05");
expect(formatClock(3725)).toBe("1:02:05");
expect(formatClock(-4)).toBe("0:00");
});
});
describe("takes of the same clip", () => {
const take = (id: string, n: number, over: Partial<Job> = {}) =>
job({ id, name: `wolf-lunch-clip-0${n}`, prompt: `[Shot 1] Wide shot. ${id}.`, num_frames: 240, ...over });
it("counts re-runs of a clip number once, not as extra clips", () => {
const [scene] = buildScenes([
take("a1", 1, { created_at: 1, status: "completed" }),
take("a2", 1, { created_at: 2, status: "completed" }),
take("a3", 1, { created_at: 3, status: "completed" }),
take("b1", 2, { created_at: 4, status: "completed" }),
]);
expect(scene.clips).toHaveLength(2);
expect(scene.totalSeconds).toBe(20);
expect(scene.shotCount).toBe(2);
expect(scene.statusCounts).toEqual({ completed: 2 });
expect(scene.clips[0].takes.map((j) => j.id)).toEqual(["a1", "a2", "a3"]);
});
it("shows the most recent completed take, even over a newer failed one", () => {
const [scene] = buildScenes([
take("old", 1, { created_at: 1, status: "completed", finished_at: 10 }),
take("good", 1, { created_at: 2, status: "completed", finished_at: 20 }),
take("bad", 1, { created_at: 3, status: "failed", finished_at: 30 }),
]);
expect(scene.clips[0].job.id).toBe("good");
});
it("falls back to the newest take when none completed", () => {
const [scene] = buildScenes([
take("p1", 1, { created_at: 1, status: "failed" }),
take("p2", 1, { created_at: 2, status: "pending" }),
]);
expect(scene.clips[0].job.id).toBe("p2");
});
it("shows the take the user chose, and lays the timeline out from it", () => {
const jobs = [
take("short", 1, { created_at: 1, status: "completed", num_frames: 120 }),
take("long", 1, { created_at: 2, status: "completed", num_frames: 480 }),
take("next", 2, { created_at: 3 }),
];
const [auto] = buildScenes(jobs);
expect(auto.clips[0].job.id).toBe("long");
expect(auto.clips[1].startSeconds).toBe(20);
const [chosen] = buildScenes(jobs, { [auto.clips[0].takeKey as string]: "short" });
expect(chosen.clips[0].job.id).toBe("short");
expect(chosen.clips[1].startSeconds).toBe(5);
});
it("ignores a choice that isn't one of the takes", () => {
const jobs = [take("a", 1, { status: "completed" })];
const [scene] = buildScenes(jobs, { "wolf-lunch#1": "no-such-job" });
expect(scene.clips[0].job.id).toBe("a");
});
it("keeps each unnumbered job as its own clip", () => {
const [scene] = buildScenes([
job({ id: "u1", name: "solo", created_at: 1 }),
job({ id: "u2", name: "solo", created_at: 2 }),
]);
expect(scene.clips.map((c) => c.job.id)).toEqual(["u1", "u2"]);
expect(scene.clips.every((c) => c.takeKey === null)).toBe(true);
});
});
+182
View File
@@ -0,0 +1,182 @@
/**
* Group jobs into scenes and lay their shots out on one timeline.
*
* A long scene is generated as many short clips, one job each. Nothing on the
* server links them, so the link is the job name: `wolf-lunch-clip-01`,
* `wolf-lunch-clip-02`, ... share the stem `wolf-lunch` and are ordered by the
* trailing number. Everything else -- shots, dialogue, subjects -- is read back
* out of each job's own prompt, so no backend support is needed.
*/
import { readField, shotTarget } from "@/lib/h3PromptFields";
import {
parseDetailedDescription,
parseSubjectLabels,
resolveShotStarts,
shotProse,
type Shot,
} from "@/lib/h3Shots";
import type { Job } from "@/lib/types";
export const UNNAMED_SCENE = "__unnamed__";
export const DEFAULT_FPS = 24;
// "<stem>[-_. ][clip|part|pt|shot|take][-_. ]<number>". "scene" is deliberately not
// a clip word: it is far more likely part of the scene's own name ("rooftop-scene-2").
const CLIP_SUFFIX_RE = /^(.*?)[\s._-]*(?:clip|part|pt|shot|take)?[\s._-]*(\d+)$/i;
export interface SceneName {
key: string;
title: string;
/** the clip number from the end of the name, or null when there isn't one */
order: number | null;
}
export function parseSceneName(name: string | undefined | null): SceneName {
const trimmed = (name ?? "").trim();
if (!trimmed) return { key: UNNAMED_SCENE, title: "Unnamed jobs", order: null };
const m = trimmed.match(CLIP_SUFFIX_RE);
const stem = m ? m[1].replace(/[\s._-]+$/, "") : "";
if (m && stem) return { key: stem.toLowerCase(), title: stem, order: Number(m[2]) };
return { key: trimmed.toLowerCase(), title: trimmed, order: null };
}
/** The shots a job's prompt describes, wherever they live in it (base, reference, or plain prompt). */
export function shotsOfJob(job: Pick<Job, "prompt">): Shot[] {
const prompt = job.prompt ?? "";
const labels = parseSubjectLabels(readField(prompt, "subject_definitions") ?? "");
return parseDetailedDescription(shotTarget(prompt).body, labels).shots;
}
export interface SceneClip {
/** the take shown for this clip */
job: Job;
/**
* Every job that claims this clip number, oldest first. Re-running a clip
* gives the same name again, so these are takes of one clip, not extra clips.
*/
takes: Job[];
/** identifies this clip for choosing a take; null for a clip with no number */
takeKey: string | null;
/** 1-based place in the scene */
position: number;
shots: Shot[];
/** seconds from the start of the scene to the start of this clip */
startSeconds: number;
durationSeconds: number;
/** each shot's cut-in time measured from the start of the scene */
shotStarts: number[];
/** this clip picks up in the middle of the previous clip's last shot */
continuation: boolean;
}
export interface Scene {
key: string;
title: string;
clips: SceneClip[];
totalSeconds: number;
shotCount: number;
statusCounts: Record<string, number>;
/** most recent created_at among the clips, for ordering scenes */
latest: number;
}
export function clipSeconds(job: Pick<Job, "num_frames" | "fps">): number {
const fps = job.fps && job.fps > 0 ? job.fps : DEFAULT_FPS;
return Math.max(0, (job.num_frames ?? 0) / fps);
}
function isContinuation(job: Job, shots: Shot[]): boolean {
if (/continues the previous shot/i.test(job.prompt ?? "")) return true;
return shots.length > 0 && /^Continuing the same shot/i.test(shotProse(shots[0]));
}
function createdAt(job: Job): number {
return job.created_at ?? 0;
}
/** The take to show: the one the user chose, else the most recent completed one, else the newest. */
function pickTake(takes: Job[], choice?: string): Job {
const chosen = choice ? takes.find((j) => j.id === choice) : undefined;
if (chosen) return chosen;
const recent = (a: Job, b: Job) => (b.finished_at ?? createdAt(b)) - (a.finished_at ?? createdAt(a));
const completed = takes.filter((j) => j.status === "completed").sort(recent);
return completed[0] ?? [...takes].sort((a, b) => createdAt(b) - createdAt(a))[0];
}
/**
* Group inference jobs into scenes, each with its clips in order and a
* scene-wide timeline. `choices` maps a clip's `takeKey` to the job id of the
* take to show in place of the default.
*/
export function buildScenes(jobs: Job[], choices: Record<string, string> = {}): Scene[] {
const groups = new Map<string, { title: string; entries: { job: Job; order: number | null }[] }>();
for (const job of jobs) {
const { key, title, order } = parseSceneName(job.name);
const group = groups.get(key) ?? { title, entries: [] };
group.entries.push({ job, order });
groups.set(key, group);
}
const scenes: Scene[] = [];
for (const [key, { title, entries }] of groups) {
// One clip per clip number (its takes together); each unnumbered job stands alone, after the numbered ones.
const byNumber = new Map<number, Job[]>();
const unnumbered: Job[][] = [];
for (const { job, order } of entries) {
if (order === null) unnumbered.push([job]);
else byNumber.set(order, [...(byNumber.get(order) ?? []), job]);
}
const slots: { takes: Job[]; takeKey: string | null }[] = [
...[...byNumber.entries()].sort(([a], [b]) => a - b).map(([order, takes]) => ({ takes, takeKey: `${key}#${order}` })),
...unnumbered.sort((a, b) => createdAt(a[0]) - createdAt(b[0])).map((takes) => ({ takes, takeKey: null })),
];
let elapsed = 0;
const statusCounts: Record<string, number> = {};
const clips: SceneClip[] = slots.map(({ takes, takeKey }, i) => {
const ordered = [...takes].sort((a, b) => createdAt(a) - createdAt(b));
const job = pickTake(ordered, takeKey ? choices[takeKey] : undefined);
const shots = shotsOfJob(job);
const durationSeconds = clipSeconds(job);
const clip: SceneClip = {
job,
takes: ordered,
takeKey,
position: i + 1,
shots,
startSeconds: elapsed,
durationSeconds,
shotStarts: resolveShotStarts(shots).map((t) => elapsed + t),
continuation: isContinuation(job, shots),
};
elapsed += durationSeconds;
statusCounts[job.status] = (statusCounts[job.status] ?? 0) + 1;
return clip;
});
scenes.push({
key,
title,
clips,
totalSeconds: elapsed,
shotCount: clips.reduce((n, c) => n + c.shots.length, 0),
statusCounts,
latest: Math.max(0, ...entries.map((e) => createdAt(e.job))),
});
}
// Newest scene first; the catch-all group of unnamed jobs goes last.
return scenes.sort((a, b) => {
if ((a.key === UNNAMED_SCENE) !== (b.key === UNNAMED_SCENE)) return a.key === UNNAMED_SCENE ? 1 : -1;
return b.latest - a.latest;
});
}
/** m:ss, or h:mm:ss past an hour -- for clip ranges and runtimes. */
export function formatClock(seconds: number): string {
const total = Math.max(0, Math.round(seconds));
const h = Math.floor(total / 3600);
const m = Math.floor((total % 3600) / 60);
const sec = String(total % 60).padStart(2, "0");
return h > 0 ? `${h}:${String(m).padStart(2, "0")}:${sec}` : `${m}:${sec}`;
}
+23
View File
@@ -2,6 +2,19 @@
export type JobType = "inference" | "finetuning" | "distillation";
/**
* One razor-cut section of a clip's kept (already start/end-trimmed) range,
* and its own color grade. `end_seconds` is seconds into that kept range, not
* the original file's timeline; only the last section in a list may leave it
* null, meaning "to the end". Sections are contiguous, in order, no gaps.
*/
export interface GradeSegment {
end_seconds: number | null;
brightness: number;
contrast: number;
saturation: number;
}
export interface Job {
id: string;
model_id: string;
@@ -11,13 +24,23 @@ export interface Job {
workload_type?: string;
status: string;
created_at: number;
/** when the job joined the queue; null unless it is queued */
queued_at?: number | null;
started_at: number | null;
finished_at: number | null;
error: string | null;
output_path: string | null;
log_file_path: string | null;
/** the range/sections last applied via the trim API; neutral/full-range for a never-edited job */
edit_start_seconds?: number;
edit_end_seconds?: number | null;
edit_segments?: GradeSegment[];
num_inference_steps: number;
num_frames: number;
/** frames per second; the API sends it, older cached objects may not have it */
fps?: number;
/** reference media (image/video/audio) attached to the job, as saved on the server */
references?: { source: string; media_type: string }[] | null;
height: number;
width: number;
guidance_scale: number;
+17
View File
@@ -16,6 +16,23 @@ if (!globalThis.crypto?.randomUUID) {
});
}
// jsdom's Blob/File predate Blob.text(); real browsers all have it.
if (!Blob.prototype.text) {
Blob.prototype.text = function text(this: Blob) {
return new Promise<string>((resolve, reject) => {
const reader = new FileReader();
reader.onload = () => resolve(String(reader.result));
reader.onerror = () => reject(reader.error);
reader.readAsText(this);
});
};
}
// jsdom doesn't implement scrollIntoView.
if (!Element.prototype.scrollIntoView) {
Element.prototype.scrollIntoView = vi.fn();
}
if (!URL.createObjectURL) {
URL.createObjectURL = vi.fn(() => 'blob:mock-url');
}
@@ -0,0 +1,278 @@
# SPDX-License-Identifier: Apache-2.0
"""Queued jobs run in order, one at a time, and a clip waits for the clip whose last frame it uses."""
from __future__ import annotations
from dataclasses import dataclass, field
from typing import Any
import pytest
from fastvideo_studio.job_queue import (
circular_dependency,
deferred_last_clip_source,
deferred_last_frame_source,
dependencies,
dependency_of,
plan_dispatch,
resolve_references,
UnresolvedReference,
)
@dataclass
class J:
id: str
status: str = "pending"
queued_at: float | None = None
references: list[dict[str, Any]] = field(default_factory=list)
name: str = ""
def after(job_id: str) -> list[dict[str, str]]:
return [{"source": deferred_last_frame_source(job_id), "media_type": "image"}]
def queue(*jobs: J) -> dict[str, J]:
for i, job in enumerate(jobs):
if job.status == "queued" and job.queued_at is None:
job.queued_at = float(i)
return {j.id: j for j in jobs}
def test_deferred_source_round_trips():
assert dependency_of({"source": deferred_last_frame_source("abc")}) == "abc"
@pytest.mark.parametrize("ref", [None, {}, {"source": "/tmp/a.png"}, {"source": "job-last-frame:"}, {"source": 3}])
def test_ordinary_references_have_no_dependency(ref):
assert dependency_of(ref) is None
def test_dependencies_keep_order_and_drop_repeats():
refs = [*after("b"), {"source": "/x.png"}, *after("a"), *after("b")]
assert dependencies(refs) == ["b", "a"]
class TestDispatchOrder:
def test_starts_the_oldest_queued_job_first(self):
jobs = queue(J("late", "queued", 5.0), J("early", "queued", 1.0))
assert plan_dispatch(jobs, 1).start == ["early"]
def test_runs_only_max_concurrent_including_running_jobs(self):
jobs = queue(J("r", "running"), J("a", "queued"), J("b", "queued"))
assert plan_dispatch(jobs, 1).start == []
assert plan_dispatch(jobs, 2).start == ["a"]
assert plan_dispatch(jobs, 3).start == ["a", "b"]
def test_ignores_jobs_that_are_not_queued(self):
jobs = queue(J("p"), J("c", "completed"), J("f", "failed"), J("s", "stopped"))
assert plan_dispatch(jobs, 4).start == []
def test_a_limit_below_one_still_runs_one_job(self):
assert plan_dispatch(queue(J("a", "queued")), 0).start == ["a"]
def test_ties_are_broken_by_id_for_a_stable_order(self):
jobs = {"b": J("b", "queued", 1.0), "a": J("a", "queued", 1.0)}
assert plan_dispatch(jobs, 2).start == ["a", "b"]
class TestWaitingOnAFrame:
def test_waits_while_the_source_clip_is_running_or_queued(self):
for source_status in ("running", "queued"):
jobs = queue(J("one", source_status), J("two", "queued", references=after("one")))
plan = plan_dispatch(jobs, 2)
assert "two" not in plan.start and not plan.fail
def test_starts_once_the_source_clip_completed(self):
jobs = queue(J("one", "completed"), J("two", "queued", references=after("one")))
assert plan_dispatch(jobs, 1).start == ["two"]
def test_waiting_does_not_hold_up_later_jobs(self):
jobs = queue(
J("one", "running"),
J("two", "queued", references=after("one")),
J("free", "queued"),
)
assert plan_dispatch(jobs, 2).start == ["free"]
def test_a_source_that_is_only_pending_keeps_waiting(self):
# The user may still queue it.
jobs = queue(J("one", "pending"), J("two", "queued", references=after("one")))
plan = plan_dispatch(jobs, 1)
assert plan.start == [] and plan.fail == {}
def test_needs_every_source(self):
refs = [*after("a"), *after("b")]
jobs = queue(J("a", "completed"), J("b", "running"), J("c", "queued", references=refs))
assert plan_dispatch(jobs, 2).start == []
jobs["b"].status = "completed"
assert plan_dispatch(jobs, 2).start == ["c"]
def test_a_chain_advances_one_clip_at_a_time(self):
jobs = queue(
J("c1", "queued"),
J("c2", "queued", references=after("c1")),
J("c3", "queued", references=after("c2")),
)
assert plan_dispatch(jobs, 5).start == ["c1"]
jobs["c1"].status = "completed"
assert plan_dispatch(jobs, 5).start == ["c2"]
jobs["c2"].status = "completed"
assert plan_dispatch(jobs, 5).start == ["c3"]
class TestImpossibleJobs:
@pytest.mark.parametrize(("status", "word"), [("failed", "failed"), ("stopped", "stopped")])
def test_fails_when_the_source_failed_or_was_stopped(self, status, word):
jobs = queue(J("one", status, name="Clip 1"), J("two", "queued", references=after("one")))
plan = plan_dispatch(jobs, 1)
assert plan.start == []
assert "Clip 1" in plan.fail["two"] and word in plan.fail["two"]
def test_fails_when_the_source_was_deleted(self):
jobs = queue(J("two", "queued", references=after("gone")))
assert "deleted" in plan_dispatch(jobs, 1).fail["two"]
def test_failure_cascades_down_the_chain(self):
jobs = queue(
J("c1", "failed"),
J("c2", "queued", references=after("c1")),
J("c3", "queued", references=after("c2")),
J("c4", "queued", references=after("c3")),
J("other", "queued"),
)
plan = plan_dispatch(jobs, 3)
assert set(plan.fail) == {"c2", "c3", "c4"}
assert plan.start == ["other"]
def test_a_failed_job_frees_its_slot_for_the_next(self):
jobs = queue(
J("c1", "failed"),
J("c2", "queued", references=after("c1")),
J("next", "queued"),
)
assert plan_dispatch(jobs, 1).start == ["next"]
def test_planning_does_not_change_the_jobs(self):
jobs = queue(J("c1", "failed"), J("c2", "queued", references=after("c1")))
plan_dispatch(jobs, 1)
assert jobs["c2"].status == "queued"
class TestCircularDependency:
def test_none_for_a_plain_chain(self):
jobs = queue(J("a"), J("b", references=after("a")), J("c", references=after("b")))
assert circular_dependency(jobs, "c") is None
def test_finds_a_loop_through_the_job(self):
jobs = queue(J("a", references=after("c")), J("b", references=after("a")), J("c", references=after("b")))
assert circular_dependency(jobs, "a") is not None
def test_a_job_that_waits_on_itself(self):
assert circular_dependency(queue(J("a", references=after("a"))), "a") is not None
def test_a_loop_elsewhere_is_not_this_jobs_problem(self):
jobs = queue(
J("a", references=after("b")),
J("b", references=after("a")),
J("c", references=after("a")),
)
assert circular_dependency(jobs, "c") is None
class TestResolveReferences:
def _jobs(self, **statuses: str) -> dict[str, J]:
return {k: J(k, v, name=f"Clip {k}") for k, v in statuses.items()}
def test_replaces_a_deferred_reference_with_the_extracted_frame(self):
jobs = self._jobs(one="completed")
stored = [{"source": "/ref.png", "media_type": "image"}, *after("one")]
resolved = resolve_references(stored, jobs.get, lambda job: f"/frames/{job.id}.png")
assert resolved == [
{"source": "/ref.png", "media_type": "image"},
{"source": "/frames/one.png", "media_type": "image"},
]
def test_leaves_the_stored_references_untouched(self):
jobs = self._jobs(one="completed")
stored = after("one")
resolve_references(stored, jobs.get, lambda job: "/f.png")
assert stored == after("one")
def test_no_references(self):
assert resolve_references(None, {}.get, lambda job: "") == []
def test_refuses_an_unfinished_source(self):
jobs = self._jobs(one="running")
with pytest.raises(UnresolvedReference, match="hasn't finished.*running"):
resolve_references(after("one"), jobs.get, lambda job: "/f.png")
def test_refuses_a_deleted_source(self):
with pytest.raises(UnresolvedReference, match="no longer exists"):
resolve_references(after("gone"), {}.get, lambda job: "/f.png")
def test_passes_on_the_extraction_error(self):
class Boom(Exception):
detail = "The video has no frames."
def fail(job):
raise Boom()
with pytest.raises(UnresolvedReference, match="no frames"):
resolve_references(after("one"), self._jobs(one="completed").get, fail)
def after_clip(job_id: str) -> list[dict[str, str]]:
return [{"source": deferred_last_clip_source(job_id), "media_type": "video"}]
class TestResolveClipReferences:
"""The same resolution, but for a trailing-clip (job-last-clip:) reference."""
def _jobs(self, **statuses: str) -> dict[str, J]:
return {k: J(k, v, name=f"Clip {k}") for k, v in statuses.items()}
def test_uses_the_clip_resolver_not_the_frame_one(self):
jobs = self._jobs(one="completed")
stored = after_clip("one")
resolved = resolve_references(
stored, jobs.get,
last_frame=lambda job: f"/frames/{job.id}.png",
last_clip=lambda job: f"/clips/{job.id}.mp4",
)
assert resolved == [{"source": "/clips/one.mp4", "media_type": "video"}]
def test_a_mix_of_frame_and_clip_references_each_use_their_own_resolver(self):
jobs = self._jobs(a="completed", b="completed")
stored = [*after("a"), *after_clip("b"), {"source": "/x.png", "media_type": "image"}]
resolved = resolve_references(
stored, jobs.get,
last_frame=lambda job: f"/frames/{job.id}.png",
last_clip=lambda job: f"/clips/{job.id}.mp4",
)
assert resolved == [
{"source": "/frames/a.png", "media_type": "image"},
{"source": "/clips/b.mp4", "media_type": "video"},
{"source": "/x.png", "media_type": "image"},
]
def test_refuses_an_unfinished_source(self):
jobs = self._jobs(one="running")
with pytest.raises(UnresolvedReference, match="hasn't finished.*running"):
resolve_references(after_clip("one"), jobs.get, last_frame=lambda j: "", last_clip=lambda j: "/c.mp4")
def test_refuses_a_deleted_source(self):
with pytest.raises(UnresolvedReference, match="no longer exists"):
resolve_references(after_clip("gone"), {}.get, last_frame=lambda j: "", last_clip=lambda j: "/c.mp4")
def test_without_a_clip_resolver_a_clip_reference_is_refused_not_ignored(self):
jobs = self._jobs(one="completed")
with pytest.raises(UnresolvedReference, match="can't resolve"):
resolve_references(after_clip("one"), jobs.get, last_frame=lambda j: "/f.png")
def test_dependencies_and_circular_checks_see_clip_references_too(self):
jobs = self._jobs(a="pending")
jobs["a"].references = after_clip("a")
assert dependency_of(after_clip("x")[0]) == "x"
assert dependencies(after_clip("a") + after("a")) == ["a"]
assert circular_dependency(jobs, "a") is not None
@@ -0,0 +1,768 @@
# SPDX-License-Identifier: Apache-2.0
"""JobRunner's queue: order, concurrency, last-frame dependencies, and surviving a restart.
Runs without a GPU: `fastvideo` is stubbed (importing it needs a working triton
driver) and jobs "run" by waiting on an event instead of generating video.
"""
from __future__ import annotations
import collections
import importlib.util
import sys
import threading
import time
import types
import uuid
from pathlib import Path
import pytest
from fastvideo_studio.database import Database
from fastvideo_studio.frames import FrameError
from fastvideo_studio.job_queue import deferred_last_frame_source
RUNNER_PATH = Path(__file__).resolve().parents[1] / "job_runner.py"
@pytest.fixture
def job_runner_module(monkeypatch):
class Manager:
def shutdown(self):
pass
stub_utils = types.ModuleType("fastvideo.utils")
stub_utils.get_mp_context = lambda: types.SimpleNamespace(Manager=Manager)
stub_pkg = types.ModuleType("fastvideo")
stub_pkg.utils = stub_utils
monkeypatch.setitem(sys.modules, "fastvideo", stub_pkg)
monkeypatch.setitem(sys.modules, "fastvideo.utils", stub_utils)
spec = importlib.util.spec_from_file_location("job_runner_under_test", RUNNER_PATH)
module = importlib.util.module_from_spec(spec)
monkeypatch.setitem(sys.modules, spec.name, module) # @dataclass looks its module up here
spec.loader.exec_module(module)
return module
class Harness:
"""Builds runners whose jobs finish only when the test says so."""
def __init__(self, module, tmp_path, monkeypatch):
self.module = module
self.tmp_path = tmp_path
self.gates: dict[str, threading.Event] = collections.defaultdict(threading.Event)
self.outcomes: dict[str, object] = {}
self.ran: list[str] = []
self.runners = []
self.thread_of: dict[str, threading.Thread] = {}
self.closed = False
harness = self
def fake_run(runner_self, job):
harness.thread_of[job.id] = threading.current_thread()
harness.ran.append(job.id)
if harness.closed:
harness.gates[job.id].set()
harness.gates[job.id].wait(10)
job.status = harness.outcomes.get(job.id, module.JobStatus.COMPLETED)
if job.status == module.JobStatus.FAILED:
job.error = "boom"
job.finished_at = time.time()
runner_self._save_job(job)
monkeypatch.setattr(module.JobRunner, "_run_job", fake_run)
def runner(self, **kwargs):
runner = self.module.JobRunner(
output_dir=str(self.tmp_path / "out"),
log_dir=str(self.tmp_path / "logs"),
database=Database(self.tmp_path / "t.db"),
upload_dir=str(self.tmp_path / "uploads"),
**kwargs,
)
self.runners.append(runner)
return runner
def make(self, runner, name, **over):
kwargs = dict(job_id=str(uuid.uuid4()), model_id="m", name=name, prompt="p")
kwargs.update(over)
return runner.create_job(**kwargs)
def finish(self, job, status=None):
if status is not None:
self.outcomes[job.id] = status
self.gates[job.id].set()
def release_all(self):
for gate in self.gates.values():
gate.set()
def wait_for(condition, timeout=5.0):
deadline = time.time() + timeout
while time.time() < deadline:
if condition():
return True
time.sleep(0.01)
return False
@pytest.fixture
def h(job_runner_module, tmp_path, monkeypatch):
harness = Harness(job_runner_module, tmp_path, monkeypatch)
yield harness
# Let every job (and the queue's follow-up launches) finish while `_run_job` is
# still patched, so no stray thread starts a real job after the test is over.
harness.closed = True
active = ("queued", "running")
for runner in harness.runners:
for _ in range(100):
harness.release_all()
if not any(j.status.value in active for j in runner.list_jobs()):
break
time.sleep(0.05)
runner._shutdown() # stops the worker threads
def statuses(runner, *jobs):
return [runner.get_job(j.id).status.value for j in jobs]
def frame_of(job):
return [{"source": deferred_last_frame_source(job.id), "media_type": "image"}]
class TestOrderAndConcurrency:
def test_one_job_runs_at_a_time_in_the_order_queued(self, h):
r = h.runner()
a, b, c = (h.make(r, n) for n in "abc")
r.enqueue_jobs([b.id, c.id, a.id])
assert wait_for(lambda: statuses(r, a, b, c) == ["queued", "running", "queued"])
h.finish(b)
assert wait_for(lambda: statuses(r, a, b, c) == ["queued", "completed", "running"])
h.finish(c)
assert wait_for(lambda: statuses(r, a, b, c) == ["running", "completed", "completed"])
h.finish(a)
assert wait_for(lambda: statuses(r, a, b, c) == ["completed"] * 3)
assert h.ran == [b.id, c.id, a.id]
def test_runs_as_many_at_once_as_allowed(self, h):
r = h.runner(max_concurrent_jobs=2)
a, b, c = (h.make(r, n) for n in "abc")
r.enqueue_jobs([a.id, b.id, c.id])
assert wait_for(lambda: statuses(r, a, b, c) == ["running", "running", "queued"])
h.finish(a)
assert wait_for(lambda: statuses(r, a, b, c) == ["completed", "running", "running"])
def test_a_failed_job_does_not_stop_the_queue(self, h):
r = h.runner()
a, b = h.make(r, "a"), h.make(r, "b")
r.enqueue_jobs([a.id, b.id])
h.finish(a, h.module.JobStatus.FAILED)
assert wait_for(lambda: statuses(r, a, b) == ["failed", "running"])
def test_queued_jobs_have_a_queue_time_and_running_ones_do_not(self, h):
r = h.runner()
a, b = h.make(r, "a"), h.make(r, "b")
r.enqueue_jobs([a.id, b.id])
assert wait_for(lambda: r.get_job(a.id).status.value == "running")
assert r.get_job(a.id).queued_at is None
assert r.get_job(b.id).queued_at is not None
assert r.get_job(b.id).to_dict()["status"] == "queued"
class TestLongLivedWorkers:
"""Model workers die with the thread that started them, so inference jobs share long-lived threads."""
def _run_all(self, h, r, n):
jobs = [h.make(r, f"j{i}") for i in range(n)]
for job in jobs:
h.gates[job.id].set() # each finishes as soon as it starts
r.enqueue_jobs([j.id for j in jobs])
assert wait_for(lambda: statuses(r, *jobs) == ["completed"] * n)
return jobs
def test_consecutive_jobs_run_on_the_same_thread_and_it_stays_alive(self, h):
r = h.runner()
jobs = self._run_all(h, r, 4)
threads = {h.thread_of[j.id] for j in jobs}
assert len(threads) == 1
assert threads.pop().is_alive()
def test_the_thread_is_not_one_made_for_the_job(self, h):
r = h.runner()
(job,) = self._run_all(h, r, 1)
assert h.thread_of[job.id].name.startswith("inference-worker")
def test_as_many_workers_as_jobs_allowed_at_once(self, h):
r = h.runner(max_concurrent_jobs=2)
jobs = self._run_all(h, r, 6)
assert len({h.thread_of[j.id] for j in jobs}) <= 2
def test_two_jobs_at_once_use_two_different_workers(self, h):
r = h.runner(max_concurrent_jobs=2)
a, b = h.make(r, "a"), h.make(r, "b")
r.enqueue_jobs([a.id, b.id])
assert wait_for(lambda: statuses(r, a, b) == ["running", "running"])
assert wait_for(lambda: len(h.thread_of) == 2)
assert h.thread_of[a.id] is not h.thread_of[b.id]
def test_a_job_that_blows_up_does_not_kill_its_worker(self, h):
r = h.runner()
bad, good = h.make(r, "bad"), h.make(r, "good")
original = h.module.JobRunner._run_job
def explode_once(self, job):
if job.id == bad.id:
raise RuntimeError("boom")
return original(self, job)
h.module.JobRunner._run_job = explode_once
h.gates[good.id].set()
r.enqueue_jobs([bad.id, good.id])
assert wait_for(lambda: statuses(r, bad, good) == ["failed", "completed"])
assert "boom" in r.get_job(bad.id).error
def test_a_direct_start_is_refused_while_the_workers_are_busy(self, h):
r = h.runner()
first, second = h.make(r, "first"), h.make(r, "second")
r.start_job(first.id)
with pytest.raises(ValueError, match="already running.*Queue this job"):
r.start_job(second.id)
assert r.get_job(second.id).status.value == "pending"
def test_a_chain_runs_clean_when_every_clip_finishes_instantly(self, h):
r = h.runner()
clips = []
for i in range(6):
clips.append(h.make(r, f"clip {i}", references=frame_of(clips[-1]) if clips else []))
for clip in clips:
h.gates[clip.id].set()
r.enqueue_jobs([c.id for c in clips])
assert wait_for(lambda: statuses(r, *clips) == ["completed"] * 6)
assert h.ran == [c.id for c in clips]
class TestGeneratorCache:
@pytest.fixture
def generators(self, job_runner_module, monkeypatch):
made = []
class FakeGenerator:
def __init__(self, model_id):
self.model_id = model_id
self.shut_down = False
made.append(self)
def shutdown(self):
self.shut_down = True
stub = types.SimpleNamespace(from_pretrained=lambda model_id, *a, **k: FakeGenerator(model_id))
monkeypatch.setattr(sys.modules["fastvideo"], "VideoGenerator", stub, raising=False)
return made
def _get(self, runner, model="m"):
return runner._get_or_create_generator(model, "i2v", 1)
def _on_thread(self, fn):
box = []
t = threading.Thread(target=lambda: box.append(fn()))
t.start()
t.join()
return box[0]
def test_the_thread_that_built_it_keeps_reusing_it(self, h, generators):
r = h.runner()
first = self._get(r)
assert self._get(r) is first
assert len(generators) == 1
def test_a_generator_from_an_exited_thread_is_dropped_and_replaced(self, h, generators):
r = h.runner()
old = self._on_thread(lambda: self._get(r)) # that thread has exited: its workers are gone
new = self._get(r)
assert new is not old
assert old.shut_down
assert len(generators) == 2
def test_threads_do_not_share_generators(self, h, generators):
# Sharing would break the second the moment the first one's thread ends.
r = h.runner()
mine = self._get(r)
other = self._on_thread(lambda: self._get(r))
assert other is not mine
assert not mine.shut_down # still owned by a live thread
def test_a_different_model_replaces_the_loaded_one_instead_of_sitting_beside_it(self, h, generators):
r = h.runner()
first = self._get(r, "model-a")
second = self._get(r, "model-b")
assert first.shut_down # its GPU memory is freed before the next load
assert not second.shut_down
assert self._get(r, "model-b") is second
def test_switching_back_reloads(self, h, generators):
r = h.runner()
a1 = self._get(r, "model-a")
self._get(r, "model-b")
a2 = self._get(r, "model-a")
assert a2 is not a1
def test_discarding_unloads_this_threads_models_only(self, h, generators):
r = h.runner()
mine = self._get(r)
gate, done = threading.Event(), threading.Event()
theirs = []
def other_thread():
theirs.append(self._get(r, "other"))
done.set()
gate.wait(5)
t = threading.Thread(target=other_thread)
t.start()
done.wait(5)
r._discard_generators()
gate.set()
t.join()
assert mine.shut_down and not theirs[0].shut_down
assert self._get(r) is not mine # reloads fresh next time
class TestLastFrameChains:
def _chain(self, h, r, n=3):
clips = []
for i in range(n):
refs = frame_of(clips[-1]) if clips else []
clips.append(h.make(r, f"clip {i + 1}", references=refs))
return clips
def test_a_scene_runs_clip_by_clip_however_many_slots_there_are(self, h):
r = h.runner(max_concurrent_jobs=4)
c1, c2, c3 = self._chain(h, r)
r.enqueue_jobs([c1.id, c2.id, c3.id])
assert wait_for(lambda: statuses(r, c1, c2, c3) == ["running", "queued", "queued"])
time.sleep(0.05)
assert statuses(r, c1, c2, c3) == ["running", "queued", "queued"] # spare slots stay unused
h.finish(c1)
assert wait_for(lambda: statuses(r, c1, c2, c3) == ["completed", "running", "queued"])
h.finish(c2)
assert wait_for(lambda: statuses(r, c1, c2, c3) == ["completed", "completed", "running"])
def test_a_clip_does_not_block_unrelated_jobs_behind_it(self, h):
r = h.runner(max_concurrent_jobs=2)
c1, c2 = self._chain(h, r, 2)
other = h.make(r, "other")
r.enqueue_jobs([c1.id, c2.id, other.id])
assert wait_for(lambda: statuses(r, c1, c2, other) == ["running", "queued", "running"])
def test_a_failed_clip_fails_the_clips_after_it_and_says_why(self, h):
r = h.runner()
c1, c2, c3 = self._chain(h, r)
r.enqueue_jobs([c1.id, c2.id, c3.id])
h.finish(c1, h.module.JobStatus.FAILED)
assert wait_for(lambda: statuses(r, c1, c2, c3) == ["failed"] * 3)
assert "clip 1" in r.get_job(c2.id).error and "failed" in r.get_job(c2.id).error
assert "clip 2" in r.get_job(c3.id).error
assert h.ran == [c1.id]
def test_a_stopped_clip_fails_the_clips_after_it(self, h):
r = h.runner()
c1, c2 = self._chain(h, r, 2)
r.enqueue_jobs([c1.id, c2.id])
h.finish(c1, h.module.JobStatus.STOPPED)
assert wait_for(lambda: statuses(r, c1, c2) == ["stopped", "failed"])
assert "stopped" in r.get_job(c2.id).error
def test_deleting_a_queued_clip_fails_the_clips_after_it(self, h):
r = h.runner()
blocker = h.make(r, "blocker")
c1, c2 = self._chain(h, r, 2)
r.enqueue_jobs([blocker.id, c1.id, c2.id])
assert wait_for(lambda: statuses(r, blocker, c1, c2) == ["running", "queued", "queued"])
assert r.delete_job(c1.id)
assert wait_for(lambda: r.get_job(c2.id).status.value == "failed")
assert "deleted" in r.get_job(c2.id).error
def test_a_failed_clip_can_be_fixed_and_the_scene_requeued(self, h):
r = h.runner()
c1, c2 = self._chain(h, r, 2)
r.enqueue_jobs([c1.id, c2.id])
h.finish(c1, h.module.JobStatus.FAILED)
assert wait_for(lambda: statuses(r, c1, c2) == ["failed", "failed"])
h.outcomes.pop(c1.id)
h.gates[c1.id].clear()
r.enqueue_jobs([c1.id, c2.id])
assert wait_for(lambda: statuses(r, c1, c2) == ["running", "queued"])
h.finish(c1)
assert wait_for(lambda: statuses(r, c1, c2) == ["completed", "running"])
def test_the_stored_reference_stays_deferred(self, h):
r = h.runner()
c1, c2 = self._chain(h, r, 2)
r.enqueue_jobs([c1.id, c2.id])
h.finish(c1)
assert wait_for(lambda: r.get_job(c2.id).status.value == "running")
assert r.get_job(c2.id).references == frame_of(c1)
class TestStartingDirectly:
def test_refuses_while_the_frame_source_is_unfinished(self, h):
r = h.runner()
c1, c2 = h.make(r, "clip 1"), None
c2 = h.make(r, "clip 2", references=frame_of(c1))
with pytest.raises(ValueError, match="clip 1.*hasn't finished.*pending"):
r.start_job(c2.id)
assert r.get_job(c2.id).status.value == "pending"
assert h.ran == []
def test_starts_once_the_frame_source_completed(self, h):
r = h.runner()
c1 = h.make(r, "clip 1")
c1.status = h.module.JobStatus.COMPLETED
c2 = h.make(r, "clip 2", references=frame_of(c1))
r.start_job(c2.id)
assert r.get_job(c2.id).status.value == "running"
def test_refuses_when_the_frame_source_was_deleted(self, h):
r = h.runner()
c1 = h.make(r, "clip 1")
c2 = h.make(r, "clip 2", references=frame_of(c1))
r.delete_job(c1.id)
with pytest.raises(ValueError, match="no longer exists"):
r.start_job(c2.id)
def test_a_direct_start_takes_a_queue_slot(self, h):
r = h.runner()
running, queued = h.make(r, "running"), h.make(r, "queued")
r.start_job(running.id)
r.enqueue_job(queued.id)
time.sleep(0.05)
assert statuses(r, running, queued) == ["running", "queued"]
h.finish(running)
assert wait_for(lambda: r.get_job(queued.id).status.value == "running")
def test_a_queued_job_cannot_also_be_started(self, h):
r = h.runner()
blocker, waiting = h.make(r, "blocker"), h.make(r, "waiting")
r.enqueue_jobs([blocker.id, waiting.id])
with pytest.raises(ValueError, match="already queued"):
r.start_job(waiting.id)
class TestQueueManagement:
def test_dequeue_returns_a_job_to_pending_and_lets_others_go(self, h):
r = h.runner()
a, b, c = (h.make(r, n) for n in "abc")
r.enqueue_jobs([a.id, b.id, c.id])
assert wait_for(lambda: r.get_job(a.id).status.value == "running")
r.dequeue_job(b.id)
assert r.get_job(b.id).status.value == "pending"
assert r.get_job(b.id).queued_at is None
h.finish(a)
assert wait_for(lambda: statuses(r, a, b, c) == ["completed", "pending", "running"])
def test_stopping_a_queued_job_dequeues_it(self, h):
r = h.runner()
a, b = h.make(r, "a"), h.make(r, "b")
r.enqueue_jobs([a.id, b.id])
assert wait_for(lambda: r.get_job(a.id).status.value == "running")
assert r.stop_job(b.id).status.value == "pending"
def test_dequeue_of_a_job_not_queued_is_refused(self, h):
r = h.runner()
a = h.make(r, "a")
with pytest.raises(ValueError, match="not queued"):
r.dequeue_job(a.id)
with pytest.raises(ValueError, match="not found"):
r.dequeue_job("nope")
def test_enqueue_is_all_or_nothing(self, h):
r = h.runner()
a, done = h.make(r, "a"), h.make(r, "done")
done.status = h.module.JobStatus.COMPLETED
with pytest.raises(ValueError, match="already completed"):
r.enqueue_jobs([a.id, done.id])
assert r.get_job(a.id).status.value == "pending"
with pytest.raises(ValueError, match="not found"):
r.enqueue_jobs([a.id, "nope"])
assert h.ran == []
def test_a_job_cannot_be_queued_twice(self, h):
r = h.runner()
blocker, a = h.make(r, "blocker"), h.make(r, "a")
r.enqueue_jobs([blocker.id, a.id])
with pytest.raises(ValueError, match="already queued"):
r.enqueue_job(a.id)
with pytest.raises(ValueError, match="more than once"):
r.enqueue_jobs([a.id, a.id])
def test_loops_of_last_frames_are_refused(self, h):
r = h.runner()
a = h.make(r, "a")
b = h.make(r, "b", references=frame_of(a))
r.update_job_config(a.id, {"references": frame_of(b)})
with pytest.raises(ValueError, match="loop"):
r.enqueue_jobs([a.id, b.id])
assert statuses(r, a, b) == ["pending", "pending"]
def test_queued_jobs_cannot_be_edited(self, h):
r = h.runner()
blocker, a = h.make(r, "blocker"), h.make(r, "a")
r.enqueue_jobs([blocker.id, a.id])
with pytest.raises(ValueError, match="queued"):
r.update_job_config(a.id, {"prompt": "changed"})
def test_requeueing_clears_the_previous_error(self, h):
r = h.runner()
a = h.make(r, "a")
h.finish(a, h.module.JobStatus.FAILED)
r.enqueue_job(a.id)
assert wait_for(lambda: r.get_job(a.id).status.value == "failed")
h.outcomes.pop(a.id)
h.gates[a.id].clear()
r.enqueue_job(a.id)
assert wait_for(lambda: r.get_job(a.id).status.value == "running")
assert r.get_job(a.id).error is None
class TestRestart:
def test_queued_jobs_resume_in_order_after_a_restart(self, h, job_runner_module):
first = h.runner()
blocker = h.make(first, "blocker")
a, b = h.make(first, "a"), h.make(first, "b")
first.enqueue_jobs([blocker.id, b.id, a.id])
assert wait_for(lambda: h.ran == [blocker.id]) # actually started, not just handed to a worker
h.ran.clear()
second = h.runner() # same database
# The job that was running is failed, exactly as before queues existed...
assert second.get_job(blocker.id).status.value == "failed"
assert "restarted" in second.get_job(blocker.id).error
# ...and the queue carries on where it was.
assert wait_for(lambda: h.ran[:1] == [b.id])
assert second.get_job(a.id).status.value == "queued"
def test_edits_survive_a_restart(self, h):
first = h.runner()
job = h.make(first, "a", fps=24, negative_prompt="neg")
first.update_job_config(job.id, {"prompt": "edited", "references": frame_of(job), "fps": 12})
second = h.runner()
restored = second.get_job(job.id)
assert restored.prompt == "edited"
assert restored.references == frame_of(job)
assert restored.fps == 12
assert restored.negative_prompt == "neg"
def seg(brightness=0.0, contrast=1.0, saturation=1.0, end_seconds=None):
"""One razor-cut section dict, as trim_job()/the API expects."""
return {"end_seconds": end_seconds, "brightness": brightness, "contrast": contrast, "saturation": saturation}
class TestTrimming:
"""trim_job/restore_job_video against a real (small, synthetic) video file."""
FPS = 24
def _write_video(self, path, n=48):
import av
import numpy as np
Path(path).parent.mkdir(parents=True, exist_ok=True)
container = av.open(path, mode="w")
try:
stream = container.add_stream("h264", rate=self.FPS)
stream.width, stream.height, stream.pix_fmt = 64, 48, "yuv420p"
for i in range(n):
array = np.full((48, 64, 3), min(i * 5, 255), dtype=np.uint8)
for packet in stream.encode(av.VideoFrame.from_ndarray(array, format="rgb24")):
container.mux(packet)
for packet in stream.encode():
container.mux(packet)
finally:
container.close()
def _completed(self, h, r, n=48):
job = h.make(r, "clip")
r.start_job(job.id)
h.finish(job)
assert wait_for(lambda: r.get_job(job.id).status.value == "completed")
live = r.get_job(job.id)
path = str(h.tmp_path / "videos" / f"{job.id}.mp4")
self._write_video(path, n)
live.output_path = path
live.num_frames = n
return live
def test_trim_updates_output_path_and_frame_count(self, h):
r = h.runner()
job = self._completed(h, r) # 2s @ 24fps
original_path = job.output_path
trimmed = r.trim_job(job.id, start_seconds=0.5, end_seconds=1.5)
assert trimmed.output_path != original_path
assert trimmed.output_path.endswith("edited.mp4")
assert trimmed.num_frames == self.FPS # 1s kept
def test_grading_updates_output_path_without_changing_frame_count(self, h):
r = h.runner()
job = self._completed(h, r)
graded = r.trim_job(job.id, start_seconds=0.0, end_seconds=None, segments=[seg(brightness=40.0)])
assert graded.output_path.endswith("edited.mp4")
assert graded.num_frames == self.FPS * 2 # full length kept, only color changed
def test_grading_and_a_trim_requested_together_both_take_effect(self, h):
import av
r = h.runner()
job = self._completed(h, r)
edited = r.trim_job(job.id, start_seconds=0.5, end_seconds=1.5, segments=[seg(brightness=80.0)])
assert edited.num_frames == self.FPS # the range applied
with av.open(edited.output_path) as c:
frame = next(c.decode(c.streams.video[0]))
# Source frame at 0.5s (index 12) is level 60; +80 brightness should clearly lighten it.
assert int(frame.to_ndarray(format="rgb24")[0, 0, 0]) > 60 + 40
def test_two_sections_grade_independently(self, h):
import av
r = h.runner()
job = self._completed(h, r) # 48 frames @ 24fps = 2s
edited = r.trim_job(
job.id, start_seconds=0.0, end_seconds=None,
segments=[seg(brightness=80.0, end_seconds=1.0), seg()],
)
with av.open(edited.output_path) as c:
frames = list(c.decode(c.streams.video[0]))
first_half_level = int(frames[0].to_ndarray(format="rgb24")[0, 0, 0])
second_half_level = int(frames[-1].to_ndarray(format="rgb24")[0, 0, 0])
assert first_half_level > 0 + 40 # source level 0, graded section
assert second_half_level == min(47 * 5, 255) # source's actual last-frame level, untouched
def test_trim_persists_across_a_restart(self, h):
first = h.runner()
job = self._completed(h, first)
trimmed = first.trim_job(job.id, start_seconds=0.5, end_seconds=1.5)
second = h.runner()
restored = second.get_job(job.id)
assert restored.output_path == trimmed.output_path
assert restored.num_frames == trimmed.num_frames
def test_trim_records_the_applied_values_on_the_job(self, h):
r = h.runner()
job = self._completed(h, r)
edited = r.trim_job(
job.id, start_seconds=0.5, end_seconds=1.5,
segments=[seg(brightness=20.0, contrast=1.4, saturation=0.5)],
)
assert edited.edit_start_seconds == 0.5
assert edited.edit_end_seconds == 1.5
assert edited.edit_segments == [seg(brightness=20.0, contrast=1.4, saturation=0.5)]
def test_a_second_trim_overwrites_the_recorded_values_of_the_first(self, h):
r = h.runner()
job = self._completed(h, r)
r.trim_job(job.id, start_seconds=0.5, end_seconds=1.5, segments=[seg(brightness=20.0)])
again = r.trim_job(job.id, start_seconds=0.0, end_seconds=None, segments=[seg(contrast=1.4)])
assert again.edit_start_seconds == 0.0
assert again.edit_end_seconds is None
assert again.edit_segments == [seg(contrast=1.4)] # not carried over from the first trim
def test_the_recorded_edit_values_persist_across_a_restart(self, h):
first = h.runner()
job = self._completed(h, first)
first.trim_job(
job.id, start_seconds=0.5, end_seconds=1.5,
segments=[seg(brightness=20.0, contrast=1.4, saturation=0.5)],
)
second = h.runner()
restored = second.get_job(job.id)
assert restored.edit_start_seconds == 0.5
assert restored.edit_end_seconds == 1.5
assert restored.edit_segments == [seg(brightness=20.0, contrast=1.4, saturation=0.5)]
def test_a_multi_section_edit_persists_across_a_restart(self, h):
first = h.runner()
job = self._completed(h, first)
sections = [seg(brightness=20.0, end_seconds=0.5), seg(contrast=1.4)]
first.trim_job(job.id, start_seconds=0.0, end_seconds=None, segments=sections)
second = h.runner()
restored = second.get_job(job.id)
assert restored.edit_segments == sections
def test_restore_resets_the_recorded_edit_values_too(self, h):
r = h.runner()
job = self._completed(h, r)
r.trim_job(
job.id, start_seconds=0.5, end_seconds=1.5,
segments=[seg(brightness=20.0, contrast=1.4, saturation=0.5)],
)
restored = r.restore_job_video(job.id)
assert restored.edit_start_seconds == 0.0
assert restored.edit_end_seconds is None
assert restored.edit_segments == []
def test_a_never_edited_job_has_neutral_recorded_values(self, h):
r = h.runner()
job = self._completed(h, r)
assert job.edit_start_seconds == 0.0
assert job.edit_end_seconds is None
assert job.edit_segments == []
def test_trimming_again_is_relative_to_the_original_not_the_last_trim(self, h):
r = h.runner()
job = self._completed(h, r)
r.trim_job(job.id, start_seconds=0.5, end_seconds=1.5)
again = r.trim_job(job.id, start_seconds=0.0, end_seconds=2.0)
assert again.num_frames == self.FPS * 2
def test_restore_undoes_a_trim(self, h):
r = h.runner()
job = self._completed(h, r)
r.trim_job(job.id, start_seconds=0.5, end_seconds=1.5)
restored = r.restore_job_video(job.id)
# Points at the untouched backup, not literally the job's original path --
# trim_job() never overwrites or renames what H3 actually generated.
assert restored.output_path.endswith("original.mp4")
assert restored.num_frames == self.FPS * 2
def test_restore_refuses_a_job_never_edited(self, h):
r = h.runner()
job = self._completed(h, r)
with pytest.raises(FrameError, match="not been edited"):
r.restore_job_video(job.id)
def test_restore_undoes_a_grade_too(self, h):
import av
r = h.runner()
job = self._completed(h, r)
graded = r.trim_job(job.id, start_seconds=0.0, end_seconds=None, segments=[seg(brightness=80.0)])
with av.open(graded.output_path) as c:
graded_level = int(next(c.decode(c.streams.video[0])).to_ndarray(format="rgb24")[0, 0, 0])
assert graded_level > 60 # the grade visibly lightened the first frame (source level 0)
restored = r.restore_job_video(job.id)
with av.open(restored.output_path) as c:
restored_level = int(next(c.decode(c.streams.video[0])).to_ndarray(format="rgb24")[0, 0, 0])
assert restored_level == 0 # back to the source's actual first-frame level
def test_trim_refuses_an_unknown_job(self, h):
r = h.runner()
with pytest.raises(ValueError, match="not found"):
r.trim_job("nope", 0.0, 1.0)
def test_trim_refuses_a_pending_job(self, h):
r = h.runner()
job = h.make(r, "pending")
with pytest.raises(FrameError, match="No output"):
r.trim_job(job.id, 0.0, 1.0)
@@ -0,0 +1,209 @@
# SPDX-License-Identifier: Apache-2.0
"""A finished clip's trailing seconds are saved as a short video the next clip can continue from."""
from __future__ import annotations
import os
import time
from enum import Enum
from types import SimpleNamespace
import numpy as np
import pytest
from fastvideo_studio import frames
from fastvideo_studio.frames import FrameError, last_clip_for_job, save_last_clip
WIDTH, HEIGHT, FPS, FRAMES = 128, 96, 24, 48 # 2 seconds
class Status(Enum): # mirrors JobStatus without importing job_runner (which needs fastvideo)
PENDING = "pending"
COMPLETED = "completed"
FAILED = "failed"
def _write_video(path: str, n: int = FRAMES, fps: int = FPS) -> None:
"""A clip whose frames are numbered by brightness (frame i is roughly i*5 grey), so the tail is identifiable."""
import av
container = av.open(path, mode="w")
try:
stream = container.add_stream("h264", rate=fps)
stream.width, stream.height, stream.pix_fmt = WIDTH, HEIGHT, "yuv420p"
for i in range(n):
level = min(i * 5, 255)
array = np.full((HEIGHT, WIDTH, 3), level, dtype=np.uint8)
frame = av.VideoFrame.from_ndarray(array, format="rgb24")
for packet in stream.encode(frame):
container.mux(packet)
for packet in stream.encode():
container.mux(packet)
finally:
container.close()
def _read_levels(path: str) -> list[int]:
import av
with av.open(path) as container:
return [int(f.to_ndarray(format="rgb24")[0, 0, 0]) for f in container.decode(container.streams.video[0])]
def _job(job_id="job-1", status=Status.COMPLETED, output_path=None):
return SimpleNamespace(id=job_id, status=status, output_path=output_path)
@pytest.fixture
def video(tmp_path):
path = str(tmp_path / "clip.mp4")
_write_video(path)
return path
@pytest.fixture
def uploads(tmp_path):
return str(tmp_path / "uploads")
class TestSaveLastClip:
def test_keeps_roughly_the_requested_seconds(self, video, tmp_path):
out = str(tmp_path / "tail.mp4")
save_last_clip(video, out, seconds=1.0)
assert os.path.isfile(out)
levels = _read_levels(out)
assert FPS - 2 <= len(levels) <= FPS + 2 # ~1s at 24fps
def test_the_muxed_frame_rate_is_correct(self, video, tmp_path):
# Regression: every frame stamped with the same pts (e.g. all left at 0,
# or all set to `None` and left uncorrected) still decodes back the right
# *number* of frames, so a frame-count-only check doesn't catch it -- but
# it corrupts the container's average-frame-rate metadata, which is what
# H3's own reference pipeline reads to resample the clip. A wrong rate
# there silently collapses a short reference clip to as little as one
# usable frame.
out = str(tmp_path / "tail.mp4")
save_last_clip(video, out, seconds=1.0)
import av
with av.open(out) as container:
rate = float(container.streams.video[0].average_rate)
assert FPS - 1 <= rate <= FPS + 1
def test_frame_timestamps_are_distinct_and_increasing(self, video, tmp_path):
out = str(tmp_path / "tail.mp4")
save_last_clip(video, out, seconds=1.0)
import av
with av.open(out) as container:
times = [f.time for f in container.decode(container.streams.video[0])]
assert len(set(times)) == len(times)
assert times == sorted(times)
def test_keeps_the_end_of_the_source_not_the_start(self, video, tmp_path):
out = str(tmp_path / "tail.mp4")
save_last_clip(video, out, seconds=1.0)
levels = _read_levels(out)
full = _read_levels(video)
assert levels[-1] == full[-1] # the very last frame is preserved
assert levels[0] > full[0] # it does not start from the source's beginning
def test_asking_for_the_whole_clip_or_more_keeps_every_frame(self, video, tmp_path):
out = str(tmp_path / "tail.mp4")
save_last_clip(video, out, seconds=10.0)
assert len(_read_levels(out)) == FRAMES
def test_a_clip_with_no_frames(self, video, tmp_path, monkeypatch):
# A container that opens fine but yields zero frames (e.g. a truncated encode).
import av
class EmptyContainer:
class _Streams:
video = [SimpleNamespace(average_rate=24, guessed_rate=24)]
streams = _Streams()
def decode(self, *_a):
return iter(())
def __enter__(self):
return self
def __exit__(self, *_a):
return False
monkeypatch.setattr(av, "open", lambda *_a, **_k: EmptyContainer())
with pytest.raises(ValueError, match="no frames"):
save_last_clip(video, str(tmp_path / "out.mp4"))
def test_a_nonpositive_duration_is_refused(self, video, tmp_path):
with pytest.raises(ValueError, match="positive"):
save_last_clip(video, str(tmp_path / "out.mp4"), seconds=0)
def test_writes_no_temp_file_leftovers(self, video, tmp_path):
out = str(tmp_path / "tail.mp4")
save_last_clip(video, out, seconds=1.0)
assert os.listdir(tmp_path) == ["clip.mp4", "tail.mp4"]
class TestLastClipForJob:
def test_extracts_and_returns_a_path(self, video, uploads):
job = _job(output_path=video)
path = last_clip_for_job(job, uploads)
assert os.path.isfile(path)
assert path.endswith("last_clip_job-1.mp4")
assert "last_clips" in path
def test_reuses_an_existing_extraction(self, video, uploads, monkeypatch):
job = _job(output_path=video)
first = last_clip_for_job(job, uploads)
mtime = os.path.getmtime(first)
calls = []
monkeypatch.setattr(frames, "save_last_clip", lambda *a, **k: calls.append(a))
second = last_clip_for_job(job, uploads)
assert second == first
assert os.path.getmtime(second) == mtime
assert calls == []
def test_re_extracts_when_the_video_changed_since(self, video, uploads, monkeypatch):
job = _job(output_path=video)
last_clip_for_job(job, uploads)
_write_video(video, n=FRAMES + 24) # a longer re-render
# Force the source to read as newer, regardless of filesystem mtime resolution.
monkeypatch.setattr(os.path, "getmtime", lambda p: (time.time() + 1) if p == video else os.stat(p).st_mtime)
calls = []
real_save = frames.save_last_clip
monkeypatch.setattr(frames, "save_last_clip", lambda *a, **k: (calls.append(a), real_save(*a, **k))[1])
last_clip_for_job(job, uploads)
assert len(calls) == 1
@pytest.mark.parametrize(
("job", "match"),
[
(_job(status=Status.PENDING), "No output"),
(_job(status=Status.COMPLETED, output_path=None), "No output"),
(_job(status=Status.COMPLETED, output_path="/nope.mp4"), "not found"),
],
)
def test_refusals(self, job, match, uploads):
with pytest.raises(FrameError, match=match):
last_clip_for_job(job, uploads)
def test_refuses_a_non_video_output(self, uploads, tmp_path):
img = tmp_path / "out.png"
img.write_bytes(b"x")
job = _job(output_path=str(img))
with pytest.raises(FrameError, match="not a video"):
last_clip_for_job(job, uploads)
def test_refuses_without_an_upload_dir(self, video):
job = _job(output_path=video)
with pytest.raises(FrameError, match="Upload directory"):
last_clip_for_job(job, "")
def test_a_decode_failure_is_reported_as_a_500(self, uploads, tmp_path):
bad = tmp_path / "bad.mp4"
bad.write_bytes(b"not a video")
job = _job(output_path=str(bad))
with pytest.raises(FrameError, match="Could not read the end of the clip") as e:
last_clip_for_job(job, uploads)
assert e.value.status_code == 500
@@ -0,0 +1,155 @@
# SPDX-License-Identifier: Apache-2.0
"""A finished clip's final frame is saved as an image the next clip can use as a reference."""
from __future__ import annotations
import os
from enum import Enum
from types import SimpleNamespace
import numpy as np
import pytest
from PIL import Image
from fastvideo_studio import frames
from fastvideo_studio.frames import FrameError, last_frame_for_job
WIDTH, HEIGHT, FRAMES = 128, 96, 12
class Status(Enum): # mirrors JobStatus without importing job_runner (which needs fastvideo)
PENDING = "pending"
RUNNING = "running"
COMPLETED = "completed"
FAILED = "failed"
STOPPED = "stopped"
def _write_video(path: str, *, last_rgb=(0, 0, 255), other_rgb=(255, 0, 0)) -> None:
"""A clip whose frames are all `other_rgb` except the last, which is `last_rgb`."""
import av
container = av.open(path, mode="w")
try:
stream = container.add_stream("h264", rate=24)
stream.width, stream.height, stream.pix_fmt = WIDTH, HEIGHT, "yuv420p"
for i in range(FRAMES):
rgb = last_rgb if i == FRAMES - 1 else other_rgb
array = np.full((HEIGHT, WIDTH, 3), rgb, dtype=np.uint8)
frame = av.VideoFrame.from_ndarray(array, format="rgb24")
for packet in stream.encode(frame):
container.mux(packet)
for packet in stream.encode():
container.mux(packet)
finally:
container.close()
def _job(job_id="job-1", status=Status.COMPLETED, output_path=None):
return SimpleNamespace(id=job_id, status=status, output_path=output_path)
@pytest.fixture
def video(tmp_path):
path = str(tmp_path / "clip.mp4")
_write_video(path)
return path
@pytest.fixture
def uploads(tmp_path):
return str(tmp_path / "uploads")
def test_saves_the_final_frame_not_the_first(video, uploads, tmp_path):
path = last_frame_for_job(_job(output_path=video), uploads)
assert path == str(tmp_path / "uploads" / "last_frames" / "last_frame_job-1.png")
assert os.path.isabs(path)
image = Image.open(path).convert("RGB")
assert image.size == (WIDTH, HEIGHT)
r, _g, b = image.getpixel((WIDTH // 2, HEIGHT // 2))
# the last frame is blue and every other frame is red -- a wrong frame would be red
assert b > 150 and r < 100, (r, b)
def test_accepts_a_plain_string_status(video, uploads):
assert last_frame_for_job(_job(status="completed", output_path=video), uploads).endswith(".png")
def test_extraction_is_reused_until_the_video_changes(video, uploads, monkeypatch):
calls = []
real = frames.save_last_frame
monkeypatch.setattr(frames, "save_last_frame", lambda v, o: (calls.append(o), real(v, o))[1])
first = last_frame_for_job(_job(output_path=video), uploads)
second = last_frame_for_job(_job(output_path=video), uploads)
assert first == second
assert len(calls) == 1
# the clip is regenerated -> the saved frame is stale and gets rebuilt
later = os.path.getmtime(first) + 10
os.utime(video, (later, later))
last_frame_for_job(_job(output_path=video), uploads)
assert len(calls) == 2
def test_each_job_gets_its_own_file(video, uploads):
a = last_frame_for_job(_job("a", output_path=video), uploads)
b = last_frame_for_job(_job("b", output_path=video), uploads)
assert a != b and os.path.isfile(a) and os.path.isfile(b)
def test_leaves_no_temp_file_behind(video, uploads):
last_frame_for_job(_job(output_path=video), uploads)
assert os.listdir(os.path.join(uploads, "last_frames")) == ["last_frame_job-1.png"]
@pytest.mark.parametrize("status", [Status.PENDING, Status.RUNNING, Status.FAILED, Status.STOPPED])
def test_a_job_that_has_not_finished_is_404(video, uploads, status):
with pytest.raises(FrameError) as e:
last_frame_for_job(_job(status=status, output_path=video), uploads)
assert e.value.status_code == 404
assert "No output" in e.value.detail
def test_a_completed_job_with_no_output_path_is_404(uploads):
with pytest.raises(FrameError) as e:
last_frame_for_job(_job(output_path=None), uploads)
assert e.value.status_code == 404
def test_missing_output_file_is_404(uploads, tmp_path):
with pytest.raises(FrameError) as e:
last_frame_for_job(_job(output_path=str(tmp_path / "gone.mp4")), uploads)
assert (e.value.status_code, "not found on disk" in e.value.detail) == (404, True)
def test_an_image_output_is_rejected(uploads, tmp_path):
png = tmp_path / "out.png"
Image.new("RGB", (8, 8)).save(png)
with pytest.raises(FrameError) as e:
last_frame_for_job(_job(output_path=str(png)), uploads)
assert e.value.status_code == 400
def test_an_unreadable_video_is_a_500_with_a_reason(uploads, tmp_path):
bad = tmp_path / "bad.mp4"
bad.write_bytes(b"not a video")
with pytest.raises(FrameError) as e:
last_frame_for_job(_job(output_path=str(bad)), uploads)
assert e.value.status_code == 500
assert "Could not read the last frame" in e.value.detail
def test_a_failed_extraction_does_not_leave_a_partial_file(uploads, tmp_path):
bad = tmp_path / "bad.mp4"
bad.write_bytes(b"not a video")
with pytest.raises(FrameError):
last_frame_for_job(_job(output_path=str(bad)), uploads)
assert os.listdir(os.path.join(uploads, "last_frames")) == []
def test_needs_an_upload_directory(video):
with pytest.raises(FrameError) as e:
last_frame_for_job(_job(output_path=video), "")
assert e.value.status_code == 503
+251
View File
@@ -0,0 +1,251 @@
# SPDX-License-Identifier: Apache-2.0
"""A scene's clips are joined, in order, into one video without re-encoding."""
from __future__ import annotations
import os
from enum import Enum
from types import SimpleNamespace
import numpy as np
import pytest
from fastvideo_studio import merge
from fastvideo_studio.merge import MergeError, find_ffmpeg, merge_clips, merge_jobs, merged_path, probe_clip
pytestmark = pytest.mark.skipif(find_ffmpeg() is None, reason="needs an ffmpeg binary")
FPS = 24
COLORS = {"red": (255, 0, 0), "green": (0, 255, 0), "blue": (0, 0, 255)}
class Status(Enum): # mirrors JobStatus without importing job_runner (which needs fastvideo)
PENDING = "pending"
COMPLETED = "completed"
FAILED = "failed"
def write_clip(path, *, frames=24, color="red", size=(128, 96), fps=FPS, audio=True, audio_extra=0.042):
"""A clip of one flat colour. Like real H3 output, its audio runs a little past the video."""
import av
container = av.open(path, mode="w")
try:
video = container.add_stream("h264", rate=fps)
video.width, video.height, video.pix_fmt = size[0], size[1], "yuv420p"
sound = None
if audio:
sound = container.add_stream("aac", rate=32000)
sound.layout = "stereo"
for _ in range(frames):
array = np.full((size[1], size[0], 3), COLORS[color], dtype=np.uint8)
for packet in video.encode(av.VideoFrame.from_ndarray(array, format="rgb24")):
container.mux(packet)
for packet in video.encode():
container.mux(packet)
if sound is not None:
total = int((frames / fps + audio_extra) * 32000)
tone = (np.sin(np.arange(total) * 2 * np.pi * 440 / 32000) * 0.2).astype(np.float32)
pts = 0
for start in range(0, total, 1024):
chunk = tone[start:start + 1024]
chunk = np.pad(chunk, (0, 1024 - len(chunk)))
frame = av.AudioFrame.from_ndarray(np.stack([chunk, chunk]), format="fltp", layout="stereo")
frame.sample_rate, frame.pts = 32000, pts
pts += 1024
for packet in sound.encode(frame):
container.mux(packet)
for packet in sound.encode():
container.mux(packet)
finally:
container.close()
def read_back(path):
"""(frame colours in order, audio seconds, video seconds) of a finished video."""
import av
with av.open(path) as c:
v = c.streams.video[0]
colours = []
for frame in c.decode(v):
r, g, b = frame.to_ndarray(format="rgb24")[48, 64]
colours.append("red" if r > 150 else "green" if g > 150 else "blue")
audio = next((s for s in c.streams if s.type == "audio"), None)
a_secs = float(audio.duration * audio.time_base) if audio and audio.duration else None
return colours, a_secs, float(v.duration * v.time_base)
def job(path, name="", status=Status.COMPLETED, job_id="j"):
return SimpleNamespace(id=job_id, name=name, status=status, output_path=path)
@pytest.fixture
def clips(tmp_path):
paths = []
for i, (color, frames) in enumerate([("red", 24), ("green", 36), ("blue", 12)]):
p = str(tmp_path / f"clip{i}.mp4")
write_clip(p, frames=frames, color=color)
paths.append(p)
return paths
class TestMergeClips:
def test_joins_clips_in_the_order_given(self, clips, tmp_path):
out = str(tmp_path / "out.mp4")
merge_clips(clips, out)
colours, _, _ = read_back(out)
assert colours == ["red"] * 24 + ["green"] * 36 + ["blue"] * 12
def test_the_order_is_the_callers_not_the_filenames(self, clips, tmp_path):
out = str(tmp_path / "out.mp4")
merge_clips([clips[2], clips[0], clips[1]], out)
colours, _, _ = read_back(out)
assert colours == ["blue"] * 12 + ["red"] * 24 + ["green"] * 36
def test_keeps_every_frame_and_the_full_length(self, clips, tmp_path):
out = str(tmp_path / "out.mp4")
seconds = merge_clips(clips, out)
colours, _, video_secs = read_back(out)
assert len(colours) == 72
assert seconds == pytest.approx(72 / FPS, abs=0.02)
assert video_secs == pytest.approx(72 / FPS, abs=0.05)
def test_audio_stays_in_step_with_the_picture(self, clips, tmp_path):
# Each clip's audio runs ~40 ms past its video; joined naively that adds up to a gap or drift.
out = str(tmp_path / "out.mp4")
merge_clips(clips, out)
_, audio_secs, video_secs = read_back(out)
assert audio_secs is not None
assert abs(audio_secs - video_secs) < 0.08 # not 3 x 0.042 and growing
def test_a_single_clip_is_fine(self, clips, tmp_path):
out = str(tmp_path / "out.mp4")
merge_clips(clips[:1], out)
assert len(read_back(out)[0]) == 24
def test_clips_without_audio_join_too(self, tmp_path):
paths = []
for i, color in enumerate(("red", "green")):
p = str(tmp_path / f"s{i}.mp4")
write_clip(p, frames=12, color=color, audio=False)
paths.append(p)
out = str(tmp_path / "out.mp4")
merge_clips(paths, out)
colours, audio_secs, _ = read_back(out)
assert colours == ["red"] * 12 + ["green"] * 12 and audio_secs is None
def test_creates_the_output_folder_and_leaves_no_temp_files(self, clips, tmp_path):
out = str(tmp_path / "new" / "dir" / "out.mp4")
merge_clips(clips, out)
assert os.listdir(os.path.dirname(out)) == ["out.mp4"]
def test_paths_with_quotes_and_spaces(self, tmp_path):
paths = []
for i, color in enumerate(("red", "blue")):
p = str(tmp_path / f"it's clip {i}.mp4")
write_clip(p, frames=6, color=color)
paths.append(p)
out = str(tmp_path / "out.mp4")
merge_clips(paths, out)
assert read_back(out)[0] == ["red"] * 6 + ["blue"] * 6
class TestRefusals:
def test_clips_of_different_sizes(self, clips, tmp_path):
odd = str(tmp_path / "odd.mp4")
write_clip(odd, size=(160, 96))
with pytest.raises(MergeError, match=r"'clip 3' \(160x96.*doesn't match 'clip 1' \(128x96") as e:
merge_clips([clips[0], clips[1], odd], str(tmp_path / "o.mp4"), labels=["'clip 1'", "'clip 2'", "'clip 3'"])
assert e.value.status_code == 409
assert not (tmp_path / "o.mp4").exists()
def test_clips_of_different_frame_rates(self, clips, tmp_path):
odd = str(tmp_path / "odd.mp4")
write_clip(odd, fps=30)
with pytest.raises(MergeError, match="doesn't match"):
merge_clips([clips[0], odd], str(tmp_path / "o.mp4"))
def test_one_clip_with_audio_and_one_without(self, clips, tmp_path):
silent = str(tmp_path / "silent.mp4")
write_clip(silent, audio=False)
with pytest.raises(MergeError, match="no audio"):
merge_clips([clips[0], silent], str(tmp_path / "o.mp4"))
def test_nothing_to_merge(self, tmp_path):
with pytest.raises(MergeError) as e:
merge_clips([], str(tmp_path / "o.mp4"))
assert e.value.status_code == 400
def test_a_file_that_is_not_a_video(self, tmp_path):
bad = tmp_path / "bad.mp4"
bad.write_bytes(b"not a video")
with pytest.raises(MergeError, match="Could not read bad.mp4"):
merge_clips([str(bad)], str(tmp_path / "o.mp4"))
def test_no_ffmpeg(self, clips, tmp_path, monkeypatch):
monkeypatch.setattr(merge, "find_ffmpeg", lambda: None)
with pytest.raises(MergeError) as e:
merge_clips(clips, str(tmp_path / "o.mp4"))
assert e.value.status_code == 503 and "ffmpeg" in e.value.detail
class TestMergeJobs:
def test_merges_the_jobs_videos_into_a_merged_folder(self, clips, tmp_path):
jobs = [job(p, name=f"clip {i + 1}", job_id=f"j{i}") for i, p in enumerate(clips)]
result = merge_jobs(jobs, str(tmp_path / "out"), name="Wolf Lunch")
assert result.clips == 3 and result.seconds == pytest.approx(3.0, abs=0.02)
assert result.filename.startswith("Wolf-Lunch-") and result.filename.endswith(".mp4")
assert result.path == str(tmp_path / "out" / "merged" / result.filename)
assert len(read_back(result.path)[0]) == 72
def test_a_name_cannot_escape_the_folder(self, clips, tmp_path):
result = merge_jobs([job(clips[0])], str(tmp_path / "out"), name="../../etc/passwd")
assert os.path.dirname(result.path) == str(tmp_path / "out" / "merged")
assert "/" not in result.filename
def test_an_empty_name_still_gets_a_filename(self, clips, tmp_path):
assert merge_jobs([job(clips[0])], str(tmp_path / "out"), name=" ").filename.startswith("scene-")
def test_refuses_a_clip_that_has_not_finished_and_names_it(self, clips, tmp_path):
jobs = [job(clips[0], name="clip 1"), job(None, name="clip 2", status=Status.PENDING)]
with pytest.raises(MergeError, match=r"'clip 2' hasn't finished \(pending\)") as e:
merge_jobs(jobs, str(tmp_path / "out"))
assert e.value.status_code == 409
assert not (tmp_path / "out").exists() # nothing partial
def test_refuses_a_video_missing_from_disk(self, clips, tmp_path):
jobs = [job(clips[0], name="a"), job(str(tmp_path / "gone.mp4"), name="b")]
with pytest.raises(MergeError, match="'b'.*missing from disk"):
merge_jobs(jobs, str(tmp_path / "out"))
def test_refuses_an_output_that_is_not_a_video(self, tmp_path):
img = tmp_path / "a.png"
img.write_bytes(b"x")
with pytest.raises(MergeError, match="isn't a video"):
merge_jobs([job(str(img), name="a")], str(tmp_path / "out"))
def test_refuses_no_jobs(self, tmp_path):
with pytest.raises(MergeError) as e:
merge_jobs([], str(tmp_path))
assert e.value.status_code == 400
class TestMergedPath:
def test_finds_a_merged_video(self, clips, tmp_path):
result = merge_jobs([job(clips[0])], str(tmp_path))
assert merged_path(str(tmp_path), result.filename) == result.path
@pytest.mark.parametrize("name", ["../secret.mp4", "a/b.mp4", "..", "x.txt", "", "nope.mp4", "a b.mp4"])
def test_refuses_anything_else(self, tmp_path, name):
(tmp_path / "merged").mkdir()
(tmp_path / "secret.mp4").write_bytes(b"x")
with pytest.raises(MergeError) as e:
merged_path(str(tmp_path), name)
assert e.value.status_code == 404
def test_probe_reads_real_clip_settings(clips):
info = probe_clip(clips[1])
assert (info.width, info.height, info.video_codec, info.pix_fmt) == (128, 96, "h264", "yuv420p")
assert info.fps == pytest.approx(24) and info.video_seconds == pytest.approx(1.5, abs=0.03)
assert info.audio == ("aac", 32000, 2)
@@ -0,0 +1,428 @@
# SPDX-License-Identifier: Apache-2.0
"""The mock API's job edit and last-frame routes behave like the real server's."""
from __future__ import annotations
import pytest
from fastapi.testclient import TestClient
from fastvideo_studio import mock_server
@pytest.fixture(autouse=True)
def clean_state():
mock_server._jobs.clear()
yield
mock_server._jobs.clear()
@pytest.fixture
def client():
return TestClient(mock_server.app)
def _create(client, **over):
body = {"model_id": "MiniMaxAI/MiniMax-H3", "prompt": "p", "workload_type": "i2v", **over}
res = client.post("/api/jobs", json=body)
assert res.status_code == 201
return res.json()
def test_edits_a_pending_jobs_prompt_and_references(client):
job = _create(client, name="clip-2", references=[{"source": "/a.png", "media_type": "image"}])
refs = job["references"] + [{"source": "/b.png", "media_type": "image"}]
res = client.patch(f"/api/jobs/{job['id']}", json={"prompt": "new prompt", "references": refs})
assert res.status_code == 200
body = res.json()
assert body["prompt"] == "new prompt"
assert [r["source"] for r in body["references"]] == ["/a.png", "/b.png"]
assert body["name"] == "clip-2" # untouched fields stay
assert client.get(f"/api/jobs/{job['id']}").json()["prompt"] == "new prompt"
def test_a_started_job_cannot_be_edited(client):
job = _create(client)
mock_server._jobs[job["id"]]["status"] = "completed"
res = client.patch(f"/api/jobs/{job['id']}", json={"prompt": "x"})
assert res.status_code == 400
assert "only pending, failed, stopped jobs can be edited" in res.json()["detail"]
@pytest.mark.parametrize("status", ["failed", "stopped"])
def test_failed_and_stopped_jobs_can_be_edited(client, status):
job = _create(client)
mock_server._jobs[job["id"]]["status"] = status
assert client.patch(f"/api/jobs/{job['id']}", json={"prompt": "again"}).status_code == 200
def test_unknown_fields_are_rejected(client):
job = _create(client)
res = client.patch(f"/api/jobs/{job['id']}", json={"status": "completed", "nonsense": 1})
assert res.status_code == 400
assert "Not editable: nonsense, status" in res.json()["detail"]
def test_editing_an_unknown_job_is_404(client):
assert client.patch("/api/jobs/nope", json={"prompt": "x"}).status_code == 404
def test_last_frame_of_a_completed_job(client):
job = _create(client)
mock_server._jobs[job["id"]].update(status="completed", output_path="/out/a.mp4")
res = client.post(f"/api/jobs/{job['id']}/last-frame")
assert res.status_code == 200
assert res.json() == {"path": f"/mock/last_frames/last_frame_{job['id']}.png", "media_type": "image"}
@pytest.mark.parametrize("status", ["pending", "failed", "stopped"])
def test_last_frame_of_an_unfinished_job_is_404(client, status):
job = _create(client)
mock_server._jobs[job["id"]]["status"] = status
res = client.post(f"/api/jobs/{job['id']}/last-frame")
assert res.status_code == 404
assert "No output" in res.json()["detail"]
def test_last_frame_of_an_unknown_job_is_404(client):
assert client.post("/api/jobs/nope/last-frame").status_code == 404
# --- queue -------------------------------------------------------------------
def _frame_of(job):
return [{"source": f"job-last-frame:{job['id']}", "media_type": "image"}]
def _statuses(client, *jobs):
return [client.get(f"/api/jobs/{j['id']}").json()["status"] for j in jobs]
def _age(job, seconds):
"""Pretend `seconds` have passed since `job` started."""
record = mock_server._jobs[job["id"]]
record["started_at"] -= seconds
def test_queued_jobs_run_one_at_a_time_in_order(client):
a, b, c = (_create(client, name=n) for n in "abc")
res = client.post("/api/jobs/queue", json={"job_ids": [b["id"], c["id"], a["id"]]})
assert res.status_code == 200
assert [j["name"] for j in res.json()] == ["b", "c", "a"]
assert _statuses(client, a, b, c) == ["queued", "running", "queued"]
_age(b, 10)
assert _statuses(client, a, b, c) == ["queued", "completed", "running"]
def test_a_long_idle_queue_is_all_done_on_the_next_poll(client):
clips = [_create(client, name=f"c{i}") for i in range(3)]
client.post("/api/jobs/queue", json={"job_ids": [c["id"] for c in clips]})
for record in mock_server._jobs.values(): # everything happened a minute ago
if record.get("queued_at"):
record["queued_at"] -= 60
if record.get("started_at"):
record["started_at"] -= 60
assert _statuses(client, *clips) == ["completed"] * 3
starts = [mock_server._jobs[c["id"]]["started_at"] for c in clips]
assert starts == sorted(starts) # each took over when the previous finished
def test_a_clip_waits_for_the_clip_whose_last_frame_it_uses(client):
one = _create(client, name="one")
two = _create(client, name="two", references=_frame_of(one))
client.post("/api/jobs/queue", json={"job_ids": [one["id"], two["id"]]})
assert _statuses(client, one, two) == ["running", "queued"]
_age(one, 10)
assert _statuses(client, one, two) == ["completed", "running"]
def test_failure_of_a_frame_source_fails_the_clips_after_it(client):
one = _create(client, name="Clip 1")
two = _create(client, name="Clip 2", references=_frame_of(one))
three = _create(client, name="Clip 3", references=_frame_of(two))
mock_server._jobs[one["id"]]["status"] = "failed"
client.post("/api/jobs/queue", json={"job_ids": [two["id"], three["id"]]})
assert _statuses(client, two, three) == ["failed", "failed"]
error = client.get(f"/api/jobs/{two['id']}").json()["error"]
assert "Clip 1" in error and "failed" in error
def test_deleting_a_frame_source_fails_the_clips_waiting_on_it(client):
blocker = _create(client)
one = _create(client, name="one")
two = _create(client, name="two", references=_frame_of(one))
client.post("/api/jobs/queue", json={"job_ids": [blocker["id"], one["id"], two["id"]]})
assert client.delete(f"/api/jobs/{one['id']}").status_code == 200
assert _statuses(client, two) == ["failed"]
assert "deleted" in client.get(f"/api/jobs/{two['id']}").json()["error"]
def test_dequeue_and_stop_return_a_queued_job_to_pending(client):
blocker, a, b = _create(client), _create(client), _create(client)
client.post("/api/jobs/queue", json={"job_ids": [blocker["id"], a["id"], b["id"]]})
assert client.post(f"/api/jobs/{a['id']}/dequeue").json()["status"] == "pending"
assert client.post(f"/api/jobs/{b['id']}/stop").json()["status"] == "pending"
assert client.post(f"/api/jobs/{a['id']}/dequeue").status_code == 409
assert client.post("/api/jobs/nope/dequeue").status_code == 404
def test_queueing_is_all_or_nothing_and_refuses_started_jobs(client):
a, done = _create(client), _create(client)
mock_server._jobs[done["id"]]["status"] = "completed"
res = client.post("/api/jobs/queue", json={"job_ids": [a["id"], done["id"]]})
assert res.status_code == 409 and "already completed" in res.json()["detail"]
assert _statuses(client, a) == ["pending"]
assert client.post("/api/jobs/queue", json={"job_ids": [a["id"], "nope"]}).status_code == 404
assert client.post("/api/jobs/queue", json={"job_ids": [a["id"], a["id"]]}).status_code == 409
assert client.post(f"/api/jobs/{a['id']}/queue").status_code == 200
assert client.post(f"/api/jobs/{a['id']}/queue").status_code == 409 # running now
def test_starting_a_clip_before_its_frame_source_finished_is_refused(client):
one = _create(client, name="Clip 1")
two = _create(client, references=_frame_of(one))
res = client.post(f"/api/jobs/{two['id']}/start")
assert res.status_code == 409
assert "Clip 1" in res.json()["detail"] and "hasn't finished" in res.json()["detail"]
mock_server._jobs[one["id"]]["status"] = "completed"
assert client.post(f"/api/jobs/{two['id']}/start").status_code == 200
def test_queued_jobs_cannot_be_edited(client):
blocker, a = _create(client), _create(client)
client.post("/api/jobs/queue", json={"job_ids": [blocker["id"], a["id"]]})
assert client.patch(f"/api/jobs/{a['id']}", json={"prompt": "x"}).status_code == 400
# --- merging a scene -----------------------------------------------------------
def _completed(client, name):
job = _create(client, name=name)
record = mock_server._jobs[job["id"]]
record.update(status="completed", output_path=f"/mock/outputs/{job['id']}/output.mp4", finished_at=1.0)
return job
def test_merges_finished_clips_into_one_downloadable_video(client):
clips = [_completed(client, f"wolf-{i}") for i in range(3)]
res = client.post("/api/scenes/merge", json={"job_ids": [c["id"] for c in clips], "name": "wolf"})
assert res.status_code == 200
body = res.json()
assert body["clips"] == 3 and body["filename"].startswith("wolf-") and body["seconds"] == pytest.approx(3.0, abs=0.1)
video = client.get(body["url"])
assert video.status_code == 200 and video.headers["content-type"] == "video/mp4"
assert len(video.content) > 1000
def test_refuses_to_merge_a_scene_with_an_unfinished_clip(client):
done, pending = _completed(client, "a"), _create(client, name="b")
res = client.post("/api/scenes/merge", json={"job_ids": [done["id"], pending["id"]]})
assert res.status_code == 409 and "'b' hasn't finished" in res.json()["detail"]
def test_merge_of_an_unknown_job_and_of_nothing(client):
assert client.post("/api/scenes/merge", json={"job_ids": ["nope"]}).status_code == 404
assert client.post("/api/scenes/merge", json={"job_ids": []}).status_code == 400
@pytest.mark.parametrize("filename", ["nope.mp4", "..%2Fetc%2Fpasswd", "x.txt"])
def test_an_unknown_merged_file_is_not_found(client, filename):
assert client.get(f"/api/merged/{filename}").status_code == 404
# --- trimming ------------------------------------------------------------------
def seg(brightness=0.0, contrast=1.0, saturation=1.0, end_seconds=None):
"""One razor-cut section dict, as the trim API expects."""
return {"end_seconds": end_seconds, "brightness": brightness, "contrast": contrast, "saturation": saturation}
def test_trims_a_completed_jobs_video_by_resizing_its_frame_count(client):
job = _completed(client, "clip")
mock_server._jobs[job["id"]]["num_frames"] = 240 # 10s @ 24fps
mock_server._jobs[job["id"]]["fps"] = 24
res = client.post(f"/api/jobs/{job['id']}/trim", json={"start_seconds": 1.0, "end_seconds": 3.0})
assert res.status_code == 200
assert res.json()["num_frames"] == 48 # 2s @ 24fps
def test_an_open_end_keeps_to_the_original_end(client):
job = _completed(client, "clip")
mock_server._jobs[job["id"]]["num_frames"] = 240
mock_server._jobs[job["id"]]["fps"] = 24
res = client.post(f"/api/jobs/{job['id']}/trim", json={"start_seconds": 8.0})
assert res.json()["num_frames"] == 48 # 10s source, 8-10s kept = 2s remaining
def test_trimming_again_is_relative_to_the_original_length_not_the_last_trim(client):
job = _completed(client, "clip")
mock_server._jobs[job["id"]]["num_frames"] = 240
mock_server._jobs[job["id"]]["fps"] = 24
client.post(f"/api/jobs/{job['id']}/trim", json={"start_seconds": 1.0, "end_seconds": 3.0})
res = client.post(f"/api/jobs/{job['id']}/trim", json={"start_seconds": 0.0, "end_seconds": 10.0})
assert res.json()["num_frames"] == 240
def test_restore_undoes_a_trim(client):
job = _completed(client, "clip")
mock_server._jobs[job["id"]]["num_frames"] = 240
mock_server._jobs[job["id"]]["fps"] = 24
client.post(f"/api/jobs/{job['id']}/trim", json={"start_seconds": 1.0, "end_seconds": 3.0})
res = client.post(f"/api/jobs/{job['id']}/restore-video")
assert res.status_code == 200
assert res.json()["num_frames"] == 240
assert client.get(f"/api/jobs/{job['id']}").json()["num_frames"] == 240
def test_restore_refuses_a_job_that_was_never_edited(client):
job = _completed(client, "clip")
res = client.post(f"/api/jobs/{job['id']}/restore-video")
assert res.status_code == 404 and "not been edited" in res.json()["detail"]
def test_trim_refuses_an_invalid_range(client):
job = _completed(client, "clip")
mock_server._jobs[job["id"]]["num_frames"] = 240
mock_server._jobs[job["id"]]["fps"] = 24
assert client.post(f"/api/jobs/{job['id']}/trim", json={"start_seconds": -1.0}).status_code == 400
assert client.post(f"/api/jobs/{job['id']}/trim", json={"start_seconds": 5.0, "end_seconds": 5.0}).status_code == 400
def test_trim_accepts_color_parameters(client):
job = _completed(client, "clip")
mock_server._jobs[job["id"]]["num_frames"] = 240
mock_server._jobs[job["id"]]["fps"] = 24
res = client.post(
f"/api/jobs/{job['id']}/trim",
json={"start_seconds": 0.0, "end_seconds": 5.0, "segments": [seg(brightness=20.0, contrast=1.5, saturation=0.5)]},
)
assert res.status_code == 200
def test_a_never_edited_job_reports_neutral_edit_values(client):
job = _completed(client, "clip")
body = client.get(f"/api/jobs/{job['id']}").json()
assert body["edit_start_seconds"] == 0.0
assert body["edit_end_seconds"] is None
assert body["edit_segments"] == []
def test_trim_response_records_the_applied_values(client):
job = _completed(client, "clip")
mock_server._jobs[job["id"]]["num_frames"] = 240
mock_server._jobs[job["id"]]["fps"] = 24
sections = [seg(brightness=20.0, contrast=1.5, saturation=0.5)]
res = client.post(
f"/api/jobs/{job['id']}/trim",
json={"start_seconds": 1.0, "end_seconds": 3.0, "segments": sections},
)
body = res.json()
assert body["edit_start_seconds"] == 1.0
assert body["edit_end_seconds"] == 3.0
assert body["edit_segments"] == sections
# And getting the job again afterwards reflects the same recorded values.
again = client.get(f"/api/jobs/{job['id']}").json()
assert again["edit_segments"] == sections
def test_trim_response_records_multiple_sections(client):
job = _completed(client, "clip")
mock_server._jobs[job["id"]]["num_frames"] = 240
mock_server._jobs[job["id"]]["fps"] = 24
sections = [seg(brightness=20.0, end_seconds=2.0), seg(contrast=1.5)]
res = client.post(f"/api/jobs/{job['id']}/trim", json={"start_seconds": 0.0, "end_seconds": 5.0,
"segments": sections})
assert res.json()["edit_segments"] == sections
def test_restore_resets_the_recorded_edit_values(client):
job = _completed(client, "clip")
mock_server._jobs[job["id"]]["num_frames"] = 240
mock_server._jobs[job["id"]]["fps"] = 24
client.post(
f"/api/jobs/{job['id']}/trim",
json={"start_seconds": 1.0, "end_seconds": 3.0, "segments": [seg(brightness=20.0, contrast=1.5, saturation=0.5)]},
)
res = client.post(f"/api/jobs/{job['id']}/restore-video")
body = res.json()
assert body["edit_start_seconds"] == 0.0
assert body["edit_end_seconds"] is None
assert body["edit_segments"] == []
@pytest.mark.parametrize(
"segments",
[
[seg(brightness=200.0)],
[seg(contrast=-1.0)],
[seg(saturation=10.0)],
],
)
def test_trim_refuses_an_out_of_range_color_value(client, segments):
job = _completed(client, "clip")
mock_server._jobs[job["id"]]["num_frames"] = 240
mock_server._jobs[job["id"]]["fps"] = 24
body = {"start_seconds": 0.0, "segments": segments}
assert client.post(f"/api/jobs/{job['id']}/trim", json=body).status_code == 400
def test_an_empty_section_list_is_not_an_error_matching_trim_py(client):
# Same as sending no `segments` at all -- defaults to one neutral section.
job = _completed(client, "clip")
mock_server._jobs[job["id"]]["num_frames"] = 240
mock_server._jobs[job["id"]]["fps"] = 24
res = client.post(f"/api/jobs/{job['id']}/trim", json={"start_seconds": 0.0, "segments": []})
assert res.status_code == 200
def test_trim_refuses_a_non_last_open_ended_section(client):
job = _completed(client, "clip")
mock_server._jobs[job["id"]]["num_frames"] = 240
mock_server._jobs[job["id"]]["fps"] = 24
body = {"start_seconds": 0.0, "segments": [seg(), seg(end_seconds=2.0)]}
res = client.post(f"/api/jobs/{job['id']}/trim", json=body)
assert res.status_code == 400 and "open-ended" in res.json()["detail"]
def test_trim_refuses_sections_out_of_order(client):
job = _completed(client, "clip")
mock_server._jobs[job["id"]]["num_frames"] = 240
mock_server._jobs[job["id"]]["fps"] = 24
body = {"start_seconds": 0.0, "segments": [seg(end_seconds=3.0), seg(end_seconds=2.0), seg()]}
res = client.post(f"/api/jobs/{job['id']}/trim", json=body)
assert res.status_code == 400 and "increasing order" in res.json()["detail"]
def test_trim_refuses_an_unfinished_job(client):
job = _create(client, name="pending")
res = client.post(f"/api/jobs/{job['id']}/trim", json={"start_seconds": 0.0, "end_seconds": 1.0})
assert res.status_code == 404
def test_trim_and_restore_of_an_unknown_job(client):
assert client.post("/api/jobs/nope/trim", json={"start_seconds": 0.0}).status_code == 404
assert client.post("/api/jobs/nope/restore-video").status_code == 404
def test_trimmed_frame_count_does_not_leak_the_internal_backup_key(client):
job = _completed(client, "clip")
mock_server._jobs[job["id"]]["num_frames"] = 240
mock_server._jobs[job["id"]]["fps"] = 24
res = client.post(f"/api/jobs/{job['id']}/trim", json={"start_seconds": 1.0, "end_seconds": 3.0})
assert not any(k.startswith("_") for k in res.json())
assert not any(k.startswith("_") for k in client.get(f"/api/jobs/{job['id']}").json())
+499
View File
@@ -0,0 +1,499 @@
# SPDX-License-Identifier: Apache-2.0
"""Cutting a finished clip to a chosen in/out range and adjusting its color, non-destructively."""
from __future__ import annotations
import os
from enum import Enum
from types import SimpleNamespace
import numpy as np
import pytest
from fastvideo_studio.frames import FrameError
from fastvideo_studio.trim import (
BRIGHTNESS_RANGE,
CONTRAST_RANGE,
EDITED_FILENAME,
ORIGINAL_FILENAME,
SATURATION_RANGE,
clip_duration_seconds,
restore_original,
trim_video,
)
WIDTH, HEIGHT, FPS, FRAMES = 128, 96, 24, 48 # 2 seconds, one frame per level 0..47
class Status(Enum): # mirrors JobStatus without importing job_runner (which needs fastvideo)
PENDING = "pending"
COMPLETED = "completed"
FAILED = "failed"
def _write_video(path: str, n: int = FRAMES, fps: int = FPS) -> None:
"""A clip whose frames are numbered by brightness (frame i is roughly i*5 grey), so a range is identifiable."""
import av
container = av.open(path, mode="w")
try:
stream = container.add_stream("h264", rate=fps)
stream.width, stream.height, stream.pix_fmt = WIDTH, HEIGHT, "yuv420p"
for i in range(n):
level = min(i * 5, 255)
frame = av.VideoFrame.from_ndarray(np.full((HEIGHT, WIDTH, 3), level, dtype=np.uint8), format="rgb24")
for packet in stream.encode(frame):
container.mux(packet)
for packet in stream.encode():
container.mux(packet)
finally:
container.close()
AUDIO_SAMPLE_RATE = 32000
AUDIO_SAMPLES_PER_FRAME = 1024
def _write_video_with_audio(path: str, n: int = FRAMES, fps: int = FPS) -> None:
"""Same as ``_write_video``, plus a silent stereo AAC track, to test that edits keep audio."""
import av
from fractions import Fraction
container = av.open(path, mode="w")
try:
vstream = container.add_stream("h264", rate=fps)
vstream.width, vstream.height, vstream.pix_fmt = WIDTH, HEIGHT, "yuv420p"
astream = container.add_stream("aac", rate=AUDIO_SAMPLE_RATE)
astream.layout = "stereo"
for i in range(n):
level = min(i * 5, 255)
frame = av.VideoFrame.from_ndarray(np.full((HEIGHT, WIDTH, 3), level, dtype=np.uint8), format="rgb24")
for packet in vstream.encode(frame):
container.mux(packet)
for packet in vstream.encode():
container.mux(packet)
audio_time_base = Fraction(1, AUDIO_SAMPLE_RATE)
n_audio_frames = int(n / fps * AUDIO_SAMPLE_RATE / AUDIO_SAMPLES_PER_FRAME) + 2
pts = 0
for _ in range(n_audio_frames):
arr = np.zeros((2, AUDIO_SAMPLES_PER_FRAME), dtype=np.float32)
aframe = av.AudioFrame.from_ndarray(arr, format="fltp", layout="stereo")
aframe.sample_rate = AUDIO_SAMPLE_RATE
aframe.pts = pts
aframe.time_base = audio_time_base
pts += AUDIO_SAMPLES_PER_FRAME
for packet in astream.encode(aframe):
container.mux(packet)
for packet in astream.encode(None):
container.mux(packet)
finally:
container.close()
def _write_colored_video(path: str, rgb: tuple[int, int, int], n: int = 12, fps: int = FPS) -> None:
"""A clip of one solid, non-gray color, so saturation changes are visible (unlike on a gray clip)."""
import av
container = av.open(path, mode="w")
try:
stream = container.add_stream("h264", rate=fps)
stream.width, stream.height, stream.pix_fmt = WIDTH, HEIGHT, "yuv420p"
for _ in range(n):
frame = av.VideoFrame.from_ndarray(np.full((HEIGHT, WIDTH, 3), rgb, dtype=np.uint8), format="rgb24")
for packet in stream.encode(frame):
container.mux(packet)
for packet in stream.encode():
container.mux(packet)
finally:
container.close()
def _read_levels(path: str) -> list[int]:
import av
with av.open(path) as container:
return [int(f.to_ndarray(format="rgb24")[0, 0, 0]) for f in container.decode(container.streams.video[0])]
def _read_mean_rgb(path: str, index: int = 0) -> np.ndarray:
import av
with av.open(path) as container:
frames = list(container.decode(container.streams.video[0]))
return frames[index].to_ndarray(format="rgb24").astype(float).mean(axis=(0, 1))
def _job(job_id="job-1", status=Status.COMPLETED, output_path=None):
return SimpleNamespace(id=job_id, status=status, output_path=output_path)
def _top_level_atom_offset(path: str, atom_type: bytes) -> int:
"""Byte offset of a top-level MP4 box (e.g. b"moov", b"mdat"), or -1 if absent."""
import struct
with open(path, "rb") as f:
f.seek(0, 2)
size = f.tell()
f.seek(0)
pos = 0
while pos < size:
f.seek(pos)
header = f.read(8)
if len(header) < 8:
break
atom_size, found_type = struct.unpack(">I4s", header)
if found_type == atom_type:
return pos
if atom_size in (0, 1):
break
pos += atom_size
return -1
def _moov_offset(path: str) -> int:
return _top_level_atom_offset(path, b"moov")
def _mdat_offset(path: str) -> int:
return _top_level_atom_offset(path, b"mdat")
def seg(brightness=0.0, contrast=1.0, saturation=1.0, end_seconds=None):
"""One razor-cut section dict, as the API/trim_video expects."""
return {"end_seconds": end_seconds, "brightness": brightness, "contrast": contrast, "saturation": saturation}
@pytest.fixture
def job_dir(tmp_path):
d = tmp_path / "job-1"
d.mkdir()
return d
@pytest.fixture
def video(job_dir):
path = str(job_dir / "wolf-lunch-clip-06.mp4")
_write_video(path)
return path
@pytest.fixture
def colored_video(job_dir):
path = str(job_dir / "wolf-lunch-clip-07.mp4")
_write_colored_video(path, (200, 100, 50)) # a warm orange, clearly non-gray
return path
@pytest.fixture
def video_with_audio(job_dir):
path = str(job_dir / "wolf-lunch-clip-08.mp4")
_write_video_with_audio(path)
return path
class TestTrimVideo:
def test_keeps_only_the_requested_range(self, video, job_dir):
path, kept = trim_video(_job(output_path=video), start_seconds=0.5, end_seconds=1.5)
assert path == str(job_dir / EDITED_FILENAME)
levels = _read_levels(path)
assert kept == len(levels) == FPS # 1 second kept
assert levels[0] == _read_levels(video)[12] # starts at 0.5s = frame 12
assert levels[-1] == _read_levels(video)[35] # ends just before 1.5s = frame 36
def test_an_open_end_keeps_to_the_end_of_the_source(self, video):
path, kept = trim_video(_job(output_path=video), start_seconds=1.0, end_seconds=None)
assert kept == FRAMES - FPS
assert _read_levels(path)[-1] == _read_levels(video)[-1]
def test_start_at_zero_keeps_from_the_beginning(self, video):
path, _ = trim_video(_job(output_path=video), start_seconds=0.0, end_seconds=1.0)
assert _read_levels(path)[0] == _read_levels(video)[0]
def test_the_original_is_preserved_byte_for_byte(self, video, job_dir):
before = open(video, "rb").read()
trim_video(_job(output_path=video), start_seconds=0.5, end_seconds=1.5)
original = job_dir / ORIGINAL_FILENAME
assert original.is_file()
assert original.read_bytes() == before
def test_trimming_again_re_cuts_from_the_original_not_the_edited_file(self, video, job_dir):
# If it re-cut the already-edited 0.5-1.5s file, asking for 0-2s again
# could not recover the parts outside that first window.
trim_video(_job(output_path=video), start_seconds=0.5, end_seconds=1.5)
path, kept = trim_video(_job(output_path=video), start_seconds=0.0, end_seconds=2.0)
assert kept == FRAMES
assert _read_levels(path) == _read_levels(video)
def test_narrowing_the_range_further_still_re_cuts_from_the_original(self, video):
trim_video(_job(output_path=video), start_seconds=0.0, end_seconds=2.0)
path, kept = trim_video(_job(output_path=video), start_seconds=1.0, end_seconds=1.5)
assert kept == FPS // 2
assert _read_levels(path)[0] == _read_levels(video)[24]
def test_muxed_frame_rate_stays_correct(self, video):
path, _ = trim_video(_job(output_path=video), start_seconds=0.0, end_seconds=1.0)
import av
with av.open(path) as c:
assert FPS - 1 <= float(c.streams.video[0].average_rate) <= FPS + 1
def test_output_is_faststart_moov_before_mdat(self, video):
# Otherwise a browser has to reach nearly the end of the file just to read
# its duration/seek table -- painfully slow over a network filesystem.
path, _ = trim_video(_job(output_path=video), start_seconds=0.0, end_seconds=1.0)
assert _moov_offset(path) < _mdat_offset(path)
@pytest.mark.parametrize(
("start", "end", "match"),
[
(-1.0, None, "negative"),
(1.0, 1.0, "after the start"),
(1.0, 0.5, "after the start"),
(100.0, None, "no frames"),
],
)
def test_refuses_an_invalid_range(self, video, start, end, match):
with pytest.raises(FrameError, match=match) as e:
trim_video(_job(output_path=video), start_seconds=start, end_seconds=end)
assert e.value.status_code == 400
@pytest.mark.parametrize(
("job", "match"),
[
(_job(status=Status.PENDING), "No output"),
(_job(status=Status.COMPLETED, output_path=None), "No output"),
(_job(status=Status.COMPLETED, output_path="/nope.mp4"), "not found"),
],
)
def test_refusals(self, job, match):
with pytest.raises(FrameError, match=match):
trim_video(job, 0.0, 1.0)
def test_refuses_a_non_video_output(self, tmp_path):
img = tmp_path / "out.png"
img.write_bytes(b"x")
with pytest.raises(FrameError, match="not a video"):
trim_video(_job(output_path=str(img)), 0.0, 1.0)
class TestGrading:
def test_neutral_values_do_not_alter_the_pixels(self, colored_video):
path, _ = trim_video(_job(output_path=colored_video), 0.0, None, segments=[seg()])
got = _read_mean_rgb(path)
want = _read_mean_rgb(colored_video)
assert np.allclose(got, want, atol=2) # h264 is lossy even unchanged
def test_neutral_values_skip_the_per_frame_numpy_round_trip(self, colored_video, monkeypatch):
import fastvideo_studio.trim as trim_module
calls = []
monkeypatch.setattr(trim_module, "_apply_grade", lambda *a: calls.append(a) or a[0])
trim_video(_job(output_path=colored_video), 0.0, None) # no segments -> the default single neutral one
assert calls == []
def test_positive_brightness_lightens_every_channel(self, colored_video):
path, _ = trim_video(_job(output_path=colored_video), 0.0, None, segments=[seg(brightness=50.0)])
got = _read_mean_rgb(path)
want = _read_mean_rgb(colored_video)
assert (got > want + 30).all()
def test_negative_brightness_darkens(self, colored_video):
path, _ = trim_video(_job(output_path=colored_video), 0.0, None, segments=[seg(brightness=-50.0)])
got = _read_mean_rgb(path)
want = _read_mean_rgb(colored_video)
assert (got < want - 30).all()
def test_zero_saturation_makes_every_channel_equal(self, colored_video):
path, _ = trim_video(_job(output_path=colored_video), 0.0, None, segments=[seg(saturation=0.0)])
r, g, b = _read_mean_rgb(path)
assert abs(r - g) < 2 and abs(g - b) < 2 # gray: channels converge on the frame's luma
def test_saturation_above_one_widens_the_channel_spread(self, colored_video):
base_r, base_g, base_b = _read_mean_rgb(colored_video)
path, _ = trim_video(_job(output_path=colored_video), 0.0, None, segments=[seg(saturation=2.0)])
r, g, b = _read_mean_rgb(path)
assert (r - b) > (base_r - base_b) # the warm/cool spread grows, not shrinks
def test_high_contrast_pushes_values_away_from_mid_gray(self, colored_video):
path, _ = trim_video(_job(output_path=colored_video), 0.0, None, segments=[seg(contrast=2.0)])
got = _read_mean_rgb(path)
want = _read_mean_rgb(colored_video)
# 128 = mid-gray. This clip's R (200) sits above it and should be pushed further up;
# its G and B (100, 50) sit below and should be pushed further down -- the spread
# around 128 widens either way.
assert got[0] > want[0] # R: 200 -> further above 128
assert got[1] < want[1] # G: 100 -> further below 128
assert got[2] < want[2] # B: 50 -> further below 128
def test_grading_and_trimming_the_same_call_compose(self, video, colored_video):
# A range trim and a color grade requested together both take effect --
# neither is silently dropped by the other.
path, kept = trim_video(
_job(output_path=colored_video), start_seconds=0.2, end_seconds=0.3, segments=[seg(brightness=40.0)])
assert kept == pytest.approx(FPS * 0.1, abs=1)
base = _read_mean_rgb(colored_video)
assert (_read_mean_rgb(path) > base + 20).all()
@pytest.mark.parametrize(
("kwargs", "bounds"),
[
({"brightness": BRIGHTNESS_RANGE[1] + 1}, BRIGHTNESS_RANGE),
({"brightness": BRIGHTNESS_RANGE[0] - 1}, BRIGHTNESS_RANGE),
({"contrast": CONTRAST_RANGE[1] + 1}, CONTRAST_RANGE),
({"contrast": CONTRAST_RANGE[0] - 1}, CONTRAST_RANGE),
({"saturation": SATURATION_RANGE[1] + 1}, SATURATION_RANGE),
({"saturation": SATURATION_RANGE[0] - 1}, SATURATION_RANGE),
],
)
def test_refuses_an_out_of_range_value(self, colored_video, kwargs, bounds):
with pytest.raises(FrameError, match="between") as e:
trim_video(_job(output_path=colored_video), 0.0, None, segments=[seg(**kwargs)])
assert e.value.status_code == 400
@pytest.mark.parametrize("kwargs", [{"brightness": BRIGHTNESS_RANGE[1]}, {"contrast": CONTRAST_RANGE[0]},
{"saturation": SATURATION_RANGE[1]}])
def test_the_edges_of_each_range_are_allowed(self, colored_video, kwargs):
trim_video(_job(output_path=colored_video), 0.0, None, segments=[seg(**kwargs)]) # does not raise
class TestSections:
"""Razor-cut sections: an ordered, contiguous list of grades within the kept range."""
def test_two_sections_grade_independently(self, colored_video):
path, kept = trim_video(
_job(output_path=colored_video), 0.0, None,
segments=[seg(brightness=80.0, end_seconds=0.25), seg()],
)
base = _read_mean_rgb(colored_video)
assert (_read_mean_rgb(path, index=0) > base + 40).all() # first section, graded
assert np.allclose(_read_mean_rgb(path, index=kept - 1), base, atol=2) # second section, untouched
def test_three_sections_each_get_their_own_grade(self, colored_video):
path, kept = trim_video(
_job(output_path=colored_video), 0.0, None,
segments=[
seg(saturation=0.0, end_seconds=4 / FPS),
seg(brightness=80.0, end_seconds=8 / FPS),
seg(),
],
)
base = _read_mean_rgb(colored_video)
r0, g0, b0 = _read_mean_rgb(path, index=0)
assert abs(r0 - g0) < 2 and abs(g0 - b0) < 2 # first section desaturated
assert (_read_mean_rgb(path, index=6) > base + 40).all() # second section brightened
assert np.allclose(_read_mean_rgb(path, index=kept - 1), base, atol=2) # third section untouched
def test_sections_compose_with_the_range_trim(self, colored_video):
# The cut point (0.1) is relative to the KEPT range, not the original file's timeline.
path, kept = trim_video(
_job(output_path=colored_video), start_seconds=0.1, end_seconds=0.4,
segments=[seg(brightness=80.0, end_seconds=0.1), seg()],
)
base = _read_mean_rgb(colored_video)
assert (_read_mean_rgb(path, index=0) > base + 40).all()
assert np.allclose(_read_mean_rgb(path, index=kept - 1), base, atol=2)
def test_an_empty_section_list_defaults_to_one_neutral_section(self, colored_video):
path, _ = trim_video(_job(output_path=colored_video), 0.0, None, segments=[])
assert np.allclose(_read_mean_rgb(path), _read_mean_rgb(colored_video), atol=2)
def test_a_single_explicit_end_that_does_not_cover_the_clip_is_refused(self, colored_video):
with pytest.raises(FrameError, match="cover the whole clip"):
trim_video(_job(output_path=colored_video), 0.0, None, segments=[seg(end_seconds=0.1)])
def test_only_the_last_section_can_be_open_ended(self, colored_video):
with pytest.raises(FrameError, match="open-ended"):
trim_video(_job(output_path=colored_video), 0.0, None, segments=[seg(), seg(end_seconds=0.25)])
def test_sections_must_be_in_increasing_order(self, colored_video):
with pytest.raises(FrameError, match="increasing order"):
trim_video(
_job(output_path=colored_video), 0.0, None,
segments=[seg(end_seconds=0.3), seg(end_seconds=0.2), seg()],
)
def test_a_section_too_short_to_keep_a_frame_is_refused(self, colored_video):
# 0.001s at 24fps rounds to 0 frames -- passes the ordering check (> 0) but keeps nothing.
with pytest.raises(FrameError, match="too short"):
trim_video(_job(output_path=colored_video), 0.0, None, segments=[seg(end_seconds=0.001), seg()])
def test_a_section_with_an_out_of_range_grade_is_refused(self, colored_video):
with pytest.raises(FrameError, match="between"):
trim_video(
_job(output_path=colored_video), 0.0, None,
segments=[seg(end_seconds=0.25), seg(brightness=500.0)],
)
def test_only_frames_in_a_graded_section_go_through_the_numpy_round_trip(self, colored_video, monkeypatch):
import fastvideo_studio.trim as trim_module
calls = []
original = trim_module._apply_grade
monkeypatch.setattr(trim_module, "_apply_grade", lambda *a: calls.append(a) or original(*a))
trim_video(
_job(output_path=colored_video), 0.0, None,
segments=[seg(end_seconds=0.25), seg(brightness=50.0)], # first neutral, second graded
)
assert 0 < len(calls) < 12 # some frames graded, not all -- the neutral section was skipped
class TestAudio:
def test_a_source_with_no_audio_stays_video_only(self, video):
path, _ = trim_video(_job(output_path=video), start_seconds=0.5, end_seconds=1.5)
import av
with av.open(path) as c:
assert [s.type for s in c.streams] == ["video"]
def test_a_range_trim_keeps_the_audio_track(self, video_with_audio):
path, _ = trim_video(_job(output_path=video_with_audio), start_seconds=0.5, end_seconds=1.5)
import av
with av.open(path) as c:
assert "audio" in [s.type for s in c.streams]
a = c.streams.audio[0]
assert a.codec_context.sample_rate == AUDIO_SAMPLE_RATE
assert sum(1 for _ in c.decode(a)) > 0
def test_grading_alongside_a_trim_also_keeps_the_audio_track(self, video_with_audio):
path, _ = trim_video(
_job(output_path=video_with_audio), start_seconds=0.0, end_seconds=1.0, segments=[seg(brightness=40.0)])
import av
with av.open(path) as c:
assert "audio" in [s.type for s in c.streams]
def test_restore_original_keeps_the_audio_track(self, video_with_audio, job_dir):
trim_video(_job(output_path=video_with_audio), start_seconds=0.5, end_seconds=1.5)
path, _ = restore_original(_job(output_path=str(job_dir / EDITED_FILENAME)))
import av
with av.open(path) as c:
assert "audio" in [s.type for s in c.streams]
class TestRestoreOriginal:
def test_restores_the_original_video_and_its_frame_count(self, video, job_dir):
trim_video(_job(output_path=video), start_seconds=0.5, end_seconds=1.5)
path, n = restore_original(_job(output_path=str(job_dir / EDITED_FILENAME)))
assert path == str(job_dir / ORIGINAL_FILENAME)
assert n == FRAMES
assert _read_levels(path) == _read_levels(video)
def test_restores_past_a_grade_too(self, colored_video, job_dir):
trim_video(_job(output_path=colored_video), 0.0, None, segments=[seg(brightness=50.0, saturation=0.0)])
path, _ = restore_original(_job(output_path=str(job_dir / EDITED_FILENAME)))
assert np.allclose(_read_mean_rgb(path), _read_mean_rgb(colored_video), atol=2)
def test_refuses_a_job_that_was_never_edited(self, video):
with pytest.raises(FrameError, match="not been edited"):
restore_original(_job(output_path=video))
def test_refuses_a_job_with_no_output(self):
with pytest.raises(FrameError, match="No output"):
restore_original(_job(output_path=None))
def test_clip_duration_seconds(video):
assert clip_duration_seconds(video) == pytest.approx(FRAMES / FPS, abs=0.05)
+310
View File
@@ -0,0 +1,310 @@
# SPDX-License-Identifier: Apache-2.0
"""Cut a finished clip to a chosen in/out range and adjust its color, non-destructively.
Kept apart from ``frames.py`` (which saves the *end* of a clip for another job
to continue from) because this edits a clip's own timeline, not a reference
for someone else. Needs only PyAV and numpy, so it can be exercised without a GPU.
Trimming and grading share one pipeline rather than being two independent
edits, on purpose: both always re-render from the untouched original in a
single pass with the *current* full set of parameters (range and color
together). Two separate "always start from the original" operations would
each undo the other's work the moment they ran -- grading, then trimming,
would silently discard the grade, because the trim would re-cut the original
as if the grade had never happened. Rendering everything from one function
each time is what lets adjusting either one leave the other in place.
Color grading is per razor-cut section, not the whole clip at once: the kept
range is split at zero or more cut points into contiguous sections (no
reordering, no gaps), each with its own brightness/contrast/saturation. A
plain range trim with no cuts is just the one-section case. Audio is one
continuous track regardless -- cuts only change which grade a video frame
gets, they don't touch the audio.
"""
from __future__ import annotations
import os
import shutil
from fractions import Fraction
from pathlib import Path
from typing import Any
import numpy as np
from fastvideo_studio.frames import VIDEO_EXTENSIONS, FrameError
ORIGINAL_FILENAME = "original.mp4"
EDITED_FILENAME = "edited.mp4"
# Neutral (no-op) values for each color control, and the range each is clamped to.
BRIGHTNESS_RANGE = (-100.0, 100.0) # added directly to 0-255 pixel values
CONTRAST_RANGE = (0.0, 3.0) # 1.0 = unchanged
SATURATION_RANGE = (0.0, 3.0) # 1.0 = unchanged, 0 = grayscale
# Perceptual luma weights, matching the ones used elsewhere in Studio for brightness checks.
_LUMA_WEIGHTS = np.array([0.2126, 0.7152, 0.0722], dtype=np.float32)
def clip_duration_seconds(video_path: str) -> float:
"""Decode ``video_path`` and return its length in seconds."""
import av
with av.open(video_path) as container:
stream = container.streams.video[0]
fps = float(stream.average_rate or stream.guessed_rate or 24)
n = sum(1 for _ in container.decode(stream))
if n == 0:
raise ValueError("The video has no frames.")
return n / fps
def _clamp_range(name: str, value: float, bounds: tuple[float, float]) -> float:
low, high = bounds
if not (low <= value <= high):
raise ValueError(f"{name} must be between {low:g} and {high:g}, got {value:g}.")
return value
def _apply_grade(array: np.ndarray, brightness: float, contrast: float, saturation: float) -> np.ndarray:
"""Adjust one RGB uint8 frame. Neutral values (0, 1, 1) are a no-op, skipped by the caller."""
pixels = array.astype(np.float32)
if saturation != 1.0:
luma = (pixels * _LUMA_WEIGHTS).sum(axis=-1, keepdims=True)
pixels = luma + (pixels - luma) * saturation
if contrast != 1.0:
pixels = (pixels - 128.0) * contrast + 128.0
if brightness != 0.0:
pixels = pixels + brightness
return np.clip(pixels, 0, 255).astype(np.uint8)
#: A neutral single section covering the whole kept range -- what a never-edited
#: clip (or a call that doesn't care about sections) gets by default.
DEFAULT_SEGMENTS: list[dict[str, Any]] = [{"end_seconds": None, "brightness": 0.0, "contrast": 1.0, "saturation": 1.0}]
class _Segment:
"""One validated, frame-indexed section: [start_frame, end_frame) plus its grade."""
__slots__ = ("start_frame", "end_frame", "brightness", "contrast", "saturation")
def __init__(self, start_frame: int, end_frame: int, brightness: float, contrast: float, saturation: float):
self.start_frame = start_frame
self.end_frame = end_frame
self.brightness = brightness
self.contrast = contrast
self.saturation = saturation
@property
def graded(self) -> bool:
return self.brightness != 0.0 or self.contrast != 1.0 or self.saturation != 1.0
def _normalize_segments(segments: list[dict[str, Any]] | None, kept_frames: int, fps: float) -> list[_Segment]:
"""Validate a razor-cut section list and convert each boundary to a frame index.
Sections are contiguous and cover the whole kept range in order -- there's
no reordering or gaps, only where to cut and what to grade each side. Cut
points are seconds into the *kept* (already start/end-trimmed) range, i.e.
the same timeline the preview video itself plays, not the original file's.
"""
segments = segments or DEFAULT_SEGMENTS
out: list[_Segment] = []
start_frame = 0
prev_end_seconds = 0.0
for i, seg in enumerate(segments):
is_last = i == len(segments) - 1
end_seconds = seg.get("end_seconds")
brightness = _clamp_range("brightness", float(seg.get("brightness", 0.0)), BRIGHTNESS_RANGE)
contrast = _clamp_range("contrast", float(seg.get("contrast", 1.0)), CONTRAST_RANGE)
saturation = _clamp_range("saturation", float(seg.get("saturation", 1.0)), SATURATION_RANGE)
if end_seconds is None:
if not is_last:
raise ValueError("Only the last section can be open-ended.")
end_frame = kept_frames
else:
if end_seconds <= prev_end_seconds:
raise ValueError("Sections must be in increasing order.")
end_frame = min(kept_frames, round(end_seconds * fps))
prev_end_seconds = end_seconds
if end_frame <= start_frame:
raise ValueError("A section is too short to keep any frames.")
out.append(_Segment(start_frame, end_frame, brightness, contrast, saturation))
start_frame = end_frame
if start_frame < kept_frames:
raise ValueError("The sections must cover the whole clip; the last one can't end early.")
return out
def _render_edit(
video_path: str,
out_path: str,
start_seconds: float,
end_seconds: float | None,
segments: list[dict[str, Any]] | None,
) -> int:
"""Re-encode [``start_seconds``, ``end_seconds``) of ``video_path`` to ``out_path``, graded per section.
Re-encoded rather than stream-copied, for a frame-accurate cut regardless
of where the source's keyframes happen to fall. Returns the frame count kept.
"""
import av
if start_seconds < 0:
raise ValueError("The start of the range can't be negative.")
if end_seconds is not None and end_seconds <= start_seconds:
raise ValueError("The end of the range must come after the start.")
with av.open(video_path) as src:
video_stream = src.streams.video[0]
audio_stream = src.streams.audio[0] if src.streams.audio else None
fps = float(video_stream.average_rate or video_stream.guessed_rate or 24)
frames: list[Any] = []
audio_frames: list[Any] = []
# One decode pass covers both streams -- the container can't be
# decoded a second time once its packets are exhausted.
decode_streams = (video_stream, audio_stream) if audio_stream else (video_stream,)
for frame in src.decode(*decode_streams):
(audio_frames if isinstance(frame, av.AudioFrame) else frames).append(frame)
if not frames:
raise ValueError("The video has no frames.")
start_i = max(0, round(start_seconds * fps))
end_i = len(frames) if end_seconds is None else min(len(frames), round(end_seconds * fps))
kept = frames[start_i:end_i]
if not kept:
raise ValueError("That range keeps no frames.")
# Audio is cut to the same frame-granular window as the video, not the raw
# requested seconds, so the two tracks stay aligned. Razor cuts only affect
# which grade a frame gets, not the audio, which stays one continuous track.
kept_audio = [f for f in audio_frames if f.time is not None and start_i / fps <= f.time < end_i / fps]
section_list = _normalize_segments(segments, len(kept), fps)
tmp_path = f"{out_path}.tmp.mp4"
# movflags=faststart puts the moov atom (duration, seek table) at the front of
# the file instead of the end -- without it, a browser has to reach nearly the
# end of the file just to read the video's metadata, which is painfully slow
# over a network filesystem.
with av.open(tmp_path, mode="w", options={"movflags": "faststart"}) as out:
out_stream = out.add_stream("h264", rate=round(fps))
out_stream.width, out_stream.height = kept[0].width, kept[0].height
out_stream.pix_fmt = "yuv420p"
# Every output stream has to be added before the first packet is muxed
# (that's what writes the container header), so the audio stream and
# resampler are set up here even though they're only used after the
# video loop below.
out_audio_stream = None
resampler = None
if kept_audio:
out_audio_stream = out.add_stream("aac", rate=audio_stream.codec_context.sample_rate)
out_audio_stream.layout = audio_stream.codec_context.layout.name
resampler = av.AudioResampler(
format=out_audio_stream.format,
layout=out_audio_stream.layout,
rate=out_audio_stream.rate,
)
# Each frame needs its own increasing pts at a known time base, or the
# muxed container's frame-rate metadata comes out wrong (see frames.py's
# save_last_clip, which hit exactly this).
frame_time_base = Fraction(1, round(fps))
section_i = 0
for i, frame in enumerate(kept):
while i >= section_list[section_i].end_frame:
section_i += 1
section = section_list[section_i]
if section.graded:
array = _apply_grade(frame.to_ndarray(format="rgb24"), section.brightness, section.contrast,
section.saturation)
frame = av.VideoFrame.from_ndarray(array, format="rgb24")
frame.pts = i
frame.time_base = frame_time_base
for packet in out_stream.encode(frame):
out.mux(packet)
for packet in out_stream.encode():
out.mux(packet)
if out_audio_stream is not None:
for frame in kept_audio:
frame.pts = None # let the resampler retime relative to the trimmed start
for rframe in resampler.resample(frame):
for packet in out_audio_stream.encode(rframe):
out.mux(packet)
for packet in out_audio_stream.encode(None):
out.mux(packet)
os.replace(tmp_path, out_path) # atomic, so a reader never sees a half-written file
return len(kept)
def _require_completed_video(job: Any) -> None:
status = getattr(job.status, "value", job.status)
if status != "completed" or not job.output_path:
raise FrameError(404, "No output available for this job")
if not os.path.isfile(job.output_path):
raise FrameError(404, "Output file not found on disk")
if Path(job.output_path).suffix.lower() not in VIDEO_EXTENSIONS:
raise FrameError(400, "This job's output is not a video")
def trim_video(
job: Any,
start_seconds: float = 0.0,
end_seconds: float | None = None,
segments: list[dict[str, Any]] | None = None,
) -> tuple[str, int]:
"""Cut ``job``'s video to [``start_seconds``, ``end_seconds``) and grade it section by section.
``segments`` is an ordered, contiguous list of razor-cut sections covering
the whole kept range -- each a ``{"end_seconds", "brightness", "contrast",
"saturation"}`` dict, ``end_seconds`` being seconds into the *kept* range
(only the last section's may be ``None``, meaning to the end). Omitted or
empty defaults to one neutral section covering everything, i.e. what a
plain range trim with no grading looks like.
Always renders from the untouched original with this *full* set of
parameters, so trimming and grading compose -- adjusting one doesn't
discard the other, and neither ever re-encodes an already-edited file.
Returns ``(new_output_path, frames_kept)``. Raises :class:`FrameError` with
the status the API should report.
"""
_require_completed_video(job)
job_dir = os.path.dirname(job.output_path)
original_path = os.path.join(job_dir, ORIGINAL_FILENAME)
if not os.path.isfile(original_path):
# First edit: keep an untouched copy of whatever is there now before
# changing anything, so every later edit (and a restore) is relative
# to this, never to a file this function already produced.
shutil.copy2(job.output_path, original_path)
edited_path = os.path.join(job_dir, EDITED_FILENAME)
try:
kept = _render_edit(original_path, edited_path, start_seconds, end_seconds, segments)
except FrameError:
raise
except Exception as e:
raise FrameError(400, str(e)) from e
return edited_path, kept
def restore_original(job: Any) -> tuple[str, int]:
"""Undo every edit: the path of ``job``'s untouched original video, and its frame count.
Undoes trimming and grading together, since they share one edit state.
Raises :class:`FrameError` if the job was never edited.
"""
if not job.output_path:
raise FrameError(404, "No output available for this job")
original_path = os.path.join(os.path.dirname(job.output_path), ORIGINAL_FILENAME)
if not os.path.isfile(original_path):
raise FrameError(404, "This job has not been edited.")
try:
import av
with av.open(original_path) as container:
stream = container.streams.video[0]
n = sum(1 for _ in container.decode(stream))
except Exception as e:
raise FrameError(500, f"Could not read the original video: {e}") from e
return original_path, n
+32 -3
View File
@@ -128,6 +128,30 @@ def _resolve_output_size(
return fallback
def _find_ffmpeg() -> str | None:
"""An ffmpeg to run: $FASTVIDEO_FFMPEG_BIN, then PATH, then imageio-ffmpeg's bundled one.
A plain PATH lookup misses environments (this one included) where ffmpeg
is only present as the copy imageio-ffmpeg installs for its own use, never
exposed on PATH -- without this fallback, every save silently drops to the
PyAV path below, which has no '+faststart', producing MP4s slow to start
playback over a network filesystem (the metadata ends up at the end of the
file instead of the front).
"""
configured = os.getenv("FASTVIDEO_FFMPEG_BIN")
if configured and (shutil.which(configured) or os.path.isfile(configured)):
return shutil.which(configured) or configured
on_path = shutil.which("ffmpeg")
if on_path:
return on_path
try:
import imageio_ffmpeg
return imageio_ffmpeg.get_ffmpeg_exe()
except Exception:
return None
def _validate_request_stage_overrides(model_path: str, request: GenerationRequest) -> None:
"""Validate typed stage overrides against the model's registered preset."""
if not request.stage_overrides:
@@ -1100,7 +1124,10 @@ class VideoGenerator:
try:
audio_int16, num_channels = cls._audio_to_int16(audio)
layout = "stereo" if num_channels == 2 else "mono"
output = av.open(output_path, mode="w")
# movflags=faststart puts the moov atom (duration, seek table) at the
# front of the file instead of the end -- this is the fallback path
# (ffmpeg unavailable), so it's the one place nothing else adds it.
output = av.open(output_path, mode="w", options={"movflags": "faststart"})
video_stream = output.add_stream("libx264", rate=fps)
video_stream.width = int(frames[0].shape[1])
video_stream.height = int(frames[0].shape[0])
@@ -1185,9 +1212,11 @@ class VideoGenerator:
sample_rate: int,
) -> bool:
"""Encode video+audio using ffmpeg via rawvideo stdin + WAV input."""
ffmpeg_bin = shutil.which(os.getenv("FASTVIDEO_FFMPEG_BIN", "ffmpeg"))
ffmpeg_bin = _find_ffmpeg()
if ffmpeg_bin is None:
logger.warning("ffmpeg not found; cannot use ffmpeg pipe save.")
logger.warning("ffmpeg not found (checked $FASTVIDEO_FFMPEG_BIN, PATH, and imageio-ffmpeg's bundled "
"binary); cannot use ffmpeg pipe save. Falling back to a PyAV save that skips "
"'+faststart', so saved clips may be slow to start playback over a network filesystem.")
return False
if not frames: