uvicorn's access log lands in the engine tail, and the UI polls several
endpoints every few seconds — the console was mostly its own GET 200 lines.
access_log=False at the server (applies on restart) + a client-side filter
for 2xx GET/HEAD access-log lines (immediate, and covers servers started
before the flag).
the mp executor's workers arm prctl(PR_SET_PDEATHSIG, SIGKILL), which on
linux fires when the CREATING THREAD exits — not the process. generators
spawned from short-lived threads (preload, per-job loader helpers) lose
every worker to a silent SIGKILL the moment that thread finishes: zombie
workers, broken pipes on the first rpc. all creations now run on a single
long-lived loader thread.
FastVideoArgs defaults it True, which disables FSDP and holds a full DiT
copy in host RAM per worker — 4 mp workers x 66GB DiT (+ the Qwen3-VL text
encoder) OOM-killed the node silently right after 'workers ready' (defunct
workers, broken pipes on first RPC). the example scripts always overrode
this; the studio never did.
sending the Manager-queue proxy over the mp executor's worker pipes
(set_log_queue RPC on each generate) breaks the pipe — that path was never
exercised while the queue was also passed at creation. single-slot now keeps
one runner-wide queue for the generator's whole life (drained per job), and
generate_video no longer sends log_queue at all.
- create-job modal: 480p/720p/1080p preset chips (/32-safe dims) + native
chip restoring the model preset + portrait/landscape toggle; sliders stay
- completed jobs stream their clip inline in the details sidebar (video tag
over the existing /video endpoint; png for t2i)
- top bar/nav: Jobs -> Studio (labels only)
- gpus page is now a cluster view over GET /api/cluster: per-node cards with
util/vram meters, temp/power, mode badge, available/total gpus; compact
per-node strip; 5s polling, paused while hidden
self-contained NVML probe scheduled on every alive ray node (worker envs
can't import apps/, so cloudpickle carries it by value); falls back to the
local host when not connected. includes ray gpu totals/available.
h3's denoising loop emitted no per-step progress at all, and ray wraps
relayed worker lines in ANSI color codes the bridge's prefix match never
survived — together the UI bar sat at 0 until completion. tqdm on rank 0
(mirroring the shared denoising stage; non-tty refreshes are plain lines ray
relays per step) + ANSI strip before matching/feeding.
on the ray backend worker tqdm can't reach the job buffer via the mp queue,
but ray already relays worker stdout to the driver — which the engine tee
captures. lines with ray's actor prefix now feed the running inference job's
log buffer, whose existing tqdm parser moves the UI progress bar. no UI
changes needed.
killed actors free their devices asynchronously — a load fired right after a
release could still see 0 GPUs on the node and hard-fail. shutdown now waits
(<=60s) for the GPUs to return to ray's ledger, and initialize_ray_cluster's
node-local GPU probe retries for 60s instead of failing instantly.
two races hit by real UI use:
- a job-triggered replace nulled the resident generator while state stayed
'ready', so a concurrent job with the old config grabbed None and died on
generate_video. every slot transition (preload / job replace / unload) now
runs under one load lock; jobs block behind in-flight loads instead of
peeking at half-replaced state.
- ray placement-group removal is async: a load right after a release raced
the teardown and failed 'no GPU available'. executor shutdown now waits
(<=30s) for the PG to actually reach REMOVED.
create-job modal now defaults its model to the resident one and, for any
selected model, populates every sampling field from GET /api/models/presets
(fixes generating H3 with wan-style defaults: guidance 5.0 and 81 frames both
fail H3's input validation). selecting the warm model also adopts the slot's
engine config so the job actually reuses the resident instance instead of
silently replacing it.
panel shows the one resident model (load/replace/unload, one load at a
time); engine console tails the server's stdout/stderr via /api/engine/logs
(collapsible, polls only while open, stick-to-bottom).
reload-after-unload failed because RayDistributedExecutor.shutdown() killed
the actors but never removed the placement group — the GPUs stayed reserved
forever. remove it on shutdown.
studio generator management simplified to a single slot: one VideoGenerator
instance at a time, one load in flight at a time (409 otherwise), loading a
new config always releases the old instance, unload shuts it down and
deletes it. jobs wait out an in-flight preload and reuse it on config match.
GET /api/engine/logs: incremental tail of the server's stdout/stderr (ray
relays worker output to the driver, so this captures every rank).
lists resident/loading/failed generators, preload from the model dropdown
using the persisted default job options (same cache key as jobs), unload with
409 surfaced as a toast. polls only while something is loading. mock server
+ playwright spec + vitest coverage.
generators were already cached across jobs but only implicitly — the first
job per config silently paid the whole model load. adds:
- POST /api/generators/preload: load a model into memory ahead of time
(async, idempotent; state loading/ready/failed)
- GET /api/generators: what's resident or in flight
- POST /api/generators/unload: shutdown + free VRAM (409 while inference
jobs run)
cache key builder shared with the job path and pinned by a test so preload
and job lookup can't drift apart.
RayDistributedExecutor became uninstantiable when set_log_queue/
clear_log_queue were added to the Executor ABC (only the mp executor got
implementations). No-op on ray: a Manager queue can't cross nodes.
- forward num_gpus to from_pretrained (was dropped: multi-gpu only ever
happened via sp_size)
- FASTVIDEO_STUDIO_EXECUTOR_BACKEND=ray env knob so a deployment on a ray
cluster can span nodes
- FASTVIDEO_STUDIO_MODEL_PATHS="id=/local/dir" serves a registered model id
from local weights (avoids re-downloading checkpoints already on shared fs)
- accept an existing local dir as model_id in POST /api/jobs
SolitaryThinker