[docs] Add Dreamverse app documentation
Apply Dreamverse monorepo changes for stack slice 2/13 from the source branch.
Source-Branch: will/dreamverse-monorepo
Source-SHA: 03d3e61df6
Dreamverse-Stack: 2/13
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
This commit is contained in:
Davids048
co-authored by
SolitaryThinker
Matthew Noto
XOR-op
Zhang Peiyuan
SolitaryThinker
Matthew Noto
XOR-op
Zhang Peiyuan
parent
4ad6880ca6
commit
2a8a7a01a9
@@ -0,0 +1,203 @@
|
||||
# Dreamverse Agent Notes
|
||||
|
||||
## Repo-local skills
|
||||
|
||||
- `.agents/skills/bootstrap-fastvideo-private-fork/`: temporary setup skill for
|
||||
cloning `git@github.com:hao-ai-lab/FastVideo-internal.git` at
|
||||
`will/rebase-nbv` into `../FastVideo-internal`, then running
|
||||
`uv sync --extra server`.
|
||||
- Prefer the bundled script in that skill instead of inventing a new private
|
||||
FastVideo bootstrap flow.
|
||||
|
||||
## Repo layout
|
||||
|
||||
Current paths:
|
||||
|
||||
- `apps/web/`: Next.js frontend, client-side stores, websocket event reduction,
|
||||
prompt-window editing, devtools UI.
|
||||
- `server/`: current Python FastAPI runtime, websocket protocol, prompt
|
||||
enhancement, prompt rewrite orchestration, GPU worker lifecycle.
|
||||
- `server/tests/`: backend unit and integration-oriented tests.
|
||||
- `server/benchmarks/`: prompt-provider latency/token benchmarking scripts.
|
||||
|
||||
Planned paths during the OSS reorg:
|
||||
|
||||
- `controller/`: local control plane for provider credentials, compute
|
||||
lifecycle, and proxying.
|
||||
- `runtime/`: eventual rename of `server/` once the controller/runtime split is
|
||||
stable.
|
||||
- `providers/`: provider adapters for local, Runpod, and Modal.
|
||||
|
||||
Important rule:
|
||||
|
||||
- Until the split lands, treat `server/` as the authoritative runtime and keep
|
||||
provider orchestration out of `apps/web`.
|
||||
|
||||
## System split
|
||||
|
||||
Dreamverse is moving toward a three-part local-first architecture.
|
||||
|
||||
- Frontend owns local UI state, drafts, inspection tools, and user-triggered
|
||||
actions.
|
||||
- Controller will own local-only credentials, compute provisioning, runtime
|
||||
lifecycle, and HTTP/websocket proxying.
|
||||
- Runtime owns generation sessions, prompt rewrite, prompt safety, and
|
||||
websocket semantics.
|
||||
|
||||
The browser should only talk to the local Dreamverse process, never directly to
|
||||
Modal or Runpod.
|
||||
|
||||
## Current runtime responsibilities
|
||||
|
||||
The runtime in `server/` is responsible for:
|
||||
|
||||
- websocket session lifecycle on `/ws`
|
||||
- queueing, GPU assignment, worker startup, and stream chunk emission
|
||||
- seed prompt memory and the active prompt window used for generation
|
||||
- prompt enhancement and prompt rewrite execution
|
||||
- prompt safety checks
|
||||
- persistence and reload of prompt system prompt files
|
||||
- curated preset append/read routes in devtools mode
|
||||
- health and readiness endpoints
|
||||
|
||||
Relevant files:
|
||||
|
||||
- `server/main.py`: websocket protocol, session state machine, REST routes
|
||||
- `server/gpu_pool.py`: FastVideo-backed generation workers
|
||||
- `server/prompt_enhancer.py`: provider clients, prompt enhancement, rewrite
|
||||
execution
|
||||
- `server/rewrite_prompt_payload.py`: canonical rewrite request body format
|
||||
- `server/config.py`: prompt file paths, provider configuration, runtime flags
|
||||
|
||||
## Frontend responsibilities
|
||||
|
||||
The frontend in `apps/web/` is responsible for:
|
||||
|
||||
- collecting user input and deciding whether to send raw prompts or rewrite
|
||||
requests
|
||||
- maintaining client-side stores for session, prompt-window, stream, rewrite,
|
||||
and UI state
|
||||
- rendering prompt history, playback state, devtools controls, and rewrite
|
||||
inspection
|
||||
- building the prompt-window snapshot sent with rewrite requests
|
||||
- reducing websocket events into UI state
|
||||
- showing compute status and controller-driven errors once the controller lands
|
||||
|
||||
Relevant files:
|
||||
|
||||
- `apps/web/src/app/page.tsx`: main orchestration, websocket connect/send paths
|
||||
- `apps/web/src/lib/ws/reducer.ts`: applies normalized websocket events to
|
||||
stores
|
||||
- `apps/web/src/stores/promptWindow.ts`: prompt window and preset/editor state
|
||||
- `apps/web/src/stores/rewrite.ts`: rewrite activity timeline and flags
|
||||
- `apps/web/src/lib/prompts/promptWindowSnapshot.ts`: rewrite snapshot
|
||||
normalization and padding
|
||||
|
||||
## Planned controller responsibilities
|
||||
|
||||
The future local controller should own:
|
||||
|
||||
- local-only provider credential loading and storage
|
||||
- provider selection
|
||||
- runtime provisioning, reuse, shutdown, and health checks
|
||||
- proxying frontend HTTP and websocket traffic to the active runtime
|
||||
- surfacing provisioning, ready, failed, and idle states to the frontend
|
||||
- durable local settings that should survive ephemeral remote runtimes
|
||||
|
||||
The controller should not own:
|
||||
|
||||
- prompt rewrite logic
|
||||
- seed prompt memory
|
||||
- generation queue semantics
|
||||
- websocket event schemas
|
||||
|
||||
## Prompt rewrite contract
|
||||
|
||||
Prompt rewrite is a shared flow with a strict ownership split.
|
||||
|
||||
Frontend responsibilities:
|
||||
|
||||
- decide when a user action should trigger `rewrite_seed_prompts` instead of
|
||||
`append_prompt`
|
||||
- send `rewrite_instruction` and a snapshot of the current prompt window
|
||||
- pad the rewrite snapshot to the runtime-expected segment count using
|
||||
`buildRewritePromptWindowSnapshotFromPrompts(...)`
|
||||
- show rewrite activity and raw LLM output in local inspection UI
|
||||
|
||||
Runtime responsibilities:
|
||||
|
||||
- validate and normalize `prompt_window_prompts`
|
||||
- choose the rewrite system prompt and provider/model/temperature
|
||||
- build the canonical LLM request body in `server/rewrite_prompt_payload.py`
|
||||
- run the rewrite through `PromptEnhancer.rewrite_prompt_sequence(...)`
|
||||
- apply safety filtering to rewritten prompts
|
||||
- replace the authoritative seed prompt memory when rewrite succeeds
|
||||
- emit `seed_prompts_updated` and `rewrite_seed_prompts_complete`
|
||||
|
||||
Controller responsibilities:
|
||||
|
||||
- proxy the request and response
|
||||
- surface runtime availability and provider lifecycle failures
|
||||
|
||||
Important rule:
|
||||
|
||||
- The frontend may suggest the prompt window to rewrite, but the runtime owns
|
||||
the actual rewritten rollout and the authoritative prompt window after
|
||||
acceptance.
|
||||
|
||||
## Rewrite modes
|
||||
|
||||
There are two runtime rewrite modes:
|
||||
|
||||
- edit existing rollout: when `prompt_window_prompts` is non-empty, rewrite the
|
||||
current rollout while preserving segment count and ordering
|
||||
- new rollout: when the prompt window is empty but there is a
|
||||
`rewrite_instruction`, generate a fresh rollout
|
||||
|
||||
The frontend should not emulate runtime rewrite behavior locally. It should
|
||||
prepare the snapshot, send it, and display the result.
|
||||
|
||||
## Prompt window ownership
|
||||
|
||||
- Frontend owns editable drafts, selected preset UI, and prompt-window
|
||||
inspection state.
|
||||
- Runtime owns the active seed prompt memory used for actual generation.
|
||||
- After any runtime event with reason `rewrite`, the frontend must replace its
|
||||
prompt window from the server payload instead of keeping a locally-derived
|
||||
version.
|
||||
|
||||
## Devtools and persistence ownership
|
||||
|
||||
Prompt config editing is runtime-owned persistence with frontend-owned forms
|
||||
today.
|
||||
|
||||
- Frontend loads and edits drafts through `/prompt-system-config`.
|
||||
- Runtime reads and writes prompt files and reloads runtime prompt config.
|
||||
|
||||
Curated presets follow the same pattern:
|
||||
|
||||
- frontend submits append requests and may update local UI optimistically from
|
||||
the response
|
||||
- runtime persists the JSON file and resolves overlay vs fallback file paths
|
||||
|
||||
During the controller reorg, avoid moving durable user settings into ephemeral
|
||||
remote runtimes. Controller-owned local persistence is preferred for anything
|
||||
that must survive provider restarts.
|
||||
|
||||
## Editing guidance
|
||||
|
||||
- Do not move rewrite logic into the frontend or controller.
|
||||
- Do not make the frontend the source of truth for the generated prompt window
|
||||
after rewrite.
|
||||
- If you change websocket message types or payload fields in `server/main.py`,
|
||||
update the reducer in `apps/web/src/lib/ws/reducer.ts` in the same change.
|
||||
- If you change rewrite request shape, update both
|
||||
`apps/web/src/lib/prompts/promptWindowSnapshot.ts` and
|
||||
`server/rewrite_prompt_payload.py`.
|
||||
- If you add controller-managed status or error payloads, keep them separate
|
||||
from runtime websocket events unless there is a strong reason to merge them.
|
||||
- If you change prompt file paths or devtools persistence, update `server/`,
|
||||
the frontend devtools UI, and any controller-owned local persistence logic
|
||||
together.
|
||||
- Keep provider adapters focused on runtime lifecycle and reachability, not on
|
||||
prompt or session semantics.
|
||||
@@ -0,0 +1,234 @@
|
||||
# Dreamverse
|
||||
|
||||
Dreamverse is the FastVideo realtime video generation & editing platform. It lives in this monorepo under `apps/dreamverse/`.
|
||||
|
||||
## Install Dreamverse
|
||||
|
||||
You can install Dreamverse using one of the methods below.
|
||||
|
||||
### Method 1: With uv pip
|
||||
|
||||
```bash
|
||||
pip install --upgrade pip
|
||||
pip install uv
|
||||
uv venv .venv --python 3.12
|
||||
source .venv/bin/activate
|
||||
uv pip install "fastvideo[dreamverse]"
|
||||
```
|
||||
|
||||
### Method 2: From source
|
||||
|
||||
```bash
|
||||
git clone https://github.com/hao-ai-lab/FastVideo.git
|
||||
cd FastVideo
|
||||
|
||||
pip install --upgrade pip
|
||||
pip install uv
|
||||
uv venv .venv --python 3.12
|
||||
source .venv/bin/activate
|
||||
uv pip install -e ".[dreamverse]"
|
||||
```
|
||||
|
||||
### Method 3: Using Docker
|
||||
|
||||
```bash
|
||||
git clone https://github.com/hao-ai-lab/FastVideo.git
|
||||
cd FastVideo
|
||||
|
||||
apps/dreamverse/docker/docker_build.sh
|
||||
```
|
||||
|
||||
See `apps/dreamverse/docker/README.md` for Docker build and run option details.
|
||||
|
||||
## Optional: Building FFmpeg For Better Performance
|
||||
|
||||
For full streaming performance in a non-Docker install, build a custom FFmpeg
|
||||
binary:
|
||||
|
||||
```bash
|
||||
bash apps/dreamverse/scripts/install_native_ffmpeg.sh
|
||||
```
|
||||
|
||||
This builds and installs into `~/opt/ffmpeg-native/` and writes
|
||||
`apps/dreamverse/scripts/ffmpeg-env.sh`. Source it before starting the backend
|
||||
so Dreamverse uses the custom FFmpeg binary:
|
||||
|
||||
```bash
|
||||
source apps/dreamverse/scripts/ffmpeg-env.sh
|
||||
dreamverse-server
|
||||
```
|
||||
|
||||
Docker images already run this FFmpeg build during image creation and source the
|
||||
generated environment file at container startup. The installer supports Linux
|
||||
`x86_64` and `aarch64`.
|
||||
|
||||
## Launch Dreamverse
|
||||
|
||||
Start the backend with the installed Dreamverse commands:
|
||||
|
||||
```bash
|
||||
dreamverse-server --port 8009
|
||||
dreamverse-mock-server --port 8009
|
||||
```
|
||||
|
||||
## Frontend Setup
|
||||
|
||||
Install the web dependencies once from the FastVideo checkout:
|
||||
|
||||
```bash
|
||||
cd apps/dreamverse/web
|
||||
pnpm install --frozen-lockfile
|
||||
```
|
||||
|
||||
The frontend package also has an npm lockfile, but the bundled launch scripts
|
||||
use `pnpm`.
|
||||
|
||||
## Quick Start: Local GPU
|
||||
|
||||
### Start Backend
|
||||
|
||||
Export the API keys used for prompt rewrite and prompt enhancement:
|
||||
|
||||
```bash
|
||||
export CEREBRAS_API_KEY=...
|
||||
export GROQ_API_KEY=...
|
||||
```
|
||||
|
||||
If you built the optional native FFmpeg binary above, source its environment
|
||||
file in the same shell before starting the backend:
|
||||
|
||||
```bash
|
||||
source apps/dreamverse/scripts/ffmpeg-env.sh
|
||||
dreamverse-server --host 0.0.0.0 --port 8009
|
||||
```
|
||||
|
||||
The Dreamverse backend defaults to `0.0.0.0:8009` and starts one GPU worker on
|
||||
the first visible GPU by default.
|
||||
|
||||
### Check Readiness
|
||||
|
||||
In another shell, verify that the backend process is alive:
|
||||
|
||||
```bash
|
||||
curl http://localhost:8009/healthz
|
||||
```
|
||||
|
||||
Then wait for GPU workers and startup warmup to finish:
|
||||
|
||||
```bash
|
||||
curl http://localhost:8009/readyz
|
||||
```
|
||||
|
||||
You can also run the same readiness path with:
|
||||
|
||||
```bash
|
||||
BACKEND_HOST=localhost BACKEND_PORT=8009 apps/dreamverse/scripts/smoke_local.sh
|
||||
```
|
||||
|
||||
If a backend is already running and you only want the script to probe it:
|
||||
|
||||
```bash
|
||||
DREAMVERSE_SMOKE_START_BACKEND=0 apps/dreamverse/scripts/smoke_local.sh
|
||||
```
|
||||
|
||||
### Start Frontend
|
||||
|
||||
Start the frontend:
|
||||
|
||||
```bash
|
||||
cd apps/dreamverse/web
|
||||
BACKEND_HOST=localhost BACKEND_PORT=8009 pnpm run dev
|
||||
```
|
||||
|
||||
Open `http://localhost:5299`.
|
||||
|
||||
## Quick Start: Mock Backend (For UI development)
|
||||
|
||||
The mock server emulates the Dreamverse backend protocol and streams a
|
||||
synthetic FFmpeg-generated fMP4 clip, so the frontend can run without a GPU.
|
||||
|
||||
```bash
|
||||
dreamverse-mock-server --latency 200 --port 8009
|
||||
```
|
||||
|
||||
## Tests
|
||||
|
||||
Run the focused backend tests that validate local startup wiring, config, GPU
|
||||
selection, and mock-server behavior:
|
||||
|
||||
```bash
|
||||
pytest apps/dreamverse/dreamverse/tests/test_config.py \
|
||||
apps/dreamverse/dreamverse/tests/test_entrypoints.py \
|
||||
apps/dreamverse/dreamverse/tests/test_gpu_pool.py \
|
||||
apps/dreamverse/dreamverse/tests/test_mock_server.py -q
|
||||
```
|
||||
|
||||
Run the broader Dreamverse backend suite:
|
||||
|
||||
```bash
|
||||
pytest apps/dreamverse/dreamverse/tests -q
|
||||
```
|
||||
|
||||
Run the frontend tests:
|
||||
|
||||
```bash
|
||||
cd apps/dreamverse/web
|
||||
pnpm test
|
||||
```
|
||||
|
||||
Run the frontend e2e tests:
|
||||
|
||||
```bash
|
||||
cd apps/dreamverse/web
|
||||
pnpm run e2e
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
`dreamverse-server` exits with an install hint
|
||||
|
||||
- install the Dreamverse extra with `uv pip install -e ".[dreamverse]"` from a
|
||||
source checkout, or `uv pip install "fastvideo[dreamverse]"` from PyPI.
|
||||
|
||||
Prompt-provider environment variable errors
|
||||
|
||||
- set `CEREBRAS_API_KEY`
|
||||
- set `GROQ_API_KEY`
|
||||
- direct `dreamverse-server` launches do not source `~/.env`; export the keys
|
||||
in the shell or use the bundled launch scripts, which source `~/.env`.
|
||||
|
||||
`/readyz` stays at `503`
|
||||
|
||||
- wait for model loading and startup warmup to finish
|
||||
- confirm a compatible CUDA GPU is visible to the process
|
||||
- check backend logs for worker startup or warmup failures
|
||||
- for a startup/debug pass without warmup, set
|
||||
`FASTVIDEO_ENABLE_STARTUP_WARMUP=0` before starting the backend
|
||||
|
||||
Only one GPU is used
|
||||
|
||||
- this is the default local behavior
|
||||
- set `FASTVIDEO_GPU_COUNT=<N>` to start N GPU worker subprocesses inside one
|
||||
backend instance
|
||||
- set `FASTVIDEO_GPU_COUNT=all` to start one worker for every visible GPU
|
||||
- use `CUDA_VISIBLE_DEVICES` first if you need to pin the visible GPU set
|
||||
|
||||
Frontend cannot connect to backend
|
||||
|
||||
- confirm the backend is running on `8009`; if not, point the frontend at it
|
||||
with `BACKEND_HOST=<host> BACKEND_PORT=<port> pnpm run dev`
|
||||
- confirm `http://localhost:8009/healthz` responds before starting the frontend
|
||||
- confirm `http://localhost:8009/readyz` returns `200` before clicking Generate
|
||||
- use `apps/dreamverse/scripts/smoke_local.sh` for a repeatable local startup
|
||||
check
|
||||
|
||||
Mock backend fails during startup
|
||||
|
||||
- install FFmpeg or set `FASTVIDEO_FFMPEG_BIN` to an FFmpeg binary
|
||||
- for non-mock local GPU streaming performance, use the native FFmpeg installer
|
||||
described above
|
||||
|
||||
## Notes
|
||||
|
||||
Dreamverse owns its backend app under `apps/dreamverse/dreamverse/`. It expects
|
||||
`dreamverse-server`, not `fastvideo serve`.
|
||||
@@ -0,0 +1,351 @@
|
||||
# Dreamverse Architecture
|
||||
|
||||
## Overview
|
||||
|
||||
Dreamverse currently has two main runtime pieces:
|
||||
|
||||
- `apps/web/`: Next.js frontend
|
||||
- `server/`: Python FastAPI runtime
|
||||
|
||||
Today, the browser talks directly to the Dreamverse runtime over HTTP and a
|
||||
single websocket on `/ws`. The frontend owns UI state and interaction flow. The
|
||||
server owns generation state, prompt rewrite, prompt safety, websocket session
|
||||
semantics, and GPU-backed execution.
|
||||
|
||||
Near-term OSS note:
|
||||
|
||||
- `server/` is the current runtime implementation.
|
||||
- A future `controller/` layer is planned for local-only compute management and
|
||||
provider orchestration, but it does not exist yet.
|
||||
|
||||
## Repo Map
|
||||
|
||||
### Frontend
|
||||
|
||||
- `apps/web/src/app/page.tsx`: main client orchestration, websocket connect,
|
||||
init payloads, send paths, and top-level app behavior
|
||||
- `apps/web/src/lib/ws/reducer.ts`: reduces normalized websocket events into
|
||||
client stores
|
||||
- `apps/web/src/stores/session.ts`: connection, mode, and top-level session UI
|
||||
- `apps/web/src/stores/promptWindow.ts`: editable prompt window and seed prompt
|
||||
UI state
|
||||
- `apps/web/src/stores/rewrite.ts`: rewrite activity timeline and inspection
|
||||
state
|
||||
- `apps/web/src/stores/stream.ts`: playback and stream-related client state
|
||||
- `apps/web/src/lib/prompts/promptWindowSnapshot.ts`: prompt-window snapshot
|
||||
building for rewrite requests
|
||||
|
||||
### Server
|
||||
|
||||
- `server/main.py`: websocket endpoint, request handling, session state
|
||||
machine, rewrite orchestration, REST routes, and stream relay
|
||||
- `server/gpu_pool.py`: GPU worker processes, warmup, model loading, and
|
||||
`generate_video()` calls through FastVideo
|
||||
- `server/prompt_enhancer.py`: prompt enhancement, rollout rewrite execution,
|
||||
provider selection, and timeout/fallback behavior
|
||||
- `server/rewrite_prompt_payload.py`: canonical rewrite request payload
|
||||
building
|
||||
- `server/config.py`: runtime flags, prompt file paths, provider settings, and
|
||||
warmup config
|
||||
- `server/session_init_image.py`: validates and persists uploaded initial
|
||||
images for segment 1
|
||||
|
||||
## Current Split Of Responsibility
|
||||
|
||||
### Frontend owns
|
||||
|
||||
- local UI state and client-side stores
|
||||
- prompt drafts and prompt window editing
|
||||
- websocket connection management
|
||||
- deciding which user action to send:
|
||||
- `session_init_v2`
|
||||
- `project_init_v1`
|
||||
- `append_prompt`
|
||||
- `rewrite_seed_prompts`
|
||||
- `simple_generate`
|
||||
- showing rewrite progress, stream status, prompt history, and devtools views
|
||||
|
||||
### Server owns
|
||||
|
||||
- websocket session lifecycle and protocol
|
||||
- GPU assignment and worker lifecycle
|
||||
- the authoritative seed prompt memory used for generation
|
||||
- prompt rewrite execution and prompt safety
|
||||
- actual generation queue semantics
|
||||
- stream chunk emission and segment lifecycle events
|
||||
- prompt config and preset persistence routes
|
||||
- health and readiness endpoints
|
||||
|
||||
Important rule:
|
||||
|
||||
- The frontend may propose prompt-window state for rewrite, but the server is
|
||||
the source of truth for the rewritten rollout and the active prompt memory
|
||||
used for generation.
|
||||
|
||||
## End-To-End Flow
|
||||
|
||||
1. The frontend opens `/ws`.
|
||||
2. The frontend sends `session_init_v2` with the initial prompt-window state,
|
||||
preset metadata, and current toggles.
|
||||
3. The server validates init data, persists an optional initial image, acquires
|
||||
a GPU slot, and emits session status such as `gpu_assigned`.
|
||||
4. The server starts or resumes project generation and emits events like
|
||||
`ltx2_stream_start`, `ltx2_segment_start`, media init/chunks, and completion
|
||||
events.
|
||||
5. The frontend reduces those websocket events into its stores and updates the
|
||||
UI.
|
||||
6. User actions such as appending prompts, rewriting seed prompts, or starting
|
||||
a single custom clip go back to the server over the same websocket.
|
||||
|
||||
## Frontend Architecture
|
||||
|
||||
The frontend is store-driven.
|
||||
|
||||
- `page.tsx` wires together websocket setup, send helpers, reducer
|
||||
application, and top-level interaction flows.
|
||||
- Store modules separate concerns like session state, rewrite state, prompt
|
||||
window state, and stream state.
|
||||
- The websocket reducer is responsible for turning normalized runtime events
|
||||
into store updates. If the server event schema changes, the reducer must
|
||||
change with it.
|
||||
|
||||
The frontend is intentionally not responsible for:
|
||||
|
||||
- generating rewritten prompts locally
|
||||
- deciding final prompt safety outcomes
|
||||
- reconstructing server session state from scratch
|
||||
- inventing its own generation semantics independent of the runtime
|
||||
|
||||
## Server Architecture
|
||||
|
||||
The current runtime is a FastAPI app with a single long-lived websocket per
|
||||
session.
|
||||
|
||||
`server/main.py` manages:
|
||||
|
||||
- websocket connect/init
|
||||
- prompt queues
|
||||
- project and segment state
|
||||
- prompt enhancement/rewrite triggers
|
||||
- stream relay from GPU workers to the browser
|
||||
- session logging and REST endpoints
|
||||
|
||||
`server/gpu_pool.py` manages:
|
||||
|
||||
- model loading through FastVideo
|
||||
- one or more worker processes
|
||||
- startup warmup
|
||||
- user join/leave commands
|
||||
- `USER_STEP` execution for each segment
|
||||
- continuation state between segments
|
||||
|
||||
`server/prompt_enhancer.py` manages:
|
||||
|
||||
- prompt enhancement for user-submitted prompts
|
||||
- rollout rewrite requests for the prompt window
|
||||
- provider selection and fallback across configured prompt providers
|
||||
- response normalization and safety-aware failure handling
|
||||
|
||||
## FastAPI Surface
|
||||
|
||||
The server is a single FastAPI application created in
|
||||
`server/main.py`.
|
||||
|
||||
Current built-in FastAPI docs are enabled:
|
||||
|
||||
- `/docs`: Swagger UI
|
||||
- `/redoc`: ReDoc
|
||||
- `/openapi.json`: OpenAPI schema
|
||||
|
||||
The runtime also mounts static content:
|
||||
|
||||
- `/server-assets`: server-local static assets rooted at `server/`
|
||||
- `/`: frontend static build when one of the configured frontend static
|
||||
directories exists
|
||||
|
||||
## HTTP API
|
||||
|
||||
The current HTTP API is small. Most realtime behavior still goes through the
|
||||
websocket.
|
||||
|
||||
### Core health and status routes
|
||||
|
||||
- `GET /healthz`
|
||||
- process liveness probe
|
||||
- returns a small payload with `status`, `service`, and timestamp
|
||||
- `GET /readyz`
|
||||
- readiness probe
|
||||
- returns `503` until prompt services are initialized and at least one GPU
|
||||
worker is ready
|
||||
- returns readiness and GPU pool summary fields such as ready workers, total
|
||||
GPUs, warmup counts, and queue size
|
||||
- `GET /status`
|
||||
- returns the current GPU pool status payload from `gpu_pool`
|
||||
- `GET /internal/monitor/sessions`
|
||||
- internal monitoring payload for session dashboards
|
||||
- includes pending session count, max available sessions, prompt provider
|
||||
success counts, and timestamp
|
||||
|
||||
### Prompt config routes
|
||||
|
||||
- `GET /prompt-system-config`
|
||||
- returns the editable prompt-system configuration currently loaded by
|
||||
`PromptEnhancer`
|
||||
- `POST /prompt-system-config`
|
||||
- saves prompt-system configuration to disk and reloads prompt config in the
|
||||
runtime
|
||||
- current editable fields include:
|
||||
- next-segment system prompt
|
||||
- auto-extension system prompt
|
||||
- rewrite-window system prompt
|
||||
- rewrite-user system prompt
|
||||
- rewrite model
|
||||
- rewrite temperature
|
||||
|
||||
### Devtools-only preset routes
|
||||
|
||||
These exist only when `DEVTOOLS_ENABLED` is true in `server/config.py`.
|
||||
|
||||
- `GET /curated-presets`
|
||||
- returns merged curated presets, applying the local overlay file on top of
|
||||
the fallback file when both exist
|
||||
- `POST /curated-presets/append`
|
||||
- appends a new curated preset to the overlay presets file
|
||||
- validates non-empty label, normalized id, and at least two non-empty
|
||||
segment prompts
|
||||
|
||||
## Websocket API
|
||||
|
||||
`WS /ws` is the main runtime API.
|
||||
|
||||
The websocket owns:
|
||||
|
||||
- session init
|
||||
- project init and reset
|
||||
- prompt append
|
||||
- prompt rewrite
|
||||
- generation toggles
|
||||
- segment lifecycle events
|
||||
- media stream delivery
|
||||
- runtime error delivery
|
||||
|
||||
The websocket is the authoritative API for realtime Dreamverse behavior. The
|
||||
HTTP routes mainly support health checks, devtools persistence, and monitoring.
|
||||
|
||||
## Prompt Rewrite Architecture
|
||||
|
||||
Prompt rewrite is a shared flow with strict ownership boundaries.
|
||||
|
||||
### Frontend responsibilities
|
||||
|
||||
- collect the rewrite instruction
|
||||
- build the prompt-window snapshot from current client state
|
||||
- send `rewrite_seed_prompts`
|
||||
- show rewrite progress, raw output, fallback state, and resulting prompt list
|
||||
|
||||
### Server responsibilities
|
||||
|
||||
- validate and normalize the prompt-window payload
|
||||
- choose rewrite model, system prompt, timeout, and temperature
|
||||
- build the canonical prompt payload in `server/rewrite_prompt_payload.py`
|
||||
- execute rewrite through `PromptEnhancer`
|
||||
- apply safety filtering to rewritten prompts
|
||||
- replace the authoritative seed prompt memory when rewrite succeeds
|
||||
- emit `seed_prompts_updated` and `rewrite_seed_prompts_complete`
|
||||
|
||||
Important rule:
|
||||
|
||||
- The frontend owns editable drafts.
|
||||
- The server owns the accepted rollout.
|
||||
|
||||
After a successful rewrite, the frontend should replace its prompt-window view
|
||||
from the server payload instead of preserving a locally-derived version.
|
||||
|
||||
## Prompt Modes
|
||||
|
||||
There are three related prompt paths in the current system:
|
||||
|
||||
### Initial rollout
|
||||
|
||||
- The frontend sends seed prompts during `session_init_v2`.
|
||||
- The server uses those prompts as the initial seed prompt memory.
|
||||
- If the rollout starts from an empty prompt window plus an initial rewrite
|
||||
instruction, the server can pause generation until rewrite completes.
|
||||
|
||||
### Live append
|
||||
|
||||
- The frontend sends `append_prompt`.
|
||||
- The server may enhance that prompt, safety-check it, enqueue it, and use it
|
||||
as the next generated segment.
|
||||
|
||||
### Rewrite
|
||||
|
||||
- The frontend sends `rewrite_seed_prompts`.
|
||||
- The server rewrites the entire seed prompt window or generates a new rollout,
|
||||
depending on the payload and current state.
|
||||
|
||||
## Initial Image And Segment Handling
|
||||
|
||||
The frontend currently sends `initial_image` as part of session init or
|
||||
`simple_generate`.
|
||||
|
||||
The server:
|
||||
|
||||
- validates and persists the image
|
||||
- uses it only for segment 1 when present
|
||||
- keeps continuation state for later segments in the GPU worker
|
||||
|
||||
This means the runtime, not the frontend, decides how segment 1 image
|
||||
conditioning and later continuation conditioning are applied.
|
||||
|
||||
## Websocket Contract
|
||||
|
||||
The websocket is the main integration surface between UI and runtime.
|
||||
|
||||
Typical incoming messages from the frontend:
|
||||
|
||||
- `session_init_v2`
|
||||
- `project_init_v1`
|
||||
- `append_prompt`
|
||||
- `rewrite_seed_prompts`
|
||||
- `simple_generate`
|
||||
- `set_enhancement`
|
||||
- `set_auto_extension`
|
||||
- `set_loop_generation`
|
||||
|
||||
Typical outgoing messages from the server:
|
||||
|
||||
- `gpu_assigned`
|
||||
- `ltx2_stream_start`
|
||||
- `ltx2_segment_start`
|
||||
- `segment_prompt_source`
|
||||
- `prompt_received`
|
||||
- `prompt_ready`
|
||||
- `prompt_enhancing`
|
||||
- `seed_prompts_updated`
|
||||
- `rewrite_seed_prompts_complete`
|
||||
- `media_init`
|
||||
- `media_segment_complete`
|
||||
- `project_idle`
|
||||
- `error`
|
||||
|
||||
Binary websocket frames carry media chunks for playback.
|
||||
|
||||
## Current And Planned Architecture
|
||||
|
||||
Current architecture:
|
||||
|
||||
- browser -> `apps/web`
|
||||
- `apps/web` -> `server/main.py`
|
||||
- `server/main.py` -> `gpu_pool.py`
|
||||
- `gpu_pool.py` -> FastVideo runtime
|
||||
|
||||
Planned architecture:
|
||||
|
||||
- browser -> `apps/web`
|
||||
- `apps/web` -> local `controller/`
|
||||
- `controller/` -> local or remote Dreamverse runtime
|
||||
- runtime -> FastVideo runtime
|
||||
|
||||
That future controller split should not move prompt rewrite, session state, or
|
||||
generation semantics out of the runtime.
|
||||
@@ -0,0 +1,594 @@
|
||||
# Dreamverse OSS Design
|
||||
|
||||
## Overview
|
||||
|
||||
Dreamverse should ship as a local-first open source application.
|
||||
|
||||
- The browser talks only to a local Dreamverse control plane on the user's
|
||||
machine.
|
||||
- Provider credentials stay local to that machine.
|
||||
- Dreamverse may provision compute on the user's behalf, but Dreamverse does
|
||||
not host that control path as a service.
|
||||
|
||||
This keeps the UX simple without turning Dreamverse into a credential-holding
|
||||
hosted platform.
|
||||
|
||||
## Goals
|
||||
|
||||
- Support three compute modes behind one product surface:
|
||||
- local GPU
|
||||
- managed remote GPU via Runpod
|
||||
- managed remote GPU via Modal
|
||||
- Keep prompt rewrite, websocket session state, and generation behavior
|
||||
consistent across providers.
|
||||
- Keep provider API keys out of browser state and out of any hosted service.
|
||||
- Make the existing runtime reusable as the common serving contract.
|
||||
- Minimize provider-specific code and isolate it behind a narrow interface.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- Do not make the frontend call provider APIs directly.
|
||||
- Do not unify providers at the level of SSH, VM, serverless, or pod
|
||||
semantics.
|
||||
- Do not move prompt rewrite logic into the frontend or controller.
|
||||
- Do not require remote compute for the basic product path.
|
||||
|
||||
## Current State
|
||||
|
||||
Today the repo contains two major pieces:
|
||||
|
||||
- `apps/web/`: Next.js frontend
|
||||
- `server/`: FastAPI runtime that owns websocket state, prompt rewrite, prompt
|
||||
safety, and GPU-backed generation
|
||||
|
||||
The current runtime already exposes useful health and streaming surfaces such
|
||||
as `/healthz`, `/readyz`, `/status`, and `/ws`.
|
||||
|
||||
## Target Architecture
|
||||
|
||||
The target open source structure should be:
|
||||
|
||||
```text
|
||||
Dreamverse/
|
||||
├── apps/
|
||||
│ └── web/ # browser UI
|
||||
├── controller/ # local control plane and provider lifecycle
|
||||
├── runtime/ # Dreamverse websocket/generation runtime
|
||||
├── providers/ # provider adapters
|
||||
├── tests/
|
||||
│ ├── contract/
|
||||
│ ├── controller/
|
||||
│ └── smoke/
|
||||
└── design.md
|
||||
```
|
||||
|
||||
Near-term note:
|
||||
|
||||
- `server/` is the current runtime implementation.
|
||||
- We can keep the code there initially and rename it to `runtime/` only after
|
||||
the controller lands.
|
||||
|
||||
## Trust Model
|
||||
|
||||
Dreamverse is local-only for control and secrets.
|
||||
|
||||
- The user launches Dreamverse on their own machine.
|
||||
- Provider API keys are entered into the local app or local CLI.
|
||||
- The controller uses those credentials to provision or connect to compute.
|
||||
- The browser never talks to Modal or Runpod directly.
|
||||
- Dreamverse-hosted infrastructure is not involved.
|
||||
|
||||
This is the key reason the provider-based path is acceptable for OSS.
|
||||
|
||||
## Responsibility Split
|
||||
|
||||
### `apps/web`
|
||||
|
||||
The frontend should own:
|
||||
|
||||
- UI state, drafts, and local interaction state
|
||||
- websocket event reduction into client stores
|
||||
- selection of compute mode and display of cost/health/status
|
||||
- local forms for provider configuration
|
||||
- sending prompt requests and rewrite requests to the local controller
|
||||
|
||||
The frontend should not own:
|
||||
|
||||
- provider credentials after submission
|
||||
- provider API calls
|
||||
- runtime lifecycle
|
||||
- authoritative prompt window after rewrite
|
||||
- prompt safety or generation policy
|
||||
|
||||
### `controller`
|
||||
|
||||
The local controller should own:
|
||||
|
||||
- provider credential loading and local-only storage
|
||||
- compute mode selection
|
||||
- provisioning, reuse, shutdown, and health monitoring of runtimes
|
||||
- reverse proxying HTTP and websocket traffic from the frontend to the active
|
||||
runtime
|
||||
- user-visible status such as provisioning, ready, failed, and idle shutdown
|
||||
- local persistence for user settings that must survive ephemeral runtimes
|
||||
|
||||
The controller should not own:
|
||||
|
||||
- prompt rewrite logic
|
||||
- seed prompt memory semantics
|
||||
- generation queue behavior
|
||||
- provider-specific UI state
|
||||
|
||||
### `runtime`
|
||||
|
||||
The runtime should remain the authoritative owner of:
|
||||
|
||||
- `/ws` session state
|
||||
- prompt rewrite execution
|
||||
- prompt safety
|
||||
- seed prompt memory and prompt-window state used for generation
|
||||
- generation orchestration and GPU worker lifecycle
|
||||
- websocket event schemas
|
||||
|
||||
This preserves the current model and avoids splitting state across layers.
|
||||
|
||||
## Runtime Contract
|
||||
|
||||
Provider abstraction should happen around a stable Dreamverse runtime contract,
|
||||
not around infrastructure details.
|
||||
|
||||
Minimum runtime surface:
|
||||
|
||||
- `GET /healthz`
|
||||
- `GET /readyz`
|
||||
- `GET /status`
|
||||
- `GET/POST /prompt-system-config` if devtools persists config through the
|
||||
runtime
|
||||
- curated preset routes if those remain runtime-backed
|
||||
- `WS /ws`
|
||||
|
||||
Important rule:
|
||||
|
||||
- The controller only needs to know how to reach a healthy runtime.
|
||||
- The runtime remains provider-agnostic.
|
||||
|
||||
## Provider Abstraction
|
||||
|
||||
Use a narrow provider interface:
|
||||
|
||||
```python
|
||||
class ComputeProvider(Protocol):
|
||||
async def ensure_runtime(self, spec: RuntimeSpec) -> RuntimeHandle: ...
|
||||
async def wait_until_ready(self, handle: RuntimeHandle) -> None: ...
|
||||
async def stop_runtime(self, handle: RuntimeHandle) -> None: ...
|
||||
```
|
||||
|
||||
`RuntimeHandle` should include:
|
||||
|
||||
- `provider`
|
||||
- `runtime_id`
|
||||
- `base_url`
|
||||
- `ws_url`
|
||||
- runtime auth headers or tokens if needed
|
||||
- lifecycle metadata
|
||||
- cost or hardware metadata for UI display
|
||||
|
||||
The controller should work only with `RuntimeHandle`, never with raw SSH hosts
|
||||
or provider-specific payloads after resolution.
|
||||
|
||||
## Provider Notes
|
||||
|
||||
### Local
|
||||
|
||||
Local mode should be the reference implementation.
|
||||
|
||||
- Start the runtime as a local subprocess or connect to an already-running
|
||||
local runtime URL.
|
||||
- Reuse the same runtime contract as remote providers.
|
||||
- Make this the first supported path and the main smoke-test target.
|
||||
|
||||
### Runpod
|
||||
|
||||
Runpod should be treated as pod lifecycle plus runtime reachability.
|
||||
|
||||
- Prefer prepared images or templates that auto-start the Dreamverse runtime.
|
||||
- Prefer exposed HTTP/TCP ports for steady-state traffic.
|
||||
- Use SSH only for bootstrap fallback, diagnostics, or repair.
|
||||
- Avoid a design where the controller shells into the pod for every action.
|
||||
|
||||
### Modal
|
||||
|
||||
Modal should be treated as deployment-based runtime hosting.
|
||||
|
||||
- Wrap the Dreamverse runtime in a thin Modal entrypoint if needed.
|
||||
- Reuse the same runtime behavior behind that wrapper.
|
||||
- Do not model Modal as a machine that Dreamverse logs into.
|
||||
- Do not force the websocket runtime into a per-request serverless handler
|
||||
shape.
|
||||
|
||||
## Config and Persistence
|
||||
|
||||
Remote compute may be ephemeral, so mutable user configuration should not live
|
||||
only inside remote runtimes.
|
||||
|
||||
Keep durable state local to the user's machine unless there is a strong reason
|
||||
otherwise:
|
||||
|
||||
- provider selection
|
||||
- provider credentials or credential references
|
||||
- default hardware preferences
|
||||
- editable prompt presets
|
||||
- prompt system prompt overrides
|
||||
- idle shutdown policy
|
||||
|
||||
Runtime-local state should be treated as disposable unless explicitly synced.
|
||||
|
||||
## Prompt Rewrite Ownership
|
||||
|
||||
Prompt rewrite remains runtime-owned even after the controller is added.
|
||||
|
||||
Frontend responsibilities:
|
||||
|
||||
- collect the rewrite instruction
|
||||
- build the prompt-window snapshot
|
||||
- display rewrite activity and results
|
||||
|
||||
Runtime responsibilities:
|
||||
|
||||
- validate and normalize the prompt window
|
||||
- choose the rewrite system prompt and model settings
|
||||
- execute rewrite
|
||||
- apply safety filtering
|
||||
- replace authoritative seed prompt memory
|
||||
- emit the canonical completion events
|
||||
|
||||
Controller responsibilities:
|
||||
|
||||
- proxy the request and response
|
||||
- surface runtime availability and failure state
|
||||
|
||||
This boundary should not move.
|
||||
|
||||
## Recommended Rollout
|
||||
|
||||
1. Finish the path reorg so docs and code agree on `apps/web`.
|
||||
2. Introduce `controller/` as a local-only API/proxy process.
|
||||
3. Keep `server/` as the runtime and adapt it behind the controller.
|
||||
4. Add `local` provider first.
|
||||
5. Add "bring your own runtime URL" as an escape hatch.
|
||||
6. Add automated Runpod provisioning.
|
||||
7. Add Modal deployment support.
|
||||
8. Rename `server/` to `runtime/` once the split is stable.
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
The implementation should start with the smallest milestone that gives users a
|
||||
working local GPU setup without forcing the controller/provider architecture
|
||||
into the first patch series.
|
||||
|
||||
### Milestone 0: Make local GPU the official baseline
|
||||
|
||||
Goal:
|
||||
|
||||
- A user with a working `fastvideo` install can run the Dreamverse backend on a
|
||||
local GPU and connect to it from `apps/web`.
|
||||
|
||||
Non-goals for this milestone:
|
||||
|
||||
- no controller process yet
|
||||
- no provider abstraction yet
|
||||
- no Runpod or Modal support yet
|
||||
- no secret-management UI yet
|
||||
|
||||
Reasoning:
|
||||
|
||||
- `server/` already is the real local GPU runtime.
|
||||
- `apps/web` already knows how to talk to a backend over `/ws` and REST
|
||||
rewrites.
|
||||
- The shortest path is to make the existing local path explicit, reliable, and
|
||||
tested before adding another layer.
|
||||
|
||||
### Milestone 0 work items
|
||||
|
||||
#### 0.1 Fix repo path assumptions after the frontend move
|
||||
|
||||
Current issue:
|
||||
|
||||
- Some paths still assume `prod-ui/`, but the frontend now lives at
|
||||
`apps/web/`.
|
||||
|
||||
Required changes:
|
||||
|
||||
- update prompt/preset path resolution in `server/config.py`
|
||||
- update docs that still mention `prod-ui`
|
||||
- audit any frontend build settings that assume the old repo root
|
||||
|
||||
This is prerequisite cleanup. Local GPU mode should not depend on stale
|
||||
monorepo paths.
|
||||
|
||||
#### 0.2 Make local runtime startup the primary supported entrypoint
|
||||
|
||||
Required outcome:
|
||||
|
||||
- one documented backend command
|
||||
- one documented frontend command
|
||||
- one clear env contract for local development
|
||||
|
||||
Expected shape:
|
||||
|
||||
```bash
|
||||
uv pip install -e ".[dreamverse]"
|
||||
dreamverse-server --host 0.0.0.0 --port 8009
|
||||
|
||||
cd apps/dreamverse/web
|
||||
npm ci
|
||||
BACKEND_HOST=localhost BACKEND_PORT=8009 npm run dev
|
||||
```
|
||||
|
||||
Optional but useful:
|
||||
|
||||
- add a small root helper script or Make target for local startup
|
||||
- add a `dreamverse-doctor` or lightweight startup check later
|
||||
|
||||
#### 0.3 Define the minimum local runtime contract
|
||||
|
||||
For Milestone 0, the frontend should rely only on the current runtime surface:
|
||||
|
||||
- `/ws`
|
||||
- `/status`
|
||||
- `/healthz`
|
||||
- `/readyz`
|
||||
- existing prompt/devtools routes
|
||||
|
||||
Do not add a second local API layer yet unless the current runtime surface is
|
||||
proven insufficient.
|
||||
|
||||
#### 0.4 Make failure states explicit in the UI
|
||||
|
||||
Local GPU mode fails in a few predictable ways:
|
||||
|
||||
- backend not reachable
|
||||
- backend reachable but not ready
|
||||
- `fastvideo` or model runtime missing
|
||||
- no compatible GPU available
|
||||
|
||||
Minimum implementation:
|
||||
|
||||
- show a clear connection error when `/ws` or `/status` fails
|
||||
- surface readiness failures in a human-readable way
|
||||
- avoid silent retry loops that hide backend startup failures
|
||||
|
||||
This is a small UI pass, not a controller project.
|
||||
|
||||
#### 0.5 Add a minimal local smoke test path
|
||||
|
||||
At this milestone, local GPU support is "done" only if there is a repeatable
|
||||
test path for the local runtime contract.
|
||||
|
||||
Minimum test additions:
|
||||
|
||||
- backend tests for `/healthz`, `/readyz`, and `/status`
|
||||
- a frontend integration test that assumes a reachable backend URL and verifies
|
||||
connection lifecycle behavior
|
||||
- one local smoke script that starts the backend and verifies readiness before
|
||||
the frontend is launched
|
||||
|
||||
### Milestone 1: Introduce a thin local controller
|
||||
|
||||
Goal:
|
||||
|
||||
- Preserve the same local GPU behavior, but place a stable local control-plane
|
||||
API in front of the runtime.
|
||||
|
||||
This should happen only after Milestone 0 is stable.
|
||||
|
||||
Scope:
|
||||
|
||||
- add `controller/`
|
||||
- proxy `/ws` and the needed REST routes to `server/`
|
||||
- expose controller-owned status for "backend starting", "runtime ready", and
|
||||
"runtime failed"
|
||||
- optionally spawn the local runtime as a subprocess
|
||||
|
||||
Non-goal:
|
||||
|
||||
- do not add remote provider logic yet
|
||||
|
||||
Reasoning:
|
||||
|
||||
- the controller earns its complexity only once it stabilizes the local
|
||||
contract that future providers will share
|
||||
|
||||
### Milestone 2: Provider abstraction on top of the controller
|
||||
|
||||
Goal:
|
||||
|
||||
- Keep the same frontend contract while allowing the controller to resolve a
|
||||
runtime via `local`, then later `runpod` and `modal`.
|
||||
|
||||
At this point:
|
||||
|
||||
- define `ComputeProvider`
|
||||
- implement `providers/local.py`
|
||||
- move local-runtime subprocess management behind the provider interface
|
||||
|
||||
The first provider should be `local`, because it is cheapest to debug and
|
||||
matches the runtime most closely.
|
||||
|
||||
## Minimal Code Change Order
|
||||
|
||||
If we want the shortest path to a working local GPU milestone, the change order
|
||||
should be:
|
||||
|
||||
1. Fix `server/config.py` and any remaining path assumptions from `prod-ui` to
|
||||
`apps/web`.
|
||||
2. Update `README.md` to document the real local GPU startup flow.
|
||||
3. Confirm `apps/dreamverse/web` connects cleanly to the local wrapper-backed backend.
|
||||
4. Improve frontend error handling for backend-not-ready and backend-missing
|
||||
cases.
|
||||
5. Add a local smoke test and keep existing backend/frontend tests green.
|
||||
6. Only then introduce `controller/`.
|
||||
|
||||
## Test Plan for the Local GPU Milestone
|
||||
|
||||
### Backend
|
||||
|
||||
Keep the current Python test suite as the base:
|
||||
|
||||
- `server/tests/test_health_endpoints.py`
|
||||
- `server/tests/test_mock_server.py`
|
||||
- `server/tests/test_prompt_enhancer.py`
|
||||
- `server/tests/test_rewrite_prompt_payload.py`
|
||||
- related config and logging tests
|
||||
|
||||
Add or tighten tests for:
|
||||
|
||||
- path resolution in `server/config.py`
|
||||
- readiness behavior when GPU pool initialization fails
|
||||
- startup error messaging when `fastvideo` is unavailable
|
||||
|
||||
### Frontend
|
||||
|
||||
Keep the current Vitest suite as the base:
|
||||
|
||||
- websocket reducer tests
|
||||
- prompt-window snapshot tests
|
||||
- integration tests under `apps/web/src/app/`
|
||||
|
||||
Add or tighten tests for:
|
||||
|
||||
- connection failure UX when backend is down
|
||||
- readiness failure UX when backend returns non-ready status
|
||||
- backend routing configuration through `BACKEND_HOST` and `BACKEND_PORT`
|
||||
|
||||
### Manual smoke path
|
||||
|
||||
The first manual smoke checklist should be:
|
||||
|
||||
1. start `dreamverse-server`
|
||||
2. confirm `GET /healthz` returns 200
|
||||
3. confirm `GET /readyz` returns 200 after warmup
|
||||
4. start `apps/web`
|
||||
5. confirm the UI opens and the websocket connects
|
||||
6. submit a prompt and verify the first generation starts
|
||||
|
||||
This checklist should be written down in the README once the milestone is
|
||||
implemented.
|
||||
|
||||
## Test Strategy
|
||||
|
||||
The test suite should preserve one rule: provider changes must not be able to
|
||||
break prompt rewrite, websocket semantics, or runtime behavior silently.
|
||||
|
||||
### 1. Runtime unit and integration tests
|
||||
|
||||
Keep and expand the current `pytest` coverage in `server/test_*.py`.
|
||||
|
||||
Focus areas:
|
||||
|
||||
- config loading
|
||||
- prompt rewrite payload normalization
|
||||
- prompt enhancement and prompt safety
|
||||
- health/readiness endpoints
|
||||
- websocket session behavior
|
||||
- session logging
|
||||
- mock runtime behavior
|
||||
|
||||
These tests should remain provider-agnostic.
|
||||
|
||||
### 2. Controller unit tests
|
||||
|
||||
Add a Python test suite for the controller state machine.
|
||||
|
||||
Key cases:
|
||||
|
||||
- provider selection and validation
|
||||
- credential loading from local config or env
|
||||
- runtime lifecycle transitions:
|
||||
- idle
|
||||
- provisioning
|
||||
- ready
|
||||
- failed
|
||||
- stopping
|
||||
- idle timeout and cleanup behavior
|
||||
- retry and backoff behavior
|
||||
- HTTP and websocket proxy routing
|
||||
|
||||
These tests should use fake providers and fake runtimes by default.
|
||||
|
||||
### 3. Provider contract tests
|
||||
|
||||
Each provider should pass the same contract tests.
|
||||
|
||||
Examples:
|
||||
|
||||
- `ensure_runtime()` returns a usable `RuntimeHandle`
|
||||
- `wait_until_ready()` surfaces timeout vs readiness correctly
|
||||
- `stop_runtime()` is safe to call twice
|
||||
- provider errors are mapped into stable controller error types
|
||||
|
||||
Use recorded fixtures or fakes wherever possible to avoid spend in CI.
|
||||
|
||||
### 4. Web contract tests
|
||||
|
||||
The frontend already has useful Vitest coverage under `apps/web/src`.
|
||||
Preserve that and expand around the controller split.
|
||||
|
||||
Priority areas:
|
||||
|
||||
- websocket event reduction
|
||||
- prompt-window snapshot construction
|
||||
- rewrite request shaping
|
||||
- compute-status UI
|
||||
- failure and reconnect UX
|
||||
|
||||
The frontend should mock the local controller API, not provider APIs.
|
||||
|
||||
### 5. Cross-layer protocol tests
|
||||
|
||||
Add contract fixtures that validate shared payloads across layers.
|
||||
|
||||
Important fixtures:
|
||||
|
||||
- websocket event payloads
|
||||
- rewrite request payloads
|
||||
- runtime status payloads
|
||||
- controller status payloads
|
||||
|
||||
These can be simple JSON fixtures validated by both Python and TypeScript
|
||||
tests. They will catch drift earlier than end-to-end tests.
|
||||
|
||||
### 6. Smoke tests
|
||||
|
||||
Add a small number of high-signal smoke tests:
|
||||
|
||||
- local provider + mock runtime
|
||||
- local provider + real runtime when GPU is available
|
||||
- controller startup + frontend health path
|
||||
|
||||
These should be cheap enough for routine local use.
|
||||
|
||||
### 7. Provider-backed manual or nightly tests
|
||||
|
||||
Real Modal and Runpod tests should be opt-in.
|
||||
|
||||
- Do not run them in default CI.
|
||||
- Gate them behind explicit credentials and flags.
|
||||
- Keep them focused on provisioning and reachability, not full product
|
||||
regression.
|
||||
|
||||
This avoids flaky and expensive CI while still validating real provider flows.
|
||||
|
||||
## Testing Recommendations for the Next Step
|
||||
|
||||
The next practical additions should be:
|
||||
|
||||
1. A controller test suite in Python using fake providers.
|
||||
2. Shared contract fixtures for websocket and rewrite payloads.
|
||||
3. A smoke test that starts the local controller against the existing mock
|
||||
runtime.
|
||||
|
||||
I would not add Playwright yet. The current web stack already has Vitest and
|
||||
integration-style component tests, which are cheaper and better aligned with
|
||||
the immediate reorg. Add browser automation only after the controller path is
|
||||
stable.
|
||||
@@ -0,0 +1,577 @@
|
||||
<mxfile host="65bd71144e">
|
||||
<diagram id="gpu-pool" name="gpu_pool.py">
|
||||
<mxGraphModel dx="1687" dy="3380" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="1300" pageHeight="1700" math="0" shadow="0">
|
||||
<root>
|
||||
<mxCell id="0"/>
|
||||
<mxCell id="1" parent="0"/>
|
||||
<mxCell id="title" value="gpu_pool.py — runtime architecture" style="text;html=1;align=center;fontSize=18;fontStyle=1;" parent="1" vertex="1">
|
||||
<mxGeometry x="350" y="-150" width="600" height="30" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="client" value="Client (WebSocket)" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontSize=13;fontStyle=1;" parent="1" vertex="1">
|
||||
<mxGeometry x="560" y="-60" width="180" height="60" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="pool" value="GPUPool" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontSize=13;fontStyle=1;" parent="1" vertex="1">
|
||||
<mxGeometry x="560" y="60" width="180" height="60" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="slot_wrap" value="GPUSlot (one per GPU)" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#e6f2e0;strokeColor=#82b366;strokeWidth=2;verticalAlign=top;align=left;spacingLeft=15;spacingTop=8;fontSize=13;fontStyle=1;" parent="1" vertex="1">
|
||||
<mxGeometry x="40" y="216" width="1220" height="455" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="m_start" value="start()" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontSize=10;align=left;spacingLeft=8;spacingTop=4;" parent="1" vertex="1">
|
||||
<mxGeometry x="110" y="270" width="60" height="30" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="m_shutdown" value="shutdown()" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontSize=10;align=left;spacingLeft=8;spacingTop=4;" parent="1" vertex="1">
|
||||
<mxGeometry x="180" y="270" width="70" height="30" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="m_join" value="join_user()" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontSize=10;align=left;spacingLeft=8;spacingTop=4;" parent="1" vertex="1">
|
||||
<mxGeometry x="499" y="260" width="70" height="40" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="m_step" value="user_step()" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontSize=10;align=left;spacingLeft=8;spacingTop=4;" parent="1" vertex="1">
|
||||
<mxGeometry x="580" y="260" width="80" height="40" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="m_leave" value="leave_user()" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontSize=10;align=left;spacingLeft=8;spacingTop=4;" parent="1" vertex="1">
|
||||
<mxGeometry x="670" y="255" width="80" height="45" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="m_send" value="_send_command()<br><br>puts on command_queue,<br>blocks on response_queue.get<div><br></div><div><b>**for user agonistic commands</b></div>" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontSize=10;align=left;spacingLeft=8;spacingTop=4;fontStyle=2;" parent="1" vertex="1">
|
||||
<mxGeometry x="79" y="370" width="230" height="80" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="m_send_tagged" value="_send_command_tagged()<br><br>creates Future; registers in<br>_pending_futures[user_id];<br>ensures response reader; <b>awaits future</b><div><br></div><div><b>**for user specific commands</b></div>" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontSize=10;align=left;spacingLeft=8;spacingTop=4;fontStyle=2;" parent="1" vertex="1">
|
||||
<mxGeometry x="350" y="374" width="300" height="90" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="m_reader" value="_response_reader()<br><br>a while loop that reads response_queue;<br>dispatch WorkerEvent:<br>• media → _stream_queues<br>• acks/errors → _pending_futures" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontSize=10;align=left;spacingLeft=8;spacingTop=4;fontStyle=2;" parent="1" vertex="1">
|
||||
<mxGeometry x="800" y="360" width="340" height="110" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="c_futures" value="«dict» _pending_futures

{user_id → asyncio.Future}

created by _send_command_tagged;
resolved + popped by _response_reader;
cleared on model reload or leave" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#fff2cc;strokeColor=#d6b656;fontSize=10;align=left;spacingLeft=8;spacingTop=4;" parent="1" vertex="1">
|
||||
<mxGeometry x="300" y="524" width="340" height="110" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="c_streams" value="«dict» _stream_queues:&nbsp;<div><span style="color: rgb(63, 63, 63); background-color: transparent;">each is a queue of metadata that tell the consumer which part of the shared&nbsp;</span></div><div><div><font color="#000000">stream buffer to read next.<br></font><br>{user_id → asyncio.Queue}<br><br>created by register_stream_queue()<br>written by _response_reader (MediaInit / Chunk / Complete);<br>consumed by&nbsp; AV loop → ws.send_bytes / send_json;<br>popped by leave_user(); cleared on reload</div></div>" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#fff2cc;strokeColor=#d6b656;fontSize=10;align=left;spacingLeft=8;spacingTop=4;" parent="1" vertex="1">
|
||||
<mxGeometry x="720" y="524" width="460" height="126" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="div_line" value="" style="line;strokeColor=#888888;strokeWidth=1;dashed=1;" parent="1" vertex="1">
|
||||
<mxGeometry x="40" y="690" width="1220" height="10" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="div_label" value="── process boundary (mp.Queue + mp.RawArray) ──" style="text;html=1;align=center;fontSize=11;fontColor=#666666;fontStyle=2;" parent="1" vertex="1">
|
||||
<mxGeometry x="450" y="688" width="400" height="20" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="cmdq" value="command_queue" style="shape=cylinder3;whiteSpace=wrap;html=1;boundedLbl=1;backgroundOutline=1;size=15;fillColor=#f8cecc;strokeColor=#b85450;fontSize=12;fontStyle=1;" parent="1" vertex="1">
|
||||
<mxGeometry x="79" y="720" width="200" height="100" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="respq" value="response_queue" style="shape=cylinder3;whiteSpace=wrap;html=1;boundedLbl=1;backgroundOutline=1;size=15;fillColor=#f8cecc;strokeColor=#b85450;fontSize=12;fontStyle=1;" parent="1" vertex="1">
|
||||
<mxGeometry x="520" y="720" width="200" height="100" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="sharedbuf" value="shared_stream_buffer
(mp.RawArray, 256 MiB)

zero-copy mp4 bytes" style="shape=cylinder3;whiteSpace=wrap;html=1;boundedLbl=1;backgroundOutline=1;size=15;fillColor=#f8cecc;strokeColor=#b85450;fontSize=11;" parent="1" vertex="1">
|
||||
<mxGeometry x="900" y="720" width="280" height="100" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="worker_ipc" value="worker_ipc.py — shared types (imported on both sides of the process boundary)

Command(type: CommandType, payload: CommandPayload | None, user_id: str | None)
CommandPayload = UserStepPayload | WarmupPayload | ReloadModelPayload
WorkerEvent = StepComplete | WorkerError | JoinAck | LeaveAck | ReloadAck | WarmupComplete | MediaInit | MediaChunk | MediaComplete | InitAck | ShutdownAck" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#e0e0e0;strokeColor=#666666;fontSize=10;align=left;spacingLeft=10;spacingTop=6;fontFamily=monospace;fontStyle=1;" parent="1" vertex="1">
|
||||
<mxGeometry x="80" y="860" width="1140" height="80" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="worker_wrap" value="Worker Subprocess - in each GPUSlot" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#fef1e0;strokeColor=#d79b00;strokeWidth=2;verticalAlign=top;align=left;spacingLeft=15;spacingTop=8;fontSize=13;fontStyle=1;" parent="1" vertex="1">
|
||||
<mxGeometry x="73" y="1071" width="1140" height="360" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="dispatcher" value="command dispatcher

gpu_worker_process() branches on
CommandType; asserts payload type

INIT / WARMUP / RELOAD_MODEL
USER_JOIN / USER_STEP / USER_LEAVE
SHUTDOWN" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#ffe6cc;strokeColor=#d79b00;fontSize=11;align=left;spacingLeft=10;spacingTop=8;fontStyle=1;" parent="1" vertex="1">
|
||||
<mxGeometry x="120" y="1120" width="240" height="120" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="do_step" value="VideoGenerationWorker.generate_step()
video_generation.py:380

reads + updates ContinuationState,
calls generator" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#e1d5e7;strokeColor=#9673a6;fontSize=11;align=left;spacingLeft=10;spacingTop=8;fontStyle=1;" parent="1" vertex="1">
|
||||
<mxGeometry x="460" y="1120" width="240" height="120" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="stream_av" value="stream_fmp4()
av_streaming.py:121

trims overlap, pipes to ffmpeg,
publishes StreamInit / StreamChunk /
StreamComplete via callback" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#b1d8d7;strokeColor=#23445d;fontSize=11;align=left;spacingLeft=10;spacingTop=8;fontStyle=1;" parent="1" vertex="1">
|
||||
<mxGeometry x="800" y="1120" width="260" height="120" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="Ot8BU52QTIb4EhyRSe7I-2" value="" style="edgeStyle=none;html=1;" parent="1" source="generator" target="Ot8BU52QTIb4EhyRSe7I-1" edge="1">
|
||||
<mxGeometry relative="1" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="generator" value="VideoGenerator (fastvideo)

LTX2 DiT + refine upsampler
FP4 quant, torch.compile

owned by VideoGenerationWorker
video_generation.py:211" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#e1d5e7;strokeColor=#9673a6;fontSize=11;" parent="1" vertex="1">
|
||||
<mxGeometry x="460" y="1300" width="240" height="100" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="ffmpeg" value="ffmpeg subprocess

libx264 / *_nvenc
fragmented mp4" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#ffffff;strokeColor=#d79b00;fontSize=11;" parent="1" vertex="1">
|
||||
<mxGeometry x="800" y="1300" width="260" height="100" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="caches" value="ContinuationState
video_generation.py:89

• video_images: list[PIL.Image]
• audio_latents: torch.Tensor (CPU)

carried across segments" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#e1d5e7;strokeColor=#9673a6;fontSize=11;align=left;spacingLeft=10;spacingTop=8;" parent="1" vertex="1">
|
||||
<mxGeometry x="120" y="1300" width="240" height="100" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="e_cp" value="acquire" style="edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;strokeColor=#6c8ebf;endArrow=classic;fontSize=11;exitX=0.5;exitY=1;exitDx=0;exitDy=0;entryX=0.5;entryY=0;entryDx=0;entryDy=0;" parent="1" source="client" target="pool" edge="1">
|
||||
<mxGeometry relative="1" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="e_ps" value="assigns slot" style="edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;strokeColor=#82b366;endArrow=classic;fontSize=11;exitX=0.5;exitY=1;exitDx=0;exitDy=0;entryX=0.5;entryY=0;entryDx=0;entryDy=0;" parent="1" source="pool" target="slot_wrap" edge="1">
|
||||
<mxGeometry relative="1" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="e_start_send" value="" style="edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;strokeColor=#82b366;endArrow=classic;fontSize=10;" parent="1" target="m_send" edge="1">
|
||||
<mxGeometry relative="1" as="geometry">
|
||||
<mxPoint x="138" y="300" as="sourcePoint"/>
|
||||
<Array as="points">
|
||||
<mxPoint x="138" y="380"/>
|
||||
<mxPoint x="138" y="380"/>
|
||||
</Array>
|
||||
</mxGeometry>
|
||||
</mxCell>
|
||||
<mxCell id="e_join_tag" value="" style="edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;strokeColor=#82b366;endArrow=classic;fontSize=10;exitX=0.5;exitY=1;exitDx=0;exitDy=0;" parent="1" source="m_join" target="m_send_tagged" edge="1">
|
||||
<mxGeometry relative="1" as="geometry">
|
||||
<mxPoint x="531.6999999999998" y="400.3600000000001" as="targetPoint"/>
|
||||
<Array as="points">
|
||||
<mxPoint x="532" y="300"/>
|
||||
</Array>
|
||||
</mxGeometry>
|
||||
</mxCell>
|
||||
<mxCell id="e_step_tag" value="" style="edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;strokeColor=#82b366;endArrow=classic;fontSize=10;exitX=0.1;exitY=1;exitDx=0;exitDy=0;" parent="1" source="m_step" target="m_send_tagged" edge="1">
|
||||
<mxGeometry relative="1" as="geometry">
|
||||
<mxPoint x="588" y="400" as="targetPoint"/>
|
||||
<Array as="points">
|
||||
<mxPoint x="588" y="337"/>
|
||||
<mxPoint x="590" y="337"/>
|
||||
</Array>
|
||||
</mxGeometry>
|
||||
</mxCell>
|
||||
<mxCell id="e_leave_tag" value="" style="edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;strokeColor=#82b366;endArrow=classic;fontSize=10;exitX=0.1;exitY=1;exitDx=0;exitDy=0;entryX=0.874;entryY=-0.002;entryDx=0;entryDy=0;entryPerimeter=0;" parent="1" source="m_leave" target="m_send_tagged" edge="1">
|
||||
<mxGeometry x="-0.1388" relative="1" as="geometry">
|
||||
<Array as="points">
|
||||
<mxPoint x="680" y="300"/>
|
||||
<mxPoint x="680" y="330"/>
|
||||
<mxPoint x="612" y="330"/>
|
||||
</Array>
|
||||
<mxPoint as="offset"/>
|
||||
</mxGeometry>
|
||||
</mxCell>
|
||||
<mxCell id="e_tag_futures" value="register future" style="edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;strokeColor=#d6b656;endArrow=classic;fontSize=10;exitX=0.397;exitY=0.989;exitDx=0;exitDy=0;entryX=0.5;entryY=0;entryDx=0;entryDy=0;exitPerimeter=0;" parent="1" source="m_send_tagged" target="c_futures" edge="1">
|
||||
<mxGeometry relative="1" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="e_reader_futures" value="<b>set future result</b>" style="edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;strokeColor=#d6b656;endArrow=classic;fontSize=10;exitX=0.008;exitY=0.908;exitDx=0;exitDy=0;entryX=1;entryY=0.3;entryDx=0;entryDy=0;exitPerimeter=0;" parent="1" source="m_reader" target="c_futures" edge="1">
|
||||
<mxGeometry x="-0.325" relative="1" as="geometry">
|
||||
<Array as="points">
|
||||
<mxPoint x="803" y="470"/>
|
||||
<mxPoint x="670" y="470"/>
|
||||
<mxPoint x="670" y="557"/>
|
||||
</Array>
|
||||
<mxPoint as="offset"/>
|
||||
</mxGeometry>
|
||||
</mxCell>
|
||||
<mxCell id="e_reader_streams" value="put(MediaEvent)" style="edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;strokeColor=#d6b656;endArrow=classic;fontSize=10;exitX=0.146;exitY=1.008;exitDx=0;exitDy=0;exitPerimeter=0;entryX=0.282;entryY=0.021;entryDx=0;entryDy=0;entryPerimeter=0;" parent="1" source="m_reader" target="c_streams" edge="1">
|
||||
<mxGeometry relative="1" as="geometry">
|
||||
<Array as="points"/>
|
||||
<mxPoint x="850" y="522" as="targetPoint"/>
|
||||
</mxGeometry>
|
||||
</mxCell>
|
||||
<mxCell id="e_send_cmdq" value="put Command" style="edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;strokeColor=#b85450;endArrow=classic;fontSize=11;entryX=0.35;entryY=0;entryDx=0;entryDy=0;" parent="1" source="m_send" target="cmdq" edge="1">
|
||||
<mxGeometry relative="1" as="geometry">
|
||||
<Array as="points">
|
||||
<mxPoint x="150" y="585"/>
|
||||
<mxPoint x="149" y="585"/>
|
||||
</Array>
|
||||
</mxGeometry>
|
||||
</mxCell>
|
||||
<mxCell id="e_tag_cmdq" value="put Command" style="edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;strokeColor=#b85450;endArrow=classic;fontSize=11;exitX=0.022;exitY=0.97;exitDx=0;exitDy=0;entryX=0.85;entryY=0;entryDx=0;entryDy=0;exitPerimeter=0;" parent="1" source="m_send_tagged" target="cmdq" edge="1">
|
||||
<mxGeometry x="0.1755" relative="1" as="geometry">
|
||||
<Array as="points">
|
||||
<mxPoint x="357" y="490"/>
|
||||
<mxPoint x="250" y="490"/>
|
||||
<mxPoint x="250" y="592"/>
|
||||
<mxPoint x="249" y="592"/>
|
||||
</Array>
|
||||
<mxPoint as="offset"/>
|
||||
</mxGeometry>
|
||||
</mxCell>
|
||||
<mxCell id="e_respq_reader" value="WorkerEvent" style="edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;strokeColor=#b85450;endArrow=classic;fontSize=11;exitX=0.753;exitY=0.056;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;exitPerimeter=0;" parent="1" source="respq" target="m_reader" edge="1">
|
||||
<mxGeometry x="0.7113" relative="1" as="geometry">
|
||||
<Array as="points">
|
||||
<mxPoint x="671" y="720"/>
|
||||
<mxPoint x="670" y="720"/>
|
||||
<mxPoint x="670" y="415"/>
|
||||
</Array>
|
||||
<mxPoint as="offset"/>
|
||||
</mxGeometry>
|
||||
</mxCell>
|
||||
<mxCell id="e_respq_send" value="blocking .get" style="edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;strokeColor=#b85450;endArrow=classic;dashed=1;fontSize=10;exitX=0;exitY=0.5;exitDx=0;exitDy=0;" parent="1" source="respq" target="m_send" edge="1">
|
||||
<mxGeometry x="-0.2187" relative="1" as="geometry">
|
||||
<Array as="points">
|
||||
<mxPoint x="520" y="680"/>
|
||||
<mxPoint x="200" y="680"/>
|
||||
</Array>
|
||||
<mxPoint as="offset"/>
|
||||
</mxGeometry>
|
||||
</mxCell>
|
||||
<mxCell id="e_cd" value="" style="edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;strokeColor=#b85450;endArrow=classic;fontSize=11;exitX=0.5;exitY=1;exitDx=0;exitDy=0;entryX=0.25;entryY=0;entryDx=0;entryDy=0;" parent="1" source="cmdq" edge="1" target="dispatcher">
|
||||
<mxGeometry relative="1" as="geometry">
|
||||
<mxPoint x="310" y="1120" as="targetPoint"/>
|
||||
<Array as="points">
|
||||
<mxPoint x="179" y="970"/>
|
||||
<mxPoint x="180" y="970"/>
|
||||
</Array>
|
||||
</mxGeometry>
|
||||
</mxCell>
|
||||
<mxCell id="e_dds" value="USER_STEP" style="edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;strokeColor=#d79b00;endArrow=classic;fontSize=10;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;" parent="1" source="dispatcher" target="do_step" edge="1">
|
||||
<mxGeometry relative="1" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="e_dr" value="StepComplete / ack types" style="edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;strokeColor=#b85450;endArrow=classic;fontSize=10;exitX=0.5;exitY=0;exitDx=0;exitDy=0;entryX=0.15;entryY=1;entryDx=0;entryDy=0;" parent="1" source="dispatcher" target="respq" edge="1">
|
||||
<mxGeometry relative="1" as="geometry">
|
||||
<Array as="points">
|
||||
<mxPoint x="240" y="1000"/>
|
||||
<mxPoint x="550" y="1000"/>
|
||||
</Array>
|
||||
</mxGeometry>
|
||||
</mxCell>
|
||||
<mxCell id="e_dsg" value="generator.generate_video()" style="edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;strokeColor=#9673a6;endArrow=classic;fontSize=10;exitX=0.5;exitY=1;exitDx=0;exitDy=0;entryX=0.5;entryY=0;entryDx=0;entryDy=0;" parent="1" source="do_step" target="generator" edge="1">
|
||||
<mxGeometry relative="1" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="e_dscache" value="read / write" style="edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;strokeColor=#d6b656;endArrow=classic;startArrow=classic;fontSize=10;exitX=0;exitY=0.8;exitDx=0;exitDy=0;entryX=1;entryY=0.2;entryDx=0;entryDy=0;" parent="1" source="do_step" target="caches" edge="1">
|
||||
<mxGeometry relative="1" as="geometry"/>
|
||||
<Array as="points">
|
||||
<mxPoint x="420" y="1058"/>
|
||||
<mxPoint x="420" y="1150"/>
|
||||
</Array>
|
||||
</mxCell>
|
||||
<mxCell id="e_dsstream" value="frames + audio" style="edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;strokeColor=#d79b00;endArrow=classic;fontSize=10;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;" parent="1" source="do_step" target="stream_av" edge="1">
|
||||
<mxGeometry relative="1" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="e_sf" value="rawvideo + wav" style="edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;strokeColor=#d79b00;endArrow=classic;fontSize=10;exitX=0.5;exitY=1;exitDx=0;exitDy=0;entryX=0.5;entryY=0;entryDx=0;entryDy=0;" parent="1" source="stream_av" target="ffmpeg" edge="1">
|
||||
<mxGeometry relative="1" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="e_ssb" value="write mp4 bytes
at shared_write_offset" style="edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;strokeColor=#b85450;endArrow=classic;fontSize=10;exitX=0.75;exitY=0;exitDx=0;exitDy=0;entryX=0.5;entryY=1;entryDx=0;entryDy=0;" parent="1" source="stream_av" target="sharedbuf" edge="1">
|
||||
<mxGeometry relative="1" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="e_sbs" value="bytes from the buffer are sent through websocket&nbsp;<div>based on info from the stream queue</div>" style="edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;strokeColor=#b85450;endArrow=classic;dashed=1;fontSize=10;exitX=0.591;exitY=0.027;exitDx=0;exitDy=0;entryX=0.75;entryY=1;entryDx=0;entryDy=0;exitPerimeter=0;" parent="1" source="sharedbuf" target="c_streams" edge="1">
|
||||
<mxGeometry x="0.0048" relative="1" as="geometry">
|
||||
<Array as="points">
|
||||
<mxPoint x="1066" y="720"/>
|
||||
</Array>
|
||||
<mxPoint as="offset"/>
|
||||
</mxGeometry>
|
||||
</mxCell>
|
||||
<mxCell id="e_scl" value="session AV loop (main.py:2124):
ws.send_bytes(chunk) — binary mp4 frames
ws.send_json({type: media_init | media_segment_complete | ...})" style="edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;strokeColor=#6c8ebf;endArrow=classic;fontSize=10;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=1;entryY=0.5;entryDx=0;entryDy=0;" parent="1" source="c_streams" target="client" edge="1">
|
||||
<mxGeometry x="0.0003" relative="1" as="geometry">
|
||||
<mxPoint as="offset"/>
|
||||
<Array as="points">
|
||||
<mxPoint x="1310" y="577"/>
|
||||
<mxPoint x="1310" y="-30"/>
|
||||
</Array>
|
||||
</mxGeometry>
|
||||
</mxCell>
|
||||
<mxCell id="e_sr" value="MediaChunk (offset, len)" style="edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;strokeColor=#b85450;endArrow=classic;fontSize=10;exitX=0.1;exitY=0;exitDx=0;exitDy=0;entryX=0.85;entryY=1;entryDx=0;entryDy=0;" parent="1" source="stream_av" target="respq" edge="1">
|
||||
<mxGeometry relative="1" as="geometry"/>
|
||||
<Array as="points">
|
||||
<mxPoint x="826" y="880"/>
|
||||
<mxPoint x="690" y="880"/>
|
||||
</Array>
|
||||
</mxCell>
|
||||
<mxCell id="legend" value="Legend

■ blue client / external
■ green main-process pool/slot
 (methods — italic label)
■ yellow containers (routing state)
■ red IPC primitives (mp.Queue, mp.RawArray)

Worker subprocess modules:
■ orange gpu_pool.py (dispatcher)
■ lavender video_generation.py
■ teal av_streaming.py
■ gray worker_ipc.py (shared types)

Flow:
 client → pool → slot
 → _send_command(_tagged) → command_queue
 → dispatcher → generate_step()
 → stream_fmp4() → ffmpeg
 → shared_buf + response_queue
 → _response_reader → futures / stream_queues
 → client awaits (via main.py AV loop)" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#f5f5f5;strokeColor=#999999;fontSize=11;align=left;spacingLeft=10;spacingTop=8;" parent="1" vertex="1">
|
||||
<mxGeometry x="39" y="-200" width="270" height="380" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="Ot8BU52QTIb4EhyRSe7I-1" value="FastVideo video_generator" style="whiteSpace=wrap;html=1;fontSize=11;fillColor=#e1d5e7;strokeColor=#9673a6;rounded=1;" parent="1" vertex="1">
|
||||
<mxGeometry x="520" y="1470" width="120" height="60" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="5" value="" style="edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;strokeColor=#82b366;endArrow=classic;fontSize=10;exitX=0.421;exitY=1.01;exitDx=0;exitDy=0;exitPerimeter=0;entryX=0.569;entryY=0.02;entryDx=0;entryDy=0;entryPerimeter=0;" edge="1" parent="1" source="m_shutdown" target="m_send">
|
||||
<mxGeometry relative="1" as="geometry">
|
||||
<mxPoint x="260" y="300" as="sourcePoint"/>
|
||||
<mxPoint x="260" y="400" as="targetPoint"/>
|
||||
<Array as="points"/>
|
||||
</mxGeometry>
|
||||
</mxCell>
|
||||
<mxCell id="6" style="edgeStyle=orthogonalEdgeStyle;html=1;entryX=0.983;entryY=0.021;entryDx=0;entryDy=0;entryPerimeter=0;strokeColor=#61A25D;" edge="1" parent="1" source="m_join" target="m_send">
|
||||
<mxGeometry relative="1" as="geometry">
|
||||
<Array as="points">
|
||||
<mxPoint x="534" y="320"/>
|
||||
<mxPoint x="306" y="320"/>
|
||||
</Array>
|
||||
</mxGeometry>
|
||||
</mxCell>
|
||||
<mxCell id="11" value="reload model" style="edgeLabel;html=1;align=center;verticalAlign=middle;resizable=0;points=[];" vertex="1" connectable="0" parent="6">
|
||||
<mxGeometry x="0.1965" relative="1" as="geometry">
|
||||
<mxPoint as="offset"/>
|
||||
</mxGeometry>
|
||||
</mxCell>
|
||||
<mxCell id="9" value="" style="group" vertex="1" connectable="0" parent="1">
|
||||
<mxGeometry x="1340" y="750" width="240" height="40" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="7" value="" style="shape=cylinder3;whiteSpace=wrap;html=1;boundedLbl=1;backgroundOutline=1;size=6.353056066176691;fillColor=#f8cecc;strokeColor=#b85450;fontSize=11;" vertex="1" parent="9">
|
||||
<mxGeometry y="4" width="21.39" height="22" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="8" value="Queue: owned by GPUSlot, passed to worker process as argument." style="text;html=1;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;" vertex="1" parent="9">
|
||||
<mxGeometry x="21.3900000000001" width="218.61" height="40" as="geometry"/>
|
||||
</mxCell>
|
||||
</root>
|
||||
</mxGraphModel>
|
||||
</diagram>
|
||||
<diagram id="gpu-pool-boot" name="boot & ws">
|
||||
<mxGraphModel grid="1" page="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" pageScale="1" pageWidth="1300" pageHeight="1900" math="0" shadow="0">
|
||||
<root>
|
||||
<mxCell id="0"/>
|
||||
<mxCell id="1" parent="0"/>
|
||||
<mxCell id="b_title" value="gpu_pool.py — boot flow & websocket session" style="text;html=1;align=center;fontSize=18;fontStyle=1;" parent="1" vertex="1">
|
||||
<mxGeometry x="350" y="20" width="600" height="30" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="secA" value="A. Call chain" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#e1f5fe;strokeColor=#039be5;strokeWidth=2;verticalAlign=top;align=left;spacingLeft=15;spacingTop=8;fontSize=13;fontStyle=1;" parent="1" vertex="1">
|
||||
<mxGeometry x="40" y="70" width="400" height="420" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="a1" value="$ uv run dreamverse-server" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#ffffff;strokeColor=#039be5;fontSize=12;fontFamily=monospace;" parent="1" vertex="1">
|
||||
<mxGeometry x="70" y="120" width="340" height="40" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="a2" value="server_entry.cli()
(pyproject.toml [project.scripts])

guards missing fastvideo extra" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#ffffff;strokeColor=#039be5;fontSize=11;align=left;spacingLeft=10;" parent="1" vertex="1">
|
||||
<mxGeometry x="70" y="180" width="340" height="70" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="a3" value="main.cli() (main.py:2325)

argparse --host/--port
uvicorn.run(app, host, port)" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#ffffff;strokeColor=#039be5;fontSize=11;align=left;spacingLeft=10;" parent="1" vertex="1">
|
||||
<mxGeometry x="70" y="270" width="340" height="80" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="a4" value="FastAPI(lifespan=lifespan)
(main.py:147)

on startup → enter lifespan ctx → Section B" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#ffffff;strokeColor=#039be5;fontSize=11;align=left;spacingLeft=10;" parent="1" vertex="1">
|
||||
<mxGeometry x="70" y="370" width="340" height="90" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="ea12" style="edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;strokeColor=#039be5;endArrow=classic;exitX=0.5;exitY=1;exitDx=0;exitDy=0;entryX=0.5;entryY=0;entryDx=0;entryDy=0;" parent="1" source="a1" target="a2" edge="1">
|
||||
<mxGeometry relative="1" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="ea23" style="edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;strokeColor=#039be5;endArrow=classic;exitX=0.5;exitY=1;exitDx=0;exitDy=0;entryX=0.5;entryY=0;entryDx=0;entryDy=0;" parent="1" source="a2" target="a3" edge="1">
|
||||
<mxGeometry relative="1" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="ea34" style="edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;strokeColor=#039be5;endArrow=classic;exitX=0.5;exitY=1;exitDx=0;exitDy=0;entryX=0.5;entryY=0;entryDx=0;entryDy=0;" parent="1" source="a3" target="a4" edge="1">
|
||||
<mxGeometry relative="1" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="secB" value="B. lifespan() zoom (main.py:116-144)" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#f3e5f5;strokeColor=#8e24aa;strokeWidth=2;verticalAlign=top;align=left;spacingLeft=15;spacingTop=8;fontSize=13;fontStyle=1;" parent="1" vertex="1">
|
||||
<mxGeometry x="480" y="70" width="780" height="420" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="b1" value="1. gpu_ids = get_available_gpus()
 respects FASTVIDEO_GPU_COUNT env" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#ffffff;strokeColor=#8e24aa;fontSize=11;align=left;spacingLeft=10;fontFamily=monospace;" parent="1" vertex="1">
|
||||
<mxGeometry x="510" y="115" width="720" height="45" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="b2" value="2. gpu_pool = GPUPool(gpu_ids)
 creates one GPUSlot per GPU (not yet started)" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#ffffff;strokeColor=#8e24aa;fontSize=11;align=left;spacingLeft=10;fontFamily=monospace;" parent="1" vertex="1">
|
||||
<mxGeometry x="510" y="170" width="720" height="45" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="b3" value="3. await gpu_pool.initialize()
 asyncio.create_task(_init_gpu(id)) for each GPU
 returns immediately — GPUs init in background → Section C" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#fff9c4;strokeColor=#8e24aa;fontSize=11;align=left;spacingLeft=10;fontFamily=monospace;" parent="1" vertex="1">
|
||||
<mxGeometry x="510" y="225" width="720" height="60" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="b4" value="4. prompt_enhancer = PromptEnhancer()
5. session_event_logger = SessionEventLogger(...)
6. prompt_safety_filter = PromptSafetyFilter() (optional)" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#ffffff;strokeColor=#8e24aa;fontSize=11;align=left;spacingLeft=10;fontFamily=monospace;" parent="1" vertex="1">
|
||||
<mxGeometry x="510" y="295" width="720" height="55" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="byield" value="yield
server is now accepting HTTP + WS
/healthz → 200 /readyz → 503 until GPUs warm up" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#c8e6c9;strokeColor=#388e3c;fontSize=11;align=left;spacingLeft=10;fontStyle=1;" parent="1" vertex="1">
|
||||
<mxGeometry x="510" y="360" width="720" height="55" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="bshut" value="[on shutdown] await gpu_pool.shutdown()" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#ffcdd2;strokeColor=#c62828;fontSize=11;align=left;spacingLeft=10;fontFamily=monospace;" parent="1" vertex="1">
|
||||
<mxGeometry x="510" y="425" width="720" height="35" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="eab" style="edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;strokeColor=#666;endArrow=classic;dashed=1;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0.2;entryDx=0;entryDy=0;" parent="1" source="a4" target="b1" edge="1">
|
||||
<mxGeometry relative="1" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="secC" value="C. GPUSlot.start() zoom — runs in background per GPU (gpu_pool.py:426)" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#fff3e0;strokeColor=#ef6c00;strokeWidth=2;verticalAlign=top;align=left;spacingLeft=15;spacingTop=8;fontSize=13;fontStyle=1;" parent="1" vertex="1">
|
||||
<mxGeometry x="40" y="530" width="1220" height="520" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="cparent" value="Parent (main) process" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#ffe0b2;strokeColor=#ef6c00;fontSize=12;fontStyle=1;verticalAlign=top;align=left;spacingLeft=10;spacingTop=5;" parent="1" vertex="1">
|
||||
<mxGeometry x="70" y="580" width="380" height="440" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="cp1" value="mp.get_context('spawn')
cmd_q, resp_q = ctx.Queue(), ctx.Queue()
(optional) shared_stream_buffer = mp.RawArray('B', 256 MiB)" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#ffffff;strokeColor=#ef6c00;fontSize=10;align=left;spacingLeft=8;fontFamily=monospace;" parent="1" vertex="1">
|
||||
<mxGeometry x="85" y="620" width="350" height="60" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="cp2" value="ctx.Process(target=gpu_worker_process,
 args=(gpu_id, cuda_device, cmd_q, resp_q, ...))
process.start()" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#ffffff;strokeColor=#ef6c00;fontSize=10;align=left;spacingLeft=8;fontFamily=monospace;" parent="1" vertex="1">
|
||||
<mxGeometry x="85" y="690" width="350" height="60" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="cp3" value="_send_command(Command(INIT), timeout=600s)
 await response on resp_q" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#ffffff;strokeColor=#ef6c00;fontSize=10;align=left;spacingLeft=8;fontFamily=monospace;" parent="1" vertex="1">
|
||||
<mxGeometry x="85" y="760" width="350" height="50" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="cp4" value="if STARTUP_WARMUP_ENABLED:
 _send_command(Command(WARMUP, prompt=...),
 timeout=STARTUP_WARMUP_TIMEOUT)" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#ffffff;strokeColor=#ef6c00;fontSize=10;align=left;spacingLeft=8;fontFamily=monospace;" parent="1" vertex="1">
|
||||
<mxGeometry x="85" y="820" width="350" height="60" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="cp5" value="slot.ready = True" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#c8e6c9;strokeColor=#388e3c;fontSize=11;fontStyle=1;fontFamily=monospace;" parent="1" vertex="1">
|
||||
<mxGeometry x="85" y="890" width="350" height="35" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="cp6" value="parent now enters pool as a ready slot;
/readyz flips to 200 once ≥1 slot is ready" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#f5f5f5;strokeColor=#888;fontSize=10;align=left;spacingLeft=8;fontStyle=2;" parent="1" vertex="1">
|
||||
<mxGeometry x="85" y="935" width="350" height="45" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="cipc" value="IPC" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#ffcdd2;strokeColor=#b71c1c;fontSize=12;fontStyle=1;verticalAlign=top;align=center;spacingTop=5;" parent="1" vertex="1">
|
||||
<mxGeometry x="480" y="580" width="150" height="440" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="ccmdq" value="command_queue
(mp.Queue)" style="shape=cylinder3;whiteSpace=wrap;html=1;boundedLbl=1;backgroundOutline=1;size=15;fillColor=#f8cecc;strokeColor=#b85450;fontSize=10;" parent="1" vertex="1">
|
||||
<mxGeometry x="490" y="650" width="130" height="80" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="crespq" value="response_queue
(mp.Queue)" style="shape=cylinder3;whiteSpace=wrap;html=1;boundedLbl=1;backgroundOutline=1;size=15;fillColor=#f8cecc;strokeColor=#b85450;fontSize=10;" parent="1" vertex="1">
|
||||
<mxGeometry x="490" y="760" width="130" height="80" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="cshared" value="shared_stream_buffer
(mp.RawArray)" style="shape=cylinder3;whiteSpace=wrap;html=1;boundedLbl=1;backgroundOutline=1;size=15;fillColor=#f8cecc;strokeColor=#b85450;fontSize=10;" parent="1" vertex="1">
|
||||
<mxGeometry x="490" y="870" width="130" height="80" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="cworker" value="Worker subprocess (gpu_worker_process, gpu_pool.py:151)" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#fef1e0;strokeColor=#d79b00;fontSize=12;fontStyle=1;verticalAlign=top;align=left;spacingLeft=10;spacingTop=5;" parent="1" vertex="1">
|
||||
<mxGeometry x="660" y="580" width="580" height="440" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="cw1" value="os.environ['CUDA_VISIBLE_DEVICES'] = cuda_device
os.environ['FASTVIDEO_ATTENTION_BACKEND'] = 'FLASH_ATTN'

must be set BEFORE importing torch" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#ffffff;strokeColor=#d79b00;fontSize=10;align=left;spacingLeft=8;fontFamily=monospace;" parent="1" vertex="1">
|
||||
<mxGeometry x="675" y="620" width="550" height="65" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="cw2" value="from fastvideo.entrypoints.video_generator import VideoGenerator
from fastvideo.models.dits.ltx2 import DEFAULT_LTX2_AUDIO_*

** Dreamverse reaches into fastvideo internals here **" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#ffe0b2;strokeColor=#d79b00;fontSize=10;align=left;spacingLeft=8;fontFamily=monospace;" parent="1" vertex="1">
|
||||
<mxGeometry x="675" y="695" width="550" height="60" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="cw3" value="on Command(INIT):
 VideoGenerationWorker.initialize() (video_generation.py:247)
 maybe_download_model(model_id)
 VideoGenerator.from_pretrained(path, FP4Config, PipelineConfig)
 load audio VAE, resolve refine upsampler
 resp_q.put(InitAck(success=True))" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#ffffff;strokeColor=#d79b00;fontSize=10;align=left;spacingLeft=8;fontFamily=monospace;" parent="1" vertex="1">
|
||||
<mxGeometry x="675" y="765" width="550" height="95" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="cw4" value="on Command(WARMUP) with WarmupPayload:
 VideoGenerationWorker.warmup(payload.prompt) (video_generation.py:518)
 two synthetic segments prime caches + torch.compile
 resp_q.put(WarmupComplete(timings=...))" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#ffffff;strokeColor=#d79b00;fontSize=10;align=left;spacingLeft=8;fontFamily=monospace;" parent="1" vertex="1">
|
||||
<mxGeometry x="675" y="870" width="550" height="55" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="cw5" value="enter main worker loop → waits for JOIN_USER / USER_STEP / LEAVE" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#c8e6c9;strokeColor=#388e3c;fontSize=11;fontStyle=1;fontFamily=monospace;" parent="1" vertex="1">
|
||||
<mxGeometry x="675" y="935" width="550" height="40" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="cea1" value="Command" style="edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;strokeColor=#b85450;endArrow=classic;fontSize=10;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;" parent="1" source="cp3" target="ccmdq" edge="1">
|
||||
<mxGeometry relative="1" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="cea2" style="edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;strokeColor=#b85450;endArrow=classic;exitX=1;exitY=0.5;exitDx=0;exitDy=0;" parent="1" source="ccmdq" edge="1">
|
||||
<mxGeometry relative="1" as="geometry">
|
||||
<mxPoint x="675" y="780" as="targetPoint"/>
|
||||
<Array as="points">
|
||||
<mxPoint x="648" y="690"/>
|
||||
<mxPoint x="648" y="780"/>
|
||||
<mxPoint x="675" y="780"/>
|
||||
</Array>
|
||||
</mxGeometry>
|
||||
</mxCell>
|
||||
<mxCell id="cea3" value="WorkerEvent" style="edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;strokeColor=#b85450;endArrow=classic;fontSize=10;exitX=0;exitY=0.5;exitDx=0;exitDy=0;entryX=0.969;entryY=0.798;entryDx=0;entryDy=0;entryPerimeter=0;" parent="1" target="crespq" edge="1">
|
||||
<mxGeometry relative="1" as="geometry">
|
||||
<mxPoint x="684" y="825.5" as="sourcePoint"/>
|
||||
<mxPoint x="629" y="813" as="targetPoint"/>
|
||||
</mxGeometry>
|
||||
</mxCell>
|
||||
<mxCell id="cea4" style="edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;strokeColor=#b85450;endArrow=classic;exitX=0;exitY=0.5;exitDx=0;exitDy=0;entryX=0.995;entryY=0.887;entryDx=0;entryDy=0;entryPerimeter=0;" parent="1" target="cp3" edge="1">
|
||||
<mxGeometry relative="1" as="geometry">
|
||||
<mxPoint x="490" y="804" as="sourcePoint"/>
|
||||
<mxPoint x="435" y="789" as="targetPoint"/>
|
||||
</mxGeometry>
|
||||
</mxCell>
|
||||
<mxCell id="secD" value="D. WebSocket session — after startup" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#e8f5e9;strokeColor=#2e7d32;strokeWidth=2;verticalAlign=top;align=left;spacingLeft=15;spacingTop=8;fontSize=13;fontStyle=1;" parent="1" vertex="1">
|
||||
<mxGeometry x="40" y="1080" width="1220" height="820" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="dc_head" value="Browser / WS" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontSize=11;fontStyle=1;" parent="1" vertex="1">
|
||||
<mxGeometry x="80" y="1130" width="160" height="40" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="ds_head" value="main.py / Session" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontSize=11;fontStyle=1;" parent="1" vertex="1">
|
||||
<mxGeometry x="290" y="1130" width="180" height="40" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="dp_head" value="GPUPool" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontSize=11;fontStyle=1;" parent="1" vertex="1">
|
||||
<mxGeometry x="520" y="1130" width="160" height="40" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="dsl_head" value="GPUSlot" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#e6f2e0;strokeColor=#82b366;fontSize=11;fontStyle=1;" parent="1" vertex="1">
|
||||
<mxGeometry x="730" y="1130" width="160" height="40" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="dw_head" value="Worker subprocess" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#fef1e0;strokeColor=#d79b00;fontSize=11;fontStyle=1;" parent="1" vertex="1">
|
||||
<mxGeometry x="940" y="1130" width="200" height="40" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="dc_life" style="endArrow=none;html=1;strokeColor=#6c8ebf;dashed=1;" parent="1" edge="1">
|
||||
<mxGeometry relative="1" as="geometry">
|
||||
<mxPoint x="160" y="1175" as="sourcePoint"/>
|
||||
<mxPoint x="160" y="1885" as="targetPoint"/>
|
||||
</mxGeometry>
|
||||
</mxCell>
|
||||
<mxCell id="ds_life" style="endArrow=none;html=1;strokeColor=#82b366;dashed=1;" parent="1" edge="1">
|
||||
<mxGeometry relative="1" as="geometry">
|
||||
<mxPoint x="380" y="1175" as="sourcePoint"/>
|
||||
<mxPoint x="380" y="1885" as="targetPoint"/>
|
||||
</mxGeometry>
|
||||
</mxCell>
|
||||
<mxCell id="dp_life" style="endArrow=none;html=1;strokeColor=#82b366;dashed=1;" parent="1" edge="1">
|
||||
<mxGeometry relative="1" as="geometry">
|
||||
<mxPoint x="600" y="1175" as="sourcePoint"/>
|
||||
<mxPoint x="600" y="1885" as="targetPoint"/>
|
||||
</mxGeometry>
|
||||
</mxCell>
|
||||
<mxCell id="dsl_life" style="endArrow=none;html=1;strokeColor=#82b366;dashed=1;" parent="1" edge="1">
|
||||
<mxGeometry relative="1" as="geometry">
|
||||
<mxPoint x="810" y="1175" as="sourcePoint"/>
|
||||
<mxPoint x="810" y="1885" as="targetPoint"/>
|
||||
</mxGeometry>
|
||||
</mxCell>
|
||||
<mxCell id="dw_life" style="endArrow=none;html=1;strokeColor=#d79b00;dashed=1;" parent="1" edge="1">
|
||||
<mxGeometry relative="1" as="geometry">
|
||||
<mxPoint x="1040" y="1175" as="sourcePoint"/>
|
||||
<mxPoint x="1040" y="1885" as="targetPoint"/>
|
||||
</mxGeometry>
|
||||
</mxCell>
|
||||
<mxCell id="dm1" value="1. WS connect /ws + session_init_v2" style="endArrow=classic;html=1;strokeColor=#6c8ebf;fontSize=10;" parent="1" edge="1">
|
||||
<mxGeometry relative="1" as="geometry">
|
||||
<mxPoint x="160" y="1200" as="sourcePoint"/>
|
||||
<mxPoint x="380" y="1200" as="targetPoint"/>
|
||||
</mxGeometry>
|
||||
</mxCell>
|
||||
<mxCell id="dm2" value="2. pool.acquire(client_id, ws)" style="endArrow=classic;html=1;strokeColor=#82b366;fontSize=10;" parent="1" edge="1">
|
||||
<mxGeometry relative="1" as="geometry">
|
||||
<mxPoint x="380" y="1240" as="sourcePoint"/>
|
||||
<mxPoint x="600" y="1240" as="targetPoint"/>
|
||||
</mxGeometry>
|
||||
</mxCell>
|
||||
<mxCell id="dm3a" value="3a. if slot free: (gpu_id, slot)" style="endArrow=classic;html=1;strokeColor=#82b366;fontSize=10;" parent="1" edge="1">
|
||||
<mxGeometry relative="1" as="geometry">
|
||||
<mxPoint x="600" y="1280" as="sourcePoint"/>
|
||||
<mxPoint x="380" y="1280" as="targetPoint"/>
|
||||
</mxGeometry>
|
||||
</mxCell>
|
||||
<mxCell id="dm3b" value="3b. else: enqueue, emit queue_status until a slot frees" style="endArrow=classic;html=1;strokeColor=#c62828;fontSize=10;dashed=1;" parent="1" edge="1">
|
||||
<mxGeometry relative="1" as="geometry">
|
||||
<mxPoint x="600" y="1320" as="sourcePoint"/>
|
||||
<mxPoint x="160" y="1320" as="targetPoint"/>
|
||||
</mxGeometry>
|
||||
</mxCell>
|
||||
<mxCell id="dm4" value="4. emit gpu_assigned" style="endArrow=classic;html=1;strokeColor=#6c8ebf;fontSize=10;" parent="1" edge="1">
|
||||
<mxGeometry relative="1" as="geometry">
|
||||
<mxPoint x="380" y="1360" as="sourcePoint"/>
|
||||
<mxPoint x="160" y="1360" as="targetPoint"/>
|
||||
</mxGeometry>
|
||||
</mxCell>
|
||||
<mxCell id="dm5" value="5. slot._send_command_tagged(Command(USER_JOIN))" style="endArrow=classic;html=1;strokeColor=#82b366;fontSize=10;" parent="1" edge="1">
|
||||
<mxGeometry relative="1" as="geometry">
|
||||
<mxPoint x="380" y="1400" as="sourcePoint"/>
|
||||
<mxPoint x="810" y="1400" as="targetPoint"/>
|
||||
</mxGeometry>
|
||||
</mxCell>
|
||||
<mxCell id="dm6" value="6. command_queue.put(JOIN_USER)" style="endArrow=classic;html=1;strokeColor=#b85450;fontSize=10;" parent="1" edge="1">
|
||||
<mxGeometry relative="1" as="geometry">
|
||||
<mxPoint x="810" y="1440" as="sourcePoint"/>
|
||||
<mxPoint x="1040" y="1440" as="targetPoint"/>
|
||||
</mxGeometry>
|
||||
</mxCell>
|
||||
<mxCell id="dssteady" value="── steady state: per prompt ──" style="text;html=1;align=center;fontSize=11;fontColor=#666;fontStyle=2;" parent="1" vertex="1">
|
||||
<mxGeometry x="300" y="1470" width="600" height="25" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="dm7" value="7. append_prompt / simple_generate" style="endArrow=classic;html=1;strokeColor=#6c8ebf;fontSize=10;" parent="1" edge="1">
|
||||
<mxGeometry relative="1" as="geometry">
|
||||
<mxPoint x="160" y="1510" as="sourcePoint"/>
|
||||
<mxPoint x="380" y="1510" as="targetPoint"/>
|
||||
</mxGeometry>
|
||||
</mxCell>
|
||||
<mxCell id="dm9" value="8. slot._send_command_tagged(Command(USER_STEP, payload=UserStepPayload(prompt, segment_idx, image_path, reset_conditioning)))" style="endArrow=classic;html=1;strokeColor=#82b366;fontSize=10;" parent="1" edge="1">
|
||||
<mxGeometry relative="1" as="geometry">
|
||||
<mxPoint x="380" y="1560" as="sourcePoint"/>
|
||||
<mxPoint x="810" y="1560" as="targetPoint"/>
|
||||
</mxGeometry>
|
||||
</mxCell>
|
||||
<mxCell id="dm10" value="9. command_queue.put(USER_STEP)" style="endArrow=classic;html=1;strokeColor=#b85450;fontSize=10;" parent="1" edge="1">
|
||||
<mxGeometry relative="1" as="geometry">
|
||||
<mxPoint x="810" y="1610" as="sourcePoint"/>
|
||||
<mxPoint x="1040" y="1610" as="targetPoint"/>
|
||||
</mxGeometry>
|
||||
</mxCell>
|
||||
<mxCell id="dm11a" value="10a. worker runs:
VideoGenerationWorker.generate_step()
 (video_generation.py:380)
 → generator.generate_video()
 → updates ContinuationState
then stream_fmp4() (av_streaming.py:121)
 → ffmpeg (rawvideo+wav → fmp4)" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#ffe0b2;strokeColor=#d79b00;fontSize=10;align=left;spacingLeft=8;fontFamily=monospace;" parent="1" vertex="1">
|
||||
<mxGeometry x="955" y="1640" width="180" height="70" as="geometry"/>
|
||||
</mxCell>
|
||||
<mxCell id="dm11" value="10b. resp_q.put(MediaInit / MediaChunk / MediaComplete / StepComplete)" style="endArrow=classic;html=1;strokeColor=#b85450;fontSize=10;labelBackgroundColor=#ffffff;" parent="1" edge="1">
|
||||
<mxGeometry x="0.0435" relative="1" as="geometry">
|
||||
<mxPoint x="1092" y="1733" as="sourcePoint"/>
|
||||
<mxPoint x="862" y="1733" as="targetPoint"/>
|
||||
<mxPoint as="offset"/>
|
||||
</mxGeometry>
|
||||
</mxCell>
|
||||
<mxCell id="dm12" value="11. _response_reader isinstance-dispatches: MediaChunk/etc → _stream_queues[user_id]; StepComplete → _pending_futures" style="endArrow=classic;html=1;strokeColor=#d6b656;fontSize=10;" parent="1" edge="1">
|
||||
<mxGeometry x="0.0014" relative="1" as="geometry">
|
||||
<mxPoint x="810" y="1760" as="sourcePoint"/>
|
||||
<mxPoint x="380" y="1760" as="targetPoint"/>
|
||||
<mxPoint as="offset"/>
|
||||
</mxGeometry>
|
||||
</mxCell>
|
||||
<mxCell id="dm13a" value="12a. binary WS frame (mp4 bytes)" style="endArrow=classic;html=1;strokeColor=#6c8ebf;fontSize=10;labelBackgroundColor=#ffffff;" parent="1" edge="1">
|
||||
<mxGeometry x="-0.0909" relative="1" as="geometry">
|
||||
<mxPoint x="380" y="1795" as="sourcePoint"/>
|
||||
<mxPoint x="160" y="1795" as="targetPoint"/>
|
||||
<mxPoint as="offset"/>
|
||||
</mxGeometry>
|
||||
</mxCell>
|
||||
<mxCell id="dm13b" value="12b. JSON WS event (media_init, media_segment_complete, prompt_ready, …)" style="endArrow=classic;html=1;strokeColor=#6c8ebf;fontSize=10;labelBackgroundColor=#ffffff;" parent="1" edge="1">
|
||||
<mxGeometry x="-0.0909" y="5" relative="1" as="geometry">
|
||||
<mxPoint x="380" y="1825" as="sourcePoint"/>
|
||||
<mxPoint x="160" y="1825" as="targetPoint"/>
|
||||
<mxPoint as="offset"/>
|
||||
</mxGeometry>
|
||||
</mxCell>
|
||||
<mxCell id="dm14" value="13. on WS close: pool.release(client_id) → Command(USER_LEAVE) → LeaveAck" style="endArrow=classic;html=1;strokeColor=#c62828;fontSize=10;dashed=1;labelBackgroundColor=#ffffff;" parent="1" edge="1">
|
||||
<mxGeometry relative="1" as="geometry">
|
||||
<mxPoint x="160" y="1865" as="sourcePoint"/>
|
||||
<mxPoint x="1040" y="1865" as="targetPoint"/>
|
||||
</mxGeometry>
|
||||
</mxCell>
|
||||
</root>
|
||||
</mxGraphModel>
|
||||
</diagram>
|
||||
</mxfile>
|
||||
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 85 KiB |
@@ -0,0 +1,41 @@
|
||||
[project]
|
||||
name = "dreamverse"
|
||||
version = "0.1.0"
|
||||
description = "Dreamverse application with FastAPI backend and Next.js frontend."
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.10"
|
||||
dependencies = [
|
||||
"fastapi>=0.115,<1",
|
||||
"fastvideo>=0.1.7",
|
||||
"pillow>=10.0",
|
||||
"uvicorn[standard]>=0.30,<1",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
server = [
|
||||
"cerebras-cloud-sdk",
|
||||
"flash-attn-cute @ git+https://github.com/XOR-op/flash-attention.git@fa4-compile#subdirectory=flash_attn/cute",
|
||||
"flashinfer-python",
|
||||
"openai>=1.40",
|
||||
]
|
||||
safety = [
|
||||
"fasttext>=0.9.2",
|
||||
]
|
||||
test = [
|
||||
"pytest>=8.0",
|
||||
]
|
||||
|
||||
[project.scripts]
|
||||
dreamverse-server = "dreamverse.server_entry:cli"
|
||||
dreamverse-mock-server = "dreamverse.mock_server:cli"
|
||||
|
||||
[tool.uv]
|
||||
package = false
|
||||
|
||||
[tool.uv.sources]
|
||||
fastvideo = { workspace = true }
|
||||
|
||||
[tool.pytest.ini_options]
|
||||
markers = [
|
||||
"gpu: requires real GPU + model weights; skip in CI",
|
||||
]
|
||||
Reference in New Issue
Block a user