From 2a8a7a01a9752cdb06c79b89e55c6e56e560198a Mon Sep 17 00:00:00 2001 From: Davids048 Date: Sun, 10 May 2026 14:41:02 -0700 Subject: [PATCH] [docs] Add Dreamverse app documentation Apply Dreamverse monorepo changes for stack slice 2/13 from the source branch. Source-Branch: will/dreamverse-monorepo Source-SHA: 03d3e61df69fe99a81eb62d86a3926a5f769f857 Dreamverse-Stack: 2/13 Co-authored-by: SolitaryThinker Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com> Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com> Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com> --- apps/dreamverse/AGENTS.md | 203 +++++++++++ apps/dreamverse/README.md | 234 +++++++++++++ apps/dreamverse/arch.md | 351 +++++++++++++++++++ apps/dreamverse/design.md | 594 ++++++++++++++++++++++++++++++++ apps/dreamverse/gpu-pool.drawio | 577 +++++++++++++++++++++++++++++++ apps/dreamverse/gpu-pool.svg | 1 + apps/dreamverse/pyproject.toml | 41 +++ 7 files changed, 2001 insertions(+) create mode 100644 apps/dreamverse/AGENTS.md create mode 100644 apps/dreamverse/README.md create mode 100644 apps/dreamverse/arch.md create mode 100644 apps/dreamverse/design.md create mode 100644 apps/dreamverse/gpu-pool.drawio create mode 100644 apps/dreamverse/gpu-pool.svg create mode 100644 apps/dreamverse/pyproject.toml diff --git a/apps/dreamverse/AGENTS.md b/apps/dreamverse/AGENTS.md new file mode 100644 index 000000000..658525c62 --- /dev/null +++ b/apps/dreamverse/AGENTS.md @@ -0,0 +1,203 @@ +# Dreamverse Agent Notes + +## Repo-local skills + +- `.agents/skills/bootstrap-fastvideo-private-fork/`: temporary setup skill for + cloning `git@github.com:hao-ai-lab/FastVideo-internal.git` at + `will/rebase-nbv` into `../FastVideo-internal`, then running + `uv sync --extra server`. +- Prefer the bundled script in that skill instead of inventing a new private + FastVideo bootstrap flow. + +## Repo layout + +Current paths: + +- `apps/web/`: Next.js frontend, client-side stores, websocket event reduction, + prompt-window editing, devtools UI. +- `server/`: current Python FastAPI runtime, websocket protocol, prompt + enhancement, prompt rewrite orchestration, GPU worker lifecycle. +- `server/tests/`: backend unit and integration-oriented tests. +- `server/benchmarks/`: prompt-provider latency/token benchmarking scripts. + +Planned paths during the OSS reorg: + +- `controller/`: local control plane for provider credentials, compute + lifecycle, and proxying. +- `runtime/`: eventual rename of `server/` once the controller/runtime split is + stable. +- `providers/`: provider adapters for local, Runpod, and Modal. + +Important rule: + +- Until the split lands, treat `server/` as the authoritative runtime and keep + provider orchestration out of `apps/web`. + +## System split + +Dreamverse is moving toward a three-part local-first architecture. + +- Frontend owns local UI state, drafts, inspection tools, and user-triggered + actions. +- Controller will own local-only credentials, compute provisioning, runtime + lifecycle, and HTTP/websocket proxying. +- Runtime owns generation sessions, prompt rewrite, prompt safety, and + websocket semantics. + +The browser should only talk to the local Dreamverse process, never directly to +Modal or Runpod. + +## Current runtime responsibilities + +The runtime in `server/` is responsible for: + +- websocket session lifecycle on `/ws` +- queueing, GPU assignment, worker startup, and stream chunk emission +- seed prompt memory and the active prompt window used for generation +- prompt enhancement and prompt rewrite execution +- prompt safety checks +- persistence and reload of prompt system prompt files +- curated preset append/read routes in devtools mode +- health and readiness endpoints + +Relevant files: + +- `server/main.py`: websocket protocol, session state machine, REST routes +- `server/gpu_pool.py`: FastVideo-backed generation workers +- `server/prompt_enhancer.py`: provider clients, prompt enhancement, rewrite + execution +- `server/rewrite_prompt_payload.py`: canonical rewrite request body format +- `server/config.py`: prompt file paths, provider configuration, runtime flags + +## Frontend responsibilities + +The frontend in `apps/web/` is responsible for: + +- collecting user input and deciding whether to send raw prompts or rewrite + requests +- maintaining client-side stores for session, prompt-window, stream, rewrite, + and UI state +- rendering prompt history, playback state, devtools controls, and rewrite + inspection +- building the prompt-window snapshot sent with rewrite requests +- reducing websocket events into UI state +- showing compute status and controller-driven errors once the controller lands + +Relevant files: + +- `apps/web/src/app/page.tsx`: main orchestration, websocket connect/send paths +- `apps/web/src/lib/ws/reducer.ts`: applies normalized websocket events to + stores +- `apps/web/src/stores/promptWindow.ts`: prompt window and preset/editor state +- `apps/web/src/stores/rewrite.ts`: rewrite activity timeline and flags +- `apps/web/src/lib/prompts/promptWindowSnapshot.ts`: rewrite snapshot + normalization and padding + +## Planned controller responsibilities + +The future local controller should own: + +- local-only provider credential loading and storage +- provider selection +- runtime provisioning, reuse, shutdown, and health checks +- proxying frontend HTTP and websocket traffic to the active runtime +- surfacing provisioning, ready, failed, and idle states to the frontend +- durable local settings that should survive ephemeral remote runtimes + +The controller should not own: + +- prompt rewrite logic +- seed prompt memory +- generation queue semantics +- websocket event schemas + +## Prompt rewrite contract + +Prompt rewrite is a shared flow with a strict ownership split. + +Frontend responsibilities: + +- decide when a user action should trigger `rewrite_seed_prompts` instead of + `append_prompt` +- send `rewrite_instruction` and a snapshot of the current prompt window +- pad the rewrite snapshot to the runtime-expected segment count using + `buildRewritePromptWindowSnapshotFromPrompts(...)` +- show rewrite activity and raw LLM output in local inspection UI + +Runtime responsibilities: + +- validate and normalize `prompt_window_prompts` +- choose the rewrite system prompt and provider/model/temperature +- build the canonical LLM request body in `server/rewrite_prompt_payload.py` +- run the rewrite through `PromptEnhancer.rewrite_prompt_sequence(...)` +- apply safety filtering to rewritten prompts +- replace the authoritative seed prompt memory when rewrite succeeds +- emit `seed_prompts_updated` and `rewrite_seed_prompts_complete` + +Controller responsibilities: + +- proxy the request and response +- surface runtime availability and provider lifecycle failures + +Important rule: + +- The frontend may suggest the prompt window to rewrite, but the runtime owns + the actual rewritten rollout and the authoritative prompt window after + acceptance. + +## Rewrite modes + +There are two runtime rewrite modes: + +- edit existing rollout: when `prompt_window_prompts` is non-empty, rewrite the + current rollout while preserving segment count and ordering +- new rollout: when the prompt window is empty but there is a + `rewrite_instruction`, generate a fresh rollout + +The frontend should not emulate runtime rewrite behavior locally. It should +prepare the snapshot, send it, and display the result. + +## Prompt window ownership + +- Frontend owns editable drafts, selected preset UI, and prompt-window + inspection state. +- Runtime owns the active seed prompt memory used for actual generation. +- After any runtime event with reason `rewrite`, the frontend must replace its + prompt window from the server payload instead of keeping a locally-derived + version. + +## Devtools and persistence ownership + +Prompt config editing is runtime-owned persistence with frontend-owned forms +today. + +- Frontend loads and edits drafts through `/prompt-system-config`. +- Runtime reads and writes prompt files and reloads runtime prompt config. + +Curated presets follow the same pattern: + +- frontend submits append requests and may update local UI optimistically from + the response +- runtime persists the JSON file and resolves overlay vs fallback file paths + +During the controller reorg, avoid moving durable user settings into ephemeral +remote runtimes. Controller-owned local persistence is preferred for anything +that must survive provider restarts. + +## Editing guidance + +- Do not move rewrite logic into the frontend or controller. +- Do not make the frontend the source of truth for the generated prompt window + after rewrite. +- If you change websocket message types or payload fields in `server/main.py`, + update the reducer in `apps/web/src/lib/ws/reducer.ts` in the same change. +- If you change rewrite request shape, update both + `apps/web/src/lib/prompts/promptWindowSnapshot.ts` and + `server/rewrite_prompt_payload.py`. +- If you add controller-managed status or error payloads, keep them separate + from runtime websocket events unless there is a strong reason to merge them. +- If you change prompt file paths or devtools persistence, update `server/`, + the frontend devtools UI, and any controller-owned local persistence logic + together. +- Keep provider adapters focused on runtime lifecycle and reachability, not on + prompt or session semantics. diff --git a/apps/dreamverse/README.md b/apps/dreamverse/README.md new file mode 100644 index 000000000..4dfb82306 --- /dev/null +++ b/apps/dreamverse/README.md @@ -0,0 +1,234 @@ +# Dreamverse + +Dreamverse is the FastVideo realtime video generation & editing platform. It lives in this monorepo under `apps/dreamverse/`. + +## Install Dreamverse + +You can install Dreamverse using one of the methods below. + +### Method 1: With uv pip + +```bash +pip install --upgrade pip +pip install uv +uv venv .venv --python 3.12 +source .venv/bin/activate +uv pip install "fastvideo[dreamverse]" +``` + +### Method 2: From source + +```bash +git clone https://github.com/hao-ai-lab/FastVideo.git +cd FastVideo + +pip install --upgrade pip +pip install uv +uv venv .venv --python 3.12 +source .venv/bin/activate +uv pip install -e ".[dreamverse]" +``` + +### Method 3: Using Docker + +```bash +git clone https://github.com/hao-ai-lab/FastVideo.git +cd FastVideo + +apps/dreamverse/docker/docker_build.sh +``` + +See `apps/dreamverse/docker/README.md` for Docker build and run option details. + +## Optional: Building FFmpeg For Better Performance + +For full streaming performance in a non-Docker install, build a custom FFmpeg +binary: + +```bash +bash apps/dreamverse/scripts/install_native_ffmpeg.sh +``` + +This builds and installs into `~/opt/ffmpeg-native/` and writes +`apps/dreamverse/scripts/ffmpeg-env.sh`. Source it before starting the backend +so Dreamverse uses the custom FFmpeg binary: + +```bash +source apps/dreamverse/scripts/ffmpeg-env.sh +dreamverse-server +``` + +Docker images already run this FFmpeg build during image creation and source the +generated environment file at container startup. The installer supports Linux +`x86_64` and `aarch64`. + +## Launch Dreamverse + +Start the backend with the installed Dreamverse commands: + +```bash +dreamverse-server --port 8009 +dreamverse-mock-server --port 8009 +``` + +## Frontend Setup + +Install the web dependencies once from the FastVideo checkout: + +```bash +cd apps/dreamverse/web +pnpm install --frozen-lockfile +``` + +The frontend package also has an npm lockfile, but the bundled launch scripts +use `pnpm`. + +## Quick Start: Local GPU + +### Start Backend + +Export the API keys used for prompt rewrite and prompt enhancement: + +```bash +export CEREBRAS_API_KEY=... +export GROQ_API_KEY=... +``` + +If you built the optional native FFmpeg binary above, source its environment +file in the same shell before starting the backend: + +```bash +source apps/dreamverse/scripts/ffmpeg-env.sh +dreamverse-server --host 0.0.0.0 --port 8009 +``` + +The Dreamverse backend defaults to `0.0.0.0:8009` and starts one GPU worker on +the first visible GPU by default. + +### Check Readiness + +In another shell, verify that the backend process is alive: + +```bash +curl http://localhost:8009/healthz +``` + +Then wait for GPU workers and startup warmup to finish: + +```bash +curl http://localhost:8009/readyz +``` + +You can also run the same readiness path with: + +```bash +BACKEND_HOST=localhost BACKEND_PORT=8009 apps/dreamverse/scripts/smoke_local.sh +``` + +If a backend is already running and you only want the script to probe it: + +```bash +DREAMVERSE_SMOKE_START_BACKEND=0 apps/dreamverse/scripts/smoke_local.sh +``` + +### Start Frontend + +Start the frontend: + +```bash +cd apps/dreamverse/web +BACKEND_HOST=localhost BACKEND_PORT=8009 pnpm run dev +``` + +Open `http://localhost:5299`. + +## Quick Start: Mock Backend (For UI development) + +The mock server emulates the Dreamverse backend protocol and streams a +synthetic FFmpeg-generated fMP4 clip, so the frontend can run without a GPU. + +```bash +dreamverse-mock-server --latency 200 --port 8009 +``` + +## Tests + +Run the focused backend tests that validate local startup wiring, config, GPU +selection, and mock-server behavior: + +```bash +pytest apps/dreamverse/dreamverse/tests/test_config.py \ + apps/dreamverse/dreamverse/tests/test_entrypoints.py \ + apps/dreamverse/dreamverse/tests/test_gpu_pool.py \ + apps/dreamverse/dreamverse/tests/test_mock_server.py -q +``` + +Run the broader Dreamverse backend suite: + +```bash +pytest apps/dreamverse/dreamverse/tests -q +``` + +Run the frontend tests: + +```bash +cd apps/dreamverse/web +pnpm test +``` + +Run the frontend e2e tests: + +```bash +cd apps/dreamverse/web +pnpm run e2e +``` + +## Troubleshooting + +`dreamverse-server` exits with an install hint + +- install the Dreamverse extra with `uv pip install -e ".[dreamverse]"` from a + source checkout, or `uv pip install "fastvideo[dreamverse]"` from PyPI. + +Prompt-provider environment variable errors + +- set `CEREBRAS_API_KEY` +- set `GROQ_API_KEY` +- direct `dreamverse-server` launches do not source `~/.env`; export the keys + in the shell or use the bundled launch scripts, which source `~/.env`. + +`/readyz` stays at `503` + +- wait for model loading and startup warmup to finish +- confirm a compatible CUDA GPU is visible to the process +- check backend logs for worker startup or warmup failures +- for a startup/debug pass without warmup, set + `FASTVIDEO_ENABLE_STARTUP_WARMUP=0` before starting the backend + +Only one GPU is used + +- this is the default local behavior +- set `FASTVIDEO_GPU_COUNT=` to start N GPU worker subprocesses inside one + backend instance +- set `FASTVIDEO_GPU_COUNT=all` to start one worker for every visible GPU +- use `CUDA_VISIBLE_DEVICES` first if you need to pin the visible GPU set + +Frontend cannot connect to backend + +- confirm the backend is running on `8009`; if not, point the frontend at it + with `BACKEND_HOST= BACKEND_PORT= pnpm run dev` +- confirm `http://localhost:8009/healthz` responds before starting the frontend +- confirm `http://localhost:8009/readyz` returns `200` before clicking Generate +- use `apps/dreamverse/scripts/smoke_local.sh` for a repeatable local startup + check + +Mock backend fails during startup + +- install FFmpeg or set `FASTVIDEO_FFMPEG_BIN` to an FFmpeg binary +- for non-mock local GPU streaming performance, use the native FFmpeg installer + described above + +## Notes + +Dreamverse owns its backend app under `apps/dreamverse/dreamverse/`. It expects +`dreamverse-server`, not `fastvideo serve`. diff --git a/apps/dreamverse/arch.md b/apps/dreamverse/arch.md new file mode 100644 index 000000000..03101a53b --- /dev/null +++ b/apps/dreamverse/arch.md @@ -0,0 +1,351 @@ +# Dreamverse Architecture + +## Overview + +Dreamverse currently has two main runtime pieces: + +- `apps/web/`: Next.js frontend +- `server/`: Python FastAPI runtime + +Today, the browser talks directly to the Dreamverse runtime over HTTP and a +single websocket on `/ws`. The frontend owns UI state and interaction flow. The +server owns generation state, prompt rewrite, prompt safety, websocket session +semantics, and GPU-backed execution. + +Near-term OSS note: + +- `server/` is the current runtime implementation. +- A future `controller/` layer is planned for local-only compute management and + provider orchestration, but it does not exist yet. + +## Repo Map + +### Frontend + +- `apps/web/src/app/page.tsx`: main client orchestration, websocket connect, + init payloads, send paths, and top-level app behavior +- `apps/web/src/lib/ws/reducer.ts`: reduces normalized websocket events into + client stores +- `apps/web/src/stores/session.ts`: connection, mode, and top-level session UI +- `apps/web/src/stores/promptWindow.ts`: editable prompt window and seed prompt + UI state +- `apps/web/src/stores/rewrite.ts`: rewrite activity timeline and inspection + state +- `apps/web/src/stores/stream.ts`: playback and stream-related client state +- `apps/web/src/lib/prompts/promptWindowSnapshot.ts`: prompt-window snapshot + building for rewrite requests + +### Server + +- `server/main.py`: websocket endpoint, request handling, session state + machine, rewrite orchestration, REST routes, and stream relay +- `server/gpu_pool.py`: GPU worker processes, warmup, model loading, and + `generate_video()` calls through FastVideo +- `server/prompt_enhancer.py`: prompt enhancement, rollout rewrite execution, + provider selection, and timeout/fallback behavior +- `server/rewrite_prompt_payload.py`: canonical rewrite request payload + building +- `server/config.py`: runtime flags, prompt file paths, provider settings, and + warmup config +- `server/session_init_image.py`: validates and persists uploaded initial + images for segment 1 + +## Current Split Of Responsibility + +### Frontend owns + +- local UI state and client-side stores +- prompt drafts and prompt window editing +- websocket connection management +- deciding which user action to send: + - `session_init_v2` + - `project_init_v1` + - `append_prompt` + - `rewrite_seed_prompts` + - `simple_generate` +- showing rewrite progress, stream status, prompt history, and devtools views + +### Server owns + +- websocket session lifecycle and protocol +- GPU assignment and worker lifecycle +- the authoritative seed prompt memory used for generation +- prompt rewrite execution and prompt safety +- actual generation queue semantics +- stream chunk emission and segment lifecycle events +- prompt config and preset persistence routes +- health and readiness endpoints + +Important rule: + +- The frontend may propose prompt-window state for rewrite, but the server is + the source of truth for the rewritten rollout and the active prompt memory + used for generation. + +## End-To-End Flow + +1. The frontend opens `/ws`. +2. The frontend sends `session_init_v2` with the initial prompt-window state, + preset metadata, and current toggles. +3. The server validates init data, persists an optional initial image, acquires + a GPU slot, and emits session status such as `gpu_assigned`. +4. The server starts or resumes project generation and emits events like + `ltx2_stream_start`, `ltx2_segment_start`, media init/chunks, and completion + events. +5. The frontend reduces those websocket events into its stores and updates the + UI. +6. User actions such as appending prompts, rewriting seed prompts, or starting + a single custom clip go back to the server over the same websocket. + +## Frontend Architecture + +The frontend is store-driven. + +- `page.tsx` wires together websocket setup, send helpers, reducer + application, and top-level interaction flows. +- Store modules separate concerns like session state, rewrite state, prompt + window state, and stream state. +- The websocket reducer is responsible for turning normalized runtime events + into store updates. If the server event schema changes, the reducer must + change with it. + +The frontend is intentionally not responsible for: + +- generating rewritten prompts locally +- deciding final prompt safety outcomes +- reconstructing server session state from scratch +- inventing its own generation semantics independent of the runtime + +## Server Architecture + +The current runtime is a FastAPI app with a single long-lived websocket per +session. + +`server/main.py` manages: + +- websocket connect/init +- prompt queues +- project and segment state +- prompt enhancement/rewrite triggers +- stream relay from GPU workers to the browser +- session logging and REST endpoints + +`server/gpu_pool.py` manages: + +- model loading through FastVideo +- one or more worker processes +- startup warmup +- user join/leave commands +- `USER_STEP` execution for each segment +- continuation state between segments + +`server/prompt_enhancer.py` manages: + +- prompt enhancement for user-submitted prompts +- rollout rewrite requests for the prompt window +- provider selection and fallback across configured prompt providers +- response normalization and safety-aware failure handling + +## FastAPI Surface + +The server is a single FastAPI application created in +`server/main.py`. + +Current built-in FastAPI docs are enabled: + +- `/docs`: Swagger UI +- `/redoc`: ReDoc +- `/openapi.json`: OpenAPI schema + +The runtime also mounts static content: + +- `/server-assets`: server-local static assets rooted at `server/` +- `/`: frontend static build when one of the configured frontend static + directories exists + +## HTTP API + +The current HTTP API is small. Most realtime behavior still goes through the +websocket. + +### Core health and status routes + +- `GET /healthz` + - process liveness probe + - returns a small payload with `status`, `service`, and timestamp +- `GET /readyz` + - readiness probe + - returns `503` until prompt services are initialized and at least one GPU + worker is ready + - returns readiness and GPU pool summary fields such as ready workers, total + GPUs, warmup counts, and queue size +- `GET /status` + - returns the current GPU pool status payload from `gpu_pool` +- `GET /internal/monitor/sessions` + - internal monitoring payload for session dashboards + - includes pending session count, max available sessions, prompt provider + success counts, and timestamp + +### Prompt config routes + +- `GET /prompt-system-config` + - returns the editable prompt-system configuration currently loaded by + `PromptEnhancer` +- `POST /prompt-system-config` + - saves prompt-system configuration to disk and reloads prompt config in the + runtime + - current editable fields include: + - next-segment system prompt + - auto-extension system prompt + - rewrite-window system prompt + - rewrite-user system prompt + - rewrite model + - rewrite temperature + +### Devtools-only preset routes + +These exist only when `DEVTOOLS_ENABLED` is true in `server/config.py`. + +- `GET /curated-presets` + - returns merged curated presets, applying the local overlay file on top of + the fallback file when both exist +- `POST /curated-presets/append` + - appends a new curated preset to the overlay presets file + - validates non-empty label, normalized id, and at least two non-empty + segment prompts + +## Websocket API + +`WS /ws` is the main runtime API. + +The websocket owns: + +- session init +- project init and reset +- prompt append +- prompt rewrite +- generation toggles +- segment lifecycle events +- media stream delivery +- runtime error delivery + +The websocket is the authoritative API for realtime Dreamverse behavior. The +HTTP routes mainly support health checks, devtools persistence, and monitoring. + +## Prompt Rewrite Architecture + +Prompt rewrite is a shared flow with strict ownership boundaries. + +### Frontend responsibilities + +- collect the rewrite instruction +- build the prompt-window snapshot from current client state +- send `rewrite_seed_prompts` +- show rewrite progress, raw output, fallback state, and resulting prompt list + +### Server responsibilities + +- validate and normalize the prompt-window payload +- choose rewrite model, system prompt, timeout, and temperature +- build the canonical prompt payload in `server/rewrite_prompt_payload.py` +- execute rewrite through `PromptEnhancer` +- apply safety filtering to rewritten prompts +- replace the authoritative seed prompt memory when rewrite succeeds +- emit `seed_prompts_updated` and `rewrite_seed_prompts_complete` + +Important rule: + +- The frontend owns editable drafts. +- The server owns the accepted rollout. + +After a successful rewrite, the frontend should replace its prompt-window view +from the server payload instead of preserving a locally-derived version. + +## Prompt Modes + +There are three related prompt paths in the current system: + +### Initial rollout + +- The frontend sends seed prompts during `session_init_v2`. +- The server uses those prompts as the initial seed prompt memory. +- If the rollout starts from an empty prompt window plus an initial rewrite + instruction, the server can pause generation until rewrite completes. + +### Live append + +- The frontend sends `append_prompt`. +- The server may enhance that prompt, safety-check it, enqueue it, and use it + as the next generated segment. + +### Rewrite + +- The frontend sends `rewrite_seed_prompts`. +- The server rewrites the entire seed prompt window or generates a new rollout, + depending on the payload and current state. + +## Initial Image And Segment Handling + +The frontend currently sends `initial_image` as part of session init or +`simple_generate`. + +The server: + +- validates and persists the image +- uses it only for segment 1 when present +- keeps continuation state for later segments in the GPU worker + +This means the runtime, not the frontend, decides how segment 1 image +conditioning and later continuation conditioning are applied. + +## Websocket Contract + +The websocket is the main integration surface between UI and runtime. + +Typical incoming messages from the frontend: + +- `session_init_v2` +- `project_init_v1` +- `append_prompt` +- `rewrite_seed_prompts` +- `simple_generate` +- `set_enhancement` +- `set_auto_extension` +- `set_loop_generation` + +Typical outgoing messages from the server: + +- `gpu_assigned` +- `ltx2_stream_start` +- `ltx2_segment_start` +- `segment_prompt_source` +- `prompt_received` +- `prompt_ready` +- `prompt_enhancing` +- `seed_prompts_updated` +- `rewrite_seed_prompts_complete` +- `media_init` +- `media_segment_complete` +- `project_idle` +- `error` + +Binary websocket frames carry media chunks for playback. + +## Current And Planned Architecture + +Current architecture: + +- browser -> `apps/web` +- `apps/web` -> `server/main.py` +- `server/main.py` -> `gpu_pool.py` +- `gpu_pool.py` -> FastVideo runtime + +Planned architecture: + +- browser -> `apps/web` +- `apps/web` -> local `controller/` +- `controller/` -> local or remote Dreamverse runtime +- runtime -> FastVideo runtime + +That future controller split should not move prompt rewrite, session state, or +generation semantics out of the runtime. diff --git a/apps/dreamverse/design.md b/apps/dreamverse/design.md new file mode 100644 index 000000000..773a0fd38 --- /dev/null +++ b/apps/dreamverse/design.md @@ -0,0 +1,594 @@ +# Dreamverse OSS Design + +## Overview + +Dreamverse should ship as a local-first open source application. + +- The browser talks only to a local Dreamverse control plane on the user's + machine. +- Provider credentials stay local to that machine. +- Dreamverse may provision compute on the user's behalf, but Dreamverse does + not host that control path as a service. + +This keeps the UX simple without turning Dreamverse into a credential-holding +hosted platform. + +## Goals + +- Support three compute modes behind one product surface: + - local GPU + - managed remote GPU via Runpod + - managed remote GPU via Modal +- Keep prompt rewrite, websocket session state, and generation behavior + consistent across providers. +- Keep provider API keys out of browser state and out of any hosted service. +- Make the existing runtime reusable as the common serving contract. +- Minimize provider-specific code and isolate it behind a narrow interface. + +## Non-goals + +- Do not make the frontend call provider APIs directly. +- Do not unify providers at the level of SSH, VM, serverless, or pod + semantics. +- Do not move prompt rewrite logic into the frontend or controller. +- Do not require remote compute for the basic product path. + +## Current State + +Today the repo contains two major pieces: + +- `apps/web/`: Next.js frontend +- `server/`: FastAPI runtime that owns websocket state, prompt rewrite, prompt + safety, and GPU-backed generation + +The current runtime already exposes useful health and streaming surfaces such +as `/healthz`, `/readyz`, `/status`, and `/ws`. + +## Target Architecture + +The target open source structure should be: + +```text +Dreamverse/ +├── apps/ +│ └── web/ # browser UI +├── controller/ # local control plane and provider lifecycle +├── runtime/ # Dreamverse websocket/generation runtime +├── providers/ # provider adapters +├── tests/ +│ ├── contract/ +│ ├── controller/ +│ └── smoke/ +└── design.md +``` + +Near-term note: + +- `server/` is the current runtime implementation. +- We can keep the code there initially and rename it to `runtime/` only after + the controller lands. + +## Trust Model + +Dreamverse is local-only for control and secrets. + +- The user launches Dreamverse on their own machine. +- Provider API keys are entered into the local app or local CLI. +- The controller uses those credentials to provision or connect to compute. +- The browser never talks to Modal or Runpod directly. +- Dreamverse-hosted infrastructure is not involved. + +This is the key reason the provider-based path is acceptable for OSS. + +## Responsibility Split + +### `apps/web` + +The frontend should own: + +- UI state, drafts, and local interaction state +- websocket event reduction into client stores +- selection of compute mode and display of cost/health/status +- local forms for provider configuration +- sending prompt requests and rewrite requests to the local controller + +The frontend should not own: + +- provider credentials after submission +- provider API calls +- runtime lifecycle +- authoritative prompt window after rewrite +- prompt safety or generation policy + +### `controller` + +The local controller should own: + +- provider credential loading and local-only storage +- compute mode selection +- provisioning, reuse, shutdown, and health monitoring of runtimes +- reverse proxying HTTP and websocket traffic from the frontend to the active + runtime +- user-visible status such as provisioning, ready, failed, and idle shutdown +- local persistence for user settings that must survive ephemeral runtimes + +The controller should not own: + +- prompt rewrite logic +- seed prompt memory semantics +- generation queue behavior +- provider-specific UI state + +### `runtime` + +The runtime should remain the authoritative owner of: + +- `/ws` session state +- prompt rewrite execution +- prompt safety +- seed prompt memory and prompt-window state used for generation +- generation orchestration and GPU worker lifecycle +- websocket event schemas + +This preserves the current model and avoids splitting state across layers. + +## Runtime Contract + +Provider abstraction should happen around a stable Dreamverse runtime contract, +not around infrastructure details. + +Minimum runtime surface: + +- `GET /healthz` +- `GET /readyz` +- `GET /status` +- `GET/POST /prompt-system-config` if devtools persists config through the + runtime +- curated preset routes if those remain runtime-backed +- `WS /ws` + +Important rule: + +- The controller only needs to know how to reach a healthy runtime. +- The runtime remains provider-agnostic. + +## Provider Abstraction + +Use a narrow provider interface: + +```python +class ComputeProvider(Protocol): + async def ensure_runtime(self, spec: RuntimeSpec) -> RuntimeHandle: ... + async def wait_until_ready(self, handle: RuntimeHandle) -> None: ... + async def stop_runtime(self, handle: RuntimeHandle) -> None: ... +``` + +`RuntimeHandle` should include: + +- `provider` +- `runtime_id` +- `base_url` +- `ws_url` +- runtime auth headers or tokens if needed +- lifecycle metadata +- cost or hardware metadata for UI display + +The controller should work only with `RuntimeHandle`, never with raw SSH hosts +or provider-specific payloads after resolution. + +## Provider Notes + +### Local + +Local mode should be the reference implementation. + +- Start the runtime as a local subprocess or connect to an already-running + local runtime URL. +- Reuse the same runtime contract as remote providers. +- Make this the first supported path and the main smoke-test target. + +### Runpod + +Runpod should be treated as pod lifecycle plus runtime reachability. + +- Prefer prepared images or templates that auto-start the Dreamverse runtime. +- Prefer exposed HTTP/TCP ports for steady-state traffic. +- Use SSH only for bootstrap fallback, diagnostics, or repair. +- Avoid a design where the controller shells into the pod for every action. + +### Modal + +Modal should be treated as deployment-based runtime hosting. + +- Wrap the Dreamverse runtime in a thin Modal entrypoint if needed. +- Reuse the same runtime behavior behind that wrapper. +- Do not model Modal as a machine that Dreamverse logs into. +- Do not force the websocket runtime into a per-request serverless handler + shape. + +## Config and Persistence + +Remote compute may be ephemeral, so mutable user configuration should not live +only inside remote runtimes. + +Keep durable state local to the user's machine unless there is a strong reason +otherwise: + +- provider selection +- provider credentials or credential references +- default hardware preferences +- editable prompt presets +- prompt system prompt overrides +- idle shutdown policy + +Runtime-local state should be treated as disposable unless explicitly synced. + +## Prompt Rewrite Ownership + +Prompt rewrite remains runtime-owned even after the controller is added. + +Frontend responsibilities: + +- collect the rewrite instruction +- build the prompt-window snapshot +- display rewrite activity and results + +Runtime responsibilities: + +- validate and normalize the prompt window +- choose the rewrite system prompt and model settings +- execute rewrite +- apply safety filtering +- replace authoritative seed prompt memory +- emit the canonical completion events + +Controller responsibilities: + +- proxy the request and response +- surface runtime availability and failure state + +This boundary should not move. + +## Recommended Rollout + +1. Finish the path reorg so docs and code agree on `apps/web`. +2. Introduce `controller/` as a local-only API/proxy process. +3. Keep `server/` as the runtime and adapt it behind the controller. +4. Add `local` provider first. +5. Add "bring your own runtime URL" as an escape hatch. +6. Add automated Runpod provisioning. +7. Add Modal deployment support. +8. Rename `server/` to `runtime/` once the split is stable. + +## Implementation Plan + +The implementation should start with the smallest milestone that gives users a +working local GPU setup without forcing the controller/provider architecture +into the first patch series. + +### Milestone 0: Make local GPU the official baseline + +Goal: + +- A user with a working `fastvideo` install can run the Dreamverse backend on a + local GPU and connect to it from `apps/web`. + +Non-goals for this milestone: + +- no controller process yet +- no provider abstraction yet +- no Runpod or Modal support yet +- no secret-management UI yet + +Reasoning: + +- `server/` already is the real local GPU runtime. +- `apps/web` already knows how to talk to a backend over `/ws` and REST + rewrites. +- The shortest path is to make the existing local path explicit, reliable, and + tested before adding another layer. + +### Milestone 0 work items + +#### 0.1 Fix repo path assumptions after the frontend move + +Current issue: + +- Some paths still assume `prod-ui/`, but the frontend now lives at + `apps/web/`. + +Required changes: + +- update prompt/preset path resolution in `server/config.py` +- update docs that still mention `prod-ui` +- audit any frontend build settings that assume the old repo root + +This is prerequisite cleanup. Local GPU mode should not depend on stale +monorepo paths. + +#### 0.2 Make local runtime startup the primary supported entrypoint + +Required outcome: + +- one documented backend command +- one documented frontend command +- one clear env contract for local development + +Expected shape: + +```bash +uv pip install -e ".[dreamverse]" +dreamverse-server --host 0.0.0.0 --port 8009 + +cd apps/dreamverse/web +npm ci +BACKEND_HOST=localhost BACKEND_PORT=8009 npm run dev +``` + +Optional but useful: + +- add a small root helper script or Make target for local startup +- add a `dreamverse-doctor` or lightweight startup check later + +#### 0.3 Define the minimum local runtime contract + +For Milestone 0, the frontend should rely only on the current runtime surface: + +- `/ws` +- `/status` +- `/healthz` +- `/readyz` +- existing prompt/devtools routes + +Do not add a second local API layer yet unless the current runtime surface is +proven insufficient. + +#### 0.4 Make failure states explicit in the UI + +Local GPU mode fails in a few predictable ways: + +- backend not reachable +- backend reachable but not ready +- `fastvideo` or model runtime missing +- no compatible GPU available + +Minimum implementation: + +- show a clear connection error when `/ws` or `/status` fails +- surface readiness failures in a human-readable way +- avoid silent retry loops that hide backend startup failures + +This is a small UI pass, not a controller project. + +#### 0.5 Add a minimal local smoke test path + +At this milestone, local GPU support is "done" only if there is a repeatable +test path for the local runtime contract. + +Minimum test additions: + +- backend tests for `/healthz`, `/readyz`, and `/status` +- a frontend integration test that assumes a reachable backend URL and verifies + connection lifecycle behavior +- one local smoke script that starts the backend and verifies readiness before + the frontend is launched + +### Milestone 1: Introduce a thin local controller + +Goal: + +- Preserve the same local GPU behavior, but place a stable local control-plane + API in front of the runtime. + +This should happen only after Milestone 0 is stable. + +Scope: + +- add `controller/` +- proxy `/ws` and the needed REST routes to `server/` +- expose controller-owned status for "backend starting", "runtime ready", and + "runtime failed" +- optionally spawn the local runtime as a subprocess + +Non-goal: + +- do not add remote provider logic yet + +Reasoning: + +- the controller earns its complexity only once it stabilizes the local + contract that future providers will share + +### Milestone 2: Provider abstraction on top of the controller + +Goal: + +- Keep the same frontend contract while allowing the controller to resolve a + runtime via `local`, then later `runpod` and `modal`. + +At this point: + +- define `ComputeProvider` +- implement `providers/local.py` +- move local-runtime subprocess management behind the provider interface + +The first provider should be `local`, because it is cheapest to debug and +matches the runtime most closely. + +## Minimal Code Change Order + +If we want the shortest path to a working local GPU milestone, the change order +should be: + +1. Fix `server/config.py` and any remaining path assumptions from `prod-ui` to + `apps/web`. +2. Update `README.md` to document the real local GPU startup flow. +3. Confirm `apps/dreamverse/web` connects cleanly to the local wrapper-backed backend. +4. Improve frontend error handling for backend-not-ready and backend-missing + cases. +5. Add a local smoke test and keep existing backend/frontend tests green. +6. Only then introduce `controller/`. + +## Test Plan for the Local GPU Milestone + +### Backend + +Keep the current Python test suite as the base: + +- `server/tests/test_health_endpoints.py` +- `server/tests/test_mock_server.py` +- `server/tests/test_prompt_enhancer.py` +- `server/tests/test_rewrite_prompt_payload.py` +- related config and logging tests + +Add or tighten tests for: + +- path resolution in `server/config.py` +- readiness behavior when GPU pool initialization fails +- startup error messaging when `fastvideo` is unavailable + +### Frontend + +Keep the current Vitest suite as the base: + +- websocket reducer tests +- prompt-window snapshot tests +- integration tests under `apps/web/src/app/` + +Add or tighten tests for: + +- connection failure UX when backend is down +- readiness failure UX when backend returns non-ready status +- backend routing configuration through `BACKEND_HOST` and `BACKEND_PORT` + +### Manual smoke path + +The first manual smoke checklist should be: + +1. start `dreamverse-server` +2. confirm `GET /healthz` returns 200 +3. confirm `GET /readyz` returns 200 after warmup +4. start `apps/web` +5. confirm the UI opens and the websocket connects +6. submit a prompt and verify the first generation starts + +This checklist should be written down in the README once the milestone is +implemented. + +## Test Strategy + +The test suite should preserve one rule: provider changes must not be able to +break prompt rewrite, websocket semantics, or runtime behavior silently. + +### 1. Runtime unit and integration tests + +Keep and expand the current `pytest` coverage in `server/test_*.py`. + +Focus areas: + +- config loading +- prompt rewrite payload normalization +- prompt enhancement and prompt safety +- health/readiness endpoints +- websocket session behavior +- session logging +- mock runtime behavior + +These tests should remain provider-agnostic. + +### 2. Controller unit tests + +Add a Python test suite for the controller state machine. + +Key cases: + +- provider selection and validation +- credential loading from local config or env +- runtime lifecycle transitions: + - idle + - provisioning + - ready + - failed + - stopping +- idle timeout and cleanup behavior +- retry and backoff behavior +- HTTP and websocket proxy routing + +These tests should use fake providers and fake runtimes by default. + +### 3. Provider contract tests + +Each provider should pass the same contract tests. + +Examples: + +- `ensure_runtime()` returns a usable `RuntimeHandle` +- `wait_until_ready()` surfaces timeout vs readiness correctly +- `stop_runtime()` is safe to call twice +- provider errors are mapped into stable controller error types + +Use recorded fixtures or fakes wherever possible to avoid spend in CI. + +### 4. Web contract tests + +The frontend already has useful Vitest coverage under `apps/web/src`. +Preserve that and expand around the controller split. + +Priority areas: + +- websocket event reduction +- prompt-window snapshot construction +- rewrite request shaping +- compute-status UI +- failure and reconnect UX + +The frontend should mock the local controller API, not provider APIs. + +### 5. Cross-layer protocol tests + +Add contract fixtures that validate shared payloads across layers. + +Important fixtures: + +- websocket event payloads +- rewrite request payloads +- runtime status payloads +- controller status payloads + +These can be simple JSON fixtures validated by both Python and TypeScript +tests. They will catch drift earlier than end-to-end tests. + +### 6. Smoke tests + +Add a small number of high-signal smoke tests: + +- local provider + mock runtime +- local provider + real runtime when GPU is available +- controller startup + frontend health path + +These should be cheap enough for routine local use. + +### 7. Provider-backed manual or nightly tests + +Real Modal and Runpod tests should be opt-in. + +- Do not run them in default CI. +- Gate them behind explicit credentials and flags. +- Keep them focused on provisioning and reachability, not full product + regression. + +This avoids flaky and expensive CI while still validating real provider flows. + +## Testing Recommendations for the Next Step + +The next practical additions should be: + +1. A controller test suite in Python using fake providers. +2. Shared contract fixtures for websocket and rewrite payloads. +3. A smoke test that starts the local controller against the existing mock + runtime. + +I would not add Playwright yet. The current web stack already has Vitest and +integration-style component tests, which are cheaper and better aligned with +the immediate reorg. Add browser automation only after the controller path is +stable. diff --git a/apps/dreamverse/gpu-pool.drawio b/apps/dreamverse/gpu-pool.drawio new file mode 100644 index 000000000..ea9fe67b5 --- /dev/null +++ b/apps/dreamverse/gpu-pool.drawio @@ -0,0 +1,577 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/apps/dreamverse/gpu-pool.svg b/apps/dreamverse/gpu-pool.svg new file mode 100644 index 000000000..aa379cf60 --- /dev/null +++ b/apps/dreamverse/gpu-pool.svg @@ -0,0 +1 @@ +
gpu_pool.py — runtime architecture
gpu_pool.py — runtime architecture
Client (WebSocket)
Client (WebSocket)
GPUPool
GPUPool
GPUSlot (one per GPU)
GPUSlot (one per GPU)
start()
start()
shutdown()
shutdown()
join_user()
join_user()
user_step()
user_step()
leave_user()
leave_user()
_send_command()

puts on command_queue,
blocks on response_queue.get

**for user agonistic commands
_send_command()...
_send_command_tagged()

creates Future; registers in
_pending_futures[user_id];
ensures response reader; awaits future

**for user specific commands
_send_command_tagged()...
_response_reader()

a while loop that reads response_queue;
dispatch WorkerEvent:
• media → _stream_queues
• acks/errors → _pending_futures
_response_reader()...
«dict» _pending_futures

{user_id → asyncio.Future}

created by _send_command_tagged;
resolved + popped by _response_reader;
cleared on model reload or leave
«dict» _pending_futures...
«dict» _stream_queues: 
each is a queue of metadata that tell the consumer which part of the shared 
stream buffer to read next.

{user_id → asyncio.Queue}

created by register_stream_queue()
written by _response_reader (MediaInit / Chunk / Complete);
consumed by  AV loop → ws.send_bytes / send_json;
popped by leave_user(); cleared on reload
«dict» _stream_queues:...
── process boundary (mp.Queue + mp.RawArray) ──
── process boundary (mp.Queue + mp.RawArray) ──
command_queue
command_queue
response_queue
response_queue
shared_stream_buffer
(mp.RawArray, 256 MiB)

zero-copy mp4 bytes
shared_stream_buffer...
worker_ipc.py — shared types (imported on both sides of the process boundary)

Command(type: CommandType, payload: CommandPayload | None, user_id: str | None)
CommandPayload = UserStepPayload | WarmupPayload | ReloadModelPayload
WorkerEvent = StepComplete | WorkerError | JoinAck | LeaveAck | ReloadAck | WarmupComplete | MediaInit | MediaChunk | MediaComplete | InitAck | ShutdownAck
worker_ipc.py — shared types (imported on both sides of the process boundary)...
Worker Subprocess - in each GPUSlot
Worker Subprocess - in each GPUSlot
command dispatcher

gpu_worker_process() branches on
CommandType; asserts payload type

INIT / WARMUP / RELOAD_MODEL
USER_JOIN / USER_STEP / USER_LEAVE
SHUTDOWN
command dispatcher...
VideoGenerationWorker.generate_step()
video_generation.py:380

reads + updates ContinuationState,
calls generator
VideoGenerationWorker.generate_step()...
stream_fmp4()
av_streaming.py:121

trims overlap, pipes to ffmpeg,
publishes StreamInit / StreamChunk /
StreamComplete via callback
stream_fmp4()...
VideoGenerator (fastvideo)

LTX2 DiT + refine upsampler
FP4 quant, torch.compile

owned by VideoGenerationWorker
video_generation.py:211
VideoGenerator (fastvideo)...
ffmpeg subprocess

libx264 / *_nvenc
fragmented mp4
ffmpeg subprocess...
ContinuationState
video_generation.py:89

• video_images: list[PIL.Image]
• audio_latents: torch.Tensor (CPU)

carried across segments
ContinuationState...
acquire
acquire
assigns slot
assigns slot
register future
register future
set future result
set future result
put(MediaEvent)
put(MediaEvent)
put Command
put Command
put Command
put Command
WorkerEvent
WorkerEvent
blocking .get
blocking .get
USER_STEP
USER_STEP
StepComplete / ack types
StepComplete / ack types
generator.generate_video()
generator.generate_video()
read / write
read / write
frames + audio
frames + audio
rawvideo + wav
rawvideo + wav
write mp4 bytes
at shared_write_offset
write mp4 bytes...
bytes from the buffer are sent through websocket 
based on info from the stream queue
bytes from the buffer are sent through websocket...
session AV loop (main.py:2124):
ws.send_bytes(chunk) — binary mp4 frames
ws.send_json({type: media_init | media_segment_complete | ...})
session AV loop (main.py:2124):...
MediaChunk (offset, len)
MediaChunk (offset, len)
Legend

■ blue client / external
■ green main-process pool/slot
(methods — italic label)
■ yellow containers (routing state)
■ red IPC primitives (mp.Queue, mp.RawArray)

Worker subprocess modules:
■ orange gpu_pool.py (dispatcher)
■ lavender video_generation.py
■ teal av_streaming.py
■ gray worker_ipc.py (shared types)

Flow:
client → pool → slot
→ _send_command(_tagged) → command_queue
→ dispatcher → generate_step()
→ stream_fmp4() → ffmpeg
→ shared_buf + response_queue
→ _response_reader → futures / stream_queues
→ client awaits (via main.py AV loop)
Legend...
FastVideo video_generator
FastVideo video_genera...
reload model
reload model
Queue: owned by GPUSlot, passed to worker process as argument.
Queue: owned by GPUSlot, passed to w...Text is not SVG - cannot display \ No newline at end of file diff --git a/apps/dreamverse/pyproject.toml b/apps/dreamverse/pyproject.toml new file mode 100644 index 000000000..cc44a3bcf --- /dev/null +++ b/apps/dreamverse/pyproject.toml @@ -0,0 +1,41 @@ +[project] +name = "dreamverse" +version = "0.1.0" +description = "Dreamverse application with FastAPI backend and Next.js frontend." +readme = "README.md" +requires-python = ">=3.10" +dependencies = [ + "fastapi>=0.115,<1", + "fastvideo>=0.1.7", + "pillow>=10.0", + "uvicorn[standard]>=0.30,<1", +] + +[project.optional-dependencies] +server = [ + "cerebras-cloud-sdk", + "flash-attn-cute @ git+https://github.com/XOR-op/flash-attention.git@fa4-compile#subdirectory=flash_attn/cute", + "flashinfer-python", + "openai>=1.40", +] +safety = [ + "fasttext>=0.9.2", +] +test = [ + "pytest>=8.0", +] + +[project.scripts] +dreamverse-server = "dreamverse.server_entry:cli" +dreamverse-mock-server = "dreamverse.mock_server:cli" + +[tool.uv] +package = false + +[tool.uv.sources] +fastvideo = { workspace = true } + +[tool.pytest.ini_options] +markers = [ + "gpu: requires real GPU + model weights; skip in CI", +]