9ff65c83b6
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com> Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com> Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com> Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
355 lines
11 KiB
Markdown
355 lines
11 KiB
Markdown
# Dreamverse Architecture
|
|
|
|
## Overview
|
|
|
|
Dreamverse currently has two main runtime pieces:
|
|
|
|
- `apps/dreamverse/web/`: Next.js frontend
|
|
- `apps/dreamverse/dreamverse/`: Python FastAPI runtime
|
|
|
|
Today, the browser talks directly to the Dreamverse runtime over HTTP and a
|
|
single websocket on `/ws`. The frontend owns UI state and interaction flow. The
|
|
server owns generation state, prompt rewrite, prompt safety, websocket session
|
|
semantics, and GPU-backed execution.
|
|
|
|
Near-term OSS note:
|
|
|
|
- `apps/dreamverse/dreamverse/` is the current runtime implementation.
|
|
- A future `controller/` layer is planned for local-only compute management and
|
|
provider orchestration, but it does not exist yet.
|
|
|
|
## Repo Map
|
|
|
|
### Frontend
|
|
|
|
- `apps/dreamverse/web/src/app/page.tsx`: main client orchestration,
|
|
websocket connect, init payloads, send paths, and top-level app behavior
|
|
- `apps/dreamverse/web/src/lib/ws/reducer.ts`: reduces normalized websocket
|
|
events into client stores
|
|
- `apps/dreamverse/web/src/stores/session.ts`: connection, mode, and top-level
|
|
session UI
|
|
- `apps/dreamverse/web/src/stores/promptWindow.ts`: editable prompt window and
|
|
seed prompt UI state
|
|
- `apps/dreamverse/web/src/stores/rewrite.ts`: rewrite activity timeline and
|
|
inspection state
|
|
- `apps/dreamverse/web/src/stores/stream.ts`: playback and stream-related
|
|
client state
|
|
- `apps/dreamverse/web/src/lib/prompts/promptWindowSnapshot.ts`: prompt-window snapshot
|
|
building for rewrite requests
|
|
|
|
### Server
|
|
|
|
- `apps/dreamverse/dreamverse/main.py`: websocket endpoint, request handling,
|
|
session state machine, rewrite orchestration, REST routes, and stream relay
|
|
- `apps/dreamverse/dreamverse/gpu_pool.py`: GPU worker processes, warmup, model
|
|
loading, and `generate_video()` calls through FastVideo
|
|
- `apps/dreamverse/dreamverse/prompt_enhancer.py`: prompt enhancement, rollout
|
|
rewrite execution, provider selection, and timeout/fallback behavior
|
|
- `apps/dreamverse/dreamverse/rewrite_prompt_payload.py`: canonical rewrite request payload
|
|
building
|
|
- `apps/dreamverse/dreamverse/config.py`: runtime flags, prompt file paths, provider settings, and
|
|
warmup config
|
|
- `apps/dreamverse/dreamverse/session_init_image.py`: validates and persists uploaded initial
|
|
images for segment 1
|
|
|
|
## Current Split Of Responsibility
|
|
|
|
### Frontend owns
|
|
|
|
- local UI state and client-side stores
|
|
- prompt drafts and prompt window editing
|
|
- websocket connection management
|
|
- deciding which user action to send:
|
|
- `session_init_v2`
|
|
- `project_init_v1`
|
|
- `append_prompt`
|
|
- `rewrite_seed_prompts`
|
|
- `simple_generate`
|
|
- showing rewrite progress, stream status, prompt history, and devtools views
|
|
|
|
### Server owns
|
|
|
|
- websocket session lifecycle and protocol
|
|
- GPU assignment and worker lifecycle
|
|
- the authoritative seed prompt memory used for generation
|
|
- prompt rewrite execution and prompt safety
|
|
- actual generation queue semantics
|
|
- stream chunk emission and segment lifecycle events
|
|
- prompt config and preset persistence routes
|
|
- health and readiness endpoints
|
|
|
|
Important rule:
|
|
|
|
- The frontend may propose prompt-window state for rewrite, but the server is
|
|
the source of truth for the rewritten rollout and the active prompt memory
|
|
used for generation.
|
|
|
|
## End-To-End Flow
|
|
|
|
1. The frontend opens `/ws`.
|
|
2. The frontend sends `session_init_v2` with the initial prompt-window state,
|
|
preset metadata, and current toggles.
|
|
3. The server validates init data, persists an optional initial image, acquires
|
|
a GPU slot, and emits session status such as `gpu_assigned`.
|
|
4. The server starts or resumes project generation and emits events like
|
|
`ltx2_stream_start`, `ltx2_segment_start`, media init/chunks, and completion
|
|
events.
|
|
5. The frontend reduces those websocket events into its stores and updates the
|
|
UI.
|
|
6. User actions such as appending prompts, rewriting seed prompts, or starting
|
|
a single custom clip go back to the server over the same websocket.
|
|
|
|
## Frontend Architecture
|
|
|
|
The frontend is store-driven.
|
|
|
|
- `page.tsx` wires together websocket setup, send helpers, reducer
|
|
application, and top-level interaction flows.
|
|
- Store modules separate concerns like session state, rewrite state, prompt
|
|
window state, and stream state.
|
|
- The websocket reducer is responsible for turning normalized runtime events
|
|
into store updates. If the server event schema changes, the reducer must
|
|
change with it.
|
|
|
|
The frontend is intentionally not responsible for:
|
|
|
|
- generating rewritten prompts locally
|
|
- deciding final prompt safety outcomes
|
|
- reconstructing server session state from scratch
|
|
- inventing its own generation semantics independent of the runtime
|
|
|
|
## Server Architecture
|
|
|
|
The current runtime is a FastAPI app with a single long-lived websocket per
|
|
session.
|
|
|
|
`apps/dreamverse/dreamverse/main.py` manages:
|
|
|
|
- websocket connect/init
|
|
- prompt queues
|
|
- project and segment state
|
|
- prompt enhancement/rewrite triggers
|
|
- stream relay from GPU workers to the browser
|
|
- session logging and REST endpoints
|
|
|
|
`apps/dreamverse/dreamverse/gpu_pool.py` manages:
|
|
|
|
- model loading through FastVideo
|
|
- one or more worker processes
|
|
- startup warmup
|
|
- user join/leave commands
|
|
- `USER_STEP` execution for each segment
|
|
- continuation state between segments
|
|
|
|
`apps/dreamverse/dreamverse/prompt_enhancer.py` manages:
|
|
|
|
- prompt enhancement for user-submitted prompts
|
|
- rollout rewrite requests for the prompt window
|
|
- provider selection and fallback across configured prompt providers
|
|
- response normalization and safety-aware failure handling
|
|
|
|
## FastAPI Surface
|
|
|
|
The server is a single FastAPI application created in
|
|
`apps/dreamverse/dreamverse/main.py`.
|
|
|
|
Current built-in FastAPI docs are enabled:
|
|
|
|
- `/docs`: Swagger UI
|
|
- `/redoc`: ReDoc
|
|
- `/openapi.json`: OpenAPI schema
|
|
|
|
The runtime also mounts the frontend static build at `/` when one of the
|
|
configured frontend static directories exists. It does not expose the backend
|
|
Python package as static content.
|
|
|
|
## HTTP API
|
|
|
|
The current HTTP API is small. Most realtime behavior still goes through the
|
|
websocket.
|
|
|
|
### Core health and status routes
|
|
|
|
- `GET /healthz`
|
|
- process liveness probe
|
|
- returns a small payload with `status`, `service`, and timestamp
|
|
- `GET /readyz`
|
|
- readiness probe
|
|
- returns `503` until prompt services are initialized and at least one GPU
|
|
worker is ready
|
|
- returns readiness and GPU pool summary fields such as ready workers, total
|
|
GPUs, warmup counts, and queue size
|
|
- `GET /status`
|
|
- returns the current GPU pool status payload from `gpu_pool`
|
|
- `GET /internal/monitor/sessions`
|
|
- internal monitoring payload for session dashboards
|
|
- includes pending session count, max available sessions, prompt provider
|
|
success counts, and timestamp
|
|
|
|
### Prompt config routes
|
|
|
|
- `GET /prompt-system-config`
|
|
- returns the editable prompt-system configuration currently loaded by
|
|
`PromptEnhancer`
|
|
- `POST /prompt-system-config`
|
|
- saves prompt-system configuration to disk and reloads prompt config in the
|
|
runtime
|
|
- current editable fields include:
|
|
- next-segment system prompt
|
|
- auto-extension system prompt
|
|
- rewrite-window system prompt
|
|
- rewrite-user system prompt
|
|
- rewrite model
|
|
- rewrite temperature
|
|
|
|
### Devtools-only preset routes
|
|
|
|
These exist only when `DEVTOOLS_ENABLED` is true in
|
|
`apps/dreamverse/dreamverse/config.py`.
|
|
|
|
- `GET /curated-presets`
|
|
- returns merged curated presets, applying the local overlay file on top of
|
|
the fallback file when both exist
|
|
- `POST /curated-presets/append`
|
|
- appends a new curated preset to the overlay presets file
|
|
- validates non-empty label, normalized id, and at least two non-empty
|
|
segment prompts
|
|
|
|
## Websocket API
|
|
|
|
`WS /ws` is the main runtime API.
|
|
|
|
The websocket owns:
|
|
|
|
- session init
|
|
- project init and reset
|
|
- prompt append
|
|
- prompt rewrite
|
|
- generation toggles
|
|
- segment lifecycle events
|
|
- media stream delivery
|
|
- runtime error delivery
|
|
|
|
The websocket is the authoritative API for realtime Dreamverse behavior. The
|
|
HTTP routes mainly support health checks, devtools persistence, and monitoring.
|
|
|
|
## Prompt Rewrite Architecture
|
|
|
|
Prompt rewrite is a shared flow with strict ownership boundaries.
|
|
|
|
### Frontend responsibilities
|
|
|
|
- collect the rewrite instruction
|
|
- build the prompt-window snapshot from current client state
|
|
- send `rewrite_seed_prompts`
|
|
- show rewrite progress, raw output, fallback state, and resulting prompt list
|
|
|
|
### Server responsibilities
|
|
|
|
- validate and normalize the prompt-window payload
|
|
- choose rewrite model, system prompt, timeout, and temperature
|
|
- build the canonical prompt payload in
|
|
`apps/dreamverse/dreamverse/rewrite_prompt_payload.py`
|
|
- execute rewrite through `PromptEnhancer`
|
|
- apply safety filtering to rewritten prompts
|
|
- replace the authoritative seed prompt memory when rewrite succeeds
|
|
- emit `seed_prompts_updated` and `rewrite_seed_prompts_complete`
|
|
|
|
Important rule:
|
|
|
|
- The frontend owns editable drafts.
|
|
- The server owns the accepted rollout.
|
|
|
|
After a successful rewrite, the frontend should replace its prompt-window view
|
|
from the server payload instead of preserving a locally-derived version.
|
|
|
|
## Prompt Modes
|
|
|
|
There are three related prompt paths in the current system:
|
|
|
|
### Initial rollout
|
|
|
|
- The frontend sends seed prompts during `session_init_v2`.
|
|
- The server uses those prompts as the initial seed prompt memory.
|
|
- If the rollout starts from an empty prompt window plus an initial rewrite
|
|
instruction, the server can pause generation until rewrite completes.
|
|
|
|
### Live append
|
|
|
|
- The frontend sends `append_prompt`.
|
|
- The server may enhance that prompt, safety-check it, enqueue it, and use it
|
|
as the next generated segment.
|
|
|
|
### Rewrite
|
|
|
|
- The frontend sends `rewrite_seed_prompts`.
|
|
- The server rewrites the entire seed prompt window or generates a new rollout,
|
|
depending on the payload and current state.
|
|
|
|
## Initial Image And Segment Handling
|
|
|
|
The frontend currently sends `initial_image` as part of session init or
|
|
`simple_generate`.
|
|
|
|
The server:
|
|
|
|
- validates and persists the image
|
|
- uses it only for segment 1 when present
|
|
- keeps continuation state for later segments in the GPU worker
|
|
|
|
This means the runtime, not the frontend, decides how segment 1 image
|
|
conditioning and later continuation conditioning are applied.
|
|
|
|
## Websocket Contract
|
|
|
|
The websocket is the main integration surface between UI and runtime.
|
|
|
|
Typical incoming messages from the frontend:
|
|
|
|
- `session_init_v2`
|
|
- `project_init_v1`
|
|
- `append_prompt`
|
|
- `rewrite_seed_prompts`
|
|
- `simple_generate`
|
|
- `set_enhancement`
|
|
- `set_auto_extension`
|
|
- `set_loop_generation`
|
|
|
|
Typical outgoing messages from the server:
|
|
|
|
- `gpu_assigned`
|
|
- `ltx2_stream_start`
|
|
- `ltx2_segment_start`
|
|
- `segment_prompt_source`
|
|
- `prompt_received`
|
|
- `prompt_ready`
|
|
- `prompt_enhancing`
|
|
- `seed_prompts_updated`
|
|
- `rewrite_seed_prompts_complete`
|
|
- `media_init`
|
|
- `media_segment_complete`
|
|
- `project_idle`
|
|
- `error`
|
|
|
|
Binary websocket frames carry media chunks for playback.
|
|
|
|
## Current And Planned Architecture
|
|
|
|
Current architecture:
|
|
|
|
- browser -> `apps/dreamverse/web`
|
|
- `apps/dreamverse/web` -> `apps/dreamverse/dreamverse/main.py`
|
|
- `apps/dreamverse/dreamverse/main.py` ->
|
|
`apps/dreamverse/dreamverse/gpu_pool.py`
|
|
- `gpu_pool.py` -> FastVideo runtime
|
|
|
|
Planned architecture:
|
|
|
|
- browser -> `apps/dreamverse/web`
|
|
- `apps/dreamverse/web` -> local `controller/`
|
|
- `controller/` -> local or remote Dreamverse runtime
|
|
- runtime -> FastVideo runtime
|
|
|
|
That future controller split should not move prompt rewrite, session state, or
|
|
generation semantics out of the runtime.
|