[docs] dreamverse-integration: drift audit + integration tradeoff doc
Comprehensive review against ../Dreamverse and ../FastVideo-internal
plus 4-option integration tradeoff analysis with recommendation.
Driven by user request to (1) verify will/ltx2_sr_port has not drifted
from the integration goal, and (2) analyze whether Dreamverse should
remain a separate repo, become a subfolder, or merge into FastVideo's
namespace entirely.
Methodology: 4 parallel exploration agents mapped Dreamverse repo
structure, FastVideo-internal residual, FastVideo public cross-repo
surfaces, and OSS service-in-monorepo precedents (vLLM, BentoML, Ray
Serve, TGI+ChatUI, Transformers.js, ComfyUI, AUTOMATIC1111). Synthesis
authored by ultrabrain category. Oracle review applied as
PASS-WITH-FIXES; critical issues (broad zero-drift wording, stale Dynamo
mapping citation, broken Ray Serve URL, missing D-8/VPO/D-12-A/D-12-B/
SBS coverage) addressed in this commit.
* integration-review.md (new, 863 lines): Part 1 drift audit, Part 2
integration tradeoffs, Part 3 action items.
Part 1 — Drift audit
- Methodology: typed-public-boundary criterion, intentional refactor
vs drift, evidence sourced from worktree files + memory dir.
- Zero core typed API drift on the integration surface — but the
realtime-runtime contract surface (health routes) IS real drift.
- 8 numbered drift items (Dreamverse README/bootstrap, prompt
enhancer fork, cerebras_ifm gap, health routes, layerwise offload,
upsampler CLI, missing example config, LTX-2 stage equivalence
verification gap).
- 6 deferred/accepted residual items.
- 17-row drift summary table with priority / effort / status /
tracked-where / next-action columns. Includes D-8 (ltx2_image_crf
flow), VPO (video_position_offset_sec semantics — overdue),
D-12-A (GpuPool docstring), D-12-B (run_async migration), SBS
(session/blob lifecycle).
Part 2 — Integration path tradeoffs
- Option A: status quo (Dreamverse separate, depends on fastvideo).
- Option B: Dreamverse as subfolder under FastVideo.
- Option C: full merge into fastvideo.entrypoints.dreamverse.*.
- Option D: hybrid — backend merges, frontend stays separate.
- Comparison matrix on 13 axes.
- 7 OSS precedent rows with citations (vLLM, BentoML, Ray Serve,
TGI, Transformers.js, ComfyUI, AUTOMATIC1111).
- Recommendation: Option D (constrained) — backend merges as
generic FastVideo streaming, frontend stays separate. Mirrors
ComfyUI's late-stage frontend split. Strong precedent: TGI +
ChatUI. Critical librarian finding: NO 1:1 precedent exists for
"Python ML library + Next.js product merged into library
namespace" — argues against Option C.
- Conditions that would change the recommendation enumerated.
Part 3 — Action items: phased migration sketch (phases 0-5) with
concrete file paths, effort estimates, and cross-references to
open-threads.md / decisions-log.md.
* README.md: register integration-review.md in deep-dive reading guide.
Bump Last reconciled to b36bdbc9 (current PR #1288 head, post
STACK.md removal + integration-review.md addition); 36 commits ahead.
* state.md: branch tip refresh — b36bdbc9 / 36 commits / now includes
integration-review.md.
* pr-roadmap.md: PR #1288 head bumped to b36bdbc9.
* open-threads.md: Last updated header bumped to b36bdbc9. Item D
reference SHA bumped to b36bdbc9.
* authors.md: PR #1288 row bumped to b36bdbc9 / 36 commits / 70 files
(post STACK.md removal).
Verified:
- Pre-commit clean (yapf/ruff/mypy/codespell skipped per memory dir
excludes; spaces-check passes).
- Oracle review: PASS-WITH-FIXES, all 4 critical issues addressed.
- Cross-references resolve: design.md, cross-repo-surfaces.md,
decisions-log.md, open-threads.md, pr-roadmap.md, state.md,
runbook.md, authors.md, streaming-server.md, quantization.md.
PR #1288: https://github.com/hao-ai-lab/FastVideo/pull/1288
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
This commit is contained in:
co-authored by
Junda Su
Matthew Noto
XOR-op
Zhang Peiyuan
parent
b36bdbc96d
commit
907620ff22
@@ -6,7 +6,7 @@ upstream, the Dreamverse switch from `FastVideo-internal` to public
|
||||
`FastVideo`, and the NVFP4 quantization landing.
|
||||
|
||||
**Last reconciled:** 2026-05-05 (strategy reversal — single mega-PR).
|
||||
FastVideo `will/ltx2_sr_port` @ `39dfa009` (34 commits ahead of main).
|
||||
FastVideo `will/ltx2_sr_port` @ `b36bdbc9` (36 commits ahead of main, post-D-17 STACK.md removal + integration-review.md addition).
|
||||
Dreamverse `will/integrate-public-fastvideo` @ `ec8ef92`.
|
||||
PRs #1257 / #1258 / #1284 / #1286 MERGED to main. **PR #1287 CLOSED
|
||||
(in favor of consolidation); PR #1288 OPEN as the single mega-PR
|
||||
@@ -68,6 +68,7 @@ file it in [open-threads.md](open-threads.md) before continuing.
|
||||
| "Why was decision X made? What's resolved vs. open?" | [decisions-log.md](decisions-log.md) |
|
||||
| "What should I work on next? Priority order?" | [open-threads.md](open-threads.md) |
|
||||
| "Who should be co-authored on commits in this scope?" | [authors.md](authors.md) |
|
||||
| "Have we drifted from the goal? Should Dreamverse merge into FastVideo?" | [integration-review.md](integration-review.md) |
|
||||
|
||||
## Repo + worktree paths
|
||||
|
||||
|
||||
@@ -41,7 +41,7 @@ across every PR in the integration scope:
|
||||
| #1284 | `will/api_7.8` (streaming auxiliaries) | ✅ merged 2026-05-04 | yes (2/2) |
|
||||
| #1286 | `will/api_7.9` (streaming router) | ✅ merged 2026-05-05 at `2aaeee2a` (squash) | yes on commits 1-3; commit `a152cb77` (`[fix] streaming: router polish`) was missing trailers but got squashed into the merge commit, so the merge commit on main inherits the trailers from the other 3. The trailerless cherry-pick partner (`40e265b8` on `will/ltx2_sr_port`) was dropped by the post-#1286 rebase — gap permanently resolved. |
|
||||
| #1287 | `will/api_7.10` (`generate_async` + `VideoEvent`) | ❌ CLOSED 2026-05-05 — superseded by #1288 per [D-17](decisions-log.md#d-17) | yes on all 3 commits (now part of #1288's chain) |
|
||||
| **#1288** | **`will/ltx2_sr_port`** (mega-PR — full stack: SR runtime + NVFP4 + generate_async + Dynamo contract + agents memory) | 🟢 OPEN, MERGEABLE at `39dfa009`, 34 commits / 71 files / +13,074 LOC | yes on all 34 commits |
|
||||
| **#1288** | **`will/ltx2_sr_port`** (mega-PR — full stack: SR runtime + NVFP4 + generate_async + Dynamo contract + agents memory + integration-review) | 🟢 OPEN, MERGEABLE at `b36bdbc9`, 36 commits / 70 files / ~+13.0k LOC (post STACK.md removal) | yes on all 36 commits |
|
||||
|
||||
Aggregate count across `will/ltx2_sr_port` (top of stack) at the time of
|
||||
writing: 32-33 commits per co-author, matching the 32 commits in the stack
|
||||
|
||||
@@ -0,0 +1,863 @@
|
||||
# Integration Review — Drift Audit + Path Forward
|
||||
|
||||
**Last updated:** 2026-05-05.
|
||||
|
||||
**Scope:** FastVideo public `will/ltx2_sr_port` at the requested audit
|
||||
anchor `b36bdbc9`; Dreamverse `will/integrate-public-fastvideo` at
|
||||
`ec8ef92`; FastVideo-internal `will/rebase-nbv` as read-only comparison.
|
||||
|
||||
**Memory-dir context:** the current integration memory snapshot tracks the
|
||||
same public mega-PR lineage as `will/ltx2_sr_port`, with PRs #1257,
|
||||
#1258, #1284, and #1286 already merged, #1287 closed, and #1288 open as
|
||||
the consolidated landing vehicle for LTX-2 SR runtime, NVFP4,
|
||||
`generate_async`, Dynamo contract, and memory-dir cleanup. Source:
|
||||
[memory index](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/README.md#L8-L19)
|
||||
and [D-17](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/decisions-log.md#L19-L45).
|
||||
|
||||
**Bottom line:** Zero core typed API drift — typed construction, typed
|
||||
continuation state, NVFP4 wiring, and Dynamo-facing async events are either
|
||||
already public or in #1288. **Real drift remains on the realtime-runtime
|
||||
contract surface (`/healthz` / `/readyz` / `/status` routes per
|
||||
[cross-repo-surfaces.md](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/cross-repo-surfaces.md#L74-L88))
|
||||
and on operational/product edges**: stale Dreamverse docs/scripts, a
|
||||
1933-LOC Dreamverse prompt-enhancer fork, two unresolved per-session
|
||||
fields (`ltx2_image_crf` D-8, `video_position_offset_sec` VPO), one
|
||||
missing example config, and two internal-only utilities whose product
|
||||
relevance is not yet proven.
|
||||
|
||||
---
|
||||
|
||||
## Part 1 — Drift audit
|
||||
|
||||
### Methodology
|
||||
|
||||
1. **Compared three repositories and branches.**
|
||||
- FastVideo public: `/home/william5lin/FastVideo`, branch
|
||||
`will/ltx2_sr_port`.
|
||||
- Dreamverse: `/home/william5lin/Dreamverse`, branch
|
||||
`will/integrate-public-fastvideo`.
|
||||
- FastVideo-internal: `/home/william5lin/FastVideo-internal`, branch
|
||||
`will/rebase-nbv`.
|
||||
- Canonical repo paths are listed in the integration memory index:
|
||||
[repo paths](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/README.md#L72-L79).
|
||||
|
||||
2. **Scoped the audit to the ultimate integration goal.**
|
||||
- Dreamverse should depend on public `fastvideo`, not
|
||||
`FastVideo-internal`.
|
||||
- FastVideo should own the reusable backend subset that Dreamverse
|
||||
currently needs from internal: streaming runtime, GPU pool, router,
|
||||
prompt enhancer, NVFP4, continuation state, and typed generation.
|
||||
- Dynamo should consume FastVideo through typed public Python APIs, not
|
||||
through private modules.
|
||||
- The three Dreamverse surfaces are documented as pipeline construction,
|
||||
realtime runtime, and continuation state:
|
||||
[cross-repo surfaces](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/cross-repo-surfaces.md#L13-L20).
|
||||
|
||||
3. **Separated intentional refactor from drift.**
|
||||
- A path rename is not drift if the public branch contains the same
|
||||
responsibility under the typed design.
|
||||
- A deleted file is not drift if the public design intentionally
|
||||
consolidated it.
|
||||
- A private alias is not drift if the public schema exposes a typed
|
||||
replacement with contract tests.
|
||||
- This matches the typed-public-boundary rule in
|
||||
[design.md](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/design.md#L24-L43).
|
||||
|
||||
4. **Used memory docs for rationale and worktree files for concrete proof.**
|
||||
- API schema and public exports:
|
||||
[schema](file:///home/william5lin/FastVideo/fastvideo/api/schema.py#L68-L85),
|
||||
[api exports](file:///home/william5lin/FastVideo/fastvideo/api/__init__.py#L49-L109).
|
||||
- Streaming server current routes:
|
||||
[build_app](file:///home/william5lin/FastVideo/fastvideo/entrypoints/streaming/server.py#L88-L160).
|
||||
- Dreamverse dependency state:
|
||||
[pyproject server extra](file:///home/william5lin/Dreamverse/pyproject.toml#L17-L22),
|
||||
[uv lock editable source](file:///home/william5lin/Dreamverse/uv.lock#L716-L722).
|
||||
- Contract tests:
|
||||
[Dreamverse shape](file:///home/william5lin/FastVideo/fastvideo/tests/contract/test_dreamverse_shape.py#L1-L26),
|
||||
[Dynamo shape](file:///home/william5lin/FastVideo/fastvideo/tests/contract/test_dynamo_shape.py#L1-L19),
|
||||
[generate_async](file:///home/william5lin/FastVideo/fastvideo/tests/contract/test_generate_async.py#L1-L7).
|
||||
|
||||
5. **Did not treat product-only Dreamverse behavior as FastVideo drift.**
|
||||
- Dreamverse keeps a local product server and Next.js UI today:
|
||||
[README baseline](file:///home/william5lin/Dreamverse/README.md#L5-L16).
|
||||
- Product-only routes, curated presets, devtools, and frontend-specific
|
||||
behavior belong in Dreamverse unless a second non-Dreamverse consumer
|
||||
needs them.
|
||||
|
||||
6. **Risk scale used below.**
|
||||
- **P0:** blocks Dreamverse from running without FastVideo-internal.
|
||||
- **P1:** blocks clean `BE_FLAVOR=fastvideo` or Dynamo/public API use.
|
||||
- **P2:** reproducibility or maintenance drag.
|
||||
- **P3:** optional parity or future memory/perf improvement.
|
||||
|
||||
### Findings: zero core typed API drift
|
||||
|
||||
The public branch is aligned with the goal on the **core typed API
|
||||
surface** (construction, request, continuation state, async events).
|
||||
The table below lists items that look like drift only if compared by
|
||||
path name or legacy field name. They are intentional public refactors
|
||||
or already guarded by tests. **Note:** the realtime-runtime _contract_
|
||||
surface (FE-required health routes) is a separate matter — see "real
|
||||
drift items" §4 below.
|
||||
|
||||
| Investigated item | Drift? | Evidence | Conclusion |
|
||||
|---|---:|---|---|
|
||||
| Dreamverse surface 1: pipeline construction | No | Dreamverse migrated from flat kwargs to typed `GeneratorConfig` at `d80c2a8`; mapping documented in [cross-repo surfaces](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/cross-repo-surfaces.md#L22-L47). | Stable public surface exists. |
|
||||
| Dreamverse surface 2: realtime runtime | No on architecture; some route work remains | Runtime migration target is public `streaming/`, not internal `realtime/`: [streaming upstream](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/streaming-server.md#L14-L31). | Rename/refactor is intentional. |
|
||||
| Dreamverse surface 3: continuation state | No | Public typed `ContinuationState` plus LTX-2 state mapping are documented in [cross-repo surfaces](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/cross-repo-surfaces.md#L108-L146). | Public state is a superset of Dreamverse's data carrier. |
|
||||
| Internal `fastvideo/entrypoints/realtime/` | No | Public design chooses parallel `fastvideo/entrypoints/streaming/`: [layout decision](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/streaming-server.md#L55-L60), [current build_app](file:///home/william5lin/FastVideo/fastvideo/entrypoints/streaming/server.py#L88-L160). | Intentional rename plus typed-config rewrite. |
|
||||
| Internal `configs/sample/` presets | No | Public PR 4 intentionally deleted `configs/sample/` and moved defaults to per-family presets plus `fastvideo/api/sampling_param.py`: [design](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/design.md#L194-L205), [PR roadmap](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/pr-roadmap.md#L21-L29). | Intentional consolidation. |
|
||||
| LTX-2 pipeline presets | No | Public target is model-owned named presets and per-family colocation: [design](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/design.md#L175-L205). | Public layout matches design. |
|
||||
| Internal `use_fp4_linear` flag | No | Public typed quant carrier is `engine.quantization.transformer_quant`; schema field exists in [schema](file:///home/william5lin/FastVideo/fastvideo/api/schema.py#L68-L85), compat resolves it in [compat.py](file:///home/william5lin/FastVideo/fastvideo/api/compat.py#L267-L279). | Replaced by typed NVFP4 surface. |
|
||||
| Public-only `transformer_quant` field | No | Public `FastVideoArgs` pins typed quant to `dit_config.quant_config`: [fastvideo_args](file:///home/william5lin/FastVideo/fastvideo/fastvideo_args.py#L220-L228), [apply logic](file:///home/william5lin/FastVideo/fastvideo/fastvideo_args.py#L260-L279). | Public superset, not drift. |
|
||||
| Internal `config_model_path` | No | Public typed home is `generator.pipeline.components.config_root`: [design mapping](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/design.md#L261-L269), [compat mapping](file:///home/william5lin/FastVideo/fastvideo/api/compat.py#L295-L299). | Alias is covered. |
|
||||
| Internal flat video request fields | No | Public `GenerationRequest` nests `inputs`, `sampling`, `runtime`, `output`, `state`, `extensions`: [schema](file:///home/william5lin/FastVideo/fastvideo/api/schema.py#L193-L204). Internal legacy fields live in internal protocol at [protocol.py](file:///home/william5lin/FastVideo-internal/fastvideo/entrypoints/openai/protocol.py#L64-L82). | Intentional request refactor. |
|
||||
| Dreamverse typed init kwargs | No | Contract test asserts current Dreamverse load kwargs all land on typed fields, not `experimental`: [test_dreamverse_shape](file:///home/william5lin/FastVideo/fastvideo/tests/contract/test_dreamverse_shape.py#L44-L135). | Guard in place. |
|
||||
| Dreamverse request path | No | Contract test asserts request fields round-trip through typed `GenerationRequest`: [test_dreamverse_shape](file:///home/william5lin/FastVideo/fastvideo/tests/contract/test_dreamverse_shape.py#L153-L197). | Guard in place. |
|
||||
| Dynamo native backend shape | No | FastVideo's only obligation is stable typed Python API: [cross-repo contract](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/cross-repo-surfaces.md#L188-L210). | Dynamo should stay out of FastVideo. |
|
||||
| `generate_async` event API | No | API exists in [video_generator](file:///home/william5lin/FastVideo/fastvideo/entrypoints/video_generator.py#L264-L332), event types exist in [results.py](file:///home/william5lin/FastVideo/fastvideo/api/results.py#L109-L164). | #1288 covers the async contract. |
|
||||
| Dynamo request mapping | No | Authoritative source is the contract test [test_dynamo_shape](file:///home/william5lin/FastVideo/fastvideo/tests/contract/test_dynamo_shape.py#L90-L175) which asserts `req.prompt`, `req.sampling.{height,width,num_frames,fps,num_inference_steps,guidance_scale,seed,negative_prompt}`, and `req.inputs.{image_path,video_path}` against the actual nested [`GenerationRequest` schema](file:///home/william5lin/FastVideo/fastvideo/api/schema.py#L193-L204). The `streaming-server.md` Dynamo mapping table mis-cites a `prompt -> sampling.prompt` path that no longer exists; the test is correct, the doc is stale and tracked for refresh. | Guard in place; companion doc needs minor refresh. |
|
||||
| Public API exports | No | `VideoEvent`, `VideoResult`, and typed schema classes are exported from [fastvideo.api](file:///home/william5lin/FastVideo/fastvideo/api/__init__.py#L49-L109). | Integration imports resolve. |
|
||||
| FastVideo-internal FP4/NVFP4 paths | No | Public NVFP4 files and roles are documented in [quantization](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/quantization.md#L24-L35); actual `NVFP4Config` documents lazy FlashInfer and public naming in [nvfp4_config.py](file:///home/william5lin/FastVideo/fastvideo/layers/quantization/nvfp4_config.py#L1-L19). | Public is typed superset. |
|
||||
| AbsMaxFP8 refactor | No for Dreamverse | Public quant registry includes `AbsMaxFP8` and `NVFP4`: [quantization init](file:///home/william5lin/FastVideo/fastvideo/layers/quantization/__init__.py#L1-L8). AbsMaxFP8 failure is tracked as separate tech debt: [open threads](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/open-threads.md#L100-L117). | Not Dreamverse blocker. |
|
||||
| Internal realtime API regression test | No | Public contract tests replace it: [Dreamverse contract](file:///home/william5lin/FastVideo/fastvideo/tests/contract/test_dreamverse_shape.py#L1-L26), [Dynamo contract](file:///home/william5lin/FastVideo/fastvideo/tests/contract/test_dynamo_shape.py#L1-L19), [generate_async tests](file:///home/william5lin/FastVideo/fastvideo/tests/contract/test_generate_async.py#L91-L230). | Better scoped guards exist. |
|
||||
| Dreamverse dependency declaration | No | `server` extra declares `fastvideo>=0.1.7`: [pyproject](file:///home/william5lin/Dreamverse/pyproject.toml#L17-L22). Dev lock resolves editable public `../FastVideo`: [uv.lock](file:///home/william5lin/Dreamverse/uv.lock#L716-L722), [package source](file:///home/william5lin/Dreamverse/uv.lock#L777-L780). | Dependency is already switched in metadata/lock. |
|
||||
|
||||
#### Core conclusion for the zero-typed-drift section
|
||||
|
||||
The public typed API no longer needs to mirror `FastVideo-internal` file
|
||||
paths. The correct test is whether Dreamverse and Dynamo can express their
|
||||
needs through public typed objects and public entrypoints. On that test,
|
||||
the **typed core** is covered (construction, request, continuation,
|
||||
async events). The **runtime contract** still has health-route gaps —
|
||||
see real drift §4. On the **typed core**:
|
||||
|
||||
- `GeneratorConfig` and `GenerationRequest` cover construction and calls:
|
||||
[schema surface](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/design.md#L45-L72).
|
||||
- `ServeConfig.streaming` covers the server envelope:
|
||||
[schema](file:///home/william5lin/FastVideo/fastvideo/api/schema.py#L244-L279).
|
||||
- `generate_async` covers streaming, OpenAI, and Dynamo on one substrate:
|
||||
[video_generator](file:///home/william5lin/FastVideo/fastvideo/entrypoints/video_generator.py#L264-L332).
|
||||
- Contract tests now encode the cross-repo shapes:
|
||||
[Dreamverse](file:///home/william5lin/FastVideo/fastvideo/tests/contract/test_dreamverse_shape.py#L70-L214),
|
||||
[Dynamo](file:///home/william5lin/FastVideo/fastvideo/tests/contract/test_dynamo_shape.py#L170-L331),
|
||||
[async events](file:///home/william5lin/FastVideo/fastvideo/tests/contract/test_generate_async.py#L91-L273).
|
||||
|
||||
### Findings: real drift items requiring action
|
||||
|
||||
#### 1. Dreamverse README and bootstrap script still point at FastVideo-internal
|
||||
|
||||
- **Priority:** P0 for a clean public-dependency story.
|
||||
- **Effort:** Small.
|
||||
- **Owner:** Dreamverse repo.
|
||||
- **Evidence:** Dreamverse metadata already points at public FastVideo:
|
||||
[pyproject](file:///home/william5lin/Dreamverse/pyproject.toml#L17-L22),
|
||||
[uv source](file:///home/william5lin/Dreamverse/pyproject.toml#L54-L61),
|
||||
[uv.lock](file:///home/william5lin/Dreamverse/uv.lock#L716-L722).
|
||||
- **Drift:** README still tells users that `uv` resolves from
|
||||
`../FastVideo-internal` and that bootstrap expects `../FastVideo-internal`:
|
||||
[README](file:///home/william5lin/Dreamverse/README.md#L76-L109).
|
||||
- **Drift:** bootstrap script still defaults to cloning the private repo and
|
||||
verifying imports from that clone:
|
||||
[script defaults](file:///home/william5lin/Dreamverse/.agents/skills/bootstrap-fastvideo-private-fork/scripts/bootstrap_fastvideo_private.sh#L7-L11),
|
||||
[script clone flow](file:///home/william5lin/Dreamverse/.agents/skills/bootstrap-fastvideo-private-fork/scripts/bootstrap_fastvideo_private.sh#L33-L63),
|
||||
[script import assertion](file:///home/william5lin/Dreamverse/.agents/skills/bootstrap-fastvideo-private-fork/scripts/bootstrap_fastvideo_private.sh#L66-L88).
|
||||
- **Action:** Replace private-fork bootstrap with public FastVideo bootstrap
|
||||
or delete the bootstrap once PyPI publication is the default path.
|
||||
- **Do not overreach:** no FastVideo code change required.
|
||||
|
||||
#### 2. Dreamverse carries a 1933-line prompt-enhancer fork
|
||||
|
||||
- **Priority:** P1.
|
||||
- **Effort:** Medium.
|
||||
- **Owner:** Dreamverse repo, after public prompt enhancer is available.
|
||||
- **Evidence:** Dreamverse local fork starts at
|
||||
[server/prompt_enhancer.py](file:///home/william5lin/Dreamverse/server/prompt_enhancer.py#L1-L80).
|
||||
- **Public replacement:** FastVideo now has provider-agnostic
|
||||
`PromptEnhancer` with `enhance`, `auto_extend`, `rewrite`, and
|
||||
`register_provider`:
|
||||
[public enhancer](file:///home/william5lin/FastVideo/fastvideo/entrypoints/streaming/prompt/enhancer.py#L66-L142).
|
||||
- **Provider extension point:** custom providers implement `LLMProvider`:
|
||||
[provider protocol](file:///home/william5lin/FastVideo/fastvideo/entrypoints/streaming/prompt/providers/base.py#L63-L75).
|
||||
- **Tracking:** DR-1 in open threads already defines the compat-shim shape:
|
||||
[DR-1](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/open-threads.md#L174-L206).
|
||||
- **Action:** Replace the fork with a small Dreamverse shim that adapts
|
||||
public `LLMResponse` to Dreamverse's product response objects and keeps
|
||||
only product-only extras.
|
||||
- **Do not overreach:** do not merge Dreamverse's full prompt product layer
|
||||
into FastVideo unless a second consumer needs the same semantics.
|
||||
|
||||
#### 3. `cerebras_ifm` provider is unresolved
|
||||
|
||||
- **Priority:** P1 if Dreamverse needs IFM in production; P2 otherwise.
|
||||
- **Effort:** Small decision plus small/medium implementation.
|
||||
- **Owner:** Team decision; implementation either Dreamverse-side or public.
|
||||
- **Public state:** `PromptEnhancerConfig.provider` is currently
|
||||
`Literal["cerebras", "groq"]`:
|
||||
[schema](file:///home/william5lin/FastVideo/fastvideo/api/schema.py#L229-L235).
|
||||
- **Design note:** public Literal excludes `cerebras_ifm` today:
|
||||
[streaming-server D-3](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/streaming-server.md#L61-L100).
|
||||
- **Tracking:** DR-2 already frames the public-vs-Dreamverse decision:
|
||||
[DR-2](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/open-threads.md#L211-L229).
|
||||
- **Recommended default:** implement IFM as a Dreamverse-side custom provider
|
||||
registered through `enhancer.register_provider(...)` unless there is a
|
||||
non-Dreamverse public user.
|
||||
|
||||
#### 4. `/healthz`, `/readyz`, and `/status` are not in public `build_app`
|
||||
|
||||
- **Priority:** P1 for `BE_FLAVOR=fastvideo` frontend compatibility.
|
||||
- **Effort:** Medium/Large because route shapes need tests.
|
||||
- **Owner:** FastVideo public.
|
||||
- **Public current state:** `build_app` exposes `GET /health` and
|
||||
`WS /v1/stream`:
|
||||
[server.py](file:///home/william5lin/FastVideo/fastvideo/entrypoints/streaming/server.py#L126-L160).
|
||||
- **Dreamverse expected state:** Dreamverse exposes `GET /healthz`,
|
||||
`GET /readyz`, and `GET /status`:
|
||||
[routes/health.py](file:///home/william5lin/Dreamverse/server/routes/health.py#L34-L79).
|
||||
- **Tracking:** open item #1 documents route ownership and files likely to
|
||||
touch:
|
||||
[open threads](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/open-threads.md#L69-L99).
|
||||
- **Design note:** `/curated-presets`, `/prompt-system-config`, and devtools
|
||||
stay Dreamverse-side, with feature detection:
|
||||
[streaming route contract](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/streaming-server.md#L232-L259).
|
||||
|
||||
#### 5. `fastvideo/models/layerwise_offload.py` exists only internally
|
||||
|
||||
- **Priority:** P3 unless memory-tight Dreamverse deployments require it.
|
||||
- **Effort:** Medium if adopted; low if documented as deferred.
|
||||
- **Owner:** FastVideo public only if a concrete deployment needs it.
|
||||
- **Internal evidence:** internal file defines async layerwise CPU offload
|
||||
manager with pinned CPU memory and prefetch stream:
|
||||
[layerwise_offload.py](file:///home/william5lin/FastVideo-internal/fastvideo/models/layerwise_offload.py#L1-L20),
|
||||
[prefetch path](file:///home/william5lin/FastVideo-internal/fastvideo/models/layerwise_offload.py#L127-L180).
|
||||
- **Public state:** no equivalent public file was identified in this audit.
|
||||
- **Action:** defer unless Dreamverse or another public deployment hits a
|
||||
memory ceiling that cannot be handled by existing offload knobs.
|
||||
- **Decision rule:** if adopted, port as a generic offload utility with
|
||||
tests; do not make it Dreamverse-specific.
|
||||
|
||||
#### 6. Standalone LTX-2 upsampler CLI exists only internally
|
||||
|
||||
- **Priority:** P2 for reproducibility; P3 for product runtime.
|
||||
- **Effort:** Small/Medium after scope decision.
|
||||
- **Owner:** FastVideo public if standalone upsampling is a supported user
|
||||
workflow.
|
||||
- **Internal utility:** `upscale_video_file(...)` reads an existing video,
|
||||
prepares frame count/resolution, loads VAE + upsampler, and writes an mp4:
|
||||
[upsample.py](file:///home/william5lin/FastVideo-internal/fastvideo/entrypoints/upsample.py#L120-L180),
|
||||
[write tail](file:///home/william5lin/FastVideo-internal/fastvideo/entrypoints/upsample.py#L181-L202).
|
||||
- **Internal CLI:** `fastvideo upsample` wrapper exists internally:
|
||||
[cli/upsample.py](file:///home/william5lin/FastVideo-internal/fastvideo/entrypoints/cli/upsample.py#L15-L35),
|
||||
[CLI args](file:///home/william5lin/FastVideo-internal/fastvideo/entrypoints/cli/upsample.py#L48-L130).
|
||||
- **Public related functionality:** LTX-2 SR refine stage covers the
|
||||
in-pipeline latent upsample/refine path:
|
||||
[ltx2_refine.py](file:///home/william5lin/FastVideo/fastvideo/pipelines/basic/ltx2/stages/ltx2_refine.py#L1-L22),
|
||||
[upsample stage](file:///home/william5lin/FastVideo/fastvideo/pipelines/basic/ltx2/stages/ltx2_refine.py#L116-L180).
|
||||
- **Action:** decide whether standalone file-to-file upsampling is a public
|
||||
CLI promise or whether the SR refine stage is sufficient.
|
||||
|
||||
#### 7. Reproducible streaming demo config lives only in Dreamverse
|
||||
|
||||
- **Priority:** P2.
|
||||
- **Effort:** Small.
|
||||
- **Owner:** FastVideo public.
|
||||
- **Evidence:** canonical demo config currently lives at
|
||||
[Dreamverse/serve_configs/streaming_demo.yaml](file:///home/william5lin/Dreamverse/serve_configs/streaming_demo.yaml#L1-L12).
|
||||
- **Config content:** it documents LTX-2 distilled model, one GPU,
|
||||
no offload, compile settings, NVFP4, refine overrides, default request,
|
||||
and streaming settings:
|
||||
[generator block](file:///home/william5lin/Dreamverse/serve_configs/streaming_demo.yaml#L31-L87),
|
||||
[streaming block](file:///home/william5lin/Dreamverse/serve_configs/streaming_demo.yaml#L108-L149).
|
||||
- **Memory pointer:** design.md already treats this as the canonical
|
||||
example:
|
||||
[design YAML example](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/design.md#L235-L239).
|
||||
- **Action:** copy/adapt it into
|
||||
`examples/serving/streaming_demo.yaml` with public-safe comments.
|
||||
|
||||
#### 8. LTX-2 stage equivalence is a verification gap, not proven drift
|
||||
|
||||
- **Priority:** P2.
|
||||
- **Effort:** Medium if parity checks are added; small if only manual audit.
|
||||
- **Owner:** FastVideo public.
|
||||
- **Public state:** model-specific LTX-2 stages are colocated under
|
||||
`fastvideo/pipelines/basic/ltx2/stages/`, consistent with the target
|
||||
layout in [design.md](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/design.md#L175-L205).
|
||||
- **Example public stage:** `ltx2_refine.py` explicitly says it is a
|
||||
public-side port of the internal stage and describes the three-stage SR
|
||||
flow:
|
||||
[ltx2_refine.py](file:///home/william5lin/FastVideo/fastvideo/pipelines/basic/ltx2/stages/ltx2_refine.py#L1-L22).
|
||||
- **Action:** verify behavior for the six internal `ltx2_*` stage files
|
||||
against public colocated stages. If a mismatch is found, file it as a
|
||||
real drift item with a failing parity test.
|
||||
|
||||
### Findings: deferred / accepted residual
|
||||
|
||||
These items should not block the public-dependency transition.
|
||||
|
||||
1. **StepVideo residual.**
|
||||
- Dreamverse's model registry is LTX-2/LTX-2.3 only:
|
||||
[Dreamverse config](file:///home/william5lin/Dreamverse/server/config.py#L28-L45).
|
||||
- Internal local tests even stub StepVideo modules to keep LTX registry
|
||||
tests focused:
|
||||
[test_ltx2_registry.py](file:///home/william5lin/FastVideo-internal/tests/local_tests/test_ltx2_registry.py#L38-L61).
|
||||
- Conclusion: accepted low-priority deferral unless Dreamverse adds a
|
||||
StepVideo model.
|
||||
|
||||
2. **Internal debug-only `FastVideoArgs` fields.**
|
||||
- Internal debug fields exist around `FastVideoArgs` and stage/model sums:
|
||||
[internal grep source](file:///home/william5lin/FastVideo-internal/fastvideo/fastvideo_args.py#L200-L203).
|
||||
- They are debug-only and not a public user-facing integration surface.
|
||||
- Conclusion: low-priority; do not add to public schema unless a debug
|
||||
workflow requires them.
|
||||
|
||||
3. **Private request aliases.**
|
||||
- Public request schema is nested and strict:
|
||||
[GenerationRequest](file:///home/william5lin/FastVideo/fastvideo/api/schema.py#L193-L204).
|
||||
- Legacy OpenAI flat fields are compatibility input, not the canonical
|
||||
public API:
|
||||
[internal protocol](file:///home/william5lin/FastVideo-internal/fastvideo/entrypoints/openai/protocol.py#L64-L82).
|
||||
- Conclusion: no action beyond current compat tests.
|
||||
|
||||
4. **`experimental["pipeline_config"]` escape hatch.**
|
||||
- Dreamverse currently uses an explicit in-memory quant config because
|
||||
typed `transformer_quant: "NVFP4"` does not expose `layer_profile`:
|
||||
[quantization](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/quantization.md#L86-L97).
|
||||
- Open thread #4 tracks `layer_profile`:
|
||||
[open threads](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/open-threads.md#L244-L260).
|
||||
- Conclusion: defer broader typed carrier design; add `layer_profile`
|
||||
first if Dreamverse needs base/refine profile selection.
|
||||
|
||||
5. **Router sticky routing and active-active semantics.**
|
||||
- Public router intentionally ships active-passive first and defers
|
||||
sticky/weighted routing:
|
||||
[D-15](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/decisions-log.md#L118-L155).
|
||||
- Follow-ups are tracked:
|
||||
[D-15 action items](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/decisions-log.md#L175-L201).
|
||||
- Conclusion: not drift; defer until load-balancing needs are real.
|
||||
|
||||
6. **AbsMaxFP8 failure.**
|
||||
- Pre-existing and not introduced by NVFP4:
|
||||
[state](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/state.md#L154-L159),
|
||||
[quantization](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/quantization.md#L202-L214).
|
||||
- Conclusion: fix separately; not a Dreamverse public-dependency blocker.
|
||||
|
||||
### Drift summary table
|
||||
|
||||
| # | Item | Priority | Effort | Status | Tracked where | Next action |
|
||||
|---:|---|---|---|---|---|---|
|
||||
| 1 | Dreamverse README still names `../FastVideo-internal` | P0 | S | Real drift | [README lines](file:///home/william5lin/Dreamverse/README.md#L76-L109) | Update docs to public FastVideo / PyPI path. |
|
||||
| 2 | Dreamverse private bootstrap clones internal repo | P0 | S | Real drift | [bootstrap script](file:///home/william5lin/Dreamverse/.agents/skills/bootstrap-fastvideo-private-fork/scripts/bootstrap_fastvideo_private.sh#L7-L11) | Replace or delete private bootstrap. |
|
||||
| 3 | Dreamverse `prompt_enhancer.py` fork | P1 | M | Real drift | [DR-1](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/open-threads.md#L174-L206) | Build compat shim over public enhancer. |
|
||||
| 4 | `cerebras_ifm` provider path | P1/P2 | S-M | Real drift / decision | [DR-2](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/open-threads.md#L211-L229) | Choose public provider vs Dreamverse custom provider. |
|
||||
| 5 | Health route mismatch | P1 | M-L | Real drift | [open item #1](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/open-threads.md#L69-L99) | Add `/healthz`, `/readyz`, `/status` to public build_app. |
|
||||
| 6 | Missing public streaming demo config | P2 | S | Real drift | [Dreamverse config](file:///home/william5lin/Dreamverse/serve_configs/streaming_demo.yaml#L1-L12) | Add `examples/serving/streaming_demo.yaml`. |
|
||||
| 7 | Standalone upsampler CLI | P2/P3 | S-M | Real drift if standalone CLI is desired | [internal CLI](file:///home/william5lin/FastVideo-internal/fastvideo/entrypoints/cli/upsample.py#L15-L35) | Decide CLI promise; port or defer. |
|
||||
| 8 | Layerwise offload utility | P3 | M | Optional internal-only residual (no Dreamverse deployment requires it today) | [internal manager](file:///home/william5lin/FastVideo-internal/fastvideo/models/layerwise_offload.py#L15-L20) | Defer until memory-tight deployment needs it. |
|
||||
| 9 | LTX-2 stage equivalence | P2 | S-M | Verification gap | [public refine stage](file:///home/william5lin/FastVideo/fastvideo/pipelines/basic/ltx2/stages/ltx2_refine.py#L1-L22) | Add targeted parity audit/test if needed. |
|
||||
| 10 | StepVideo | P3 | M | Accepted residual | [Dreamverse model registry](file:///home/william5lin/Dreamverse/server/config.py#L28-L45) | No action unless Dreamverse adds StepVideo. |
|
||||
| 11 | Debug-only fields | P3 | S | Accepted residual | [internal args](file:///home/william5lin/FastVideo-internal/fastvideo/fastvideo_args.py#L200-L203) | Do not publicize unless needed. |
|
||||
| 12 | `layer_profile` typed quant knob | P2 | M | Tracked gap | [open item #4](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/open-threads.md#L244-L260) | Add typed layer profile if Dreamverse drops escape hatch. |
|
||||
| 13 | `ltx2_image_crf` per-segment field flow (D-8) | P1 | S | Open verification gap — Dreamverse still passes `ltx2_image_crf=0.0` per [Dreamverse video_generation.py](file:///home/william5lin/Dreamverse/server/video_generation.py#L420-L435); needs trace-through to confirm it lands on `request.stage_overrides.refine.image_crf` rather than being silently dropped | [D-8 in open-threads](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/open-threads.md#L52-L68) | 10-min trace + add a Dreamverse-shape contract test pinning the field. |
|
||||
| 14 | `video_position_offset_sec` semantics (VPO) | P1 | S | Open decision — persistent-vs-per-segment ambiguity unresolved | [VPO in open-threads](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/open-threads.md#L120-L144) | Confirm semantics with audio team; document + add test. Decision deadline was "before PR 7.6 emits state" — that PR (7.6 / #1257) is now MERGED, so the decision is overdue. |
|
||||
| 15 | `GpuPool` ABC docstring missing experimental caveat (D-12-A) | P3 | trivial | Tracked gap — `GpuPool` ABC at [gpu_pool.py:74-83](file:///home/william5lin/FastVideo/fastvideo/entrypoints/streaming/gpu_pool.py#L74-L83) lacks the "API may change post-PR-7.10; experimental / server-internal" caveat | [D-12-A](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/open-threads.md#L301-L313) | Edit docstring; trivial. |
|
||||
| 16 | `GpuPool.run_async()` migration (D-12-B) | P2 | M | Tracked gap — `GpuPool.run() -> Any` should become `run_async() -> AsyncIterator[VideoEvent]` per D-12 / D-12-B | [D-12-B](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/open-threads.md#L317-L327) | Land alongside #1288 merge or in immediate follow-up. |
|
||||
| 17 | `SessionStore` / `BlobStore` lifecycle policy (SBS) | P2 | M | Tracked gap — in-memory defaults have no eviction/TTL/blob-cleanup policy | [SBS](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/open-threads.md#L278-L294) | Streaming server design pass needed before high-traffic deployment. |
|
||||
| 13 | Router sticky / active-active | P3 | M | Deferred | [D-15](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/decisions-log.md#L175-L201) | Defer until reconnect/load evidence. |
|
||||
| 14 | AbsMaxFP8 test failure | P2 | S | Separate tech debt | [state](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/state.md#L154-L159) | Fix outside Dreamverse migration. |
|
||||
| 15 | Dynamo backend package | P1 | External | Not FastVideo drift | [Dynamo contract](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/cross-repo-surfaces.md#L188-L210) | Reopen Dynamo-side PR after public API lands. |
|
||||
|
||||
---
|
||||
|
||||
## Part 2 — Integration path tradeoffs
|
||||
|
||||
### The four options
|
||||
|
||||
#### Option A — Status quo: Dreamverse stays separate and depends on `fastvideo`
|
||||
|
||||
**Shape**
|
||||
|
||||
- FastVideo remains the Python library and reusable backend runtime.
|
||||
- Dreamverse remains the product repo with FastAPI product glue and Next.js
|
||||
frontend.
|
||||
- Dreamverse `server` extra depends on `fastvideo>=0.1.7`:
|
||||
[pyproject](file:///home/william5lin/Dreamverse/pyproject.toml#L17-L22).
|
||||
- Local development can keep using editable `../FastVideo` until PyPI
|
||||
publication catches up:
|
||||
[uv.lock](file:///home/william5lin/Dreamverse/uv.lock#L716-L722).
|
||||
|
||||
**What it solves**
|
||||
|
||||
- Directly satisfies "Dreamverse depends on public FastVideo".
|
||||
- Keeps frontend release cadence independent.
|
||||
- Keeps product-specific prompts, routes, and UI in the product repo.
|
||||
- Minimizes FastVideo packaging and CI growth.
|
||||
|
||||
**What it does not solve by itself**
|
||||
|
||||
- Does not remove Dreamverse prompt-enhancer fork unless DR-1 is executed.
|
||||
- Does not give Dreamverse FE compatibility with public `build_app` until
|
||||
health routes migrate.
|
||||
- Does not make Dreamverse server itself reusable as a public entrypoint.
|
||||
|
||||
**Best fit**
|
||||
|
||||
- Default for the next release if the goal is to stop using
|
||||
FastVideo-internal quickly and safely.
|
||||
|
||||
#### Option B — Dreamverse as a subfolder under FastVideo
|
||||
|
||||
**Shape**
|
||||
|
||||
- One repository: FastVideo contains `dreamverse/server/` and
|
||||
`dreamverse/apps/web/`.
|
||||
- Dreamverse can remain a separate package in the same repo, or FastVideo's
|
||||
build can ignore Dreamverse by default.
|
||||
- CI must understand Python library tests plus Next.js install/build/test.
|
||||
|
||||
**What it solves**
|
||||
|
||||
- Eliminates sibling-checkout drift.
|
||||
- Makes cross-repo integration changes atomic.
|
||||
- Easier for a single reviewer to see library and product changes together.
|
||||
|
||||
**Costs**
|
||||
|
||||
- Adds frontend dependency management to a Python ML library repo.
|
||||
- Couples clone size, CI setup, issue tracking, and review load.
|
||||
- Forces maintainers to decide whether product assets are included in source
|
||||
distributions, wheels, docs, and release notes.
|
||||
|
||||
**Best fit**
|
||||
|
||||
- Only if Dreamverse becomes the primary FastVideo product surface and the
|
||||
team accepts a product monorepo.
|
||||
|
||||
#### Option C — Full merge into `fastvideo.entrypoints.dreamverse.*`
|
||||
|
||||
**Shape**
|
||||
|
||||
- Dreamverse backend becomes FastVideo code.
|
||||
- Public import becomes something like
|
||||
`from fastvideo.entrypoints.dreamverse import build_app`.
|
||||
- CLI becomes `fastvideo dreamverse-serve --config dreamverse.yaml`.
|
||||
- Frontend either ships as static assets in the package or as a frontend
|
||||
extra.
|
||||
|
||||
**What it solves**
|
||||
|
||||
- One namespace and one release train for library plus product backend.
|
||||
- No dependency boundary between Dreamverse server and FastVideo internals.
|
||||
- Product route contract can be tested entirely inside FastVideo CI.
|
||||
|
||||
**Costs**
|
||||
|
||||
- Maximally expands FastVideo's public/security surface.
|
||||
- Locks product experiments to FastVideo release cadence.
|
||||
- Makes private prompt/provider/product assumptions look like framework API.
|
||||
- Has weak precedent for a Python ML library plus Next.js product being merged
|
||||
into the library namespace.
|
||||
|
||||
**Best fit**
|
||||
|
||||
- Only if Dreamverse is no longer a separate product and becomes the
|
||||
canonical FastVideo UI/serving mode.
|
||||
|
||||
#### Option D — Hybrid: backend merges, frontend stays separate
|
||||
|
||||
**Shape**
|
||||
|
||||
- Reusable backend components merge into public FastVideo.
|
||||
- Frontend stays in a separate Dreamverse UI repo or Dreamverse product repo.
|
||||
- The backend should be generic where possible: `fastvideo.entrypoints.streaming`,
|
||||
not product-only names, unless product-only routes are intentionally
|
||||
accepted as public API.
|
||||
- This matches the current trajectory: streaming server, GPU pool, prompt
|
||||
enhancer, safety/rewrite/session logging, router, NVFP4, and
|
||||
`generate_async` are public-side work already tracked in the PR roadmap:
|
||||
[pr-roadmap](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/pr-roadmap.md#L19-L42).
|
||||
|
||||
**What it solves**
|
||||
|
||||
- Removes FastVideo-internal dependency for reusable backend pieces.
|
||||
- Keeps product frontend cadence independent.
|
||||
- Gives non-Dreamverse users a streaming backend and typed API without
|
||||
carrying the Dreamverse app.
|
||||
- Gives Dynamo a stable library API while leaving Dynamo package code in
|
||||
Dynamo:
|
||||
[Dynamo contract](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/cross-repo-surfaces.md#L188-L210).
|
||||
|
||||
**Costs**
|
||||
|
||||
- Requires careful boundary discipline: generic streaming/server code in
|
||||
FastVideo; product routes/prompts/presets in Dreamverse.
|
||||
- Requires contract tests to prevent drift.
|
||||
- Some Dreamverse compatibility routes may become public and need support.
|
||||
|
||||
**Best fit**
|
||||
|
||||
- Best long-term target if the team wants FastVideo to own serving/runtime
|
||||
infrastructure while keeping Dreamverse as a separately evolving product.
|
||||
|
||||
### Comparison matrix
|
||||
|
||||
| Criterion | A. Separate dep | B. Subfolder monorepo | C. Full namespace merge | D. Hybrid backend merge |
|
||||
|---|---|---|---|---|
|
||||
| Alignment with stated goal | High: Dreamverse depends on public package | Medium: no external dep, but product becomes repo-local | Medium: dependency disappears by absorption | High: reusable backend in public, product separate |
|
||||
| Time to remove `FastVideo-internal` | Fastest | Medium | Slowest | Medium-fast |
|
||||
| Build complexity | Low | High: Python + Next.js in one repo | High: Python package plus static/frontend extras | Medium: Python backend only in FastVideo |
|
||||
| Release cadence | Independent | Coupled clone; releases can still be separate but more friction | Fully coupled | Backend coupled to FastVideo, frontend independent |
|
||||
| Security surface in FastVideo | Low | Medium/High | Highest | Medium |
|
||||
| Contributor friction | Low for both repos | Higher for library contributors | Highest; product assumptions in library | Medium; clear backend boundary needed |
|
||||
| Dependency management | Normal package pin | Workspace/monorepo tooling needed | FastVideo extras/static asset decisions needed | FastVideo extras for backend; FE out-of-tree |
|
||||
| CI cost | Low/medium | High | High | Medium |
|
||||
| Contract-test value | High; cross-repo contract tests are essential | Medium; same repo but still useful | Medium; less boundary pressure | High; generic backend vs product boundary |
|
||||
| Precedent strength | Strong: library/server plus external UI patterns exist | Mixed | Weak for Python ML library + Next.js inside namespace | Strongest match: in-tree server/backend, external UI |
|
||||
| Packaging risk | Low | Medium/high | High | Medium |
|
||||
| Future Dynamo fit | Strong | Strong if API remains clean | Risky if product API bleeds in | Strong |
|
||||
| Frontend iteration speed | Highest | Lower | Lowest | Highest |
|
||||
| Risk of product-specific API leakage | Low | Medium | High | Medium; controllable with naming discipline |
|
||||
| Reversibility | High | Medium | Low | Medium/high |
|
||||
|
||||
### OSS precedents (with citations)
|
||||
|
||||
| Pattern | Project | What it supports | Citation |
|
||||
|---|---|---|---|
|
||||
| Library plus in-tree server | vLLM | A Python ML library can ship an in-tree OpenAI-compatible server while clients remain external. | https://github.com/vllm-project/vllm/blob/bcf5cac9fb956788f649d1f5297b74c886a9d6d3/README.md#L64-L74 |
|
||||
| Service packaging | BentoML | Packaging model + service + dependencies is supported, but CWD packaging creates discipline needs. | https://github.com/bentoml/BentoML/blob/32230a5276a8da8b23c4a06a9ec6272c1993451a/docs/source/build-with-bentoml/asgi.rst#L5-L18 |
|
||||
| YAML-driven production serving | Ray Serve | Production updates should avoid in-place mutation; use new deployment/traffic switch. | https://docs.ray.io/en/latest/serve/advanced-guides/inplace-updates.html |
|
||||
| Library/server plus external UI | TGI + ChatUI | Server can live with backend project while UI is separate. | https://github.com/huggingface/text-generation-inference/blob/b4adbf2f6e2e721280bd0ea5f91d70f7d033f5ed/docs/source/basic_tutorials/consuming_tgi.md#L182-L186 |
|
||||
| Lean library plus examples elsewhere | Transformers.js | Library stays lean; demos/examples can live outside core. | https://github.com/huggingface/transformers.js/blob/f7487c737aa8cafbc106c9adf69dc9578c8f3fe0/README.md#L26-L34 |
|
||||
| Product monorepo that later split frontend | ComfyUI | Product UI/server monorepo can hit release-cadence mismatch and split FE later. | https://github.com/comfyanonymous/ComfyUI/blob/fed8d5efa6b70d5b24c4c33cb643bfccc39d45b5/README.md#L131-L149 and https://github.com/Comfy-Org/ComfyUI_frontend/blob/60f789d58070a9d1d789b260f83c36d7293a39f0/README.md#L31-L60 |
|
||||
| Tightly coupled UI/server product | AUTOMATIC1111 SD WebUI | Product repos can couple UI/server tightly, but security surface becomes product-sized. | https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/82a973c04367123ae98bd9abdf80d9eda9b910e2/webui.py#L48-L104 |
|
||||
|
||||
#### Precedent synthesis
|
||||
|
||||
- Strong precedents exist for a Python ML library shipping a server entrypoint.
|
||||
- Strong precedents exist for keeping frontend/product UI out of the backend
|
||||
library repo.
|
||||
- The cited set does not contain a clean precedent for merging a Next.js
|
||||
product into a Python ML library namespace.
|
||||
- The most applicable pattern is **backend/server in the ML project,
|
||||
product UI outside**.
|
||||
|
||||
### Recommendation
|
||||
|
||||
#### Recommend Option D, constrained: backend merges as generic FastVideo streaming; frontend stays separate
|
||||
|
||||
Recommendation: follow **Option D** as the long-term architecture, but keep
|
||||
the backend merge generic. In practice, this means continuing the current
|
||||
public FastVideo path:
|
||||
|
||||
- `fastvideo.entrypoints.streaming.*` owns reusable streaming runtime.
|
||||
- `fastvideo.entrypoints.streaming.gpu_pool` owns generic GPU worker pools.
|
||||
- `fastvideo.entrypoints.streaming.prompt.*` owns provider-agnostic prompt
|
||||
operations.
|
||||
- `fastvideo.entrypoints.streaming.router.*` owns FastVideo-aware routing.
|
||||
- `fastvideo.api` owns typed construction, requests, results, events, and
|
||||
continuation state.
|
||||
- Dreamverse keeps product-only FE, curated presets, prompt UX, product
|
||||
routes, and launch scripts.
|
||||
|
||||
This is effectively the path already underway in PRs #1257, #1258, #1284,
|
||||
#1286, and #1288:
|
||||
[PR roadmap](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/pr-roadmap.md#L21-L42).
|
||||
|
||||
#### Why not Option A as the final answer?
|
||||
|
||||
Option A is the fastest near-term release posture and should be used as the
|
||||
immediate migration posture. However, plain status quo is not enough for
|
||||
the ultimate goal because reusable backend pieces still need to live in
|
||||
public FastVideo so Dreamverse can stop reaching into internal code. That
|
||||
work is already partly complete:
|
||||
|
||||
- GPU pool: [D-12](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/decisions-log.md#L47-L116).
|
||||
- Prompt enhancer: [PR roadmap 7.7](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/pr-roadmap.md#L32-L35).
|
||||
- Streaming auxiliaries: [PR roadmap 7.8](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/pr-roadmap.md#L35-L36).
|
||||
- Router: [D-15](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/decisions-log.md#L118-L155).
|
||||
- `generate_async`: [streaming-server unlock PR](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/streaming-server.md#L314-L345).
|
||||
|
||||
So the practical answer is:
|
||||
|
||||
- **Near term:** Option A operationally, after docs/scripts are fixed.
|
||||
- **Architecture target:** Option D, with generic backend ownership in
|
||||
FastVideo and product ownership in Dreamverse.
|
||||
|
||||
#### Why not Option B?
|
||||
|
||||
Option B makes cross-repo coordination easier but imports frontend build,
|
||||
package, and CI complexity into FastVideo. That is unnecessary while a
|
||||
normal package dependency plus contract tests can guard the integration.
|
||||
FastVideo's current repo structure is a Python package with examples and
|
||||
docs, not a product monorepo:
|
||||
[codebase map](file:///home/william5lin/FastVideo/.agents/memory/codebase-map/README.md#L5-L75).
|
||||
|
||||
#### Why not Option C?
|
||||
|
||||
Option C makes the product backend a public FastVideo namespace. That is
|
||||
only appropriate if the team wants to support Dreamverse as a first-class
|
||||
FastVideo product surface. Today the known public obligations are generic:
|
||||
typed requests, streaming server, GPU pool, prompt provider protocol,
|
||||
router, NVFP4, and Dynamo event APIs. Product-only Dreamverse behavior does
|
||||
not need to become framework API.
|
||||
|
||||
#### Conditions that would change the recommendation
|
||||
|
||||
Move from constrained D toward **C** only if all of these become true:
|
||||
|
||||
1. Dreamverse is declared the canonical FastVideo serving product.
|
||||
2. Product routes such as curated presets and prompt-system config are
|
||||
accepted as public FastVideo API.
|
||||
3. FastVideo maintainers accept the security and support surface.
|
||||
4. Release cadence for product UX and FastVideo core is intentionally
|
||||
coupled.
|
||||
5. Frontend packaging/static asset strategy is explicitly owned by
|
||||
FastVideo.
|
||||
|
||||
Move from constrained D back toward **A** if any of these become true:
|
||||
|
||||
1. Prompt enhancement, router, or GPU pool turn out to be Dreamverse-only.
|
||||
2. No second user appears for the streaming backend outside Dreamverse.
|
||||
3. FastVideo maintainers want to minimize serving surface and publish only
|
||||
Python library APIs.
|
||||
4. Dreamverse needs product changes faster than FastVideo can release.
|
||||
5. Security review rejects in-tree serving/router responsibilities.
|
||||
|
||||
### Migration sketch for the recommended path
|
||||
|
||||
#### Phase 0 — Land the public backend stack
|
||||
|
||||
- **Effort:** Large, already in flight.
|
||||
- **Owner:** FastVideo public.
|
||||
- **Files:** #1288 scope, especially `fastvideo/api/`,
|
||||
`fastvideo/entrypoints/video_generator.py`,
|
||||
`fastvideo/entrypoints/streaming/`, LTX-2 pipeline stages, NVFP4 files,
|
||||
and contract tests.
|
||||
- **Exit criteria:** #1288 merges; public `fastvideo.api.VideoEvent` and
|
||||
`VideoGenerator.generate_async` are available:
|
||||
[results.py](file:///home/william5lin/FastVideo/fastvideo/api/results.py#L109-L164),
|
||||
[video_generator.py](file:///home/william5lin/FastVideo/fastvideo/entrypoints/video_generator.py#L264-L332).
|
||||
|
||||
#### Phase 1 — Fix Dreamverse dependency docs and bootstrap
|
||||
|
||||
- **Effort:** Small.
|
||||
- **Owner:** Dreamverse.
|
||||
- **Files:**
|
||||
- [README.md](file:///home/william5lin/Dreamverse/README.md#L76-L109)
|
||||
- [bootstrap script](file:///home/william5lin/Dreamverse/.agents/skills/bootstrap-fastvideo-private-fork/scripts/bootstrap_fastvideo_private.sh#L7-L11)
|
||||
- [pyproject.toml](file:///home/william5lin/Dreamverse/pyproject.toml#L17-L22)
|
||||
- [uv.lock](file:///home/william5lin/Dreamverse/uv.lock#L716-L722)
|
||||
- **Exit criteria:** no user-facing docs or scripts mention
|
||||
`FastVideo-internal` as the expected dependency path.
|
||||
|
||||
#### Phase 2 — Add public health/readiness/status route compatibility
|
||||
|
||||
- **Effort:** Medium/Large.
|
||||
- **Owner:** FastVideo public.
|
||||
- **Files likely to touch:**
|
||||
- `fastvideo/entrypoints/streaming/server.py::build_app`
|
||||
- new `fastvideo/entrypoints/streaming/health.py`
|
||||
- tests under `fastvideo/tests/entrypoints/streaming/`
|
||||
- **Source route shapes:**
|
||||
[Dreamverse health routes](file:///home/william5lin/Dreamverse/server/routes/health.py#L34-L79).
|
||||
- **Tracking:** [open item #1](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/open-threads.md#L69-L99).
|
||||
- **Exit criteria:** Dreamverse FE can target public `build_app` for
|
||||
`/healthz`, `/readyz`, `/status`, and `/v1/stream`; product-only routes
|
||||
remain feature-detected.
|
||||
|
||||
#### Phase 3 — Replace Dreamverse prompt enhancer fork
|
||||
|
||||
- **Effort:** Medium.
|
||||
- **Owner:** Dreamverse.
|
||||
- **Files likely to touch:**
|
||||
- new `Dreamverse/server/prompting/_internal_compat.py`
|
||||
- `Dreamverse/server/runtime.py`
|
||||
- `Dreamverse/server/main.py`
|
||||
- `Dreamverse/server/prompt_enhancer.py`
|
||||
- **Public API:**
|
||||
[PromptEnhancer](file:///home/william5lin/FastVideo/fastvideo/entrypoints/streaming/prompt/enhancer.py#L66-L142),
|
||||
[LLMProvider](file:///home/william5lin/FastVideo/fastvideo/entrypoints/streaming/prompt/providers/base.py#L63-L75).
|
||||
- **Tracking:** [DR-1](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/open-threads.md#L174-L206).
|
||||
- **Exit criteria:** Dreamverse no longer carries a full local fork for the
|
||||
generic prompt operations public FastVideo already owns.
|
||||
|
||||
#### Phase 4 — Decide and implement `cerebras_ifm`
|
||||
|
||||
- **Effort:** Small decision plus small/medium implementation.
|
||||
- **Owner:** Team decision, then Dreamverse or FastVideo.
|
||||
- **Default recommendation:** Dreamverse-side custom provider.
|
||||
- **Public schema source:**
|
||||
[PromptEnhancerConfig](file:///home/william5lin/FastVideo/fastvideo/api/schema.py#L229-L235).
|
||||
- **Tracking:** [DR-2](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/open-threads.md#L211-L229).
|
||||
- **Exit criteria:** Dreamverse IFM provider works after prompt fork removal.
|
||||
|
||||
#### Phase 5 — Move streaming demo config into FastVideo examples
|
||||
|
||||
- **Effort:** Small.
|
||||
- **Owner:** FastVideo public.
|
||||
- **Source:**
|
||||
[Dreamverse streaming_demo.yaml](file:///home/william5lin/Dreamverse/serve_configs/streaming_demo.yaml#L1-L149).
|
||||
- **Target:** `examples/serving/streaming_demo.yaml`.
|
||||
- **Exit criteria:** users can reproduce the typed streaming path from the
|
||||
FastVideo repo without checking out Dreamverse.
|
||||
|
||||
#### Phase 6 — Remove `experimental["pipeline_config"]` where practical
|
||||
|
||||
- **Effort:** Medium for `layer_profile`; Large for a full typed
|
||||
`dit_config.quant_config` carrier.
|
||||
- **Owner:** FastVideo public, then Dreamverse cleanup.
|
||||
- **Tracking:**
|
||||
[open item #4](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/open-threads.md#L244-L260),
|
||||
[quantization follow-up](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/quantization.md#L175-L200).
|
||||
- **Exit criteria:** Dreamverse can express its quant layer profile through
|
||||
typed config instead of in-memory mutation.
|
||||
|
||||
#### Phase 7 — Decide standalone upsampler CLI
|
||||
|
||||
- **Effort:** Small/Medium.
|
||||
- **Owner:** FastVideo public.
|
||||
- **Input:** internal standalone utility
|
||||
[upsample.py](file:///home/william5lin/FastVideo-internal/fastvideo/entrypoints/upsample.py#L120-L180)
|
||||
and internal CLI
|
||||
[cli/upsample.py](file:///home/william5lin/FastVideo-internal/fastvideo/entrypoints/cli/upsample.py#L48-L130).
|
||||
- **Public alternative:** SR refine stage already covers in-pipeline latent
|
||||
upsampling:
|
||||
[ltx2_refine.py](file:///home/william5lin/FastVideo/fastvideo/pipelines/basic/ltx2/stages/ltx2_refine.py#L116-L180).
|
||||
- **Exit criteria:** explicit decision: port CLI, document refine-stage-only
|
||||
support, or defer.
|
||||
|
||||
#### Phase 8 — Validate LTX-2 stage parity and offload residuals
|
||||
|
||||
- **Effort:** Small/Medium for stage parity; Medium for layerwise offload.
|
||||
- **Owner:** FastVideo public.
|
||||
- **Stage source:**
|
||||
[public LTX-2 stages](file:///home/william5lin/FastVideo/fastvideo/pipelines/basic/ltx2/stages/).
|
||||
- **Offload source:**
|
||||
[internal layerwise offload](file:///home/william5lin/FastVideo-internal/fastvideo/models/layerwise_offload.py#L15-L20).
|
||||
- **Exit criteria:** no known behavior gap between internal and public LTX-2
|
||||
stages; offload is either deliberately deferred or ported with tests.
|
||||
|
||||
### Open questions
|
||||
|
||||
1. **Which provider path for `cerebras_ifm`?**
|
||||
- Public provider or Dreamverse-side custom provider?
|
||||
- Default recommendation: Dreamverse-side unless there is another user.
|
||||
- Source: [DR-2](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/open-threads.md#L211-L229).
|
||||
|
||||
2. **Should public FastVideo support standalone LTX-2 file upsampling?**
|
||||
- If yes, port internal CLI.
|
||||
- If no, document that SR support is pipeline-refine only.
|
||||
- Sources: [internal CLI](file:///home/william5lin/FastVideo-internal/fastvideo/entrypoints/cli/upsample.py#L15-L35),
|
||||
[public refine stage](file:///home/william5lin/FastVideo/fastvideo/pipelines/basic/ltx2/stages/ltx2_refine.py#L1-L22).
|
||||
|
||||
3. **Does Dreamverse need layerwise CPU offload?**
|
||||
- If memory-tight deployments require it, port as generic FastVideo.
|
||||
- Otherwise defer.
|
||||
- Source: [internal offload manager](file:///home/william5lin/FastVideo-internal/fastvideo/models/layerwise_offload.py#L15-L20).
|
||||
|
||||
4. **How much Dreamverse route surface should FastVideo own?**
|
||||
- Health/readiness/status should migrate because they are part of
|
||||
streaming-server compatibility.
|
||||
- Curated presets and prompt-system config should stay Dreamverse-side
|
||||
unless product policy changes.
|
||||
- Source: [route contract](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/streaming-server.md#L232-L259).
|
||||
|
||||
5. **Should `layer_profile` be the only near-term quant typed addition?**
|
||||
- Adding `layer_profile` is bounded.
|
||||
- A typed carrier for arbitrary mutated `PipelineConfig` is larger design
|
||||
work.
|
||||
- Source: [quantization follow-ups](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/quantization.md#L175-L200).
|
||||
|
||||
6. **When does Option D become Option C?**
|
||||
- Only if Dreamverse backend routes become public FastVideo product API.
|
||||
- Until then, keep generic streaming code in FastVideo and product code in
|
||||
Dreamverse.
|
||||
|
||||
---
|
||||
|
||||
## Part 3 — Action items
|
||||
|
||||
1. **P0 / S — Update Dreamverse README dependency notes.**
|
||||
- Replace `../FastVideo-internal` with public FastVideo instructions.
|
||||
- Preserve local editable `../FastVideo` dev flow where useful.
|
||||
- Source: [README stale lines](file:///home/william5lin/Dreamverse/README.md#L76-L109).
|
||||
|
||||
2. **P0 / S — Replace or remove private FastVideo bootstrap script.**
|
||||
- Current script clones `FastVideo-internal` and verifies imports from it.
|
||||
- Source: [script](file:///home/william5lin/Dreamverse/.agents/skills/bootstrap-fastvideo-private-fork/scripts/bootstrap_fastvideo_private.sh#L7-L11).
|
||||
|
||||
3. **P1 / M-L — Add `/healthz`, `/readyz`, and `/status` to public `build_app`.**
|
||||
- Keep `/curated-presets` and prompt-system config in Dreamverse.
|
||||
- Source: [open item #1](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/open-threads.md#L69-L99).
|
||||
|
||||
4. **P1 / M — Replace Dreamverse prompt-enhancer fork with compat shim.**
|
||||
- Wrap public `PromptEnhancer`.
|
||||
- Keep only Dreamverse-specific metadata and product fallback behavior.
|
||||
- Source: [DR-1](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/open-threads.md#L174-L206).
|
||||
|
||||
5. **P1 / S-M — Decide `cerebras_ifm` provider path.**
|
||||
- Default: Dreamverse custom provider via `register_provider`.
|
||||
- Source: [DR-2](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/open-threads.md#L211-L229).
|
||||
|
||||
6. **P2 / S — Add `examples/serving/streaming_demo.yaml` to FastVideo.**
|
||||
- Start from Dreamverse config and remove Dreamverse-private comments.
|
||||
- Source: [streaming_demo.yaml](file:///home/william5lin/Dreamverse/serve_configs/streaming_demo.yaml#L1-L149).
|
||||
|
||||
7. **P2 / M — Verify each public LTX-2 colocated stage against internal behavior.**
|
||||
- Start with refine, denoising, latent prep, image conditioning, text
|
||||
encoding, and audio decoding.
|
||||
- Source: [public refine stage](file:///home/william5lin/FastVideo/fastvideo/pipelines/basic/ltx2/stages/ltx2_refine.py#L1-L22).
|
||||
|
||||
8. **P2 / M — Add typed `transformer_quant_layer_profile` if Dreamverse needs it.**
|
||||
- Thread schema → compat → `FastVideoArgs._apply_transformer_quant`.
|
||||
- Source: [open item #4](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/open-threads.md#L244-L260).
|
||||
|
||||
9. **P2 / S-M — Decide standalone LTX-2 upsampler CLI support.**
|
||||
- Port internal CLI only if file-to-file upsampling is a public workflow.
|
||||
- Source: [internal upsample CLI](file:///home/william5lin/FastVideo-internal/fastvideo/entrypoints/cli/upsample.py#L15-L35).
|
||||
|
||||
10. **P2 / S — Fix pre-existing AbsMaxFP8 test failure separately.**
|
||||
- Do not block Dreamverse migration on it.
|
||||
- Source: [open item #2](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/open-threads.md#L100-L117).
|
||||
|
||||
11. **P2 / S-M — Document public streaming install extras and dependencies.**
|
||||
- Include router `websockets`, prompt enhancer provider SDKs, and optional
|
||||
safety classifier extras.
|
||||
- Source: [D-16 dependency note](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/decisions-log.md#L224-L242).
|
||||
|
||||
12. **P2 / S — Keep contract tests in the FastVideo CI path.**
|
||||
- Guard Dreamverse shape, Dynamo shape, and async events.
|
||||
- Sources: [Dreamverse test](file:///home/william5lin/FastVideo/fastvideo/tests/contract/test_dreamverse_shape.py#L1-L26),
|
||||
[Dynamo test](file:///home/william5lin/FastVideo/fastvideo/tests/contract/test_dynamo_shape.py#L1-L19),
|
||||
[async test](file:///home/william5lin/FastVideo/fastvideo/tests/contract/test_generate_async.py#L1-L7).
|
||||
|
||||
13. **P3 / M — Defer layerwise offload until a deployment needs it.**
|
||||
- Port only as generic FastVideo utility with tests.
|
||||
- Source: [internal offload](file:///home/william5lin/FastVideo-internal/fastvideo/models/layerwise_offload.py#L15-L20).
|
||||
|
||||
14. **P3 / M — Defer StepVideo public parity for this integration.**
|
||||
- Dreamverse model registry is LTX-2/LTX-2.3 only.
|
||||
- Source: [Dreamverse config](file:///home/william5lin/Dreamverse/server/config.py#L28-L45).
|
||||
|
||||
15. **P3 / S — Do not add debug-only fields to public schema by default.**
|
||||
- Keep them private unless there is a user-facing debugging workflow.
|
||||
- Source: [internal debug args](file:///home/william5lin/FastVideo-internal/fastvideo/fastvideo_args.py#L200-L203).
|
||||
|
||||
16. **P3 / M — Keep router active-active and sticky routing deferred.**
|
||||
- Add only when session-routing evidence justifies it.
|
||||
- Source: [D-15 action items](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/decisions-log.md#L175-L201).
|
||||
|
||||
17. **P3 / S — Preserve Dynamo as an external backend package.**
|
||||
- FastVideo should expose typed API; Dynamo code lives in Dynamo.
|
||||
- Source: [Dynamo contract](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/cross-repo-surfaces.md#L188-L210).
|
||||
|
||||
18. **P3 / S — After #1288 merges, update memory-dir state.**
|
||||
- Mark item D resolved and update branch tips.
|
||||
- Source: [runbook post-merge steps](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/runbook.md#L51-L70).
|
||||
|
||||
19. **P3 / S — Remove stale split-PR mental model from follow-up docs.**
|
||||
- #1288 is the current vehicle; split bookmarks are historical.
|
||||
- Source: [D-17 implications](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/decisions-log.md#L34-L45).
|
||||
|
||||
20. **P3 / S — Keep product-only Dreamverse frontend out of FastVideo unless explicitly re-scoped.**
|
||||
- This preserves release cadence and avoids packaging bloat.
|
||||
- Source: [Dreamverse baseline](file:///home/william5lin/Dreamverse/README.md#L5-L16).
|
||||
@@ -7,7 +7,7 @@ For why each item is open see [decisions-log.md](decisions-log.md). For
|
||||
PR-level context see [pr-roadmap.md](pr-roadmap.md).
|
||||
|
||||
**Last updated:** 2026-05-05 (strategy reversal — PR #1287 CLOSED, replaced
|
||||
by mega-PR #1288 on `will/ltx2_sr_port` @ `39dfa009` covering the full
|
||||
by mega-PR #1288 on `will/ltx2_sr_port` @ `b36bdbc9` covering the full
|
||||
6-layer stack at once. See [decisions-log.md D-17](decisions-log.md#d-17).
|
||||
Item D resolution gate is now #1288 merge instead of #1287; same content,
|
||||
different vehicle.).
|
||||
@@ -20,7 +20,7 @@ different vehicle.).
|
||||
| **1** | High | Migrate `/healthz`+`/readyz`+`/status` into FastVideo `build_app` | M-L | Closes BE_FLAVOR=fastvideo FE-compatibility; closes streaming-upstream contract debt |
|
||||
| **2** | High | Fix pre-existing AbsMaxFP8 test failure | S | Self-contained quantization tech debt |
|
||||
| **VPO** | High | Decide `video_position_offset_sec` semantics (a vs b) | 30 min | Unblocks PR 7.6 state emission |
|
||||
| **D** | 🟢 in flight | Implement `generate_async` — content shipped in mega-PR **#1288** on `will/ltx2_sr_port` @ `39dfa009` (was #1287, CLOSED + re-routed per [D-17](decisions-log.md#d-17)) | L | Closes Q-5/Q-9/PR-7.5 TODOs simultaneously; enables Dynamo backend; unblocks audio re-encode; enables `GpuPool.run_async()` migration (D-12-B). Resolution gate: #1288 merge. |
|
||||
| **D** | 🟢 in flight | Implement `generate_async` — content shipped in mega-PR **#1288** on `will/ltx2_sr_port` @ `b36bdbc9` (was #1287, CLOSED + re-routed per [D-17](decisions-log.md#d-17)) | L | Closes Q-5/Q-9/PR-7.5 TODOs simultaneously; enables Dynamo backend; unblocks audio re-encode; enables `GpuPool.run_async()` migration (D-12-B). Resolution gate: #1288 merge. |
|
||||
| **DR-1** | High | Dreamverse: create `prompting/_internal_compat.py` shim + replace local `prompt_enhancer.py` (1933 LOC) — **PR #1258 has merged (`f673423b`); now actionable** | M (~150-200 LOC shim, replace upstream wiring) | Lets Dreamverse stop carrying a 1933-LOC fork |
|
||||
| **DR-2** | Med | Decide `cerebras_ifm` provider path: (a) public Literal + `CerebrasIFMProvider` shipped, OR (b) Dreamverse-side custom provider via `enhancer.register_provider(...)` | S (decision) + S-M (impl) | Resolves the cerebras_ifm gap left by PR #1258. Same item as legacy #3 below; DR-2 is the Dreamverse-side framing. |
|
||||
| **3** | Med | Add `cerebras_ifm` to `PromptEnhancerConfig.provider` Literal + provider | S-M | Public-side resolution if DR-2 picks (a) |
|
||||
|
||||
@@ -39,7 +39,7 @@ work that runs parallel to this sequence see [quantization.md](quantization.md).
|
||||
|
||||
| # | PR | Status | Branch | Scope |
|
||||
|---|---|---|---|---|
|
||||
| **mega** | **#1288** | 🟢 OPEN, MERGEABLE | `will/ltx2_sr_port` (head `39dfa009`) | **Single consolidated landing of the full `will/ltx2_sr_port` chain.** Was originally planned as 6 stacked PRs (slices 1-3 / 4-6 / 7-15 / 16-21 / 22-23 / 24-34). Now landing as one PR — see [decisions-log.md D-17](decisions-log.md#d-17) for the strategy decision. **Contents** (commit-ordered): (1) streaming `generate_async` + `VideoEvent` + Dynamo backend contract (3 commits, was PR 7.10/#1287 closed); (2) server contract docs + Dreamverse/Dynamo shape tests (3 commits, was PR 8); (3) LTX-2 SR runtime port + i2v conditioning + alignment harness (9 commits); (4) NVFP4 wire-up + per-component compile + typed `transformer_quant` flow (6 commits); (5) LTX-2 post-handoff parity fixes — Gemma `to()`, list-of-generators (2 commits); (6) `.agents/memory/dreamverse-integration/` knowledge base + agents Phase 1 cleanup (11 commits). 34 commits total, 71 files, +13,074/-583 LOC. |
|
||||
| **mega** | **#1288** | 🟢 OPEN, MERGEABLE | `will/ltx2_sr_port` (head `b36bdbc9`) | **Single consolidated landing of the full `will/ltx2_sr_port` chain.** Was originally planned as 6 stacked PRs (slices 1-3 / 4-6 / 7-15 / 16-21 / 22-23 / 24-34). Now landing as one PR — see [decisions-log.md D-17](decisions-log.md#d-17) for the strategy decision. **Contents** (commit-ordered): (1) streaming `generate_async` + `VideoEvent` + Dynamo backend contract (3 commits, was PR 7.10/#1287 closed); (2) server contract docs + Dreamverse/Dynamo shape tests (3 commits, was PR 8); (3) LTX-2 SR runtime port + i2v conditioning + alignment harness (9 commits); (4) NVFP4 wire-up + per-component compile + typed `transformer_quant` flow (6 commits); (5) LTX-2 post-handoff parity fixes — Gemma `to()`, list-of-generators (2 commits); (6) `.agents/memory/dreamverse-integration/` knowledge base + agents Phase 1 cleanup (11 commits). 34 commits total, 71 files, +13,074/-583 LOC. |
|
||||
|
||||
## Closed PRs in this scope
|
||||
|
||||
|
||||
@@ -11,7 +11,7 @@ roster of co-authors to credit on every commit see
|
||||
|
||||
| Repo | Branch | Tip | Distance |
|
||||
|---|---|---|---|
|
||||
| FastVideo | `will/ltx2_sr_port` (**PR #1288 head**) | `39dfa009` | 34 commits ahead of `origin/main`; OPEN, MERGEABLE |
|
||||
| FastVideo | `will/ltx2_sr_port` (**PR #1288 head**) | `b36bdbc9` | 36 commits ahead of `origin/main`; OPEN, MERGEABLE; +integration-review.md (Part 1 drift + Part 2 tradeoffs + recommended Option D) |
|
||||
| FastVideo | `will/api_7.10` | `6ae7a99f` | **deprecated** — PR #1287 closed in favor of #1288. Branch can be deleted on origin and locally; kept for now as historical reference. |
|
||||
| FastVideo | `will/api_8`, `will/ltx2_sr_runtime`, `will/ltx2_nvfp4`, `will/ltx2_post_fixes`, `will/agents_cleanup` | (various) | **deprecated** split bookmarks. Strategy reversed to single mega-PR (D-17). Safe to delete locally; not pushed to origin. |
|
||||
| FastVideo | `will/ltx2_sr_port-pre-1286-rebase` | `1baa60bb` | **local-only safety backup** of pre-rebase chain (37 commits); keep until next slice merges |
|
||||
|
||||
Reference in New Issue
Block a user