nodes/hao-ai-lab-FastVideo
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

[docs] dreamverse-integration: align test strategy with B+ + GPU4 hook

Browse Source 
Audit found the test strategy in integration-plan.md was partially stale
from the prior Option D / split-PR drafting and missing an explicit
local-GPU verification path. This commit reconciles testing with the
single-mega-PR (D-17) + Option B+ monorepo (D-18) reality and makes the
GPU4-on-this-node hook explicit.

* New "Test strategy" subsection (added before "File-by-file migration
  map"): test taxonomy table (unit / integration-with-fakes / live-GPU
  / FE build / FE Playwright / contract / SSIM), pytest marker
  scheme (`@pytest.mark.gpu`), pyproject.toml addopts to default-skip
  GPU tests in ubuntu-latest CI, and a per-phase responsibility matrix
  (which class runs in which gate per Phase 0..7).

* GPU4 local verification hook documented: this dev node has 8x B200;
  GPU4 currently held by Dreamverse-side server PID 2453227 / port
  8009; concrete kill / redeploy / smoke-test / cleanup commands so
  any phase touching the live-service path (Phase 2/3/4/5) can be
  validated locally without waiting for Buildkite-Modal.

* Phase 2 verification gate: `uv run` command upgraded to
  `--locked --extra test pytest ... -m 'not gpu'` (matches Phase 1
  `--locked` enforcement). Added explicit MANUAL GPU4 QA gate
  required, not optional: kill PID 2453227, deploy `fastvideo serve`
  on GPU4 from this branch, hit `/health`, run `pytest -m gpu`
  against the live deploy, capture output in PR. Rollback expanded
  to handle GPU4-discovered bugs.

* Phase 3 verification gate: Playwright in CI is now explicitly
  DEFERRED to Phase 4 (matches `ci-dreamverse-frontend.yml` scaffold
  which already comments out Playwright steps until health routes
  land). Removed the contradictory "or PR note explains" clause.
  Added recommended manual GPU4 Playwright `frontend-shell.spec.ts`
  against the Phase 2 GPU4 deploy. Same `--locked --extra test`
  upgrade for the backend regression check.

* Phase 4 verification gate: Frontend CI Playwright RE-ENABLED at
  end of this phase (the natural re-baseline once health routes
  land). Manual GPU4 full E2E for all 3 Playwright specs
  (backend-health + frontend-shell + preset-prompt-generation) is
  required to merge.

* OSS precedent table: clarified that chainlit's "split CI" is a
  per-language CI split (frontend vs backend workflows), not a
  PR-level split — independent of D-17's single-mega-PR decision.
  Removes any reading-confusion that the testing strategy carries
  forward from the abandoned split.

No claims, references, or commands tied to the abandoned split-PR
branches (`will/api_7.10`, `will/api_8`, `will/ltx2_sr_runtime`,
`will/ltx2_nvfp4`, `will/ltx2_post_fixes`, `will/agents_cleanup`)
remain in the plan doc. All test commands route through
`apps/dreamverse/server/` and `apps/dreamverse/web/` per the B+
layout.

PR #1288: https://github.com/hao-ai-lab/FastVideo/pull/1288

Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
This commit is contained in:
SolitaryThinker
2026-05-05 10:43:21 -07:00
co-authored by Junda Su Matthew Noto XOR-op Zhang Peiyuan
parent 1e66c74aec
commit d23e71c2f3
1 changed files with 139 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.agents/memory/dreamverse-integration/integration-plan.md
+139 -10
View File
@@ -603,13 +603,110 @@ monorepo CI path.
|---|---|---|
| uv workspaces | https://docs.astral.sh/uv/concepts/workspaces/ | Authoritative Python workspace model. |
| Hatch monorepo | https://hatch.pypa.io/latest/how-to/environment/workspace/ | Alternative workspace model; not selected. |
| chainlit | https://github.com/Chainlit/chainlit | uv workspace precedent plus split frontend/backend CI. |
| chainlit | https://github.com/Chainlit/chainlit | uv workspace precedent plus **per-language CI split** (separate `check-frontend.yaml` / `check-backend.yaml` workflows path-filtered by directory). Not a PR-level split — independent of D-17 single-mega-PR decision. |
| open-webui | https://github.com/open-webui/open-webui | Frontend path filtering (`paths-ignore` on backend-only changes) and separate release tracks. **Note:** open-webui has a root `package.json`; we are choosing standalone-pnpm despite the precedent, to avoid forcing Python-only contributors to install Node. |
| streamlit | https://github.com/streamlit/streamlit | Split Python and JS testing in one repo. |
| gradio | https://github.com/gradio-app/gradio | Python package plus JS workspace precedent. |
| full-stack-fastapi-template-nextjs | https://github.com/nemanjam/full-stack-fastapi-template-nextjs | Separate frontend/backend build and deploy workflows. |
| rerun-io/rerun, microsoft/autogen, langgenius/dify | various | uv workspace examples with workspace members and sources. |
### Test strategy
The migration spans CPU-only logic, GPU-required inference, and end-to-end
WebSocket flows. Each phase's verification gate must specify which test
class runs where, because not all tests can run on `ubuntu-latest` CI.
#### Test taxonomy
| Class | Marker | Runs in | What it covers | Examples |
|---|---|---|---|---|
| **Unit** | (none / `unit`) | `ci-dreamverse-backend.yml` (ubuntu-latest CI) + locally | Pure logic, no GPU, no live service. Mocked FastVideo backends, schema validation, helper functions. | `test_config.py`, `test_rewrite_prompt_payload.py`, `test_session_init_image.py`, the new `test_import_contract.py` |
| **Integration (fakes)** | `integration` | `ci-dreamverse-backend.yml` + locally | FastAPI test client + in-process fakes/mocks for GPU pool. Validates routes, request/response shapes, session state machine. | `test_health_endpoints.py`, `test_mock_server.py`, `test_entrypoints.py`, `test_prompt_safety.py`, `test_batching.py` (deleted) |
| **Live-service GPU** | `gpu` (skip-by-default in CI) | **Local GPU4 manual QA** + Buildkite-Modal (when added) | Real `fastvideo serve` process + real model weights + real WebSocket round-trips. Validates LTX-2 streaming, NVFP4 wiring, continuation state, frame emission. | `test_realtime_stress.py` (947 LOC), `test_session_logging.py` (1278 LOC) — these spin up real workers per their current shape |
| **Frontend unit / build** | (n/a — pnpm) | `ci-dreamverse-frontend.yml` (ubuntu-latest, no GPU) | Vitest + tsc + Next.js build. No backend needed. | `apps/dreamverse/web/src/**/*.test.ts(x)` |
| **Frontend Playwright E2E** | (n/a — pnpm) | **Local GPU4 manual QA** until Phase 4 lands public health routes; then `ci-dreamverse-frontend.yml` against a mock backend OR a deployed staging | Real browser → real backend WebSocket flow. Requires `/healthz`, `/readyz`, `/status`, `/prompt-system-config`, `/curated-presets`, `/v1/stream`. | `apps/dreamverse/web/e2e/{backend-health,frontend-shell,preset-prompt-generation}.spec.ts` |
| **FastVideo public contract** | (none) | Existing FastVideo CI (`ci-precommit` + Buildkite for GPU) | Schema/shape guards that this migration must not break. | `fastvideo/tests/contract/test_dreamverse_shape.py`, `test_dynamo_shape.py`, `test_generate_async.py` |
| **FastVideo SSIM regression** | (Buildkite path-filter) | Buildkite-Modal | Inference-quality gates for ported models. | `fastvideo/tests/ssim/test_*.py` |
#### Adding the `gpu` marker
In Phase 1, add to root `pyproject.toml`:
```diff
[tool.pytest.ini_options]
+markers = [
+ "gpu: requires a real GPU + model weights; skip in ubuntu-latest CI",
+]
```
In `apps/dreamverse/server/pyproject.toml` `[tool.pytest.ini_options]`, set
the default for the backend test command to skip GPU tests:
```toml
[tool.pytest.ini_options]
addopts = "-m 'not gpu'"
markers = ["gpu: requires real GPU"]
```
GPU-dependent tests must add `@pytest.mark.gpu` at module or function level
during the Phase 2 move. Specifically: `test_realtime_stress.py` and
`test_session_logging.py` per their current LOC and live-service shape.
#### Local GPU4 verification hook
This dev node has 8× B200 GPUs. **GPU4** is currently held by the running
`dreamverse-server` (PID 2453227, port 8009 — see
[`state.md`](file:///home/william5lin/FastVideo/.agents/memory/dreamverse-integration/state.md#L80-L86)).
For migration verification, GPU4 can be reclaimed:
```bash
# Stop the running Dreamverse-side server holding GPU4
sudo kill 2453227 # or use the supervisor that owns it
# Confirm GPU4 is free
nvidia-smi --query-gpu=index,memory.used --format=csv | grep -E '^4,'
# Deploy the new public FastVideo streaming server pinned to GPU4
CUDA_VISIBLE_DEVICES=4 uv run --locked --package dreamverse-server \
fastvideo serve --config apps/dreamverse/serve_configs/streaming_demo.yaml \
--host 0.0.0.0 --port 8009
# Smoke-test from another terminal
curl -s http://localhost:8009/health | jq .
curl -s http://localhost:8009/readyz | jq . # Phase 4+ only
# Drive the FE against it: cd apps/dreamverse/web && pnpm run dev
```
This is the **manual QA gate** for any phase that touches the live-service
path (Phase 2 backend move, Phase 3 frontend move, Phase 4 health-route
promotion, Phase 5 prompt enhancer retirement). The verification gate for
each phase calls out whether the GPU4 smoke is required or optional.
**Cleanup after each test session:**
```bash
# Kill the test deployment
pkill -f 'fastvideo serve --config'
# Restart the original Dreamverse-side server if it should remain canonical
# during early phases (until Phase 6e CI freeze)
cd /home/william5lin/Dreamverse && bash scripts/smoke_local.sh # or
cd /home/william5lin/Dreamverse && uv run server # depending on op convention
```
#### Per-phase test responsibilities (summary)
| Phase | Unit | Integration (fakes) | GPU live (GPU4) | Frontend build | Frontend E2E (Playwright) |
|---|:---:|:---:|:---:|:---:|:---:|
| 0 (#1288 land) | Existing FastVideo suite | Existing | Optional sanity | n/a | n/a |
| 1 (skeleton) | New empty pkg `pytest --collect-only` | n/a | n/a | n/a | n/a |
| 2 (backend move) | **Required** in CI | **Required** in CI | **Required manual** on GPU4 (smoke) | n/a | n/a |
| 3 (FE move) | Backend still green | Backend still green | Optional | **Required** in CI | **Manual on GPU4 only** (FE CI Playwright deferred to Phase 4) |
| 4 (promote pending) | Streaming tests must add coverage | Required | **Required manual** on GPU4 | Required | **Required manual** on GPU4 → re-enable FE-CI Playwright at end |
| 5 (DR-1 / DR-2) | Required | Prompt-shim tests required | **Required manual** prompt-flow on GPU4 | Required | **Required manual** preset-prompt-generation spec on GPU4 |
| 6a-6f | All CI green | All CI green | Optional during 6b/6c deploy dry runs | All CI green | Required against staging in 6c |
| 7 (archive) | n/a | n/a | n/a | n/a | n/a |
---
## File-by-file migration map
@@ -870,8 +967,10 @@ These are **explicit shims** — Phase 4 is responsible for their promotion to p
**Verification gate:**
- `uv run --package dreamverse-server pytest apps/dreamverse/server/tests/ -q` succeeds.
- Import contract test passes.
- **CI / unit / integration**:
`uv run --locked --package dreamverse-server --extra test pytest apps/dreamverse/server/tests/ -m 'not gpu' -q`
succeeds (skips GPU-marked tests; runs in `ci-dreamverse-backend.yml`).
- Import contract test (`test_import_contract.py`) passes.
- `lsp_diagnostics` is clean for changed Python files.
- `pre-commit run --files apps/dreamverse/server/*.py apps/dreamverse/server/session/controller.py apps/dreamverse/server/session/messages.py apps/dreamverse/server/routes/presets.py apps/dreamverse/server/tests/*.py` succeeds.
- D-8 trace is documented in the PR body and either passes or opens a blocking
@@ -879,6 +978,16 @@ These are **explicit shims** — Phase 4 is responsible for their promotion to p
- No imports from internal `fastvideo.pipelines`, `fastvideo.models`,
`fastvideo.layers`, or `fastvideo.worker` remain under
`apps/dreamverse/server/`.
- **Manual QA on GPU4** (required, not optional):
- Reclaim GPU4 (kill running dreamverse-server PID 2453227).
- `CUDA_VISIBLE_DEVICES=4 uv run --locked --package dreamverse-server fastvideo serve --config apps/dreamverse/serve_configs/streaming_demo.yaml --host 0.0.0.0 --port 8009` boots cleanly.
- `curl -s http://localhost:8009/health` returns 200.
- `pytest apps/dreamverse/server/tests/ -m gpu -q` against the live deploy passes
(this exercises `test_realtime_stress.py` and `test_session_logging.py`
with a real GPU + real model weights).
- Capture output in PR description.
- Cleanup: `pkill -f 'fastvideo serve --config'` and restart the canonical
Dreamverse-side server.
**Rollback:**
@@ -887,6 +996,8 @@ These are **explicit shims** — Phase 4 is responsible for their promotion to p
rollback is needed.
- If only one public import is missing, add a temporary product-local shim only
if it is explicitly deleted in Phase 4 or Phase 5.
- If the GPU4 manual QA reveals a real-service bug, the Phase 2 PR is
blocked; fix the bug and re-run the GPU4 smoke before merging.
### Phase 3 — Move FE + content
@@ -929,10 +1040,19 @@ scripts, and product docs after backend tests are green.
- `cd apps/dreamverse/web && pnpm install --frozen-lockfile` succeeds.
- `cd apps/dreamverse/web && pnpm run build` succeeds.
- `cd apps/dreamverse/web && pnpm run test --if-present` succeeds.
- `cd apps/dreamverse/web && pnpm exec playwright test` succeeds, or a PR note
explains which backend endpoint is required for product E2E.
- `uv run --package dreamverse-server pytest apps/dreamverse/server/tests/ -q`
- `cd apps/dreamverse/web && pnpm run test --if-present` succeeds (Vitest + tsc).
- **Frontend CI Playwright is intentionally DEFERRED to Phase 4** — at Phase 3
the public `build_app` does not yet expose `/healthz`+`/readyz`+`/status`+
`/prompt-system-config`+`/curated-presets`. The `ci-dreamverse-frontend.yml`
scaffold from Phase 1 keeps Playwright steps commented out until Phase 4
reactivates them. No PR note required.
- **Manual GPU4 Playwright smoke** (recommended): on this dev node, run
`apps/dreamverse/web/e2e/frontend-shell.spec.ts` against the GPU4-deployed
backend from Phase 2 manual QA + a `pnpm run dev` frontend at port 5274
to confirm shell hydration. `backend-health.spec.ts` and
`preset-prompt-generation.spec.ts` will fail until Phase 4 — that is
expected; document as deferred.
- `uv run --locked --package dreamverse-server --extra test pytest apps/dreamverse/server/tests/ -m 'not gpu' -q`
still succeeds.
- `examples/serving/streaming_demo.yaml` parses with FastVideo serve config.
@@ -966,14 +1086,23 @@ Dreamverse product code.
**Verification gate:**
- FastVideo streaming tests pass.
- Dreamverse backend tests pass.
- FastVideo streaming tests pass (existing CI suite).
- Dreamverse backend tests pass (`-m 'not gpu'` in CI; `-m gpu` on GPU4 manual).
- Contract tests for `/healthz`, `/readyz`, `/status` cover both healthy and
not-ready states.
not-ready states (added in this phase to `fastvideo/tests/entrypoints/streaming/`).
- `run_async()` cancellation test proves client disconnect can stop mid-work.
- VPO decision is documented in code comments/tests.
- SessionStore/BlobStore lifecycle policy has tests for replacement cleanup and
disconnect expiry if implemented.
- **Frontend CI Playwright RE-ENABLED** at end of this phase: uncomment the
Playwright steps in `ci-dreamverse-frontend.yml` (deferred since Phase 1)
AND change CI to either (a) run against the mock backend
`fastvideo.entrypoints.streaming.mock_server`, or (b) require a deployed
staging URL via env var.
- **Manual GPU4 full E2E**: deploy `fastvideo serve` on GPU4 with the new
health routes, run all 3 Playwright specs (`backend-health`, `frontend-shell`,
`preset-prompt-generation`) against it. All 3 must pass — this is the
re-baseline after Phase 3's deferral. Capture output in PR.
**Rollback:**