Compare commits
40
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
c62a8514b0 | ||
|
|
3eb8081801 | ||
|
|
3505d09564 | ||
|
|
570607945c | ||
|
|
d3a821cdcf | ||
|
|
3f24578139 | ||
|
|
89fcf08378 | ||
|
|
c2930b2aa1 | ||
|
|
019239690b | ||
|
|
d6119c1f82 | ||
|
|
84214c80bb | ||
|
|
c2d7143c72 | ||
|
|
afdb6fbfa5 | ||
|
|
ba4c02d883 | ||
|
|
2c137931f3 | ||
|
|
36682797a0 | ||
|
|
0ef1357a77 | ||
|
|
a75d19786a | ||
|
|
be548a78ea | ||
|
|
6a610e2bc9 | ||
|
|
321d5112b4 | ||
|
|
ba75ad82db | ||
|
|
58caa5109f | ||
|
|
2f3ca8aaad | ||
|
|
3a67319cb6 | ||
|
|
266fa044b3 | ||
|
|
f3398db868 | ||
|
|
fda02036bc | ||
|
|
68179cd752 | ||
|
|
2dd5760291 | ||
|
|
af2ee9c78a | ||
|
|
1c80371b27 | ||
|
|
eef473225d | ||
|
|
44fb84ef6a | ||
|
|
e8597b7448 | ||
|
|
e2252c0a5e | ||
|
|
72cb427cd9 | ||
|
|
63030cf6ec | ||
|
|
773d44b875 | ||
|
|
1df513922f |
@@ -0,0 +1,41 @@
|
||||
---
|
||||
date: 2026-05-22
|
||||
experiment: PR #1386 DreamVerse app CI backend tests
|
||||
category: infrastructure
|
||||
severity: important
|
||||
---
|
||||
|
||||
# DreamVerse App CI Streaming Imports Need GPU
|
||||
|
||||
## What Happened
|
||||
|
||||
DreamVerse app CI backend pytest collection imports FastVideo streaming surfaces.
|
||||
When those tests run in a CPU-only Modal environment, collection can fail before
|
||||
any app assertions run with Triton reporting:
|
||||
|
||||
```text
|
||||
RuntimeError: 0 active drivers
|
||||
```
|
||||
|
||||
## Root Cause
|
||||
|
||||
Some streaming import paths can import `fastvideo_kernel` at module import time.
|
||||
Triton then probes for an active GPU driver during pytest collection. A CPU-only
|
||||
Modal container has no active driver, so the failure appears as an import-time
|
||||
collection error rather than a DreamVerse app behavior failure.
|
||||
|
||||
## Fix / Workaround
|
||||
|
||||
For PR #1386, use a surgical CI fix: allocate a GPU to
|
||||
`run_dreamverse_app_tests`. Do not refactor core streaming/kernel imports just to
|
||||
unstick this app CI path.
|
||||
|
||||
Keep `build_kernel=False` for this job. The DreamVerse app backend test imports
|
||||
streaming surfaces but does not need to rebuild or exercise custom kernels.
|
||||
|
||||
## Prevention
|
||||
|
||||
When adding or modifying DreamVerse app CI jobs that import FastVideo streaming
|
||||
modules, make the GPU requirement explicit if the import graph may touch
|
||||
`fastvideo_kernel`. Prefer small CI resource fixes for app test collection issues
|
||||
unless the product code genuinely requires lazy import cleanup.
|
||||
@@ -31,7 +31,7 @@ FastVideo-WorldModel/
|
||||
│ │ │ └── distribution_matching/ # DMD2Method, SelfForcingMethod
|
||||
│ │ ├── models/ # Per-role model wrappers (ModelBase, CausalModelBase)
|
||||
│ │ │ ├── wan/ # WanModel, WanCausalModel
|
||||
│ │ │ └── matrixgame/ # MatrixGameModel, MatrixGameCausalModel
|
||||
│ │ │ └── matrixgame2/ # MatrixGame2Model, MatrixGame2CausalModel
|
||||
│ │ ├── callbacks/ # Composable hooks (grad_clip, ema, validation)
|
||||
│ │ └── utils/ # Config, builder, checkpoint, optimizer, tracking
|
||||
│ ├── training/ # Legacy training infrastructure (being phased out)
|
||||
@@ -44,7 +44,7 @@ FastVideo-WorldModel/
|
||||
│ │ ├── wan_distillation_pipeline.py # Wan distillation
|
||||
│ │ ├── self_forcing_distillation_pipeline.py # Self-forcing distill
|
||||
│ │ ├── ltx2_training_pipeline.py # LTX-2 training
|
||||
│ │ └── matrixgame_training_pipeline.py # MatrixGame training
|
||||
│ │ └── matrixgame2_training_pipeline.py # Matrix-Game 2.0 training
|
||||
│ ├── attention/ # Attention backends
|
||||
│ ├── distributed/ # Sequence/tensor parallel utilities
|
||||
│ ├── layers/ # Tensor-parallel layers
|
||||
@@ -95,7 +95,7 @@ FastVideo-WorldModel/
|
||||
| Wan distillation (DMD) | `fastvideo/training/wan_distillation_pipeline.py` | `torchrun --nproc_per_node N` |
|
||||
| Self-forcing distill | `fastvideo/training/wan_self_forcing_distillation_pipeline.py` | `torchrun --nproc_per_node N` |
|
||||
| LTX-2 finetune | `fastvideo/training/ltx2_training_pipeline.py` | `torchrun --nproc_per_node N` |
|
||||
| MatrixGame | `fastvideo/training/matrixgame_training_pipeline.py` | `torchrun --nproc_per_node N` |
|
||||
| Matrix-Game 2.0 | `fastvideo/training/matrixgame2_training_pipeline.py` | `torchrun --nproc_per_node N` |
|
||||
|
||||
## W&B Integration
|
||||
|
||||
|
||||
@@ -8,6 +8,8 @@ follow-up actions see [open-threads.md](open-threads.md).
|
||||
|
||||
**Last updated:** 2026-05-06 (added D-21 — chunk-stutter root cause is software libx264 encoding consuming ~22% of segment wall-time, NOT a migration regression; verified by 3 parallel explore agents that NVFP4 + torch.compile coverage matches FastVideo-internal exactly; landed opt-in NVENC build path in install_native_ffmpeg.sh + `--nvenc`/`--no-nvenc` flag in dreamverse-deploy.sh + `apps/dreamverse/server/benchmarks/benchmark_av_streaming.py` regression test + memory dir update; default codec stays `libx264` for backward compat, opt-in via `--nvenc`. Earlier: added D-12 — GpuPool layer separation, Oracle review post-#1257-merge; added D-13 — prompt enhancer / LLMProvider abstraction shape, Oracle review pre-#1258-merge; added D-14 — streaming auxiliaries cohesion, Oracle review during #1284 review cycle; added D-15 — streaming router placement + sticky/active-active deferral, Oracle review during #1286 review cycle; added D-16 — streaming router polish round 2, second-pass review on top of D-15 covering bridge cancellation hygiene, registry state machine, httpx hard-fail, replica YAML parsing, and `websockets` dep; added D-17 — strategy reversal: abandon 6-PR split in favor of single mega-PR #1288 on `will/ltx2_sr_port`; added D-18 — Option B+ chosen: Dreamverse FE+product-server move into FastVideo as `apps/dreamverse/` subfolder while generic backend stays at `fastvideo.entrypoints.streaming.*`; integration-review.md deprecated, integration-plan.md is the executable migration plan; added D-19 — D-18 executed: 5 commits land on `will/dreamverse-monorepo`, fix-up commits corrected the integration-plan's invalid "delete generic-merged, import public substitutes" assumption — generic-merged files carried product-local instead, e2e passes against migrated code with /proc-verified evidence; added D-20 — segment-2 BrokenPipe root cause was a TWO-direction silent drop of LTX-2 audio kwargs in public `VideoGenerator`).
|
||||
|
||||
**Update 2026-05:** Dreamverse frontend tooling migrated from standalone pnpm to standalone npm. `apps/dreamverse/web/package-lock.json` is authoritative; see PR #1385.
|
||||
|
||||
## Status legend
|
||||
|
||||
- ✅ **Resolved** — decision made and implementation complete (or no implementation needed)
|
||||
@@ -297,14 +299,14 @@ Playwright (8/8 PASS in 5.1s):
|
||||
- **Python ML library** stays at root: `fastvideo/`, `fastvideo-kernel/`.
|
||||
- **Generic backend** stays at `fastvideo.entrypoints.streaming.*` (already there per #1257/#1258/#1284/#1286/#1288).
|
||||
- **Dreamverse product** moves into `apps/dreamverse/{server,web,prompts,serve_configs,scripts}/`.
|
||||
- **Tooling**: uv workspace for Python (`[tool.uv.workspace] members = ["apps/dreamverse/server"]`), standalone pnpm for the FE (no root `package.json`), split CI workflows with path-filter triggers.
|
||||
- **Tooling**: uv workspace for Python (`[tool.uv.workspace] members = ["apps/dreamverse/server"]`), standalone npm for the FE (no root `package.json`), split CI workflows with path-filter triggers.
|
||||
|
||||
**Rationale:**
|
||||
|
||||
- Drops the cross-repo coordination overhead identified in the post-#1286 rebase cycle (D-17 handled by consolidating into mega-PR; D-18 prevents the next round of cross-repo coordination from happening).
|
||||
- Keeps the architectural separation Option D recommended (FastVideo owns reusable runtime; product owns product). The boundary is now `apps/dreamverse/` directory rather than two repos.
|
||||
- Single repo means atomic cross-cutting refactors (e.g. GpuPool API change + Dreamverse adoption) ship as one PR.
|
||||
- OSS precedents support the shape (chainlit uv-workspace + pnpm; open-webui Python + Svelte with paths-ignore CI). The librarian explicitly noted no precedent for "Python ML library + Next.js product merged into library namespace" — but this isn't that pattern. Dreamverse goes into a sibling directory, NOT into `fastvideo.entrypoints.dreamverse.*`. Library namespace stays clean.
|
||||
- OSS precedents support the shape (chainlit uv-workspace + frontend package manager; open-webui Python + Svelte with paths-ignore CI). The librarian explicitly noted no precedent for "Python ML library + Next.js product merged into library namespace" — but this isn't that pattern. Dreamverse goes into a sibling directory, NOT into `fastvideo.entrypoints.dreamverse.*`. Library namespace stays clean.
|
||||
|
||||
**Why not Option D (separate repos):**
|
||||
|
||||
|
||||
@@ -311,7 +311,7 @@ Notes:
|
||||
development and CI.
|
||||
- Product server release is Docker/deploy workflow, not PyPI.
|
||||
|
||||
### Frontend build: standalone pnpm
|
||||
### Frontend build: standalone npm
|
||||
|
||||
Do not add a root `package.json`.
|
||||
|
||||
@@ -319,14 +319,14 @@ Keep all frontend tooling under:
|
||||
|
||||
```text
|
||||
apps/dreamverse/web/package.json
|
||||
apps/dreamverse/web/pnpm-lock.yaml
|
||||
apps/dreamverse/web/package-lock.json
|
||||
apps/dreamverse/web/playwright.config.*
|
||||
```
|
||||
|
||||
Rationale:
|
||||
|
||||
- FastVideo remains primarily a Python ML library.
|
||||
- Python contributors should not need Node or pnpm for normal work.
|
||||
- Python contributors should not need Node or npm for normal work.
|
||||
- This intentionally diverges from chainlit's root JS workspace pattern and
|
||||
follows the simpler open-webui-style split.
|
||||
|
||||
@@ -455,31 +455,24 @@ jobs:
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
# IMPORTANT: pnpm/action-setup MUST run BEFORE setup-node when using
|
||||
# cache: pnpm — setup-node otherwise can't find pnpm to populate cache.
|
||||
- name: Setup pnpm
|
||||
uses: pnpm/action-setup@v4
|
||||
with:
|
||||
version: 9
|
||||
|
||||
- name: Setup Node
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '22'
|
||||
cache: pnpm
|
||||
cache-dependency-path: apps/dreamverse/web/pnpm-lock.yaml
|
||||
cache: npm
|
||||
cache-dependency-path: apps/dreamverse/web/package-lock.json
|
||||
|
||||
- name: Install dependencies
|
||||
run: pnpm install --frozen-lockfile
|
||||
run: npm ci
|
||||
|
||||
- name: Typecheck
|
||||
run: pnpm run typecheck --if-present
|
||||
run: npm run typecheck --if-present
|
||||
|
||||
- name: Unit tests
|
||||
run: pnpm run test --if-present
|
||||
run: npm run test --if-present
|
||||
|
||||
- name: Build
|
||||
run: pnpm run build
|
||||
run: npm run build
|
||||
|
||||
# NOTE: Playwright tests require a running backend. Until Phase 4 lands
|
||||
# `/healthz`/`/readyz`/`/status`/`/prompt-system-config`/`/curated-presets`
|
||||
@@ -489,13 +482,13 @@ jobs:
|
||||
# `fastvideo.entrypoints.streaming.build_app` and add product routes).
|
||||
#
|
||||
# - name: Install Playwright browsers
|
||||
# run: pnpm exec playwright install --with-deps chromium
|
||||
# run: npm exec -- playwright install --with-deps chromium
|
||||
#
|
||||
# - name: Playwright (re-enable in Phase 4)
|
||||
# run: pnpm exec playwright test
|
||||
# run: npm exec -- playwright test
|
||||
```
|
||||
|
||||
**Playwright config update needed** when moving FE: `Dreamverse/apps/web/playwright.config.ts` line 39 currently uses `npm run dev`; change to `pnpm run dev` post-move ([source](file:///home/william5lin/Dreamverse/apps/web/playwright.config.ts#L39)).
|
||||
**Playwright config update needed** when moving FE: keep `Dreamverse/apps/web/playwright.config.ts` line 39 on `npm run dev` after the move ([source](file:///home/william5lin/Dreamverse/apps/web/playwright.config.ts#L39)).
|
||||
|
||||
#### `.github/workflows/ci-dreamverse-backend.yml`
|
||||
|
||||
@@ -604,7 +597,7 @@ monorepo CI path.
|
||||
| uv workspaces | https://docs.astral.sh/uv/concepts/workspaces/ | Authoritative Python workspace model. |
|
||||
| Hatch monorepo | https://hatch.pypa.io/latest/how-to/environment/workspace/ | Alternative workspace model; not selected. |
|
||||
| chainlit | https://github.com/Chainlit/chainlit | uv workspace precedent plus **per-language CI split** (separate `check-frontend.yaml` / `check-backend.yaml` workflows path-filtered by directory). Not a PR-level split — independent of D-17 single-mega-PR decision. |
|
||||
| open-webui | https://github.com/open-webui/open-webui | Frontend path filtering (`paths-ignore` on backend-only changes) and separate release tracks. **Note:** open-webui has a root `package.json`; we are choosing standalone-pnpm despite the precedent, to avoid forcing Python-only contributors to install Node. |
|
||||
| open-webui | https://github.com/open-webui/open-webui | Frontend path filtering (`paths-ignore` on backend-only changes) and separate release tracks. **Note:** open-webui has a root `package.json`; we are choosing standalone npm under `apps/dreamverse/web/` despite the precedent, to avoid forcing Python-only contributors to install Node. |
|
||||
| streamlit | https://github.com/streamlit/streamlit | Split Python and JS testing in one repo. |
|
||||
| gradio | https://github.com/gradio-app/gradio | Python package plus JS workspace precedent. |
|
||||
| full-stack-fastapi-template-nextjs | https://github.com/nemanjam/full-stack-fastapi-template-nextjs | Separate frontend/backend build and deploy workflows. |
|
||||
@@ -623,8 +616,8 @@ class runs where, because not all tests can run on `ubuntu-latest` CI.
|
||||
| **Unit** | (none / `unit`) | `ci-dreamverse-backend.yml` (ubuntu-latest CI) + locally | Pure logic, no GPU, no live service. Mocked FastVideo backends, schema validation, helper functions. | `test_config.py`, `test_rewrite_prompt_payload.py`, `test_session_init_image.py`, the new `test_import_contract.py` |
|
||||
| **Integration (fakes)** | `integration` | `ci-dreamverse-backend.yml` + locally | FastAPI test client + in-process fakes/mocks for GPU pool. Validates routes, request/response shapes, session state machine. | `test_health_endpoints.py`, `test_mock_server.py`, `test_entrypoints.py`, `test_prompt_safety.py`, `test_batching.py` (deleted) |
|
||||
| **Live-service GPU** | `gpu` (skip-by-default in CI) | **Local GPU4 manual QA** + Buildkite-Modal (when added) | Real `fastvideo serve` process + real model weights + real WebSocket round-trips. Validates LTX-2 streaming, NVFP4 wiring, continuation state, frame emission. | `test_realtime_stress.py` (947 LOC), `test_session_logging.py` (1278 LOC) — these spin up real workers per their current shape |
|
||||
| **Frontend unit / build** | (n/a — pnpm) | `ci-dreamverse-frontend.yml` (ubuntu-latest, no GPU) | Vitest + tsc + Next.js build. No backend needed. | `apps/dreamverse/web/src/**/*.test.ts(x)` |
|
||||
| **Frontend Playwright E2E** | (n/a — pnpm) | **Local GPU4 manual QA** until Phase 4 lands public health routes; then `ci-dreamverse-frontend.yml` against a mock backend OR a deployed staging | Real browser → real backend WebSocket flow. Requires `/healthz`, `/readyz`, `/status`, `/prompt-system-config`, `/curated-presets`, `/v1/stream`. | `apps/dreamverse/web/e2e/{backend-health,frontend-shell,preset-prompt-generation}.spec.ts` |
|
||||
| **Frontend unit / build** | (n/a — npm) | `ci-dreamverse-frontend.yml` (ubuntu-latest, no GPU) | Vitest + tsc + Next.js build. No backend needed. | `apps/dreamverse/web/src/**/*.test.ts(x)` |
|
||||
| **Frontend Playwright E2E** | (n/a — npm) | **Local GPU4 manual QA** until Phase 4 lands public health routes; then `ci-dreamverse-frontend.yml` against a mock backend OR a deployed staging | Real browser → real backend WebSocket flow. Requires `/healthz`, `/readyz`, `/status`, `/prompt-system-config`, `/curated-presets`, `/v1/stream`. | `apps/dreamverse/web/e2e/{backend-health,frontend-shell,preset-prompt-generation}.spec.ts` |
|
||||
| **FastVideo public contract** | (none) | Existing FastVideo CI (`ci-precommit` + Buildkite for GPU) | Schema/shape guards that this migration must not break. | `fastvideo/tests/contract/test_dreamverse_shape.py`, `test_dynamo_shape.py`, `test_generate_async.py` |
|
||||
| **FastVideo SSIM regression** | (Buildkite path-filter) | Buildkite-Modal | Inference-quality gates for ported models. | `fastvideo/tests/ssim/test_*.py` |
|
||||
|
||||
@@ -709,7 +702,7 @@ CUDA_VISIBLE_DEVICES=4 uv run --locked --package dreamverse-server \
|
||||
# Smoke-test from another terminal
|
||||
curl -s http://localhost:8009/health | jq .
|
||||
curl -s http://localhost:8009/readyz | jq . # Phase 4+ only
|
||||
# Drive the FE against it: cd apps/dreamverse/web && pnpm run dev
|
||||
# Drive the FE against it: cd apps/dreamverse/web && npm run dev
|
||||
```
|
||||
|
||||
This is the **manual QA gate** for any phase that touches the live-service
|
||||
@@ -907,7 +900,7 @@ mass move. This should be a small PR.
|
||||
6. Add `apps/dreamverse/web/.*` to pre-commit global exclude.
|
||||
7. Add `docs/contributing/dreamverse-development.md` with local dev commands:
|
||||
backend `uv run --locked --package dreamverse-server --extra test pytest ...`;
|
||||
frontend `cd apps/dreamverse/web && pnpm install && pnpm run build`.
|
||||
frontend `cd apps/dreamverse/web && npm ci && npm run build`.
|
||||
8. Run `uv lock` to regenerate `uv.lock` with the new workspace member; commit
|
||||
the lock change in the same PR.
|
||||
9. Land D-12-A docstring caveat: mark `GpuPool` experimental/server-internal
|
||||
@@ -990,7 +983,7 @@ These are **explicit shims** — Phase 4 is responsible for their promotion to p
|
||||
|
||||
- [`config.py:13`](file:///home/william5lin/Dreamverse/server/config.py#L13): `_APP_ROOT / "apps" / "web"` → `_APP_ROOT / "web"` (since `_APP_ROOT` will resolve to `apps/dreamverse/` in the new layout).
|
||||
- [`apps/web/next.config.ts:11`](file:///home/william5lin/Dreamverse/apps/web/next.config.ts#L11): `outputFileTracingRoot: path.resolve(__dirname, "../..")` → `path.resolve(__dirname, "../../..")` (one extra `..` since the FE is one level deeper in the monorepo).
|
||||
- [`playwright.config.ts:39`](file:///home/william5lin/Dreamverse/apps/web/playwright.config.ts#L39): `command: "npm run dev"` → `command: "pnpm run dev"` (matches Phase 1 tooling decision).
|
||||
- [`playwright.config.ts:39`](file:///home/william5lin/Dreamverse/apps/web/playwright.config.ts#L39): keep `command: "npm run dev"` (matches Phase 1 tooling decision).
|
||||
- Any hardcoded `../FastVideo` paths in scripts/configs → make repo-root-relative since they now share a repo.
|
||||
|
||||
**Steps:**
|
||||
@@ -1084,9 +1077,9 @@ scripts, and product docs after backend tests are green.
|
||||
|
||||
**Verification gate:**
|
||||
|
||||
- `cd apps/dreamverse/web && pnpm install --frozen-lockfile` succeeds.
|
||||
- `cd apps/dreamverse/web && pnpm run build` succeeds.
|
||||
- `cd apps/dreamverse/web && pnpm run test --if-present` succeeds (Vitest + tsc).
|
||||
- `cd apps/dreamverse/web && npm ci` succeeds.
|
||||
- `cd apps/dreamverse/web && npm run build` succeeds.
|
||||
- `cd apps/dreamverse/web && npm run test --if-present` succeeds (Vitest + tsc).
|
||||
- **Frontend CI Playwright is intentionally DEFERRED to Phase 4** — at Phase 3
|
||||
the public `build_app` does not yet expose `/healthz`+`/readyz`+`/status`+
|
||||
`/prompt-system-config`+`/curated-presets`. The `ci-dreamverse-frontend.yml`
|
||||
@@ -1094,7 +1087,7 @@ scripts, and product docs after backend tests are green.
|
||||
reactivates them. No PR note required.
|
||||
- **Manual GPU4 Playwright smoke** (recommended): on this dev node, run
|
||||
`apps/dreamverse/web/e2e/frontend-shell.spec.ts` against the GPU4-deployed
|
||||
backend from Phase 2 manual QA + a `pnpm run dev` frontend at port 5274
|
||||
backend from Phase 2 manual QA + a `npm run dev` frontend at port 5274
|
||||
to confirm shell hydration. `backend-health.spec.ts` and
|
||||
`preset-prompt-generation.spec.ts` will fail until Phase 4 — that is
|
||||
expected; document as deferred.
|
||||
@@ -1249,7 +1242,7 @@ product-specific prompt orchestration.
|
||||
| CI cost increase | Medium | Medium | Add Dreamverse path-specific workflows; add `paths-ignore` to broad workflows; rely on Buildkite monorepo diff watch lists. | CI owner |
|
||||
| FastVideo PyPI release accidentally includes Dreamverse app | High | Low | Add `apps*` to `[tool.setuptools.packages.find]` and `[tool.wheel]` excludes; verify built wheel contents. | Release owner |
|
||||
| Release cadence coupling | Medium | Medium | Keep FastVideo PyPI version release unchanged; Dreamverse uses Docker/Vercel deploy from app paths. | Release owner |
|
||||
| Frontend tooling drift | Medium | Medium | Pin pnpm lockfile in `apps/dreamverse/web/`; no root JS workspace. | Frontend owner |
|
||||
| Frontend tooling drift | Medium | Medium | Pin npm lockfile in `apps/dreamverse/web/`; no root JS workspace. | Frontend owner |
|
||||
| Security surface enlargement | Medium | Medium | Product routes stay in `apps/dreamverse/server`; only generic health/streaming routes go into FastVideo. | Backend owner |
|
||||
| Product-specific API leakage into `fastvideo.*` | High | Medium | Enforce import/module boundary; keep curated presets and prompt UX product-local. | Architecture owner |
|
||||
| Migration regression | High | Medium | Phase gates; backend before frontend; can stop after any phase with Dreamverse repo still usable until Phase 7. | Migration owner |
|
||||
|
||||
@@ -12,7 +12,7 @@ _Last updated: 2026-03-02_
|
||||
|
||||
| Metric | Category | Status | Location | Trust |
|
||||
|--------|----------|--------|----------|-------|
|
||||
| **FVD** | Distribution | ✅ Implemented | `benchmarks/fvd/` | High |
|
||||
| **FVD** | Distribution | ✅ Implemented | `fastvideo/eval/metrics/common/fvd/` | High |
|
||||
| **SSIM** | Reference | ✅ Implemented | `fastvideo/tests/ssim/` | High |
|
||||
| **LPIPS** | Perceptual | ✅ Implemented | `scripts/lora_extraction/` | Medium |
|
||||
| **Loss trajectory** | Training signal | ✅ Implemented | W&B `train_loss` | Medium |
|
||||
@@ -27,8 +27,8 @@ _Last updated: 2026-03-02_
|
||||
### FVD — Fréchet Video Distance
|
||||
|
||||
**Category**: Distribution-level quality metric
|
||||
**Status**: ✅ Fully implemented in `benchmarks/fvd/`
|
||||
**Trust**: High — standard protocol, I3D feature extractor
|
||||
**Status**: ✅ Registered as the `common.fvd` eval metric in `fastvideo/eval/metrics/common/fvd/`
|
||||
**Trust**: High — standard protocol, I3D feature extractor (CLIP / VideoMAE backbones also available, research-grade)
|
||||
|
||||
#### What It Measures
|
||||
FVD measures the distance between the **distribution** of generated videos and
|
||||
@@ -59,30 +59,39 @@ Lower FVD = generated videos are more statistically similar to real videos.
|
||||
#### How to Use
|
||||
|
||||
```python
|
||||
# Programmatic
|
||||
from benchmarks.fvd import compute_fvd_with_config, FVDConfig
|
||||
# Programmatic — drive the metric directly for custom kwargs
|
||||
from fastvideo.eval import get_metric
|
||||
|
||||
config = FVDConfig.fvd2048_16f() # Standard: 2048 videos, 16 frames
|
||||
results = compute_fvd_with_config('data/real/', 'outputs/gen/', config)
|
||||
print(f"FVD: {results['fvd']:.2f}")
|
||||
metric = get_metric("common.fvd", extractor="i3d") # or "clip" / "videomae"
|
||||
metric.to("cuda")
|
||||
metric.setup()
|
||||
metric.reset()
|
||||
|
||||
# First sample carries the reference set; later samples reuse the cache.
|
||||
metric.accumulate({"video": gen_tensors[0], "reference": real_tensors})
|
||||
for gen in gen_tensors[1:]:
|
||||
metric.accumulate({"video": gen})
|
||||
|
||||
result = metric.finalize()
|
||||
print(f"FVD: {result.score:.2f}")
|
||||
```
|
||||
|
||||
```bash
|
||||
# CLI
|
||||
python -m benchmarks.fvd.cli \
|
||||
--real-path data/real/ \
|
||||
--gen-path outputs/gen/ \
|
||||
--protocol fvd2048_16f
|
||||
# CLI — folder of generated mp4s vs a reference folder
|
||||
python examples/inference/eval/eval_fvd.py \
|
||||
--gen-dir outputs/gen/ \
|
||||
--reference-dir data/real/ \
|
||||
--extractor i3d \
|
||||
--output fvd_scores.json
|
||||
```
|
||||
|
||||
**Preset protocols**:
|
||||
| Protocol | Videos | Frames | Use Case |
|
||||
|----------|--------|--------|----------|
|
||||
| `fvd2048_16f` | 2048 | 16 | Standard benchmark (papers) |
|
||||
| `fvd2048_128f` | 2048 | 128 | Long video evaluation |
|
||||
| `quick_test` | 100 | 16 | Fast dev iteration |
|
||||
**Feature extractors**: `i3d` (default, standard FVD spec used in papers),
|
||||
`clip`, `videomae` (research-grade; not directly comparable to published
|
||||
FVD numbers).
|
||||
|
||||
**Feature extractors**: `i3d` (default, standard), `clip`, `videomae`
|
||||
**Protocol**: standard FVD uses 2048 generated + 2048 reference videos at
|
||||
16 frames each. A warning fires below 256 — the score becomes
|
||||
statistically unreliable.
|
||||
|
||||
#### Interpretation
|
||||
| FVD Range | Interpretation |
|
||||
|
||||
@@ -14,7 +14,7 @@ based on **Wan2.1** (SkyReels-V2) DiT models with causal attention for
|
||||
auto-regressive streaming generation.
|
||||
|
||||
**Key techniques you will work with:**
|
||||
- Full finetuning and LoRA on Wan / LTX-2 / MatrixGame models
|
||||
- Full finetuning and LoRA on Wan / LTX-2 / Matrix-Game 2.0 models
|
||||
- DMD-based distillation (few-step generation)
|
||||
- Self-Forcing distillation (causal streaming)
|
||||
- Diffusion-Forcing SFT (DFSFT) for causal models
|
||||
@@ -225,10 +225,10 @@ callbacks:
|
||||
|----------|-----------|----------|
|
||||
| Wan T2V finetune | `fastvideo/training/wan_training_pipeline.py` | Standard text-to-video finetune / LoRA |
|
||||
| Wan I2V finetune | `fastvideo/training/wan_i2v_training_pipeline.py` | Image-to-video (first frame conditioned) |
|
||||
| MatrixGame finetune | `fastvideo/training/matrixgame_training_pipeline.py` | Action-conditioned world model |
|
||||
| MatrixGame AR diffusion | `fastvideo/training/matrixgame_ar_diffusion_pipeline.py` | AR diffusion-forcing training |
|
||||
| MatrixGame ODE-init | `fastvideo/training/matrixgame_ode_causal_pipeline.py` | ODE-trajectory init |
|
||||
| MatrixGame self-forcing distill | `fastvideo/training/matrixgame_self_forcing_distillation_pipeline.py` | Self-forcing distillation |
|
||||
| Matrix-Game 2.0 finetune | `fastvideo/training/matrixgame2_training_pipeline.py` | Action-conditioned world model |
|
||||
| Matrix-Game 2.0 AR diffusion | `fastvideo/training/matrixgame2_ar_diffusion_pipeline.py` | AR diffusion-forcing training |
|
||||
| Matrix-Game 2.0 ODE-init | `fastvideo/training/matrixgame2_ode_causal_pipeline.py` | ODE-trajectory init |
|
||||
| Matrix-Game 2.0 self-forcing distill | `fastvideo/training/matrixgame2_self_forcing_distillation_pipeline.py` | Self-forcing distillation |
|
||||
| LTX-2 finetune | `fastvideo/training/ltx2_training_pipeline.py` | LTX-2 architecture finetuning |
|
||||
| Wan DMD distillation | `fastvideo/training/wan_distillation_pipeline.py` | Few-step distillation via DMD |
|
||||
| Self-Forcing distill | `fastvideo/training/wan_self_forcing_distillation_pipeline.py` | Causal streaming distillation |
|
||||
@@ -258,7 +258,7 @@ Read `.agents/memory/evaluation-registry/README.md` for the full metric catalog.
|
||||
|--------|-------------|-------|
|
||||
| **Loss trajectory** | Every run, real-time from W&B | Medium |
|
||||
| **SSIM** | When comparing against reference outputs | High |
|
||||
| **FVD** | For benchmarking model quality (`benchmarks/fvd/`) | High |
|
||||
| **FVD** | For benchmarking model quality (`common.fvd` eval metric; example: `examples/inference/eval/eval_fvd.py`) | High |
|
||||
| **LPIPS** | LoRA merge validation | Medium |
|
||||
| **Human preference** | Major checkpoints | Highest |
|
||||
|
||||
@@ -278,8 +278,8 @@ Read `.agents/memory/evaluation-registry/README.md` for the full metric catalog.
|
||||
|
||||
## World Model–Specific Concepts
|
||||
|
||||
### Action Injection (MatrixGame)
|
||||
The MatrixGame pipeline adds **action modules** to each DiT block, enabling
|
||||
### Action Injection (Matrix-Game 2.0)
|
||||
The Matrix-Game 2.0 pipeline adds **action modules** to each DiT block, enabling
|
||||
frame-level mouse/keyboard input conditioning. The action sequence is injected
|
||||
per-frame alongside the latent video tokens.
|
||||
|
||||
|
||||
@@ -197,6 +197,12 @@ Tolerance guide:
|
||||
Element-wise `assert_close` alone is not enough for deep full-DiT parity. Also
|
||||
log global abs-mean drift and per-modality summaries.
|
||||
|
||||
When a non-skip component parity run is numerically red after weight/input
|
||||
checks, invoke `../add-model-08-trace/SKILL.md` before adding bespoke forward
|
||||
hooks. Use `docs/contributing/activation_trace.md` to keep
|
||||
`FASTVIDEO_TRACE_LAYERS`, `FASTVIDEO_TRACE_STATS`, and `FASTVIDEO_TRACE_STEPS`
|
||||
identical across FastVideo and upstream traces.
|
||||
|
||||
Useful local commands:
|
||||
|
||||
```bash
|
||||
|
||||
@@ -94,8 +94,12 @@ Run the shared parity-debug loop. The component test command is:
|
||||
pytest <parity_test> -v -s
|
||||
```
|
||||
|
||||
For numerical drift, narrow the first divergent block with per-block hooks or
|
||||
intermediate tensor comparisons before changing layers.
|
||||
For numerical drift, use `../add-model-08-trace/SKILL.md` before writing bespoke
|
||||
hooks. Start with FastVideo's activation trace (`fastvideo/hooks/activation_trace.py`;
|
||||
`docs/contributing/activation_trace.md`) and a block-level regex such as
|
||||
`FASTVIDEO_TRACE_LAYERS="^block\.layers\.[0-9]+$"`. Only fall back to custom
|
||||
per-block hooks if the needed boundary or statistic is not exposed by
|
||||
`FASTVIDEO_TRACE_STATS`.
|
||||
|
||||
## Escape Hatches
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: add-model-08-trace
|
||||
description: Use during /add-model Phase 6 when component parity has failed and root cause requires layer-by-layer divergence analysis. Instruments both the official reference and FastVideo port with forward hooks to find the first numerical divergence point.
|
||||
description: Use during /add-model Phase 6 when component parity has failed and root cause requires layer-by-layer divergence analysis. Uses FastVideo activation trace first, falling back to custom hooks only for boundaries or stats the utility cannot observe.
|
||||
---
|
||||
|
||||
# Add-Model Trace
|
||||
@@ -38,104 +38,90 @@ Required inputs before starting:
|
||||
- Shared deterministic test inputs (same tensors on both sides).
|
||||
- The component parity test file path and its current failure output.
|
||||
|
||||
## Hard Rules: Instrumentation Hierarchy
|
||||
## Primary Path: FastVideo Activation Trace
|
||||
|
||||
Apply these in priority order. Use the highest-priority method that works for
|
||||
the target site.
|
||||
Use FastVideo's first-class activation trace before writing custom hooks:
|
||||
`fastvideo/hooks/activation_trace.py`, documented in
|
||||
`docs/contributing/activation_trace.md`.
|
||||
|
||||
### (1) Forward hooks (PREFERRED)
|
||||
Pipeline runs attach trace to the transformer during pipeline initialization.
|
||||
Component-only parity harnesses may call `attach_activation_trace(model)` from
|
||||
local test/debug code; do not add trace calls to production model code.
|
||||
|
||||
Prefix the failing parity command with a tight layer regex:
|
||||
|
||||
```bash
|
||||
FASTVIDEO_TRACE_ACTIVATIONS=1 \
|
||||
FASTVIDEO_TRACE_LAYERS="^block\.layers\.[0-9]+$" \
|
||||
FASTVIDEO_TRACE_STATS="abs_mean,sum,max,shape" \
|
||||
FASTVIDEO_TRACE_STEPS="0" \
|
||||
FASTVIDEO_TRACE_OUTPUT="/tmp/opencode/fv_trace.jsonl" \
|
||||
pytest tests/local_tests -k "parity" -v -s
|
||||
```
|
||||
|
||||
Match the layer regex to the actual `model.named_modules()` names. Empty or
|
||||
broad regexes are expensive; prefer block-level names first, then narrow to
|
||||
submodules after the first divergent block is known.
|
||||
|
||||
## Trace Compare Contract
|
||||
|
||||
One JSONL file per side. FastVideo output should use `FASTVIDEO_TRACE_OUTPUT`;
|
||||
the upstream harness should emit the same JSONL shape:
|
||||
|
||||
```json
|
||||
{"module":"block.layers.0","tensor":"out","step":0,"abs_mean":0.0123,"sum":1.0,"max":0.5,"shape":[1,16,32]}
|
||||
```
|
||||
|
||||
Compare rows by `(module, step, tensor)`. The first row whose `shape`,
|
||||
`abs_mean`, or `max` diverges beyond the component tolerance is the first broken
|
||||
boundary. Keep `FASTVIDEO_TRACE_LAYERS`, `FASTVIDEO_TRACE_STATS`, and
|
||||
`FASTVIDEO_TRACE_STEPS` identical between sides; if row order differs, sort or
|
||||
normalize before diffing.
|
||||
|
||||
## Drill-Down Loop
|
||||
|
||||
**Initial run:** trace every top-level block (`^block\.layers\.[0-9]+$` or the
|
||||
family's equivalent). Identify the first block index where `abs_mean` or `max`
|
||||
drifts beyond tolerance while earlier blocks match.
|
||||
|
||||
**Drill run:** tighten `FASTVIDEO_TRACE_LAYERS` to submodules inside the first
|
||||
divergent block: attention output, MLP projections, norm outputs, modality
|
||||
adapters, or other named boundaries exposed by `named_modules()`.
|
||||
|
||||
**Iterate:** if the first divergent operation is a free function or tensor op not
|
||||
visible as an `nn.Module`, use the fallback instrumentation hierarchy below.
|
||||
|
||||
The loop ends when the first divergent submodule or operation is identified with
|
||||
a file:line citation in the official source.
|
||||
|
||||
## Fallback Instrumentation Hierarchy
|
||||
|
||||
Use these only when activation trace cannot observe the needed boundary or
|
||||
statistic.
|
||||
|
||||
### (1) Custom forward hooks
|
||||
|
||||
`module.register_forward_hook(...)` and `register_forward_pre_hook(...)`.
|
||||
Always within `try/finally` with `handle.remove()`. Zero source residue.
|
||||
|
||||
```python
|
||||
handle = module.register_forward_hook(fn)
|
||||
try:
|
||||
output = model(inputs)
|
||||
finally:
|
||||
handle.remove()
|
||||
```
|
||||
|
||||
### (2) Runtime monkey-patch (PREFERRED over source edits)
|
||||
### (2) Runtime monkey-patch
|
||||
|
||||
`module.attr = wrapped_func` or `cls.method = wrapped_method`, restored via
|
||||
`try/finally` (save original first). Use for free functions and non-Module
|
||||
sites such as activation functions (`swiglu`, `apply_rotary_emb`) that cannot
|
||||
be hooked as `nn.Module` submodules.
|
||||
|
||||
```python
|
||||
original = cls.method
|
||||
cls.method = wrapped
|
||||
try:
|
||||
output = model(inputs)
|
||||
finally:
|
||||
cls.method = original
|
||||
```
|
||||
`try/finally` (save original first). Use for free functions and non-Module sites
|
||||
such as activation functions (`swiglu`, `apply_rotary_emb`).
|
||||
|
||||
### (3) Source edits in FastVideo's own code
|
||||
|
||||
Only when (1) and (2) are insufficient. Track all edits within a single named
|
||||
`git stash` boundary OR a temporary branch. Run `git diff` before closing the
|
||||
investigation to confirm the stash or branch is clean. The cleanup gate
|
||||
enforces this.
|
||||
investigation to confirm cleanup.
|
||||
|
||||
### (4) Source edits in official repo source
|
||||
|
||||
Allowed if EITHER:
|
||||
|
||||
- (a) The official repo is a git-tracked clone (e.g. `daVinci-MagiHuman/` at
|
||||
the repo root): use `git diff` in the clone path to verify cleanup.
|
||||
- (b) It's installed editable (`pip install -e .`): use `git diff` in the
|
||||
editable source path to verify cleanup.
|
||||
|
||||
If the official repo is installed non-editable in site-packages: back up the
|
||||
target file (`cp original.py original.py.trace-backup`) before editing, then
|
||||
restore from backup at the end (or `pip install --force-reinstall <pkg>`).
|
||||
The cleanup gate verifies via diff-against-backup or zero-diff-in-clone.
|
||||
|
||||
## Logging Contract
|
||||
|
||||
One log file per side. Paths:
|
||||
|
||||
```
|
||||
/tmp/opencode/<family>_<component>_up_layers.log
|
||||
/tmp/opencode/<family>_<component>_fv_layers.log
|
||||
```
|
||||
|
||||
Format: one line per captured tensor, space-separated:
|
||||
|
||||
```
|
||||
<name> <shape> <abs_mean> <sum> <min> <max>
|
||||
```
|
||||
|
||||
Example:
|
||||
|
||||
```
|
||||
block[00] (1,512,1024) 0.012345 6.3210 -0.4321 0.4321
|
||||
```
|
||||
|
||||
Keep the format diff-friendly. Running `diff /tmp/opencode/x_up.log
|
||||
/tmp/opencode/x_fv.log` should highlight the first divergent line directly.
|
||||
Retain side-by-side stdout output alongside the per-side files for human
|
||||
review.
|
||||
|
||||
## Drill-Down Loop
|
||||
|
||||
**Initial run:** attach hooks to every top-level block (`model.block.layers[i]`
|
||||
or equivalent). Identify the first block index `NN` where abs_mean relative
|
||||
drift exceeds 0.5% compared to the previous block.
|
||||
|
||||
**Drill run:** set `<FAMILY>_DEBUG_DRILL_LAYER=NN` and re-run. The script
|
||||
attaches submodule hooks inside block `NN`: attention output, mlp.pre_norm,
|
||||
mlp.up_gate_proj, mlp.down_proj input (via pre-hook) and output, mlp output,
|
||||
attn_post_norm (if present), mlp_post_norm (if present).
|
||||
|
||||
**Iterate:** if the drill run points to a free function (e.g. an activation
|
||||
not wrapped in an `nn.Module`), switch to a monkey-patch (method 2) to
|
||||
intercept its output via the next module's pre-hook.
|
||||
|
||||
The loop ends when the first divergent submodule is identified with a
|
||||
file:line citation in the official source.
|
||||
Allowed only when hook and monkey-patch approaches cannot capture the site.
|
||||
For git-tracked or editable official clones, use `git diff` in the clone path to
|
||||
verify cleanup. For non-editable site-packages, back up the target file before
|
||||
editing and restore it before handoff.
|
||||
|
||||
## Hypothesis Toggles
|
||||
|
||||
@@ -194,11 +180,13 @@ Escalate to the calling bucket skill when:
|
||||
|
||||
Return to the calling subagent with:
|
||||
|
||||
- File paths to per-side logs (`/tmp/opencode/<family>_<component>_{up,fv}_layers.log`).
|
||||
- The identified first divergent layer or submodule name.
|
||||
- FastVideo trace JSONL path and upstream trace JSONL path.
|
||||
- Trace settings used: `FASTVIDEO_TRACE_LAYERS`, `FASTVIDEO_TRACE_STATS`, and
|
||||
`FASTVIDEO_TRACE_STEPS`.
|
||||
- The first divergent `(module, step, tensor)` row and observed drift.
|
||||
- The upstream file:line citation where the divergence originates.
|
||||
- Hypothesis verdict if an A/B toggle was used (e.g. "PATCH_LINEAR=1 closes
|
||||
the gap, confirming dtype-cast difference in PackedExpertLinear").
|
||||
- Fallback hook/patch verdict if activation trace could not observe the boundary.
|
||||
- Hypothesis verdict if an A/B toggle was used, for example `PATCH_LINEAR=1`.
|
||||
- Cleanup-gate status: `[cleanup-gate] PASS` or a list of unresolved items.
|
||||
|
||||
The calling agent uses this to scope the production fix in the FastVideo
|
||||
@@ -206,10 +194,14 @@ component file.
|
||||
|
||||
## References
|
||||
|
||||
- `templates/block_trace_debug.py` in this skill directory: the canonical
|
||||
template this skill generalizes.
|
||||
- `docs/contributing/activation_trace.md` for canonical activation-trace env vars,
|
||||
JSONL output, cost model, and troubleshooting.
|
||||
- `fastvideo/hooks/activation_trace.py` for the implementation and
|
||||
`attach_activation_trace(model)` entry point.
|
||||
- `templates/block_trace_debug.py` in this skill directory: fallback custom-hook
|
||||
template when activation trace cannot observe the needed boundary or stat.
|
||||
- `tests/local_tests/transformers/_debug_magi_human_block_parity.py` in the
|
||||
FastVideo3 repo: the worked magi-human example this skill was extracted from.
|
||||
FastVideo3 repo: historical worked example for custom hook/patch debugging.
|
||||
- `add-model/SKILL.md` Phase 6: the calling context for this skill.
|
||||
- `add-model-03-port-dit/SKILL.md`, `add-model-04-port-vae/SKILL.md`,
|
||||
`add-model-05-port-encoder/SKILL.md`, `add-model-06-port-generic/SKILL.md`:
|
||||
|
||||
@@ -157,6 +157,13 @@ Debug pipeline drift in this order:
|
||||
channel order, sample rate, FPS, and final slicing.
|
||||
7. Add targeted stage-level diagnostics to identify the first divergent stage.
|
||||
|
||||
If stage diagnostics show the first bad stage is transformer/denoising or a
|
||||
mid-DiT block, enable activation trace before adding ad hoc pipeline prints; see
|
||||
`docs/contributing/activation_trace.md` and `../add-model-08-trace/SKILL.md`.
|
||||
Keep `FASTVIDEO_TRACE_LAYERS`, `FASTVIDEO_TRACE_STATS`, and
|
||||
`FASTVIDEO_TRACE_STEPS` identical across reruns so pipeline parity traces diff
|
||||
one-to-one.
|
||||
|
||||
If the first divergence belongs to component implementation, strict loading, or
|
||||
conversion mapping, stop pipeline edits and return `next_step=return_to_phase_6`
|
||||
with the exact failing evidence. Do not patch conversion from this skill.
|
||||
|
||||
@@ -267,6 +267,11 @@ conversion, route it through `../add-model-07-conversion/SKILL.md` with a retry
|
||||
request matching `contracts/conversion_request.md`, then resume the component
|
||||
skill with the updated conversion handoff.
|
||||
|
||||
When a component failure narrows to layer-by-layer numerical drift, load
|
||||
`../add-model-08-trace/SKILL.md` before writing custom hooks. It uses
|
||||
`fastvideo/hooks/activation_trace.py`; canonical env vars and JSONL format are
|
||||
documented in `docs/contributing/activation_trace.md`.
|
||||
|
||||
Phase 6 ends only when every required component handoff reports
|
||||
`parity_status=non_skip_pass`, or when a precise blocker or escape hatch is
|
||||
recorded in `port_state_file`.
|
||||
|
||||
@@ -22,7 +22,7 @@ both `/readyz` and the FE root to return 200.
|
||||
`cerebras-cloud-sdk`, `openai` installed (override the default path with
|
||||
`DREAMVERSE_PYTHON=/path/to/python`)
|
||||
- `~/.env` exporting `CEREBRAS_API_KEY`, `GROQ_API_KEY`, etc.
|
||||
- pnpm installed at `/home/william5lin/.local/share/pnpm/pnpm` (or in `$PATH`)
|
||||
- npm available in `$PATH` (or set `NPM=/path/to/npm`)
|
||||
- `gcc-13` + `g++-13` at `/usr/bin/` (workaround for nvcc gcc-15 rejection)
|
||||
- **Recommended:** native ffmpeg env file at `apps/dreamverse/scripts/ffmpeg-env.sh`
|
||||
(built once via `bash apps/dreamverse/scripts/install_native_ffmpeg.sh`).
|
||||
@@ -110,7 +110,7 @@ Flags can appear in any position relative to the positional args. Explicit flag
|
||||
5. Launches the backend via `apps/dreamverse/scripts/dreamverse-server` in a
|
||||
detached `setsid` session, captures PID.
|
||||
6. Polls `/readyz` until 200 (max 5 min).
|
||||
7. Launches the frontend via `pnpm run dev:devtools` in a detached session,
|
||||
7. Launches the frontend via `npm run dev:devtools` in a detached session,
|
||||
captures PID.
|
||||
8. Polls FE `/` until 200 (max 60s).
|
||||
9. Prints URLs, PIDs, and log paths.
|
||||
@@ -125,7 +125,7 @@ Flags can appear in any position relative to the positional args. Explicit flag
|
||||
PLAYWRIGHT_SKIP_WEBSERVER=1 BACKEND_URL=http://127.0.0.1:8009 \
|
||||
PLAYWRIGHT_BASE_URL=http://127.0.0.1:5274 \
|
||||
NEXT_PUBLIC_INCLUDE_DEVTOOLS=1 \
|
||||
pnpm exec playwright test
|
||||
npm exec -- playwright test
|
||||
```
|
||||
The fast suite (8 specs, ~5s) runs by default; the long-running
|
||||
two-segment audio-continuation spec is gated behind
|
||||
@@ -151,7 +151,7 @@ PLAYWRIGHT_SKIP_WEBSERVER=1 \
|
||||
PLAYWRIGHT_BASE_URL=http://127.0.0.1:5274 \
|
||||
NEXT_PUBLIC_INCLUDE_DEVTOOLS=1 \
|
||||
PLAYWRIGHT_LONG_RUNNING=1 \
|
||||
pnpm exec playwright test e2e/long-running-segments.spec.ts
|
||||
npm exec -- playwright test e2e/long-running-segments.spec.ts
|
||||
```
|
||||
|
||||
Expected runtime: ~7-9 minutes on a B200 (torch.compile max-autotune
|
||||
|
||||
@@ -185,17 +185,10 @@ CONDA_ENV_PYTHON="${DREAMVERSE_PYTHON:-${HOME}/miniconda3/envs/fv-main/bin/pytho
|
||||
"${CONDA_ENV_PYTHON}" -c 'import flashinfer' 2>/dev/null \
|
||||
|| bail "flashinfer-python not installed in ${CONDA_ENV_PYTHON} (run: ${CONDA_ENV_PYTHON} -m pip install flashinfer-python --no-build-isolation)"
|
||||
|
||||
PNPM="${PNPM:-}"
|
||||
if [[ -n "${PNPM}" ]]; then
|
||||
PNPM_REQUESTED="${PNPM}"
|
||||
PNPM="$(command -v "${PNPM}" 2>/dev/null || true)"
|
||||
[[ -n "${PNPM}" ]] || bail "pnpm not executable or not in PATH: ${PNPM_REQUESTED} (set PNPM to override)"
|
||||
elif [[ -x "${HOME}/.local/share/pnpm/pnpm" ]]; then
|
||||
PNPM="${HOME}/.local/share/pnpm/pnpm"
|
||||
else
|
||||
PNPM="$(command -v pnpm 2>/dev/null || true)"
|
||||
fi
|
||||
[[ -n "${PNPM}" ]] && [[ -x "${PNPM}" ]] || bail "pnpm not found. Set PNPM, install at ${HOME}/.local/share/pnpm/pnpm, or add pnpm to PATH"
|
||||
NPM="${NPM:-npm}"
|
||||
NPM_REQUESTED="${NPM}"
|
||||
NPM="$(command -v "${NPM}" 2>/dev/null || true)"
|
||||
[[ -n "${NPM}" ]] && [[ -x "${NPM}" ]] || bail "npm not executable or not in PATH: ${NPM_REQUESTED} (set NPM to override)"
|
||||
|
||||
GCC13="$(command -v "${GCC13:-gcc-13}" 2>/dev/null || true)"
|
||||
GPP13="$(command -v "${GPP13:-g++-13}" 2>/dev/null || true)"
|
||||
@@ -386,7 +379,7 @@ frontend_log="${LOG_DIR}/frontend-port${FRONTEND_PORT}.log"
|
||||
# requested port differs, run `next dev --port` directly with devtools env.
|
||||
fe_cmd="run dev:devtools"
|
||||
if [[ "${FRONTEND_PORT}" != "5274" ]]; then
|
||||
fe_cmd="exec next dev --port ${FRONTEND_PORT}"
|
||||
fe_cmd="exec -- next dev --port ${FRONTEND_PORT}"
|
||||
fi
|
||||
|
||||
setsid bash -c "
|
||||
@@ -395,7 +388,7 @@ setsid bash -c "
|
||||
export BACKEND_URL=http://127.0.0.1:${BACKEND_PORT}
|
||||
export BACKEND_HOST=127.0.0.1
|
||||
export BACKEND_PORT=${BACKEND_PORT}
|
||||
exec '${PNPM}' ${fe_cmd}
|
||||
exec '${NPM}' ${fe_cmd}
|
||||
" > "${frontend_log}" 2>&1 < /dev/null &
|
||||
disown
|
||||
|
||||
@@ -462,5 +455,5 @@ cat <<SUMMARY
|
||||
BACKEND_URL=http://127.0.0.1:${BACKEND_PORT} \\
|
||||
PLAYWRIGHT_BASE_URL=http://127.0.0.1:${FRONTEND_PORT} \\
|
||||
NEXT_PUBLIC_INCLUDE_DEVTOOLS=1 \\
|
||||
pnpm exec playwright test
|
||||
npm exec -- playwright test
|
||||
SUMMARY
|
||||
|
||||
@@ -6,4 +6,5 @@
|
||||
{"name": "seed-ssim-references", "description": "Run a new or updated fastvideo/tests/ssim/ test on Modal, pull generated videos, and upload them to FastVideo/ssim-reference-videos so the test has a regression baseline", "path": "seed-ssim-references/SKILL.md", "status": "draft", "trust": "low"}
|
||||
{"name": "reseed-ssim-references", "description": "Re-seed (overwrite) HF reference videos for an existing fastvideo/tests/ssim/ test and a single model id on Modal L40S. Always backs up current refs first, regenerates on Modal, pauses for the user to eyeball before-vs-after, then uploads with --force scoped to --model-id. Sister skill to seed-ssim-references; use when intentional code change has invalidated existing refs", "path": "reseed-ssim-references/SKILL.md", "status": "draft", "trust": "low"}
|
||||
{"name": "decompose-pipeline-pr", "description": "Decompose an oversized FastVideo pipeline PR into a stack of independently-reviewable PRs. Tiers the diff by blast radius (invisible / dead code / cross-cutting infra / activation), produces a branch graph and worktree bootstrap, drafts the AGENTS.md manifest, flags missing tests on cross-cutting infra changes, and extracts lessons from the PR body. Worked example: PR #1280 daVinci-MagiHuman (9.8k LOC) decomposed into 10 stacked PRs.", "path": "decompose-pipeline-pr/SKILL.md", "status": "tested", "trust": "medium"}
|
||||
{"name": "reseed-performance-baseline", "description": "Re-seed the HF performance-tracking baseline for an intentional runtime, dependency, or environment-caused benchmark shift. Use when performance CI fails because metrics such as latency, throughput, component time, or peak memory changed for an accepted reason and the rolling median baseline must be advanced by replicating one reviewed shifted source result into three success=true records, or five records when explicitly requested", "path": "reseed-performance-baseline/SKILL.md", "status": "draft", "trust": "low"}
|
||||
{"name": "add-model", "description": "Add a new model (or variant) to FastVideo: DiT + configs + pipeline + presets + registry + tests. Walks through FastVideo's single stage-based pipeline architecture with exact file paths and registration hooks.", "path": "add-model/SKILL.md", "status": "draft", "trust": "low"}
|
||||
|
||||
@@ -38,7 +38,7 @@ entrypoint, and applying defaults from the closest example script.
|
||||
| `finetune` (Wan T2V) | `fastvideo/training/wan_training_pipeline.py` |
|
||||
| `finetune` (Wan I2V) | `fastvideo/training/wan_i2v_training_pipeline.py` |
|
||||
| `finetune` (LTX-2) | `fastvideo/training/ltx2_training_pipeline.py` |
|
||||
| `finetune` (MatrixGame) | `fastvideo/training/matrixgame_training_pipeline.py` |
|
||||
| `finetune` (Matrix-Game 2.0) | `fastvideo/training/matrixgame2_training_pipeline.py` |
|
||||
| `distill-dmd` | `fastvideo/training/wan_distillation_pipeline.py` |
|
||||
| `self-forcing` | `fastvideo/training/wan_self_forcing_distillation_pipeline.py` |
|
||||
|
||||
|
||||
@@ -0,0 +1,476 @@
|
||||
---
|
||||
name: reseed-performance-baseline
|
||||
description: Re-seed the HF performance-tracking baseline for an intentional runtime, dependency, or environment-caused benchmark shift using one or more reviewed normalized performance JSONs. Use when performance CI fails because metrics such as latency, throughput, component time, or peak memory changed for an accepted reason and the rolling median baseline in FastVideo/performance-tracking must be advanced from a consistent batch of reviewed source results. The workflow backs up existing history under /tmp, validates all source JSONs for the same (model_id, gpu_type), rejects internally inconsistent source batches, uploads one success=true reseed record per accepted source JSON, and offers to clean local temp state after a successful upload.
|
||||
---
|
||||
|
||||
# Re-seed Performance Baseline
|
||||
|
||||
## Purpose
|
||||
|
||||
Replace or advance the rolling performance baseline for a single
|
||||
`(model_id, gpu_type)` pair in the HF dataset
|
||||
`FastVideo/performance-tracking`.
|
||||
|
||||
Performance comparison uses the median of up to the last 5 successful records
|
||||
for the same model and GPU. Failed records are useful audit history, but they
|
||||
do not move the future baseline because `compare_baseline.py` loads records
|
||||
with `successful_only=True`.
|
||||
|
||||
This skill now reseeds from a reviewed batch of one or more source performance
|
||||
JSONs. It uploads one new `success=true` record per accepted source JSON; it
|
||||
does not blindly replicate one measurement into 3 or 5 records. The effective
|
||||
reseed size is therefore dynamic and equals the number of provided, validated,
|
||||
internally consistent source JSONs.
|
||||
|
||||
If the operator provides fewer than 3 records, call out that the last-5 rolling
|
||||
median may not move immediately. If the operator provides 3 consistent shifted
|
||||
records, the rolling median usually moves immediately. If the operator provides
|
||||
5 consistent shifted records, the last-5 window is effectively reset to the new
|
||||
runtime profile.
|
||||
|
||||
These records are intentional operator-approved baseline resets, not ordinary
|
||||
independent main-branch persistence. Mark them clearly with provenance fields
|
||||
so the HF history remains auditable.
|
||||
|
||||
Use this skill when a performance test fails for an intentional and reviewed
|
||||
reason, such as a torch/runtime/container upgrade that legitimately increases
|
||||
peak memory or changes timings. This is the performance equivalent of
|
||||
`reseed-ssim-references`: backup first, scope tightly, require explicit human
|
||||
approval, then upload reviewed accepted baseline records.
|
||||
|
||||
## When to use
|
||||
|
||||
- A PR or main run failed the rolling performance comparison by more than the
|
||||
allowed regression threshold, and maintainers agree the shift is caused by
|
||||
an intentional runtime, dependency, hardware image, or benchmark environment
|
||||
change rather than a FastVideo logic regression.
|
||||
- One or more shifted source result JSONs have been reviewed and accepted, and
|
||||
the operator wants to use those exact reviewed results to advance the rolling
|
||||
baseline.
|
||||
- The source batch is internally consistent: no provided source JSON regresses
|
||||
against the batch median by more than the configured tolerance.
|
||||
|
||||
## When not to use
|
||||
|
||||
- The benchmark failure might be a real code regression. Fix or investigate
|
||||
the code path first.
|
||||
- The fixed benchmark thresholds in
|
||||
`.buildkite/performance-benchmarks/tests/*.json` are too low. Those are a
|
||||
separate gate from the rolling HF baseline and may need a code review change.
|
||||
- There is no clear source run, commit, and rationale. Baseline history is a
|
||||
production signal; do not edit it without provenance.
|
||||
- The provided source JSONs disagree materially with each other. Rerun or
|
||||
investigate instead of uploading a noisy reseed batch.
|
||||
|
||||
## Inputs
|
||||
|
||||
| Parameter | Required | Description |
|
||||
|-----------|----------|-------------|
|
||||
| `model_id` | Yes | Benchmark id, e.g. `wan-t2v-1.3b-2gpu`. This maps to the HF subdirectory after `sanitize(model_id)`. |
|
||||
| `gpu_type` | Yes | Exact GPU device string from the performance record, e.g. the L40S device name emitted by CI. Baselines are GPU-specific. |
|
||||
| `source_results` | Yes | One or more local paths or Buildkite artifact URLs for accepted shifted performance JSONs. Prefer normalized `normalized_perf_*.json` artifacts emitted by `compare_baseline.py`. Accept `source_result` as an alias only for a single JSON. |
|
||||
| `max_intra_batch_regression` | No | Maximum allowed regression of any source JSON against the source batch median. Default: `PERF_MAX_REGRESSION` if set, otherwise `0.05` (5%). |
|
||||
| `intent_rationale` | Yes | One-line explanation for why the baseline shift is legitimate. This is written into provenance and should be reused in the PR. |
|
||||
|
||||
Hardcoded defaults:
|
||||
|
||||
- HF repo: `FastVideo/performance-tracking` (`HF_REPO_ID` override is
|
||||
supported by the code, but use the default unless the user explicitly asks).
|
||||
- Local sync root: `/tmp/perf-tracking` (`PERFORMANCE_TRACKING_ROOT` override
|
||||
is supported).
|
||||
- Backup root: `/tmp/performance_reseed_backup`.
|
||||
- Download scratch root for source artifact URLs: `/tmp/performance_reseed_source`.
|
||||
- Baseline window: last 5 `success=true` records for the same
|
||||
`(model_id, gpu_type)`.
|
||||
- Reseed count: dynamic. Upload exactly one accepted seed record per validated
|
||||
source JSON.
|
||||
|
||||
## Steps
|
||||
|
||||
### 1. Validate the target and source results
|
||||
|
||||
Normalize `source_results` to a list. If the user passes a single
|
||||
`source_result`, treat it as a one-element `source_results` list and report
|
||||
that a single record may not move the last-5 median immediately.
|
||||
|
||||
If any source result is a Buildkite artifact URL, download it first into a
|
||||
local scratch directory under `/tmp/performance_reseed_source/` and use the
|
||||
downloaded JSON path for the rest of the workflow. If the agent cannot access
|
||||
the artifact because Buildkite authentication is missing, ask the user to
|
||||
download the artifact manually and provide the local path.
|
||||
|
||||
Prefer the normalized Buildkite artifact emitted by `compare_baseline.py`:
|
||||
|
||||
```text
|
||||
perf_reports/results/normalized_perf_*.json
|
||||
```
|
||||
|
||||
That file is already in the HF tracking schema. Load each normalized JSON
|
||||
directly:
|
||||
|
||||
```python
|
||||
import json
|
||||
|
||||
with open(source_result, encoding="utf-8") as f:
|
||||
record = json.load(f)
|
||||
```
|
||||
|
||||
Stop if any normalized record's `model_id` or `gpu_type` does not match the
|
||||
requested `model_id` and `gpu_type`.
|
||||
|
||||
The source records may have `success: false` when they came from failed
|
||||
rolling baseline comparisons. That is expected; only the reviewed reseed
|
||||
records become new `success: true` baseline records after explicit approval.
|
||||
|
||||
Sort validated source records by their original `timestamp` ascending before
|
||||
preparing the seed records. If a source timestamp is missing or unparsable,
|
||||
preserve input order for those records and print a warning. This makes the
|
||||
fresh reseed timestamps deterministic and makes it clear which records enter
|
||||
the last-5 window when more than 5 source JSONs are provided.
|
||||
|
||||
Check that `HF_API_KEY` is exported. The sync path may be public, but the
|
||||
upload path requires write access.
|
||||
|
||||
### 1a. Check source batch consistency
|
||||
|
||||
Before syncing or preparing uploads, reject source batches that are internally
|
||||
inconsistent. Use the same metric direction as `compare_baseline.py`:
|
||||
|
||||
- Lower is better: `latency`, `memory`, `text_encoder_time_s`, `dit_time_s`,
|
||||
`vae_decode_time_s`.
|
||||
- Higher is better: `throughput`.
|
||||
|
||||
For each metric with at least two non-null source values:
|
||||
|
||||
1. Compute the source batch median.
|
||||
2. For lower-is-better metrics, compute `(source_value - batch_median) / batch_median`.
|
||||
3. For `throughput`, compute `(batch_median - source_value) / batch_median`.
|
||||
4. Stop if any source record regresses against the batch median by more than
|
||||
`max_intra_batch_regression`.
|
||||
|
||||
Default `max_intra_batch_regression` to `PERF_MAX_REGRESSION` when set,
|
||||
otherwise `0.05`. Print a table with per-source values, batch median, and
|
||||
worst intra-batch regression.
|
||||
|
||||
This check prevents uploading a mixed batch where one JSON is materially
|
||||
slower or faster than the others. If the batch fails this check, ask the user
|
||||
to provide a cleaner batch or explicitly investigate the variance. Do not
|
||||
silently drop outliers unless the user gives a concrete reviewed reason and a
|
||||
new source list.
|
||||
|
||||
### 1b. How to obtain source results from CI
|
||||
|
||||
The performance CI exports normalized source results for failed rolling
|
||||
baseline comparisons when `compare_baseline.py` ran. The preferred artifacts
|
||||
come from:
|
||||
|
||||
```text
|
||||
perf_reports/results/normalized_perf_*.json
|
||||
```
|
||||
|
||||
The normal operator flow is:
|
||||
|
||||
1. Open the failed Buildkite performance job or several reruns of the same
|
||||
benchmark after the accepted environment shift.
|
||||
2. Download the `normalized_perf_*.json` artifacts for the target benchmark.
|
||||
3. Pass all reviewed local paths or artifact URLs as `source_results`.
|
||||
|
||||
Do not scrape the Markdown performance summary to reconstruct JSON. The
|
||||
normalized JSON artifacts are the only supported source of truth for reseed
|
||||
metrics and provenance. Raw `fastvideo/tests/performance/results/perf_*.json`
|
||||
artifacts are not accepted by this skill. If no normalized JSON artifact is
|
||||
present, that run is not a valid source for baseline reseeding.
|
||||
|
||||
### 2. Sync and back up existing HF records under /tmp
|
||||
|
||||
Use `fastvideo/tests/performance/hf_store.py` helpers directly. Do **not** use
|
||||
`compare_baseline.py` as a sync shortcut; on full main runs it can persist
|
||||
records, while this step must only fetch and back up existing history.
|
||||
|
||||
The sync command pattern is:
|
||||
|
||||
```bash
|
||||
export PERFORMANCE_TRACKING_ROOT="${PERFORMANCE_TRACKING_ROOT:-/tmp/perf-tracking}"
|
||||
export HF_REPO_ID="${HF_REPO_ID:-FastVideo/performance-tracking}"
|
||||
PYTHONPATH=fastvideo/tests/performance python -c 'from hf_store import sync_from_hf; import os; sync_from_hf(os.environ["PERFORMANCE_TRACKING_ROOT"], strict=True)'
|
||||
```
|
||||
|
||||
Then back up only the sanitized model directory under `/tmp`:
|
||||
|
||||
```bash
|
||||
SHORT_COMMIT=$(git rev-parse --short=12 HEAD)
|
||||
TIMESTAMP=$(date -u +%Y%m%d_%H%M%S)
|
||||
MODEL_SAFE=$(PYTHONPATH=fastvideo/tests/performance python - <<'PY'
|
||||
from hf_store import sanitize
|
||||
print(sanitize("<model_id>"))
|
||||
PY
|
||||
)
|
||||
BACKUP_DIR="/tmp/performance_reseed_backup/${TIMESTAMP}_${SHORT_COMMIT}_${MODEL_SAFE}"
|
||||
mkdir -p "$BACKUP_DIR"
|
||||
cp -R "${PERFORMANCE_TRACKING_ROOT}/${MODEL_SAFE}" "$BACKUP_DIR/" 2>/dev/null || true
|
||||
```
|
||||
|
||||
Write provenance next to the backup:
|
||||
|
||||
```bash
|
||||
cat > "$BACKUP_DIR/PROVENANCE.txt" <<EOF
|
||||
model_id: <model_id>
|
||||
gpu_type: <gpu_type>
|
||||
source_results:
|
||||
- <source_result_1>
|
||||
- <source_result_2>
|
||||
reseed_record_count: <len(source_results)>
|
||||
max_intra_batch_regression: <threshold>
|
||||
head_commit: $(git rev-parse HEAD)
|
||||
timestamp_utc: $(date -u +%FT%TZ)
|
||||
reason: <intent_rationale>
|
||||
EOF
|
||||
```
|
||||
|
||||
If the backup has no prior records, this is not a destructive reseed; it is a
|
||||
first baseline seed. Continue, but report that baseline history was empty.
|
||||
|
||||
### 3. Compute old baseline and candidate shift
|
||||
|
||||
Load the last 5 successful records for the target:
|
||||
|
||||
```python
|
||||
from hf_store import load_records_for_model
|
||||
|
||||
records = load_records_for_model(
|
||||
"/tmp/perf-tracking",
|
||||
"<model_id>",
|
||||
"<gpu_type>",
|
||||
last_n=5,
|
||||
successful_only=True,
|
||||
)
|
||||
```
|
||||
|
||||
Print a small table showing old medians, source batch medians, candidate
|
||||
medians after appending the proposed seed records, and source batch spread for:
|
||||
|
||||
- `latency`
|
||||
- `throughput`
|
||||
- `memory`
|
||||
- `text_encoder_time_s`
|
||||
- `dit_time_s`
|
||||
- `vae_decode_time_s`
|
||||
|
||||
Also print how many successful old records exist. Make clear:
|
||||
|
||||
- 1 seed record usually does not move a last-5 median by itself.
|
||||
- 3 consistent seed records usually move the last-5 median immediately.
|
||||
- 5 consistent seed records effectively reset the last-5 window.
|
||||
- The records are intentional approved baseline resets and must be labeled
|
||||
that way.
|
||||
|
||||
### 4. Confirm intent
|
||||
|
||||
Require an explicit confirmation phrase before preparing the upload:
|
||||
|
||||
> About to RE-SEED performance baseline for `<model_id>` on `<gpu_type>`.
|
||||
> This will upload `<N>` new `success=true` records to
|
||||
> `FastVideo/performance-tracking/<sanitize(model_id)>/`, one per accepted
|
||||
> source JSON.
|
||||
>
|
||||
> Reason: `<intent_rationale>`
|
||||
> Source results: `<source_results>`
|
||||
> Reseed record count: `<N>`
|
||||
> Max intra-batch regression: `<threshold>`
|
||||
> Note: these records come from a reviewed source batch and are intended to
|
||||
> move the rolling median to the accepted runtime profile. They are not
|
||||
> ordinary main-branch persistence.
|
||||
> HEAD: `<git rev-parse --short=12 HEAD>`
|
||||
> Backup: `<BACKUP_DIR>`
|
||||
>
|
||||
> Reply `confirm performance reseed` to proceed, anything else to abort.
|
||||
|
||||
Do not continue unless the user types exactly `confirm performance reseed`.
|
||||
|
||||
### 5. Create the accepted seed records
|
||||
|
||||
Create one seed record from each normalized source result. Do not copy the
|
||||
source JSON wholesale.
|
||||
|
||||
Infer the baseline field allowlist from all existing HF records for the target
|
||||
`(model_id, gpu_type)` after syncing, including both `success=true` and
|
||||
`success=false` records. Use the union of non-provenance keys present in those
|
||||
target records, preserving only fields that also exist in the normalized
|
||||
source record or are explicitly set by the reseed workflow. Always include
|
||||
`model_id`, `timestamp`, and `success` because the upload path and baseline
|
||||
loader depend on them. Always set `timestamp` to a fresh reseed timestamp and
|
||||
`success` to `true`. Do not include unrelated source-only fields that are
|
||||
absent from existing HF records.
|
||||
|
||||
Exclude existing provenance or operator metadata from the inferred baseline
|
||||
field allowlist. At minimum, exclude keys prefixed with `baseline_reseed` and
|
||||
any fields known to be local-only audit metadata.
|
||||
|
||||
If there are no previous HF records for the target model/GPU, fall back to this
|
||||
default baseline field list:
|
||||
|
||||
- `model_id`
|
||||
- `timestamp`
|
||||
- `commit_sha`
|
||||
- `gpu_type`
|
||||
- `latency`
|
||||
- `throughput`
|
||||
- `memory`
|
||||
- `text_encoder_time_s`
|
||||
- `dit_time_s`
|
||||
- `vae_decode_time_s`
|
||||
- `success`
|
||||
|
||||
Do not upload extra fields from the source artifact.
|
||||
|
||||
Optional provenance fields are allowed and useful:
|
||||
|
||||
- `baseline_reseed: true`
|
||||
- `baseline_reseed_reason`
|
||||
- `baseline_reseed_source_result`
|
||||
- `baseline_reseed_source_timestamp`
|
||||
- `baseline_reseed_source_success`
|
||||
- `baseline_reseed_batch_size`
|
||||
- `baseline_reseed_batch_index`
|
||||
- `baseline_reseed_operator`
|
||||
- `baseline_reseed_max_intra_batch_regression`
|
||||
|
||||
Use a fresh reseed timestamp for each seed record, not the original source
|
||||
result timestamp. This is required because
|
||||
`load_records_for_model(..., last_n=5)` keeps the last records after loading
|
||||
the model directory; stale filenames/timestamps may not enter the last-5
|
||||
window and therefore may not move the median. Preserve the original source
|
||||
timestamp in `baseline_reseed_source_timestamp`.
|
||||
|
||||
Use the existing filename convention from `_write_tracking_record()`:
|
||||
`<sanitize(timestamp)>_<sanitize(commit_sha)>.json` under the sanitized model
|
||||
directory, but include a deterministic suffix such as `_reseed_01`,
|
||||
`_reseed_02`, and so on before `.json` so multiple records from the same
|
||||
batch do not overwrite each other.
|
||||
|
||||
If a source record already exists on HF with `success=false`, do not edit it
|
||||
in place unless the user explicitly asked for an audit-preserving correction.
|
||||
Prefer uploading new accepted seed records so failed history remains visible.
|
||||
|
||||
### 6. Pause before upload
|
||||
|
||||
Print:
|
||||
|
||||
- Backup directory path under `/tmp`.
|
||||
- Prepared local record paths under `PERFORMANCE_TRACKING_ROOT`.
|
||||
- HF paths that will receive the new records.
|
||||
- Old rolling medians.
|
||||
- Source batch medians, source batch spread, reseed count, and candidate
|
||||
medians.
|
||||
- Rationale.
|
||||
|
||||
Ask the user to reply exactly `upload`. Anything else aborts and leaves the
|
||||
prepared records plus backup on disk.
|
||||
|
||||
### 7. Upload only the scoped records
|
||||
|
||||
Use the shared storage helper so the path and repo type match CI:
|
||||
|
||||
```python
|
||||
from hf_store import upload_record
|
||||
|
||||
upload_record("<local_record_path>", record, strict=True)
|
||||
```
|
||||
|
||||
Run it once per prepared record. Each upload goes to:
|
||||
|
||||
```text
|
||||
FastVideo/performance-tracking/<sanitize(model_id)>/<record_filename>.json
|
||||
```
|
||||
|
||||
Never bulk upload the whole tracking root. Never modify another model's
|
||||
directory in the same operation.
|
||||
|
||||
### 8. Report outcome and offer cleanup
|
||||
|
||||
Report:
|
||||
|
||||
- Uploaded HF paths.
|
||||
- Backup directory under `/tmp`.
|
||||
- Local tracking root, usually `/tmp/perf-tracking`.
|
||||
- Old baseline window count and medians.
|
||||
- Source batch medians, source batch spread, reseed count, and candidate
|
||||
medians.
|
||||
- Expected effect based on reseed count.
|
||||
- Any separate threshold changes still needed in
|
||||
`.buildkite/performance-benchmarks/tests/*.json`.
|
||||
|
||||
Include the `intent_rationale` in the PR or follow-up comment so reviewers can
|
||||
distinguish an accepted baseline shift from a hidden regression.
|
||||
|
||||
After the upload is verified, ask whether the user wants to clear temporary
|
||||
local state. Explain what each directory is for:
|
||||
|
||||
- `PERFORMANCE_TRACKING_ROOT`, usually `/tmp/perf-tracking`: local synced
|
||||
mirror of `FastVideo/performance-tracking` plus the prepared local seed
|
||||
records used for scoped upload.
|
||||
- `/tmp/performance_reseed_backup/<...>`: local backup of the target model's
|
||||
pre-reseed HF history plus `PROVENANCE.txt`, kept so a bad reseed can be
|
||||
audited or corrected.
|
||||
- `/tmp/performance_reseed_source/<...>` when used: downloaded source JSON
|
||||
artifacts from Buildkite URLs.
|
||||
|
||||
Ask:
|
||||
|
||||
> Reseed succeeded. Do you want me to delete the local temp tracking mirror,
|
||||
> source downloads, and reseed backup under `/tmp`? These files are local
|
||||
> safety/audit artifacts only; HF already has the uploaded records.
|
||||
>
|
||||
> Reply `cleanup reseed temp` to delete them, anything else to keep them.
|
||||
|
||||
Do not delete anything unless the user replies exactly
|
||||
`cleanup reseed temp`. If cleanup is requested, remove only the specific
|
||||
directories created for this reseed. Never remove unrelated `/tmp` contents.
|
||||
|
||||
## Failure modes and handling
|
||||
|
||||
- **`HF_API_KEY` unset.** Stop before upload. Do not create an untracked
|
||||
process that appears to have reseeded but never reached HF.
|
||||
- **Source result does not match target.** Stop. The wrong benchmark or GPU
|
||||
would poison a separate baseline.
|
||||
- **Source batch is internally inconsistent.** Stop if any source regresses
|
||||
against the source batch median by more than `max_intra_batch_regression`.
|
||||
Ask for cleaner sources or a reviewed explanation before continuing.
|
||||
- **Too few source records to move the median.** Continue only after making
|
||||
clear that one or two records may not immediately move the last-5 median.
|
||||
- **The source results are noisy or suspicious.** Stop. Reseeding amplifies
|
||||
those measurements into the baseline, so they must be reviewed first.
|
||||
- **HF sync fails.** Stop for destructive reseeds. A stale or empty sync can
|
||||
make the old baseline look missing.
|
||||
- **Candidate still violates fixed thresholds.** Report that this skill only
|
||||
handles the rolling HF baseline; update benchmark JSON thresholds in code
|
||||
review if maintainers accept the new absolute limit.
|
||||
- **The user aborts at either confirmation.** Leave the backup and prepared
|
||||
records on disk. Nothing should be uploaded.
|
||||
- **The user declines cleanup.** Keep `/tmp/perf-tracking`, the source
|
||||
download directory if any, and `/tmp/performance_reseed_backup/<...>` in
|
||||
place for audit/debugging.
|
||||
- **A bad seed was uploaded.** Use the backup and HF history to identify the
|
||||
uploaded file, then remove or supersede it with an explicitly reviewed
|
||||
corrective record. Do not silently rewrite unrelated history.
|
||||
|
||||
## References
|
||||
|
||||
- `.agents/skills/reseed-ssim-references/SKILL.md` — safety pattern for
|
||||
intentional baseline replacement.
|
||||
- `fastvideo/tests/performance/compare_baseline.py` — normalization, rolling
|
||||
median comparison, and persistence rules.
|
||||
- `fastvideo/tests/performance/hf_store.py` — HF sync, record loading,
|
||||
`sanitize()`, and `upload_record()`.
|
||||
- `fastvideo/tests/performance/test_inference_performance.py` — source result
|
||||
JSON schema.
|
||||
- `.buildkite/performance-benchmarks/tests/*.json` — fixed absolute benchmark
|
||||
thresholds, separate from rolling baseline comparisons.
|
||||
|
||||
## Changelog
|
||||
|
||||
| Date | Change |
|
||||
|------|--------|
|
||||
| 2026-05-03 | Initial version. Sister workflow to `reseed-ssim-references`, scoped to one performance `(model_id, gpu_type)` baseline seed with backup, confirmation, provenance, and `success=true` upload. |
|
||||
| 2026-05-03 | Previous policy: replicate one approved shifted source result into 3 success records by default, or 5 only when explicitly requested. Add provenance marker for replicated-source reseeds. Superseded by the 2026-05-08 dynamic multi-source policy. |
|
||||
| 2026-05-08 | Replace fixed 3/5 replication with dynamic multi-source reseeding: upload one seed record per reviewed source JSON, validate intra-batch consistency, move backup/source scratch under `/tmp`, and ask whether to clean temp state after successful upload. |
|
||||
@@ -89,5 +89,4 @@ The following land in follow-up PRs:
|
||||
|
||||
- **MIND** metrics (depends on a separate `vipe` submodule).
|
||||
- **VBench-2.0** sibling package.
|
||||
- Native conversion of **FVD** under `fastvideo/eval/metrics/fvd/`.
|
||||
- The training-time `EvalCallback`.
|
||||
|
||||
@@ -36,7 +36,10 @@
|
||||
"thresholds": {
|
||||
"L40S": {
|
||||
"max_generation_time_s": 34.0,
|
||||
"max_peak_memory_mb": 11000.0
|
||||
"max_peak_memory_mb": 11000.0,
|
||||
"max_text_encoder_time_s": 5.0,
|
||||
"max_dit_time_s": 10.0,
|
||||
"max_vae_decode_time_s": 10.0
|
||||
},
|
||||
"default": {
|
||||
"max_generation_time_s": 120.0,
|
||||
|
||||
@@ -77,6 +77,17 @@ steps:
|
||||
limit: 2
|
||||
agents:
|
||||
queue: "default"
|
||||
- label: ":microscope: DreamVerse App Tests"
|
||||
if: build.env("TEST_SCOPE") == "direct" && build.env("TEST_TYPE") == "dreamverse_app"
|
||||
command: "timeout 90m .buildkite/scripts/pr_test.sh"
|
||||
retry:
|
||||
automatic:
|
||||
- exit_status: 128
|
||||
limit: 3
|
||||
- exit_status: -1
|
||||
limit: 2
|
||||
agents:
|
||||
queue: "default"
|
||||
|
||||
# --- Full-suite-scope direct tests ---
|
||||
- label: ":bar_chart: SSIM Tests"
|
||||
@@ -289,6 +300,16 @@ steps:
|
||||
- TEST_TYPE=unit_test
|
||||
agents:
|
||||
queue: "default"
|
||||
- path:
|
||||
- "apps/dreamverse/**"
|
||||
- "pyproject.toml"
|
||||
config:
|
||||
command: "timeout 30m .buildkite/scripts/pr_test.sh"
|
||||
label: ":microscope: DreamVerse App Tests"
|
||||
env:
|
||||
- TEST_TYPE=dreamverse_app
|
||||
agents:
|
||||
queue: "default"
|
||||
|
||||
# ============================================================
|
||||
# Full Suite: Runs when TEST_SCOPE=full
|
||||
|
||||
@@ -121,6 +121,19 @@ upload_performance_artifacts() {
|
||||
fi
|
||||
}
|
||||
|
||||
_upload_normalized_perf_results() {
|
||||
local found=0
|
||||
while IFS= read -r -d '' target; do
|
||||
found=1
|
||||
log "Found normalized performance result: $target. Uploading to Buildkite..."
|
||||
buildkite-agent artifact upload "$target"
|
||||
done < <(find "$LOCAL_DIR" -path "*/results/normalized_perf_*.json" -print0)
|
||||
|
||||
if [ "$found" -eq 0 ]; then
|
||||
log "No normalized performance result artifacts found. This is expected when the rolling performance comparison did not run."
|
||||
fi
|
||||
}
|
||||
|
||||
_cleanup_modal_volume() {
|
||||
log "Cleaning up perf_reports/ from Modal Volume..."
|
||||
if modal volume rm hf-model-weights "perf_reports/" --recursive; then
|
||||
@@ -139,6 +152,7 @@ upload_performance_artifacts() {
|
||||
_download_reports || { _cleanup_local; return 1; }
|
||||
_upload_dashboard
|
||||
_upload_perf_summary
|
||||
_upload_normalized_perf_results
|
||||
_cleanup_modal_volume
|
||||
_cleanup_local
|
||||
}
|
||||
@@ -197,6 +211,10 @@ case "$TEST_TYPE" in
|
||||
log "Running unit tests..."
|
||||
MODAL_COMMAND="$MODAL_ENV python3 -m modal run $MODAL_TEST_FILE::run_unit_test"
|
||||
;;
|
||||
"dreamverse_app")
|
||||
log "Running DreamVerse app tests..."
|
||||
MODAL_COMMAND="$MODAL_ENV python3 -m modal run $MODAL_TEST_FILE::run_dreamverse_app_tests"
|
||||
;;
|
||||
"train_framework")
|
||||
log "Running fastvideo.train framework tests..."
|
||||
MODAL_COMMAND="$MODAL_ENV HF_API_KEY=$HF_API_KEY python3 -m modal run $MODAL_TEST_FILE::run_train_framework_tests"
|
||||
|
||||
@@ -12,6 +12,18 @@ on:
|
||||
tag_suffix:
|
||||
required: true
|
||||
type: string
|
||||
image_name:
|
||||
required: false
|
||||
type: string
|
||||
default: fastvideo-dev
|
||||
build_args:
|
||||
required: false
|
||||
type: string
|
||||
default: ''
|
||||
include_latest_tags:
|
||||
required: false
|
||||
type: boolean
|
||||
default: true
|
||||
|
||||
jobs:
|
||||
build-and-push:
|
||||
@@ -67,11 +79,14 @@ jobs:
|
||||
run: |
|
||||
SHORT_SHA=$(echo ${{ github.sha }} | cut -c1-7)
|
||||
|
||||
TAGS="type=raw,value=${{ inputs.tag_suffix }}-latest"
|
||||
TAGS="${TAGS}\ntype=raw,value=${{ inputs.tag_suffix }}-sha-${SHORT_SHA}"
|
||||
|
||||
TAGS="type=raw,value=${{ inputs.tag_suffix }}-sha-${SHORT_SHA}"
|
||||
|
||||
if [[ "${{ inputs.include_latest_tags }}" == "true" ]]; then
|
||||
TAGS="type=raw,value=${{ inputs.tag_suffix }}-latest\n${TAGS}"
|
||||
fi
|
||||
|
||||
# Set Python 3.10 as the default image
|
||||
if [[ "${{ inputs.python_version }}" == "3.10" ]]; then
|
||||
if [[ "${{ inputs.include_latest_tags }}" == "true" && "${{ inputs.python_version }}" == "3.10" ]]; then
|
||||
TAGS="${TAGS}\ntype=raw,value=latest"
|
||||
fi
|
||||
|
||||
@@ -85,7 +100,7 @@ jobs:
|
||||
id: meta
|
||||
uses: docker/metadata-action@v5
|
||||
with:
|
||||
images: ghcr.io/${{ github.repository }}/fastvideo-dev
|
||||
images: ghcr.io/${{ github.repository }}/${{ inputs.image_name }}
|
||||
tags: ${{ steps.prepare-tags.outputs.tags }}
|
||||
|
||||
- name: Build and push Docker image
|
||||
@@ -97,10 +112,11 @@ jobs:
|
||||
push: true
|
||||
tags: ${{ steps.meta.outputs.tags }}
|
||||
labels: ${{ steps.meta.outputs.labels }}
|
||||
build-args: ${{ inputs.build_args }}
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
|
||||
- name: Success message
|
||||
run: |
|
||||
echo "✅ Python ${{ inputs.python_version }} image successfully built and pushed to ghcr.io/${{ github.repository }}/fastvideo-dev:${{ inputs.tag_suffix }}-latest"
|
||||
echo "To run tests with this image, manually trigger the 'Run Tests' workflow."
|
||||
echo "✅ Python ${{ inputs.python_version }} image successfully built and pushed to ghcr.io/${{ github.repository }}/${{ inputs.image_name }}:${{ inputs.tag_suffix }}-sha-${GITHUB_SHA::7}"
|
||||
echo "To run tests with this image, manually trigger the 'Run Tests' workflow."
|
||||
|
||||
@@ -125,7 +125,7 @@ jobs:
|
||||
set -euo pipefail
|
||||
TEST_NAME=$(echo "$COMMENT" | grep -oP '(?<=/test\s)\S+' | head -1 || true)
|
||||
|
||||
VALID="encoder vae transformer kernel unit ssim training lora-inference lora-training distillation self-forcing vsa vmoba performance api train-framework full fastcheck pre-commit"
|
||||
VALID="encoder vae transformer kernel unit dreamverse ssim training lora-inference lora-training distillation self-forcing vsa vmoba performance api train-framework full fastcheck pre-commit"
|
||||
if [ -z "$TEST_NAME" ] || ! echo "$VALID" | grep -qw "$TEST_NAME"; then
|
||||
echo "Unknown test: '$TEST_NAME'. Valid: $VALID"
|
||||
exit 1
|
||||
@@ -133,7 +133,7 @@ jobs:
|
||||
|
||||
declare -A MAP=(
|
||||
[encoder]=encoder [vae]=vae [transformer]=transformer
|
||||
[kernel]=kernel_tests [unit]=unit_test
|
||||
[kernel]=kernel_tests [unit]=unit_test [dreamverse]=dreamverse_app
|
||||
[ssim]=ssim [training]=training
|
||||
[lora-inference]=inference_lora [lora-training]=training_lora
|
||||
[distillation]=distillation_dmd [self-forcing]=self_forcing
|
||||
|
||||
@@ -23,6 +23,11 @@ on:
|
||||
required: false
|
||||
default: false
|
||||
type: boolean
|
||||
dreamverse_cuda_12_9:
|
||||
description: 'Build Dreamverse CUDA 12.9 backend-only and UI images'
|
||||
required: false
|
||||
default: false
|
||||
type: boolean
|
||||
|
||||
|
||||
permissions:
|
||||
@@ -65,3 +70,27 @@ jobs:
|
||||
dockerfile_path: docker/Dockerfile.python3.12.cuda12.9.1
|
||||
tag_suffix: py3.12-cuda12.9.1
|
||||
secrets: inherit
|
||||
|
||||
build-dreamverse-backend-cuda-12-9:
|
||||
if: ${{ github.event.inputs.dreamverse_cuda_12_9 == 'true' }}
|
||||
uses: ./.github/workflows/_template-build-image.yml
|
||||
with:
|
||||
python_version: '3.12'
|
||||
dockerfile_path: apps/dreamverse/docker/Dockerfile
|
||||
tag_suffix: dreamverse-backend-cuda12.9.1
|
||||
image_name: dreamverse
|
||||
build_args: BUILD_DREAMVERSE_UI=0
|
||||
include_latest_tags: false
|
||||
secrets: inherit
|
||||
|
||||
build-dreamverse-ui-cuda-12-9:
|
||||
if: ${{ github.event.inputs.dreamverse_cuda_12_9 == 'true' }}
|
||||
uses: ./.github/workflows/_template-build-image.yml
|
||||
with:
|
||||
python_version: '3.12'
|
||||
dockerfile_path: apps/dreamverse/docker/Dockerfile
|
||||
tag_suffix: dreamverse-ui-cuda12.9.1
|
||||
image_name: dreamverse
|
||||
build_args: BUILD_DREAMVERSE_UI=1
|
||||
include_latest_tags: false
|
||||
secrets: inherit
|
||||
|
||||
@@ -42,6 +42,8 @@ FastVideo has the following features:
|
||||
- Support H100, A100, 4090
|
||||
- Support Linux, Windows, MacOS
|
||||
- See this [page](https://hao-ai-lab.github.io/FastVideo/inference/support_matrix/) for full list of supported models, hardware assumptions, and optimization compatibility.
|
||||
- Realtime video generation & editing
|
||||
- [Dreamverse](apps/dreamverse/README.md): stream and "vibe direct" video in realtime ([live demo](https://dreamverse.fastvideo.org/)), deployable on local GPU, a self-hosted B200 server, Docker, or serverless Modal
|
||||
|
||||
## Getting Started
|
||||
|
||||
@@ -69,6 +71,19 @@ See below for recipes and datasets:
|
||||
| [FastWan2.1-T2V-1.3B](https://huggingface.co/FastVideo/FastWan2.1-T2V-1.3B-Diffusers) | [Recipe](https://github.com/hao-ai-lab/FastVideo/tree/main/examples/distill/Wan2.1-T2V/Wan-Syn-Data-480P) | [FastVideo Synthetic Wan2.1 480P](https://huggingface.co/datasets/FastVideo/Wan-Syn_77x448x832_600k) |
|
||||
| [FastWan2.2-TI2V-5B](https://huggingface.co/FastVideo/FastWan2.2-TI2V-5B-Diffusers) | [Recipe](https://github.com/hao-ai-lab/FastVideo/tree/main/examples/distill/Wan2.2-TI2V-5B-Diffusers/Data-free) | [FastVideo Synthetic Wan2.2 720P](https://huggingface.co/datasets/FastVideo/Wan2.2-Syn-121x704x1280_32k) |
|
||||
|
||||
## Dreamverse — Realtime Video Generation & Editing
|
||||
|
||||
[Dreamverse](apps/dreamverse/README.md) is FastVideo's realtime video generation
|
||||
and editing platform — "vibe directing" a video as it streams. It lives in the
|
||||
monorepo under [`apps/dreamverse/`](apps/dreamverse/) and ships its own backend
|
||||
(`dreamverse-server`) plus a web UI.
|
||||
|
||||
Try the [live demo](https://dreamverse.fastvideo.org/), read the
|
||||
[blog](https://haoailab.com/blogs/dreamverse/), or run it yourself. Dreamverse
|
||||
deploys on a local GPU, a self-hosted B200 server over SSH, Docker, or
|
||||
serverless [Modal](apps/dreamverse/scripts/modal/README.md) — see the
|
||||
[Dreamverse README](apps/dreamverse/README.md).
|
||||
|
||||
## Inference
|
||||
|
||||
### Generating Your First Video
|
||||
|
||||
@@ -2,6 +2,8 @@
|
||||
|
||||
Dreamverse is the FastVideo realtime video generation & editing platform. It lives in this monorepo under `apps/dreamverse/`.
|
||||
|
||||
**Deploy on:** [local GPU](#quick-start-local-gpu) · [self-hosted B200 (SSH)](#server-b200-deployment-ssh) · [Docker](docker/README.md) · [Modal](scripts/modal/README.md)
|
||||
|
||||
## Install Dreamverse
|
||||
|
||||
You can install Dreamverse using one of the methods below.
|
||||
@@ -93,17 +95,24 @@ dreamverse-server --port 8009
|
||||
dreamverse-mock-server --port 8009
|
||||
```
|
||||
|
||||
> **Expect a slow first boot.** With `torch.compile` and startup warmup enabled
|
||||
> (the default), the backend compiles the segment 1 and segment 2 inference
|
||||
> paths before it reports ready — this can take **tens of minutes on a cold
|
||||
> cache**, regardless of how you deploy (local, server, Docker, or Modal).
|
||||
> `/healthz` responds as soon as the process is up; `/readyz` stays `503` until
|
||||
> warmup finishes. For a faster, uncompiled startup while testing, set
|
||||
> `FASTVIDEO_ENABLE_STARTUP_WARMUP=0` before starting the backend.
|
||||
|
||||
## Frontend Setup
|
||||
|
||||
Install the web dependencies once from the FastVideo checkout:
|
||||
|
||||
```bash
|
||||
cd apps/dreamverse/web
|
||||
pnpm install --frozen-lockfile
|
||||
npm ci
|
||||
```
|
||||
|
||||
The frontend package also has an npm lockfile, but the bundled launch scripts
|
||||
use `pnpm`.
|
||||
The frontend package uses `package-lock.json`; use npm for installs and scripts.
|
||||
|
||||
## Quick Start: Local GPU
|
||||
|
||||
@@ -159,11 +168,37 @@ Start the frontend:
|
||||
|
||||
```bash
|
||||
cd apps/dreamverse/web
|
||||
BACKEND_HOST=localhost BACKEND_PORT=8009 pnpm run dev
|
||||
BACKEND_HOST=localhost BACKEND_PORT=8009 npm run dev
|
||||
```
|
||||
|
||||
Open `http://localhost:5299`.
|
||||
|
||||
## Server B200 deployment (SSH)
|
||||
|
||||
Deploying on a remote GPU host (for example a B200 box) is a local install run
|
||||
over SSH, plus a few server-specific concerns. Two paths:
|
||||
|
||||
### Option A: Native (source install)
|
||||
|
||||
SSH in, then follow [Install → From source](#method-2-from-source) and
|
||||
(recommended) [Building FFmpeg](#optional-building-ffmpeg-for-better-performance),
|
||||
then start the backend as in [Quick Start: Local GPU](#quick-start-local-gpu).
|
||||
For a remote host, a few things differ from localhost:
|
||||
|
||||
- Bind all interfaces: `dreamverse-server --host 0.0.0.0 --port 8009`.
|
||||
- Point the frontend/client at the host: `BACKEND_HOST=<b200-host> BACKEND_PORT=8009 npm run dev`.
|
||||
- Keep the backend alive across SSH sessions (`tmux` / `systemd` / `nohup`).
|
||||
- Expose / firewall port `8009`, or front it with a reverse proxy + auth.
|
||||
|
||||
### Option B: Docker (on the server)
|
||||
|
||||
SSH in, then follow [Install → Using Docker](#method-3-using-docker) and the run
|
||||
steps in [`docker/README.md`](docker/README.md):
|
||||
|
||||
```bash
|
||||
CEREBRAS_API_KEY="<key>" GROQ_API_KEY="<key>" apps/dreamverse/docker/docker_run.sh
|
||||
```
|
||||
|
||||
## Quick Start: Mock Backend (For UI development)
|
||||
|
||||
The mock server emulates the Dreamverse backend protocol and streams a
|
||||
@@ -195,14 +230,14 @@ Run the frontend tests:
|
||||
|
||||
```bash
|
||||
cd apps/dreamverse/web
|
||||
pnpm test
|
||||
npm test
|
||||
```
|
||||
|
||||
Run the frontend e2e tests:
|
||||
|
||||
```bash
|
||||
cd apps/dreamverse/web
|
||||
pnpm run e2e
|
||||
npm run e2e
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
@@ -238,7 +273,7 @@ Only one GPU is used
|
||||
Frontend cannot connect to backend
|
||||
|
||||
- confirm the backend is running on `8009`; if not, point the frontend at it
|
||||
with `BACKEND_HOST=<host> BACKEND_PORT=<port> pnpm run dev`
|
||||
with `BACKEND_HOST=<host> BACKEND_PORT=<port> npm run dev`
|
||||
- confirm `http://localhost:8009/healthz` responds before starting the frontend
|
||||
- confirm `http://localhost:8009/readyz` returns `200` before clicking Generate
|
||||
- use `apps/dreamverse/scripts/smoke_local.sh` for a repeatable local startup
|
||||
|
||||
@@ -1,40 +0,0 @@
|
||||
.git/
|
||||
**/.git/
|
||||
.venv/
|
||||
**/.venv/
|
||||
**/__pycache__/
|
||||
**/*.pyc
|
||||
**/*.pyo
|
||||
**/*.egg-info/
|
||||
.pytest_cache/
|
||||
**/.pytest_cache/
|
||||
.mypy_cache/
|
||||
.ruff_cache/
|
||||
.cache/
|
||||
|
||||
apps/dreamverse/web/node_modules/
|
||||
apps/dreamverse/web/.next/
|
||||
apps/dreamverse/web/out/
|
||||
apps/dreamverse/web/dist/
|
||||
apps/dreamverse/web/test-results/
|
||||
apps/dreamverse/web/playwright-report/
|
||||
|
||||
apps/dreamverse/outputs/
|
||||
apps/dreamverse/dreamverse/outputs/
|
||||
apps/dreamverse/dreamverse/prompts.local/
|
||||
apps/dreamverse/logs/
|
||||
outputs/
|
||||
logs/
|
||||
slurm-logs/
|
||||
wandb/
|
||||
|
||||
.env
|
||||
.env.*
|
||||
**/prompts.local/
|
||||
.codex/
|
||||
.agents/exploration/
|
||||
.vscode/
|
||||
.idea/
|
||||
*.log
|
||||
*.tmp
|
||||
*.pdf
|
||||
@@ -3,6 +3,7 @@ ARG CUDA_TAG=12.9.1-cudnn-devel-ubuntu22.04
|
||||
FROM nvidia/cuda:${CUDA_TAG}
|
||||
|
||||
ARG BUILD_FASTVIDEO_KERNEL_FROM_SOURCE=0
|
||||
ARG BUILD_DREAMVERSE_UI=0
|
||||
|
||||
ENV DEBIAN_FRONTEND=noninteractive \
|
||||
PYTHONUNBUFFERED=1 \
|
||||
@@ -50,6 +51,18 @@ RUN if [[ "${BUILD_FASTVIDEO_KERNEL_FROM_SOURCE}" == "1" ]]; then \
|
||||
echo "Skipping source fastvideo-kernel build; using installed fastvideo-kernel package."; \
|
||||
fi
|
||||
|
||||
RUN if [[ "${BUILD_DREAMVERSE_UI}" == "1" ]]; then \
|
||||
curl -fsSL https://deb.nodesource.com/setup_22.x | bash - \
|
||||
&& apt-get install -y --no-install-recommends nodejs \
|
||||
&& rm -rf /var/lib/apt/lists/* \
|
||||
&& cd /opt/FastVideo/apps/dreamverse/web \
|
||||
&& npm ci --ignore-scripts \
|
||||
&& NEXT_OUTPUT_EXPORT=1 npm run build \
|
||||
&& rm -rf node_modules .next; \
|
||||
else \
|
||||
echo "Skipping Dreamverse UI build; Building backend-only image."; \
|
||||
fi
|
||||
|
||||
# The monorepo ffmpeg installer force-selects conda compiler triplets for
|
||||
# local dev shells. Inside this image we explicitly opt into the system
|
||||
# gcc/g++ toolchain.
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
**/.git/
|
||||
.venv/
|
||||
**/.venv/
|
||||
.*dreamverse*/
|
||||
**/__pycache__/
|
||||
**/*.pyc
|
||||
**/*.pyo
|
||||
@@ -24,6 +25,9 @@ apps/dreamverse/dreamverse/outputs/
|
||||
apps/dreamverse/dreamverse/prompts.local/
|
||||
apps/dreamverse/logs/
|
||||
outputs/
|
||||
outputs_video/
|
||||
quality_check_outputs/
|
||||
data/
|
||||
logs/
|
||||
slurm-logs/
|
||||
wandb/
|
||||
@@ -32,7 +36,9 @@ wandb/
|
||||
.env.*
|
||||
**/prompts.local/
|
||||
.codex/
|
||||
.agents/exploration/
|
||||
.agents/
|
||||
.opencode/
|
||||
.agent_tmp/
|
||||
.vscode/
|
||||
.idea/
|
||||
*.log
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
# Dreamverse Docker Image
|
||||
|
||||
This folder contains the backend-only Docker image for Dreamverse inside the
|
||||
FastVideo monorepo. Build commands use the FastVideo repository root as the
|
||||
Docker context, so run the helper scripts from this folder or from any path in
|
||||
the checkout.
|
||||
This folder contains the Docker image for Dreamverse inside the FastVideo
|
||||
monorepo. Build commands use the FastVideo repository root as the Docker
|
||||
context, so run the helper scripts from this folder or from any path in the
|
||||
checkout.
|
||||
|
||||
## Build
|
||||
|
||||
@@ -17,6 +17,16 @@ The image defaults to `dreamverse:dev`. Override it with:
|
||||
DREAMVERSE_IMAGE=dreamverse:local apps/dreamverse/docker/docker_build.sh
|
||||
```
|
||||
|
||||
Backend-only remains the default image. To include the static Dreamverse UI
|
||||
served by the backend, set `BUILD_DREAMVERSE_UI=1` and choose a specific image
|
||||
tag:
|
||||
|
||||
```bash
|
||||
BUILD_DREAMVERSE_UI=1 DREAMVERSE_IMAGE=<image-tag> apps/dreamverse/docker/docker_build.sh
|
||||
```
|
||||
|
||||
Prefer SHA-specific tags for deployable images; avoid `latest`.
|
||||
|
||||
The Dockerfile builds a CUDA 12.9.1 image, installs FastVideo from this
|
||||
checkout with the `dreamverse` extra, installs the FA4
|
||||
flash-attention fork, builds native FFmpeg, and installs FlashInfer for NVFP4
|
||||
|
||||
@@ -4,13 +4,44 @@ set -euo pipefail
|
||||
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
||||
REPO_ROOT="$(cd -- "${SCRIPT_DIR}/../../.." && pwd)"
|
||||
IMAGE="${DREAMVERSE_IMAGE:-dreamverse:dev}"
|
||||
ROOT_DOCKERIGNORE="${REPO_ROOT}/.dockerignore"
|
||||
DOCKERFILE_DOCKERIGNORE="${SCRIPT_DIR}/Dockerfile.dockerignore"
|
||||
CREATED_ROOT_DOCKERIGNORE_SYMLINK=0
|
||||
|
||||
cleanup_root_dockerignore_symlink() {
|
||||
if [[ "${CREATED_ROOT_DOCKERIGNORE_SYMLINK}" == "1" && -L "${ROOT_DOCKERIGNORE}" ]] && \
|
||||
[[ "$(readlink "${ROOT_DOCKERIGNORE}")" == "${DOCKERFILE_DOCKERIGNORE}" ]]; then
|
||||
rm -- "${ROOT_DOCKERIGNORE}"
|
||||
fi
|
||||
}
|
||||
trap cleanup_root_dockerignore_symlink EXIT INT TERM
|
||||
|
||||
if [[ -z "${DOCKER_BUILDKIT:-}" ]] && docker buildx version >/dev/null 2>&1; then
|
||||
export DOCKER_BUILDKIT=1
|
||||
fi
|
||||
|
||||
build_args=()
|
||||
[[ -n "${CUDA_TAG:-}" ]] && build_args+=(--build-arg "CUDA_TAG=${CUDA_TAG}")
|
||||
[[ -n "${BUILD_FASTVIDEO_KERNEL_FROM_SOURCE:-}" ]] && \
|
||||
build_args+=(--build-arg "BUILD_FASTVIDEO_KERNEL_FROM_SOURCE=${BUILD_FASTVIDEO_KERNEL_FROM_SOURCE}")
|
||||
build_args+=(--build-arg "BUILD_DREAMVERSE_UI=${BUILD_DREAMVERSE_UI:-0}")
|
||||
|
||||
exec docker build \
|
||||
if [[ "${DOCKER_BUILDKIT:-}" == "1" ]]; then
|
||||
if [[ -L "${ROOT_DOCKERIGNORE}" ]] && \
|
||||
[[ "$(readlink "${ROOT_DOCKERIGNORE}")" == "${DOCKERFILE_DOCKERIGNORE}" ]]; then
|
||||
rm -- "${ROOT_DOCKERIGNORE}"
|
||||
printf 'Removed stale temporary root .dockerignore symlink: %s\n' "${ROOT_DOCKERIGNORE}"
|
||||
fi
|
||||
elif [[ -e "${ROOT_DOCKERIGNORE}" || -L "${ROOT_DOCKERIGNORE}" ]]; then
|
||||
printf 'Using existing root .dockerignore: %s\n' "${ROOT_DOCKERIGNORE}"
|
||||
else
|
||||
ln -s -- "${DOCKERFILE_DOCKERIGNORE}" "${ROOT_DOCKERIGNORE}"
|
||||
CREATED_ROOT_DOCKERIGNORE_SYMLINK=1
|
||||
printf 'Created temporary root .dockerignore symlink for legacy Docker builder: %s -> %s\n' \
|
||||
"${ROOT_DOCKERIGNORE}" "${DOCKERFILE_DOCKERIGNORE}"
|
||||
fi
|
||||
|
||||
docker build \
|
||||
-f "${SCRIPT_DIR}/Dockerfile" \
|
||||
-t "${IMAGE}" \
|
||||
"${build_args[@]}" \
|
||||
|
||||
@@ -4,6 +4,7 @@ from pathlib import Path
|
||||
_REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||
_SERVER_ROOT = Path(__file__).resolve().parent
|
||||
_FASTVIDEO_DREAMVERSE_HOME = os.environ.get("FASTVIDEO_DREAMVERSE_HOME")
|
||||
_FASTVIDEO_DREAMVERSE_FRONTEND_ROOT = os.environ.get("FASTVIDEO_DREAMVERSE_FRONTEND_ROOT")
|
||||
_XDG_STATE_HOME = os.environ.get("XDG_STATE_HOME")
|
||||
_DEFAULT_STATE_ROOT = (Path(_FASTVIDEO_DREAMVERSE_HOME) if _FASTVIDEO_DREAMVERSE_HOME else
|
||||
(Path(_XDG_STATE_HOME) if _XDG_STATE_HOME else Path.home() / ".local/state") /
|
||||
@@ -16,18 +17,39 @@ _APP_ROOT = _REPO_ROOT
|
||||
|
||||
def _resolve_frontend_root() -> Path:
|
||||
for candidate in (
|
||||
Path(_FASTVIDEO_DREAMVERSE_FRONTEND_ROOT) if _FASTVIDEO_DREAMVERSE_FRONTEND_ROOT else None,
|
||||
_APP_ROOT / "web",
|
||||
_APP_ROOT / "prod-ui",
|
||||
):
|
||||
if candidate.is_dir():
|
||||
if candidate is not None and candidate.is_dir():
|
||||
return candidate
|
||||
if _FASTVIDEO_DREAMVERSE_FRONTEND_ROOT:
|
||||
return Path(_FASTVIDEO_DREAMVERSE_FRONTEND_ROOT)
|
||||
return _APP_ROOT / "web"
|
||||
|
||||
|
||||
def _resolve_frontend_static_dir_candidates() -> tuple[str, ...]:
|
||||
roots = (
|
||||
FRONTEND_ROOT,
|
||||
Path.cwd() / "apps/dreamverse/web",
|
||||
Path.cwd() / "web",
|
||||
Path("/opt/FastVideo/apps/dreamverse/web"),
|
||||
)
|
||||
candidates: list[str] = []
|
||||
seen: set[Path] = set()
|
||||
for root in roots:
|
||||
resolved_root = root.resolve(strict=False)
|
||||
if resolved_root in seen:
|
||||
continue
|
||||
seen.add(resolved_root)
|
||||
candidates.extend(str(resolved_root / dirname) for dirname in ("out", "dist"))
|
||||
return tuple(candidates)
|
||||
|
||||
|
||||
FRONTEND_ROOT = _resolve_frontend_root()
|
||||
_CLIENT_PROMPTS_ROOT = FRONTEND_ROOT / "prompts"
|
||||
_CLIENT_PROMPTS_LOCAL_ROOT = FRONTEND_ROOT / "prompts.local"
|
||||
FRONTEND_STATIC_DIR_CANDIDATES = tuple(str(FRONTEND_ROOT / dirname) for dirname in ("out", "dist"))
|
||||
FRONTEND_STATIC_DIR_CANDIDATES = _resolve_frontend_static_dir_candidates()
|
||||
|
||||
# Model registry
|
||||
MODEL_REGISTRY = {
|
||||
@@ -35,18 +57,24 @@ MODEL_REGISTRY = {
|
||||
"name": "FastLTX2",
|
||||
"model_path": "FastVideo/LTX2-Distilled-Diffusers",
|
||||
"config_model_path": "FastVideo/LTX2-Distilled-Diffusers",
|
||||
"lora_repo": "FastVideo/LTX2-OmniNFT-LoRA",
|
||||
},
|
||||
"fast-ltx23": {
|
||||
"name": "FastLTX23",
|
||||
"model_path": "FastVideo/LTX-2.3-Distilled-Diffusers",
|
||||
"config_model_path": "FastVideo/LTX-2.3-Distilled-Diffusers",
|
||||
"lora_repo": "FastVideo/LTX-2.3-OmniNFT-LoRA",
|
||||
},
|
||||
}
|
||||
|
||||
DEFAULT_MODEL_ID = "fast-ltx2"
|
||||
|
||||
ACTIVE_MODEL_ID = (os.getenv("DREAMVERSE_MODEL_ID", "").strip() or DEFAULT_MODEL_ID)
|
||||
if ACTIVE_MODEL_ID not in MODEL_REGISTRY:
|
||||
ACTIVE_MODEL_ID = DEFAULT_MODEL_ID
|
||||
|
||||
# Active model configuration
|
||||
MODEL_CONFIG = MODEL_REGISTRY[DEFAULT_MODEL_ID]
|
||||
MODEL_CONFIG = MODEL_REGISTRY[ACTIVE_MODEL_ID]
|
||||
|
||||
# Generation limits
|
||||
SESSION_TIMEOUT_SECONDS = 300
|
||||
@@ -142,6 +170,76 @@ def _optional_env(*names: str) -> str | None:
|
||||
|
||||
DEVTOOLS_ENABLED = _env_bool("FASTVIDEO_ENABLE_DEVTOOLS", False)
|
||||
PROMPT_SAFETY_ENABLED = _env_bool("FASTVIDEO_ENABLE_PROMPT_SAFETY", False)
|
||||
DREAMVERSE_MAX_AUTOTUNE = _env_bool("DREAMVERSE_MAX_AUTOTUNE", True)
|
||||
DREAMVERSE_SP_SIZE = max(1, _env_int("DREAMVERSE_SP_SIZE", 1))
|
||||
|
||||
DREAMVERSE_MODEL_PATH = (os.getenv("DREAMVERSE_MODEL_PATH", "").strip() or None)
|
||||
if DREAMVERSE_MODEL_PATH:
|
||||
MODEL_CONFIG = {
|
||||
**MODEL_CONFIG,
|
||||
"model_path": DREAMVERSE_MODEL_PATH,
|
||||
"config_model_path": DREAMVERSE_MODEL_PATH,
|
||||
}
|
||||
|
||||
AVAILABLE_LORAS = {
|
||||
"pixar": {
|
||||
"repo": "vrgamedevgirl84/LTX_2.3_Pixar_Toon_Style_LoRa",
|
||||
"trigger": "P1x4r",
|
||||
"model": "fast-ltx23",
|
||||
"position": "prepend",
|
||||
"label": "Pixar Toon",
|
||||
},
|
||||
"transition": {
|
||||
"repo": "valiantcat/LTX-2.3-Transition-LORA",
|
||||
"trigger": "zhuanchang",
|
||||
"model": "fast-ltx23",
|
||||
"position": "append",
|
||||
"label": "Transition",
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def _active_model_key() -> str:
|
||||
return ACTIVE_MODEL_ID
|
||||
|
||||
|
||||
def _available_styles_for_active_model() -> list[str]:
|
||||
active = _active_model_key()
|
||||
return [k for k, v in AVAILABLE_LORAS.items() if v.get("model") == active]
|
||||
|
||||
|
||||
def _resolve_lora_spec(spec: str) -> str | None:
|
||||
spec = (spec or "").strip()
|
||||
if not spec:
|
||||
return None
|
||||
if spec.lower() == "omninft":
|
||||
return MODEL_CONFIG.get("lora_repo")
|
||||
if spec.lower() in AVAILABLE_LORAS:
|
||||
return AVAILABLE_LORAS[spec.lower()]["repo"]
|
||||
return spec
|
||||
|
||||
|
||||
def _parse_lora_stack(raw: str) -> list[tuple[str, float]]:
|
||||
out: list[tuple[str, float]] = []
|
||||
for item in (raw or "").split(","):
|
||||
item = item.strip()
|
||||
if not item:
|
||||
continue
|
||||
spec, _, stren = item.partition("@")
|
||||
resolved = _resolve_lora_spec(spec)
|
||||
if resolved:
|
||||
try:
|
||||
strength = float(stren) if stren.strip() else 1.0
|
||||
except ValueError:
|
||||
strength = 1.0
|
||||
out.append((resolved, strength))
|
||||
return out
|
||||
|
||||
|
||||
DREAMVERSE_LORA_PATH = _resolve_lora_spec(os.getenv("DREAMVERSE_LORA_PATH", ""))
|
||||
DREAMVERSE_LORA_NICKNAME = (os.getenv("DREAMVERSE_LORA_NICKNAME", "omninft").strip() or "omninft")
|
||||
DREAMVERSE_LORA_STRENGTH = _env_float("DREAMVERSE_LORA_STRENGTH", 1.0)
|
||||
DREAMVERSE_LORA_STACK = _parse_lora_stack(os.getenv("DREAMVERSE_LORA_STACK", ""))
|
||||
|
||||
|
||||
def _resolve_devtools_paths(
|
||||
|
||||
@@ -13,6 +13,7 @@ from multiprocessing import Process, Queue
|
||||
|
||||
from dreamverse.config import (
|
||||
DEFAULT_MODEL_ID,
|
||||
DREAMVERSE_SP_SIZE,
|
||||
MODEL_REGISTRY,
|
||||
STARTUP_WARMUP_ENABLED,
|
||||
STARTUP_WARMUP_PROMPT,
|
||||
@@ -33,6 +34,8 @@ from dreamverse.worker_ipc import (
|
||||
InitAck,
|
||||
JoinAck,
|
||||
LeaveAck,
|
||||
LoraAck,
|
||||
LoraStackPayload,
|
||||
MediaChunk,
|
||||
MediaComplete,
|
||||
MediaInit,
|
||||
@@ -80,6 +83,7 @@ class CommandType(Enum):
|
||||
USER_STEP = "user_step"
|
||||
USER_LEAVE = "user_leave"
|
||||
RELOAD_MODEL = "reload_model"
|
||||
APPLY_LORA = "apply_lora"
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -282,6 +286,25 @@ def gpu_worker_process(
|
||||
message=str(e),
|
||||
))
|
||||
|
||||
elif cmd.type == CommandType.APPLY_LORA:
|
||||
try:
|
||||
assert isinstance(cmd.payload, LoraStackPayload), (f"APPLY_LORA requires LoraStackPayload, "
|
||||
f"got {type(cmd.payload).__name__}")
|
||||
trigger, position = worker.apply_lora_stack(cmd.payload.stack)
|
||||
response_queue.put(
|
||||
LoraAck(
|
||||
user_id=cmd.user_id,
|
||||
style_trigger=trigger,
|
||||
style_trigger_position=position,
|
||||
))
|
||||
except Exception as e:
|
||||
print(f"[GPU {gpu_id}] Apply LoRA error: {e}")
|
||||
traceback.print_exc()
|
||||
response_queue.put(WorkerError(
|
||||
user_id=cmd.user_id,
|
||||
message=str(e),
|
||||
))
|
||||
|
||||
return True # Continue loop
|
||||
|
||||
if first_cmd is not None:
|
||||
@@ -359,6 +382,25 @@ def gpu_worker_process(
|
||||
message=str(e),
|
||||
))
|
||||
|
||||
elif cmd.type == CommandType.APPLY_LORA:
|
||||
try:
|
||||
assert isinstance(cmd.payload, LoraStackPayload), (f"APPLY_LORA requires LoraStackPayload, "
|
||||
f"got {type(cmd.payload).__name__}")
|
||||
trigger, position = worker.apply_lora_stack(cmd.payload.stack)
|
||||
response_queue.put(
|
||||
LoraAck(
|
||||
user_id=cmd.user_id,
|
||||
style_trigger=trigger,
|
||||
style_trigger_position=position,
|
||||
))
|
||||
except Exception as e:
|
||||
print(f"[GPU {gpu_id}] Apply LoRA error: {e}")
|
||||
traceback.print_exc()
|
||||
response_queue.put(WorkerError(
|
||||
user_id=cmd.user_id,
|
||||
message=str(e),
|
||||
))
|
||||
|
||||
elif cmd.type in (CommandType.USER_JOIN, CommandType.USER_STEP, CommandType.USER_LEAVE):
|
||||
event_loop(first_cmd=cmd)
|
||||
break
|
||||
@@ -729,6 +771,25 @@ class GPUSlot:
|
||||
raise RuntimeError(f"Unexpected step response for {user_id[:8]}: "
|
||||
f"{type(response).__name__}")
|
||||
|
||||
async def apply_lora_stack(
|
||||
self,
|
||||
stack: list[tuple[str, float]],
|
||||
) -> LoraAck:
|
||||
"""Re-apply a runtime LoRA stack on this GPU's worker."""
|
||||
self._active = True
|
||||
user_id = "__lora__"
|
||||
payload = LoraStackPayload(stack=stack)
|
||||
response = await self._send_command_tagged(Command(CommandType.APPLY_LORA, payload=payload, user_id=user_id),
|
||||
timeout=120.0)
|
||||
match response:
|
||||
case LoraAck() as ack:
|
||||
return ack
|
||||
case WorkerError(message=msg):
|
||||
raise RuntimeError(f"Apply LoRA failed on GPU {self.gpu_id}: {msg}")
|
||||
case _:
|
||||
raise RuntimeError(f"Unexpected LoRA response on GPU {self.gpu_id}: "
|
||||
f"{type(response).__name__}")
|
||||
|
||||
async def leave_user(self, user_id: str) -> None:
|
||||
"""Remove a user from this GPU."""
|
||||
try:
|
||||
@@ -785,8 +846,16 @@ class GPUPool:
|
||||
"""Manages multiple GPU worker subprocesses."""
|
||||
|
||||
def __init__(self, gpu_ids: list[int]):
|
||||
self.gpu_ids = gpu_ids
|
||||
self.slots: dict[int, GPUSlot] = {gpu_id: GPUSlot(gpu_id, str(gpu_id)) for gpu_id in gpu_ids}
|
||||
sp_size = DREAMVERSE_SP_SIZE
|
||||
groups = [gpu_ids[i:i + sp_size] for i in range(0, len(gpu_ids), sp_size)]
|
||||
groups = [g for g in groups if len(g) == sp_size]
|
||||
if not groups:
|
||||
raise RuntimeError(f"Not enough GPUs for DREAMVERSE_SP_SIZE={sp_size}: available={gpu_ids}")
|
||||
if sp_size > 1:
|
||||
print(f"[INFO] Sequence-parallel slots (sp_size={sp_size}): " + ", ".join("{" + ",".join(map(str, g)) + "}"
|
||||
for g in groups))
|
||||
self.gpu_ids = [g[0] for g in groups]
|
||||
self.slots: dict[int, GPUSlot] = {g[0]: GPUSlot(g[0], ",".join(str(x) for x in g)) for g in groups}
|
||||
self.waiting_list: list[tuple[str, asyncio.Event, WebSocket]] = []
|
||||
self.client_gpu_map: dict[str, int] = {}
|
||||
self._pool_lock = asyncio.Lock()
|
||||
@@ -903,6 +972,17 @@ class GPUPool:
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
async def apply_lora_stack(
|
||||
self,
|
||||
stack: list[tuple[str, float]],
|
||||
) -> dict[int, str | None]:
|
||||
"""Re-apply a runtime LoRA stack across all ready GPU workers."""
|
||||
ready_slots = [slot for slot in self.slots.values() if slot.ready]
|
||||
if not ready_slots:
|
||||
raise RuntimeError("No ready GPU workers to apply LoRA stack.")
|
||||
results = await asyncio.gather(*(slot.apply_lora_stack(stack) for slot in ready_slots))
|
||||
return {slot.gpu_id: ack.style_trigger for slot, ack in zip(ready_slots, results, strict=False)}
|
||||
|
||||
async def shutdown(self):
|
||||
"""Shutdown all GPU workers."""
|
||||
print("Shutting down GPU pool...")
|
||||
|
||||
@@ -6,18 +6,22 @@ import os
|
||||
from contextlib import asynccontextmanager
|
||||
from pathlib import Path
|
||||
|
||||
from fastapi import FastAPI, WebSocket
|
||||
from fastapi import FastAPI, HTTPException, WebSocket
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
from fastapi.staticfiles import StaticFiles
|
||||
from pydantic import BaseModel
|
||||
from fastvideo.entrypoints.streaming import build_health_router
|
||||
from dreamverse.gpu_pool import GPUPool, get_available_gpus
|
||||
from dreamverse.session_logger import SessionEventLogger
|
||||
|
||||
from dreamverse.config import (
|
||||
AVAILABLE_LORAS,
|
||||
DEVTOOLS_ENABLED,
|
||||
FRONTEND_STATIC_DIR_CANDIDATES,
|
||||
PROMPT_SAFETY_ENABLED,
|
||||
SESSION_LOG_ROOT,
|
||||
_available_styles_for_active_model,
|
||||
_resolve_lora_spec,
|
||||
)
|
||||
from dreamverse.prompt_enhancer import PromptEnhancer
|
||||
from dreamverse.prompt_safety import PromptSafetyFilter
|
||||
@@ -104,6 +108,83 @@ async def websocket_endpoint(websocket: WebSocket):
|
||||
await controller.run()
|
||||
|
||||
|
||||
class LoraRequest(BaseModel):
|
||||
strength: float = 1.0
|
||||
styles: dict[str, float] = {}
|
||||
style: str = ""
|
||||
|
||||
|
||||
@app.get("/lora/options")
|
||||
async def lora_options() -> dict:
|
||||
if not DEVTOOLS_ENABLED:
|
||||
raise HTTPException(status_code=404, detail="Not found")
|
||||
styles = _available_styles_for_active_model()
|
||||
has_base_lora = _resolve_lora_spec("omninft") is not None
|
||||
labels = {"none": "None"}
|
||||
labels.update({k: AVAILABLE_LORAS[k].get("label", k) for k in styles})
|
||||
return {
|
||||
"styles": ["none", *styles],
|
||||
"labels": labels,
|
||||
"has_base_lora": has_base_lora,
|
||||
}
|
||||
|
||||
|
||||
@app.post("/lora")
|
||||
async def apply_lora(request: LoraRequest) -> dict:
|
||||
if not DEVTOOLS_ENABLED:
|
||||
raise HTTPException(status_code=404, detail="Not found")
|
||||
if runtime.gpu_pool is None:
|
||||
raise HTTPException(status_code=503, detail="GPU pool not ready")
|
||||
|
||||
strength = max(0.0, min(1.0, float(request.strength)))
|
||||
allowed_styles = _available_styles_for_active_model()
|
||||
|
||||
requested = dict(request.styles) if request.styles else {}
|
||||
if not requested and request.style and request.style.strip().lower() != "none":
|
||||
requested = {request.style: 1.0}
|
||||
|
||||
styles_map: dict[str, float] = {}
|
||||
for name, intensity in requested.items():
|
||||
key = str(name).strip().lower()
|
||||
if key in ("", "none"):
|
||||
continue
|
||||
if key not in allowed_styles:
|
||||
raise HTTPException(status_code=400, detail=f"Unknown style for active model: {key}")
|
||||
value = max(0.0, min(1.0, float(intensity)))
|
||||
if value <= 0.0:
|
||||
continue
|
||||
styles_map[key] = value
|
||||
|
||||
stack: list[tuple[str, float]] = []
|
||||
if _resolve_lora_spec("omninft") is not None:
|
||||
stack.append(("omninft", strength))
|
||||
for key, intensity in styles_map.items():
|
||||
stack.append((key, intensity))
|
||||
|
||||
if not stack:
|
||||
raise HTTPException(status_code=400,
|
||||
detail="Active model has no OmniNFT LoRA and no style selected; nothing to apply.")
|
||||
|
||||
try:
|
||||
triggers = await runtime.gpu_pool.apply_lora_stack(stack)
|
||||
except Exception as exc:
|
||||
raise HTTPException(status_code=500, detail=str(exc)) from exc
|
||||
|
||||
return {
|
||||
"applied": True,
|
||||
"strength": strength,
|
||||
"styles": styles_map,
|
||||
"triggers": {
|
||||
key: AVAILABLE_LORAS[key]["trigger"]
|
||||
for key in styles_map
|
||||
},
|
||||
"gpus": {
|
||||
str(gpu_id): trigger
|
||||
for gpu_id, trigger in triggers.items()
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
# Serve an exported frontend bundle when present.
|
||||
for static_dir in FRONTEND_STATIC_DIR_CANDIDATES:
|
||||
if os.path.isdir(static_dir):
|
||||
|
||||
@@ -41,7 +41,7 @@ MOCK_FPS = 24
|
||||
MOCK_DURATION_SECONDS = 5.0
|
||||
MOCK_STREAM_CHUNK_SIZE_BYTES = 256 * 1024
|
||||
MOCK_STREAM_CHUNK_DELAY_MS = 15
|
||||
MOCK_AV_MIME = 'video/mp4; codecs="avc1.42E01E"'
|
||||
MOCK_AV_MIME = 'video/mp4; codecs="avc1.42E01E,mp4a.40.2"'
|
||||
FFMPEG_BIN = shutil.which(os.getenv("FASTVIDEO_FFMPEG_BIN", "ffmpeg"))
|
||||
MOCK_SEGMENT_BYTES: bytes | None = None
|
||||
|
||||
@@ -75,9 +75,14 @@ def _build_mock_segment_bytes() -> bytes:
|
||||
"lavfi",
|
||||
"-i",
|
||||
f"testsrc2=size={MOCK_FRAME_WIDTH}x{MOCK_FRAME_HEIGHT}:rate={MOCK_FPS}",
|
||||
# Silent stereo AAC track. We don't need audible content — the FE's
|
||||
# audio-decode coverage only needs a real AAC track present
|
||||
"-f",
|
||||
"lavfi",
|
||||
"-i",
|
||||
"anullsrc=channel_layout=stereo:sample_rate=48000",
|
||||
"-t",
|
||||
f"{MOCK_DURATION_SECONDS}",
|
||||
"-an",
|
||||
"-c:v",
|
||||
"libx264",
|
||||
"-preset",
|
||||
@@ -90,6 +95,12 @@ def _build_mock_segment_bytes() -> bytes:
|
||||
"baseline",
|
||||
"-level",
|
||||
"3.0",
|
||||
"-c:a",
|
||||
"aac",
|
||||
"-b:a",
|
||||
"128k",
|
||||
"-ar",
|
||||
"48000",
|
||||
"-movflags",
|
||||
"+empty_moov+default_base_moof+frag_keyframe",
|
||||
"-frag_duration",
|
||||
@@ -183,6 +194,55 @@ async def status():
|
||||
"queue_size": 0,
|
||||
"available_gpus": 1,
|
||||
"total_gpus": 1,
|
||||
"warmup_enabled": False,
|
||||
"warmup_successful_gpus": 1,
|
||||
"warmup_failed_gpus": 0,
|
||||
"gpu_status": {
|
||||
0: {
|
||||
"ready": True,
|
||||
"available": True,
|
||||
"client_count": 0,
|
||||
"current_model_id": "mock-ltx2",
|
||||
"process_alive": True,
|
||||
"warmup_enabled": False,
|
||||
"warmup_success": True,
|
||||
"warmup_error": None,
|
||||
"warmup_timings": {},
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
@app.get("/prompt-system-config")
|
||||
async def prompt_system_config():
|
||||
return {
|
||||
"next_segment_system_prompt": "Mock next segment prompt.",
|
||||
"auto_extension_system_prompt": "Mock auto extension prompt.",
|
||||
"rewrite_window_system_prompt": "Mock rewrite window prompt.",
|
||||
"rewrite_user_system_prompt": "Mock rewrite user prompt.",
|
||||
"rewrite_model": "mock-rewrite-model",
|
||||
"rewrite_temperature": 0.0,
|
||||
}
|
||||
|
||||
|
||||
@app.get("/curated-presets")
|
||||
async def curated_presets():
|
||||
presets = [
|
||||
{
|
||||
"id": "mock_story",
|
||||
"label": "Mock Story",
|
||||
"description": "CI-safe mock story preset.",
|
||||
"segment_prompts": [
|
||||
"A tiny robot opens a glowing door.",
|
||||
"The robot waves at a friendly moon.",
|
||||
],
|
||||
},
|
||||
]
|
||||
return {
|
||||
"presets": presets,
|
||||
"count": len(presets),
|
||||
"file_path": None,
|
||||
"fallback_file_path": None,
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -124,12 +124,21 @@ def test_config_uses_local_overlay_paths_when_devtools_enabled(monkeypatch, tmp_
|
||||
assert module.CURATED_PRESETS_FALLBACK_FILE_PATH.endswith(
|
||||
"apps/dreamverse/web/prompts/selected_ltx2_continuation_story_presets.json"
|
||||
)
|
||||
assert module.FRONTEND_STATIC_DIR_CANDIDATES == (
|
||||
assert module.FRONTEND_STATIC_DIR_CANDIDATES[:2] == (
|
||||
str(module.FRONTEND_ROOT / "out"),
|
||||
str(module.FRONTEND_ROOT / "dist"),
|
||||
)
|
||||
|
||||
|
||||
def test_config_includes_docker_checkout_static_frontend_candidates(monkeypatch):
|
||||
_set_required_prompt_keys(monkeypatch)
|
||||
|
||||
module = _load_config_module()
|
||||
|
||||
assert "/opt/FastVideo/apps/dreamverse/web/out" in module.FRONTEND_STATIC_DIR_CANDIDATES
|
||||
assert "/opt/FastVideo/apps/dreamverse/web/dist" in module.FRONTEND_STATIC_DIR_CANDIDATES
|
||||
|
||||
|
||||
def test_config_enables_prompt_safety_when_requested(monkeypatch):
|
||||
_set_required_prompt_keys(monkeypatch)
|
||||
monkeypatch.setenv("FASTVIDEO_ENABLE_PROMPT_SAFETY", "true")
|
||||
|
||||
@@ -171,3 +171,69 @@ def test_mock_server_cli_updates_latency(monkeypatch):
|
||||
assert mock_server.LATENCY_MS == 321
|
||||
finally:
|
||||
mock_server.LATENCY_MS = old_latency_ms
|
||||
|
||||
|
||||
def _lora_client(monkeypatch, captured):
|
||||
server_main = _import_server_main(monkeypatch)
|
||||
monkeypatch.setattr(server_main, "DEVTOOLS_ENABLED", True)
|
||||
monkeypatch.setattr(server_main, "_available_styles_for_active_model", lambda: ["pixar", "transition"])
|
||||
monkeypatch.setattr(
|
||||
server_main,
|
||||
"_resolve_lora_spec",
|
||||
lambda spec: "FastVideo/OmniNFT" if str(spec).strip().lower() == "omninft" else spec,
|
||||
)
|
||||
|
||||
class _FakePool:
|
||||
|
||||
async def apply_lora_stack(self, stack):
|
||||
captured["stack"] = list(stack)
|
||||
return {0: "trigger"}
|
||||
|
||||
server_main.runtime.gpu_pool = _FakePool()
|
||||
return TestClient(server_main.app)
|
||||
|
||||
|
||||
def test_apply_lora_stacks_multiple_styles_with_intensities(monkeypatch):
|
||||
captured: dict = {}
|
||||
client = _lora_client(monkeypatch, captured)
|
||||
response = client.post("/lora", json={"strength": 0.8, "styles": {"pixar": 0.45, "transition": 0.5}})
|
||||
assert response.status_code == 200
|
||||
body = response.json()
|
||||
assert body["applied"] is True
|
||||
assert body["styles"] == {"pixar": 0.45, "transition": 0.5}
|
||||
assert captured["stack"] == [("omninft", 0.8), ("pixar", 0.45), ("transition", 0.5)]
|
||||
|
||||
|
||||
def test_apply_lora_rejects_unknown_style(monkeypatch):
|
||||
captured: dict = {}
|
||||
client = _lora_client(monkeypatch, captured)
|
||||
response = client.post("/lora", json={"strength": 0.5, "styles": {"watercolor": 0.5}})
|
||||
assert response.status_code == 400
|
||||
assert "stack" not in captured
|
||||
|
||||
|
||||
def test_apply_lora_accepts_legacy_single_style(monkeypatch):
|
||||
captured: dict = {}
|
||||
client = _lora_client(monkeypatch, captured)
|
||||
response = client.post("/lora", json={"strength": 0.6, "style": "pixar"})
|
||||
assert response.status_code == 200
|
||||
assert captured["stack"] == [("omninft", 0.6), ("pixar", 1.0)]
|
||||
|
||||
|
||||
def test_apply_lora_clamps_and_drops_zero_intensity(monkeypatch):
|
||||
captured: dict = {}
|
||||
client = _lora_client(monkeypatch, captured)
|
||||
response = client.post("/lora", json={"strength": 1.5, "styles": {"pixar": 2.0, "transition": 0.0}})
|
||||
assert response.status_code == 200
|
||||
body = response.json()
|
||||
assert body["strength"] == 1.0
|
||||
assert body["styles"] == {"pixar": 1.0}
|
||||
assert captured["stack"] == [("omninft", 1.0), ("pixar", 1.0)]
|
||||
|
||||
|
||||
def test_lora_endpoints_hidden_without_devtools(monkeypatch):
|
||||
server_main = _import_server_main(monkeypatch)
|
||||
monkeypatch.setattr(server_main, "DEVTOOLS_ENABLED", False)
|
||||
client = TestClient(server_main.app)
|
||||
assert client.get("/lora/options").status_code == 404
|
||||
assert client.post("/lora", json={"strength": 0.5}).status_code == 404
|
||||
|
||||
@@ -20,9 +20,11 @@ os.environ.setdefault("GROQ_API_KEY", "dummy")
|
||||
|
||||
import dreamverse.main as server_main # noqa: E402
|
||||
import dreamverse.runtime as runtime # noqa: E402
|
||||
from dreamverse.config import PROMPT_TIMEOUT_MS # noqa: E402
|
||||
from dreamverse.session_logger import SessionEventLogger # noqa: E402
|
||||
from dreamverse.worker_ipc import MediaChunk, MediaComplete, MediaInit # noqa: E402
|
||||
from dreamverse.session import controller as session_controller # noqa: E402
|
||||
from dreamverse.utils import PROMPT_EXTENSION_FAILURE_USER_MESSAGE # noqa: E402
|
||||
|
||||
pytestmark = pytest.mark.gpu
|
||||
|
||||
@@ -338,7 +340,7 @@ def test_session_event_logger_initializes_hostname_folder_and_utc_filename(
|
||||
logger = SessionEventLogger(tmp_path)
|
||||
assert logger.directory == tmp_path / socket.gethostname()
|
||||
assert logger.directory.is_dir()
|
||||
assert re.match(r"^\d{6}_\d{6}\.jsonl$", logger.path.name)
|
||||
assert re.match(r"^\d{6}_\d{6}_\d{6}\.jsonl$", logger.path.name)
|
||||
assert logger.path.is_file()
|
||||
|
||||
|
||||
@@ -833,7 +835,7 @@ def test_initial_custom_rollout_prompt_generates_seed_window_before_streaming():
|
||||
"rewrite_instruction": "A moonbase corridor thriller with flooding",
|
||||
"rewrite_model": "gpt-4.1-mini",
|
||||
"rewrite_temperature": 0.4,
|
||||
"timeout_ms": server_main.PROMPT_TIMEOUT_MS,
|
||||
"timeout_ms": PROMPT_TIMEOUT_MS,
|
||||
"system_prompt_override": "Session-specific rewrite prompt",
|
||||
}
|
||||
assert fake_pool._slot.calls
|
||||
@@ -1256,7 +1258,7 @@ def test_single5s_enhancement_fallback_does_not_start_generation():
|
||||
assert fallback_payloads[0]["prompt_id"] == "simple_custom_prompt"
|
||||
assert (
|
||||
fallback_payloads[0]["error"]
|
||||
== server_main.PROMPT_EXTENSION_FAILURE_USER_MESSAGE
|
||||
== PROMPT_EXTENSION_FAILURE_USER_MESSAGE
|
||||
)
|
||||
assert fallback_payloads[0]["source"] == "user_enhancement_failed"
|
||||
|
||||
|
||||
@@ -12,6 +12,7 @@ module import time.
|
||||
# mypy: ignore-errors
|
||||
import gc
|
||||
import os
|
||||
import re
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
@@ -20,11 +21,19 @@ import numpy as np
|
||||
import torch
|
||||
|
||||
from dreamverse.config import (
|
||||
AVAILABLE_LORAS,
|
||||
FRAME_HEIGHT,
|
||||
FRAME_WIDTH,
|
||||
MODEL_CONFIG,
|
||||
NUM_FRAMES,
|
||||
NUM_INFERENCE_STEPS,
|
||||
DREAMVERSE_MAX_AUTOTUNE,
|
||||
DREAMVERSE_SP_SIZE,
|
||||
DREAMVERSE_LORA_PATH,
|
||||
DREAMVERSE_LORA_NICKNAME,
|
||||
DREAMVERSE_LORA_STRENGTH,
|
||||
DREAMVERSE_LORA_STACK,
|
||||
_resolve_lora_spec,
|
||||
)
|
||||
|
||||
# Multi-frame decoded continuation defaults from
|
||||
@@ -62,6 +71,15 @@ DEFAULT_LTX2_AUDIO_HOP_LENGTH = 160
|
||||
DEFAULT_LTX2_AUDIO_DOWNSAMPLE = 4
|
||||
|
||||
|
||||
def _reset_lora_registry(worker) -> dict:
|
||||
pipeline = getattr(worker, "pipeline", None)
|
||||
if pipeline is None:
|
||||
return {"status": "no_pipeline"}
|
||||
pipeline.cur_adapter_name = ""
|
||||
pipeline.cur_adapter_strength = 1.0
|
||||
return {"status": "lora_registry_reset"}
|
||||
|
||||
|
||||
@dataclass
|
||||
class StepResult:
|
||||
"""Output of one generation step.
|
||||
@@ -199,6 +217,8 @@ class VideoGenerationWorker:
|
||||
self.continuation = ContinuationState()
|
||||
self.audio_encoder_module = None
|
||||
self.audio_processor_module = None
|
||||
self.active_style_prepend: list[str] = []
|
||||
self.active_style_append: list[str] = []
|
||||
|
||||
def _gpu_mem(self) -> str:
|
||||
a = torch.cuda.memory_allocated() / 1024**3
|
||||
@@ -257,6 +277,7 @@ class VideoGenerationWorker:
|
||||
or self.current_model_config["model_path"])
|
||||
|
||||
enable_compile = os.getenv("ENABLE_TORCH_COMPILE", "1") == "1"
|
||||
compile_mode = "max-autotune-no-cudagraphs" if DREAMVERSE_MAX_AUTOTUNE else None
|
||||
|
||||
components = ComponentConfig(
|
||||
config_root=config_model_path,
|
||||
@@ -269,7 +290,7 @@ class VideoGenerationWorker:
|
||||
generator_config = GeneratorConfig(
|
||||
model_path=model_root,
|
||||
engine=EngineConfig(
|
||||
num_gpus=1,
|
||||
num_gpus=DREAMVERSE_SP_SIZE,
|
||||
offload=OffloadConfig(
|
||||
dit=False,
|
||||
dit_layerwise=False,
|
||||
@@ -280,9 +301,10 @@ class VideoGenerationWorker:
|
||||
compile=CompileConfig(
|
||||
enabled=enable_compile,
|
||||
text_encoder_enabled=enable_compile,
|
||||
vae_enabled=enable_compile,
|
||||
backend="inductor",
|
||||
fullgraph=True,
|
||||
mode="max-autotune-no-cudagraphs",
|
||||
mode=compile_mode,
|
||||
dynamic=False,
|
||||
),
|
||||
use_fsdp_inference=False,
|
||||
@@ -304,9 +326,88 @@ class VideoGenerationWorker:
|
||||
|
||||
self.generator = VideoGenerator.from_pretrained(config=generator_config)
|
||||
print(f"[GPU {self.gpu_id}] After model load: {self._gpu_mem()}")
|
||||
|
||||
lora_stack = DREAMVERSE_LORA_STACK or ([(DREAMVERSE_LORA_PATH,
|
||||
DREAMVERSE_LORA_STRENGTH)] if DREAMVERSE_LORA_PATH else [])
|
||||
for i, (lora_path, lora_strength) in enumerate(lora_stack):
|
||||
nickname = DREAMVERSE_LORA_NICKNAME if i == 0 else f"{DREAMVERSE_LORA_NICKNAME}_{i}"
|
||||
print(f"[GPU {self.gpu_id}] Applying LoRA '{nickname}' from {lora_path} "
|
||||
f"@{lora_strength} accumulate={i > 0}")
|
||||
self.generator.set_lora_adapter(
|
||||
lora_nickname=nickname,
|
||||
lora_path=lora_path,
|
||||
strength=lora_strength,
|
||||
accumulate=(i > 0),
|
||||
)
|
||||
if lora_stack:
|
||||
print(f"[GPU {self.gpu_id}] LoRA stack applied ({len(lora_stack)})")
|
||||
|
||||
self._load_audio_encoder(model_root)
|
||||
print(f"[GPU {self.gpu_id}] LTX2 model loaded (warmup pending)")
|
||||
|
||||
def apply_lora_stack(
|
||||
self,
|
||||
stack: list[tuple[str, float]],
|
||||
) -> tuple[str | None, str | None]:
|
||||
"""Re-apply a runtime LoRA stack and update the active style trigger."""
|
||||
if self.generator is None:
|
||||
raise RuntimeError("Generator not initialized; cannot apply LoRA stack.")
|
||||
|
||||
try:
|
||||
self.generator.unmerge_lora_weights()
|
||||
except Exception as exc:
|
||||
print(f"[GPU {self.gpu_id}] unmerge_lora_weights skipped: {exc}")
|
||||
|
||||
try:
|
||||
self.generator.executor.collective_rpc(_reset_lora_registry)
|
||||
except Exception as exc:
|
||||
print(f"[GPU {self.gpu_id}] lora registry reset skipped: {exc}")
|
||||
|
||||
resolved_stack: list[tuple[str, float]] = []
|
||||
for spec, strength in stack:
|
||||
resolved = _resolve_lora_spec(spec)
|
||||
if resolved:
|
||||
resolved_stack.append((resolved, float(strength)))
|
||||
|
||||
for i, (lora_path, lora_strength) in enumerate(resolved_stack):
|
||||
nickname = DREAMVERSE_LORA_NICKNAME if i == 0 else f"{DREAMVERSE_LORA_NICKNAME}_{i}"
|
||||
print(f"[GPU {self.gpu_id}] Re-applying LoRA '{nickname}' from {lora_path} "
|
||||
f"@{lora_strength} accumulate={i > 0}")
|
||||
self.generator.set_lora_adapter(
|
||||
lora_nickname=nickname,
|
||||
lora_path=lora_path,
|
||||
strength=lora_strength,
|
||||
accumulate=(i > 0),
|
||||
)
|
||||
print(f"[GPU {self.gpu_id}] Runtime LoRA stack applied ({len(resolved_stack)})")
|
||||
|
||||
prepend: list[str] = []
|
||||
append: list[str] = []
|
||||
for spec, _ in stack:
|
||||
key = (spec or "").strip().lower()
|
||||
if key in AVAILABLE_LORAS:
|
||||
trigger = AVAILABLE_LORAS[key]["trigger"]
|
||||
if AVAILABLE_LORAS[key].get("position", "append") == "prepend":
|
||||
prepend.append(trigger)
|
||||
else:
|
||||
append.append(trigger)
|
||||
self.active_style_prepend = prepend
|
||||
self.active_style_append = append
|
||||
combined = " ".join(prepend + append) or None
|
||||
return combined, None
|
||||
|
||||
def _inject_style_trigger(self, prompt: str) -> str:
|
||||
result = prompt
|
||||
for trigger in self.active_style_prepend:
|
||||
trigger = (trigger or "").strip()
|
||||
if trigger and not re.search(rf"\b{re.escape(trigger)}\b", result, re.IGNORECASE):
|
||||
result = f"{trigger} {result}".strip()
|
||||
for trigger in self.active_style_append:
|
||||
trigger = (trigger or "").strip()
|
||||
if trigger and not re.search(rf"\b{re.escape(trigger)}\b", result, re.IGNORECASE):
|
||||
result = f"{result} {trigger}".strip()
|
||||
return result
|
||||
|
||||
def _load_audio_encoder(self, model_root: str) -> None:
|
||||
if not ENABLE_AUDIO_RE_ENCODE:
|
||||
return
|
||||
@@ -375,6 +476,8 @@ class VideoGenerationWorker:
|
||||
"""Execute one generation step; snapshot state for the next segment."""
|
||||
timings: dict = {}
|
||||
|
||||
prompt = self._inject_style_trigger(prompt)
|
||||
|
||||
request_kwargs = dict(
|
||||
prompt=prompt,
|
||||
negative_prompt="",
|
||||
|
||||
@@ -48,6 +48,13 @@ class ReloadAck:
|
||||
user_id: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class LoraAck:
|
||||
user_id: str | None
|
||||
style_trigger: str | None
|
||||
style_trigger_position: str | None
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class WarmupComplete:
|
||||
user_id: str | None
|
||||
@@ -117,6 +124,7 @@ WorkerEvent = (StepComplete
|
||||
| JoinAck
|
||||
| LeaveAck
|
||||
| ReloadAck
|
||||
| LoraAck
|
||||
| WarmupComplete
|
||||
| MediaInit
|
||||
| MediaChunk
|
||||
@@ -154,4 +162,9 @@ class ReloadModelPayload:
|
||||
model_config: dict
|
||||
|
||||
|
||||
CommandPayload = UserStepPayload | WarmupPayload | ReloadModelPayload
|
||||
@dataclass(frozen=True)
|
||||
class LoraStackPayload:
|
||||
stack: list[tuple[str, float]]
|
||||
|
||||
|
||||
CommandPayload = (UserStepPayload | WarmupPayload | ReloadModelPayload | LoraStackPayload)
|
||||
|
||||
@@ -28,7 +28,7 @@ NO_BROWSER=1 apps/dreamverse/scripts/launch/launch_demo.sh
|
||||
`launch_backend_dreamverse.sh` starts the full Dreamverse backend path used by
|
||||
the web app.
|
||||
|
||||
`launch_frontend.sh` starts the Next.js frontend and installs `pnpm`
|
||||
`launch_frontend.sh` starts the Next.js frontend and installs npm
|
||||
dependencies when `node_modules/` is missing.
|
||||
|
||||
`launch_backend_fastvideo.sh` starts the typed `fastvideo serve --config` path
|
||||
|
||||
@@ -7,8 +7,8 @@
|
||||
# FRONTEND_MODE=dev bash launch_frontend.sh # plain dev (5299, no devtools)
|
||||
# FRONTEND_MODE=single5s bash launch_frontend.sh # single-5s product mode
|
||||
#
|
||||
# The script ``cd``'s into ``web`` and shells out to pnpm. It runs
|
||||
# ``pnpm install --frozen-lockfile`` only when ``node_modules/`` is missing so repeat
|
||||
# The script ``cd``'s into ``web`` and shells out to npm. It runs
|
||||
# ``npm ci`` only when ``node_modules/`` is missing so repeat
|
||||
# launches are fast.
|
||||
|
||||
set -euo pipefail
|
||||
@@ -34,21 +34,21 @@ fi
|
||||
cd "${WEB_ROOT}"
|
||||
|
||||
if [[ ! -d node_modules ]]; then
|
||||
echo "[launch-demo] node_modules missing — running pnpm install --frozen-lockfile"
|
||||
pnpm install --frozen-lockfile
|
||||
echo "[launch-demo] node_modules missing — running npm ci"
|
||||
npm ci
|
||||
fi
|
||||
|
||||
case "${FRONTEND_MODE}" in
|
||||
devtools)
|
||||
echo "[launch-demo] starting Next.js dev:devtools (port 5274)"
|
||||
exec pnpm run dev:devtools -- "$@"
|
||||
exec npm run dev:devtools -- "$@"
|
||||
;;
|
||||
dev)
|
||||
echo "[launch-demo] starting Next.js dev (port 5299)"
|
||||
exec pnpm run dev -- "$@"
|
||||
exec npm run dev -- "$@"
|
||||
;;
|
||||
single5s)
|
||||
echo "[launch-demo] starting Next.js dev:single5s (port 5274)"
|
||||
exec pnpm run dev:single5s -- "$@"
|
||||
exec npm run dev:single5s -- "$@"
|
||||
;;
|
||||
esac
|
||||
|
||||
@@ -0,0 +1,127 @@
|
||||
# Dreamverse Modal deployment
|
||||
|
||||
This directory contains the Modal wrapper for running the registry-built
|
||||
Dreamverse Docker image on a Modal B200 container. The wrapper pulls a prebuilt Docker image and exposes the dreamverse application running inside.
|
||||
|
||||
|
||||
## 1. Install Modal CLI
|
||||
|
||||
Install and configure the Modal CLI for the target workspace/profile:
|
||||
|
||||
```bash
|
||||
pip install modal
|
||||
modal token set --token-id <token-id> --token-secret <token-secret> --profile=<profile-name>
|
||||
modal profile activate <profile-name>
|
||||
```
|
||||
|
||||
## 2. Create the API key secret
|
||||
|
||||
Dreamverse needs several API keys for prompt-rewriter LLMs and model access in the Modal secret named
|
||||
`dreamverse-api-keys`. Create or replace it with placeholders like this:
|
||||
|
||||
```bash
|
||||
modal secret create dreamverse-api-keys \
|
||||
CEREBRAS_API_KEY=<your-cerebras-key> \
|
||||
GROQ_API_KEY=<your-groq-key> \
|
||||
HF_TOKEN=<your-huggingface-token> \
|
||||
--force
|
||||
```
|
||||
|
||||
## 3. Deploy with Docker image
|
||||
|
||||
`DREAMVERSE_IMAGE` is required at deploy time:
|
||||
|
||||
```bash
|
||||
DREAMVERSE_IMAGE=ghcr.io/<org>/<repo>/dreamverse:<tag> \
|
||||
modal deploy apps/dreamverse/scripts/modal/modal_app.py
|
||||
```
|
||||
|
||||
Use a SHA-specific tag, not `latest`. Use a `dreamverse-backend-cuda12.9.1-sha-*`
|
||||
tag for backend-only deploys, or a `dreamverse-ui-cuda12.9.1-sha-*` tag for an
|
||||
image that includes the static UI served by the backend.
|
||||
|
||||
For local image build details, see
|
||||
`apps/dreamverse/docker/README.md` and `apps/dreamverse/docker/docker_build.sh`.
|
||||
|
||||
|
||||
## 4. Validate the deployment
|
||||
|
||||
Set the URL returned by `modal deploy`, then probe the backend:
|
||||
|
||||
```bash
|
||||
URL=https://<workspace>--dreamverse-b200-serve.modal.run
|
||||
curl -fsS --max-time 300 "$URL/healthz"
|
||||
curl -fsS --max-time 300 "$URL/status"
|
||||
curl -fsS --max-time 300 "$URL/readyz"
|
||||
```
|
||||
|
||||
Check logs with function and container IDs so you can confirm only one container
|
||||
is active:
|
||||
|
||||
```bash
|
||||
modal app logs dreamverse-b200 \
|
||||
--tail 200 \
|
||||
--timestamps \
|
||||
--show-function-id \
|
||||
--show-container-id
|
||||
```
|
||||
|
||||
Useful Modal commands:
|
||||
|
||||
```bash
|
||||
modal app list
|
||||
modal app list --json
|
||||
modal billing report --for today --resolution h
|
||||
modal app stop dreamverse-b200
|
||||
```
|
||||
|
||||
## Runtime details
|
||||
|
||||
### Volumes and assumed locations
|
||||
|
||||
`modal_app.py` creates the named volumes if they do not exist and mounts them at
|
||||
the locations expected by the image:
|
||||
|
||||
| Modal volume | Container path | Environment variable | Purpose |
|
||||
| --- | --- | --- | --- |
|
||||
| `dreamverse-hf-cache` | `/root/.cache/huggingface` | `HF_HOME` | Hugging Face model/cache data |
|
||||
| `dreamverse-state` | `/var/lib/dreamverse` | `FASTVIDEO_DREAMVERSE_HOME` | Dreamverse outputs, session logs, and runtime state |
|
||||
|
||||
### Useful dev env vars
|
||||
|
||||
Most deploys only need the Modal secret and required `DREAMVERSE_IMAGE`.
|
||||
`DREAMVERSE_IMAGE` is read from your local shell at deploy time; the runtime
|
||||
knobs below live in the image or `modal_app.py` unless you intentionally change them:
|
||||
|
||||
- `DREAMVERSE_IMAGE`: deploy a prebuilt SHA-specific image tag without editing `modal_app.py`.
|
||||
- `FASTVIDEO_ENABLE_DEVTOOLS`: enable Dreamverse devtools behavior.
|
||||
- `FASTVIDEO_PROMPT_PROVIDER` / `FASTVIDEO_PROMPT_*_MODEL`: try prompt-rewriter provider or model choices.
|
||||
- `FASTVIDEO_ENABLE_STARTUP_WARMUP`: trade slower startup for a warmer first request.
|
||||
- `DREAMVERSE_MAX_AUTOTUNE`: enable or disable PyTorch Inductor max-autotune for the compiled Dreamverse runtime.
|
||||
|
||||
The Modal wrapper defaults to torch compile with Inductor max-autotune enabled
|
||||
for the fastest generation after startup warmup. If you need shorter
|
||||
compile/warmup time, disable max-autotune via:
|
||||
|
||||
```bash
|
||||
DREAMVERSE_IMAGE=ghcr.io/<org>/<repo>/dreamverse:<tag> \
|
||||
DREAMVERSE_MAX_AUTOTUNE=0 \
|
||||
modal deploy apps/dreamverse/scripts/modal/modal_app.py
|
||||
```
|
||||
|
||||
### Autoscaling and cost safety
|
||||
|
||||
The current deployed script keeps exactly one B200 container warm by setting
|
||||
`min_containers=1` and `max_containers=1`. `min_containers=1` prevents Modal
|
||||
from scaling the deployment down to zero after idle periods, which avoids paying
|
||||
the expensive torch-compile/startup-warmup cost again on the next request.
|
||||
`max_containers=1` caps concurrency so concurrent requests queue onto the warm
|
||||
container instead of spawning additional B200 containers and multiplying cost.
|
||||
|
||||
### Common gotchas
|
||||
|
||||
- Modal profile/workspace matters: create the secret and deploy with the same active profile.
|
||||
- `modal secret create ... --force` replaces the existing `dreamverse-api-keys` secret.
|
||||
- Use the exact URL printed by `modal deploy`; the placeholder URL above is only an example shape.
|
||||
- B200 cold starts and first model downloads can make `/readyz` slow. Check logs before redeploying.
|
||||
- `modal app stop dreamverse-b200` stops the app; it is not a read-only inspection command.
|
||||
@@ -0,0 +1,77 @@
|
||||
# pyright: reportAttributeAccessIssue=false
|
||||
"""Modal deployment entrypoint for the Dreamverse B200 backend."""
|
||||
|
||||
import os
|
||||
import subprocess
|
||||
|
||||
import modal
|
||||
|
||||
IMAGE = os.environ.get("DREAMVERSE_IMAGE")
|
||||
if not IMAGE:
|
||||
raise RuntimeError(
|
||||
"DREAMVERSE_IMAGE is required. Set it to a published SHA-specific Dreamverse image, "
|
||||
"for example a dreamverse-backend-cuda12.9.1-sha-* tag or a "
|
||||
"dreamverse-ui-cuda12.9.1-sha-* tag if serving the static UI."
|
||||
)
|
||||
|
||||
# ``@modal.web_server`` invokes ``serve()`` directly and bypasses the image
|
||||
# ENTRYPOINT (``docker/docker_entrypoint.sh``). That entrypoint normally
|
||||
# ``:?``-validates these secret entries and ``source``-s ``ffmpeg-env.sh``;
|
||||
# neither runs on Modal, so we replicate both here.
|
||||
_REQUIRED_SECRET_KEYS: tuple[str, ...] = ("CEREBRAS_API_KEY", "GROQ_API_KEY")
|
||||
|
||||
image = modal.Image.from_registry(IMAGE)
|
||||
image = image.env({
|
||||
"DREAMVERSE_IMAGE": IMAGE,
|
||||
"HF_HOME": "/root/.cache/huggingface",
|
||||
"FASTVIDEO_DREAMVERSE_HOME": "/var/lib/dreamverse",
|
||||
"FASTVIDEO_ENABLE_STARTUP_WARMUP": "1",
|
||||
"FASTVIDEO_GPU_COUNT": "1",
|
||||
"ENABLE_TORCH_COMPILE": "1",
|
||||
"DREAMVERSE_MAX_AUTOTUNE": os.environ.get("DREAMVERSE_MAX_AUTOTUNE", "1"),
|
||||
"STREAM_MODE": "av_fmp4",
|
||||
# Native ffmpeg is built into ``/opt/ffmpeg-native`` by the Dockerfile but
|
||||
# is not on the image's ``PATH``. Without these, ``av_streaming``'s
|
||||
# ``shutil.which("ffmpeg")`` returns ``None`` and ``av_fmp4`` muxing has
|
||||
# no encoder. ``ffmpeg-env.sh`` would normally set them.
|
||||
"FASTVIDEO_FFMPEG_BIN": "/opt/ffmpeg-native/bin/ffmpeg",
|
||||
"FASTVIDEO_VIDEO_CODEC": "libx264",
|
||||
})
|
||||
|
||||
app = modal.App("dreamverse-b200")
|
||||
|
||||
hf_cache = modal.Volume.from_name("dreamverse-hf-cache", create_if_missing=True)
|
||||
dreamverse_state = modal.Volume.from_name("dreamverse-state", create_if_missing=True)
|
||||
|
||||
|
||||
@app.function(
|
||||
image=image,
|
||||
gpu="B200",
|
||||
cpu=16,
|
||||
memory=65536,
|
||||
timeout=7200,
|
||||
startup_timeout=4800,
|
||||
min_containers=1,
|
||||
max_containers=1,
|
||||
secrets=[modal.Secret.from_name("dreamverse-api-keys")],
|
||||
volumes={
|
||||
"/root/.cache/huggingface": hf_cache,
|
||||
"/var/lib/dreamverse": dreamverse_state,
|
||||
},
|
||||
)
|
||||
@modal.web_server(8009, startup_timeout=4800)
|
||||
def serve():
|
||||
# ``or ""`` collapses ``None`` (unset) into an empty string, ``.strip()``
|
||||
# collapses whitespace-only values (e.g. ``" "``) — both should be
|
||||
# treated as missing.
|
||||
missing = [
|
||||
k for k in _REQUIRED_SECRET_KEYS
|
||||
if not (os.environ.get(k) or "").strip()
|
||||
]
|
||||
if missing:
|
||||
raise RuntimeError(
|
||||
"dreamverse-api-keys secret is missing required entries: "
|
||||
f"{', '.join(missing)}. Add them with `modal secret create "
|
||||
"dreamverse-api-keys ... --force` and redeploy "
|
||||
"(see apps/dreamverse/scripts/modal/README.md).")
|
||||
subprocess.Popen(["dreamverse-server", "--host", "0.0.0.0", "--port", "8009"])
|
||||
@@ -87,9 +87,9 @@ if [[ "${started_backend}" == "1" ]]; then
|
||||
echo "Backend log: ${BACKEND_LOG_PATH}"
|
||||
echo "Backend is still running so you can launch the frontend:"
|
||||
echo " cd ${ROOT_DIR}/web"
|
||||
echo " BACKEND_HOST=${HOST} BACKEND_PORT=${PORT} pnpm run dev"
|
||||
echo " BACKEND_HOST=${HOST} BACKEND_PORT=${PORT} npm run dev"
|
||||
else
|
||||
echo "You can now launch the frontend:"
|
||||
echo " cd ${ROOT_DIR}/web"
|
||||
echo " BACKEND_HOST=${HOST} BACKEND_PORT=${PORT} pnpm run dev"
|
||||
echo " BACKEND_HOST=${HOST} BACKEND_PORT=${PORT} npm run dev"
|
||||
fi
|
||||
|
||||
@@ -11,7 +11,7 @@ test.describe('backend health', () => {
|
||||
expect(response.ok()).toBeTruthy();
|
||||
const body = await response.json();
|
||||
expect(body.status).toBe('ok');
|
||||
expect(body.service).toBe('ltx2-streaming-backend');
|
||||
expect(['ltx2-streaming-backend', 'ltx2-streaming-mock-server']).toContain(body.service);
|
||||
});
|
||||
|
||||
test('readyz reports gpu pool state', async ({ request }) => {
|
||||
|
||||
@@ -18,7 +18,7 @@ import { test, expect, type WebSocket as PWWebSocket } from '@playwright/test';
|
||||
* PLAYWRIGHT_BASE_URL=http://127.0.0.1:5274 \
|
||||
* NEXT_PUBLIC_INCLUDE_DEVTOOLS=1 \
|
||||
* PLAYWRIGHT_LONG_RUNNING=1 \
|
||||
* pnpm exec playwright test e2e/long-running-segments.spec.ts
|
||||
* npm exec -- playwright test e2e/long-running-segments.spec.ts
|
||||
*
|
||||
* The full run takes ~7-9 minutes on a B200: ~3-4min for torch.compile
|
||||
* max-autotune to warm both DiT + text-encoder graphs, then ~30s for
|
||||
|
||||
@@ -0,0 +1,311 @@
|
||||
import { stat } from 'node:fs/promises';
|
||||
|
||||
import { test, expect, type WebSocket as PWWebSocket } from '@playwright/test';
|
||||
|
||||
test.describe('mock-backed generation smoke', () => {
|
||||
test('accepts a custom prompt and enters the streaming state', async ({ page, request }) => {
|
||||
const health = await request.get('/healthz');
|
||||
const body = health.ok() ? await health.json() : {};
|
||||
test.skip(
|
||||
body.service !== 'ltx2-streaming-mock-server',
|
||||
'Mock-backed smoke requires dreamverse.mock_server on BACKEND_PORT (default 8009).',
|
||||
);
|
||||
|
||||
const wsEventTypes: string[] = [];
|
||||
|
||||
page.on('websocket', (ws: PWWebSocket) => {
|
||||
ws.on('framereceived', ({ payload }) => {
|
||||
if (typeof payload !== 'string') return;
|
||||
try {
|
||||
const parsed = JSON.parse(payload);
|
||||
if (typeof parsed?.type === 'string') wsEventTypes.push(parsed.type);
|
||||
} catch {
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
await page.goto('/');
|
||||
|
||||
await expect(page.getByRole('img', { name: 'FastVideo' })).toBeVisible();
|
||||
|
||||
const continuation = page.getByLabel('Continuation prompt');
|
||||
await expect(continuation).toBeVisible();
|
||||
await continuation.fill('A glass whale swims above a neon forest');
|
||||
|
||||
await page.getByRole('button', { name: /^generate$/i }).click();
|
||||
|
||||
await expect(continuation).toBeDisabled({ timeout: 30_000 });
|
||||
await expect(continuation).toHaveAttribute('placeholder', /generating video/i);
|
||||
await expect(page.getByRole('button', { name: /^leave$/i })).toBeVisible({ timeout: 30_000 });
|
||||
await expect(page.locator('video').first()).toHaveCount(1);
|
||||
|
||||
await expect.poll(() => wsEventTypes).toEqual(
|
||||
expect.arrayContaining(['ltx2_stream_start', 'ltx2_segment_start', 'media_init']),
|
||||
);
|
||||
});
|
||||
|
||||
test('streams, plays, and surfaces a downloadable clip', async ({ page, request }) => {
|
||||
const health = await request.get('/healthz');
|
||||
const body = health.ok() ? await health.json() : {};
|
||||
test.skip(
|
||||
body.service !== 'ltx2-streaming-mock-server',
|
||||
'Mock-backed playback assertions require dreamverse.mock_server on BACKEND_PORT (default 8009).',
|
||||
);
|
||||
|
||||
await page.addInitScript(() => {
|
||||
(window as unknown as { __sharedFiles: unknown }).__sharedFiles = null;
|
||||
const stub = async (data: { files?: File[] }) => {
|
||||
const files = Array.isArray(data?.files) ? data.files : [];
|
||||
(window as unknown as { __sharedFiles: unknown }).__sharedFiles = files.map((f) => ({
|
||||
name: f.name,
|
||||
type: f.type,
|
||||
size: f.size,
|
||||
}));
|
||||
};
|
||||
Object.defineProperty(Navigator.prototype, 'share', {
|
||||
value: stub,
|
||||
configurable: true,
|
||||
writable: true,
|
||||
});
|
||||
Object.defineProperty(Navigator.prototype, 'canShare', {
|
||||
value: (data: { files?: File[] }) => Array.isArray(data?.files),
|
||||
configurable: true,
|
||||
writable: true,
|
||||
});
|
||||
});
|
||||
|
||||
await page.goto('/');
|
||||
|
||||
const continuation = page.getByLabel('Continuation prompt');
|
||||
await expect(continuation).toBeVisible();
|
||||
await continuation.fill('A glass whale swims above a neon forest');
|
||||
await page.getByRole('button', { name: /^generate$/i }).click();
|
||||
|
||||
const liveVideo = page.locator('video:not(.hidden)');
|
||||
await expect(liveVideo).toHaveCount(1);
|
||||
|
||||
await test.step('MSE pipeline attaches the live <video>', async () => {
|
||||
// Chromium/Firefox attach via blob: URL; Safari uses srcObject with ManagedMediaSource.
|
||||
await expect
|
||||
.poll(
|
||||
async () =>
|
||||
liveVideo.evaluate(
|
||||
(v: HTMLVideoElement) =>
|
||||
v.src.startsWith('blob:') || v.srcObject !== null,
|
||||
),
|
||||
{ timeout: 30_000 },
|
||||
)
|
||||
.toBe(true);
|
||||
});
|
||||
|
||||
await test.step('SourceBuffer accepts fMP4 and decoder produces frames', async () => {
|
||||
await expect
|
||||
.poll(
|
||||
async () =>
|
||||
liveVideo.evaluate((v: HTMLVideoElement) =>
|
||||
v.buffered.length > 0 ? v.buffered.end(0) : 0,
|
||||
),
|
||||
{ timeout: 60_000 },
|
||||
)
|
||||
.toBeGreaterThan(0);
|
||||
|
||||
// readyState >= 3 = HAVE_FUTURE_DATA.
|
||||
await expect
|
||||
.poll(
|
||||
async () => liveVideo.evaluate((v: HTMLVideoElement) => v.readyState),
|
||||
{ timeout: 60_000 },
|
||||
)
|
||||
.toBeGreaterThanOrEqual(3);
|
||||
});
|
||||
|
||||
await test.step('<video> playback advances past the first second', async () => {
|
||||
await liveVideo.evaluate(async (v: HTMLVideoElement) => {
|
||||
if (v.paused) {
|
||||
try {
|
||||
await v.play();
|
||||
} catch {
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
await expect
|
||||
.poll(
|
||||
async () => liveVideo.evaluate((v: HTMLVideoElement) => v.currentTime),
|
||||
{ timeout: 30_000 },
|
||||
)
|
||||
.toBeGreaterThan(0);
|
||||
|
||||
await expect
|
||||
.poll(
|
||||
async () =>
|
||||
liveVideo.evaluate((v: HTMLVideoElement) =>
|
||||
v.ended ? v.duration : v.currentTime,
|
||||
),
|
||||
{ timeout: 10_000 },
|
||||
)
|
||||
.toBeGreaterThanOrEqual(1.0);
|
||||
});
|
||||
|
||||
await test.step('the AAC audio track is demuxed and decoded', async () => {
|
||||
// The mock emits AAC (mp4a.40.2), matching the real backend
|
||||
// (av_streaming.py). Proves the FE decoded the audio track.
|
||||
await expect
|
||||
.poll(
|
||||
async () =>
|
||||
liveVideo.evaluate((el: HTMLVideoElement) => {
|
||||
const v = el as HTMLVideoElement & {
|
||||
audioTracks?: { length: number };
|
||||
mozHasAudio?: boolean;
|
||||
webkitAudioDecodedByteCount?: number;
|
||||
};
|
||||
return (
|
||||
(v.audioTracks?.length ?? 0) > 0 ||
|
||||
v.mozHasAudio === true ||
|
||||
(v.webkitAudioDecodedByteCount ?? 0) > 0
|
||||
);
|
||||
}),
|
||||
{ timeout: 10_000 },
|
||||
)
|
||||
.toBe(true);
|
||||
});
|
||||
|
||||
await test.step('completed clip surfaces a working download', async () => {
|
||||
const downloadButton = page.getByRole('button', { name: /download video|share video/i });
|
||||
await expect(downloadButton).toBeVisible({ timeout: 30_000 });
|
||||
|
||||
const buttonLabel = (await downloadButton.getAttribute('aria-label')) ?? '';
|
||||
const isShareFlow = /share video/i.test(buttonLabel);
|
||||
|
||||
if (isShareFlow) {
|
||||
await downloadButton.click();
|
||||
await expect
|
||||
.poll(
|
||||
async () => page.evaluate(() => (window as { __sharedFiles?: unknown }).__sharedFiles),
|
||||
{ timeout: 10_000 },
|
||||
)
|
||||
.not.toBeNull();
|
||||
const shared = (await page.evaluate(
|
||||
() => (window as { __sharedFiles?: Array<{ name: string; type: string; size: number }> }).__sharedFiles,
|
||||
)) ?? [];
|
||||
expect(shared).toHaveLength(1);
|
||||
expect(shared[0].name).toMatch(/\.(mp4|webm)$/);
|
||||
expect(shared[0].size).toBeGreaterThan(0);
|
||||
} else {
|
||||
const downloadPromise = page.waitForEvent('download');
|
||||
await downloadButton.click();
|
||||
const download = await downloadPromise;
|
||||
expect(download.suggestedFilename()).toMatch(/\.(mp4|webm)$/);
|
||||
|
||||
const savedPath = await download.path();
|
||||
expect(savedPath).not.toBeNull();
|
||||
const { size } = await stat(savedPath!);
|
||||
expect(size).toBeGreaterThan(0);
|
||||
}
|
||||
});
|
||||
|
||||
await test.step('project history sidebar lists the current session', async () => {
|
||||
const sidebar = page.getByRole('complementary', { name: 'Project history' });
|
||||
await page.getByRole('button', { name: 'Toggle sidebar' }).click();
|
||||
|
||||
await expect(sidebar).toBeInViewport();
|
||||
|
||||
await expect(sidebar.getByText('Current', { exact: true })).toBeVisible();
|
||||
await expect(sidebar.getByText('Active', { exact: true })).toBeVisible();
|
||||
});
|
||||
|
||||
const lastError = await liveVideo.evaluate((v: HTMLVideoElement) =>
|
||||
v.error ? `${v.error.code}: ${v.error.message ?? ''}` : null,
|
||||
);
|
||||
expect(lastError).toBeNull();
|
||||
});
|
||||
|
||||
test('starts a new project and switches back to the prior session', async ({ page, request }) => {
|
||||
const health = await request.get('/healthz');
|
||||
const body = health.ok() ? await health.json() : {};
|
||||
test.skip(
|
||||
body.service !== 'ltx2-streaming-mock-server',
|
||||
'Mock-backed project lifecycle assertions require dreamverse.mock_server on BACKEND_PORT (default 8009).',
|
||||
);
|
||||
|
||||
await page.goto('/');
|
||||
|
||||
await test.step('complete a generation so there is a project to save', async () => {
|
||||
const continuation = page.getByLabel('Continuation prompt');
|
||||
await expect(continuation).toBeVisible();
|
||||
await continuation.fill('Aurora over a frozen lake');
|
||||
await page.getByRole('button', { name: /^generate$/i }).click();
|
||||
await expect(
|
||||
page.getByRole('button', { name: /download video|share video/i }),
|
||||
).toBeVisible({ timeout: 60_000 });
|
||||
});
|
||||
|
||||
const sidebar = page.getByRole('complementary', { name: 'Project history' });
|
||||
|
||||
await test.step('"New project" closes the sidebar and resets the composer', async () => {
|
||||
await page.getByRole('button', { name: 'Toggle sidebar' }).click();
|
||||
await expect(sidebar).toBeInViewport();
|
||||
await sidebar.getByRole('button', { name: /^new project$/i }).click();
|
||||
await expect(sidebar).not.toBeInViewport();
|
||||
await expect(page.getByRole('button', { name: /^generate$/i })).toBeVisible({ timeout: 30_000 });
|
||||
const continuation = page.getByLabel('Continuation prompt');
|
||||
await expect(continuation).toBeEnabled();
|
||||
await expect(continuation).toHaveValue('');
|
||||
});
|
||||
|
||||
await test.step('the prior session appears under timeline', async () => {
|
||||
await page.getByRole('button', { name: 'Toggle sidebar' }).click();
|
||||
await expect(sidebar).toBeInViewport();
|
||||
await expect(sidebar.getByText('Previous', { exact: true })).toBeVisible({ timeout: 30_000 });
|
||||
await expect(sidebar.getByText(/^(just now|\d+m ago)$/).first()).toBeVisible();
|
||||
});
|
||||
|
||||
await test.step('clicking the prior session enters viewing mode', async () => {
|
||||
const priorRow = sidebar.locator('div[role="button"]').filter({ hasText: /just now|\d+m ago/ }).first();
|
||||
await priorRow.click();
|
||||
await expect(sidebar).not.toBeInViewport();
|
||||
await expect(page.locator('video[autoplay][loop]')).toBeVisible({ timeout: 30_000 });
|
||||
});
|
||||
});
|
||||
|
||||
test('saved projects persist across a page reload', async ({ page, request }) => {
|
||||
const health = await request.get('/healthz');
|
||||
const body = health.ok() ? await health.json() : {};
|
||||
test.skip(
|
||||
body.service !== 'ltx2-streaming-mock-server',
|
||||
'Mock-backed persistence assertions require dreamverse.mock_server on BACKEND_PORT (default 8009).',
|
||||
);
|
||||
|
||||
await page.goto('/');
|
||||
|
||||
await test.step('complete a generation', async () => {
|
||||
const continuation = page.getByLabel('Continuation prompt');
|
||||
await expect(continuation).toBeVisible();
|
||||
await continuation.fill('Aurora over a frozen lake');
|
||||
await page.getByRole('button', { name: /^generate$/i }).click();
|
||||
await expect(
|
||||
page.getByRole('button', { name: /download video|share video/i }),
|
||||
).toBeVisible({ timeout: 60_000 });
|
||||
});
|
||||
|
||||
const sidebar = page.getByRole('complementary', { name: 'Project history' });
|
||||
|
||||
await test.step('persist via "New project" and confirm it lands under Previous', async () => {
|
||||
await page.getByRole('button', { name: 'Toggle sidebar' }).click();
|
||||
await expect(sidebar).toBeInViewport();
|
||||
await sidebar.getByRole('button', { name: /^new project$/i }).click();
|
||||
await expect(page.getByRole('button', { name: /^generate$/i })).toBeVisible({ timeout: 30_000 });
|
||||
await page.getByRole('button', { name: 'Toggle sidebar' }).click();
|
||||
await expect(sidebar).toBeInViewport();
|
||||
await expect(sidebar.getByText('Previous', { exact: true })).toBeVisible({ timeout: 30_000 });
|
||||
await expect(sidebar.getByText(/^(just now|\d+m ago)$/).first()).toBeVisible();
|
||||
});
|
||||
|
||||
await test.step('after page reload, the prior project is still in Previous', async () => {
|
||||
await page.reload();
|
||||
await page.getByRole('button', { name: 'Toggle sidebar' }).click();
|
||||
await expect(sidebar).toBeInViewport();
|
||||
await expect(sidebar.getByText('Previous', { exact: true })).toBeVisible({ timeout: 30_000 });
|
||||
await expect(sidebar.getByText(/^(just now|\d+m ago)$/).first()).toBeVisible();
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -6,10 +6,13 @@ const backendHost = process.env.BACKEND_HOST || '127.0.0.1';
|
||||
const backendPort = Number(process.env.BACKEND_PORT) || 8009;
|
||||
const backendUrl = `http://${backendHost}:${backendPort}`;
|
||||
const configDir = path.dirname(fileURLToPath(import.meta.url));
|
||||
const staticExport = process.env.NEXT_OUTPUT_EXPORT === '1';
|
||||
|
||||
const nextConfig: NextConfig = {
|
||||
...(staticExport ? { output: 'export' as const } : {}),
|
||||
...(staticExport ? { images: { unoptimized: true } } : {}),
|
||||
outputFileTracingRoot: path.join(configDir, '..', '..', '..'),
|
||||
async rewrites() {
|
||||
...(staticExport ? {} : { async rewrites() {
|
||||
return [
|
||||
{
|
||||
source: '/ws',
|
||||
@@ -47,8 +50,16 @@ const nextConfig: NextConfig = {
|
||||
source: '/curated-presets/:path*',
|
||||
destination: `${backendUrl}/curated-presets/:path*`,
|
||||
},
|
||||
{
|
||||
source: '/lora',
|
||||
destination: `${backendUrl}/lora`,
|
||||
},
|
||||
{
|
||||
source: '/lora/:path*',
|
||||
destination: `${backendUrl}/lora/:path*`,
|
||||
},
|
||||
];
|
||||
},
|
||||
} }),
|
||||
webpack: (config) => {
|
||||
config.module.rules.push({
|
||||
test: /\.jsonl$/,
|
||||
|
||||
Generated
+11669
-11443
File diff suppressed because it is too large
Load Diff
@@ -38,7 +38,7 @@
|
||||
"geist": "^1.7.0",
|
||||
"lucide-react": "^0.577.0",
|
||||
"mp4box": "^2.3.0",
|
||||
"next": "^15.3.3",
|
||||
"next": "15.5.18",
|
||||
"radix-ui": "^1.4.3",
|
||||
"react": "^19.1.0",
|
||||
"react-dom": "^19.1.0",
|
||||
@@ -59,9 +59,16 @@
|
||||
"@vitest/coverage-v8": "^3.2.4",
|
||||
"jsdom": "^26.1.0",
|
||||
"mock-socket": "^9.3.1",
|
||||
"postcss": "^8.5.8",
|
||||
"postcss": "8.5.10",
|
||||
"tailwindcss": "^4.2.1",
|
||||
"typescript": "^5.8.3",
|
||||
"vitest": "^3.2.4"
|
||||
},
|
||||
"overrides": {
|
||||
"esbuild": "0.25.12",
|
||||
"picomatch": "4.0.4",
|
||||
"postcss": "8.5.10",
|
||||
"rollup": "4.59.0",
|
||||
"vite": "7.3.2"
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,12 +1,16 @@
|
||||
import { defineConfig, devices } from '@playwright/test';
|
||||
|
||||
const backendHost = process.env.BACKEND_HOST || '127.0.0.1';
|
||||
const backendPort = process.env.BACKEND_PORT || '8009';
|
||||
|
||||
/**
|
||||
* Playwright config for Dreamverse end-to-end tests.
|
||||
*
|
||||
* Tests assume the Dreamverse Python server is reachable at
|
||||
* BACKEND_HOST:BACKEND_PORT (default 127.0.0.1:8009) and the Next.js frontend
|
||||
* runs on port 5299. The webServer block boots `pnpm run dev` if no
|
||||
* server is already listening so tests work both locally and in CI.
|
||||
* runs on port 5299. The webServer block boots the mock backend and
|
||||
* `npm run dev` if no server is already listening so tests work both
|
||||
* locally and in CI.
|
||||
*/
|
||||
export default defineConfig({
|
||||
testDir: './e2e',
|
||||
@@ -26,23 +30,34 @@ export default defineConfig({
|
||||
viewport: { width: 1280, height: 720 },
|
||||
screenshot: 'only-on-failure',
|
||||
trace: 'retain-on-failure',
|
||||
video: process.env.CI ? 'retain-on-failure' : 'on',
|
||||
},
|
||||
projects: [
|
||||
{
|
||||
name: 'chromium',
|
||||
use: { ...devices['Desktop Chrome'] },
|
||||
},
|
||||
{ name: 'chromium', use: { ...devices['Desktop Chrome'] } },
|
||||
{ name: 'webkit', use: { ...devices['Desktop Safari'] } },
|
||||
{ name: 'firefox', use: { ...devices['Desktop Firefox'] } },
|
||||
{ name: 'msedge', use: { ...devices['Desktop Edge'], channel: 'msedge' } },
|
||||
{ name: 'mobile-safari', use: { ...devices['iPhone 14'] } },
|
||||
{ name: 'mobile-chromium', use: { ...devices['Pixel 7'] } },
|
||||
],
|
||||
webServer: process.env.PLAYWRIGHT_SKIP_WEBSERVER
|
||||
? undefined
|
||||
: {
|
||||
command: 'pnpm run dev',
|
||||
url: 'http://127.0.0.1:5299',
|
||||
reuseExistingServer: true,
|
||||
timeout: 120_000,
|
||||
env: {
|
||||
BACKEND_HOST: process.env.BACKEND_HOST || '127.0.0.1',
|
||||
BACKEND_PORT: process.env.BACKEND_PORT || '8009',
|
||||
: [
|
||||
{
|
||||
command: `PYTHONPATH=.. python3 -m uvicorn dreamverse.mock_server:app --host ${backendHost} --port ${backendPort}`,
|
||||
url: `http://${backendHost}:${backendPort}/healthz`,
|
||||
reuseExistingServer: true,
|
||||
timeout: 120_000,
|
||||
},
|
||||
},
|
||||
{
|
||||
command: 'npm run dev',
|
||||
url: 'http://127.0.0.1:5299',
|
||||
reuseExistingServer: true,
|
||||
timeout: 120_000,
|
||||
env: {
|
||||
BACKEND_HOST: backendHost,
|
||||
BACKEND_PORT: backendPort,
|
||||
},
|
||||
},
|
||||
],
|
||||
});
|
||||
|
||||
Generated
-5199
File diff suppressed because it is too large
Load Diff
@@ -2489,6 +2489,8 @@ export default function Page() {
|
||||
if (devtoolsMode) {
|
||||
return (
|
||||
<DevtoolsShell
|
||||
videoRef={videoRefCallback}
|
||||
archivedPlaybackRef={archivedPlaybackRefCallback}
|
||||
connected={connected as boolean}
|
||||
gpuAssigned={gpuAssigned as boolean}
|
||||
sessionStarted={sessionStarted as boolean}
|
||||
|
||||
@@ -161,6 +161,7 @@ export default function VideoPlayer({
|
||||
}}
|
||||
size="icon"
|
||||
variant="outline"
|
||||
aria-label={canShare ? "Share video" : "Download video"}
|
||||
className="absolute top-3 left-3 z-10 cursor-pointer bg-slate-800/50 text-white/90 shadow-md backdrop-blur-sm transition-all border-white/30 hover:bg-slate-800/85 hover:border-white/50 hover:text-white hover:scale-105"
|
||||
>
|
||||
{canShare ? <Share className="size-5" /> : <Download className="size-5" />}
|
||||
|
||||
@@ -9,6 +9,7 @@ import { Label } from '@/components/ui/label';
|
||||
import { ScrollArea } from '@/components/ui/scroll-area';
|
||||
import { Textarea } from '@/components/ui/textarea';
|
||||
import { cn } from '@/lib/utils';
|
||||
import LoraControls from '@/components/devtools/LoraControls';
|
||||
|
||||
interface DevtoolsDrawerProps {
|
||||
editableMode?: boolean;
|
||||
@@ -126,6 +127,8 @@ export default function DevtoolsDrawer({
|
||||
</div>
|
||||
|
||||
<div className="grid gap-4 xl:grid-cols-2">
|
||||
<LoraControls />
|
||||
|
||||
{/* Prompt window memory */}
|
||||
<details className="group overflow-hidden rounded-2xl border border-border bg-card shadow-sm" open>
|
||||
<summary className="flex cursor-pointer list-none items-start justify-between gap-4 px-5 py-4">
|
||||
|
||||
@@ -113,6 +113,8 @@ interface DevtoolsShellProps {
|
||||
formatTime?: (seconds: number) => string;
|
||||
formatDurationMs?: (durationMs: number) => string;
|
||||
onPlaying?: () => void;
|
||||
videoRef?: React.RefCallback<HTMLVideoElement>;
|
||||
archivedPlaybackRef?: React.RefCallback<HTMLVideoElement>;
|
||||
}
|
||||
|
||||
export default function DevtoolsShell({
|
||||
@@ -209,6 +211,8 @@ export default function DevtoolsShell({
|
||||
formatTime = (seconds) => `${seconds}`,
|
||||
formatDurationMs = (durationMs) => `${durationMs}`,
|
||||
onPlaying = () => {},
|
||||
videoRef,
|
||||
archivedPlaybackRef,
|
||||
}: DevtoolsShellProps) {
|
||||
return (
|
||||
<WorkspaceShell
|
||||
@@ -235,6 +239,8 @@ export default function DevtoolsShell({
|
||||
workspace={
|
||||
<div className="flex flex-col gap-4">
|
||||
<VideoPlayer
|
||||
videoRef={videoRef}
|
||||
archivedPlaybackRef={archivedPlaybackRef}
|
||||
activeClip={activeClip}
|
||||
sessionStarted={sessionStarted}
|
||||
avPlaybackStarted={avPlaybackStarted}
|
||||
|
||||
@@ -0,0 +1,181 @@
|
||||
'use client';
|
||||
|
||||
import React, { useEffect, useRef, useState } from 'react';
|
||||
|
||||
import { Badge } from '@/components/ui/badge';
|
||||
import { Label } from '@/components/ui/label';
|
||||
|
||||
const STYLE_LABELS: Record<string, string> = {
|
||||
none: 'None',
|
||||
pixar: 'Pixar Toon',
|
||||
transition: 'Transition',
|
||||
};
|
||||
|
||||
export default function LoraControls() {
|
||||
const [styleKeys, setStyleKeys] = useState<string[]>([]);
|
||||
const [labels, setLabels] = useState<Record<string, string>>(STYLE_LABELS);
|
||||
const [strength, setStrength] = useState(0.8);
|
||||
const [enabled, setEnabled] = useState<Record<string, boolean>>({});
|
||||
const [intensity, setIntensity] = useState<Record<string, number>>({});
|
||||
const [status, setStatus] = useState('idle');
|
||||
|
||||
const strengthRef = useRef(0.8);
|
||||
const enabledRef = useRef<Record<string, boolean>>({});
|
||||
const intensityRef = useRef<Record<string, number>>({});
|
||||
const debounceRef = useRef<ReturnType<typeof setTimeout> | null>(null);
|
||||
const requestIdRef = useRef(0);
|
||||
|
||||
useEffect(() => {
|
||||
fetch('/lora/options')
|
||||
.then((response) => (response.ok ? response.json() : null))
|
||||
.then((data) => {
|
||||
if (data && Array.isArray(data.styles)) {
|
||||
const keys = data.styles.filter((s: string) => s !== 'none');
|
||||
setStyleKeys(keys);
|
||||
const initIntensity: Record<string, number> = {};
|
||||
keys.forEach((k: string) => {
|
||||
initIntensity[k] = 1.0;
|
||||
});
|
||||
setIntensity(initIntensity);
|
||||
intensityRef.current = initIntensity;
|
||||
}
|
||||
if (data && data.labels && typeof data.labels === 'object') {
|
||||
setLabels((prev) => ({ ...prev, ...data.labels }));
|
||||
}
|
||||
})
|
||||
.catch(() => setStatus('options unavailable'));
|
||||
|
||||
return () => {
|
||||
if (debounceRef.current) {
|
||||
clearTimeout(debounceRef.current);
|
||||
}
|
||||
};
|
||||
}, []);
|
||||
|
||||
const applyNow = () => {
|
||||
const reqId = ++requestIdRef.current;
|
||||
const stylesPayload: Record<string, number> = {};
|
||||
for (const key of Object.keys(enabledRef.current)) {
|
||||
if (enabledRef.current[key]) {
|
||||
stylesPayload[key] = intensityRef.current[key] ?? 1.0;
|
||||
}
|
||||
}
|
||||
setStatus('applying…');
|
||||
fetch('/lora', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ strength: strengthRef.current, styles: stylesPayload }),
|
||||
})
|
||||
.then(async (response) => {
|
||||
if (!response.ok) {
|
||||
throw new Error(await response.text());
|
||||
}
|
||||
return response.json();
|
||||
})
|
||||
.then((data) => {
|
||||
if (reqId !== requestIdRef.current) return;
|
||||
const active = Object.keys(data?.styles ?? {});
|
||||
const desc = active.length
|
||||
? active.map((k) => `${labels[k] ?? k}@${data.styles[k]}`).join(' + ')
|
||||
: 'no style';
|
||||
setStatus(`applied OmniNFT@${data.strength} + ${desc}`);
|
||||
})
|
||||
.catch((error) => {
|
||||
if (reqId !== requestIdRef.current) return;
|
||||
setStatus(`error: ${String(error).slice(0, 120)}`);
|
||||
});
|
||||
};
|
||||
|
||||
const scheduleApply = () => {
|
||||
if (debounceRef.current) {
|
||||
clearTimeout(debounceRef.current);
|
||||
}
|
||||
debounceRef.current = setTimeout(applyNow, 250);
|
||||
};
|
||||
|
||||
return (
|
||||
<details
|
||||
className="group overflow-hidden rounded-2xl border border-border bg-card shadow-sm xl:col-span-2"
|
||||
open
|
||||
>
|
||||
<summary className="flex cursor-pointer list-none items-start justify-between gap-4 px-5 py-4">
|
||||
<div className="space-y-1">
|
||||
<span className="block text-lg font-semibold text-foreground">
|
||||
LoRA stack
|
||||
</span>
|
||||
<span className="block text-sm leading-6 text-muted-foreground">
|
||||
OmniNFT strength + stackable style adapters, each with its own intensity (live).
|
||||
</span>
|
||||
</div>
|
||||
<Badge variant="secondary">{status}</Badge>
|
||||
</summary>
|
||||
<div className="space-y-5 border-t border-border px-5 py-4">
|
||||
<div className="space-y-2">
|
||||
<Label htmlFor="lora-strength">
|
||||
OmniNFT strength · {strength.toFixed(2)}
|
||||
</Label>
|
||||
<input
|
||||
id="lora-strength"
|
||||
type="range"
|
||||
min={0}
|
||||
max={1}
|
||||
step={0.05}
|
||||
value={strength}
|
||||
onChange={(event) => {
|
||||
const next = Number(event.target.value);
|
||||
setStrength(next);
|
||||
strengthRef.current = next;
|
||||
scheduleApply();
|
||||
}}
|
||||
className="w-full accent-sky-400"
|
||||
/>
|
||||
</div>
|
||||
|
||||
{styleKeys.length === 0 ? (
|
||||
<p className="text-sm text-muted-foreground">
|
||||
No style adapters for the active model.
|
||||
</p>
|
||||
) : (
|
||||
styleKeys.map((key) => (
|
||||
<div key={key} className="space-y-2">
|
||||
<div className="flex items-center justify-between">
|
||||
<label className="flex cursor-pointer items-center gap-2 text-sm font-medium text-foreground">
|
||||
<input
|
||||
type="checkbox"
|
||||
checked={!!enabled[key]}
|
||||
onChange={(event) => {
|
||||
const next = { ...enabledRef.current, [key]: event.target.checked };
|
||||
enabledRef.current = next;
|
||||
setEnabled(next);
|
||||
scheduleApply();
|
||||
}}
|
||||
className="accent-sky-400"
|
||||
/>
|
||||
{labels[key] ?? STYLE_LABELS[key] ?? key}
|
||||
</label>
|
||||
<span className="text-xs text-muted-foreground">
|
||||
{(intensity[key] ?? 1).toFixed(2)}
|
||||
</span>
|
||||
</div>
|
||||
<input
|
||||
type="range"
|
||||
min={0}
|
||||
max={1}
|
||||
step={0.05}
|
||||
value={intensity[key] ?? 1}
|
||||
disabled={!enabled[key]}
|
||||
onChange={(event) => {
|
||||
const next = { ...intensityRef.current, [key]: Number(event.target.value) };
|
||||
intensityRef.current = next;
|
||||
setIntensity(next);
|
||||
scheduleApply();
|
||||
}}
|
||||
className="w-full accent-sky-400 disabled:opacity-40"
|
||||
/>
|
||||
</div>
|
||||
))
|
||||
)}
|
||||
</div>
|
||||
</details>
|
||||
);
|
||||
}
|
||||
@@ -1,106 +0,0 @@
|
||||
# FVD (Fréchet Video Distance) Benchmark
|
||||
|
||||
Evaluate generated video quality using FVD with the I3D feature extractor.
|
||||
|
||||
## Quick Start
|
||||
|
||||
**Run the benchmark:**
|
||||
|
||||
```bash
|
||||
bash benchmarks/scripts/run.sh
|
||||
```
|
||||
|
||||
That's it! The script auto-installs dependencies and runs the benchmark.
|
||||
|
||||
**To customize:** Edit `benchmarks/fvd/run_fvd.py` to change:
|
||||
- Video paths (`real_dir`, `gen_dir`)
|
||||
- Number of videos, frames, sampling strategy
|
||||
- Device, batch size, caching, etc.
|
||||
|
||||
## Advanced Usage (CLI)
|
||||
|
||||
For more control without editing Python files, use the CLI.
|
||||
|
||||
**First-time setup** (one-time per pod/environment):
|
||||
|
||||
```bash
|
||||
bash benchmarks/scripts/setup_fvd.sh
|
||||
```
|
||||
|
||||
Then run any configuration you want:
|
||||
|
||||
```bash
|
||||
# Custom configuration
|
||||
python -m benchmarks.fvd.cli \
|
||||
--real-path data/real/ \
|
||||
--gen-path outputs/gen/ \
|
||||
--num-videos 1024 \
|
||||
--num-frames 32 \
|
||||
--clip-strategy random \
|
||||
--batch-size 32 \
|
||||
--seed 42 \
|
||||
--extractor clip
|
||||
```
|
||||
|
||||
**Standard protocols:**
|
||||
|
||||
```bash
|
||||
# Use predefined protocols
|
||||
python -m benchmarks.fvd.cli \
|
||||
--real-path data/real/ \
|
||||
--gen-path outputs/gen/ \
|
||||
--protocol fvd2048_16f # or fvd2048_128f, quick_test, etc.
|
||||
```
|
||||
|
||||
This would use i3d model by default as the feature extractor
|
||||
|
||||
**Feature caching** (speed up repeated evaluations):
|
||||
|
||||
```bash
|
||||
python -m benchmarks.fvd.cli \
|
||||
--real-path data/real/ \
|
||||
--gen-path outputs/gen/ \
|
||||
--protocol fvd2048_16f \
|
||||
--cache-real-features fvd-cache/extractor_name # Directory path (will save/load fvd-cache/extractor_name/extractor-name_real_features.pkl)
|
||||
```
|
||||
|
||||
Run `python -m benchmarks.fvd.cli --help` for all options.
|
||||
|
||||
## Available Protocols
|
||||
|
||||
- `fvd2048_16f` - Standard (2048 videos, 16 frames)
|
||||
- `fvd2048_128f` - Long videos (128 frames)
|
||||
- `fvd2048_128f_subsample8` - Subsampled long videos
|
||||
- `quick_test` - Fast testing (10 videos)
|
||||
|
||||
## Configuration Options
|
||||
|
||||
Key options in `FVDConfig`:
|
||||
|
||||
```python
|
||||
num_videos=2048, # Videos to evaluate
|
||||
num_frames_per_clip=16, # Frames per clip
|
||||
clip_strategy='beginning', # beginning|random|uniform|middle|sliding
|
||||
frame_stride=1, # Frame subsampling
|
||||
batch_size=32, # GPU batch size
|
||||
device='cuda', # cuda|cpu
|
||||
cache_real_features=None, # Cache path for speed
|
||||
seed=42, # Reproducibility
|
||||
extractor='i3d', # i3d|clip|videomae
|
||||
```
|
||||
|
||||
## Programmatic Usage
|
||||
|
||||
```python
|
||||
from benchmarks.fvd import compute_fvd_with_config, FVDConfig
|
||||
|
||||
config = FVDConfig.fvd2048_16f() # or custom config
|
||||
results = compute_fvd_with_config('data/real/', 'outputs/gen/', config)
|
||||
print(f"FVD: {results['fvd']:.2f}")
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- Requires minimum 10 frames per clip
|
||||
- Supports both video files (.mp4, .avi, etc.) and frame directories
|
||||
- `--cache-real-features` expects a **directory path** (e.g., `cache/real`), it will automatically create/load `real_features.pkl` inside that directory
|
||||
@@ -1,37 +0,0 @@
|
||||
"""
|
||||
FastVideo Frechet Video Distance (FVD) Benchmark Module.
|
||||
>>> from fastvideo.benchmarks.fvd import compute_fvd_with_config, FVDConfig
|
||||
>>> config = FVDConfig.fvd2048_16f() # Standard protocol
|
||||
>>> results = compute_fvd_with_config('data/real/', 'outputs/gen/', config)
|
||||
>>> print(f"FVD: {results['fvd']:.2f}")
|
||||
"""
|
||||
|
||||
from .fvd import (
|
||||
compute_fvd,
|
||||
compute_fvd_with_config,
|
||||
compute_frechet_distance,
|
||||
compute_statistics,
|
||||
FVDConfig,
|
||||
)
|
||||
from .feature_extractors import (BaseFeatureExtractor, I3DFeatureExtractor, load_extractor)
|
||||
from .video_utils import (
|
||||
load_video_auto,
|
||||
sample_clips_from_video,
|
||||
load_video_clips_streaming,
|
||||
ClipSamplingStrategy,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
'compute_fvd',
|
||||
'compute_fvd_with_config',
|
||||
'compute_frechet_distance',
|
||||
'compute_statistics',
|
||||
'FVDConfig',
|
||||
'BaseFeatureExtractor',
|
||||
'I3DFeatureExtractor',
|
||||
'load_extractor',
|
||||
'load_video_auto',
|
||||
'sample_clips_from_video',
|
||||
'load_video_clips_streaming',
|
||||
'ClipSamplingStrategy',
|
||||
]
|
||||
@@ -1,77 +0,0 @@
|
||||
import argparse
|
||||
import sys
|
||||
import traceback
|
||||
from .fvd import compute_fvd_with_config, FVDConfig
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description='Compute Fréchet Video Distance (FVD)')
|
||||
|
||||
# Required arguments
|
||||
parser.add_argument('--real-path', type=str, required=True, help='Path to real videos')
|
||||
parser.add_argument('--gen-path', type=str, required=True, help='Path to generated videos')
|
||||
|
||||
# Extractor selection
|
||||
parser.add_argument('--extractor',
|
||||
type=str,
|
||||
default='i3d',
|
||||
choices=['i3d', 'clip', 'videomae'],
|
||||
help='Feature extractor model to use (default: i3d)')
|
||||
|
||||
# Standard args
|
||||
parser.add_argument('--seed', type=int, default=None, help='Random seed for reproducibility')
|
||||
parser.add_argument('--protocol',
|
||||
type=str,
|
||||
default=None,
|
||||
choices=['fvd2048_16f', 'fvd2048_128f', 'quick_test'],
|
||||
help='Use standard protocol (overrides other settings)')
|
||||
parser.add_argument('--num-videos', type=int, default=2048, help='Number of videos to use')
|
||||
parser.add_argument('--num-frames', type=int, default=16, help='Number of frames per clip')
|
||||
parser.add_argument('--clip-strategy', type=str, default='beginning', help='Clip sampling strategy')
|
||||
parser.add_argument('--batch-size', type=int, default=32, help='Batch size for feature extraction')
|
||||
parser.add_argument('--device', type=str, default='cuda', help='Device to use (cuda or cpu)')
|
||||
parser.add_argument('--cache-real-features', type=str, default=None, help='Path to cache real video features')
|
||||
parser.add_argument('--quiet', action='store_true', help='Suppress progress output')
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
# Create config
|
||||
if args.protocol:
|
||||
protocol_map = {
|
||||
'fvd2048_16f': FVDConfig.fvd2048_16f,
|
||||
'fvd2048_128f': FVDConfig.fvd2048_128f,
|
||||
'quick_test': FVDConfig.quick_test,
|
||||
}
|
||||
config = protocol_map[args.protocol]()
|
||||
# Apply overrides
|
||||
config.device = args.device
|
||||
config.cache_real_features = args.cache_real_features
|
||||
config.extractor_model = args.extractor # Apply extractor arg
|
||||
else:
|
||||
config = FVDConfig(
|
||||
num_videos=args.num_videos,
|
||||
num_frames_per_clip=args.num_frames,
|
||||
extractor_model=args.extractor, # Apply extractor arg
|
||||
clip_strategy=args.clip_strategy,
|
||||
batch_size=args.batch_size,
|
||||
device=args.device,
|
||||
cache_real_features=args.cache_real_features,
|
||||
seed=args.seed)
|
||||
|
||||
try:
|
||||
_ = compute_fvd_with_config(
|
||||
args.real_path, # noqa: F841
|
||||
args.gen_path,
|
||||
config,
|
||||
verbose=not args.quiet)
|
||||
|
||||
return 0
|
||||
|
||||
except Exception as e:
|
||||
print(f"Error: {e}", file=sys.stderr)
|
||||
traceback.print_exc(file=sys.stderr)
|
||||
return 1
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
sys.exit(main())
|
||||
@@ -1,232 +0,0 @@
|
||||
"""
|
||||
Pluggable Feature Extractors for FVD Computation.
|
||||
Supports I3D (standard), CLIP, and VideoMAE via a common interface.
|
||||
"""
|
||||
|
||||
import torch
|
||||
import torch.nn as nn
|
||||
import torch.nn.functional as F
|
||||
from abc import ABC, abstractmethod
|
||||
from huggingface_hub import hf_hub_download
|
||||
from tqdm import tqdm
|
||||
|
||||
try:
|
||||
from transformers import CLIPModel, CLIPProcessor, VideoMAEModel
|
||||
TRANSFORMERS_AVAILABLE = True
|
||||
except ImportError:
|
||||
TRANSFORMERS_AVAILABLE = False
|
||||
|
||||
|
||||
class BaseFeatureExtractor(ABC, nn.Module):
|
||||
"""Abstract base class for all video feature extractors."""
|
||||
|
||||
def __init__(self, device: str = 'cuda'):
|
||||
super().__init__()
|
||||
self.device = torch.device(device if torch.cuda.is_available() else 'cpu')
|
||||
|
||||
@property
|
||||
@abstractmethod
|
||||
def feature_dim(self) -> int:
|
||||
"""Dimension of the output feature vector."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def preprocess(self, videos: torch.Tensor) -> torch.Tensor:
|
||||
"""
|
||||
Args:
|
||||
videos: [B, T, C, H, W] in [0, 255] range.
|
||||
Returns:
|
||||
Preprocessed tensor ready for the model.
|
||||
"""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def extract_features_batch(self, videos: torch.Tensor) -> torch.Tensor:
|
||||
"""
|
||||
Extract features for a single batch.
|
||||
Args:
|
||||
videos: [B, T, C, H, W] (raw input)
|
||||
Returns:
|
||||
Features: [B, feature_dim]
|
||||
"""
|
||||
pass
|
||||
|
||||
@torch.no_grad()
|
||||
def extract_features(self, videos: torch.Tensor, batch_size: int = 32, verbose: bool = True) -> torch.Tensor:
|
||||
"""
|
||||
Extract features for a large tensor of videos by batching.
|
||||
"""
|
||||
N = len(videos)
|
||||
all_features = []
|
||||
|
||||
iterator = range(0, N, batch_size)
|
||||
if verbose:
|
||||
iterator = tqdm(iterator, desc=f"Extracting features ({self.__class__.__name__})")
|
||||
|
||||
for i in iterator:
|
||||
batch = videos[i:i + batch_size].to(self.device)
|
||||
features = self.extract_features_batch(batch)
|
||||
all_features.append(features.cpu())
|
||||
|
||||
return torch.cat(all_features, dim=0)
|
||||
|
||||
|
||||
# 1. I3D Extractor (The Standard FVD Metric)
|
||||
class I3DFeatureExtractor(BaseFeatureExtractor):
|
||||
REPO_ID = 'flateon/FVD-I3D-torchscript'
|
||||
MODEL_FILENAME = 'i3d_torchscript.pt'
|
||||
|
||||
def __init__(self, device: str = 'cuda', cache_dir: str | None = None):
|
||||
super().__init__(device)
|
||||
self.cache_dir = cache_dir
|
||||
self.model = self._load_model()
|
||||
self.model.eval()
|
||||
self.model.to(self.device)
|
||||
|
||||
@property
|
||||
def feature_dim(self) -> int:
|
||||
return 400
|
||||
|
||||
def _load_model(self) -> torch.nn.Module:
|
||||
try:
|
||||
model_path = hf_hub_download(repo_id=self.REPO_ID, filename=self.MODEL_FILENAME, cache_dir=self.cache_dir)
|
||||
return torch.jit.load(model_path, map_location=self.device)
|
||||
except Exception as e:
|
||||
raise RuntimeError(f"Failed to load I3D model: {e}") from e
|
||||
|
||||
def preprocess(self, videos: torch.Tensor) -> torch.Tensor:
|
||||
"""Standard I3D preprocessing: Resize to 224, Norm to [-1, 1]."""
|
||||
B, T, C, H, W = videos.shape
|
||||
|
||||
if T < 10:
|
||||
raise ValueError(f"I3D requires at least 10 frames, got {T}")
|
||||
|
||||
# Normalize to [0, 1]
|
||||
if videos.max() > 1.0:
|
||||
videos = videos / 255.0
|
||||
|
||||
# Scale to [-1, 1]
|
||||
videos = videos * 2.0 - 1.0
|
||||
|
||||
# Resize to 224x224
|
||||
if H != 224 or W != 224:
|
||||
videos = videos.reshape(B * T, C, H, W)
|
||||
videos = F.interpolate(videos, size=(224, 224), mode='bilinear', align_corners=False)
|
||||
videos = videos.reshape(B, T, C, 224, 224)
|
||||
|
||||
# [B, T, C, H, W] -> [B, C, T, H, W]
|
||||
return videos.permute(0, 2, 1, 3, 4).contiguous()
|
||||
|
||||
def extract_features_batch(self, videos: torch.Tensor) -> torch.Tensor:
|
||||
batch = self.preprocess(videos)
|
||||
# TorchScript I3D returns raw logits when return_features=True
|
||||
return self.model(batch, rescale=False, resize=False, return_features=True)
|
||||
|
||||
|
||||
# 2. CLIP Extractor (Semantic/Content Quality)
|
||||
class CLIPFeatureExtractor(BaseFeatureExtractor):
|
||||
|
||||
def __init__(self, device: str = 'cuda', model_name: str = "openai/clip-vit-base-patch32"):
|
||||
if not TRANSFORMERS_AVAILABLE:
|
||||
raise ImportError("Please install transformers: uv pip install transformers")
|
||||
super().__init__(device)
|
||||
self.processor = CLIPProcessor.from_pretrained(model_name)
|
||||
self.model = CLIPModel.from_pretrained(model_name).to(self.device)
|
||||
self.model.eval()
|
||||
self._feature_dim = self.model.config.projection_dim
|
||||
|
||||
@property
|
||||
def feature_dim(self) -> int:
|
||||
return self._feature_dim
|
||||
|
||||
def preprocess(self, videos: torch.Tensor) -> torch.Tensor:
|
||||
# Ensure values are [0, 255]
|
||||
if videos.max() <= 1.0:
|
||||
videos = videos * 255.0
|
||||
|
||||
return videos.to(torch.uint8)
|
||||
|
||||
def extract_features_batch(self, videos: torch.Tensor) -> torch.Tensor:
|
||||
# Input: [B, T, C, H, W]
|
||||
B, T, C, H, W = videos.shape
|
||||
videos = self.preprocess(videos)
|
||||
|
||||
# Flatten B*T to treat frames as images
|
||||
images = videos.view(B * T, C, H, W)
|
||||
|
||||
# HF Processor
|
||||
inputs = self.processor(images=images, return_tensors="pt", padding=True)
|
||||
inputs = {k: v.to(self.device) for k, v in inputs.items()}
|
||||
|
||||
# Extract features [B*T, Dim]
|
||||
outputs = self.model.get_image_features(**inputs)
|
||||
|
||||
# Reshape [B, T, Dim] and Average Pooling over time
|
||||
outputs = outputs.view(B, T, -1)
|
||||
return outputs.mean(dim=1)
|
||||
|
||||
|
||||
# 3. VideoMAE Extractor (Structure/Motion Quality)
|
||||
class VideoMAEFeatureExtractor(BaseFeatureExtractor):
|
||||
|
||||
def __init__(self, device: str = 'cuda', model_name: str = "MCG-NJU/videomae-base"):
|
||||
if not TRANSFORMERS_AVAILABLE:
|
||||
raise ImportError("Please install transformers: uv pip install transformers")
|
||||
super().__init__(device)
|
||||
self.model = VideoMAEModel.from_pretrained(model_name).to(self.device)
|
||||
self.model.eval()
|
||||
|
||||
self.register_buffer('mean', torch.tensor([0.485, 0.456, 0.406], device=self.device).view(1, 1, 3, 1, 1))
|
||||
self.register_buffer('std', torch.tensor([0.229, 0.224, 0.225], device=self.device).view(1, 1, 3, 1, 1))
|
||||
|
||||
@property
|
||||
def feature_dim(self) -> int:
|
||||
return self.model.config.hidden_size
|
||||
|
||||
def preprocess(self, videos: torch.Tensor) -> torch.Tensor:
|
||||
"""
|
||||
Efficient GPU-based preprocessing.
|
||||
Input: [B, T, C, H, W] in range [0, 255]
|
||||
"""
|
||||
B, T, C, H, W = videos.shape
|
||||
|
||||
# 1. Resize to 224x224
|
||||
if H != 224 or W != 224:
|
||||
videos = videos.view(B * T, C, H, W)
|
||||
videos = F.interpolate(videos, size=(224, 224), mode='bilinear', align_corners=False)
|
||||
videos = videos.view(B, T, C, 224, 224)
|
||||
|
||||
# 2. Normalize to [0, 1]
|
||||
if videos.dtype != torch.float32:
|
||||
videos = videos.float()
|
||||
|
||||
if videos.max() > 1.0:
|
||||
videos = videos / 255.0
|
||||
|
||||
# 3. Apply ImageNet Mean/Std
|
||||
return (videos - self.mean) / self.std
|
||||
|
||||
def extract_features_batch(self, videos: torch.Tensor) -> torch.Tensor:
|
||||
# Input: [B, T, C, H, W]
|
||||
|
||||
# Fast GPU Preprocessing
|
||||
pixel_values = self.preprocess(videos)
|
||||
|
||||
# Forward pass
|
||||
outputs = self.model(pixel_values)
|
||||
|
||||
# Global Average Pooling of last hidden state [B, T_patches, 768] -> [B, 768]
|
||||
return outputs.last_hidden_state.mean(dim=1)
|
||||
|
||||
|
||||
# Factory
|
||||
def load_extractor(name: str, device: str = 'cuda') -> BaseFeatureExtractor:
|
||||
name = name.lower()
|
||||
if name == 'i3d':
|
||||
return I3DFeatureExtractor(device)
|
||||
elif name == 'clip':
|
||||
return CLIPFeatureExtractor(device)
|
||||
elif name == 'videomae':
|
||||
return VideoMAEFeatureExtractor(device)
|
||||
else:
|
||||
raise ValueError(f"Unknown extractor: {name}. Options: i3d, clip, videomae")
|
||||
@@ -1,384 +0,0 @@
|
||||
import numpy as np
|
||||
import scipy.linalg
|
||||
import torch
|
||||
from pathlib import Path
|
||||
from collections.abc import Iterator
|
||||
import pickle
|
||||
from dataclasses import dataclass, field
|
||||
from .feature_extractors import BaseFeatureExtractor, load_extractor
|
||||
from .video_utils import ClipSamplingStrategy, load_video_clips_streaming
|
||||
|
||||
|
||||
def compute_statistics(features: np.ndarray) -> tuple[np.ndarray, np.ndarray]:
|
||||
"""Compute mean and covariance."""
|
||||
mu = np.mean(features, axis=0)
|
||||
sigma = np.cov(features, rowvar=False)
|
||||
return mu, sigma
|
||||
|
||||
|
||||
def compute_frechet_distance(mu1: np.ndarray,
|
||||
sigma1: np.ndarray,
|
||||
mu2: np.ndarray,
|
||||
sigma2: np.ndarray,
|
||||
eps: float = 1e-6) -> float:
|
||||
"""
|
||||
Compute Fréchet distance between two Gaussians.
|
||||
"""
|
||||
sigma1 = sigma1 + eps * np.eye(sigma1.shape[0])
|
||||
sigma2 = sigma2 + eps * np.eye(sigma2.shape[0])
|
||||
|
||||
diff = mu1 - mu2
|
||||
mean_distance = np.sum(diff**2)
|
||||
|
||||
trace_sum = np.trace(sigma1 + sigma2)
|
||||
|
||||
covmean = scipy.linalg.sqrtm(sigma1 @ sigma2)
|
||||
|
||||
if np.iscomplexobj(covmean):
|
||||
if not np.allclose(np.diagonal(covmean).imag, 0, atol=1e-3):
|
||||
print(f"Warning: Imaginary component: {np.max(np.abs(covmean.imag))}")
|
||||
covmean = covmean.real
|
||||
|
||||
trace_product = np.trace(covmean)
|
||||
|
||||
fvd = mean_distance + trace_sum - 2 * trace_product
|
||||
|
||||
return float(fvd)
|
||||
|
||||
|
||||
@dataclass
|
||||
class FVDConfig:
|
||||
# default configuration for FVD computation:
|
||||
|
||||
# Video selection
|
||||
num_videos: int = 2048
|
||||
|
||||
# Feature Extractor Selection
|
||||
extractor_model: str = 'i3d' # Options: 'i3d', 'clip', 'videomae'
|
||||
|
||||
# Clip sampling
|
||||
num_frames_per_clip: int = 16
|
||||
num_clips_per_video: int = 1
|
||||
clip_strategy: str | ClipSamplingStrategy = 'beginning'
|
||||
|
||||
# Temporal subsampling
|
||||
frame_stride: int = 1 # 1=no subsampling, 2=every 2nd, 8=every 8th
|
||||
temporal_stride: int = 1 # For sliding window clips
|
||||
|
||||
# Data processing
|
||||
video_extensions: list[str] = field(default_factory=lambda: ['.mp4', '.avi', '.mov', '.mkv'])
|
||||
support_frame_dirs: bool = True
|
||||
|
||||
# Computation
|
||||
batch_size: int = 32
|
||||
device: str = 'cuda'
|
||||
|
||||
use_streaming: bool = True
|
||||
resize_before_extraction: bool = True
|
||||
|
||||
# Caching
|
||||
cache_real_features: str | None = None
|
||||
i3d_model_path: str | None = None
|
||||
|
||||
# Reproducibility
|
||||
seed: int | None = None
|
||||
|
||||
@classmethod
|
||||
def fvd2048_16f(cls) -> 'FVDConfig':
|
||||
"""Standard FVD protocol: 2048 videos, 16 frames, beginning clip."""
|
||||
return cls(num_videos=2048, num_frames_per_clip=16, clip_strategy='beginning', use_streaming=True)
|
||||
|
||||
@classmethod
|
||||
def fvd2048_128f(cls) -> 'FVDConfig':
|
||||
"""Long video protocol: 2048 videos, 128 frames."""
|
||||
return cls(num_videos=2048, num_frames_per_clip=128, clip_strategy='beginning', use_streaming=True)
|
||||
|
||||
@classmethod
|
||||
def quick_test(cls) -> 'FVDConfig':
|
||||
"""Quick test config: 100 videos, 16 frames."""
|
||||
return cls(num_videos=100, num_frames_per_clip=16, clip_strategy='beginning')
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
"""Export config to dict for logging"""
|
||||
d = self.__dict__.copy()
|
||||
d['clip_strategy'] = str(self.clip_strategy)
|
||||
return d
|
||||
|
||||
def __str__(self) -> str:
|
||||
"""Human-readable protocol name"""
|
||||
desc = f"FVD_{self.extractor_model.upper()}_{self.num_videos}_{self.num_frames_per_clip}f"
|
||||
if self.frame_stride > 1:
|
||||
desc += f"_subsample{self.frame_stride}"
|
||||
if self.num_clips_per_video > 1:
|
||||
desc += f"_{self.num_clips_per_video}clips"
|
||||
if self.clip_strategy != 'beginning':
|
||||
desc += f"_{self.clip_strategy}"
|
||||
return desc
|
||||
|
||||
|
||||
def extract_features_streaming(video_generator: Iterator[torch.Tensor],
|
||||
extractor: BaseFeatureExtractor,
|
||||
batch_size: int = 32,
|
||||
max_clips: int | None = None,
|
||||
verbose: bool = True) -> np.ndarray:
|
||||
"""
|
||||
Extract features from a video clip generator using streaming.
|
||||
"""
|
||||
all_features = []
|
||||
batch = []
|
||||
|
||||
if verbose:
|
||||
print(f"Extracting features with batch_size={batch_size}...")
|
||||
|
||||
with torch.no_grad():
|
||||
for clip_count, clip in enumerate(video_generator):
|
||||
batch.append(clip)
|
||||
|
||||
# Process batch when full
|
||||
if len(batch) == batch_size:
|
||||
batch_tensor = torch.stack(batch).to(extractor.device)
|
||||
features = extractor.extract_features_batch(batch_tensor)
|
||||
|
||||
all_features.append(features.detach().cpu().numpy())
|
||||
batch = []
|
||||
|
||||
if verbose and clip_count % (batch_size * 10) == 0:
|
||||
print(f"Processed {clip_count} clips...")
|
||||
|
||||
if max_clips is not None and clip_count >= max_clips:
|
||||
break
|
||||
|
||||
# Process remaining clips
|
||||
if len(batch) > 0:
|
||||
batch_tensor = torch.stack(batch).to(extractor.device)
|
||||
features = extractor.extract_features_batch(batch_tensor)
|
||||
all_features.append(features.detach().cpu().numpy())
|
||||
|
||||
if len(all_features) == 0:
|
||||
raise RuntimeError("No features extracted - check video loading")
|
||||
|
||||
features = np.concatenate(all_features, axis=0)
|
||||
|
||||
if verbose:
|
||||
print(f"Extracted {len(features)} feature vectors")
|
||||
|
||||
return features
|
||||
|
||||
|
||||
def load_or_compute_features(videos: str | Path | torch.Tensor,
|
||||
extractor: BaseFeatureExtractor,
|
||||
config: FVDConfig,
|
||||
cache_path: str | None = None,
|
||||
cache_name: str = "real_features") -> np.ndarray:
|
||||
"""Load features from cache or compute (with streaming support)"""
|
||||
|
||||
if cache_path is not None:
|
||||
script_dir = Path(__file__).parent
|
||||
cache_dir = script_dir / cache_path
|
||||
cache_file = cache_dir / f"{config.extractor_model}_{cache_name}.pkl"
|
||||
|
||||
if cache_file.exists():
|
||||
print(f"Loading cached features from {cache_file}")
|
||||
with open(cache_file, 'rb') as f:
|
||||
features = pickle.load(f)
|
||||
|
||||
# Validate and limit based on config
|
||||
max_features = config.num_videos * config.num_clips_per_video
|
||||
|
||||
if len(features) < max_features:
|
||||
print(f"WARNING: Cache has {len(features)} features but need {max_features}")
|
||||
print("Cached features insufficient - will recompute...")
|
||||
elif len(features) > max_features:
|
||||
print(f"Using {max_features} features from cache (truncated from {len(features)})")
|
||||
features = features[:max_features]
|
||||
return features
|
||||
else:
|
||||
print(f"Using all {len(features)} cached features")
|
||||
return features
|
||||
|
||||
print("Computing features from scratch...")
|
||||
|
||||
if isinstance(videos, (str | Path)):
|
||||
target_size = (224, 224) if config.resize_before_extraction else None
|
||||
|
||||
video_generator = load_video_clips_streaming(videos,
|
||||
num_frames=config.num_frames_per_clip,
|
||||
max_videos=config.num_videos,
|
||||
clip_strategy=config.clip_strategy,
|
||||
frame_stride=config.frame_stride,
|
||||
num_clips_per_video=config.num_clips_per_video,
|
||||
video_extensions=config.video_extensions,
|
||||
support_frame_dirs=config.support_frame_dirs,
|
||||
target_size=target_size,
|
||||
verbose=True)
|
||||
|
||||
max_clips = config.num_videos * config.num_clips_per_video
|
||||
features = extract_features_streaming(video_generator,
|
||||
extractor,
|
||||
batch_size=config.batch_size,
|
||||
max_clips=max_clips,
|
||||
verbose=True)
|
||||
else:
|
||||
print(f"Extracting features from {len(videos)} video tensors...")
|
||||
features = extractor.extract_features(videos, batch_size=config.batch_size, verbose=True)
|
||||
features = features.numpy()
|
||||
|
||||
# Validate feature count
|
||||
expected_count = config.num_videos * config.num_clips_per_video
|
||||
if len(features) < expected_count:
|
||||
raise ValueError(f"ERROR: Only extracted {len(features)} features, but need {expected_count}!\n"
|
||||
f"Found fewer videos than expected. Check your video directory.")
|
||||
elif len(features) > expected_count:
|
||||
print(f"Truncating {len(features)} features to {expected_count}")
|
||||
features = features[:expected_count]
|
||||
|
||||
# Cache features if requested
|
||||
if cache_path is not None:
|
||||
script_dir = Path(__file__).parent
|
||||
cache_dir = script_dir / cache_path
|
||||
cache_dir.mkdir(parents=True, exist_ok=True)
|
||||
cache_file = cache_dir / f"{config.extractor_model}_{cache_name}.pkl"
|
||||
print(f"Caching features to {cache_file}")
|
||||
with open(cache_file, 'wb') as f:
|
||||
pickle.dump(features, f)
|
||||
|
||||
return features
|
||||
|
||||
|
||||
def compute_fvd_with_config(real_videos: str | Path | torch.Tensor,
|
||||
gen_videos: str | Path | torch.Tensor,
|
||||
config: FVDConfig,
|
||||
verbose: bool = True) -> dict:
|
||||
"""
|
||||
Compute FVD using a standardized configuration.
|
||||
|
||||
This is the recommended way to compute FVD for reproducibility.
|
||||
|
||||
Args:
|
||||
real_videos: Path or tensors
|
||||
gen_videos: Path or tensors
|
||||
config: FVDConfig specifying protocol
|
||||
verbose: Print progress
|
||||
|
||||
Returns:
|
||||
results: Dictionary with:
|
||||
- 'fvd': FVD score (float)
|
||||
- 'protocol': Protocol name (str)
|
||||
- 'model': Feature extractor model name (str)
|
||||
- 'config': Configuration dict
|
||||
|
||||
Example:
|
||||
>>> config = FVDConfig.fvd2048_16f()
|
||||
>>> results = compute_fvd_with_config('data/real/', 'outputs/gen/', config)
|
||||
>>> print(f"FVD: {results['fvd']:.2f}")
|
||||
"""
|
||||
|
||||
# Seed for reproducibility
|
||||
if config.seed is not None:
|
||||
import random as _rnd
|
||||
_rnd.seed(config.seed)
|
||||
np.random.seed(config.seed)
|
||||
torch.manual_seed(config.seed)
|
||||
if torch.cuda.is_available():
|
||||
torch.cuda.manual_seed_all(config.seed)
|
||||
|
||||
if verbose:
|
||||
print("=" * 70)
|
||||
print(f"Computing FVD with protocol: {config}")
|
||||
print(f"Model: {config.extractor_model.upper()}")
|
||||
print("=" * 70)
|
||||
print("\nConfiguration:")
|
||||
for key, value in config.to_dict().items():
|
||||
print(f" {key}: {value}")
|
||||
print()
|
||||
|
||||
# Initialize Extractor using Factory
|
||||
if verbose:
|
||||
print(f"\nInitializing {config.extractor_model.upper()} model on {config.device}...")
|
||||
|
||||
extractor = load_extractor(config.extractor_model, device=config.device)
|
||||
|
||||
# Extract features
|
||||
if verbose:
|
||||
print(f"\n{'='*70}")
|
||||
print("Extracting REAL video features...")
|
||||
print(f"{'='*70}")
|
||||
|
||||
real_features = load_or_compute_features(videos=real_videos,
|
||||
extractor=extractor,
|
||||
config=config,
|
||||
cache_path=config.cache_real_features,
|
||||
cache_name="real_features")
|
||||
|
||||
if verbose:
|
||||
print(f"\n{'='*70}")
|
||||
print("Extracting GENERATED video features...")
|
||||
print(f"{'='*70}")
|
||||
|
||||
gen_features = load_or_compute_features(videos=gen_videos,
|
||||
extractor=extractor,
|
||||
config=config,
|
||||
cache_path=None,
|
||||
cache_name="gen_features")
|
||||
|
||||
if verbose:
|
||||
print(f"\nReal videos/clips: {len(real_features)}")
|
||||
print(f"Generated videos/clips: {len(gen_features)}")
|
||||
print(f"\n{'='*70}")
|
||||
print("Computing statistics...")
|
||||
print(f"{'='*70}")
|
||||
|
||||
mu_real, sigma_real = compute_statistics(real_features)
|
||||
mu_gen, sigma_gen = compute_statistics(gen_features)
|
||||
|
||||
if verbose:
|
||||
print(f"\n{'='*70}")
|
||||
print("Computing Fréchet distance...")
|
||||
print(f"{'='*70}")
|
||||
|
||||
fvd = compute_frechet_distance(mu_real, sigma_real, mu_gen, sigma_gen)
|
||||
|
||||
if verbose:
|
||||
print(f"\n{'='*70}")
|
||||
print(f"FVD Score ({config.extractor_model.upper()}): {fvd:.4f}")
|
||||
print(f"Protocol: {config}")
|
||||
print(f"{'='*70}\n")
|
||||
|
||||
results = {
|
||||
'fvd': fvd,
|
||||
'protocol': str(config),
|
||||
'model': config.extractor_model,
|
||||
'config': config.to_dict(),
|
||||
}
|
||||
|
||||
return results
|
||||
|
||||
|
||||
def compute_fvd(real_videos: str | Path | torch.Tensor,
|
||||
gen_videos: str | Path | torch.Tensor,
|
||||
num_frames: int = 16,
|
||||
batch_size: int = 32,
|
||||
device: str = 'cuda',
|
||||
num_videos: int | None = 2048,
|
||||
cache_real_features: str | None = None,
|
||||
i3d_model_path: str | None = None,
|
||||
seed: int | None = None,
|
||||
verbose: bool = True) -> float:
|
||||
"""
|
||||
Backward compatibility wrapper for computing FVD (defaults to I3D).
|
||||
"""
|
||||
num_videos = num_videos if num_videos is not None else 2048
|
||||
|
||||
config = FVDConfig(
|
||||
num_videos=num_videos,
|
||||
num_frames_per_clip=num_frames,
|
||||
extractor_model='i3d', # Default to I3D
|
||||
batch_size=batch_size,
|
||||
device=device,
|
||||
cache_real_features=cache_real_features,
|
||||
i3d_model_path=i3d_model_path,
|
||||
seed=seed,
|
||||
)
|
||||
|
||||
result = compute_fvd_with_config(real_videos, gen_videos, config, verbose)
|
||||
return result['fvd']
|
||||
@@ -1,124 +0,0 @@
|
||||
"""I3D Feature Extractor for FVD Computation"""
|
||||
|
||||
import torch
|
||||
import torch.nn as nn
|
||||
import torch.nn.functional as F
|
||||
from pathlib import Path
|
||||
from huggingface_hub import hf_hub_download
|
||||
from tqdm import tqdm
|
||||
from contextlib import suppress
|
||||
|
||||
|
||||
class I3DFeatureExtractor(nn.Module):
|
||||
"""
|
||||
I3D feature extractor for FVD computation.
|
||||
Extracts 400-dimensional features from videos using I3D model
|
||||
trained on Kinetics-400.
|
||||
"""
|
||||
|
||||
REPO_ID = 'flateon/FVD-I3D-torchscript'
|
||||
MODEL_FILENAME = 'i3d_torchscript.pt'
|
||||
|
||||
def __init__(self, device: str = 'cuda', cache_dir: str | Path | None = None):
|
||||
super().__init__()
|
||||
|
||||
self.device_str = device
|
||||
if device == 'cuda' and not torch.cuda.is_available():
|
||||
print("Warning: CUDA requested but not available – falling back to CPU")
|
||||
self.device = torch.device('cpu')
|
||||
else:
|
||||
self.device = torch.device(device)
|
||||
|
||||
self.cache_dir: str | None
|
||||
if cache_dir is not None:
|
||||
self.cache_dir = str(Path(cache_dir).resolve())
|
||||
else:
|
||||
self.cache_dir = None # Use HF default cache
|
||||
|
||||
self.model = self._load_model()
|
||||
self.model.eval()
|
||||
|
||||
with suppress(Exception):
|
||||
self.model.to(self.device)
|
||||
|
||||
def _load_model(self) -> torch.nn.Module:
|
||||
"""Download and load I3D TorchScript model from Hugging Face Hub."""
|
||||
print(f"Loading I3D model from Hugging Face Hub ({self.REPO_ID})...")
|
||||
|
||||
try:
|
||||
# Download model from Hugging Face Hub
|
||||
model_path = hf_hub_download(repo_id=self.REPO_ID, filename=self.MODEL_FILENAME, cache_dir=self.cache_dir)
|
||||
|
||||
# Load directly to chosen device
|
||||
model = torch.jit.load(model_path, map_location=self.device)
|
||||
print("I3D model loaded successfully")
|
||||
return model
|
||||
|
||||
except Exception as e:
|
||||
raise RuntimeError(f"Failed to load I3D model from Hugging Face Hub. Error: {e}\n"
|
||||
f"Ensure you have internet connection and huggingface_hub installed:\n"
|
||||
f"uv pip install huggingface_hub") from e
|
||||
|
||||
def preprocess(self, videos: torch.Tensor) -> torch.Tensor:
|
||||
"""
|
||||
Preprocess videos for I3D.
|
||||
|
||||
Args:
|
||||
videos: [B, T, C, H, W], values in [0, 255]
|
||||
|
||||
Returns:
|
||||
Preprocessed videos [B, C, T, 224, 224] (normalized and resized)
|
||||
"""
|
||||
B, T, C, H, W = videos.shape
|
||||
|
||||
if T < 10:
|
||||
raise ValueError(f"I3D requires at least 10 frames, got {T}")
|
||||
|
||||
# Normalize to [0, 1] if needed
|
||||
if videos.max() > 1.0:
|
||||
videos = videos / 255.0
|
||||
|
||||
# Resize to 224x224 if needed
|
||||
if H != 224 or W != 224:
|
||||
videos = videos.reshape(B * T, C, H, W)
|
||||
videos = F.interpolate(videos, size=(224, 224), mode='bilinear', align_corners=False)
|
||||
videos = videos.reshape(B, T, C, 224, 224)
|
||||
|
||||
# Convert to [B, C, T, H, W] format
|
||||
videos = videos.permute(0, 2, 1, 3, 4).contiguous()
|
||||
|
||||
return videos
|
||||
|
||||
@torch.no_grad()
|
||||
def extract_features(self, videos: torch.Tensor, batch_size: int = 32, verbose: bool = True) -> torch.Tensor:
|
||||
"""
|
||||
Extract I3D features
|
||||
|
||||
Args:
|
||||
videos: [N, T, C, H, W], values in [0, 255]
|
||||
batch_size: Batch size for processing
|
||||
verbose: Show progress bar
|
||||
|
||||
Returns:
|
||||
Features [N, 400]
|
||||
"""
|
||||
N = len(videos)
|
||||
all_features = []
|
||||
|
||||
iterator = range(0, N, batch_size)
|
||||
if verbose:
|
||||
iterator = tqdm(iterator, desc="Extracting I3D features")
|
||||
|
||||
for i in iterator:
|
||||
batch = videos[i:i + batch_size].to(self.device)
|
||||
batch = self.preprocess(batch) # Now returns [B, C, T, H, W]
|
||||
|
||||
# Use the HF model without rescale/resize (we handle it in preprocess)
|
||||
features = self.model(batch, rescale=False, resize=False, return_features=True)
|
||||
|
||||
all_features.append(features.cpu())
|
||||
|
||||
return torch.cat(all_features, dim=0)
|
||||
|
||||
def __call__(self, videos: torch.Tensor, batch_size: int = 32) -> torch.Tensor:
|
||||
return self.extract_features(videos, batch_size=batch_size)
|
||||
@@ -1,51 +0,0 @@
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
root_dir = Path(__file__).parent.parent.parent
|
||||
sys.path.insert(0, str(root_dir))
|
||||
|
||||
from benchmarks.fvd.fvd import FVDConfig, compute_fvd_with_config # noqa: E402
|
||||
|
||||
|
||||
def main() -> None:
|
||||
script_dir = Path(__file__).parent.resolve()
|
||||
|
||||
# Define directories
|
||||
real_dir = "benchmarks/data/real_videos"
|
||||
gen_dir = "benchmarks/data/generated_videos"
|
||||
|
||||
# Compare all 3 models
|
||||
models_to_test = ['i3d', 'clip', 'videomae']
|
||||
|
||||
print(f"\n{'='*60}")
|
||||
print("STARTING COMPARISON BENCHMARK")
|
||||
print(f"{'='*60}")
|
||||
|
||||
for model_name in models_to_test:
|
||||
print(f"\n>>> Running evaluation with {model_name.upper()}...")
|
||||
|
||||
try:
|
||||
cfg = FVDConfig(
|
||||
num_videos=650,
|
||||
num_frames_per_clip=16,
|
||||
extractor_model=model_name,
|
||||
clip_strategy='beginning',
|
||||
device='cuda',
|
||||
seed=42,
|
||||
# Use separate cache folders for each model to avoid conflicts
|
||||
cache_real_features=str(script_dir / f'fvd-cache/{model_name}'),
|
||||
)
|
||||
|
||||
results = compute_fvd_with_config(real_dir, gen_dir, cfg, verbose=False)
|
||||
print(f"FVD: {results['fvd']}\nModel: {results['model']}")
|
||||
|
||||
except Exception as e:
|
||||
print(f"{model_name.upper()} Failed: {e}")
|
||||
|
||||
print(f"\n{'='*60}")
|
||||
print("BENCHMARK COMPLETE")
|
||||
print(f"{'='*60}")
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -1,89 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
import sys
|
||||
from pathlib import Path
|
||||
import shutil
|
||||
import random
|
||||
from fvd import compute_fvd_with_config, FVDConfig
|
||||
|
||||
script_path = Path(__file__).resolve()
|
||||
fastvideo_root = script_path.parent.parent.parent
|
||||
sys.path.insert(0, str(fastvideo_root))
|
||||
|
||||
|
||||
def split_videos(video_dir: Path, n_per_subset: int = 128, seed: int = 42):
|
||||
subset_a = video_dir.parent / 'bair_full_subset_A'
|
||||
subset_b = video_dir.parent / 'bair_full_subset_B'
|
||||
|
||||
if subset_a.exists():
|
||||
shutil.rmtree(subset_a)
|
||||
if subset_b.exists():
|
||||
shutil.rmtree(subset_b)
|
||||
|
||||
subset_a.mkdir(parents=True)
|
||||
subset_b.mkdir(parents=True)
|
||||
|
||||
videos = sorted(video_dir.glob('*.mp4'))
|
||||
|
||||
random.seed(seed)
|
||||
shuffled = list(videos)
|
||||
random.shuffle(shuffled)
|
||||
|
||||
needed = n_per_subset * 2
|
||||
if len(shuffled) > needed:
|
||||
shuffled = shuffled[:needed]
|
||||
|
||||
mid = len(shuffled) // 2
|
||||
|
||||
print(f"\nSplitting {len(shuffled)} BAIR FULL videos:")
|
||||
print(f" Subset A: {mid} videos")
|
||||
print(f" Subset B: {len(shuffled) - mid} videos")
|
||||
|
||||
for v in shuffled[:mid]:
|
||||
shutil.copy2(v, subset_a / v.name)
|
||||
|
||||
for v in shuffled[mid:]:
|
||||
shutil.copy2(v, subset_b / v.name)
|
||||
|
||||
return subset_a, subset_b, mid
|
||||
|
||||
|
||||
def validate_fvd(subset_a: Path, subset_b: Path, num_videos: int):
|
||||
config = FVDConfig(num_videos=num_videos,
|
||||
num_frames_per_clip=16,
|
||||
clip_strategy='beginning',
|
||||
batch_size=8,
|
||||
device='cuda',
|
||||
seed=42)
|
||||
|
||||
print("\n" + "=" * 70)
|
||||
print("TEST 1: Identity Test")
|
||||
print("=" * 70)
|
||||
|
||||
result1 = compute_fvd_with_config(real_videos=str(subset_a), gen_videos=str(subset_a), config=config, verbose=False)
|
||||
fvd_identity = result1['fvd']
|
||||
print(f"\nIdentity FVD: {fvd_identity:.2f}")
|
||||
|
||||
print("\n" + "=" * 70)
|
||||
print("TEST 2: Real vs Real")
|
||||
print("=" * 70)
|
||||
|
||||
result2 = compute_fvd_with_config(real_videos=str(subset_a), gen_videos=str(subset_b), config=config, verbose=False)
|
||||
fvd_real = result2['fvd']
|
||||
print(f"\nReal vs Real FVD: {fvd_real:.2f}")
|
||||
|
||||
print("\n" + "=" * 70)
|
||||
print("RESULTS")
|
||||
print("=" * 70)
|
||||
print(f"Identity: {fvd_identity:.2f}")
|
||||
print(f"Real vs Real: {fvd_real:.2f}")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
bair_dir = Path('benchmarks/data/bair_full_videos')
|
||||
|
||||
subset_a, subset_b, count = split_videos(bair_dir, n_per_subset=128, seed=42)
|
||||
validate_fvd(subset_a, subset_b, count)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -1,460 +0,0 @@
|
||||
import torch
|
||||
import cv2
|
||||
import numpy as np
|
||||
from pathlib import Path
|
||||
from collections.abc import Iterator
|
||||
from tqdm import tqdm
|
||||
from enum import Enum
|
||||
|
||||
|
||||
class ClipSamplingStrategy(Enum):
|
||||
"""Clip sampling strategies for FVD evaluation."""
|
||||
BEGINNING = 'beginning' # Take first N frames (most common)
|
||||
RANDOM = 'random' # Random N consecutive frames
|
||||
UNIFORM = 'uniform' # Uniformly spaced frames across video
|
||||
MIDDLE = 'middle' # Middle N frames
|
||||
SLIDING = 'sliding' # Multiple sliding windows
|
||||
ALL = 'all' # All possible clips
|
||||
|
||||
|
||||
def _load_video_cv2(video_path: str | Path,
|
||||
num_frames: int | None = 16,
|
||||
sample_strategy: str = 'uniform') -> torch.Tensor:
|
||||
"""
|
||||
Load video from video file using OpenCV.
|
||||
|
||||
Args:
|
||||
video_path: Path to video file (MP4, AVI, MOV, MKV)
|
||||
num_frames: Number of frames to extract
|
||||
sample_strategy: 'uniform' or 'random'
|
||||
|
||||
Returns:
|
||||
video: [T, C, H, W]
|
||||
"""
|
||||
video_path = str(video_path)
|
||||
cap = cv2.VideoCapture(video_path)
|
||||
|
||||
if not cap.isOpened():
|
||||
raise RuntimeError(f"Cannot open video: {video_path}")
|
||||
|
||||
frames = []
|
||||
total_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
|
||||
|
||||
if num_frames is None:
|
||||
# Read all available frames
|
||||
while True:
|
||||
ret, frame = cap.read()
|
||||
if not ret:
|
||||
break
|
||||
frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
|
||||
frames.append(frame)
|
||||
|
||||
cap.release()
|
||||
if len(frames) == 0:
|
||||
raise RuntimeError(f"Video has 0 frames: {video_path}")
|
||||
|
||||
frames = np.stack(frames) # [T, H, W, C]
|
||||
frames = torch.from_numpy(frames).permute(0, 3, 1, 2).float() # [T, C, H, W]
|
||||
return frames
|
||||
|
||||
if total_frames == 0:
|
||||
raise RuntimeError(f"Video has 0 frames: {video_path}")
|
||||
|
||||
# Determine frame indices for sampling
|
||||
if total_frames < num_frames:
|
||||
frame_indices = list(range(total_frames)) + [total_frames - 1] * (num_frames - total_frames)
|
||||
elif sample_strategy == 'uniform':
|
||||
frame_indices = np.linspace(0, total_frames - 1, num_frames, dtype=int).tolist()
|
||||
elif sample_strategy == 'random':
|
||||
frame_indices = sorted(np.random.choice(total_frames, num_frames, replace=False))
|
||||
else:
|
||||
raise ValueError(f"Unknown sample_strategy: {sample_strategy}")
|
||||
|
||||
# Extract frames
|
||||
for idx in frame_indices:
|
||||
cap.set(cv2.CAP_PROP_POS_FRAMES, idx)
|
||||
ret, frame = cap.read()
|
||||
|
||||
if not ret:
|
||||
if len(frames) > 0:
|
||||
frames.append(frames[-1].copy())
|
||||
else:
|
||||
h = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
|
||||
w = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
|
||||
frames.append(np.zeros((h, w, 3), dtype=np.uint8))
|
||||
continue
|
||||
|
||||
frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
|
||||
frames.append(frame)
|
||||
|
||||
cap.release()
|
||||
|
||||
frames = np.stack(frames) # [T, H, W, C]
|
||||
frames = torch.from_numpy(frames).permute(0, 3, 1, 2).float() # [T, C, H, W]
|
||||
|
||||
return frames
|
||||
|
||||
|
||||
def _load_video_from_frames(frame_dir: str | Path,
|
||||
num_frames: int | None = 16,
|
||||
sample_strategy: str = 'uniform',
|
||||
frame_extensions: list[str] | None = None) -> torch.Tensor:
|
||||
"""
|
||||
Load video from directory of frame images.
|
||||
|
||||
Args:
|
||||
frame_dir: Directory containing frames
|
||||
num_frames: Number of frames to sample
|
||||
sample_strategy: 'uniform' or 'random'
|
||||
frame_extensions: Image file extensions to look for
|
||||
|
||||
Returns:
|
||||
video: [T, C, H, W]
|
||||
"""
|
||||
if frame_extensions is None:
|
||||
frame_extensions = ['.jpg', '.png', '.jpeg', '.bmp']
|
||||
|
||||
frame_dir = Path(frame_dir)
|
||||
|
||||
if not frame_dir.exists():
|
||||
raise FileNotFoundError(f"Frame directory not found: {frame_dir}")
|
||||
|
||||
# Find all frames
|
||||
frame_files: list[Path] = []
|
||||
for ext in frame_extensions:
|
||||
frame_files.extend(frame_dir.glob(f"*{ext}"))
|
||||
|
||||
if len(frame_files) == 0:
|
||||
raise ValueError(f"No frames found in {frame_dir} with extensions {frame_extensions}")
|
||||
|
||||
frame_files = sorted(frame_files, key=lambda x: x.name)
|
||||
total_frames = len(frame_files)
|
||||
|
||||
# Determine frame indices
|
||||
if num_frames is None:
|
||||
frame_indices = list(range(total_frames))
|
||||
else:
|
||||
if total_frames < num_frames:
|
||||
frame_indices = list(range(total_frames)) + [total_frames - 1] * (num_frames - total_frames)
|
||||
elif sample_strategy == 'uniform':
|
||||
frame_indices = np.linspace(0, total_frames - 1, num_frames, dtype=int).tolist()
|
||||
elif sample_strategy == 'random':
|
||||
frame_indices = sorted(np.random.choice(total_frames, num_frames, replace=False))
|
||||
else:
|
||||
raise ValueError(f"Unknown sample_strategy: {sample_strategy}")
|
||||
|
||||
# Load frames
|
||||
frames = []
|
||||
for idx in frame_indices:
|
||||
frame_path = frame_files[idx]
|
||||
frame = cv2.imread(str(frame_path))
|
||||
|
||||
if frame is None:
|
||||
raise RuntimeError(f"Failed to load frame: {frame_path}")
|
||||
|
||||
frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
|
||||
frames.append(frame)
|
||||
|
||||
# Stack and convert to tensor
|
||||
frames = np.stack(frames) # [T, H, W, C]
|
||||
frames = torch.from_numpy(frames).permute(0, 3, 1, 2).float() # [T, C, H, W]
|
||||
|
||||
return frames
|
||||
|
||||
|
||||
def _detect_video_format(path: str | Path) -> str:
|
||||
"""
|
||||
Detect if path is a video file or frame directory.
|
||||
|
||||
Returns:
|
||||
'video_file', 'frame_directory', or 'unknown'
|
||||
"""
|
||||
path = Path(path)
|
||||
|
||||
if path.is_file():
|
||||
return 'video_file'
|
||||
elif path.is_dir():
|
||||
# Check if contains image files
|
||||
image_extensions = ['.jpg', '.jpeg', '.png', '.bmp']
|
||||
for ext in image_extensions:
|
||||
if list(path.glob(f"*{ext}")):
|
||||
return 'frame_directory'
|
||||
return 'unknown'
|
||||
else:
|
||||
raise ValueError(f"Path does not exist: {path}")
|
||||
|
||||
|
||||
def load_video_auto(video_path: str | Path,
|
||||
num_frames: int | None = 16,
|
||||
sample_strategy: str = 'uniform') -> torch.Tensor:
|
||||
"""
|
||||
Automatically detect format and load video.
|
||||
|
||||
Supports:
|
||||
- Video files (MP4, AVI, MOV, MKV)
|
||||
- Frame directories (JPG, PNG)
|
||||
|
||||
Args:
|
||||
video_path: Path to video file or frame directory
|
||||
num_frames: Number of frames to extract
|
||||
sample_strategy: 'uniform' or 'random'
|
||||
|
||||
Returns:
|
||||
video: [T, C, H, W]
|
||||
"""
|
||||
format_type = _detect_video_format(video_path)
|
||||
|
||||
if format_type == 'video_file':
|
||||
return _load_video_cv2(video_path, num_frames, sample_strategy)
|
||||
elif format_type == 'frame_directory':
|
||||
return _load_video_from_frames(video_path, num_frames, sample_strategy)
|
||||
else:
|
||||
raise ValueError(f"Unknown video format at {video_path}")
|
||||
|
||||
|
||||
def sample_clips_from_video(video: torch.Tensor,
|
||||
num_frames_per_clip: int = 16,
|
||||
num_clips: int = 1,
|
||||
strategy: str | ClipSamplingStrategy = ClipSamplingStrategy.BEGINNING,
|
||||
frame_stride: int = 1,
|
||||
temporal_stride: int = 1) -> list[torch.Tensor]:
|
||||
"""
|
||||
Sample clips from a video with various strategies.
|
||||
|
||||
Args:
|
||||
video: [T, C, H, W] full video
|
||||
num_frames_per_clip: Frames per clip
|
||||
num_clips: Number of clips to extract
|
||||
strategy: ClipSamplingStrategy or string ('beginning', 'random', etc.)
|
||||
frame_stride: Skip frames (FPS control: 1=all, 2=every 2nd, 8=every 8th)
|
||||
temporal_stride: Stride between clips for sliding window
|
||||
|
||||
Returns:
|
||||
List of clips, each [num_frames_per_clip, C, H, W]
|
||||
|
||||
Examples:
|
||||
>>> # Beginning clip (most common for FVD)
|
||||
>>> clips = sample_clips_from_video(video, 16, strategy='beginning')
|
||||
|
||||
>>> # Multiple random clips
|
||||
>>> clips = sample_clips_from_video(video, 16, num_clips=4, strategy='random')
|
||||
|
||||
>>> # Subsample FPS by 2x (every 2nd frame)
|
||||
>>> clips = sample_clips_from_video(video, 16, frame_stride=2)
|
||||
|
||||
>>> # Sliding window with overlap
|
||||
>>> clips = sample_clips_from_video(video, 16, strategy='sliding', temporal_stride=8)
|
||||
"""
|
||||
# Convert string to enum if needed
|
||||
if isinstance(strategy, str):
|
||||
strategy = ClipSamplingStrategy(strategy)
|
||||
|
||||
T, C, H, W = video.shape
|
||||
|
||||
# Apply frame stride (FPS subsampling)
|
||||
if frame_stride > 1:
|
||||
video = video[::frame_stride]
|
||||
T = len(video)
|
||||
|
||||
effective_clip_length = num_frames_per_clip
|
||||
|
||||
# Handle videos shorter than clip length
|
||||
if effective_clip_length > T:
|
||||
pad_length = effective_clip_length - T
|
||||
last_frame = video[-1:].repeat(pad_length, 1, 1, 1)
|
||||
video = torch.cat([video, last_frame], dim=0)
|
||||
T = len(video)
|
||||
|
||||
clips = []
|
||||
|
||||
if strategy == ClipSamplingStrategy.BEGINNING:
|
||||
# Take first clip (most common for FVD evaluation)
|
||||
clip = video[:effective_clip_length]
|
||||
clips.append(clip)
|
||||
|
||||
elif strategy == ClipSamplingStrategy.MIDDLE:
|
||||
# Take middle clip
|
||||
start = (T - effective_clip_length) // 2
|
||||
clip = video[start:start + effective_clip_length]
|
||||
clips.append(clip)
|
||||
|
||||
elif strategy == ClipSamplingStrategy.RANDOM:
|
||||
# Sample N random clips
|
||||
for _ in range(num_clips):
|
||||
start = 0 if effective_clip_length == T else np.random.randint(0, T - effective_clip_length + 1)
|
||||
clip = video[start:start + effective_clip_length]
|
||||
clips.append(clip)
|
||||
|
||||
elif strategy == ClipSamplingStrategy.UNIFORM:
|
||||
# Uniformly spaced clips
|
||||
if num_clips == 1:
|
||||
# Single clip from middle
|
||||
start = (T - effective_clip_length) // 2
|
||||
clip = video[start:start + effective_clip_length]
|
||||
clips.append(clip)
|
||||
else:
|
||||
# Multiple uniformly spaced clips
|
||||
step = (T - effective_clip_length) / (num_clips - 1) if num_clips > 1 else 0
|
||||
for i in range(num_clips):
|
||||
start = int(i * step)
|
||||
start = min(start, T - effective_clip_length)
|
||||
clip = video[start:start + effective_clip_length]
|
||||
clips.append(clip)
|
||||
|
||||
elif strategy == ClipSamplingStrategy.SLIDING:
|
||||
# Sliding window with stride
|
||||
for start in range(0, T - effective_clip_length + 1, temporal_stride):
|
||||
clip = video[start:start + effective_clip_length]
|
||||
clips.append(clip)
|
||||
if len(clips) >= num_clips:
|
||||
break
|
||||
|
||||
elif strategy == ClipSamplingStrategy.ALL:
|
||||
# All possible clips (overlapping)
|
||||
for start in range(T - effective_clip_length + 1):
|
||||
clip = video[start:start + effective_clip_length]
|
||||
clips.append(clip)
|
||||
|
||||
else:
|
||||
raise ValueError(f"Unknown strategy: {strategy}")
|
||||
|
||||
return clips
|
||||
|
||||
|
||||
def load_video_clips_streaming(directory: str | Path,
|
||||
num_frames: int = 16,
|
||||
max_videos: int | None = None,
|
||||
clip_strategy: str
|
||||
| ClipSamplingStrategy = 'beginning',
|
||||
frame_stride: int = 1,
|
||||
num_clips_per_video: int = 1,
|
||||
video_extensions: list[str] | None = None,
|
||||
support_frame_dirs: bool = True,
|
||||
target_size: tuple[int, int] | None = (224, 224),
|
||||
verbose: bool = True) -> Iterator[torch.Tensor]:
|
||||
"""
|
||||
This generator yields clips one-by-one instead of loading all videos into RAM.
|
||||
Perfect for large datasets where memory is limited.
|
||||
|
||||
Args:
|
||||
directory: Path to directory with videos
|
||||
num_frames: Frames per clip
|
||||
max_videos: Max videos to load
|
||||
clip_strategy: 'beginning', 'random', 'uniform', etc.
|
||||
frame_stride: Frame skip (1=all, 2=every 2nd, 8=every 8th)
|
||||
num_clips_per_video: Number of clips per video
|
||||
video_extensions: Video file extensions
|
||||
support_frame_dirs: Also load frame directories
|
||||
target_size: Resize clips to (H, W). If None, keep original size.
|
||||
verbose: Show progress
|
||||
|
||||
Yields:
|
||||
clip: [T, C, H, W] individual clips
|
||||
|
||||
Example:
|
||||
>>> for clip in load_video_clips_streaming('data/videos/', num_frames=16):
|
||||
>>> features = model.extract_features(clip.unsqueeze(0))
|
||||
>>> # Process one clip at a time - low memory usage!
|
||||
"""
|
||||
if video_extensions is None:
|
||||
video_extensions = ['.mp4', '.avi', '.mov', '.mkv']
|
||||
|
||||
directory = Path(directory)
|
||||
|
||||
if not directory.exists():
|
||||
raise FileNotFoundError(f"Directory not found: {directory}")
|
||||
|
||||
# Find video paths
|
||||
video_paths: list[Path] = []
|
||||
|
||||
# Find video files
|
||||
for ext in video_extensions:
|
||||
video_paths.extend(directory.glob(f"**/*{ext}"))
|
||||
|
||||
# Find frame directories if enabled
|
||||
if support_frame_dirs:
|
||||
for subdir in directory.iterdir():
|
||||
if subdir.is_dir():
|
||||
# Check if it contains frames
|
||||
image_extensions = ['.jpg', '.jpeg', '.png', '.bmp']
|
||||
for ext in image_extensions:
|
||||
if list(subdir.glob(f"*{ext}")):
|
||||
video_paths.append(subdir)
|
||||
break
|
||||
|
||||
if len(video_paths) == 0:
|
||||
raise ValueError(f"No videos found in {directory}")
|
||||
|
||||
video_paths = sorted(video_paths)
|
||||
|
||||
if max_videos is not None:
|
||||
video_paths = video_paths[:max_videos]
|
||||
|
||||
if verbose:
|
||||
print(f"Found {len(video_paths)} videos in {directory}")
|
||||
if num_clips_per_video > 1:
|
||||
print(f"Extracting {num_clips_per_video} clips per video...")
|
||||
if frame_stride > 1:
|
||||
print(f"Subsampling frames with stride {frame_stride}...")
|
||||
if target_size:
|
||||
print(f"Resizing clips to {target_size}...")
|
||||
|
||||
# Track statistics
|
||||
failed_count = 0
|
||||
total_clips = 0
|
||||
|
||||
iterator = tqdm(video_paths, desc="Loading videos") if verbose else video_paths
|
||||
|
||||
for video_path in iterator:
|
||||
try:
|
||||
# Load full video
|
||||
video = load_video_auto(video_path, num_frames=None, sample_strategy='uniform')
|
||||
|
||||
# Sample clips from video
|
||||
clips = sample_clips_from_video(video,
|
||||
num_frames_per_clip=num_frames,
|
||||
num_clips=num_clips_per_video,
|
||||
strategy=clip_strategy,
|
||||
frame_stride=frame_stride)
|
||||
|
||||
if target_size is not None:
|
||||
resized_clips = []
|
||||
for clip in clips:
|
||||
T, C, H, W = clip.shape
|
||||
if target_size != (H, W):
|
||||
# Resize to target size
|
||||
clip = clip.contiguous() # Fix non-contiguous tensors first
|
||||
clip_flat = clip.view(T * C, H, W).unsqueeze(0) # [1, T*C, H, W]
|
||||
clip_resized = torch.nn.functional.interpolate(clip_flat,
|
||||
size=target_size,
|
||||
mode='bilinear',
|
||||
align_corners=False)
|
||||
clip = clip_resized.squeeze(0).view(T, C, target_size[0],
|
||||
target_size[1]) # Back to [T, C, H, W]
|
||||
resized_clips.append(clip)
|
||||
clips = resized_clips
|
||||
|
||||
# Yield clips one by one
|
||||
for clip in clips:
|
||||
yield clip
|
||||
total_clips += 1
|
||||
|
||||
# Free memory
|
||||
del video, clips
|
||||
|
||||
except Exception as e:
|
||||
failed_count += 1
|
||||
if verbose:
|
||||
print(f"\nWarning: Failed to load {video_path}: {e}")
|
||||
continue
|
||||
|
||||
# Validate
|
||||
if total_clips == 0:
|
||||
raise RuntimeError(f"Failed to load any videos from {directory}")
|
||||
|
||||
failure_rate = failed_count / len(video_paths)
|
||||
if failure_rate > 0.1: # More than 10% failed
|
||||
print(f"\nWARNING: {failure_rate:.1%} of videos failed to load ({failed_count}/{len(video_paths)})")
|
||||
|
||||
if verbose:
|
||||
print(f"\nSuccessfully loaded {total_clips} clips from {len(video_paths) - failed_count} videos")
|
||||
@@ -1,7 +0,0 @@
|
||||
#!/bin/bash
|
||||
|
||||
# 1. Install missing dependency
|
||||
uv pip install -q opencv-python-headless transformers huggingface_hub
|
||||
|
||||
# 2. Run FVD script
|
||||
python benchmarks/fvd/run_fvd.py
|
||||
@@ -1,4 +0,0 @@
|
||||
#!/bin/bash
|
||||
|
||||
# 1. Install missing dependency
|
||||
uv pip install -q opencv-python-headless
|
||||
@@ -85,6 +85,124 @@ Each line in `/tmp/fv_trace.jsonl` is a JSON record:
|
||||
5. The first divergent line identifies the first layer where FastVideo and the
|
||||
upstream produce different outputs. Start debugging there.
|
||||
|
||||
## Performance impact
|
||||
|
||||
When the master toggle is off, overhead is nil. When on, the cost is
|
||||
proportional to how broadly the layer regex matches.
|
||||
|
||||
### What runs when tracing is on
|
||||
|
||||
For every match against `model.named_modules()`, FastVideo registers an
|
||||
`ActivationStatHook` that runs after the module's forward returns:
|
||||
|
||||
1. Walks the (possibly nested) output `tuple` / `list` / `dict` and extracts
|
||||
every `torch.Tensor` leaf.
|
||||
2. Computes each enabled stat on the tensor's `.detach().float()` view — the
|
||||
cast is required for bf16 inputs because some reductions are not stable
|
||||
in bf16.
|
||||
3. Writes one JSON record per tensor to the line-buffered JSONL sink.
|
||||
|
||||
Cost is roughly `O(num_matched_modules × num_output_tensors × num_stats × tensor_numel)`
|
||||
per forward, dominated by `tensor_numel` for value-stats (`abs_mean`, `sum`,
|
||||
`min`, `max`, `mean`, `std`). The `shape` and `dtype` stats are O(1).
|
||||
|
||||
### Cost-shaping knobs
|
||||
|
||||
The default config (empty `FASTVIDEO_TRACE_LAYERS` is treated as `.*`, default
|
||||
two stats, all steps) is intentionally blunt — useful only for a one-shot
|
||||
smoke run. For real debugging, scope down:
|
||||
|
||||
| Knob | Effect |
|
||||
|---|---|
|
||||
| Tighten `FASTVIDEO_TRACE_LAYERS` to a regex matching <50 modules | Linear reduction in hook count |
|
||||
| Drop unused stats from `FASTVIDEO_TRACE_STATS` (`mean`, `std`, `min`, `max`) if you only need divergence detection | One full reduction per stat saved per matched module per forward |
|
||||
| Set `FASTVIDEO_TRACE_STEPS="0,15,31"` for a 32-step run | ~10x reduction vs all-steps (the hook still fires but exits early when the step doesn't match) |
|
||||
| Use `shape` + `dtype` only on layers where you only care about layout | Skips tensor reductions entirely on those layers |
|
||||
|
||||
### Disk
|
||||
|
||||
Output is line-buffered (`open(..., buffering=1)`), so every record flushes
|
||||
on write. A typical 32-step run that traces 40 DiT blocks with 4 stats
|
||||
writes roughly 5K records — about 1 MB of JSONL. Point
|
||||
`FASTVIDEO_TRACE_OUTPUT` at a fast local disk for parity runs; slow network
|
||||
mounts will dominate runtime once tracing is on.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "I set the env var but no JSONL file appears"
|
||||
|
||||
Three things to check, in order:
|
||||
|
||||
1. **Toggle semantics.** `FASTVIDEO_TRACE_ACTIVATIONS` uses a strict
|
||||
not-equal-to-`"0"` test. Setting it to `1`, `true`, or even `""` all
|
||||
enable tracing. Only an unset variable or `FASTVIDEO_TRACE_ACTIVATIONS=0`
|
||||
disables it.
|
||||
2. **Module exposure.** The hook attaches to `pipeline.modules.get("transformer")`
|
||||
at the end of `post_init`. If your pipeline does not expose a module
|
||||
under that key (e.g. a non-standard custom pipeline whose DiT is
|
||||
reachable only via `pipeline.modules["sr_transformer"]`), the trace is
|
||||
silently a no-op. Check the
|
||||
`Activation trace attached to N modules` log line at startup; if
|
||||
`N=0`, either the regex didn't match anything or the expected module
|
||||
isn't exposed.
|
||||
3. **Output path failure.** The parent of `FASTVIDEO_TRACE_OUTPUT` is
|
||||
auto-created. If creation fails (permissions, read-only mount),
|
||||
`JsonlSink.__init__` raises at startup — look for an `OSError` early
|
||||
in the log.
|
||||
|
||||
### "My regex isn't filtering the way I expect"
|
||||
|
||||
`FASTVIDEO_TRACE_LAYERS` is compiled with `re.compile(spec)` and matched
|
||||
with `pattern.search(name)`. Two consequences:
|
||||
|
||||
- `search`, not `fullmatch`. `block.layers` matches
|
||||
`transformer.block.layers.0.attn`. Anchor with `^...$` if you want
|
||||
exact matches.
|
||||
- Module names use Python dot notation (`transformer.block.layers.0`),
|
||||
not slashes. The `.` in your regex is a metacharacter — escape it as
|
||||
`\.` if you want a literal dot.
|
||||
|
||||
Run once with `FASTVIDEO_LOGGING_LEVEL=DEBUG` to see the
|
||||
`Activation trace attached to N modules (pattern=...)` line. `N` is the
|
||||
ground truth for how many modules survived your regex.
|
||||
|
||||
### "Stats are NaN or `<error: ...>` for some layers"
|
||||
|
||||
Some outputs hit numerical edge cases:
|
||||
|
||||
- `mean` / `std` on a 0-dim tensor returns NaN.
|
||||
- `abs_mean` on an empty tensor or one full of `inf` returns NaN.
|
||||
- Non-tensor outputs (e.g. a Python `bool` from a verification gate) are
|
||||
silently skipped — the hook only walks `torch.Tensor` leaves.
|
||||
|
||||
Dropping `mean`/`std` and using `abs_mean`/`max` is more robust. For
|
||||
layout-only debugging, `shape`+`dtype` never fail.
|
||||
|
||||
### "Tracing slows the run by 5x"
|
||||
|
||||
You're probably matching too broadly. An empty `FASTVIDEO_TRACE_LAYERS` is
|
||||
treated as `.*` and matches every named module — for a 15B-param DiT
|
||||
that's hundreds of submodules, each running stat reductions on every
|
||||
forward. Tighten to a single block depth: `^transformer\.blocks\.\d+$`
|
||||
typically matches a few dozen modules, which is a manageable trace.
|
||||
|
||||
### "Trace records are missing tensors I expect"
|
||||
|
||||
The hook walks `tuple` / `list` / `dict` outputs recursively but does
|
||||
**not** unpack custom dataclasses or named tuples — those are silently
|
||||
skipped. If your module returns
|
||||
`BlockOutput(hidden_states=..., attn_logits=...)`, no records are
|
||||
emitted. Workaround: either return a plain `dict` (`{"hidden_states": ..., "attn_logits": ...}`)
|
||||
or attach the hook to a deeper module that already returns a raw tensor.
|
||||
|
||||
### "I want tracing inside a `torch.compile`'d region"
|
||||
|
||||
Module forward hooks run on the eager wrapper. If the entire module is
|
||||
compiled, the hook sees only the wrapped op's output, not internal FX
|
||||
nodes. This is by design — Extension 1 (FX backend rewrite) in
|
||||
[Future extensions](#future-extensions-design-only-not-yet-implemented)
|
||||
is the planned path when inside-graph granularity is required.
|
||||
|
||||
## Architecture (Extension 0: module forward hooks)
|
||||
|
||||
At pipeline initialization, `attach_activation_trace()` reads the env vars once.
|
||||
|
||||
@@ -73,6 +73,7 @@ replicate CI results before pushing.
|
||||
| Transformer Tests | `transformer` | `fastvideo/models/dits/**`, `fastvideo/models/loader/**`, `fastvideo/tests/transformers/**`, `fastvideo/layers/**`, `fastvideo/attention/**`, `pyproject.toml`, `docker/Dockerfile.python3.12` |
|
||||
| Kernel Tests | `kernel_tests` | `fastvideo-kernel/**`, `pyproject.toml`, `docker/Dockerfile.python3.12` |
|
||||
| Unit Tests | `unit_test` | `fastvideo/**`, `.buildkite/**`, `.github/**`, `pyproject.toml`, `docker/Dockerfile.python3.12` |
|
||||
| DreamVerse App Tests | `dreamverse_app` | `apps/dreamverse/**`, `pyproject.toml` |
|
||||
|
||||
A Fastcheck failure means a component-level regression. Check the Buildkite build log for the
|
||||
failing test's output.
|
||||
@@ -99,7 +100,7 @@ failing test's output.
|
||||
| LoRA Training Tests | `training_lora` | 15 min |
|
||||
| Training Tests VSA | `training_vsa` | 15 min |
|
||||
| Inference Tests VMoBA | `inference_vmoba` | 15 min |
|
||||
| Performance Tests | `performance` | 30 min |
|
||||
| [Performance Tests](performance_benchmarks.md) | `performance` | 30 min |
|
||||
| API Server Tests | `api_server` | 30 min |
|
||||
| Train Framework Tests | `train_framework` | 30 min |
|
||||
|
||||
@@ -268,6 +269,7 @@ Triggers a specific Buildkite test or suite on the current PR branch.
|
||||
| `/test transformer` | Transformer Tests (Fastcheck) | `transformer` |
|
||||
| `/test kernel` | Kernel Tests (Fastcheck) | `kernel_tests` |
|
||||
| `/test unit` | Unit Tests (Fastcheck) | `unit_test` |
|
||||
| `/test dreamverse` | DreamVerse App Tests (Fastcheck) | `dreamverse_app` |
|
||||
| `/test ssim` | SSIM regression tests | `ssim` |
|
||||
| `/test training` | Training pipeline tests | `training` |
|
||||
| `/test lora-inference` | LoRA inference tests | `inference_lora` |
|
||||
|
||||
@@ -34,9 +34,9 @@ Run frontend commands from the standalone web app:
|
||||
|
||||
```bash
|
||||
cd apps/dreamverse/web
|
||||
pnpm install --frozen-lockfile
|
||||
pnpm run build
|
||||
pnpm run test
|
||||
npm ci
|
||||
npm run build
|
||||
npm test
|
||||
```
|
||||
|
||||
Playwright is intentionally run against a live backend as part of the GPU4
|
||||
|
||||
@@ -554,10 +554,6 @@ scores ± tolerance, and add a calibration test under
|
||||
|
||||
## 10) When not to add a metric
|
||||
|
||||
- **Set-vs-set distribution metrics** (FVD, FID-style) do not fit
|
||||
`BaseMetric.compute(sample)` cleanly; they need a population.
|
||||
Adding them requires a stateful accumulator interface that does
|
||||
not exist yet. Open an issue first.
|
||||
- **Metrics requiring a single-GPU model larger than available
|
||||
memory.** Eval is not the place for tensor-parallel sharding;
|
||||
metrics are expected to fit on one GPU.
|
||||
|
||||
@@ -0,0 +1,312 @@
|
||||
# Performance Benchmarks
|
||||
|
||||
FastVideo's performance benchmark suite measures end-to-end inference latency,
|
||||
throughput, peak GPU memory, and component-level pipeline timings for
|
||||
representative pipeline configurations. It tracks those metrics over time
|
||||
against a rolling baseline stored on the Hugging Face Hub.
|
||||
|
||||
It serves three audiences:
|
||||
|
||||
* **CI** — gates pull requests against a per-GPU static threshold and a
|
||||
rolling-median regression check.
|
||||
* **Maintainers** — surfaces regressions in a Markdown summary on every
|
||||
performance build and a long-form Plotly dashboard.
|
||||
* **Local developers** — lets you run the same benchmark on your own machine,
|
||||
then compare against the historical baseline for the same model and GPU.
|
||||
|
||||
## Quick start (local)
|
||||
|
||||
```bash
|
||||
# Run all benchmarks; writes raw perf_*.json under
|
||||
# fastvideo/tests/performance/results/
|
||||
pytest fastvideo/tests/performance/ -vs
|
||||
|
||||
# Optional: compare against the rolling HF baseline (read-only outside CI).
|
||||
# PERF_REPORTS_DIR defaults to /root/data/perf_reports for Modal/CI, so
|
||||
# override it when running outside the container.
|
||||
PERF_REPORTS_DIR=/tmp/fastvideo_perf_reports \
|
||||
python fastvideo/tests/performance/compare_baseline.py
|
||||
|
||||
# Optional: build the Plotly dashboard locally.
|
||||
PERF_REPORTS_DIR=/tmp/fastvideo_perf_reports \
|
||||
python fastvideo/tests/performance/dashboard.py
|
||||
```
|
||||
|
||||
The pytest run never uploads anything. `compare_baseline.py` only writes to
|
||||
the HF dataset when `TEST_SCOPE=full` *and* `BUILDKITE_BRANCH=main`, so local
|
||||
runs are always read-only. The report directory default is container-oriented;
|
||||
set `PERF_REPORTS_DIR` to a writable local path when generating dashboards or
|
||||
when you want local Markdown/normalized-result artifacts from the comparator.
|
||||
`compare_baseline.py` reads every `perf_*.json` currently present in
|
||||
`fastvideo/tests/performance/results/`; remove stale result files if you only
|
||||
want to compare the latest local run.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
.buildkite/performance-benchmarks/tests/*.json
|
||||
└── per-benchmark configs: model, gen kwargs, per-GPU thresholds
|
||||
|
||||
fastvideo/tests/performance/
|
||||
├── test_inference_performance.py
|
||||
│ └── pytest test that runs each config, writes perf_*.json
|
||||
│ with latency, memory, throughput, and component timings
|
||||
├── compare_baseline.py
|
||||
│ └── normalizes raw results, compares against HF rolling baseline,
|
||||
│ writes Markdown summary + (optionally) uploads new records
|
||||
├── dashboard.py
|
||||
│ └── builds time-series Plotly HTML from HF history
|
||||
└── hf_store.py # shared HF I/O + DataFrame helpers
|
||||
```
|
||||
|
||||
The HF dataset (`FastVideo/performance-tracking` by default) holds one
|
||||
normalized JSON per `(model_id, gpu_type, run)` tuple. The rolling baseline is
|
||||
the median of the last 5 successful records for that model+GPU.
|
||||
|
||||
## Planned Coverage
|
||||
|
||||
The current rollout tracks a small set of representative inference workloads.
|
||||
Broader coverage is planned for additional models, GPU types, attention
|
||||
backends, workload shapes, and inference recipes. As that coverage lands, the
|
||||
performance tracking system will also add environment-specific considerations
|
||||
so comparisons remain meaningful across hardware, runtime, attention backend,
|
||||
and recipe changes instead of treating all records for a model as equivalent.
|
||||
|
||||
## Metrics
|
||||
|
||||
Each benchmark records six metrics:
|
||||
|
||||
| Metric | Raw key | Normalized key | Direction |
|
||||
|---|---|---|---|
|
||||
| End-to-end generation latency | `avg_generation_time_s` | `latency` | Lower is better |
|
||||
| Video throughput | `throughput_fps` | `throughput` | Higher is better |
|
||||
| Peak GPU memory | `max_peak_memory_mb` | `memory` | Lower is better |
|
||||
| Text encoder time | `text_encoder_time_s` | `text_encoder_time_s` | Lower is better |
|
||||
| DiT denoising time | `dit_time_s` | `dit_time_s` | Lower is better |
|
||||
| VAE decode time | `vae_decode_time_s` | `vae_decode_time_s` | Lower is better |
|
||||
|
||||
`test_inference_performance.py` temporarily sets `FASTVIDEO_STAGE_LOGGING=1`
|
||||
while it runs so pipeline stage execution times are available in
|
||||
`generate_video(...).logging_info`. It maps `TextEncodingStage` to
|
||||
`text_encoder_time_s`, `DenoisingStage` and `DmdDenoisingStage` to
|
||||
`dit_time_s`, and `DecodingStage` to `vae_decode_time_s`. If a pipeline does
|
||||
not report one of those stages, that component metric is stored as `null` and
|
||||
is skipped by the static threshold and rolling baseline checks.
|
||||
|
||||
## The two gates
|
||||
|
||||
There are **two independent regression gates** — they protect against
|
||||
different failure modes and are not redundant.
|
||||
|
||||
### Static thresholds (per-GPU)
|
||||
|
||||
Defined in `.buildkite/performance-benchmarks/tests/<benchmark>.json` under
|
||||
`thresholds`. Example:
|
||||
|
||||
```json
|
||||
"thresholds": {
|
||||
"L40S": {
|
||||
"max_generation_time_s": 34.0,
|
||||
"max_peak_memory_mb": 11000.0,
|
||||
"max_text_encoder_time_s": 5.0,
|
||||
"max_dit_time_s": 10.0,
|
||||
"max_vae_decode_time_s": 10.0
|
||||
},
|
||||
"default": { "max_generation_time_s": 120.0, "max_peak_memory_mb": 30000.0 }
|
||||
}
|
||||
```
|
||||
|
||||
Selection: `_get_thresholds(cfg)` matches the current GPU name (substring
|
||||
match) against keys; falls back to `default` if no GPU matches.
|
||||
|
||||
`max_generation_time_s` and `max_peak_memory_mb` are required for every
|
||||
selected threshold block. Component limits are optional: if
|
||||
`max_text_encoder_time_s`, `max_dit_time_s`, or `max_vae_decode_time_s` is
|
||||
absent, the pytest static-threshold gate skips that component.
|
||||
|
||||
These are **fail-safes** — they catch order-of-magnitude regressions,
|
||||
unrealistic memory growth, and optionally large component-specific slowdowns
|
||||
even when the rolling baseline is empty. They are hand-set with generous
|
||||
headroom and almost never need touching.
|
||||
|
||||
### Rolling baseline (per `(model_id, gpu_type)`)
|
||||
|
||||
`compare_baseline.py` loads the last 5 successful records for the same
|
||||
`(model_id, gpu_type)` from the HF dataset, computes the median for each
|
||||
available metric, and fails if the current run regresses by more than
|
||||
`PERF_MAX_REGRESSION` (default 5%). For latency, memory, and component times,
|
||||
higher values are regressions. For throughput, lower values are regressions.
|
||||
|
||||
This is the **drift detector** — it catches sub-threshold regressions that
|
||||
slowly add up. It only persists new records when running the full suite on
|
||||
`main`. Local and pull-request runs can compare against the HF baseline, but
|
||||
they do not update it.
|
||||
|
||||
When the baseline shifts for a legitimate reason (torch upgrade, kernel
|
||||
change, etc.) and CI starts failing, use the
|
||||
[`reseed-performance-baseline`](https://github.com/hao-ai-lab/FastVideo/blob/main/.agents/skills/reseed-performance-baseline/SKILL.md)
|
||||
agent skill to advance the rolling median.
|
||||
|
||||
## Schemas
|
||||
|
||||
### Raw record (`results/perf_*.json`)
|
||||
|
||||
Written by `test_inference_performance.py`. One file per benchmark run.
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"benchmark_id": "wan-t2v-1.3b-2gpu",
|
||||
"model_short_name": "Wan2.1-T2V-1.3B-Diffusers",
|
||||
"device": "NVIDIA L40S",
|
||||
"num_gpus": 2,
|
||||
"num_warmup_runs": 1,
|
||||
"num_measurement_runs": 3,
|
||||
"avg_generation_time_s": 28.4,
|
||||
"individual_times_s": [28.5, 28.3, 28.4],
|
||||
"throughput_fps": 1.58,
|
||||
"max_peak_memory_mb": 10840.0,
|
||||
"individual_peak_memories_mb": [10840.0, 10822.0, 10833.0],
|
||||
"thresholds": {
|
||||
"max_generation_time_s": 34.0,
|
||||
"max_peak_memory_mb": 11000.0,
|
||||
"max_text_encoder_time_s": 5.0,
|
||||
"max_dit_time_s": 10.0,
|
||||
"max_vae_decode_time_s": 10.0
|
||||
},
|
||||
"commit": "<full sha>",
|
||||
"pr_number": "1234",
|
||||
"timestamp": "2026-05-08T22:00:00+00:00",
|
||||
"text_encoder_time_s": 2.141,
|
||||
"dit_time_s": 8.437,
|
||||
"vae_decode_time_s": 3.208
|
||||
}
|
||||
```
|
||||
|
||||
### Normalized record (HF dataset, also dumped as `normalized_perf_*.json`)
|
||||
|
||||
Written by `compare_baseline.py:_normalize_record`. One file per benchmark
|
||||
result, used as the rolling-baseline source of truth.
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"model_id": "wan-t2v-1.3b-2gpu",
|
||||
"timestamp": "2026-05-08T22:00:00+00:00",
|
||||
"commit_sha": "<full sha>",
|
||||
"gpu_type": "NVIDIA L40S",
|
||||
"latency": 28.4,
|
||||
"throughput": 1.58,
|
||||
"memory": 10840.0,
|
||||
"text_encoder_time_s": 2.141,
|
||||
"dit_time_s": 8.437,
|
||||
"vae_decode_time_s": 3.208,
|
||||
"success": true
|
||||
}
|
||||
```
|
||||
|
||||
### Compatibility with legacy records
|
||||
|
||||
Older records in the HF dataset may not have component timing fields. The
|
||||
comparator ignores missing or `null` metrics when computing a median, and the
|
||||
dashboard lists skipped plots for metric series that have no non-null values.
|
||||
|
||||
## Environment variable reference
|
||||
|
||||
| Variable | Default | Used by | Purpose |
|
||||
|---|---|---|---|
|
||||
| `PERF_MAX_REGRESSION` | `0.05` | `compare_baseline.py` | Per-metric regression fraction that fails the build. |
|
||||
| `PERFORMANCE_TRACKING_ROOT` | `/tmp/perf-tracking` | `compare_baseline.py`, `dashboard.py` | Local directory the HF dataset is synced to. |
|
||||
| `PERF_REPORTS_DIR` | `/root/data/perf_reports` | `compare_baseline.py`, `dashboard.py` | Where the Markdown summary and Plotly HTML get written for Buildkite to pick up. |
|
||||
| `HF_REPO_ID` | `FastVideo/performance-tracking` | `hf_store.py` | HF dataset repo holding rolling-baseline records. |
|
||||
| `HF_API_KEY` | unset | `hf_store.py` | Required for upload (main-branch full-suite only); reads work without it. |
|
||||
| `TEST_SCOPE` | unset | `compare_baseline.py` | Set to `full` together with `BUILDKITE_BRANCH=main` to enable HF persistence. |
|
||||
| `BUILDKITE_BRANCH`, `BUILDKITE_COMMIT`, `BUILDKITE_PULL_REQUEST` | unset | `compare_baseline.py`, `test_inference_performance.py` | CI metadata stamped into records. |
|
||||
| `DASHBOARD_DAYS` | `30` | `dashboard.py` | Lookback window for the Plotly trend pages. |
|
||||
| `PERFORMANCE_TRACKING_SYNC_REUSE_TTL_SECONDS` | `3600` | `hf_store.py` | Freshness window for reusing an existing HF sync when requested by dashboard consumers. |
|
||||
| `FASTVIDEO_STAGE_LOGGING` | set by the pytest test | `test_inference_performance.py` | Enables pipeline stage timing capture for component metrics during benchmark runs. |
|
||||
|
||||
## CI integration
|
||||
|
||||
The performance step can run on demand with `/test performance` and as part of
|
||||
the Full Suite (see [CI Architecture](ci_architecture.md)). The Modal entry
|
||||
point is `fastvideo/tests/modal/pr_test.py:run_performance_tests` and the
|
||||
Buildkite artifact upload is in
|
||||
`.buildkite/scripts/pr_test.sh:upload_performance_artifacts`.
|
||||
|
||||
Each performance build runs pytest first. If that fixed-threshold phase fails,
|
||||
`compare_baseline.py` is skipped, so Markdown summaries and normalized JSON
|
||||
artifacts are not emitted. The dashboard still runs best-effort for
|
||||
observability. When pytest passes, the rolling-baseline phase emits:
|
||||
|
||||
* **Markdown summary** — appended to `$GITHUB_STEP_SUMMARY` when that variable
|
||||
is set, and written as `perf_<sha>_<ts>.md` for Buildkite upload. Contains a
|
||||
per-benchmark row with current vs. baseline values for latency, throughput,
|
||||
memory, text encoder time, DiT time, and VAE decode time.
|
||||
* **Plotly dashboard** — `dashboard_<sha>_<ts>.html` showing time-series for
|
||||
each metric grouped by `(model_id, gpu_type)`.
|
||||
* **Normalized records** — `normalized_perf_*.json`, one per benchmark.
|
||||
Useful as input to the
|
||||
[`reseed-performance-baseline`](https://github.com/hao-ai-lab/FastVideo/blob/main/.agents/skills/reseed-performance-baseline/SKILL.md)
|
||||
skill.
|
||||
|
||||
## Adding a new benchmark
|
||||
|
||||
1. Drop a new JSON config into
|
||||
`.buildkite/performance-benchmarks/tests/<name>.json`. Required keys:
|
||||
|
||||
```json
|
||||
{
|
||||
"benchmark_id": "<unique-id>",
|
||||
"model": { "model_path": "...", "model_short_name": "..." },
|
||||
"init_kwargs": { "num_gpus": 1, ... },
|
||||
"generation_kwargs": { "num_frames": 45, ... },
|
||||
"test_prompts": ["..."],
|
||||
"run_config": { "required_gpus": 1,
|
||||
"num_warmup_runs": 1, "num_measurement_runs": 3 },
|
||||
"thresholds": {
|
||||
"L40S": {
|
||||
"max_generation_time_s": 34.0,
|
||||
"max_peak_memory_mb": 11000.0,
|
||||
"max_text_encoder_time_s": 5.0,
|
||||
"max_dit_time_s": 10.0,
|
||||
"max_vae_decode_time_s": 10.0
|
||||
},
|
||||
"default": { "max_generation_time_s": 120.0, "max_peak_memory_mb": 30000.0 }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
2. The pytest test auto-discovers all configs — no test code needed. CI
|
||||
picks it up on the next `/test performance` run.
|
||||
|
||||
3. The first persisted main-branch run with no HF history initializes the
|
||||
baseline (passes automatically). Subsequent runs compare against it. Local
|
||||
and pull-request runs with no HF history also pass, but they do not seed the
|
||||
shared baseline.
|
||||
|
||||
4. If the benchmark targets a GPU not currently in `thresholds`, either add
|
||||
that GPU as a key or rely on the `default` block. Note that `default` is
|
||||
intended for slower fallback GPUs, so its values should be relaxed
|
||||
relative to the fastest entry.
|
||||
|
||||
5. Add component thresholds only when the stage timing is stable enough to be
|
||||
a useful fixed gate. The rolling baseline will still track component times
|
||||
when static component thresholds are omitted.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**"No baseline for ... Initializing"** — first run for this `(model_id,
|
||||
gpu_type)`. Run will pass and (if persisting) seed the first record.
|
||||
|
||||
**Persistent failure right after a torch / kernel / image upgrade** —
|
||||
genuine regression *or* baseline drift. Compare the failing normalized record
|
||||
with recent successful records in the HF dataset. If the shift is expected and
|
||||
reviewed, use the `reseed-performance-baseline` skill.
|
||||
|
||||
**Dashboard reports skipped metric plots** — the loaded records do not have
|
||||
non-null values for that metric. This is expected for older records or for
|
||||
pipelines that did not report a mapped component stage.
|
||||
|
||||
**Component timing is `null`** — the generated result did not include a mapped
|
||||
stage in `logging_info.stages`. Check that the pipeline emits stage logging
|
||||
and that the stage name is listed in `STAGE_METRIC_MAP` in
|
||||
`test_inference_performance.py`.
|
||||
@@ -51,3 +51,11 @@ Traces can be visualized using <https://ui.perfetto.dev/>.
|
||||
- Keep the profiled step count small; traces can be large and slow down job shutdown while the profiler flushes data.
|
||||
- After profiling, clean up trace directories to avoid filling disk storage.
|
||||
- When adding new regions, register them in `fastvideo.profiler` and wrap the corresponding code block with `with self.profiler_controller.region("your_region"):` or the `@profile_region` decorator.
|
||||
|
||||
## Related: Activation Trace Mode
|
||||
|
||||
For per-layer **numerical-divergence** debugging (parity bring-up against an
|
||||
upstream reference), use the env-gated [Activation Trace Mode](activation_trace.md)
|
||||
instead of the torch profiler. Activation trace dumps per-tensor stats to
|
||||
JSONL for offline `diff`-ing; the torch profiler captures kernel timing.
|
||||
They solve different problems and can run together if needed.
|
||||
|
||||
@@ -6,10 +6,12 @@ This guide explains how to add and run tests in FastVideo. The testing suite is
|
||||
|
||||
* **Unit Tests**: Located in `fastvideo/tests/api`, `fastvideo/tests/dataset`, `fastvideo/tests/entrypoints`, `fastvideo/tests/workflow`, and the CPU-only subset of `fastvideo/tests/train` (callbacks, utils). These test individual functions and classes.
|
||||
* **Component Tests**: Located in `fastvideo/tests/encoders`, `fastvideo/tests/transformers`, and `fastvideo/tests/vaes`. These verify the loading and basic functionality of model components.
|
||||
* **Train Framework Tests** (GPU): Located in `fastvideo/tests/train/models`. Cover model loading + forward smoke for the new `fastvideo/train/` framework. Triggered via `/test train-framework` or as part of the Full Suite.
|
||||
* **Train Framework Tests** (GPU): Located in `fastvideo/tests/train/models` (model loading + forward smoke) and `fastvideo/tests/train/methods` (per-method single training step). Cover the new `fastvideo/train/` framework end-to-end on real checkpoints with tiny synthetic batches. Triggered via `/test train-framework` or as part of the Full Suite.
|
||||
* **SSIM Tests**: Located in `fastvideo/tests/ssim`. These are regression tests that compare generated videos against reference videos using the Structural Similarity Index Measure (SSIM) to detect quality degradation.
|
||||
* **Training Tests**: Located in `fastvideo/tests/training`. These validate training loops, loss calculations, and specific training techniques like LoRA, Distillation, and VSA.
|
||||
* **Inference Tests**: Located in `fastvideo/tests/inference`. These test specialized inference pipelines and optimizations (e.g., VSA, V-MoBA).
|
||||
* **Performance Tests**: Located in `fastvideo/tests/performance`. End-to-end latency, throughput, and peak-memory benchmarks gated against per-GPU static thresholds and a rolling Hugging Face baseline. See [Performance Benchmarks](performance_benchmarks.md) for the full workflow, schema and future environment-specific considerations.
|
||||
* **DreamVerse App Tests**: Located in `apps/dreamverse`. These validate the DreamVerse app frontend and related app code.
|
||||
|
||||
For now, we will focus on **SSIM Tests**.
|
||||
|
||||
@@ -212,6 +214,7 @@ from a PR comment. The workflow reacts with a 🚀 emoji to confirm the command
|
||||
/test transformer # Transformer / DiT tests (Fastcheck)
|
||||
/test kernel # CUDA kernel tests (Fastcheck)
|
||||
/test unit # Unit tests (Fastcheck)
|
||||
/test dreamverse # DreamVerse app tests (Fastcheck)
|
||||
/test full # Entire Full Suite
|
||||
/test fastcheck # Entire Fastcheck suite
|
||||
```
|
||||
|
||||
@@ -261,8 +261,8 @@ surfaces:
|
||||
- fastvideo.configs.pipelines.turbodiffusion.TurboDiffusionT2V_1_3B_Config
|
||||
- fastvideo.configs.pipelines.wan.FastWan2_1_T2V_480P_Config
|
||||
- fastvideo.configs.pipelines.wan.FastWan2_2_TI2V_5B_Config
|
||||
- fastvideo.configs.pipelines.wan.MatrixGameBaseI2V480PConfig
|
||||
- fastvideo.configs.pipelines.wan.MatrixGameI2V480PConfig
|
||||
- fastvideo.configs.pipelines.matrixgame2.MatrixGame2BaseI2V480PConfig
|
||||
- fastvideo.configs.pipelines.matrixgame2.MatrixGame2I2V480PConfig
|
||||
- fastvideo.configs.pipelines.wan.SelfForcingWan2_2_T2V480PConfig
|
||||
- fastvideo.configs.pipelines.wan.SelfForcingWanT2V480PConfig
|
||||
- fastvideo.configs.pipelines.wan.WANV2VConfig
|
||||
@@ -284,8 +284,8 @@ surfaces:
|
||||
- fastvideo.configs.pipelines.turbodiffusion.TurboDiffusionT2V_1_3B_Config
|
||||
- fastvideo.configs.pipelines.wan.FastWan2_1_T2V_480P_Config
|
||||
- fastvideo.configs.pipelines.wan.FastWan2_2_TI2V_5B_Config
|
||||
- fastvideo.configs.pipelines.wan.MatrixGameBaseI2V480PConfig
|
||||
- fastvideo.configs.pipelines.wan.MatrixGameI2V480PConfig
|
||||
- fastvideo.configs.pipelines.matrixgame2.MatrixGame2BaseI2V480PConfig
|
||||
- fastvideo.configs.pipelines.matrixgame2.MatrixGame2I2V480PConfig
|
||||
- fastvideo.configs.pipelines.wan.SelfForcingWan2_2_T2V480PConfig
|
||||
- fastvideo.configs.pipelines.wan.SelfForcingWanT2V480PConfig
|
||||
- fastvideo.configs.pipelines.wan.WANV2VConfig
|
||||
@@ -333,9 +333,9 @@ surfaces:
|
||||
- fastvideo.configs.pipelines.wan.FastWan2_2_TI2V_5B_Config
|
||||
- fastvideo.configs.pipelines.wan.Wan2_2_TI2V_5B_Config
|
||||
context_noise:
|
||||
sources: [fastvideo.configs.pipelines.wan.MatrixGameI2V480PConfig]
|
||||
sources: [fastvideo.configs.pipelines.matrixgame2.MatrixGame2I2V480PConfig]
|
||||
num_frames_per_block:
|
||||
sources: [fastvideo.configs.pipelines.wan.MatrixGameI2V480PConfig]
|
||||
sources: [fastvideo.configs.pipelines.matrixgame2.MatrixGame2I2V480PConfig]
|
||||
audio_channels:
|
||||
sources:
|
||||
- fastvideo.configs.pipelines.stable_audio.StableAudioT2AConfig
|
||||
|
||||
@@ -226,7 +226,7 @@ Dataclass carrying all pipeline state between stages. Key field groups:
|
||||
`width_latents`.
|
||||
- **Scheduler**: `timesteps`, `num_inference_steps`, `guidance_scale`,
|
||||
`sigmas`.
|
||||
- **Task-specific**: `mouse_cond`/`keyboard_cond` (MatrixGame), `pose`
|
||||
- **Task-specific**: `mouse_cond`/`keyboard_cond` (Matrix-Game 2.0), `pose`
|
||||
(HYWorld), `camera_states` (GameCraft), `c2ws_plucker_emb`
|
||||
(LingBotWorld).
|
||||
- **Output**: `output: Tensor | None`.
|
||||
@@ -252,7 +252,7 @@ Standard stages (typical execution order):
|
||||
|
||||
Specialized variants: `CausalDenoisingStage`, `LTX2DenoisingStage`,
|
||||
`LongCatDenoisingStage`, `GameCraftDenoisingStage`,
|
||||
`HYWorldDenoisingStage`, `MatrixGameDenoisingStage`,
|
||||
`HYWorldDenoisingStage`, `MatrixGame2CausalDenoisingStage`,
|
||||
`SRDenoisingStage`, `LTX2AudioDecodingStage`, `SD35ConditioningStage`,
|
||||
`LTX2TextEncodingStage`, `LTX2LatentPreparationStage`.
|
||||
|
||||
|
||||
+147
-12
@@ -11,6 +11,9 @@ This page describes the various options for speeding up generation times in Fast
|
||||
- [Sliding Tile Attention (Archived)](#sliding-tile-attention-archived)
|
||||
- [Sage Attention](#sage-attention)
|
||||
- [Sage Attention 3](#sage-attention-3)
|
||||
- [Adaptive Guidance (CFG gating)](#adaptive-guidance-cfg-gating)
|
||||
|
||||
- [torch.compile](#torch-compile)
|
||||
|
||||
## Attention Backends
|
||||
|
||||
@@ -69,9 +72,9 @@ python setup.py install
|
||||
|
||||
### FP4 Flash Attention 4 (Blackwell only)
|
||||
|
||||
**`FLASH_ATTN`** with **`FASTVIDEO_NVFP4_FA4=1`**
|
||||
**`FLASH_ATTN`** with **`--nvfp4_fa4`**
|
||||
|
||||
On Blackwell GPUs (B200/B300), you can enable FP4 quantized Q/K attention for up to **1.39x kernel speedup** over BF16 FA4, peaking at **1801 TFLOPS**. This quantizes Q and K to NVFP4 E2M1 with per-block E4M3 scale factors while keeping V in BF16.
|
||||
On Blackwell GPUs (B200/B300), you can enable FP4 quantized Q/K attention for up to **1.31x kernel speedup** over BF16 FA4, peaking at **2018 TFLOPS**. This quantizes Q and K to NVFP4 E2M1 with per-block E4M3 scale factors while keeping V in BF16 or FP8.
|
||||
|
||||
See the [Attn-QAT paper](https://arxiv.org/abs/2603.00040) and [flash-attention-fp4 benchmark results](https://github.com/hao-ai-lab/flash-attention-fp4/blob/fp4/flash_attn/cute/README.md) for details.
|
||||
|
||||
@@ -83,32 +86,30 @@ See the [Attn-QAT paper](https://arxiv.org/abs/2603.00040) and [flash-attention-
|
||||
|
||||
#### Installation
|
||||
|
||||
Install the FP4 flash attention kernel and its dependencies:
|
||||
Install the FP4 flash attention kernel (without upgrading your existing torch):
|
||||
|
||||
```bash
|
||||
pip install "git+ssh://git@github.com/hao-ai-lab/flash-attention-fp4.git@fp4#subdirectory=flash_attn/cute"
|
||||
pip install --no-deps "git+ssh://git@github.com/hao-ai-lab/flash-attention-fp4.git@fp4#subdirectory=flash_attn/cute"
|
||||
pip install "nvidia-cutlass-dsl>=4.4.2" apache-tvm-ffi flashinfer-python
|
||||
```
|
||||
|
||||
This installs the FP4 kernel and all dependencies (nvidia-cutlass-dsl, flashinfer-python, apache-tvm-ffi).
|
||||
The `--no-deps` flag prevents upgrading torch/torchvision. The kernel requires torch >= 2.4 with CUDA 12.8+ support (already present in FastVideo's environment).
|
||||
|
||||
#### Usage
|
||||
|
||||
Enable FP4 attention via environment variables:
|
||||
Enable FP4 attention via the `--nvfp4_fa4` flag:
|
||||
|
||||
```bash
|
||||
FASTVIDEO_NVFP4_FA4=1 CUTE_DSL_ENABLE_TVM_FFI=1 python examples/inference/optimizations/fp4_attn_wan2_1_1_3b.py --nvfp4_fa4
|
||||
python examples/inference/optimizations/fp4_attn_wan2_1_1_3b.py --nvfp4_fa4
|
||||
```
|
||||
|
||||
Or in Python:
|
||||
Or in Python via the `nvfp4_fa4` kwarg (sets env vars automatically):
|
||||
|
||||
```python
|
||||
import os
|
||||
os.environ["FASTVIDEO_NVFP4_FA4"] = "1"
|
||||
os.environ["CUTE_DSL_ENABLE_TVM_FFI"] = "1"
|
||||
|
||||
from fastvideo import VideoGenerator
|
||||
gen = VideoGenerator.from_pretrained(
|
||||
"Wan-AI/Wan2.1-T2V-1.3B-Diffusers",
|
||||
nvfp4_fa4=True,
|
||||
num_gpus=1,
|
||||
use_fsdp_inference=False, # FSDP is incompatible with FP4 pointer path
|
||||
)
|
||||
@@ -177,6 +178,106 @@ These backends are model-specific and require the corresponding kernels and
|
||||
dependencies. Use the support matrix and model examples to confirm compatibility
|
||||
before enabling them.
|
||||
|
||||
<a id="torch-compile"></a>
|
||||
|
||||
## torch.compile
|
||||
|
||||
FastVideo can `torch.compile` the DiT (transformer) for a substantial
|
||||
end-to-end speedup. It is **off by default** and enabled per-run.
|
||||
|
||||
### Enabling
|
||||
|
||||
```python
|
||||
generator = VideoGenerator.from_pretrained(
|
||||
"Wan-AI/Wan2.1-T2V-1.3B-Diffusers",
|
||||
enable_torch_compile=True,
|
||||
)
|
||||
```
|
||||
|
||||
A complete A/B example (eager vs compiled, warmup excluded) is in
|
||||
[`examples/inference/optimizations/torch_compile_example.py`](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/optimizations/torch_compile_example.py).
|
||||
|
||||
`fastvideo generate` is config-file driven; to enable `torch.compile`
|
||||
from the CLI, set the relevant field in your run config and pass it via
|
||||
`fastvideo generate --config run.yaml`. There is no top-level
|
||||
`--enable-torch-compile` flag on the subcommand.
|
||||
|
||||
Only DiT submodules that declare `_compile_conditions` are compiled
|
||||
(most shipped models). The text encoder and VAE are not compiled by this
|
||||
flag.
|
||||
|
||||
### What to expect
|
||||
|
||||
| Config | Effect |
|
||||
|---|---|
|
||||
| Wan2.1-T2V-1.3B, A100-80GB, 480×832×81f, 50 steps | end-to-end **259.7s → 198.1s (−23.7%)**; per-step **4.91 → 3.78 s/it** |
|
||||
|
||||
The speedup is **configuration-dependent** — it varies with model,
|
||||
resolution, step count, and GPU. Treat the number above as one measured
|
||||
data point, not a guarantee; benchmark your own config (recipe below).
|
||||
|
||||
There is a **one-time graph-build cost** on the first generation (tens of
|
||||
seconds to minutes, model-dependent). It amortizes over subsequent
|
||||
generations with the same input shapes. Always exclude the first
|
||||
(warmup) generation when measuring steady-state latency — measuring the
|
||||
warmup is the most common way to wrongly conclude "compile is slower".
|
||||
|
||||
**Numerics.** Inductor's lowering is designed to preserve eager
|
||||
semantics within floating-point tolerance, but per-model equivalence is
|
||||
not asserted by any standing SSIM regression here — the SSIM tests in
|
||||
[`fastvideo/tests/ssim/`](https://github.com/hao-ai-lab/FastVideo/tree/main/fastvideo/tests/ssim)
|
||||
run with `enable_torch_compile` disabled. If you depend on compile
|
||||
output staying close to eager (or your previous compiled run), run an
|
||||
MS-SSIM gate on *your* config, especially when combining
|
||||
`enable_torch_compile=True` with other numerics-affecting flags
|
||||
(quantized attention backends, FP4, layerwise offload edge cases).
|
||||
|
||||
### Known interactions
|
||||
|
||||
- **Layerwise CPU offload** (`dit_layerwise_offload=True`, the default):
|
||||
the offload hook previously caused an implicit graph break once per
|
||||
transformer layer, fragmenting the compiled region. Addressed in
|
||||
hao-ai-lab/FastVideo#1365 — keep that fix to get a clean compiled
|
||||
region under the default offload path.
|
||||
- **`mode="reduce-overhead"` / CUDA graphs**: not yet supported
|
||||
end-to-end. The attention dispatch is an untraceable custom op and
|
||||
still breaks the graph, which CUDA-graph trees cannot span. Use the
|
||||
default inductor mode (shown above) until that is resolved.
|
||||
|
||||
Extra `torch.compile` options are passed through `torch_compile_kwargs`
|
||||
(a dict), accepted by `VideoGenerator.from_pretrained(...)` and by the
|
||||
CLI as a JSON string via `--torch-compile-kwargs`. Example (currently
|
||||
**not** recommended — see the CUDA-graphs caveat above):
|
||||
|
||||
```python
|
||||
VideoGenerator.from_pretrained(
|
||||
"Wan-AI/Wan2.1-T2V-1.3B-Diffusers",
|
||||
enable_torch_compile=True,
|
||||
torch_compile_kwargs={"mode": "reduce-overhead"}, # may error today
|
||||
)
|
||||
```
|
||||
|
||||
### Benchmarking torch.compile
|
||||
|
||||
Same discipline as attention backends — same prompt, same seed, same
|
||||
config; **discard the first generation** (graph build):
|
||||
|
||||
```python
|
||||
import time
|
||||
from fastvideo import VideoGenerator
|
||||
|
||||
gen = VideoGenerator.from_pretrained("your-model-id", enable_torch_compile=True)
|
||||
req = {"prompt": "Your prompt", "sampling": {"seed": 1024},
|
||||
"output": {"save_video": False}}
|
||||
gen.generate(req) # warmup: graph build, discard
|
||||
t0 = time.perf_counter()
|
||||
gen.generate(req) # measured: shapes reused
|
||||
print(f"compiled steady-state: {time.perf_counter() - t0:.2f}s")
|
||||
```
|
||||
|
||||
See `examples/inference/optimizations/torch_compile_example.py` for a
|
||||
baseline-vs-compile A/B with the warmup correctly excluded.
|
||||
|
||||
## Benchmarking different optimizations
|
||||
|
||||
To benchmark backend performance, generate the same prompt with the same seed and compare end-to-end generation times:
|
||||
@@ -198,3 +299,37 @@ for backend in ["TORCH_SDPA", "FLASH_ATTN", "SAGE_ATTN"]:
|
||||
```
|
||||
|
||||
Note: reinstantiate `VideoGenerator` after changing `FASTVIDEO_ATTENTION_BACKEND`.
|
||||
|
||||
## Adaptive Guidance (CFG gating)
|
||||
|
||||
CFG gating accelerates classifier-free guidance by reusing the cached
|
||||
`noise_pred_cond - noise_pred_uncond` delta after a configurable fraction of
|
||||
the denoising schedule, skipping the unconditional model forward for the
|
||||
remaining steps. The technique is the LinearAG variant of Adaptive Guidance
|
||||
(Castillo et al. 2023, [arXiv:2312.12487](https://arxiv.org/abs/2312.12487)).
|
||||
|
||||
### Enabling
|
||||
|
||||
Set the `FASTVIDEO_CFG_GATE_STEP` environment variable to a float in `[0, 1]`:
|
||||
|
||||
| Value | Behavior |
|
||||
|-------|----------|
|
||||
| `1.0` (default) | Disabled — legacy two-pass CFG every step. |
|
||||
| `0.5` | Cache the delta after `len(timesteps) * 0.5` steps; reuse for the rest. |
|
||||
| `0.0` | Cache from the very first step (most aggressive). |
|
||||
|
||||
```bash
|
||||
export FASTVIDEO_CFG_GATE_STEP=0.5
|
||||
```
|
||||
|
||||
### Trade-offs
|
||||
|
||||
- **Memory**: one extra model-output-sized tensor per rank held during the
|
||||
gating window.
|
||||
- **Quality**: VBench-measured quality is preserved within noise on 4 of 5
|
||||
dimensions at `FASTVIDEO_CFG_GATE_STEP=0.5` for Wan T2V 1.3B per the PR's
|
||||
reported numbers (see [#1372](https://github.com/hao-ai-lab/FastVideo/pull/1372)).
|
||||
- **Speed**: ~22% e2e on 4xL40S and ~24% on 1xH100 at the same settings.
|
||||
|
||||
Default behavior is byte-for-byte equivalent to the legacy two-pass CFG path;
|
||||
the feature is fully opt-in.
|
||||
|
||||
@@ -70,9 +70,10 @@ pipeline initialization and sampling.
|
||||
| TurboWan2.1 T2V 14B | `loayrashid/TurboWan2.1-T2V-14B-Diffusers` | 480P, 720P | ⭕ | ⭕ | ⭕ | ⭕ | ⭕ |
|
||||
| TurboWan2.2 I2V A14B | `loayrashid/TurboWan2.2-I2V-A14B-Diffusers` | 480P<br>720P | ⭕ | ⭕ | ⭕ | ⭕ | ⭕ |
|
||||
| LongCat T2V 13.6B | See note** | 480P<br>720P | ❌ | ❌ | ❌ | ⭕ | ✅ |
|
||||
| Matrix Game 2.0 Base | `FastVideo/Matrix-Game-2.0-Base-Diffusers` | 352x640 | ⭕ | ⭕ | ⭕ | ⭕ | ⭕ |
|
||||
| Matrix Game 2.0 GTA | `FastVideo/Matrix-Game-2.0-GTA-Diffusers` | 352x640 | ⭕ | ⭕ | ⭕ | ⭕ | ⭕ |
|
||||
| Matrix Game 2.0 TempleRun | `FastVideo/Matrix-Game-2.0-TempleRun-Diffusers` | 352x640 | ⭕ | ⭕ | ⭕ | ⭕ | ⭕ |
|
||||
| Matrix Game 2.0 Base Distilled | `FastVideo/Matrix-Game-2.0-Base-Distilled-Diffusers` | 352x640 | ⭕ | ⭕ | ⭕ | ⭕ | ⭕ |
|
||||
| Matrix Game 2.0 GTA Distilled | `FastVideo/Matrix-Game-2.0-GTA-Distilled-Diffusers` | 352x640 | ⭕ | ⭕ | ⭕ | ⭕ | ⭕ |
|
||||
| Matrix Game 2.0 TempleRun Distilled | `FastVideo/Matrix-Game-2.0-TempleRun-Distilled-Diffusers` | 352x640 | ⭕ | ⭕ | ⭕ | ⭕ | ⭕ |
|
||||
| Matrix Game 3.0 Base Distilled | `FastVideo/Matrix-Game-3.0-Base-Distilled-Diffusers` | 720x1280 | ⭕ | ⭕ | ⭕ | ⭕ | ⭕ |
|
||||
| GEN3C Cosmos 7B | `FastVideo/GEN3C-Cosmos-7B-Diffusers` | 704px1280p | ❌ | ❌ | ❌ | ⭕ | ⭕ |
|
||||
|
||||
**Note**: Wan2.2 TI2V 5B has some quality issues when performing I2V generation. We are working on fixing this issue.
|
||||
|
||||
@@ -30,6 +30,27 @@ Useful variables:
|
||||
`TORCH_SDPA`, `FLASH_ATTN`, `SAGE_ATTN_THREE`, or
|
||||
`ATTN_QAT_INFER`, or `ATTN_QAT_TRAIN`)
|
||||
|
||||
## Layer-by-Layer Activation Tracing
|
||||
|
||||
For numerical-divergence debugging — typically when porting a new model and
|
||||
needing to find the first layer where FastVideo and an upstream reference
|
||||
produce different outputs — use the env-gated activation trace mode:
|
||||
|
||||
```bash
|
||||
FASTVIDEO_TRACE_ACTIVATIONS=1 \
|
||||
FASTVIDEO_TRACE_LAYERS="^transformer\.blocks\.\d+$" \
|
||||
FASTVIDEO_TRACE_OUTPUT=/tmp/fv_trace.jsonl \
|
||||
python your_script.py
|
||||
```
|
||||
|
||||
The trace dumps per-tensor stats (`abs_mean`, `sum`, `shape`, etc.) to a
|
||||
JSONL file. Run the same workload with tracing on the upstream side, then
|
||||
`diff` the two files to localize the first divergent layer.
|
||||
|
||||
See [Activation Trace Mode](../contributing/activation_trace.md) for the
|
||||
full guide (env var reference, JSONL output schema, parity-debug workflow,
|
||||
performance impact, and troubleshooting).
|
||||
|
||||
## Common Failure Modes
|
||||
|
||||
### Out-of-memory
|
||||
|
||||
@@ -8,18 +8,18 @@ RUN_NAME=$(date +"%m%d_%H%M")
|
||||
echo "RUN_NAME: $RUN_NAME"
|
||||
|
||||
# Model paths for Self-Forcing DMD distillation.
|
||||
GENERATOR_MODEL_PATH="FastVideo/Matrix-Game-2.0-Base-Diffusers"
|
||||
REAL_SCORE_MODEL_PATH="FastVideo/Matrix-Game-2.0-Foundation-Diffusers" # Teacher
|
||||
FAKE_SCORE_MODEL_PATH="FastVideo/Matrix-Game-2.0-Foundation-Diffusers" # Critic
|
||||
GENERATOR_MODEL_PATH="FastVideo/Matrix-Game-2.0-Base-Distilled-Diffusers"
|
||||
REAL_SCORE_MODEL_PATH="FastVideo/Matrix-Game-2.0-Base-Diffusers" # Teacher
|
||||
FAKE_SCORE_MODEL_PATH="FastVideo/Matrix-Game-2.0-Base-Diffusers" # Critic
|
||||
|
||||
DATA_DIR="data/matrixgame"
|
||||
DATA_DIR="data/matrixgame2"
|
||||
VALIDATION_DATASET_FILE="examples/distill/MatrixGame2.0/validation.json"
|
||||
NUM_GPUS=1
|
||||
|
||||
# Training arguments
|
||||
training_args=(
|
||||
--tracker_project_name "matrixgame_sf"
|
||||
--output_dir "checkpoints/matrixgame_sf_${RUN_NAME}"
|
||||
--tracker_project_name "matrixgame2_sf"
|
||||
--output_dir "checkpoints/matrixgame2_sf_${RUN_NAME}"
|
||||
--wandb_run_name "${RUN_NAME}_test"
|
||||
--max_train_steps 5
|
||||
--train_batch_size 1
|
||||
@@ -114,7 +114,7 @@ self_forcing_args=(
|
||||
torchrun \
|
||||
--nnodes 1 \
|
||||
--nproc_per_node $NUM_GPUS \
|
||||
fastvideo/training/matrixgame_self_forcing_distillation_pipeline.py \
|
||||
fastvideo/training/matrixgame2_self_forcing_distillation_pipeline.py \
|
||||
"${parallel_args[@]}" \
|
||||
"${model_args[@]}" \
|
||||
"${dataset_args[@]}" \
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
#!/bin/bash
|
||||
#SBATCH --job-name=matrixgame_sf_dmd
|
||||
#SBATCH --job-name=matrixgame2_sf_dmd
|
||||
#SBATCH --partition=main
|
||||
#SBATCH --nodes=1
|
||||
#SBATCH --ntasks=1
|
||||
@@ -7,8 +7,8 @@
|
||||
#SBATCH --gres=gpu:4
|
||||
#SBATCH --cpus-per-task=128
|
||||
#SBATCH --mem=1440G
|
||||
#SBATCH --output=logs/slurm/matrixgame_sf_dmd_%j.out
|
||||
#SBATCH --error=logs/slurm/matrixgame_sf_dmd_%j.err
|
||||
#SBATCH --output=logs/slurm/matrixgame2_sf_dmd_%j.out
|
||||
#SBATCH --error=logs/slurm/matrixgame2_sf_dmd_%j.err
|
||||
#SBATCH --exclusive
|
||||
|
||||
set -euo pipefail
|
||||
@@ -56,16 +56,16 @@ echo "TOTAL_GPUS: ${TOTAL_GPUS}"
|
||||
RUN_NAME="$(date +"%m%d_%H%M")"
|
||||
echo "RUN_NAME: ${RUN_NAME}"
|
||||
|
||||
GENERATOR_MODEL_PATH="FastVideo/Matrix-Game-2.0-Base-Diffusers"
|
||||
REAL_SCORE_MODEL_PATH="FastVideo/Matrix-Game-2.0-Foundation-Diffusers"
|
||||
FAKE_SCORE_MODEL_PATH="FastVideo/Matrix-Game-2.0-Foundation-Diffusers"
|
||||
GENERATOR_MODEL_PATH="FastVideo/Matrix-Game-2.0-Base-Distilled-Diffusers"
|
||||
REAL_SCORE_MODEL_PATH="FastVideo/Matrix-Game-2.0-Base-Diffusers"
|
||||
FAKE_SCORE_MODEL_PATH="FastVideo/Matrix-Game-2.0-Base-Diffusers"
|
||||
|
||||
DATA_DIR="data/matrixgame"
|
||||
DATA_DIR="data/matrixgame2"
|
||||
VALIDATION_DATASET_FILE="examples/distill/MatrixGame2.0/validation.json"
|
||||
|
||||
training_args=(
|
||||
--tracker_project_name "matrixgame_sf"
|
||||
--output_dir "checkpoints/matrixgame_sf_${RUN_NAME}"
|
||||
--tracker_project_name "matrixgame2_sf"
|
||||
--output_dir "checkpoints/matrixgame2_sf_${RUN_NAME}"
|
||||
--wandb_run_name "${RUN_NAME}_test"
|
||||
--max_train_steps 1200
|
||||
--train_batch_size 1
|
||||
@@ -156,7 +156,7 @@ srun python -m torch.distributed.run \
|
||||
--node_rank "${SLURM_PROCID}" \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint="${MASTER_ADDR}:${MASTER_PORT}" \
|
||||
fastvideo/training/matrixgame_self_forcing_distillation_pipeline.py \
|
||||
fastvideo/training/matrixgame2_self_forcing_distillation_pipeline.py \
|
||||
"${parallel_args[@]}" \
|
||||
"${model_args[@]}" \
|
||||
"${dataset_args[@]}" \
|
||||
|
||||
+4
-4
@@ -1,5 +1,5 @@
|
||||
from fastvideo import VideoGenerator
|
||||
from fastvideo.models.dits.matrixgame.utils import create_action_presets
|
||||
from fastvideo.models.dits.matrixgame2.utils import create_action_presets
|
||||
|
||||
import torch
|
||||
|
||||
@@ -13,17 +13,17 @@ MODEL_VARIANT = "base_distilled_model"
|
||||
# Variant-specific settings
|
||||
VARIANT_CONFIG = {
|
||||
"base_distilled_model": {
|
||||
"model_path": "FastVideo/Matrix-Game-2.0-Base-Diffusers",
|
||||
"model_path": "FastVideo/Matrix-Game-2.0-Base-Distilled-Diffusers",
|
||||
"keyboard_dim": 4,
|
||||
"image_url": "https://raw.githubusercontent.com/SkyworkAI/Matrix-Game/main/Matrix-Game-2/demo_images/universal/0000.png",
|
||||
},
|
||||
"gta_distilled_model": {
|
||||
"model_path": "FastVideo/Matrix-Game-2.0-GTA-Diffusers",
|
||||
"model_path": "FastVideo/Matrix-Game-2.0-GTA-Distilled-Diffusers",
|
||||
"keyboard_dim": 2,
|
||||
"image_url": "https://raw.githubusercontent.com/SkyworkAI/Matrix-Game/main/Matrix-Game-2/demo_images/gta_drive/0000.png",
|
||||
},
|
||||
"templerun_distilled_model": {
|
||||
"model_path": "FastVideo/Matrix-Game-2.0-TempleRun-Diffusers",
|
||||
"model_path": "FastVideo/Matrix-Game-2.0-TempleRun-Distilled-Diffusers",
|
||||
"keyboard_dim": 7,
|
||||
"image_url": "https://raw.githubusercontent.com/SkyworkAI/Matrix-Game/main/Matrix-Game-2/demo_images/temple_run/0000.png",
|
||||
},
|
||||
+4
-4
@@ -1,5 +1,5 @@
|
||||
from fastvideo.entrypoints.streaming_generator import StreamingVideoGenerator
|
||||
from fastvideo.models.dits.matrixgame.utils import get_current_action_async, expand_action_to_frames
|
||||
from fastvideo.models.dits.matrixgame2.utils import get_current_action_async, expand_action_to_frames
|
||||
|
||||
import torch
|
||||
import asyncio
|
||||
@@ -14,19 +14,19 @@ MODEL_VARIANT = "base_distilled_model"
|
||||
# Variant-specific settings
|
||||
VARIANT_CONFIG = {
|
||||
"base_distilled_model": {
|
||||
"model_path": "FastVideo/Matrix-Game-2.0-Base-Diffusers",
|
||||
"model_path": "FastVideo/Matrix-Game-2.0-Base-Distilled-Diffusers",
|
||||
"keyboard_dim": 4,
|
||||
"mode": "universal",
|
||||
"image_url": "https://raw.githubusercontent.com/SkyworkAI/Matrix-Game/main/Matrix-Game-2/demo_images/universal/0000.png",
|
||||
},
|
||||
"gta_distilled_model": {
|
||||
"model_path": "FastVideo/Matrix-Game-2.0-GTA-Diffusers",
|
||||
"model_path": "FastVideo/Matrix-Game-2.0-GTA-Distilled-Diffusers",
|
||||
"keyboard_dim": 2,
|
||||
"mode": "gta_drive",
|
||||
"image_url": "https://raw.githubusercontent.com/SkyworkAI/Matrix-Game/main/Matrix-Game-2/demo_images/gta_drive/0000.png",
|
||||
},
|
||||
"templerun_distilled_model": {
|
||||
"model_path": "FastVideo/Matrix-Game-2.0-TempleRun-Diffusers",
|
||||
"model_path": "FastVideo/Matrix-Game-2.0-TempleRun-Distilled-Diffusers",
|
||||
"keyboard_dim": 7,
|
||||
"mode": "templerun",
|
||||
"image_url": "https://raw.githubusercontent.com/SkyworkAI/Matrix-Game/main/Matrix-Game-2/demo_images/temple_run/0000.png",
|
||||
@@ -0,0 +1,35 @@
|
||||
from fastvideo import VideoGenerator
|
||||
|
||||
MODEL_PATH = "FastVideo/Matrix-Game-3.0-Base-Distilled-Diffusers"
|
||||
IMAGE_URL = "https://raw.githubusercontent.com/SkyworkAI/Matrix-Game/main/Matrix-Game-3/demo_images/001/image.png"
|
||||
PROMPT = "A colorful, animated cityscape with a gas station and various buildings."
|
||||
OUTPUT_PATH = "video_samples_matrixgame3"
|
||||
|
||||
|
||||
def main():
|
||||
generator = VideoGenerator.from_pretrained(
|
||||
MODEL_PATH,
|
||||
num_gpus=1,
|
||||
use_fsdp_inference=False,
|
||||
dit_cpu_offload=False,
|
||||
vae_cpu_offload=False,
|
||||
text_encoder_cpu_offload=True,
|
||||
pin_cpu_memory=True,
|
||||
)
|
||||
|
||||
generator.generate_video(
|
||||
prompt=PROMPT,
|
||||
image_path=IMAGE_URL,
|
||||
height=720,
|
||||
width=1280,
|
||||
num_frames=57,
|
||||
num_inference_steps=3,
|
||||
guidance_scale=1.0,
|
||||
seed=42,
|
||||
output_path=OUTPUT_PATH,
|
||||
save_video=True,
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,90 @@
|
||||
"""Compute Fréchet Video Distance (FVD) over a folder of generated videos.
|
||||
|
||||
Run::
|
||||
|
||||
pip install -e .[eval]
|
||||
python examples/inference/eval/eval_fvd.py \\
|
||||
--gen-dir path/to/generated_videos/ \\
|
||||
--reference-dir path/to/real_videos/ \\
|
||||
--output fvd_scores.json
|
||||
|
||||
Demonstrates the canonical pattern: turn two directories into a samples
|
||||
list with :func:`samples_from`, hand it to :class:`Evaluator`. The
|
||||
Evaluator decodes through its :class:`VideoPool` and runs FVD's
|
||||
``accumulate`` / ``finalize`` end-to-end.
|
||||
|
||||
The first call extracts I3D features over every file under
|
||||
``--reference-dir`` and caches them to
|
||||
``${FASTVIDEO_EVAL_CACHE}/fvd/real_features_i3d.pt``. Subsequent runs
|
||||
reuse the cache — omit ``--reference-dir`` to score new generations
|
||||
against the cached reference set.
|
||||
|
||||
For paper-grade FVD scores use at least 256 generated + 256 reference
|
||||
videos.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
import torch
|
||||
|
||||
from fastvideo.eval import create_evaluator, samples_from
|
||||
|
||||
|
||||
def main() -> None:
|
||||
p = argparse.ArgumentParser(description=__doc__,
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
p.add_argument("--gen-dir", type=Path, required=True,
|
||||
help="Directory of generated videos (.mp4, .avi, .mov, .mkv, .gif).")
|
||||
p.add_argument("--reference-dir", type=Path, default=None,
|
||||
help="Directory of reference videos. Omit to score against the cached "
|
||||
"reference features (built on a previous run).")
|
||||
p.add_argument("--device", default="cuda:0" if torch.cuda.is_available() else "cpu")
|
||||
p.add_argument("--num-gpus", type=int, default=1,
|
||||
help="Number of GPU replicas. >1 fans extraction out across devices.")
|
||||
p.add_argument("--cache-path", type=Path, default=None,
|
||||
help="Override the reference-feature cache path. "
|
||||
"Defaults to ${FASTVIDEO_EVAL_CACHE}/fvd/real_features_i3d.pt.")
|
||||
p.add_argument("--output", type=Path, default=None,
|
||||
help="Write the result as JSON to this path (default: stdout only).")
|
||||
args = p.parse_args()
|
||||
|
||||
if args.cache_path is not None:
|
||||
# FVDMetric resolves cache_path via this env-var when no constructor
|
||||
# kwarg is set; create_evaluator doesn't forward kwargs through to
|
||||
# the metric, so the env-var is the only path here.
|
||||
os.environ["FASTVIDEO_FVD_REF_FEATURES"] = str(args.cache_path)
|
||||
|
||||
samples = samples_from(
|
||||
video=args.gen_dir,
|
||||
reference=args.reference_dir,
|
||||
)
|
||||
|
||||
ev = create_evaluator(metrics=["common.fvd"], device=args.device, num_gpus=args.num_gpus)
|
||||
results = ev.evaluate(samples=samples)
|
||||
result = results.corpus["common.fvd"]
|
||||
|
||||
if result.score is None:
|
||||
print(f"\nFVD: SKIPPED — {result.details.get('skipped')}")
|
||||
else:
|
||||
print(f"\nFVD: {result.score:.4f}")
|
||||
print(f" details: {result.details}")
|
||||
|
||||
if args.output is not None:
|
||||
payload = {
|
||||
"metric": result.name,
|
||||
"score": result.score,
|
||||
"details": result.details,
|
||||
"gen_dir": str(args.gen_dir),
|
||||
"reference_dir": str(args.reference_dir) if args.reference_dir else None,
|
||||
}
|
||||
args.output.parent.mkdir(parents=True, exist_ok=True)
|
||||
args.output.write_text(json.dumps(payload, indent=2))
|
||||
print(f" wrote {args.output}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,84 @@
|
||||
# FastLTX-2.3 Gradio local demo
|
||||
|
||||
Local Gradio + FastAPI demo for FastLTX-2.3 text-to-video generation. This
|
||||
directory is a package (`gradio_local_demo_ltx2_3/`) split out from the
|
||||
original single-file version to make each concern independently reviewable.
|
||||
The folder name matches the sibling `gradio_local_demo*.py` demos in this
|
||||
directory so both flat and packaged demos read consistently.
|
||||
|
||||
> **Status: draft.** This package is structurally in place but will not run
|
||||
> against the current upstream `fastvideo` package. See *Blocking prereqs*
|
||||
> below.
|
||||
|
||||
## Layout
|
||||
|
||||
| File | Purpose |
|
||||
| --- | --- |
|
||||
| `app.py` | `main()` — CLI args, `VideoGenerator` / `SamplingParam` boot, FastAPI mount with logo/favicon/generated-clip routes, `uvicorn.run`. |
|
||||
| `config.py` | Constants, defaults, env-var resolution, Inductor tuning flags, `setup_model_environment`, `resolve_model_path`, `resolve_refine_upsampler_path`, `apply_ltx2_defaults`. |
|
||||
| `safety.py` | fastText NSFW + hate-speech classifiers, `PromptSafetyCheck`, `get_prompt_safety_check`. |
|
||||
| `prompt_rewrite.py` | Cerebras-backed prompt enhancer wrapper (`maybe_enhance_prompt`, `get_prompt_enhancer`). Curated prompts bypass enhancement. |
|
||||
| `prompt_enhancer.py` | Cerebras API client used by `prompt_rewrite.py`. |
|
||||
| `rendering.py` | HTML helpers: timing cards, error cards, completed-clip gallery, image-upload status. |
|
||||
| `examples.py` | `load_example_prompts` — reads `selected_ltx2_prompts.jsonl`. |
|
||||
| `ui.py` | `create_gradio_interface` — Gradio Blocks, CSS, event wiring, generation closure. |
|
||||
| `__main__.py` | Enables `python -m gradio_local_demo_ltx2_3`. |
|
||||
| `__init__.py` | Re-exports `main` from `app`. |
|
||||
| `selected_ltx2_prompts.jsonl` | Curated example prompts. |
|
||||
| `prompts/prompt_extension_system_prompt.md` | System prompt for the Cerebras enhancer. |
|
||||
| `download_fasttext_classifiers.py` | Helper to download NSFW/hate-speech classifier binaries from Hugging Face Hub. |
|
||||
|
||||
## How to run
|
||||
|
||||
```bash
|
||||
cd examples/inference/gradio/local
|
||||
python -m gradio_local_demo_ltx2_3 --port 7860
|
||||
```
|
||||
|
||||
GPU requirement: a single FP4-capable GPU (B200 or comparable) for the
|
||||
"real-time 1080p" speed claim. Lower tiers will still run but slower.
|
||||
|
||||
## Blocking prereqs (why this draft PR cannot be merged yet)
|
||||
|
||||
The upstream `fastvideo` package is missing three pieces that the demo
|
||||
currently depends on verbatim. Each needs its own upstreaming PR before this
|
||||
demo can actually boot:
|
||||
|
||||
1. **`fastvideo.layers.quantization.fp4_config.FP4Config`** — the demo sets
|
||||
`pipeline_config.dit_config.quant_config = FP4Config()` in `app.py`.
|
||||
Upstream only ships `absmax_fp8.py` and `base_config.py` under
|
||||
`fastvideo/layers/quantization/`.
|
||||
2. **LTX-2.3 refine / image-conditioning kwargs on `VideoGenerator`** —
|
||||
`ltx2_refine_enabled`, `ltx2_refine_upsampler_path`, `ltx2_refine_lora_path`,
|
||||
`ltx2_refine_num_inference_steps`, `ltx2_refine_guidance_scale`,
|
||||
`ltx2_refine_add_noise`, `ltx2_images`, `ltx2_image_crf`. Upstream
|
||||
`fastvideo/fastvideo_args.py` currently wires only `ltx2_vae_tiling`.
|
||||
The backing stages (`ltx2_refine.py`, `ltx2_i2v_conditioning.py`) are
|
||||
also missing from `fastvideo/pipelines/stages/`.
|
||||
3. **`fastvideo.configs.sample.base.SamplingParam`** — the import path used
|
||||
by this demo. Upstream moved sampling params to
|
||||
`fastvideo.api.sampling_param`. A re-export shim at the old path, or an
|
||||
import update here once the other two prereqs land, will resolve it.
|
||||
|
||||
## Environment variables
|
||||
|
||||
| Var | Default | Purpose |
|
||||
| --- | --- | --- |
|
||||
| `LTX2_3_MODEL_PATH` | `FastVideo/LTX-2.3-Distilled-Diffusers` | Model ID or local snapshot. |
|
||||
| `LTX2_CLASSIFIER_DIR` | package dir | Where to look for fastText classifiers. |
|
||||
| `LTX2_NSFW_CLASSIFIER_PATH` | — | Explicit path to NSFW classifier `.bin`. |
|
||||
| `LTX2_HATESPEECH_CLASSIFIER_PATH` | — | Explicit path to hate-speech classifier `.bin`. |
|
||||
| `LTX2_REFINE_UPSAMPLER_PATH` | — | Explicit path to the spatial upsampler dir. |
|
||||
| `FASTVIDEO_PROMPT_API_KEY` / `CEREBRAS_API_KEY` | — | Cerebras API key for prompt enhancement. When missing, enhancer returns the raw prompt. |
|
||||
| `LTX2_PROMPT_MODEL` | `gpt-oss-120b` | Cerebras model name for the enhancer. |
|
||||
| `LTX2_PROMPT_TEMPERATURE` | `1.0` | Enhancer LLM temperature. |
|
||||
| `LTX2_PROMPT_EXTENSION_SYSTEM_PROMPT_PATH` | `prompts/prompt_extension_system_prompt.md` | System prompt for the enhancer. |
|
||||
|
||||
## Fetching the safety classifiers
|
||||
|
||||
```bash
|
||||
python examples/inference/gradio/local/gradio_local_demo_ltx2_3/download_fasttext_classifiers.py
|
||||
```
|
||||
|
||||
The classifiers come from `allenai/dolma-jigsaw-fasttext-bigrams-{nsfw,hatespeech}`
|
||||
on Hugging Face Hub.
|
||||
@@ -0,0 +1,9 @@
|
||||
"""FastLTX-2.3 Gradio local demo package.
|
||||
|
||||
Split from the monolithic gradio_local_demo_ltx2_3.py for maintainability.
|
||||
Runtime entrypoint is `main` in .app.
|
||||
"""
|
||||
|
||||
from .app import main
|
||||
|
||||
__all__ = ["main"]
|
||||
@@ -0,0 +1,4 @@
|
||||
from .app import main
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,225 @@
|
||||
import argparse
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
import gradio as gr
|
||||
|
||||
from fastvideo.configs.pipelines.base import PipelineConfig
|
||||
from fastvideo.configs.sample.base import SamplingParam
|
||||
from fastvideo.entrypoints.video_generator import VideoGenerator
|
||||
from fastvideo.layers.quantization.fp4_config import FP4Config
|
||||
from fastvideo.utils import maybe_download_model
|
||||
|
||||
from .config import (
|
||||
GENERATED_CLIP_ROOT,
|
||||
MODEL_ID,
|
||||
apply_ltx2_defaults,
|
||||
resolve_model_path,
|
||||
resolve_refine_upsampler_path,
|
||||
setup_model_environment,
|
||||
)
|
||||
from .ui import create_gradio_interface
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="FastVideo Gradio Local Demo")
|
||||
parser.add_argument("--t2v_model_paths", type=str,
|
||||
default=MODEL_ID,
|
||||
help="Comma separated list of paths to the T2V model(s)")
|
||||
parser.add_argument("--host", type=str, default="0.0.0.0",
|
||||
help="Host to bind to")
|
||||
parser.add_argument("--port", type=int, default=7860,
|
||||
help="Port to bind to")
|
||||
args = parser.parse_args()
|
||||
gradio_temp_dir = os.path.abspath("outputs/gradio_tmp")
|
||||
os.makedirs(gradio_temp_dir, exist_ok=True)
|
||||
os.environ["GRADIO_TEMP_DIR"] = gradio_temp_dir
|
||||
generators = {}
|
||||
default_params = {}
|
||||
model_paths = args.t2v_model_paths.split(",")
|
||||
for model_path in model_paths:
|
||||
print(f"Loading model: {model_path}")
|
||||
setup_model_environment(model_path)
|
||||
resolved_model_input = str(resolve_model_path(model_path))
|
||||
model_root = maybe_download_model(resolved_model_input)
|
||||
resolved_model_path = Path(model_root)
|
||||
|
||||
pipeline_config = PipelineConfig.from_pretrained(str(resolved_model_path))
|
||||
pipeline_config.dit_config.quant_config = FP4Config()
|
||||
refine_upsampler_path = resolve_refine_upsampler_path(resolved_model_path)
|
||||
print(f"Using refine upsampler: {refine_upsampler_path}")
|
||||
|
||||
generators[model_path] = VideoGenerator.from_pretrained(
|
||||
str(resolved_model_path),
|
||||
num_gpus=1,
|
||||
ltx2_refine_enabled=True,
|
||||
ltx2_refine_upsampler_path=str(refine_upsampler_path),
|
||||
ltx2_refine_lora_path="", # disable refine LoRA for distilled model
|
||||
ltx2_refine_num_inference_steps=2,
|
||||
ltx2_refine_guidance_scale=1.0,
|
||||
ltx2_refine_add_noise=True,
|
||||
pipeline_config=pipeline_config,
|
||||
enable_torch_compile=True,
|
||||
enable_torch_compile_text_encoder=True,
|
||||
torch_compile_kwargs={
|
||||
"backend": "inductor",
|
||||
"fullgraph": True,
|
||||
"mode": "max-autotune-no-cudagraphs",
|
||||
"dynamic": False,
|
||||
},
|
||||
dit_cpu_offload=False,
|
||||
vae_cpu_offload=False,
|
||||
text_encoder_cpu_offload=False,
|
||||
ltx2_vae_tiling=False,
|
||||
)
|
||||
default_params[model_path] = apply_ltx2_defaults(
|
||||
SamplingParam.from_pretrained(str(resolved_model_path))
|
||||
)
|
||||
demo = create_gradio_interface(default_params, generators)
|
||||
print(f"Starting Gradio frontend at http://{args.host}:{args.port}")
|
||||
print(f"T2V Models: {args.t2v_model_paths}")
|
||||
|
||||
from fastapi import FastAPI, Request, HTTPException
|
||||
from fastapi.responses import HTMLResponse, FileResponse
|
||||
import uvicorn
|
||||
|
||||
app = FastAPI()
|
||||
|
||||
@app.get("/logo.png")
|
||||
def get_logo():
|
||||
return FileResponse(
|
||||
"assets/full.svg",
|
||||
media_type="image/svg+xml",
|
||||
headers={
|
||||
"Cache-Control": "public, max-age=3600",
|
||||
"Access-Control-Allow-Origin": "*"
|
||||
}
|
||||
)
|
||||
|
||||
@app.get("/nvidia.png")
|
||||
def get_nvidia_logo():
|
||||
return FileResponse(
|
||||
"assets/nv.png",
|
||||
media_type="image/png",
|
||||
headers={
|
||||
"Cache-Control": "public, max-age=3600",
|
||||
"Access-Control-Allow-Origin": "*"
|
||||
}
|
||||
)
|
||||
|
||||
@app.get("/favicon.ico")
|
||||
def get_favicon():
|
||||
favicon_path = "assets/icon-simple.svg"
|
||||
|
||||
if os.path.exists(favicon_path):
|
||||
return FileResponse(
|
||||
favicon_path,
|
||||
media_type="image/svg+xml",
|
||||
headers={
|
||||
"Cache-Control": "public, max-age=3600",
|
||||
"Access-Control-Allow-Origin": "*"
|
||||
}
|
||||
)
|
||||
else:
|
||||
raise HTTPException(status_code=404, detail="Favicon not found")
|
||||
|
||||
@app.get("/generated-clips/{clip_path:path}")
|
||||
def get_generated_clip(clip_path: str):
|
||||
root = GENERATED_CLIP_ROOT.resolve()
|
||||
resolved_path = (root / clip_path).resolve()
|
||||
|
||||
if root not in resolved_path.parents or not resolved_path.is_file():
|
||||
raise HTTPException(status_code=404, detail="Clip not found")
|
||||
|
||||
return FileResponse(
|
||||
resolved_path,
|
||||
media_type="video/mp4",
|
||||
headers={
|
||||
"Cache-Control": "no-store",
|
||||
"Access-Control-Allow-Origin": "*",
|
||||
},
|
||||
)
|
||||
|
||||
@app.get("/", response_class=HTMLResponse)
|
||||
def index(request: Request):
|
||||
base_url = str(request.base_url).rstrip('/')
|
||||
return f"""
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
|
||||
<title>FastLTX-2.3</title>
|
||||
<meta name="title" content="FastLTX-2.3">
|
||||
<meta name="description" content="Make video generation go blurrrrrrr">
|
||||
<meta name="keywords" content="FastVideo, video generation, AI, machine learning, FastLTX-2.3">
|
||||
|
||||
<meta property="og:type" content="website">
|
||||
<meta property="og:url" content="{base_url}/">
|
||||
<meta property="og:title" content="FastLTX-2.3">
|
||||
<meta property="og:description" content="Make video generation go blurrrrrrr">
|
||||
<meta property="og:image" content="{base_url}/logo.png">
|
||||
<meta property="og:image:width" content="1200">
|
||||
<meta property="og:image:height" content="630">
|
||||
<meta property="og:site_name" content="FastLTX-2.3">
|
||||
|
||||
<meta property="twitter:card" content="summary_large_image">
|
||||
<meta property="twitter:url" content="{base_url}/">
|
||||
<meta property="twitter:title" content="FastLTX-2.3">
|
||||
<meta property="twitter:description" content="Make video generation go blurrrrrrr">
|
||||
<meta property="twitter:image" content="{base_url}/logo.png">
|
||||
<link rel="icon" type="image/png" sizes="32x32" href="/favicon.ico">
|
||||
<link rel="icon" type="image/png" sizes="16x16" href="/favicon.ico">
|
||||
<link rel="apple-touch-icon" href="/favicon.ico">
|
||||
<style>
|
||||
body, html {{
|
||||
margin: 0;
|
||||
padding: 0;
|
||||
min-height: 100%;
|
||||
width: 100%;
|
||||
background: #000;
|
||||
background-color: #000;
|
||||
background-image: none;
|
||||
overscroll-behavior-y: auto;
|
||||
scroll-behavior: smooth;
|
||||
}}
|
||||
body {{
|
||||
position: relative;
|
||||
}}
|
||||
body::before {{
|
||||
content: "";
|
||||
position: fixed;
|
||||
inset: 0;
|
||||
background: #000;
|
||||
pointer-events: none;
|
||||
z-index: -1;
|
||||
}}
|
||||
iframe {{
|
||||
display: block;
|
||||
width: 100%;
|
||||
height: 100vh;
|
||||
background: #000;
|
||||
background-color: #000;
|
||||
background-image: none;
|
||||
border: none;
|
||||
}}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<iframe src="/gradio" width="100%" height="100%" style="border: none;"></iframe>
|
||||
</body>
|
||||
</html>
|
||||
"""
|
||||
|
||||
app = gr.mount_gradio_app(
|
||||
app,
|
||||
demo,
|
||||
path="/gradio",
|
||||
allowed_paths=[
|
||||
os.path.abspath("outputs"),
|
||||
os.path.abspath("outputs_video"),
|
||||
os.path.abspath("fastvideo-logos"),
|
||||
]
|
||||
)
|
||||
|
||||
uvicorn.run(app, host=args.host, port=args.port)
|
||||
@@ -0,0 +1,90 @@
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
import torch
|
||||
import torch._inductor.config
|
||||
|
||||
from fastvideo.configs.sample.base import SamplingParam
|
||||
|
||||
LOCAL_DEMO_DIR = Path(__file__).resolve().parent
|
||||
CLASSIFIER_DIR = Path(
|
||||
os.path.expandvars(
|
||||
os.path.expanduser(
|
||||
os.getenv("LTX2_CLASSIFIER_DIR", str(LOCAL_DEMO_DIR))
|
||||
)
|
||||
)
|
||||
)
|
||||
|
||||
MODEL_ID = os.path.expandvars(
|
||||
os.path.expanduser(
|
||||
os.getenv("LTX2_3_MODEL_PATH", "FastVideo/LTX-2.3-Distilled-Diffusers")
|
||||
)
|
||||
)
|
||||
MODEL_PATH_MAPPING = {
|
||||
"FastLTX-2.3": MODEL_ID,
|
||||
}
|
||||
|
||||
DEFAULT_HEIGHT = 1088
|
||||
DEFAULT_WIDTH = 1920
|
||||
DEFAULT_NUM_FRAMES = 121
|
||||
DEFAULT_FPS = 24
|
||||
DEFAULT_GUIDANCE_SCALE = 1.0
|
||||
DEFAULT_NUM_INFERENCE_STEPS = 5
|
||||
DEFAULT_SEED = 10
|
||||
DEFAULT_NEGATIVE_PROMPT = ""
|
||||
REFINE_UPSAMPLER_PATH = "converted/ltx2_spatial_upscaler"
|
||||
REPO_ROOT = Path(__file__).resolve().parents[5]
|
||||
OUTPUT_DIR = REPO_ROOT / "outputs_video" / "ltx2_basic_new"
|
||||
GENERATED_CLIP_ROOT = REPO_ROOT / "outputs_video"
|
||||
MAX_SESSION_CLIPS = 24
|
||||
|
||||
config = torch._inductor.config
|
||||
config.conv_1x1_as_mm = True
|
||||
config.coordinate_descent_tuning = True
|
||||
config.coordinate_descent_check_all_directions = True
|
||||
config.epilogue_fusion = False
|
||||
|
||||
def apply_ltx2_defaults(params: SamplingParam) -> SamplingParam:
|
||||
params.height = DEFAULT_HEIGHT
|
||||
params.width = DEFAULT_WIDTH
|
||||
params.num_frames = DEFAULT_NUM_FRAMES
|
||||
params.fps = DEFAULT_FPS
|
||||
params.guidance_scale = DEFAULT_GUIDANCE_SCALE
|
||||
params.num_inference_steps = DEFAULT_NUM_INFERENCE_STEPS
|
||||
params.seed = DEFAULT_SEED
|
||||
params.negative_prompt = DEFAULT_NEGATIVE_PROMPT
|
||||
return params
|
||||
|
||||
def resolve_model_path(model_path: str) -> Path:
|
||||
return Path(os.path.expandvars(os.path.expanduser(model_path)))
|
||||
|
||||
def resolve_refine_upsampler_path(model_path: Path) -> Path:
|
||||
candidates = [
|
||||
model_path / "spatial_upscaler",
|
||||
model_path / "spatial_upsampler",
|
||||
Path(os.path.expandvars(os.path.expanduser(REFINE_UPSAMPLER_PATH))),
|
||||
REPO_ROOT / REFINE_UPSAMPLER_PATH,
|
||||
]
|
||||
|
||||
env_path = os.getenv("LTX2_REFINE_UPSAMPLER_PATH")
|
||||
if env_path:
|
||||
candidates.insert(
|
||||
0, Path(os.path.expandvars(os.path.expanduser(env_path)))
|
||||
)
|
||||
|
||||
for candidate in candidates:
|
||||
if (candidate / "config.json").is_file():
|
||||
return candidate
|
||||
|
||||
checked = "\n".join(f" - {candidate}" for candidate in candidates)
|
||||
raise FileNotFoundError(
|
||||
"Could not find an LTX2 refine upsampler directory.\n"
|
||||
"Checked:\n"
|
||||
f"{checked}\n"
|
||||
"Set LTX2_REFINE_UPSAMPLER_PATH or update REFINE_UPSAMPLER_PATH."
|
||||
)
|
||||
|
||||
def setup_model_environment(model_path: str) -> None:
|
||||
_ = model_path
|
||||
os.environ["FASTVIDEO_ATTENTION_BACKEND"] = "FLASH_ATTN"
|
||||
os.environ["FASTVIDEO_STAGE_LOGGING"] = "1"
|
||||
+174
@@ -0,0 +1,174 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import shutil
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
|
||||
from huggingface_hub import hf_hub_download
|
||||
|
||||
LOCAL_DEMO_DIR = Path(__file__).resolve().parent
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ClassifierSpec:
|
||||
name: str
|
||||
repo_id: str
|
||||
source_filename: str
|
||||
target_filename: str
|
||||
env_var: str
|
||||
|
||||
|
||||
CLASSIFIERS = {
|
||||
"nsfw": ClassifierSpec(
|
||||
name="nsfw",
|
||||
repo_id="allenai/dolma-jigsaw-fasttext-bigrams-nsfw",
|
||||
source_filename="model.bin",
|
||||
target_filename="jigsaw_fasttext_bigrams_nsfw_final.bin",
|
||||
env_var="LTX2_NSFW_CLASSIFIER_PATH",
|
||||
),
|
||||
"hatespeech": ClassifierSpec(
|
||||
name="hatespeech",
|
||||
repo_id="allenai/dolma-jigsaw-fasttext-bigrams-hatespeech",
|
||||
source_filename="model.bin",
|
||||
target_filename="jigsaw_fasttext_bigrams_hatespeech_final.bin",
|
||||
env_var="LTX2_HATESPEECH_CLASSIFIER_PATH",
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def expand_path(path: str | Path) -> Path:
|
||||
return Path(os.path.expandvars(os.path.expanduser(str(path))))
|
||||
|
||||
|
||||
def default_output_dir() -> Path:
|
||||
raw_value = os.getenv("LTX2_CLASSIFIER_DIR", str(LOCAL_DEMO_DIR))
|
||||
return expand_path(raw_value)
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(
|
||||
description=(
|
||||
"Download the fastText safety classifiers used by "
|
||||
"gradio_local_demo_ltx2_3.py."
|
||||
)
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output-dir",
|
||||
type=Path,
|
||||
default=default_output_dir(),
|
||||
help=(
|
||||
"Directory where the demo looks for classifiers. Defaults to "
|
||||
"LTX2_CLASSIFIER_DIR or the local Gradio demo directory."
|
||||
),
|
||||
)
|
||||
parser.add_argument(
|
||||
"--classifier",
|
||||
choices=sorted(CLASSIFIERS),
|
||||
nargs="+",
|
||||
default=sorted(CLASSIFIERS),
|
||||
help="Subset of classifiers to download. Defaults to both.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--cache-dir",
|
||||
type=Path,
|
||||
default=None,
|
||||
help="Optional Hugging Face cache directory.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--force",
|
||||
action="store_true",
|
||||
help="Redownload and overwrite existing classifier files.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--token",
|
||||
default=None,
|
||||
help="Optional Hugging Face token for authenticated downloads.",
|
||||
)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def materialize_download(source: Path, destination: Path) -> None:
|
||||
if destination.exists() or destination.is_symlink():
|
||||
destination.unlink()
|
||||
|
||||
shutil.copy2(source, destination)
|
||||
|
||||
|
||||
def download_classifier(
|
||||
spec: ClassifierSpec,
|
||||
output_dir: Path,
|
||||
cache_dir: Path | None,
|
||||
force: bool,
|
||||
token: str | None,
|
||||
) -> Path:
|
||||
destination = output_dir / spec.target_filename
|
||||
if destination.is_file() and not force:
|
||||
print(f"[skip] {spec.name}: {destination}")
|
||||
return destination
|
||||
|
||||
print(
|
||||
f"[download] {spec.name}: "
|
||||
f"{spec.repo_id}/{spec.source_filename}"
|
||||
)
|
||||
cached_path = Path(
|
||||
hf_hub_download(
|
||||
repo_id=spec.repo_id,
|
||||
filename=spec.source_filename,
|
||||
cache_dir=cache_dir,
|
||||
force_download=force,
|
||||
token=token,
|
||||
)
|
||||
)
|
||||
materialize_download(cached_path, destination)
|
||||
print(f"[ready] {spec.name}: {destination}")
|
||||
return destination
|
||||
|
||||
|
||||
def print_summary(
|
||||
selected_specs: list[ClassifierSpec],
|
||||
output_dir: Path,
|
||||
downloaded_paths: list[Path],
|
||||
) -> None:
|
||||
print("\nDownloaded classifier paths:")
|
||||
for path in downloaded_paths:
|
||||
print(f" - {path}")
|
||||
|
||||
print("\nThese filenames are auto-discovered by gradio_local_demo_ltx2_3.py.")
|
||||
|
||||
if output_dir != default_output_dir():
|
||||
print(
|
||||
"\nBecause you used a custom output directory, point the demo to it:"
|
||||
)
|
||||
print(f' export LTX2_CLASSIFIER_DIR="{output_dir}"')
|
||||
|
||||
for spec, path in zip(selected_specs, downloaded_paths, strict=True):
|
||||
print(f' export {spec.env_var}="{path}"')
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = parse_args()
|
||||
output_dir = expand_path(args.output_dir).resolve()
|
||||
cache_dir = None
|
||||
if args.cache_dir is not None:
|
||||
cache_dir = expand_path(args.cache_dir).resolve()
|
||||
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
selected_specs = [CLASSIFIERS[name] for name in args.classifier]
|
||||
downloaded_paths = [
|
||||
download_classifier(
|
||||
spec=spec,
|
||||
output_dir=output_dir,
|
||||
cache_dir=cache_dir,
|
||||
force=args.force,
|
||||
token=args.token,
|
||||
)
|
||||
for spec in selected_specs
|
||||
]
|
||||
print_summary(selected_specs, output_dir, downloaded_paths)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,67 @@
|
||||
import json
|
||||
import os
|
||||
|
||||
def load_example_prompts():
|
||||
examples: list[str] = []
|
||||
example_labels: list[str] = []
|
||||
prompts_path = os.path.join(
|
||||
os.path.dirname(__file__),
|
||||
"selected_ltx2_prompts.jsonl",
|
||||
)
|
||||
|
||||
try:
|
||||
with open(prompts_path, encoding="utf-8") as f:
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
entry = json.loads(line)
|
||||
prompt = entry.get("video_prompt", "")
|
||||
if not isinstance(prompt, str):
|
||||
continue
|
||||
prompt = prompt.strip()
|
||||
if not prompt:
|
||||
continue
|
||||
examples.append(prompt)
|
||||
example_labels.append(
|
||||
prompt[:100] + "..." if len(prompt) > 100 else prompt
|
||||
)
|
||||
except Exception as e:
|
||||
print(f"Warning: Could not read {prompts_path}: {e}")
|
||||
|
||||
if not examples:
|
||||
# Backward-compatible fallback to validation captions.
|
||||
validation_path = os.path.join(
|
||||
os.path.dirname(__file__),
|
||||
"..",
|
||||
"..",
|
||||
"..",
|
||||
"..",
|
||||
"distill",
|
||||
"LTX2",
|
||||
"validation.json",
|
||||
)
|
||||
try:
|
||||
with open(validation_path, encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
for entry in data.get("data", []):
|
||||
caption = entry.get("caption", "")
|
||||
if not isinstance(caption, str):
|
||||
continue
|
||||
caption = caption.strip()
|
||||
if not caption:
|
||||
continue
|
||||
examples.append(caption)
|
||||
example_labels.append(
|
||||
caption[:100] + "..." if len(caption) > 100 else caption
|
||||
)
|
||||
except Exception as e:
|
||||
print(f"Warning: Could not read {validation_path}: {e}")
|
||||
|
||||
if not examples:
|
||||
examples = [
|
||||
"A crowded rooftop bar buzzes with energy, the city skyline twinkling like a field of stars in the background."
|
||||
]
|
||||
example_labels = ["Crowded rooftop bar at night"]
|
||||
|
||||
return examples, example_labels
|
||||
@@ -0,0 +1,378 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
try:
|
||||
from cerebras.cloud.sdk import Cerebras
|
||||
except ImportError: # pragma: no cover - optional dependency
|
||||
Cerebras = None # type: ignore[assignment]
|
||||
|
||||
LOCAL_DEMO_DIR = Path(__file__).resolve().parent
|
||||
DEFAULT_SYSTEM_PROMPT_PATH = (
|
||||
LOCAL_DEMO_DIR / "prompts" / "prompt_extension_system_prompt.md"
|
||||
)
|
||||
DEFAULT_PROVIDER = "cerebras"
|
||||
DEFAULT_MODEL = "gpt-oss-120b"
|
||||
DEFAULT_TEMPERATURE = 1.0
|
||||
|
||||
|
||||
def _enhance_print(level: str, message: str) -> None:
|
||||
print(f"[ENHANCE][{level}] {message}", flush=True)
|
||||
|
||||
|
||||
@dataclass
|
||||
class EnhanceResult:
|
||||
prompt: str
|
||||
fallback_used: bool
|
||||
error: str | None
|
||||
provider: str
|
||||
model: str
|
||||
latency_ms: float
|
||||
|
||||
|
||||
def _env_value(*names: str) -> str | None:
|
||||
for name in names:
|
||||
value = os.getenv(name)
|
||||
if not isinstance(value, str):
|
||||
continue
|
||||
normalized = value.strip()
|
||||
if normalized:
|
||||
return normalized
|
||||
return None
|
||||
|
||||
|
||||
def _env_float(name: str, default: float) -> float:
|
||||
value = os.getenv(name)
|
||||
if value is None:
|
||||
return default
|
||||
try:
|
||||
return float(value)
|
||||
except ValueError:
|
||||
return default
|
||||
|
||||
|
||||
def _preview_text(text: str, *, limit: int = 160) -> str:
|
||||
normalized = text.replace("\n", "\\n")
|
||||
if len(normalized) <= limit:
|
||||
return normalized
|
||||
return normalized[:limit] + "..."
|
||||
|
||||
|
||||
def _expand_prompt_candidate_paths(path: str) -> list[Path]:
|
||||
prompt_path = Path(path)
|
||||
candidate_paths = [prompt_path]
|
||||
if prompt_path.suffix == ".txt":
|
||||
candidate_paths.append(prompt_path.with_suffix(".md"))
|
||||
elif prompt_path.suffix == ".md":
|
||||
candidate_paths.append(prompt_path.with_suffix(".txt"))
|
||||
return candidate_paths
|
||||
|
||||
|
||||
def _load_prompt_required(path: str, prompt_name: str) -> str:
|
||||
candidate_paths = _expand_prompt_candidate_paths(path)
|
||||
|
||||
for candidate in candidate_paths:
|
||||
if not candidate.is_file():
|
||||
continue
|
||||
try:
|
||||
text = candidate.read_text(encoding="utf-8").strip()
|
||||
except OSError as exc:
|
||||
raise RuntimeError(
|
||||
f"Failed to read {prompt_name} system prompt: {candidate}"
|
||||
) from exc
|
||||
if not text:
|
||||
raise RuntimeError(
|
||||
f"{prompt_name} system prompt file is empty: {candidate}"
|
||||
)
|
||||
return text
|
||||
|
||||
tried = ", ".join(str(candidate) for candidate in candidate_paths)
|
||||
raise RuntimeError(
|
||||
f"{prompt_name} system prompt file not found. Tried: {tried}"
|
||||
)
|
||||
|
||||
|
||||
def _extract_assistant_content(response_json: dict[str, Any]) -> str:
|
||||
choices = response_json.get("choices")
|
||||
if not isinstance(choices, list) or not choices:
|
||||
raise ValueError("Missing choices in chat completion response.")
|
||||
|
||||
message = choices[0].get("message", {})
|
||||
content = message.get("content")
|
||||
if isinstance(content, str):
|
||||
return content
|
||||
if isinstance(content, list):
|
||||
chunks: list[str] = []
|
||||
for item in content:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
text = item.get("text")
|
||||
if isinstance(text, str):
|
||||
chunks.append(text)
|
||||
continue
|
||||
if isinstance(text, dict):
|
||||
value = text.get("value")
|
||||
if isinstance(value, str):
|
||||
chunks.append(value)
|
||||
continue
|
||||
alt_text = item.get("output_text")
|
||||
if isinstance(alt_text, str):
|
||||
chunks.append(alt_text)
|
||||
if chunks:
|
||||
return "".join(chunks)
|
||||
|
||||
finish_reason = choices[0].get("finish_reason")
|
||||
refusal = message.get("refusal")
|
||||
raise ValueError(
|
||||
"Missing assistant content in chat completion response. "
|
||||
f"finish_reason={finish_reason!r}, refusal={refusal!r}"
|
||||
)
|
||||
|
||||
|
||||
def _dump_response_json(response: Any) -> dict[str, Any]:
|
||||
if hasattr(response, "model_dump"):
|
||||
payload = response.model_dump(mode="json")
|
||||
elif isinstance(response, dict):
|
||||
payload = response
|
||||
elif hasattr(response, "dict"):
|
||||
payload = response.dict()
|
||||
else:
|
||||
raise TypeError(
|
||||
"Unsupported chat completion response type. "
|
||||
f"type={type(response)!r}"
|
||||
)
|
||||
|
||||
if not isinstance(payload, dict):
|
||||
raise TypeError(
|
||||
"Chat completion response did not serialize to a JSON object."
|
||||
)
|
||||
return payload
|
||||
|
||||
|
||||
def _parse_json_response(content: str) -> dict[str, Any]:
|
||||
text = content.strip()
|
||||
if not text:
|
||||
raise ValueError("Assistant response is empty.")
|
||||
|
||||
try:
|
||||
parsed = json.loads(text)
|
||||
if isinstance(parsed, dict):
|
||||
return parsed
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
|
||||
fence_pattern = re.compile(
|
||||
r"```(?:json)?\s*([\s\S]*?)```",
|
||||
flags=re.IGNORECASE,
|
||||
)
|
||||
for match in fence_pattern.finditer(text):
|
||||
block = match.group(1).strip()
|
||||
if not block:
|
||||
continue
|
||||
try:
|
||||
parsed = json.loads(block)
|
||||
if isinstance(parsed, dict):
|
||||
return parsed
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
|
||||
decoder = json.JSONDecoder()
|
||||
for idx, char in enumerate(text):
|
||||
if char != "{":
|
||||
continue
|
||||
try:
|
||||
parsed, _ = decoder.raw_decode(text[idx:])
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
if isinstance(parsed, dict):
|
||||
return parsed
|
||||
|
||||
raise ValueError("No JSON object found in assistant response.")
|
||||
|
||||
|
||||
def _normalize_prompt(value: Any) -> str:
|
||||
if not isinstance(value, str):
|
||||
return ""
|
||||
return value.strip()
|
||||
|
||||
|
||||
class PromptEnhancer:
|
||||
def __init__(self) -> None:
|
||||
self.provider = DEFAULT_PROVIDER
|
||||
self.provider_label = DEFAULT_PROVIDER
|
||||
self.api_key = _env_value(
|
||||
"FASTVIDEO_PROMPT_API_KEY",
|
||||
"CEREBRAS_API_KEY",
|
||||
)
|
||||
self.model = _env_value("LTX2_PROMPT_MODEL") or DEFAULT_MODEL
|
||||
self.temperature = _env_float(
|
||||
"LTX2_PROMPT_TEMPERATURE",
|
||||
DEFAULT_TEMPERATURE,
|
||||
)
|
||||
self.system_prompt_path = _env_value(
|
||||
"LTX2_PROMPT_EXTENSION_SYSTEM_PROMPT_PATH"
|
||||
) or str(DEFAULT_SYSTEM_PROMPT_PATH)
|
||||
self.client: Any | None = None
|
||||
self.system_prompt: str | None = None
|
||||
self.unavailable_reason: str | None = None
|
||||
|
||||
try:
|
||||
self.client = self._build_client()
|
||||
self.system_prompt = _load_prompt_required(
|
||||
self.system_prompt_path,
|
||||
"prompt-extension",
|
||||
)
|
||||
except Exception as exc:
|
||||
self.unavailable_reason = str(exc)
|
||||
_enhance_print(
|
||||
"WARN",
|
||||
f"Prompt enhancement unavailable: {self.unavailable_reason}",
|
||||
)
|
||||
|
||||
def _build_client(self) -> Any:
|
||||
if Cerebras is None:
|
||||
raise RuntimeError(
|
||||
"Cerebras SDK is not installed. Install "
|
||||
"'cerebras-cloud-sdk' to use prompt enhancement."
|
||||
)
|
||||
if not self.api_key:
|
||||
raise RuntimeError(
|
||||
"Missing FASTVIDEO_PROMPT_API_KEY or CEREBRAS_API_KEY."
|
||||
)
|
||||
return Cerebras(api_key=self.api_key)
|
||||
|
||||
def _build_body(
|
||||
self,
|
||||
*,
|
||||
system_prompt: str,
|
||||
user_prompt: str,
|
||||
model: str | None = None,
|
||||
) -> dict[str, Any]:
|
||||
user_payload = {
|
||||
"request": (
|
||||
"Expand the user prompt into one detailed prompt for a "
|
||||
"single 5-second LTX-2.3 video. Respond with valid JSON "
|
||||
'only as {"prompt": "..."}.' # noqa: E501
|
||||
),
|
||||
"user_prompt": user_prompt,
|
||||
}
|
||||
return {
|
||||
"model": model or self.model,
|
||||
"temperature": self.temperature,
|
||||
"messages": [
|
||||
{
|
||||
"role": "system",
|
||||
"content": system_prompt,
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": json.dumps(user_payload, ensure_ascii=False),
|
||||
},
|
||||
],
|
||||
}
|
||||
|
||||
def _request_content(
|
||||
self,
|
||||
*,
|
||||
system_prompt: str,
|
||||
user_prompt: str,
|
||||
model: str | None = None,
|
||||
) -> tuple[dict[str, Any], str]:
|
||||
if self.client is None:
|
||||
raise RuntimeError(
|
||||
self.unavailable_reason
|
||||
or "Prompt enhancement client is not initialized."
|
||||
)
|
||||
|
||||
body = self._build_body(
|
||||
system_prompt=system_prompt,
|
||||
user_prompt=user_prompt,
|
||||
model=model,
|
||||
)
|
||||
response = self.client.chat.completions.create(
|
||||
model=body["model"],
|
||||
messages=body["messages"],
|
||||
temperature=body["temperature"],
|
||||
)
|
||||
response_json = _dump_response_json(response)
|
||||
return response_json, _extract_assistant_content(response_json)
|
||||
|
||||
def _require_prompt_field(
|
||||
self,
|
||||
parsed: dict[str, Any],
|
||||
field_name: str,
|
||||
) -> str:
|
||||
value = parsed.get(field_name)
|
||||
if not isinstance(value, str):
|
||||
raise ValueError(f"Missing {field_name} string.")
|
||||
prompt = value.strip()
|
||||
if not prompt:
|
||||
raise ValueError(f"{field_name} is empty.")
|
||||
return prompt
|
||||
|
||||
def enhance_prompt(self, prompt: str) -> EnhanceResult:
|
||||
cleaned = _normalize_prompt(prompt)
|
||||
if not cleaned:
|
||||
return EnhanceResult(
|
||||
prompt="",
|
||||
fallback_used=True,
|
||||
error="No valid prompt provided.",
|
||||
provider=self.provider_label,
|
||||
model=self.model,
|
||||
latency_ms=0.0,
|
||||
)
|
||||
|
||||
if self.system_prompt is None:
|
||||
return EnhanceResult(
|
||||
prompt="",
|
||||
fallback_used=True,
|
||||
error=(
|
||||
self.unavailable_reason
|
||||
or "Prompt enhancement system prompt is unavailable."
|
||||
),
|
||||
provider=self.provider_label,
|
||||
model=self.model,
|
||||
latency_ms=0.0,
|
||||
)
|
||||
|
||||
t0 = time.perf_counter()
|
||||
response_content: str | None = None
|
||||
try:
|
||||
_enhance_print("INFO", f"Enhancing prompt: {cleaned}")
|
||||
_, response_content = self._request_content(
|
||||
system_prompt=self.system_prompt,
|
||||
user_prompt=cleaned,
|
||||
)
|
||||
parsed = _parse_json_response(response_content)
|
||||
enhanced_prompt = self._require_prompt_field(parsed, "prompt")
|
||||
latency_ms = (time.perf_counter() - t0) * 1000.0
|
||||
return EnhanceResult(
|
||||
prompt=enhanced_prompt,
|
||||
fallback_used=False,
|
||||
error=None,
|
||||
provider=self.provider_label,
|
||||
model=self.model,
|
||||
latency_ms=latency_ms,
|
||||
)
|
||||
except Exception as exc:
|
||||
error_detail = str(exc)
|
||||
if isinstance(response_content, str) and response_content.strip():
|
||||
error_detail = (
|
||||
f"{error_detail} | assistant_response="
|
||||
f"{_preview_text(response_content, limit=240)}"
|
||||
)
|
||||
latency_ms = (time.perf_counter() - t0) * 1000.0
|
||||
return EnhanceResult(
|
||||
prompt="",
|
||||
fallback_used=True,
|
||||
error=error_detail,
|
||||
provider=self.provider_label,
|
||||
model=self.model,
|
||||
latency_ms=latency_ms,
|
||||
)
|
||||
@@ -0,0 +1,42 @@
|
||||
from functools import lru_cache
|
||||
|
||||
from .prompt_enhancer import PromptEnhancer
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def get_prompt_enhancer() -> PromptEnhancer:
|
||||
return PromptEnhancer()
|
||||
|
||||
def maybe_enhance_prompt(
|
||||
prompt: str,
|
||||
curated_prompts: set[str],
|
||||
) -> str:
|
||||
normalized_prompt = prompt.strip()
|
||||
if not normalized_prompt:
|
||||
return normalized_prompt
|
||||
|
||||
if normalized_prompt in curated_prompts:
|
||||
print(
|
||||
"[ENHANCE][INFO] Skipping prompt enhancement for curated prompt."
|
||||
)
|
||||
return normalized_prompt
|
||||
|
||||
enhancer = get_prompt_enhancer()
|
||||
if not enhancer.api_key:
|
||||
raise RuntimeError(
|
||||
"Prompt enhancement is enabled for custom prompts, but "
|
||||
"FASTVIDEO_PROMPT_API_KEY or CEREBRAS_API_KEY is not set."
|
||||
)
|
||||
|
||||
result = enhancer.enhance_prompt(normalized_prompt)
|
||||
if result.fallback_used or not result.prompt.strip():
|
||||
print(
|
||||
"[ENHANCE][WARN] Falling back to raw prompt "
|
||||
f"error={result.error}"
|
||||
)
|
||||
return normalized_prompt
|
||||
|
||||
print(
|
||||
"[ENHANCE][INFO] Prompt enhanced "
|
||||
f"latency={result.latency_ms:.2f}ms model={result.model}"
|
||||
)
|
||||
return result.prompt.strip()
|
||||
+138
@@ -0,0 +1,138 @@
|
||||
SYSTEM_PROMPT = """
|
||||
You are a prompt extender for LTX-2.3 video generation.
|
||||
|
||||
Your job is to expand a short user idea into a detailed, production-ready prompt for a single 5-second bidirectional video clip.
|
||||
|
||||
<context>
|
||||
LTX-2.3 responds strongly to detailed prompting. It performs best when prompts clearly specify:
|
||||
- the subject
|
||||
- the action
|
||||
- the environment
|
||||
- spatial layout
|
||||
- lighting
|
||||
- camera behavior
|
||||
- audio
|
||||
|
||||
LTX-2.3 is more faithful to prompt details than earlier versions. It can follow specific acting beats, pauses, physical reactions, camera directions, and environmental details more reliably.
|
||||
|
||||
For a 5-second clip, the prompt should still feel like one short, continuous cinematic moment, but it should be richly described.
|
||||
</context>
|
||||
|
||||
<task>
|
||||
Given a short user prompt, expand it into a detailed cinematic prompt optimized for a single 5-second LTX-2.3 video.
|
||||
|
||||
You must preserve the user’s subject, intent, and core action.
|
||||
You may enrich the scene, acting, environment, audio, and camera work, but you must not change the core premise.
|
||||
</task>
|
||||
|
||||
<core_principles>
|
||||
1. Be specific and descriptive
|
||||
- Add concrete visual details rather than vague summaries.
|
||||
- Include age, clothing, hair, material texture, lighting, atmosphere, and setting when relevant.
|
||||
|
||||
2. Direct the scene
|
||||
- Be explicit about spatial layout and orientation when useful:
|
||||
left, right, foreground, background, near, far, facing toward, facing away.
|
||||
|
||||
3. Use cinematic language
|
||||
- Use camera and film language naturally:
|
||||
medium shot, close-up, wide shot, low angle, over-the-shoulder, slow push in, pans across, tracks, shallow depth of field, handheld, golden hour, cold fluorescent, etc.
|
||||
|
||||
4. Use verbs for motion
|
||||
- Clearly describe who moves, what moves, how they move, and what the camera does.
|
||||
- Motion must be visible and physically plausible.
|
||||
|
||||
5. Describe audio clearly
|
||||
- If audio is relevant, describe ambient sound, dialogue tone, acoustic texture, and synced sounds.
|
||||
|
||||
6. Show emotion through physical performance
|
||||
- Prefer visible cues over abstract labels.
|
||||
- Use pauses, glances, small gestures, posture shifts, jaw tension, blinking, hand movement, breath, or voice quality.
|
||||
|
||||
7. Keep internal consistency
|
||||
- Do not introduce contradictory lighting, tone, or action.
|
||||
- Do not overload the shot with too many unrelated events.
|
||||
</core_principles>
|
||||
|
||||
<prompt_structure>
|
||||
Write one flowing paragraph in natural English.
|
||||
|
||||
The prompt should usually include:
|
||||
1. Shot type and subject
|
||||
2. Environment and spatial layout
|
||||
3. Lighting, palette, and texture
|
||||
4. Main action
|
||||
5. Small follow-up beat or reaction
|
||||
6. Camera movement if useful
|
||||
7. Audio and dialogue if relevant
|
||||
8. A stable ending image
|
||||
|
||||
For 5-second clips, the scene should feel like:
|
||||
- one continuous shot
|
||||
- one main action beat
|
||||
- one smaller reaction or follow-up beat
|
||||
- a stable visual hold at the end
|
||||
</prompt_structure>
|
||||
|
||||
<rules>
|
||||
1. Single continuous shot
|
||||
- Do not describe cuts or multiple scenes.
|
||||
- Treat the prompt as one short cinematic take.
|
||||
|
||||
2. Rich detail is encouraged
|
||||
- LTX-2.3 benefits from longer, more descriptive prompts.
|
||||
- Add enough detail to fully specify the 5-second clip.
|
||||
|
||||
3. Dialogue handling
|
||||
- If dialogue is present, put spoken words in quotation marks.
|
||||
- Break dialogue into short phrases when appropriate.
|
||||
- Insert visible acting directions between spoken phrases when useful.
|
||||
- Example pattern:
|
||||
He looks to the side and says, "I thought this was handled." He pauses, tightens his jaw, then adds, "Apparently not."
|
||||
- Keep dialogue natural and synchronized with visible action.
|
||||
|
||||
4. Physical acting
|
||||
- Prefer visible acting beats:
|
||||
pauses, eye shifts, hand adjustments, posture changes, small reactions.
|
||||
- Do not rely on internal thoughts or abstract emotional labels.
|
||||
|
||||
5. Camera movement
|
||||
- If camera movement is used, describe it clearly relative to the subject.
|
||||
- Use natural camera language, not technical numeric instructions.
|
||||
- For a 5-second clip, keep camera movement controlled and readable.
|
||||
|
||||
6. Texture and material
|
||||
- When useful, describe material qualities:
|
||||
glossy metal, worn fabric, fine hair strands, rough stone, wet pavement, polished floor, matte plastic, brushed steel.
|
||||
|
||||
7. Lighting
|
||||
- Use one coherent lighting logic:
|
||||
warm tungsten, cool fluorescent, golden hour sunlight, neon glow, moonlight, etc.
|
||||
- Avoid conflicting light descriptions.
|
||||
|
||||
8. Audio
|
||||
- Tie sound to visible action.
|
||||
- Keep audio specific:
|
||||
console beeps, chair creak, rain on glass, fluorescent hum, footsteps on tile, fabric rustle, distant chatter.
|
||||
- If dialogue is present, describe voice tone when useful.
|
||||
|
||||
9. Avoid
|
||||
- vague prompts
|
||||
- still-photo descriptions with no action
|
||||
- overloaded scenes with too many simultaneous actions
|
||||
- conflicting instructions
|
||||
- abstract emotional summaries
|
||||
- unreadable text/logo dependence
|
||||
- overly numerical constraints
|
||||
|
||||
10. Ending stability
|
||||
- End on a stable, readable frame.
|
||||
- The final image should feel visually settled rather than abruptly cut off.
|
||||
</rules>
|
||||
|
||||
<output_format>
|
||||
Return only the final extended prompt as a single paragraph in natural English.
|
||||
|
||||
Do not include headings, explanations, bullet points, or commentary.
|
||||
</output_format>
|
||||
"""
|
||||
@@ -0,0 +1,170 @@
|
||||
import html
|
||||
from pathlib import Path
|
||||
|
||||
from .config import DEFAULT_FPS, GENERATED_CLIP_ROOT, MAX_SESSION_CLIPS
|
||||
|
||||
def create_timing_display(inference_time, total_time, stage_execution_times, num_frames):
|
||||
timing_html = f"""
|
||||
<div class="timing-shell">
|
||||
<div style="display: grid; grid-template-columns: repeat(2, 1fr); gap: 10px; margin-bottom: 10px;">
|
||||
<div class="timing-card">
|
||||
<div style="font-size: 20px;">🎬</div>
|
||||
<div style="font-weight: bold; margin: 3px 0; font-size: 14px;">Video Generation Time</div>
|
||||
<div style="font-size: 18px; color: #2563eb;">{inference_time:.1f}s</div>
|
||||
</div>
|
||||
<div class="timing-card timing-card-highlight">
|
||||
<div style="font-size: 20px;">📊</div>
|
||||
<div style="font-weight: bold; margin: 3px 0; font-size: 14px;">E2E Latency</div>
|
||||
<div style="font-size: 18px; color: #0277bd;">{total_time:.1f}s</div>
|
||||
</div>
|
||||
</div>"""
|
||||
|
||||
if inference_time > 0:
|
||||
fps = num_frames / inference_time
|
||||
timing_html += f"""
|
||||
<div class="performance-card" style="margin-top: 15px;">
|
||||
<span style="font-weight: bold;">Generation Speed: </span>
|
||||
<span style="font-size: 18px; color: #6366f1; font-weight: bold;">{fps:.1f} frames/second</span>
|
||||
</div>"""
|
||||
|
||||
return timing_html + "</div>"
|
||||
|
||||
def create_timing_placeholder() -> str:
|
||||
return """
|
||||
<div class="timing-shell">
|
||||
<div style="display: grid; grid-template-columns: repeat(2, 1fr); gap: 10px; margin-bottom: 10px;">
|
||||
<div class="timing-card">
|
||||
<div style="font-weight: bold; margin: 3px 0; font-size: 14px;">Video Generation Time</div>
|
||||
<div style="font-size: 18px; color: #4f8cff;">--</div>
|
||||
</div>
|
||||
<div class="timing-card timing-card-highlight">
|
||||
<div style="font-weight: bold; margin: 3px 0; font-size: 14px;">E2E Latency</div>
|
||||
<div style="font-size: 18px; color: #4f8cff;">--</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="performance-card">
|
||||
<span style="font-weight: bold;">Generation Speed: </span>
|
||||
<span style="font-size: 18px; color: #4f8cff; font-weight: bold;">--</span>
|
||||
</div>
|
||||
</div>
|
||||
"""
|
||||
|
||||
def _truncate_text(value: str, max_chars: int) -> str:
|
||||
if len(value) <= max_chars:
|
||||
return value
|
||||
return value[: max_chars - 1].rstrip() + "..."
|
||||
|
||||
def _clip_duration_seconds(num_frames: int, fps: int) -> int:
|
||||
return max(1, round(num_frames / max(fps, 1)))
|
||||
|
||||
def _make_clip_public_path(output_path: str) -> str:
|
||||
resolved_path = Path(output_path).resolve()
|
||||
relative_path = resolved_path.relative_to(GENERATED_CLIP_ROOT.resolve())
|
||||
return f"/generated-clips/{relative_path.as_posix()}"
|
||||
|
||||
def _record_session_clip(
|
||||
session_clips: list[dict[str, str | int | float]],
|
||||
*,
|
||||
output_path: str,
|
||||
prompt: str,
|
||||
model_name: str,
|
||||
num_frames: int,
|
||||
generation_time: float,
|
||||
) -> list[dict[str, str | int | float]]:
|
||||
clip_entry = {
|
||||
"video_url": _make_clip_public_path(output_path),
|
||||
"prompt": prompt.strip(),
|
||||
"prompt_preview": _truncate_text(prompt.strip(), 120),
|
||||
"model_name": model_name,
|
||||
"duration_label": f"{_clip_duration_seconds(num_frames, DEFAULT_FPS)} Sec",
|
||||
"num_frames": num_frames,
|
||||
"generation_time": generation_time,
|
||||
}
|
||||
updated_clips = list(session_clips)
|
||||
updated_clips.append(clip_entry)
|
||||
if len(updated_clips) > MAX_SESSION_CLIPS:
|
||||
updated_clips = updated_clips[-MAX_SESSION_CLIPS:]
|
||||
return updated_clips
|
||||
|
||||
def render_completed_clips(clips: list[dict[str, str | int | float]]) -> str:
|
||||
if not clips:
|
||||
return """
|
||||
<div class="completed-clips-empty">
|
||||
<div class="completed-clips-empty-title">Nothing in gallery yet</div>
|
||||
<div class="completed-clips-empty-copy">
|
||||
Build your personal gallery for this browser session by creating videos.
|
||||
</div>
|
||||
</div>
|
||||
"""
|
||||
|
||||
cards: list[str] = []
|
||||
for clip in reversed(clips):
|
||||
video_url = html.escape(str(clip["video_url"]), quote=True)
|
||||
prompt = str(clip["prompt"])
|
||||
prompt_preview = html.escape(str(clip["prompt_preview"]))
|
||||
model_name = html.escape(str(clip["model_name"]))
|
||||
duration_label = html.escape(str(clip["duration_label"]))
|
||||
cards.append(
|
||||
f"""
|
||||
<article class="completed-clip-card">
|
||||
<div class="completed-clip-video-shell">
|
||||
<video class="completed-clip-video" src="{video_url}" controls preload="metadata" playsinline></video>
|
||||
</div>
|
||||
<div class="completed-clip-body">
|
||||
<div class="completed-clip-title" title="{html.escape(prompt, quote=True)}">{prompt_preview}</div>
|
||||
<div class="completed-clip-meta">
|
||||
<span class="completed-clip-badge">{model_name}</span>
|
||||
<span class="completed-clip-badge completed-clip-duration">{duration_label}</span>
|
||||
</div>
|
||||
<details class="completed-clip-prompt">
|
||||
<summary>Prompt</summary>
|
||||
<div>{html.escape(prompt)}</div>
|
||||
</details>
|
||||
</div>
|
||||
</article>
|
||||
"""
|
||||
)
|
||||
|
||||
return f"""
|
||||
<div class="completed-clips-grid">
|
||||
{''.join(cards)}
|
||||
</div>
|
||||
"""
|
||||
|
||||
def render_error_message(message: str) -> str:
|
||||
return f"""
|
||||
<div class="stage-error-card">
|
||||
<div class="stage-error-title">Error</div>
|
||||
<div class="stage-error-copy">{html.escape(message)}</div>
|
||||
</div>
|
||||
"""
|
||||
|
||||
def render_prompt_blocked_message(
|
||||
message: str,
|
||||
category: str | None = None,
|
||||
) -> str:
|
||||
details = ""
|
||||
if category:
|
||||
details = (
|
||||
'<div class="stage-error-copy">'
|
||||
f"Policy: {html.escape(category)}"
|
||||
"</div>"
|
||||
)
|
||||
return f"""
|
||||
<div class="stage-error-card">
|
||||
<div class="stage-error-title">Prompt Blocked</div>
|
||||
{details}
|
||||
<div class="stage-error-copy">{html.escape(message)}</div>
|
||||
</div>
|
||||
"""
|
||||
|
||||
def render_input_image_status(input_image: str | None) -> str:
|
||||
if not input_image:
|
||||
return ""
|
||||
|
||||
image_name = html.escape(Path(str(input_image)).name)
|
||||
return (
|
||||
"<div class='image-upload-status'>"
|
||||
f"Image ready: {image_name}"
|
||||
"</div>"
|
||||
)
|
||||
@@ -0,0 +1,165 @@
|
||||
import os
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from functools import lru_cache
|
||||
from pathlib import Path
|
||||
|
||||
import fasttext
|
||||
|
||||
from .config import CLASSIFIER_DIR
|
||||
|
||||
def resolve_classifier_path(
|
||||
classifier_kind: str,
|
||||
env_var: str,
|
||||
filename: str,
|
||||
legacy_path: str,
|
||||
shared_filename: str,
|
||||
) -> str:
|
||||
candidates: list[Path] = []
|
||||
env_path = os.getenv(env_var)
|
||||
if env_path:
|
||||
candidates.append(
|
||||
Path(os.path.expandvars(os.path.expanduser(env_path)))
|
||||
)
|
||||
candidates.extend(
|
||||
[
|
||||
CLASSIFIER_DIR / filename,
|
||||
Path(f"/home/shared/{shared_filename}"),
|
||||
Path(legacy_path),
|
||||
]
|
||||
)
|
||||
|
||||
for candidate in candidates:
|
||||
if candidate.is_file():
|
||||
return str(candidate)
|
||||
|
||||
checked = "\n".join(f" - {candidate}" for candidate in candidates)
|
||||
raise FileNotFoundError(
|
||||
f"Could not find the {classifier_kind} classifier.\n"
|
||||
f"Checked:\n{checked}\n"
|
||||
"Run "
|
||||
"`python examples/inference/gradio/local/download_fasttext_classifiers.py` "
|
||||
"or set the appropriate classifier path environment variable."
|
||||
)
|
||||
|
||||
def fasttext_predict(
|
||||
model_path: str,
|
||||
text: str,
|
||||
classifier_name: str,
|
||||
) -> tuple[str, float]:
|
||||
model = load_fasttext_model(model_path)
|
||||
text = text.replace('\n', ' ')
|
||||
start_time = time.perf_counter()
|
||||
try:
|
||||
labels, probs = model.predict(text)
|
||||
except ValueError as error:
|
||||
if "Unable to avoid copy while creating an array" not in str(error):
|
||||
raise
|
||||
predictions = model.f.predict(f"{text}\n", 1, 0.0, "strict")
|
||||
if not predictions:
|
||||
raise ValueError("fastText returned no predictions") from error
|
||||
probs, labels = zip(*predictions)
|
||||
latency_ms = (time.perf_counter() - start_time) * 1000.0
|
||||
identifier = labels[0].replace('__label__', '')
|
||||
confidence = probs[0]
|
||||
print(
|
||||
"[safety] "
|
||||
f"{classifier_name} fastText latency={latency_ms:.2f}ms "
|
||||
f"label={identifier} confidence={float(confidence):.4f}"
|
||||
)
|
||||
return identifier, confidence
|
||||
|
||||
@lru_cache(maxsize=None)
|
||||
def load_fasttext_model(model_path: str):
|
||||
return fasttext.load_model(model_path)
|
||||
|
||||
def classify_nsfw(text: str) -> tuple[str, float]:
|
||||
return fasttext_predict(
|
||||
resolve_classifier_path(
|
||||
"NSFW",
|
||||
"LTX2_NSFW_CLASSIFIER_PATH",
|
||||
"jigsaw_fasttext_bigrams_nsfw_final.bin",
|
||||
"/data/classifiers/dolma_fasttext_nsfw_jigsaw_model.bin",
|
||||
"dolma-jigsaw-fasttext-bigrams-nsfw-final.bin",
|
||||
),
|
||||
text,
|
||||
"nsfw",
|
||||
)
|
||||
|
||||
def classify_toxic_speech(text: str) -> tuple[str, float]:
|
||||
return fasttext_predict(
|
||||
resolve_classifier_path(
|
||||
"hate speech",
|
||||
"LTX2_HATESPEECH_CLASSIFIER_PATH",
|
||||
"jigsaw_fasttext_bigrams_hatespeech_final.bin",
|
||||
"/data/classifiers/dolma_fasttext_hatespeech_jigsaw_model.bin",
|
||||
"dolma-jigsaw-fasttext-bigrams-hatespeech-final.bin",
|
||||
),
|
||||
text,
|
||||
"hate_speech",
|
||||
)
|
||||
|
||||
def _normalize_classifier_label(identifier: str) -> str:
|
||||
return identifier.strip().lower().replace("-", "_").replace(" ", "_")
|
||||
|
||||
def _label_matches(
|
||||
identifier: str,
|
||||
blocked_markers: tuple[str, ...],
|
||||
safe_markers: tuple[str, ...],
|
||||
) -> bool:
|
||||
normalized = _normalize_classifier_label(identifier)
|
||||
if any(marker in normalized for marker in safe_markers):
|
||||
return False
|
||||
return any(marker in normalized for marker in blocked_markers)
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class PromptSafetyCheck:
|
||||
blocked: bool
|
||||
category: str | None = None
|
||||
message: str | None = None
|
||||
|
||||
def get_prompt_safety_check(prompt: str) -> PromptSafetyCheck:
|
||||
normalized_prompt = prompt.strip()
|
||||
if not normalized_prompt:
|
||||
return PromptSafetyCheck(blocked=False)
|
||||
|
||||
nsfw_label, _ = classify_nsfw(normalized_prompt)
|
||||
if _label_matches(
|
||||
nsfw_label,
|
||||
blocked_markers=("nsfw",),
|
||||
safe_markers=("sfw", "safe"),
|
||||
):
|
||||
return PromptSafetyCheck(
|
||||
blocked=True,
|
||||
category="NSFW",
|
||||
message=(
|
||||
"This request was blocked by the safety filter because it "
|
||||
"appears to contain NSFW content. Please revise the prompt "
|
||||
"and try again."
|
||||
),
|
||||
)
|
||||
|
||||
hate_label, _ = classify_toxic_speech(normalized_prompt)
|
||||
if _label_matches(
|
||||
hate_label,
|
||||
blocked_markers=("hatespeech", "hate", "toxic", "offensive", "abusive"),
|
||||
safe_markers=(
|
||||
"non_hatespeech",
|
||||
"not_hatespeech",
|
||||
"non_toxic",
|
||||
"not_toxic",
|
||||
"safe",
|
||||
"clean",
|
||||
),
|
||||
):
|
||||
return PromptSafetyCheck(
|
||||
blocked=True,
|
||||
category="Hate Speech",
|
||||
message=(
|
||||
"This request was blocked by the safety filter because it "
|
||||
"appears to contain hate speech or abusive content. Please "
|
||||
"revise the prompt and try again."
|
||||
),
|
||||
)
|
||||
|
||||
return PromptSafetyCheck(blocked=False)
|
||||
@@ -0,0 +1,48 @@
|
||||
{"id": "vidprom_semantic_unique_gpt-5-mini-2025-08-07_14", "video_prompt": "An old man in his late 70s with a short grey beard and a worn brown coat is sitting with his back against the wide, rough bark of a big oak tree, legs stretched on the cool grass as three children—an 8-year-old girl with braided hair in a denim jacket, a 10-year-old boy in a red hoodie, and a 6-year-old girl in a yellow dress—are clustered close, leaning in and looking up at him under a clear, starry sky. Soft moonlight is washing the scene in pale blue while dozens of fireflies are flickering around them, casting small points of warm yellow light that pulse in time with a gentle breeze rustling the oak leaves; the breeze causes a soft, continuous rustle and occasional low creaks from the branches. The old man is speaking slowly and warmly, his weathered hand gesturing toward a knot in the tree as he tells the story; beneath his voice a quiet night soundscape is present—distant cricket chirps, mild wind through grass, and the subtle shuffle of the children as they shift closer. Old Man (deep, slow, warm): \"Long ago this tree used to hold a lantern that guided lost travelers...\" he says, pausing to tap the knot and smile, his voice carrying low and steady over the soft night sounds. Boy (bright, quick, curious): \"Did they ever find their way without the lantern?\" the boy asks, eyes wide, leaning forward and brushing a blade of grass as the fireflies flicker near his hand. Old Man (soft, amused, steady): \"Sometimes they did, sometimes they learned to follow the stars,\" he replies, chuckling softly and nodding, his words blending with the rustle of leaves; the children exhale in a small, collective whisper of wonder and a brief, delighted giggle rises as a firefly glows nearby. Throughout, the audio remains intimate and natural—clear, close-up dialogue with the old man's voice dominant, layered over gentle ambient night SFX (wind through leaves, distant insects) and occasional tiny, bright pops of light from the fireflies visually accenting beats in the conversation."}
|
||||
{"id": "vidprom_semantic_unique_gpt-5-mini-2025-08-07_16", "video_prompt": "A slender alien with pale green skin, sparse white tendrils for hair, and large dark eyes is crouched before an old cathode-ray TV sitting on a scratched metal crate in a dim, cluttered observation alcove; they wear a simple gray tunic and lean forward with a focused, curious expression, fingertips hovering over the TV's worn knobs as a soft amber glow from the curved screen washes across their face. The CRT displays a grainy, monochrome rotating Earth with visible scanlines and intermittent static; as the alien turns a dial the globe sharpens briefly then jitters with horizontal rolling lines, while the TV cabinet emits a low electrical hum, a steady mechanical whine, and sharp, brief pops from the speaker. The alien tilts their head, squints, and taps the side of the set, then speaks in a soft, slow, curious voice, \"That look like home?\" A crackly, low-pitched announcer voice from the TV replies in a distorted, monotone cadence, \"Signal detected: human transmissions—faint, local.\" The alien exhales a small, puzzled sound and murmurs back in a quieter, puzzled tone, \"Listen close… what are they saying?\" Tiny ambient sounds layer under the exchange: distant ship machinery humming, the gentle rattle of tools in the alcove, the TV's hiss and white noise filling pauses, and a faint, muffled snippet of Earth's city ambience—distant car horns and a passing voice—bleeding through the static as the alien leans in while the scene holds for a brief, attentive moment."}
|
||||
{"id": "vidprom_semantic_unique_gpt-5-mini-2025-08-07_29", "video_prompt": "A young girl with shoulder-length brown hair tied in a bouncing ponytail is running down a sunlit suburban street, wearing a knee-length pink skirt and a fitted blue t-shirt; her white sneakers slam in quick, rhythmic strikes against rough asphalt as the skirt flutters and her ponytail snaps with each stride. Soft late-afternoon light casts long, gentle shadows across the concrete curb and the texture of small pebbles in the road is visible; a few parked cars line the sidewalk and a stray leaf skitters along at her feet. Footstep SFX: rapid sneaker impacts, light skirt swish, and a short intake of breath on each stride; ambient sound: distant traffic hum, a passing car whoosh and faint bird calls. Mid-run she glances over her shoulder, pausing her pace just enough to call out in a breathy, urgent, slightly high-pitched voice, “Wait up!” (spoken with quick cadence), then exhales with a soft pant and pushes forward again, her arms pumping and shoes kicking up tiny dust puffs as the street ambience continues—occasional distant horn and a muted dog bark—while the sound of her footsteps and breath remain prominent and in sync with her motion."}
|
||||
{"id": "vidprom_semantic_unique_gpt-5-mini-2025-08-07_37", "video_prompt": "A medium close-up of a middle-aged moderator at a wooden podium, wearing a dark navy suit and thin-rim glasses, short salt-and-pepper hair slightly tousled, smiling with bright eyes as he gestures with one hand and leans slightly forward; soft overhead conference lighting casts even, neutral illumination and a large LED screen behind him shows a pulsing schematic of interconnected nodes labeled Intelligent Neural Net in plain white text. Moderator — excited, slightly breathless, fast pace, mid-high pitch: \"Moderator: (Excitedly) Finally, we have succeeded in building the most advanced super AI system, the "Intelligent Neural Net"! It will be the most powerful AI system in human history!\" He speaks the first sentence with a rising inflection while raising both hands, then taps the side of the podium on the last phrase, causing a brief synthesized chime and the LED schematic to glow brighter; a soft microphone pop precedes his voice, and immediately as he finishes the line a swell of applause and cheers rises from the off-screen audience, mixed with a low, steady server-rack hum and the faint clicking of camera shutters. Subtle ambient conference sounds (murmur, chair rustle) sit under the action while the projection pulses in time with the chime, and the moderator holds his smile, breathing slightly faster, as the applause continues."}
|
||||
{"id": "vidprom_semantic_unique_gpt-5-mini-2025-08-07_38", "video_prompt": "A small village of stone cottages with a mix of thatched and slate roofs is nestled in a shallow valley between low, misty hills under a serene moonlit night; pale moonlight is casting a cool silver wash over dewy grass and slate tiles while soft, warm light is spilling from leaded glass windows onto narrow cobblestone lanes. Thin wisps of mist are drifting down the hill slopes and curling through the streets as thin streams of smoke are rising from chimneys and curling up into the moonlit air; lanterns hanging from wrought-iron brackets are gently swinging and candle flames behind shutters are flickering, casting subtle shadows across textured stone walls and wooden doors. In the background, a narrow brook is murmuring over stones and a distant church bell is tolling once, while crickets are chirping steadily and an occasional owl is calling from the dark hillside; a low breeze is rustling the leaves of a lone elm and causing reeds by the water to whisper. The overall palette is cool silvers and soft blues from the moon, contrasted with warm amber glows from windows and lanterns, with visible details like moss on stones, chipped plaster, and wet cobbles reflecting scattered light, creating a calm, intimate nighttime scene."}
|
||||
{"id": "vidprom_semantic_unique_gpt-5-mini-2025-08-07_38", "video_prompt": "A small village of stone cottages with a mix of thatched and slate roofs is nestled in a shallow valley between low, misty hills under a serene moonlit night; pale moonlight is casting a cool silver wash over dewy grass and slate tiles while soft, warm light is spilling from leaded glass windows onto narrow cobblestone lanes. Thin wisps of mist are drifting down the hill slopes and curling through the streets as thin streams of smoke are rising from chimneys and curling up into the moonlit air; lanterns hanging from wrought-iron brackets are gently swinging and candle flames behind shutters are flickering, casting subtle shadows across textured stone walls and wooden doors. In the background, a narrow brook is murmuring over stones and a distant church bell is tolling once, while crickets are chirping steadily and an occasional owl is calling from the dark hillside; a low breeze is rustling the leaves of a lone elm and causing reeds by the water to whisper. The overall palette is cool silvers and soft blues from the moon, contrasted with warm amber glows from windows and lanterns, with visible details like moss on stones, chipped plaster, and wet cobbles reflecting scattered light, creating a calm, intimate nighttime scene."}
|
||||
{"id": "vidprom_semantic_unique_gpt-5-mini-2025-08-07_51", "video_prompt": "A static medium shot of a woman in her late 20s with shoulder-length dark hair, wearing a plain green shirt and a flowery midi skirt, standing against a bright white background; she is holding a white ceramic pot at chest level that contains a large plant with oversized round leaves in alternating orange and green, the leaves showing a smooth, slightly waxy texture and gentle midrib veins. Soft studio lights create even, shadow-free illumination and crisp color separation; she is lifting the pot slightly and tilting it toward the camera as the leaves shift a little from the motion. Ambient audio is a quiet studio room tone with a soft, unobtrusive acoustic guitar loop underlining the moment; as she moves there is a faint rustle of fabric and the light sound of her hands adjusting on the pot. Woman (warm, medium pace, mid pitch): \"Look at these leaves—aren't they lovely?\" she smiles and holds the tilt, then pauses to glance down and smooth a leaf with her fingertip. Woman (brightening, slightly quicker): \"The orange really pops against the green,\" she says while rotating the pot a quarter turn so the leaf edges catch the light; a subtle short reverb on her voice and a small, gentle exhale sync with her final nod as she offers the plant to the viewer."}
|
||||
{"id": "vidprom_semantic_unique_gpt-5-mini-2025-08-07_52", "video_prompt": "Goku in his Super Saiyan 5 form stands on a cracked rocky plain under soft overcast light; a muscular adult male with long white-silver spiky hair falling past his shoulders, teal-green eyes, and faint red fur along his forearms and shoulders, wearing a torn orange gi with a blue undershirt and blue wristbands. A static medium full shot frames him chest-up as he is powering up, fists clenched at his sides, shoulders rising and falling with heavy breaths, hair lifting and the white-silver aura with subtle purple edges pulsing and crackling outward while small rocks and dust swirl and lift from the ground. Ambient audio begins with a low earth rumble and distant wind whoosh; as the aura intensifies a rising electrical crackle and bright synth sweep build in pitch, small stones clatter and a thin metallic ringing emerges. He inhales sharply, then exhales and speaks aloud: Goku (raw, strained, rising pitch): \"Haa...!\"—he tightens his grip and the aura spikes; immediately he releases a forceful shout that syncs to a sharp air snap and percussion hit: Goku (forceful, loud, high pitch): \"Kaaah!\"—the shout launches a brief sonic burst that bends nearby dust and leaves a brief shimmer in the air. After the shout the electrical crackle falls into a sustained metallic hum and a distant thunder roll as the energy settles, leaving faint floating embers and a quiet wind rustle while the static camera holds the charged pose for the final beat."}
|
||||
{"id": "vidprom_semantic_unique_gpt-5-mini-2025-08-07_60001", "video_prompt": "In a tight close-up, a tall clear glass filled with crushed ice, a pineapple wedge on the rim and a small paper umbrella is sitting on a weathered wooden bar top under soft late-afternoon light; a clear glass carafe is tilting above the rim and is pouring bright orange juice in a steady stream that is cascading into the glass, splashing against the ice and creating a swirling motion that lifts tiny bubbles toward the surface as the liquid level rises. As the carafe is pulled back, condensation beads are forming and slowly sliding down the glass while the umbrella trembles slightly and the pineapple wedge leans inward. The audio starts with the mid-range SFX of a liquid pour and crisp clinks of ice, layered under a low-volume tropical soundscape of distant ocean waves and soft steel-drum chords; when the juice hits the ice there is a short hollow ring of the glass and a faint fizz of bubbles, then a delicate tap as the carafe is set down and the ambient waves continue quietly in the background."}
|
||||
{"id": "vidprom_semantic_unique_gpt-5-mini-2025-08-07_60024", "video_prompt": "In a low, first-person view hovering just above the grainy ocean floor, soft blue-green shafts of light are filtering down through rippling surface patterns, revealing ridged sand, scattered broken shells, and jagged coral; fine silt is drifting upward and tiny bioluminescent plankton are pulsing like faint motes. A thin trail of disturbed sand is marking where something has been crawling, and a small crab is scuttling left to right across the frame, its legs kicking up micro-puffs of sand while a pale starfish clings to a nearby rock. Low, muffled water thrum fills the background, distant whale calls are resonant and slow, and measured regulator breaths are audible in steady intervals with occasional single bubbles popping as they rise. A companion voice, low and amused over faint radio static, says, \"You've been crawling around the ocean floor all day,\" timed with a slight ripple of light from above; after a brief pause and a tiny sift of sand, your voice, breathy and tired, replies, \"Yeah... can't tell if it's the cold or the tide,\" followed by a soft exhale and a larger bubble that drifts up through the plankton. Nearby, the crab's shell clicks softly against shell fragments and a faint metallic beep from a dive instrument punctuates the soundscape as the plankton pulse continues to drift."}
|
||||
{"id": "vidprom_semantic_unique_gpt-5-mini-2025-08-07_60025", "video_prompt": "Style: anime. A beautiful girl in a flowing white dress is standing in a sunlit forest glade, her long dark hair gently swaying as a soft breeze lifts the sheer skirt and lace hem; cel-shaded anime rendering with clean linework, soft pastel greens and warm brown trunks, and subtle rim lighting that makes the foliage and scattered wildflowers look slightly magical — the background feels fabulous with drifting pollen motes, pale blue bokeh orbs, and faint shafts of dappled sunlight filtering through leaves. She is standing center-frame with hands loosely clasped at her waist, large expressive eyes gazing upward and a small, wistful smile on her face, while nearby ferns glisten with tiny dewdrops. Ambient audio is layered: light leaves rustling, distant multi-voice birdsong that resolves into a single clear chirp, a faint stream trickle and a gentle wind whoosh; a delicate piano arpeggio with soft bell tones is playing quietly to lift the mood. She inhales audibly (soft breath SFX) and then speaks in a soft, reflective, slow voice: \"It's so quiet here...\" as she tilts her head and closes her eyes; after a brief pause she murmurs in a light, hopeful, slightly higher voice, \"I never thought I'd find this place,\" timed with a small smile and the dress whispering on the next breeze (fabric rustle SFX). A bright bird chirp answers in the soundscape just after her second line, and she gives a soft, airy laugh (gentle laugh SFX) as the piano bell lingers and the forest ambience continues."}
|
||||
{"id": "vidprom_semantic_unique_gpt-5-mini-2025-08-07_60080", "video_prompt": "A medium shot of a young female brown bear named Joy standing in a small sun-dappled forest clearing, her dense brown fur catching soft afternoon light and a simple red scarf tied around her neck; she is stepping forward with gentle, deliberate paws on a leaf-strewn floor (soft padding on dry leaves), head tilted and bright eyes focused as she listens, then she leans slightly forward and extends one paw in a friendly, open gesture (scarf rustle). Background ambience is warm forest sound—distant birdsong, a faint brook burble, and a light breeze moving leaves—synchronized so the footsteps occur as she enters and the paw stretch aligns with a quiet rustle. Joy speaks in a warm, low, friendly voice, clear and patient: \"Hi, I'm Joy. I can understand you, and I'm here to help,\" followed by a soft, amused chuckle; her mouth moves in time with the words, then she nods once and offers a small reassuring smile while lowering her paw as the forest ambience continues under a final gentle exhale. "}
|
||||
{"id": "vidprom_semantic_unique_gpt-5-mini-2025-08-07_60113", "video_prompt": "A wide view over rolling green Welsh hills is bathed in low soft sunlight from the west, the sun is shining through a thin veil of high clouds and skimming the rounded ridges; short grass and scattered heather are bending and rippling under a steady wind that is sweeping across the slopes, and small puffs of dust are lifting from thin paths as the breeze moves; a low grey drystone wall with lichen-streaked stones is tracing the contour of a slope, its rough texture catching side light while patches of gorse with small yellow flowers are trembling and shedding a few dry petals; thin white clouds are drifting across the pale blue sky and momentarily sliding cool shadows down the hillsides as the sun reappears; audio: a constant low whoosh of wind is filling the scene as the grasses bend, layered with the near rustle of stems and the soft creak and tumble of a loose stone shifting in the wall, and a clear distant skylark is trilling a sustained phrase overhead, its high note rising and fading while the wind continues; the frame is steady, showing only natural small motions—grass leaning, flowers quivering, light moving across the slopes—conveying a calm, windy sunlit moment on the Welsh hills."}
|
||||
{"id": "vidprom_semantic_unique_gpt-5-mini-2025-08-07_60133", "video_prompt": "In a static medium-wide shot from slightly behind and to the boy's right, a boy of about 12 with short dark hair and a navy hoodie is sitting cross-legged on a wool blanket on a low grassy knoll, watching the night sky as the Milky Way is arching overhead as a pale white band with a faint purple haze that is slowly drifting westward; soft starlight and a thin crescent moon are casting a cool, diffuse glow across his face and the textured knit of his hoodie. He is tilting his head back and tracking the shifting band with steady eyes, fingers tapping once on his knee, then shifting his weight so the blanket rustles. Ambient night sounds fill the scene: a gentle wind whispering through tall grass, steady cricket chirps, a distant owl hoot, and the soft rustle of fabric; his breath is audible as a small inhale. Boy (soft, awed, low voice): \"It's... actually moving.\" He exhales, smiles slightly, then leans forward and points with one hand toward a brighter patch of stars. Boy (quiet, slow, wonder): \"Look at that—like a river in the sky.\" A subtle, warm ambient pad underlies the natural soundscape, supporting the moment without overpowering the night ambience as the Milky Way continues its slow motion and the shot holds the quiet tableau."}
|
||||
{"id": "vidprom_semantic_unique_gpt-5-mini-2025-08-07_60194", "video_prompt": "A tight close-up on a young person in their early 20s with short dark hair and subtle freckles, eyes wide and open to a cool wind that is ruffling short hair and moving a loose strand across the cheek; soft side light from a low sun casts gentle highlights on the skin and a faint rim light on the hair, while the out-of-focus background shows a pale grey sky and blurred treetops. Their gaze is slightly upward, pupils slowly widening as air moves across the face, eyelashes trembling and then blinking; they inhale quietly, shoulders shifting in a small, visible breath, and exhale as the wind lifts finer hairs. They speak twice in a close, intimate delivery synchronized with the breaths: They (soft, breathy, low-pitched) says, \"It's okay...\" while looking upward and letting a slow blink hold, then after a brief pause They (quieter, steady, low-pitched) says, \"I'm here,\" as the lips part and the wind pushes the lashes. Audio layers: foreground wind whooshes that vary with the hair movement, small crisp rustles of distant leaves, a subtle distant urban hum under the wind, close-mic'd inhalation and exhalation, and a single sustaining cello tone that rises gently under the second line and fades with the wind; all sound is timed to the visible breaths, hair movement, and spoken lines. Textures and details are visible in the five-second moment: moist eyes reflecting the sky, fine skin pores, soft sheen on the lips, and individual hair fibers moving against the cheek."}
|
||||
{"id": "vidprom_semantic_unique_gpt-5-mini-2025-08-07_60198", "video_prompt": "Rendered in ultra-detailed 8k, a LEGO Super Mario minifigure (red hat with an M, blue overalls, white gloves, brown mustache, printed cheerful face) is standing on a glossy green baseplate with visible studs; soft overhead studio light and a cool rim light are creating small specular highlights on the molded plastic. He is raising his right arm and pressing a small red action button on the baseplate; precise plastic clicks and a short electronic beep occur as he moves. As he presses, a circular portal is opening behind him — a translucent ring of shifting teal and purple energy with swirling particle motes and a subtle grid-like shimmer that casts colored reflections across Mario's face; a deep harmonic drone is rising and a whoosh of wind-like synths layers with a brief chiptune arpeggio. The single camera is zooming in slowly from a medium shot toward an extreme close-up on Mario's face and the rim of the portal, tightening on the reflection of the swirling colors in his printed eyes while a quiet mechanical whirr from the zoom accompanies the sound. Mario (bright, energetic, mid-pitched, quick) says as he presses the button and leans forward, \"Let's-a go!\" — his voice is clear and playful and matches the moment the first light blooms. The portal answers with a low, resonant, echoing whisper (slow, hollow), \"Come...\" timed as tendrils of energy unfurl, and Mario blinks and inhales sharply, then exclaims (surprised, higher, short), \"Whoa!\" as the camera tightens on his expression; throughout, small plastic clacks track his movements, the portal rings with sporadic crystalline chimes, and the underlying electronic drone settles into a lower, distant hum that suggests another world beyond."}
|
||||
{"id": "vidprom_semantic_unique_gpt-5-mini-2025-08-07_60209", "video_prompt": "In a quiet medium shot, a woman in her late 20s with chestnut hair tied in a loose bun is sitting alone on a flat rock at the edge of a calm, glassy lake, wearing a light-gray sweater, dark jeans, and scuffed brown boots; soft late-afternoon light is falling across her face and the water while the surrounding trees are full green and gently rustling. She is sitting cross-legged and leaning slightly forward, shoulders relaxed, eyes fixed on her reflection, then reaches a hand down and is lightly tracing the water surface with her fingertips, sending small concentric ripples as she breathes slowly and lets out a soft sigh. Ambient audio layers are detailed and timed to motion: steady, gentle lapping of water against stone, leaves rustling in a light breeze, distant single bird calls and a low insect hum, all underscored by a sparse piano motif—slow, single notes at low volume—that swells softly as she moves her hand; when her fingertip touches the water a delicate splash and the faint rustle of her sweater are audible. In a low, steady voice she says, \"I don't know who I'll be next,\" pausing to look at the ripples and glance up toward the treeline (a short, thoughtful silence follows), then in a softer, accepting tone she adds, \"But maybe that's okay,\" as she tilts her head, exhales audibly, and allows a small, contemplative smile to form while her gaze returns to the lake."}
|
||||
{"id": "vidprom_semantic_unique_gpt-5-mini-2025-08-07_60225", "video_prompt": "In a medium close-up, a skeleton is seated on a low white cloud in a bright heaven, wearing a simple white linen sash over one shoulder; its bone surfaces are smooth with faint cracks and the jaw is articulated so the expression reads relaxed. Soft golden backlight and diffuse white cloud light wash the scene while distant pearly spires and floating islands sit out of focus behind. The skeleton is lifting a polished silver fork in its right hand and a carving knife in its left, slicing a medium-rare steak on a porcelain plate balanced on its lap; the steak shows a seared brown crust and a warm pink center that gives a quiet sizzle as the knife passes. As it brings a forkful to its jaw, a dry, gentle clack of bone is audible, followed by a muted, contented chewing sound; it speaks in a low, amused voice, slow and warm, \"Well, this is unexpected,\" then pauses, glancing down at the plate and answers itself in a softer, wry tone, deliberate and mid-pitch, \"Heaven could use better menus,\" while a thin harp arpeggio and a distant choral pad swell beneath, soft wind through clouds and faint bell chimes punctuating the air; knife-on-plate clink, fork lift, and the final swallow are audible as the skeleton settles back slightly, a small puff of cloud compressing under its weight."}
|
||||
{"id": "vidprom_semantic_unique_gpt-5-mini-2025-08-07_60254", "video_prompt": "A static medium close-up of a cottage window at night, pale full moonlight is washing a cool rectangle across the weathered wooden sill and slightly wavy glass; the right-hand casement is slowly swinging outward on aged iron hinges, the chipped white paint and rough grain are visible in the moon glow. As the pane is swinging open, a low creak from the hinges is audible, then a soft scrape as the old latch releases; a thin linen curtain is fluttering inward and its edge is brushing the sill with a quiet rustle. Outside, steady cricket chirps are underscoring the moment while a distant owl hoots once and a light breeze is whispering through nearby leaves so branches are sighing faintly. The widening opening is letting more moonlight spill into the dark interior, highlighting dust motes drifting in slow arcs and casting a pale band that is sliding across the floor."}
|
||||
{"id": "vidprom_semantic_unique_gpt-5-mini-2025-08-07_60265", "video_prompt": "In an extreme close-up, a shallow pile of whole almonds and hazelnuts rests on a matte dark slate surface under soft overhead light that picks out subtle ridges on the almond skins and the rounded texture of hazelnut shells; a thick stream of melted dark chocolate is pouring in from above and is splashing down onto the pile, coating some nuts and sending tiny droplets outward. As the chocolate is striking the nuts, a few almonds and hazelnuts are nudged and spin slightly while a small piece of hazelnut shell is flicked aside; droplets arc and hang briefly in slow motion, catching highlights on their glossy surfaces, then fall and merge into a spreading pool around the nuts. Visual emphasis is on the contrast between the glossy chocolate and the matte nuts, with one hazelnut showing a cracked interior as chocolate runs over it. Audio begins with the deep, viscous pour of chocolate—a low glug and a soft, sticky slap on contact—immediately joined by crisp, dry clacks as nuts bump one another and a faint high-frequency tinkle as tiny droplets hit the slate; a muted kitchen ambience (distant HVAC hum and soft background murmur) sits underneath, then the sound settles into a gentle wet spread and quiet drips as chocolate spreads around the almonds and hazelnuts."}
|
||||
{"id": "vidprom_semantic_unique_gpt-5-mini-2025-08-07_60278", "video_prompt": "Dong Yuhui is standing in a vast open plain under a pale sky, soft late-afternoon light falling across the scene; he is a man in his early 30s with short black hair, the breeze is blowing through his hair and ruffling the edges of a light gray jacket over a white shirt, and he is holding an open hardcover book in his left hand with the pages slightly fluttering. His face is turned toward the horizon, eyes firm and full of hope, the corners of his mouth slightly raised in a positive, energetic smile; his posture is upright, chest lifted and shoulders back, his whole body language is full of vitality and vigor, conveying clear confidence and optimism. While the wind makes a low whoosh through the grasses and the book emits a soft paper rustle, he breathes in, glances down at the open page, then looks up and speaks in a steady, warm, confident voice, \"We can do this,\" the words timed with a small, assured nod. Underneath, a gentle single-note piano chord swells as distant bird calls punctuate the air, the wind continues to whisper, and the final sound is the quiet flutter of pages as his smile holds, leaving a calm, positive, upward feeling."}
|
||||
{"id": "vidprom_semantic_unique_gpt-5-mini-2025-08-07_60291", "video_prompt": "A medium close-up of a man in his late 50s with short graying hair and a neat trimmed beard, seated behind a low glass console in a futuristic presidential chamber that blends dark wood, brushed steel and frosted glass; he wears a tailored dark gray suit with a high-collar shirt and a small stylized tricolor lapel pin, his expression calm and purposeful as translucent holographic maps and data widgets float a foot above the console. He reaches out and taps a hovering map, which ripples into sharper focus while a thin blue route highlights across Eurasia; his fingers move with deliberate, practiced gestures and his eyes track the route as if reading several layers of data at once. Ambient sound is a low, steady hum of climate systems and distant city traffic through thick glass, punctuated by soft glassy chimes on each tap and a brief electronic confirmation beep when he activates a layer; a sparse, subdued string motif plays quietly under the scene, rising slightly when the map refocuses. President (voice: measured, low, paced): \"Activate strategic overlay, scale to national,\" he says, then pauses, glances briefly to the side as if checking a monitor, fingers hovering above the interface. President (voice: softer, deliberate): \"Keep civilian channels open — alert level stable,\" he adds, and presses his palm once, sending a succinct confirming tone as the hologram contracts; his face tightens for a beat, then relaxes as the room returns to the steady ambient hum."}
|
||||
{"id": "vidprom_semantic_unique_gpt-5-mini-2025-08-07_60317", "video_prompt": "A medium close-up static shot of an androgynous angelic being is hovering a few feet above a smooth pale floor, wearing a flowing white robe with thin silver embroidery and long silver-white hair falling over the shoulders, face composed and gently focused. Soft cool backlight and a warm rim light are outlining the figure as large luminescent wings are slowly unfurling behind them, each feather translucent with an inner soft glow that pulses in pale gold and cool blue while tiny motes of light are drifting off the wing tips and catching on the silk texture of the robe. As they are extending both hands forward, the aura around them is shimmering in slow, concentric waves and their robe is fluttering slightly from a light upward lift; they are tilting their head and allowing a quiet, serene smile to form. A sustained, gentle choral pad is filling the air underneath the scene, with a single bell-like glissando punctuating the moment the wings open; soft rustle of fabric and whisper of feathers are synchronized to the wing motion, a faint whoosh of displaced air accompanies the hover, and subtle harmonic overtones are rising and falling with the aura's pulse. (voice: calm, warm, mezzo) \"I am here,\" they say, voice steady and low as their palms open, the words timed to the wings' full spread; a brief pause lets the motes scatter. (voice: soft, distant, reverent) Then they adds, voice trailing as they incline their head and close their eyes briefly, \"Stay near the light,\" while the glow around them is gently pulsing one last time."}
|
||||
{"id": "vidprom_semantic_unique_gpt-5-mini-2025-08-07_65", "video_prompt": "A young woman in her early 20s is standing in a medium full shot on a narrow stone alley lit by paper lanterns, wearing a fitted knee-length red silk qipao with pale pink peony embroidery and smooth white stockings; soft warm lantern light grazes the silk while cool evening blue fills the alleyside shadows, the stone underfoot showing a faint sheen as if damp. She has long black hair gathered into a loose chignon with a simple silver hairpin, subtle makeup, and a calm, attentive expression as she shifts her weight onto one foot and smooths the fabric at her hip, then reaches out briefly to brush a lantern's fringe with two fingers while her skirt whispers against her thigh. Ambient alley sound settles beneath—low murmur of distant conversation, a far-off bicycle bell, a small trickle of water from a nearby drain—while a single-note erhu phrase plays softly and periodically, matching her gentle movements; each step she takes produces a soft footstep on wet stone and a quiet rustle of silk. She breathes out, then speaks in a soft, warm voice, \"The lanterns look quiet tonight,\" as she glances down and then up toward the lights, a small, thoughtful smile forming; fabric rustle and a light creak from the lantern sway punctuate the moment, and the erhu holds a final quiet phrase as she tilts her head and the scene lingers on her composed, serene posture."}
|
||||
{"id": "vidprom_semantic_unique_gpt-5-mini-2025-08-07_70", "video_prompt": "A medium close-up of Lucas, early 30s, clean-shaven with short dark hair slightly tousled, wearing a navy blazer over a white shirt, standing on a small stage in a dim conference hall under a soft spotlight with a faint technical schematic projected on a screen behind him; he is leaning slightly forward, eyes bright, smiling, right hand lifting in an open, inviting gesture while his left holds a handheld microphone. Lucas (eager, breathy, mid-tempo voice): \"This is it, my friends. Humanity's next step.\" As he finishes the line he steps a half pace forward and lifts both hands briefly, adding in a quick, confident tone, \"We're ready to begin,\" (confident, slightly faster) while a low audience murmur swells into a single short cheer; ambient room sound includes a soft HVAC hum, distant chair scrape, and a subtle microphone rustle when he moves the mic. The stage light casts soft shadows across his jaw and the projector's faint flicker glints on the nearest rows; his brows lift and a slight smile tightens at the corners of his mouth, the fabric of his jacket shifting as he breathes and the crowd reacts, all within a compact, energetic five-second moment."}
|
||||
{"id": "vidprom_semantic_unique_gpt-5-mini-2025-08-07_74", "video_prompt": "A medium wide shot of a low, pebbled shoreline in soft late-afternoon light, a boy about eight with short brown hair and a faint smudge on his cheek is walking barefoot toward the water wearing a damp navy windbreaker and tan shorts, taking small, careful steps across wet stones while his jacket sleeves brush his arms. Gentle lap of small waves provides the ambient sound, with distant gull calls and a light breeze whispering through nearby reeds; as he draws within two meters the gravel crunches under his feet, a soft fabric rustle from his jacket is audible. He slows, eyes on the moving water, inhales audibly, and in a quiet, hesitant voice says, \"Okay... here goes,\" then leans forward and steps so his toes skim the shallow edge, producing a soft splash and a brief spray; the water makes a delicate fizzing sound against his foot and he lets out a small relieved breath while the wave withdraws and pebbles settle back into place."}
|
||||
{"id": "vidprom_semantic_unique_gpt-5-mini-2025-08-07_76", "video_prompt": "A stationary overhead camera looks straight down through clear, shallow blue water as a large adult whale shark is gliding slowly from left to right across the frame, its broad head, pale gray-blue skin patterned with white spots and faint stripes, and a tall dorsal fin visible beneath a gently rippling surface; soft sunlight is casting moving caustic bands across its back and the sandy seafloor below, the shark's rough, ridged skin showing subtle barnacles and a few old scars. As it swims, the shark is opening its wide mouth slightly to filter-feed while its tail is undulating in smooth, powerful strokes, a dorsal fin brushing the surface and sending tiny concentric ripples; a small group of pilot fish is trailing close behind, then a nearby cluster of baitfish is scattering in a quick burst, darting away in sharp, synchronized motions. Ambient audio begins with a low, muffled ocean hum and a distant, slow whale-like call underlining the scene; as the shark passes a soft whoosh of displaced water and a deep, low rumble follow each tail flick, faint bubble pings are audible near its mouth, and the baitfish scattering produces brief, high-frequency splashes and quick metallic plinks. The overall mood is calm and observant, the overhead view holding steady as the whale shark continues gliding out of frame."}
|
||||
{"id": "vidprom_semantic_unique_gpt-5-mini-2025-08-07_86", "video_prompt": "A medium close-up of a young man in his mid-20s, based directly on image:sketch (1).jpeg as visual reference, standing slightly angled toward camera with his shoulders relaxed; his dark, slightly long hair is moving as a steady wind is sweeping across the scene, ruffling strands and lifting the collar of his loose grey shirt. A soft blue-white glow is surrounding him like a faint halo, the glow is pulsing gently and casting a cool rim light on the edges of his face and hair while the rest of the background remains a muted charcoal sketch texture; small ink-like particles are drifting through the air and catching the light as they float. He is breathing out slowly, exhaling as a gust of wind pushes his hair back, and his expression is calm, eyes narrowing slightly as if listening. Soundscape: a clear, close wind whoosh is present and is rising and falling with each gust, soft rustling of fabric and hair on each breath, and a low, steady electrical hum is synchronizing with the glow pulses; beneath that, a faint sketching scratch like pencil on paper is barely audible to tie to the reference image. Dialogue synced to actions: He (voice: low, steady, medium pace) says, \"I can feel it,\" as he exhales and lets his chin lift; an inner voice (voice: soft, breathy, slow) replies, \"It's starting,\" as the glow brightens for one pulse; he (voice: low, steady) answers again with a softer tone, \"Then stay with it,\" as his hair settles slightly and his shoulders shift; the inner voice (voice: airy, distant) whispers, \"I am,\" timed with the final, small gust that makes the particles drift away. The overall color palette is cool greys and blue-white light, textures are drawn-paper and soft fabric, and all motion is continuous and subtle to fit within a brief five-second moment."}
|
||||
{"id": "vidprom_semantic_unique_gpt-5-mini-2025-08-07_93", "video_prompt": "In a wide shot, a small round bunny with soft white fur and faint gray patches is sitting near the crest of a gently sloping hill blanketed in short green grass and scattered wildflowers—daisies, buttercups, and small bluebells—bathed in soft midday sunlight; the bunny twitches its nose and lifts slightly, then springs forward in a quick, joyful bound, ears tilting back and hind legs tucking under as it clears a patch of flowers, petals fluttering down and a few stems bending under its passage, then it lands lightly with forefeet touching first and hind feet following, sending a muted thump and a soft rustle through the grass. Ambient sound begins with a light breeze through the grass and distant birdsong, joined by a steady, close bee buzz around the blossoms; as the bunny pushes off there is a small puff of displaced air and the faint crunch of stems, and on landing the bunny lets out a short, bright chirp. Bunny (soft, high-pitched, quick): \"peep,\" synchronized with the landing and a brief head tilt; it then sniffs a daisy, nose wrinkling and whiskers brushing petals, ears flicking, while the breeze and bees continue softly in the background and a few petals settle back onto the hill."}
|
||||
{"id": "vidprom_semantic_unique_gpt-5-mini-2025-08-07_60343", "video_prompt": "Wide shot of a small village under late afternoon light: clusters of low stone cottages with red tile roofs, a low church steeple, and a narrow cobbled lane between them; warm, soft light casts long, thin shadows and brings out the rough texture of stone and the matte clay tiles. Several groups of birds are flying across the pale sky, each flock is shifting shape as birds are soaring, banking, wheeling, splitting off, then rejoining—some individuals are gliding on outstretched wings while others are beating rapidly to gain altitude; their moving shadows skim across rooftops and the lane below as a group arcs together over the steeple. Soundscape: close, soft flapping of wings layered with overlapping bird calls—higher, quick chirps interspersed with occasional low caws—while a single distant church bell tolls slowly once, a light wind is rustling through poplar leaves, a loose wooden shutter creaks and a clay chimney pot clicks as the breeze passes, all synchronized so the wingbeats and calls rise as the flocks bank and briefly swell, then fall away as they split and move out of frame."}
|
||||
{"id": "vidprom_semantic_unique_gpt-5-mini-2025-08-07_60350", "video_prompt": "In a medium close-up, a man in his late 20s with short dark hair and light stubble is cupping a 3-week-old black and white tuxedo kitten with blue eyes in both hands, wearing a light gray cotton shirt; soft window light is falling across their faces and a warm indoor tone is bathing the scene. He is holding the kitten gently, palms cradling its tiny body as he is stroking the soft downy fur with his thumbs; the kitten is calm and relaxed, blinking slowly, paws tucked against his palms and its tiny pink nose twitching. The man smiles and speaks in a low, warm voice, slow pace, “Hey little one…” (he leans in slightly, eyes soft, fabric rustle audible), then the kitten replies with a soft, high mew (quiet, brief) while blinking and beginning a steady, quiet purr that grows slightly louder as it settles against his hands. The man responds in a tender hush, gentle pace, “You’re so small, aren’t you?” as he exhales and brushes his thumb along its back; the kitten emits another faint mew and nuzzles his thumb, purring more audibly. Ambient audio layers are intimate and close: a muted distant street hum through the window, a faint clock tick, soft breathing from the man, light fabric rustle when he shifts, the kitten’s small mews and continuous low purr present and clear, and a subtle room reverb to convey a small, quiet indoor space."}
|
||||
{"id": "vidprom_semantic_unique_gpt-5-mini-2025-08-07_60360", "video_prompt": "Style: Tim Burton Style Characters. In a dim, low-lit room a lanky, slightly elongated Santa is standing beside a tall, narrow window, peering at a smartphone held in his gloved hand; he has a narrow pale face, wide dark eyes, a scraggly white beard, and a worn red coat with thin fur trim and long sleeves that brush his knuckles. Soft moonlight is filtering through the window, casting thin shadows across his coat while a small warm desk lamp on the sill gives a muted rim light; a cinematic lens with shallow depth of field is rendering the distant streetlights outside as soft bokeh and subtle film grain. The smartphone screen is glowing with a map full of clustered red dots and tiny car icons showing a traffic jam; he is zooming and tapping the screen, brow furrowing as the map shifts. Room audio is quiet: a faint, steady traffic rumble and occasional car horn from the street below, a soft creak as he shifts his weight, and a delicate notification chime from the phone. In a low, gravelly, slow voice he mutters, \"Forty-two minutes? That's too long,\" then he is tapping to try an alternate route as a neutral, clipped phone assistant voice says, \"Delay ahead: 42 minutes, suggested detour adds ten minutes,\" synchronized with the map redrawing under his thumb; he glances up through the window at the stalled tail lights, exhales, and in a short, resigned tone replies, \"All right, take the detour,\" then tucks the phone into his coat as the distant traffic hum persists and the lamp casts a soft, low shadow across his face."}
|
||||
{"id": "vidprom_semantic_unique_gpt-5-mini-2025-08-07_60390", "video_prompt": "A wide static view of a coastal landscape on a sunny day: the sun is high in the pale sky, casting soft overhead light and subtle shadows across smooth wet sand and scattered pale pebbles while small waves are lapping at the shore and thin white foam is retreating with a gentle hiss as sunlight glints on the ripple tops. As a light breeze is bending the dune grass and a few palm fronds, several seagulls are circling low and calling with short sharp cries, and a distant sailboat is drifting near the horizon with its canvas lightly billowing. The soundscape matches the motion—soft rhythmic surf washing onto sand, the thin hiss of foam receding, wind whispering through the grass, intermittent gull calls, and an occasional soft metallic clink from the sailboat rigging, all underscored by a faint, slow acoustic guitar picking a quiet two-bar motif to add calm warmth to the scene."}
|
||||
{"id": "vidprom_semantic_unique_gpt-5-mini-2025-08-07_60404", "video_prompt": "A medium close-up shows Eveline, a young sorceress in her early 20s with waist-length dark brown hair braided loosely over one shoulder, wearing a moss-green cloak over a simple linen dress and scuffed leather boots, kneeling on soft emerald moss in a serene sun-dappled forest; soft overhead light filters through tall oaks, casting moving leaf shadows and the scene is textured with damp bark, tiny wildflowers, and dewy spiderwebs. Ambient forest sounds—distant birdsong, a gentle breeze rustling leaves, a nearby creek's quiet trickle—form a calm background; her soft footsteps on moss and the quiet inhale of her breath punctuate the space as she is experimenting with magic, tracing a small sigil in the air with a trembling finger while murmuring an incantation, \"Focus... steady,\" (soft, hesitant, low) and tiny golden motes are gathering at her fingertip with a faint bell-like chime and gentle electrical crackle. She is tilting her head, squinting in concentration, then is pushing her palm forward and a warm pale orb of light is lifting from her hand, spinning slowly as leaves and dust begin orbiting it; the orb is emitting a subtle harmonic drone and delicate tinkling SFX as it is stabilizing. Eveline is exhaling sharply, \"It's... it's working,\" (breathless, surprised, slightly high) and she is leaning closer, eyes widening and mouth parting in visible awe while small vines at her feet are curling toward the light. She is straightening with a small, stunned smile, whispering, \"I can do this,\" (quiet, resolute, calm) as the forest ambience swells—a soft wind gust, a brief chorus of birds—and the orb is pulsing once with a warm luminous beat that is bathing her face in soft gold, capturing the brief, clear moment of realization and possibility."}
|
||||
{"id": "vidprom_semantic_unique_gpt-5-mini-2025-08-07_60416", "video_prompt": "Style: cartoon. In a static medium-wide shot, a cartoon girl of about 10 sits on a smooth gray stone ledge at the base of a narrow waterfall, legs dangling and hands resting on the rock; she wears a teal hoodie and denim shorts, brown shoulder-length hair with a small side braid, and a soft, contented expression as she looks out at the view of a tree-lined valley beyond. Soft midday light filters through green leaves, the waterfall is drawn with clear blue-green water and white froth, and a fine mist makes tiny droplets bead on her knees; a light breeze moves her braid and the hem of her hoodie. The water is continuously audible as a steady, low roar; higher, crisp splashes hit the stones and a few bright bird calls and a faint rustle of leaves sit beneath the falls. She shifts her weight, tucks her braid behind her ear, blinks, and exhales slowly as the mist lands on her face (subtle wet SFX). Girl (soft, reflective, slow): \"It's so peaceful here...\" (her voice is intimate, slightly breathy; waterfall audio ducks gently while she speaks). She smiles, presses her palms to the stone, and says with a small laugh, Girl (light, quick): \"Maybe I should come back with a sketchbook.\" After her last word the waterfall volume returns to full, droplets patter, and the scene holds with distant birds and steady water ambience."}
|
||||
{"id": "vidprom_semantic_unique_gpt-5-mini-2025-08-07_60454", "video_prompt": "In a static medium-wide shot, a woman in her late 20s with wind-tousled dark hair is sitting on a smooth driftwood log at the sandy shore, wearing a light gray knit sweater and worn jeans, facing the sea as soft late-afternoon light washes the scene and low sun glances off the wet sand. Small waves are lapping and pulling at the shoreline, seafoam is pulsing around scattered pebbles, and a cool breeze is ruffling her hair and sweater; distant gull calls punctuate the air and a low foghorn is sounding far off. She is watching the horizon, fingers lightly tapping the log, then draws a slow breath and says in a quiet, steady voice, \"Calmer today, huh?\" She glances down at a phone in her lap; an off-screen voice, warm and slightly amused, replies quickly, \"Yeah — you found the quiet spot.\" She lets out a small smile, tucks hair behind her ear, answers softly, \"I needed this,\" and turns her gaze back to the sea while the rhythmic hiss of waves and gentle wind continue under the brief exchange, with occasional seabird calls and a faint distant dog bark adding texture to the seaside ambience."}
|
||||
{"id": "vidprom_semantic_unique_gpt-5-mini-2025-08-07_60489", "video_prompt": "In a medium close-up, Alexander Lukashenko, a white man in his early 70s with short gray hair and a clean-shaven face, wearing a dark navy suit and a striped tie, sits at a low podium against an out-of-focus backdrop showing a red-and-green flag; soft overhead light and a subtle rim light outline his shoulders as the scene focuses on his face. He is smiling, then leans forward slightly, then tilts his head back and throws it back as he bursts into a short, throaty laugh—his shoulders shaking and eyes crinkling—while one hand comes up briefly to his mouth and then drops to the podium. The audio layer presents a close, gravelly laugh in the foreground, layered with light applause and murmured reactions from an off-screen audience, the faint rustle of papers, and a soft microphone rustle; as the laugh begins he utters one dry, measured line in a low, even tone, \"Well, that's unexpected,\" (low, dry, slow), then follows with a quick, breathy chuckle (deep, warm) that overlaps the room noise. The camera remains steady in the medium close-up, capturing the sequence of smile, voiced remark, and sudden laugh while ambient sounds — soft clapping, hushed voices, a single camera shutter click — sit behind the laugh and then settle as the laughter fades."}
|
||||
{"id": "vidprom_semantic_unique_gpt-5-mini-2025-08-07_60498", "video_prompt": "A static close-up frames a Spider-Man action figure standing on a worn wooden shelf, painted red and blue suit with glossy plastic highlights and visible articulated joints; soft overhead light casts mild reflections across the molded webbing and a shallow depth of field blurs the background books. He is slowly tilting his head to the left as his white eye lenses curve upward into a clear smiling expression, then he raises his right hand in a short, friendly wave while a crisp plastic joint click and a faint squeak register with each motion. Room ambience carries a distant city hum and soft indoor air movement; as his lenses tighten into the smile a brief, gentle synth twinkle accents the moment. Spider-Man (playful, slightly high-pitched, quick) says, \"Hey—ready?\" as he lifts his hand, the voice timed to the click of the wrist joint; he pauses, leans forward a hair, and the painted smile seems to deepen with a tiny plastic rasp. Spider-Man (warm, amused, medium pace) follows, \"You always are, right?\" while he tilts his head back and his shoulder rotates with a soft mechanical click. Spider-Man (confident, brisk, low) finishes, \"Okay—on three,\" as he nods once and the light catches a small scuff on the painted chest; a short, bright chime punctuates the final grin and the ambient city hum continues under the closing moment."}
|
||||
{"id": "vidprom_semantic_unique_gpt-5-mini-2025-08-07_60508", "video_prompt": "A matte black Porsche 911 is cruising along a narrow, leaf-strewn forest road in the evening, its low headlights cutting through dim blue-gray dusk while long cool shadows from pine and birch stretch across the pavement; soft, warm red taillights smear over wet leaves as the car moves, and small sprays of leaves lift and scatter behind the rear tires. The engine is emitting a low, measured purr that rises briefly as the driver downshifts while negotiating a gentle curve, tires softly crunching on damp leaves and loose gravel beneath the chassis; occasional low branches brush the hood and send a faint metallic whisper along the bodywork. Ambient sound is a quiet forest layer—wind rustling needles and leaves in the canopy, a distant bird call, and the echo of the engine bouncing between tree trunks—then the subtle click of the turn signal and the whisper of airflow as the Porsche arcs past, remaining alone in the calm evening woods."}
|
||||
{"id": "vidprom_semantic_unique_gpt-5-mini-2025-08-07_60528", "video_prompt": "A quaint village nestled among rolling hills and surrounded by peaceful trees is bathed in a sky of orange and pink as the sun begins to set, soft late-afternoon light warming stone cottages and small gardens while gentle rustling of leaves and the sweet melodies of birds fill the air. After two seconds, the screen slowly zooms in, then over the next three seconds the camera slowly zooms in and pans, offering a picturesque view of the clustered homes before gracefully pointing toward a nearby pond that is shimmering in the warm sunlight; the pond's surface mirrors the colored sky and surrounding trees with glinting highlights moving across the water. As the focus narrows on the pond, three vibrant orange fishes are swimming in graceful arcs just below the surface, their scales glistening as sunlight catches them while they glide in small, coordinated loops and send soft ripples outward. Audio is layered to match the motion: continuous leaf rustle and clear bird song in the foreground, then gentle water lapping and faint, synchronized splashes as each fish breaks the surface of a ripple, keeping the overall soundscape calm and harmonious with the village remaining quiet in the background."}
|
||||
{"id": "vidprom_semantic_unique_gpt-5-mini-2025-08-07_60532", "video_prompt": "A medium close-up of a young woman in her late 20s with short black hair tucked behind her ear, wearing a light gray blouse, sitting at a small wooden table at a street-side food stall as soft late-afternoon light falls across a glossy ceramic bowl of mie ayam—yellow noodles piled with shredded braised chicken, chopped scallions and a few green vegetables, dark sauce pooling at the bottom; she is lifting a tangle of noodles with wooden chopsticks, twirling them once, then slurping them into her mouth while her free hand is scooping broth and chicken with a metal spoon, her eyes closing briefly and a small smile forming. Ambient street audio is present—distant traffic hum, a motorbike idling, low vendor chatter and occasional clink of dishes—while close SFX highlight wooden chopsticks tapping porcelain, a wet noodle slurp, the soft clink of spoon on bowl and a quiet breath; a gentle instrumental with soft percussive tones plays under the ambient mix. As she sets the chopsticks down she says in a warm, measured voice, \"Looks comforting,\" pauses and glances around, then in a soft, content voice adds, \"Just what I needed,\" and reaches for another bite."}
|
||||
{"id": "vidprom_semantic_unique_gpt-5-mini-2025-08-07_60538", "video_prompt": "A static medium-wide shot frames a low, moss-covered rock ledge and the cool pool below, soft afternoon light filtering through a leafy canopy and casting dappled patches across dark green water; the rock is slick with thin rivulets and small, pale leaves cluster at the edge. A small, smooth pebble is teetering on the lip, then tumbles and plunges into the cool pool below, breaking the mirror surface and sending concentric ripples outward while tiny droplets arc upward and scatter. Under the surface a brief ring of bubbles is rising and dissolving, and as the ripple reaches floating leaves they bob and shift. Audio: an immediate, sharp splash on impact, followed by spreading, gentle slaps of water against stone, a soft underwater gurgle as bubbles release, faint steady trickle from the rock face, distant birds calling and a low insect hum; after the initial splash the water calms and small droplets patter on leaves while the echo of the impact fades."}
|
||||
{"id": "vidprom_semantic_unique_gpt-5-mini-2025-08-07_60546", "video_prompt": "A happy child of about eight years old with short curly hair and subtle freckles is sitting cross-legged on a small red picnic blanket in a sunlit backyard, wearing a striped t-shirt and denim shorts, holding a small soprano ukulele with a light wood grain and glossy finish; warm late-afternoon light is casting soft shadows and a few leaves are drifting in a light breeze. The child is smiling and is strumming a bright, bouncy chord progression with the right hand while the left hand is forming a simple C-to-G chord change, the ukulele strings ringing with clear plucked tones; the child is gently swaying and bobbing their head as they sing one short, cheerful line in a light, singing voice, Kid (singing, bright, slightly high): \"Sunshine on my day, play along, hey!\"—they finish the phrase with a quick, playful flourish across the strings, then laugh and look up, Kid (delighted, breathy): \"Again!\" as they tap the uke body with their thumb and launch into another upbeat strum. Audio includes close, intimate ukulele sound with crisp string attack and soft resonance, the child's voice layered slightly forward, faint backyard ambience—distant birdsong, a soft breeze through leaves, a muted lawnmower far off—and the small SFX of fingers on wood and a quick giggle; everything is synchronized so the vocal lines match the visible strums and the final tap coincides with the child's delighted exclamation."}
|
||||
{"id": "vidprom_semantic_unique_gpt-5-mini-2025-08-07_60571", "video_prompt": "In a wide, low-angle shot from the highway shoulder, three super-fast sports cars are racing side-by-side along a wet night highway, a red low-slung coupe with a white racing stripe on the left is surging forward, a matte black aggressive supercar in the center is pulling slightly ahead with sharp LED accents cutting through light mist, and a metallic blue targa on the right is darting forward; glossy body panels catch and reflect neon from the big city glowing in the distance, soft sodium streetlamps pool on the slick asphalt, and heat shimmer ripples above each exhaust. As they accelerate, engine roars dominate the mix— the red coupe emits a high, razor-edge rev, the black car answers with a deep, throaty growl, and the blue car adds a fast turbo whistle—interleaved with rapid upshifts and short throttle blips; tires hiss and briefly squeal during tight lane corrections, small pebbles crack against wheel wells, and a sharp whoosh of wind whistles past side mirrors. Ahead, the distant city skyline of clustered towers and billboard neon provides a low, constant urban hum with faint car horns and a muted subway rumble beneath, while the Doppler shift bends each engine pitch as the cars edge forward and then tuck in, motion blur streaking headlights and building lights to emphasize raw speed for the five-second burst."}
|
||||
{"id": "vidprom_semantic_unique_gpt-5-mini-2025-08-07_60572", "video_prompt": "Style: hyper realism 4k. A static tight close-up of a single candle sitting in an aged brass candlestick on a dark wooden surface, the creamy white wax is melting into a glossy pool that is slowly running down the stem while the blackened wick is holding a steady amber flame; 4k detail reveals the fine char texture of the wick, tiny beads of molten wax, and subtle fingerprints and patina on the brass. Soft warm light from the flame is casting gentle reflections on the candlestick and subtle, narrow shadows on the wood as a faint draft moves through the frame, making the flame flicker and lean; the wick crackles quietly and a thin metallic spark flicks at the tip as the flame thins, then a short airy whoosh is heard exactly as the flame collapses and goes out. A slender plume of blue-gray smoke is rising in a slow spiral from the still-glowing ember, the glowing tip dimming to a dull black nub while a few droplets of settling wax produce soft, muted clicks; ambient room hush underlies the moment with a low, unobtrusive room tone and a distant faint creak, and the residual sound of the extinguish — a soft breath-like hiss and the last faint pop — lingers as the scene holds on the quiet candlestick and the smoke disperses."}
|
||||
{"id": "vidprom_semantic_unique_gpt-5-mini-2025-08-07_60589", "video_prompt": "A tight close-up of a brown-gray tabby cat with distinct black stripes and a faint white chin, lying on a soft cream knit blanket under soft side light from a nearby window; the cat blinks slowly, then leans in and presses its cheek and forehead into the blanket in a series of gentle, deliberate nuzzles while its whiskers brush the fabric and its ears flick slightly. Its green eyes remain half-closed and its shoulders shift forward as it rubs, then tilts its head and repeats another short, purposeful rub; the fur has a soft, slightly glossy texture and small tufts move with each motion. Ambient indoor room noise is quiet—a low, steady HVAC hum and distant muted street sound—then a low, steady purr begins as the cat first nuzzles, rising into a warm, continuous rumble synchronized with the rubbing. Each contact is accompanied by a soft fabric rustle and a faint breathy exhale; after the final nuzzle the cat releases and emits a brief, contented trill before settling, the purr continuing gently under the room ambience."}
|
||||
{"id": "vidprom_semantic_unique_gpt-5-mini-2025-08-07_60601", "video_prompt": "In a medium close-up, a couple in their late 20s is sitting across a small round wooden table in a cozy cafe, the woman with shoulder-length dark hair and a cream sweater holding a white ceramic cup, the man with short brown hair, light stubble, and a navy button-down resting his forearms on the table; soft warm yellow pendant lights overhead are reflecting gently on their faces and creating small highlights on the cup and the glass vase between them, while the background is softly blurred with other patrons and warm wood tones. Low indie acoustic music is playing quietly from the cafe speakers as steady ambient murmur, occasional silverware clinks, and a distant espresso machine hiss form a continuous soundbed. She is smiling and laughing, tucks a strand of hair behind her ear, and says in a warm, laugh-tinged voice, \"I'm really glad we came out tonight,\" with a short pause and a quick smile toward him; a soft clink occurs as he sets his cup down. He is smiling back, leaning in slightly, and replies in a soft, even voice, \"Me too—it's been nice,\" pausing to meet her eyes while a chair scrapes gently in the background. She is glancing down at her cup, then looking up with a brighter smile and says in a quiet, upbeat voice, \"Let's do this again,\" as the music swells a fraction and the cafe ambience continues beneath their voices, leaving them both smiling and holding eye contact."}
|
||||
{"id": "vidprom_semantic_unique_gpt-5-mini-2025-08-07_60632", "video_prompt": "A wide view of a lush forest of mixed hardwoods and fruit-bearing trees rising beside a calm lake, soft late-afternoon light filtering through green leaves and dappling mossy trunks. Low branches are heavy with red apples and yellow pears, their glossy skins catching the light as a gentle breeze is moving through the canopy—leaves are rustling and small clusters of fruit are swaying while a few water droplets bead on leaf edges. At the lake edge, reeds are bending slightly and the grey-blue surface is forming narrow ripples where the wind touches it; a dragonfly is skimming the water and a small fish is breaking the surface with a soft plop, sending a brief circular ripple outward. Ambient audio is layered and synced with action: the wind is creating a steady soft rustle through leaves, clear birdsong (quick sparrow-like chirps and a distant thrush) is punctuating the treetops, a low insect buzz is underscoring the scene, and gentle water lapping is meeting the pebbled shore; when a fruit brushes a branch a muted thud and the faint crunch of leaf litter register. The scene feels tranquil but active, natural motions continuing throughout the short clip."}
|
||||
File diff suppressed because it is too large
Load Diff
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user