Source: Dreamverse repo at will/integrate-public-fastvideo @ ec8ef92.
Cross-repo move; original history retained in archived Dreamverse repo.
Per .agents/memory/dreamverse-integration/integration-plan.md Phase 3.
* Frontend moved to apps/dreamverse/web with Next/Vitest/Playwright config
rewired for the monorepo location.
* Product docs, scripts, launch helpers, prompt assets, and Dreamverse
diagrams/assets moved under apps/dreamverse/.
* Public streaming demo config added at examples/serving/streaming_demo.yaml
with non-private placeholders.
* Launch/smoke scripts now resolve DREAMVERSE_ROOT from apps/dreamverse
and run frontend commands from apps/dreamverse/web.
* Dreamverse asset allow-list narrowed in .gitignore so copied image/video
assets can be staged without broadly tracking generated build output.
* Vitest excludes Playwright e2e specs; obsolete page.integration assertions
remain skipped until updated for the current migrated shell.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
Live smoke test on GPU4 confirms public fastvideo serve from will/ltx2_sr_port HEAD d23e71c2 boots cleanly with bf16 fallback (~24s) and empirically validates drift item #4 (only /health exists; /healthz, /readyz, /status, /prompt-system-config, /curated-presets all 404 — Phase 4 promotion target).
Plan doc updates:
* CUDA_VISIBLE_DEVICES=4 pin convention documented (original Dreamverse-side server uses this; smoke commands now consistently include it).
* flashinfer-python + flash_attn marked as Phase 0 prerequisites for production-equivalent NVFP4 smoke; bf16 fallback path documented for hosts without these deps.
* Cleanup commands updated with the verified setsid/disown launch pattern that survived the smoke session.
* Smoke-test evidence (boot time, /health JSON, endpoint matrix) recorded inline as a 2026-05-05 verification anchor.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
Audit found the test strategy in integration-plan.md was partially stale
from the prior Option D / split-PR drafting and missing an explicit
local-GPU verification path. This commit reconciles testing with the
single-mega-PR (D-17) + Option B+ monorepo (D-18) reality and makes the
GPU4-on-this-node hook explicit.
* New "Test strategy" subsection (added before "File-by-file migration
map"): test taxonomy table (unit / integration-with-fakes / live-GPU
/ FE build / FE Playwright / contract / SSIM), pytest marker
scheme (`@pytest.mark.gpu`), pyproject.toml addopts to default-skip
GPU tests in ubuntu-latest CI, and a per-phase responsibility matrix
(which class runs in which gate per Phase 0..7).
* GPU4 local verification hook documented: this dev node has 8x B200;
GPU4 currently held by Dreamverse-side server PID 2453227 / port
8009; concrete kill / redeploy / smoke-test / cleanup commands so
any phase touching the live-service path (Phase 2/3/4/5) can be
validated locally without waiting for Buildkite-Modal.
* Phase 2 verification gate: `uv run` command upgraded to
`--locked --extra test pytest ... -m 'not gpu'` (matches Phase 1
`--locked` enforcement). Added explicit MANUAL GPU4 QA gate
required, not optional: kill PID 2453227, deploy `fastvideo serve`
on GPU4 from this branch, hit `/health`, run `pytest -m gpu`
against the live deploy, capture output in PR. Rollback expanded
to handle GPU4-discovered bugs.
* Phase 3 verification gate: Playwright in CI is now explicitly
DEFERRED to Phase 4 (matches `ci-dreamverse-frontend.yml` scaffold
which already comments out Playwright steps until health routes
land). Removed the contradictory "or PR note explains" clause.
Added recommended manual GPU4 Playwright `frontend-shell.spec.ts`
against the Phase 2 GPU4 deploy. Same `--locked --extra test`
upgrade for the backend regression check.
* Phase 4 verification gate: Frontend CI Playwright RE-ENABLED at
end of this phase (the natural re-baseline once health routes
land). Manual GPU4 full E2E for all 3 Playwright specs
(backend-health + frontend-shell + preset-prompt-generation) is
required to merge.
* OSS precedent table: clarified that chainlit's "split CI" is a
per-language CI split (frontend vs backend workflows), not a
PR-level split — independent of D-17's single-mega-PR decision.
Removes any reading-confusion that the testing strategy carries
forward from the abandoned split.
No claims, references, or commands tied to the abandoned split-PR
branches (`will/api_7.10`, `will/api_8`, `will/ltx2_sr_runtime`,
`will/ltx2_nvfp4`, `will/ltx2_post_fixes`, `will/agents_cleanup`)
remain in the plan doc. All test commands route through
`apps/dreamverse/server/` and `apps/dreamverse/web/` per the B+
layout.
PR #1288: https://github.com/hao-ai-lab/FastVideo/pull/1288
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
The 10-PR / 6-remaining-slice split tracked by STACK.md is no longer the
landing strategy. Per [decisions-log.md D-17](.agents/memory/dreamverse-integration/decisions-log.md#d-17),
the remaining `will/ltx2_sr_port` content ships as a single mega-PR
(#1288) instead of 6 stacked PRs. The bulk-rebase formula and split
bookmark table in STACK.md no longer reflect reality, so keeping the
file in the repo would mislead future agents/contributors.
Already deleted:
* PR #1287 (was first slice of split) — closed
* origin/will/api_7.10 — remote branch deleted
* Local split bookmarks (will/api_7.10, will/api_8, will/ltx2_sr_runtime,
will/ltx2_nvfp4, will/ltx2_post_fixes, will/agents_cleanup) — deleted
Kept:
* CO-AUTHORS.md — still the canonical roster reference (mirrored in
.agents/memory/dreamverse-integration/authors.md but the top-level
file is the source of truth for the broader stack history).
* will/ltx2_sr_port-pre-1286-rebase — local safety backup of the
pre-rebase chain, scheduled for deletion after #1288 merges.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
User decision after the post-#1286 rebase + re-slice cycle. Closing
#1287 (slice 1-3); opening #1288 on will/ltx2_sr_port covering the full
34-commit chain (71 files, +13,074/-583 LOC) at once. STACK.md model
deprecated; split bookmarks no longer maintained.
* decisions-log.md: add D-17 with rationale (review-coordination
overhead vs structural benefit, layers not actually independent,
CI/merge-queue simplicity), implications (STACK.md deprecated,
bookmarks stale, runbook protocol replaced), and watch-outs (PR
size, fallback to re-split if main moves significantly). Bumps
Last updated header.
* README.md: switch narrative — single mega-PR #1288 on
will/ltx2_sr_port @ 39dfa009 replaces the 6-PR split plan. PR #1287
CLOSED. Cross-link to D-17.
* state.md: header reflects strategy reversal. Branch tips collapse
the deprecated split bookmarks into one row noting D-17 deprecation.
* pr-roadmap.md: "In flight" is now the mega-PR #1288. New "Closed
PRs in this scope" section captures #1287's history. New
"Deprecated split bookmarks (D-17)" section names the abandoned
branches. "Planned" section trimmed to post-#1288 work.
* open-threads.md: header updated. Item D resolution gate flipped
from #1287 merge to #1288 merge — same content, different vehicle.
* runbook.md: branch-topology section reflects single-PR model.
"After a PR merges (re-slice protocol)" replaced by "After PR
#1288 merges" with simpler 6-step cleanup. The deprecated
re-slice protocol is preserved in git history at b34d9704 and
referenced from state.md.
* authors.md: PR #1287 row marked CLOSED; new PR #1288 row added
with full 34-commit / 71-file / +13,074 LOC scope. All 34 commits
carry the 4 co-author trailers (verified per the per-commit
trailer block in every commit message).
PR #1288: https://github.com/hao-ai-lab/FastVideo/pull/1288
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
PR #1286 merged at 2aaeee2a (squash). will/ltx2_sr_port rebased onto
new origin/main, dropping 4 commits whose content is now in main:
cd76cf51 + 1ac1e732 + b0b7f59c + 40e265b8. Backup preserved on
will/ltx2_sr_port-pre-1286-rebase. PR #1287 opened on will/api_7.10
@ 6ae7a99f (slice 1-3, generate_async + VideoEvent + tests).
* README.md: bump Last reconciled to post-#1286 merge + rebase. Note
new working tip b34d9704 on will/ltx2_sr_port (33 commits ahead),
PR #1287 OPEN MERGEABLE, backup branch preserved.
* state.md: rename header to "post-#1286 merge + rebase". Branch-tips
table refreshed with all 8 relevant branches and the safety backup.
Add "Post-#1286 rebase summary" describing the 4 dropped commits +
zero conflicts. Add "New linearized chain" table mapping every
surviving slice (7.10/8/LTX-2 SR/NVFP4/post-fixes/agents_cleanup)
to its tip SHA. Mark the historical layered-chain section as
pre-rebase narrative (SHAs only valid on the backup branch).
* pr-roadmap.md: promote PR #1284 (7.8) and PR #1286 (7.9) to Landed
with merge SHAs eb3a3942 and 2aaeee2a respectively. Move PR #1287
(7.10) from Planned to In flight with full scope description.
Refresh remaining Planned rows with new slice indices and new tip
SHAs from the rebased chain.
* open-threads.md: bump Last updated header to capture the merge +
rebase + #1287 opening. Item D (Implement generate_async) flipped
from "High pri" to "🟢 in flight" with the live PR #1287 reference;
resolution gate is the merge of #1287 alongside Q-5/Q-9/PR-7.5/
D-12-B chain.
* runbook.md: bump Last updated. Branch-topology section reflects
PR #1286 -> merged, PR #1287 -> active. Add a new "After a PR
merges (re-slice protocol)" section with the 10-step recipe used
in the post-#1286 rebase, derived from the canonical STACK.md
bulk-rebase formula adapted for squash-merge drops.
* authors.md: bump Last updated. PR #1286 row promoted to merged with
squash-merge note about how the trailerless cherry-pick a152cb77
was absorbed cleanly. PR #1287 row added with all 4 trailers
verified on every commit. Known-gaps section rewritten — the two
trailerless commits are no longer reachable from any active branch
(a152cb77 absorbed by squash, 40e265b8 dropped by rebase). Gap
permanently resolved; backup branch preserves them archeologically.
Verified: 17/17 router tests pass on rebased branch (router code now
loaded from origin/main). 221 passed across api/contract/NVFP4 suites.
Pre-commit clean (memory dir is yapf/ruff/mypy excluded; only
spaces-check runs). PR #1287 created at
https://github.com/hao-ai-lab/FastVideo/pull/1287, MERGEABLE.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
Make the memory dir self-sufficient as an onboarding package — a fresh
agent should be able to resume work end-to-end (verify, commit, push,
propagate to PR #1286) using only files in this directory.
* runbook.md (new): operational how-to. Covers worktree contract,
branch topology, verification (pre-commit binary path, pytest,
lsp_diagnostics, gh CLI), commit workflow (subject/body conventions,
required co-author trailers, the multi-`-m` trailer-parsing trap and
its `-F` workaround), push + PR propagation (cherry-pick from
ltx2_sr_port to api_7.9 to avoid force-push), memory-dir maintenance
table, common pitfalls (pre-commit binary location, stash@{0} not
ours, AbsMaxFP8 pre-existing failure, untracked nested clones, live
ports, branch-switch by other agents, force-push policy, two known
trailerless commits in PR #1286), 8-question self-test, and a "first
60 seconds" copy-paste orientation block.
* README.md: replace single "Reading guide" with a two-tier structure.
New "Fresh-context onboarding (read in order)" section gives 5
ordered steps for an agent picking up the work for the first time —
worktree confirmation via runbook, then state.md, pr-roadmap.md,
open-threads.md, runbook.md end-to-end, finishing with the runbook
self-test as a context-loaded check. Original table preserved as
"Deep-dive reading guide" with a new entry for runbook.md. Bumps
"Last reconciled" header to 2026-05-05 / `09647a30` and updates the
PR status line (PR #1284 merged; PR #1286 open at `a152cb77`,
MERGEABLE).
* state.md: bump "Current State" header from 2026-05-03 to 2026-05-05.
Branch-tips table refreshed — `will/ltx2_sr_port` now at `09647a30`
(was `156103b9`), 32 commits ahead of i2v base; new row for
`will/api_7.9` (PR #1286 head `a152cb77`) noting it's an ancestor of
the working branch. Added cross-links to runbook.md and authors.md
in the front matter, plus a note about other agents sharing the
worktree (covers the branch-switch scenario without inventing
reconciliation logic).
Verified: pre-commit clean (memory dir is yapf/ruff/mypy excluded; only
spaces-check runs). All cross-links resolve to siblings in this
directory or to top-level CO-AUTHORS.md / STACK.md / AGENTS.md as
appropriate.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
* authors.md (new): dreamverse-integration-scoped mirror of the top-level
CO-AUTHORS.md. Documents the 4 human co-authors credited on every commit
in the integration scope, verified against PRs #1257/#1258/#1284/#1286.
Self-contained so the memory dir is discoverable without traversing to
the repo root. Cross-references CO-AUTHORS.md and STACK.md. Includes a
'Known gaps' section flagging that commits a152cb77 / 40e265b8 (the
[fix] streaming: router polish pair from earlier in this session) are
missing trailers and need an amend + force-push to fix.
* README.md: register authors.md in the reading guide table.
* decisions-log.md: add D-16 — Streaming router polish round 2 —
capturing the 5 second-pass fixes applied on top of D-15's pre-merge
polishes (bridge cancellation hygiene, registry UNKNOWN->HEALTHY
immediate, httpx hard-fail, URL path/query/fragment/duplicate
validation, per-index YAML parser errors, explicit websockets
[streaming] dep, +7 test cases). Bumps Last updated header.
* open-threads.md: bump Last updated header to mention the second-pass
router commit and link to D-16.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
Adds D-15 to decisions-log.md: Oracle review of PR #1286 streaming
router. Verdict — keep current shape (Alt A): in-repo at
fastvideo/entrypoints/streaming/router/, FastAPI-based, single-primary
failover, lazy httpx/websockets imports. Reject Alt B (separate
package), Alt C (fold into server), Alt D (delegate to
nginx/envoy/HAProxy as the sole answer), Alt E (sticky now), Alt F
(round-robin now).
Captures the 4 gemini review fixes:
* High: select() docstring claimed round-robin but always returns [0]
— rewrote with explicit MVP semantics
* Medium: per-probe httpx.AsyncClient prevented connection reuse —
refactored run_health_check_loop to share one client via
_build_default_probe() async context manager
* Medium: sequential probes could fall behind interval — now use
asyncio.gather for parallel per-cycle probes
* Medium: parser duplication — kept manual mapping intentionally
because YAML has nested health_check: block while RouterConfig is
flat; restructuring is beyond this PR's scope
Plus 1 Oracle pre-merge polish:
* RouterConfig.__post_init__ validation — empty replicas, non-positive
intervals/timeouts, thresholds < 1, non-http(s) URLs, and >1
primary all raise ValueError so misconfigurations surface at
config-load instead of confusing runtime failures
Plus 1 prep-rebase item carried in:
* Migrated FastAPI @app.on_event(startup/shutdown) to a single
@contextlib.asynccontextmanager-based _lifespan() handler
(deprecated API in newer FastAPI; was the 7.9 caveat in
pr-roadmap.md)
Adds open-threads.md items #13/#14/#15:
* #13: sticky session routing extensibility (forward-compat from D-15)
* #14: bridge backpressure note for high-scale deployments
* #15: multi-primary semantics if active-active becomes required
Adds D-14 to decisions-log.md: Oracle review of PR #1284 streaming
auxiliaries. Verdict — keep current shape (Alt A): single PR, 4 modules
under streaming/, mock_server in production module path, concrete
PromptSafetyFilter. Reject Alt B (split into 4 PRs), Alt C (move mock
to tests/), Alt D (premature observability extraction), Alts E/F
(premature Protocol-ization).
Captures the 4 gemini review fixes (high: session_logger race vs close;
medium: rewrite regex in hot path, safety _ensure_loaded race,
streaming extra missing prompt-safety) and 2 pre-merge polishes from
Oracle (remove inert RewriteOptions.user_system_prompt_override field,
sanitize session_id for filename safety as defense-in-depth).
Adds open-threads.md item #12 (Low): when streaming server starts using
PromptSafetyFilter, ensure operator-visible logging on
SafetyDecision.UNAVAILABLE results — surfaces degraded-safety state per
D-14's Watch-Out note.
Updates following the PR landings during this session:
* pr-roadmap.md: move PR 7.5 (#1251 merged 2026-04-26 as 95fd29e0),
PR 7.6 (#1257 merged 2026-05-04 as eb0a4152), and PR 7.7 (#1258
merged 2026-05-04 as f673423b) into 'Landed PRs' section. Add merge
commit refs and link to D-12 / D-13 architecture reviews. Add PR 7.8
(#1284 OPEN) as the new 'In flight' entry. Add LTX-2 SR / NVFP4 /
agents-cleanup branches to the Planned section as out-of-band streams
ready to open.
* open-threads.md: mark items #9 + #10 RESOLVED (commit-message
cleanups bundled into the will/api_7.8 prep rebase). DR-1 (Dreamverse
compat shim) marked actionable now that #1258 has merged. Recommended
pull order updated.
* README.md: bump last-reconciled date and current branch tip
(89a6484d post-rebase). Note PRs 1257 + 1258 merged; PR 1284 open.
Adds D-13 to decisions-log.md mirroring D-12's format: PR #1258 prompt
enhancer / LLMProvider abstraction shape review by Oracle. Verdict —
keep current shape (Alt A); promote to top-level fastvideo.prompt.* only
when a second non-streaming consumer exists. Don't convert Protocol →
ABC. Three deferred polishes captured.
Adds 9 new items to open-threads.md priority table:
* DR-1 (High): Dreamverse must create prompting/_internal_compat.py
shim and delete most of the 1933-LOC local prompt_enhancer.py once
PR #1258 merges.
* DR-2 (Med): Decide cerebras_ifm provider path — public Literal vs.
Dreamverse-side custom provider via register_provider().
* D-12-A (Med): GpuPool ABC docstring — mark experimental /
server-internal until PR 7.10 cycle.
* D-12-B (Med): Replace GpuPool.run() -> Any with run_async() ->
AsyncIterator[VideoEvent] in PR 7.10 cycle.
* D-13-A (Med): Document streaming/prompt/* as streaming-scoped in
user-facing docs; avoid framework-level framing.
* D-12-C (Low): Avoid locking PoolAssignment.gpu_id: int as public;
prefer worker_id (already exists) or device_ids: list[int] for
future multi-GPU-per-worker.
* D-13-B (Low): Optional client_factory parameter for httpx pooling
if metrics justify.
* #9 (Low): PR 8's three commits still have [8/n] Improve API:
prefix — regex didn't match single-digit version. Cleanup when
PR 8 is opened.
* #10 (Low): PR 7.8/7.9 commits have streaming: streaming X
duplication. Cleanup when 7.8/7.9 are opened.
* #11 (Low): Promote LTX-2 prompt orchestration (locked segments,
segment_prompts JSON shape) to public when a second consumer
appears.
Updates recommended pull order to include the new items.
Adds D-12 to decisions-log.md, documenting the Oracle review of whether
GpuPool (PR #1257, just merged) should be folded into VideoGenerator.
Decision summary: keep GpuPool separate from VideoGenerator (Alt A) as
interim, evolve to Alt C (thin async executor over generate_async) once
PR 7.10 lands. Do not pursue Alt B (folding into VideoGenerator).
Captures:
* Three alternatives evaluated (A: status quo, B: VideoGenerator absorbs
pool role, C: async executor over generate_async)
* Key finding: MultiprocExecutor and SubprocessGpuPool are orthogonal,
not redundant — both spawn subprocesses because CUDA contexts demand
process boundaries, but they solve different problems (one-call-fast
vs. many-sessions-throughput)
* Sticky binding stays in pool, NOT in VideoGenerator (different
consumers want different policies — sticky for streaming, lease for
HTTP, queue for per-frame real-time)
* Risks flagged: GpuPool.run() sync shape, PoolAssignment.gpu_id: int
freeze, and the 'is this canonical serving API?' framing question
Action items deferred to PR 7.10 cycle: replace run() with run_async()
returning AsyncIterator[VideoEvent], clarify worker_id vs gpu_id field
naming, mark GpuPool docstring as experimental until 7.10.
* STACK.md tracks the 10-PR split of will/ltx2_sr_port: per-PR slice
ranges, tip SHAs (live, not hardcoded), independence map, landing
strategy, fast-track candidates, and re-slice commands to run after
any rebase or trailer injection.
* CO-AUTHORS.md documents the 4 GitHub users credited as co-authors
via Co-authored-by trailers on every commit in the stack:
- @Davids048 (Junda Su)
- @RandNMR73 (Matthew Noto)
- @XOR-op
- @jzhang38 (Zhang Peiyuan)
All 4 verified active in FastVideo-internal git history. Trailer
emails use GitHub's <id>+<username>@users.noreply.github.com form
for reliable account linkage.
Both files are temporary trackers (delete after the full stack merges
or as documented at the bottom of each file).
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
Phase 1 of .agents/ cleanup. All deletions are stale, redundant, or
unused; .agents/ memory + skills + workflows remain functional.
Removed:
* STATUS.md — hand-maintained dashboard, last synced 2026-03-02 with
wrong counts (claimed 8 skills/4 workflows/4 memory; actual 9/5/5)
and references to non-existent snake_case filenames. Strictly
redundant with .agents/{memory,skills}/index.jsonl.
* workflows/sync-dashboard.md — SOP that maintained the deleted
STATUS.md, with obsolete pre-PR-4 file paths.
* skills/index-related-work/, skills/search-related-work/ — vapor
skills operating on the empty .agents/memory/related-work/ registry.
The search skill literally requires 'index has entries' as a
prerequisite but none have ever been added. Re-introduce when the
registry is actually populated.
Updated skills/index.jsonl to drop the two removed entries (was 9
entries; now 7).
.agents/scripts/sync-skills.sh ran post-deletion and pruned 2 stale
.claude/skills/ symlinks pointing at the removed skill directories.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
Wrap the device move in an equality guard so Dynamo can DCE it under
fullgraph=True (model.to() goes through _parse_to, which returns a
non-Tensor torch.device that Dynamo can't trace). The model is already
placed on the target device by prepare_for_compile, so the guard is a
runtime no-op. Also drop the orig_device save/restore — the encoder
should stay resident on the compute device between calls.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
Two new contract test files plus a few stale-name doc fixups so the
LTX-2 NVFP4 path can't silently regress.
* ``test_nvfp4_ltx2_wiring.py`` (6 tests): asserts that
- ``LTXSelfAttention``'s ``to_q/to_k/to_v/to_out`` are
``ReplicatedLinear`` (not plain ``nn.Linear``);
- ``NVFP4Config()`` attaches ``NVFP4QuantizeMethod`` to the
quantized subset and the ``layer_prefix`` matches the
``ltx2.blocks.<i>.<sub>.<proj>`` paths in
``NVFP4Config.fp4_layers``;
- non-tagged projections (cross-attn K/V, audio attn, audio FFN)
fall back to ``UnquantizedLinearMethod`` instead of crashing on
the quant_method assert;
- ``BasicAVTransformerBlock`` propagates ``quant_config`` and
``prefix`` correctly to all 4 attention modules + FFN at once.
These are CPU-only and don't need flashinfer.
* ``test_typed_quant_flow.py`` (4 tests): asserts that
- typed ``engine.quantization.transformer_quant: "NVFP4"`` resolves
through ``compat.py`` to a concrete ``NVFP4Config()`` instance;
- omitting the typed surface leaves the ``transformer_quant``
carrier ``None`` so legacy callers that mutate
``pipeline_config.dit_config.quant_config`` directly keep working;
- ``__post_init__._apply_transformer_quant`` pins the carrier onto
``dit_config.quant_config``;
- explicit ``dit_config.quant_config = …`` is preserved (the
explicit setter wins over the typed carrier).
Plus stale ``FP4Config`` → ``NVFP4Config`` doc updates in
``api/compat.py``, ``fastvideo_args.py``, and ``layers/linear.py``
that the previous rename commit missed (comments and docstrings only,
no behavior change).
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
The FP4 implementation is specifically NVIDIA's block-scaled FP4
(e2m1 mantissa, fp32 alpha, ``layout_128x4`` scale layout, group
size 16) backed by FlashInfer's ``nvfp4_quantize`` and ``mm_fp4``
kernels. Naming the public surface ``FP4`` would collide with other
FP4 variants we may want to support later (OCP-FP4 / MX-FP4 / e3m0).
Mechanical rename, no behavior change:
* ``fp4_config.py`` → ``nvfp4_config.py``
* ``FP4Config`` → ``NVFP4Config``; ``get_name()`` returns ``"nvfp4"``
* ``FP4QuantizeMethod`` → ``NVFP4QuantizeMethod``
* ``convert_model_to_fp4`` → ``convert_model_to_nvfp4``
* ``QuantizationMethods`` literal: ``"FP4"`` → ``"NVFP4"``
* registered buffer names: ``_fp4_weight``/``_fp4_alpha`` →
``_nvfp4_weight``/``_nvfp4_alpha``
* loader helper ``_maybe_convert_model_to_fp4`` →
``_maybe_convert_model_to_nvfp4``
* test file rename + symbol updates
* Dreamverse worker updates its single import + call site
Internal-scope torch op namespace ``fastvideo_fp4::*`` and
``_get_ltx2_fp4_stage_profile`` left as-is (purely internal naming
that mirrors FastVideo-internal).
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
The public LTX-2 path was previously running full bf16 even when
callers set ``pipeline_config.dit_config.quant_config = FP4Config()``,
because:
* the DiT used plain ``torch.nn.Linear`` for FP4-eligible projections
(``attn1/attn2.to_q/k/v/out``, ``audio_to_video_attn``,
``video_to_audio_attn``, FFN ``fc_in``/``fc_out``), so
``LinearBase.quant_method.apply`` was never reached;
* the loader did not call ``convert_model_to_fp4`` after weights
loaded, so even matched layers had no ``_fp4_weight*`` buffers;
* ``QuantizationMethods`` did not register ``"FP4"``, so the typed
``engine.quantization.transformer_quant`` surface had no way to
select it.
This commit completes the inference wire-up through the existing
``fastvideo.layers.quantization`` registry — no parallel pathway:
* swap ``nn.Linear`` → ``ReplicatedLinear`` for the LTX-2 attention
``to_q``/``to_k``/``to_v``/``to_out``/``to_gate_compress`` and the
FFN ``fc_in``/``fc_out`` (``GELUApprox.proj`` and ``FeedForward``
``project_out``). Other linears (timestep MLP, caption proj,
patchify, AdaLN scale/shift) stay ``nn.Linear`` to match internal.
* port ``_supports_prequantized_input`` and
``_linear_project_with_optional_prequant`` so the attention forward
quantizes input once and reuses the ``(x_fp4, x_scale, x_global_sf)``
tuple across q/k/v projections — matches internal exactly.
* plumb ``quant_config`` and ``prefix`` through
``BasicAVTransformerBlock`` → ``_init_transformer_blocks`` →
``LTXModel`` → ``LTX2Transformer3DModel`` so each
``ReplicatedLinear`` gets the correct
``ltx2.blocks.<i>.<sub>.<proj>`` prefix that
``FP4Config.get_quant_method`` matches against.
* register ``"FP4"`` in ``QuantizationMethods`` literal +
``get_quantization_config`` registry so typed
``engine.quantization.transformer_quant: "FP4"`` resolves to a
concrete ``FP4Config()`` instance.
* add ``transformer_quant`` carrier on ``FastVideoArgs`` and
``__post_init__._apply_transformer_quant`` to pin the resolved
quant config onto ``dit_config.quant_config`` (without
overwriting an explicit setter on the dit_config).
* add ``_maybe_convert_model_to_fp4`` in ``fsdp_load.py`` that
walks the loaded model once and registers
``_fp4_weight*``/``_fp4_alpha``/``_weight_global_sf`` buffers on
layers whose ``quant_method`` is ``FP4QuantizeMethod``. flashinfer
is imported lazily inside ``convert_model_to_fp4`` so this is a
no-op on hosts without the FP4 kernels.
* restore ``LinearBase`` fallback to ``UnquantizedLinearMethod`` when
``quant_config.get_quant_method`` returns ``None`` — ``FP4Config``
only tags a curated subset of layers, and the previous
``assert quant_method is not None`` would crash any non-tagged
layer that received a quant_config.
Non-FP4 callers are unaffected: ``UnquantizedLinearMethod.apply``
runs the standard ``F.linear`` path, and parameter names
(``to_q.weight``, ``to_out.0.weight`` from ``nn.ModuleList``) are
identical to the previous ``nn.Sequential`` layout.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
Bring the public ``ComposedPipelineBase.post_init`` compile dispatch
to feature parity with FastVideo-internal:
* Compile ``transformer_refine`` alongside ``transformer`` and
``transformer_2`` whenever the DiT compile flag is on. Without this
the LTX-2 stage-2 refine pass silently runs eager while the main
transformer is compiled — different inductor fusion choices vs.
eager execution can produce last-bit divergences in bf16.
* Drive the new per-component compile flags from ``FastVideoArgs``:
text encoder (``enable_torch_compile_text_encoder``), VAE
(``enable_torch_compile_vae``), audio VAE
(``enable_torch_compile_audio_vae``). Each picks its own kwargs
dict (``torch_compile_kwargs_*``) and falls back to the master
``torch_compile_kwargs`` when empty.
* Call ``module.prepare_for_compile()`` on each compiled submodule
before invoking ``torch.compile`` so model-specific external state
(e.g. lazy HF loads) lands outside Dynamo's tracer. ``Gemma3``
implements this hook to materialize the Gemma weights up front.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
Extend the typed inference API so callers can drive torch.compile per
component without falling back to the legacy flat-kwargs pathway.
* ``CompileConfig`` now exposes ``vae_enabled``, ``audio_vae_enabled``,
``dit_kwargs``, ``text_encoder_kwargs``, ``vae_kwargs``,
``audio_vae_kwargs`` alongside the existing ``enabled`` /
``text_encoder_enabled`` switches. The master ``backend``/
``fullgraph``/``mode``/``dynamic``/``extras`` continue to apply to
every compiled submodule unless a per-component kwargs dict is
non-empty (in which case it overrides entirely — matches the
FastVideo-internal precedent).
* Add the matching runtime carrier fields on ``FastVideoArgs``:
``enable_torch_compile_text_encoder/vae/audio_vae`` plus
``torch_compile_kwargs_dit/text_encoder/vae/audio_vae``. These let
the legacy flat-kwargs path round-trip through the typed config.
* ``api/compat.legacy_from_pretrained_to_config`` now lifts the new
flat kwargs into ``CompileConfig``, and
``generator_config_to_fastvideo_args`` writes them back out so
``VideoGenerator.from_pretrained(model_path, ...)`` callers keep
working unchanged.
No behavior change yet — these surfaces are ports of carriers only.
The composed pipeline base will start consuming them in the next
commit.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
Three small parity fixes uncovered while comparing the LTX-2 distilled
path against FastVideo-internal:
* `FastVideoArgs.__post_init__` now calls `_resolve_refine_args()` to
copy the public-facing generic `refine_*` knobs onto their
`ltx2_refine_*` runtime carriers (mirrors internal lines 303-322).
Without this, callers that set `refine_lora_path=...` via the typed
CLI/config surface would silently drop the value, surfacing later as
the "applied to 0 layers" warning.
* Revert the patch-noise sampler in `_randn_ltx2_video_latents` from
`randn_tensor` back to `torch.randn` to bit-match internal under
single-generator inference. `randn_tensor` is identical for a single
`torch.Generator` but diverges for `list[Generator]` (per-sample
seeds), which is the only place the two paths could disagree.
* Classify the 19 previously-unclassified `refine_*` / `ltx2_refine_*`
/ `ltx2_audio_latent_path` / `ltx2_images` / `ltx2_image_crf` /
`ltx2_conditioning_latent_*` / `ltx2_video_conditions` fields in the
schema-parity inventory yaml so the parity test passes.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
Replaces the T2V-only stub with the full internal port:
* resolve_ltx2_images / _resize_and_center_crop / CRF re-encode
* load_ltx2_conditioning_image + load_ltx2_conditioning_video_clip
* _extract_video_latent / _insert_conditioning_latent
* build_ltx2_image_conditioning composes (clean_latent, denoise_mask)
from images + video clips + continuation latents (stage1 vs stage2)
— returns None for plain T2V
ForwardBatch + SamplingParam gain the missing fields the builder
reads (ltx2_images, ltx2_image_crf, ltx2_conditioning_latent_stage1/2,
ltx2_video_conditions). generate_video kwargs flow through
sampling_param.update -> ForwardBatch(**shallow_asdict(...)) so
Dreamverse's ltx2_image_crf=0.0 and continuation latents land on the
batch correctly.
68/68 LTX-2 tests still green.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
The model-name detector loop in get_model_name_for_path tests both
``model_path.lower()`` AND ``pipeline_name`` against each detector,
OR-ing the result. The pipeline_name for the distilled checkpoint is
``ltx2pipeline`` — a string that contains no "distilled" marker — so
the *base* detector's ``and "distilled" not in path`` predicate
returns True against it, even when the absolute model path clearly
contains "distilled".
Result: the resolver matches both LTX-2 base and LTX-2 distilled,
warns "Multiple models matched … Using the first matched: '0'", and
falls back to whichever was registered first. That used to be base
(ltx2_base preset → cfg=3.0 / mod=3.0 / rescale=0.7 / stg=1.0), so
SamplingParam.from_pretrained on a distilled-checkpoint path silently
loaded the *full* LTX-2 sampling defaults — exactly the divergence
the public-vs-internal Dreamverse alignment surfaced.
Reorder so the distilled entry registers first. The registry's
"first matched wins" tiebreak now lands on the more specific preset
when both detectors fire, and SamplingParam.from_pretrained reaches
the internal-aligned distilled defaults (mod=1.0 / rescale=0.0 /
stg=0.0 / cfg_video=1.0 / cfg_audio=1.0).
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
Internal FastVideo declares two LTX-2 SamplingParam variants:
LTX2SamplingParam (mod=3.0/rescale=0.7/stg=1.0 — full LTX-2) and
LTX2DistilledSamplingParam (mod=1.0/rescale=0.0/stg=0.0). Dreamverse
production runs against LTX2-Distilled-Diffusers, so the distilled
defaults are what the runtime should land on.
The public package only has the single SamplingParam class, and its
class-level defaults for these knobs were the *full* LTX-2 values.
Result: when Dreamverse (or any other consumer) called
``SamplingParam.from_pretrained("FastVideo/LTX2-Distilled-Diffusers")``,
the LTX2_DISTILLED preset's defaults dict didn't override these
fields (it didn't list them) so they fell through to the class
defaults — silently picking up *full*-model CFG / modality / rescale /
STG instead of the distilled ones. Streamed video output diverged
from the internal-ui reference.
Switch the class defaults to the distilled values (1.0 / 0.0 / 0.0).
The LTX2_BASE preset already overrides them explicitly to 3.0/0.7/1.0
in its ``defaults`` dict, so users selecting the full-LTX-2 preset
keep the full-model behavior.
The numerical alignment harness still reproduces the same 42.72 dB
PSNR / 52.87% exact pixel match because that test pinned the values
explicitly via legacy kwargs; this fix moves the *default* path on
the public side onto the same numerical track.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
Initial alignment showed PSNR ~12 dB because public's LTX2_BASE preset
sets ``ltx2_modality_scale_video=3.0 / rescale=0.7 / stg_video=1.0``
while internal's default ForwardBatch lands at ``1.0 / 0.0 / 0.0``.
Pinning these explicitly on both runs aligns the denoising mechanics
so the diff measures pipeline parity, not preset divergence.
Result with both sides pinned to mod=1.0/rescale=0.0/stg=0.0:
max_abs_diff = 203
mean_abs_diff = 0.788
rms_diff = 1.864
psnr_db = 42.72 dB
exact pixels = 52.87%
within 1 lvl = 86.94%
within 4 lvls = 97.93%
The ~47% near-but-not-exact bucket is consistent with bf16 reduction
non-determinism on B200 plus the fact that the public LoRA matcher
emitted "applied to 0 layers" for the LTX2-Distilled refine LoRA
(internal side did not). Both runs effectively skip the refine LoRA,
so they share the same effective stage-2 model — the residual diff is
the right ballpark for "same algorithm, different bf16 reduction
schedule".
The typed-API path also sets the same overrides via setattr on the
GenerationRequest so the typed path stays comparable. Once the typed
SamplingConfig grows ltx2-specific fields the setattr workaround can
go.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
LTX-2 SR pipeline now passes module load + stage graph build on the
public side. Each retry of scripts/ltx2_sr_alignment.py surfaced one
more gap; this commit closes them so the pipeline progresses to the
upsample → stage-2 → decode tail.
* harness: PipelineSelection (real class name; PipelineConfig was a typo)
* FastVideoArgs: generic refine_* fields (preferred user-facing API)
* LTX2Pipeline.initialize_pipeline: getattr-with-default for
debug_model_sums / debug_model_detail (public args lack the debug
fields; LTX-2 still flips the env-var toggles when set)
* VAEConfig: use_temporal_scaling_frames default True (LTX-2 latent
prep reads this in the schedule-shift gate)
* LTX2LatentPreparationStage._randn_ltx2_video_latents: switch to
randn_tensor so list-of-generators batches work (matches internal
sampling order)
* ModelRegistry: register LTX2LatentUpsampler in _UPSAMPLERS
* component_loader: register spatial_upsampler / temporal_upsampler
with UpsamplerLoader (was hitting the generic stub loader);
UpsamplerLoader gains LTX-2 fallback when pipeline_config.upsampler_config
isn't a multi-config tuple, plus the 'model.'-prefix-stripping
weight load path the internal version uses
7 retries -> public side now reaches the upsample stage.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
Single script with three modes:
* --label internal --output /tmp/ltx2_sr_internal.pt : runs the legacy
from_pretrained kwargs path and saves frames + metadata. Used against
PYTHONPATH=../FastVideo-internal as the reference run.
* --label public --use-typed-api --output /tmp/ltx2_sr_public.pt : runs
through the new GeneratorConfig + GenerationRequest typed API with
pipeline.preset_overrides["refine"] driving the SR path. Used against
PYTHONPATH=../FastVideo (the public package) for the candidate run.
* --diff --reference R --candidate C : loads both .pt files and reports
shape parity + max_abs / mean_abs / rms / psnr / pixel-bucket
histograms over the uint8 frames.
Pinned alignment fixture: PROMPT + seed + 1088x1920x121x24fps + 8 base
steps + 3 refine steps mirrors basic_ltx2_upscale.py upstream so we
exercise the same code path the user has been generating with.
Deliberately skips FP4 (no quant config) and Dreamverse runtime knobs
to keep the bf16 path clean per request.
The two runs are intentionally invoked in separate processes via
PYTHONPATH overrides because both repos register as the `fastvideo`
package and can't be imported together. The diff script then runs in
either env to load both .pt files and compute the alignment metrics.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
Brings the public LTX2Pipeline into parity with the internal SR-capable
version end-to-end:
* LTX2DenoisingStage gains the stage-2 control surface
(sigmas_override, num_inference_steps_override, force_guidance_scale,
initial_audio_latents_key) plus the i2v conditioning round-trip
(clean_latent + denoise_mask via LTX2_VIDEO_*_KEY) and the audio
refine path. The file is now ~1:1 with the internal stage; only the
i2v_conditioning import path differs (rerouted to the public
ltx2_image_conditioning module).
* LTX2LatentPreparationStage takes vae=, exposes the official LTX-2
noise sampling order via _randn_ltx2_video_latents (patchify -> noise
in token order -> unpatchify), and applies image conditioning when
build_ltx2_image_conditioning returns a state. Matches internal.
* LTX2Pipeline switches its base from ComposedPipelineBase to
LoRAPipeline (refine-LoRA support); create_pipeline_stages adds the
refine_init / upsample / refine_lora / refine_denoising chain when
ltx2_refine_enabled is True; load_modules pulls
spatial_upsampler (and optionally transformer_refine) from the
resolved upsampler path, with model_index.json defaults
(fastvideo_refine_*) feeding the FastVideoArgs.ltx2_refine_*
carriers.
stages/__init__.py re-exports the new refine stages so importers
outside the basic/ltx2 package keep working. 68/68 LTX-2-related tests
still pass after the rewrite (api translation, gpu_pool, contract,
streaming).
Reachability: setting pipeline.preset_overrides["refine"] = {"enabled":
True, ...} and components.upsampler_weights flows through compat.py
into the new ltx2_refine_* args, the pipeline branches, and load_modules
brings in the upsampler. T2V SR is now end-to-end runnable; numerical
alignment harness comes next.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
Lays the runtime-side pieces the LTX2_TWO_STAGE preset will exercise.
Behaviour matches FastVideo-internal 1:1 for the text-to-video SR path;
i2v / continuation conditioning is stubbed with a NotImplementedError
that fires only on those code paths so the public T2V SR run is clean.
* fastvideo/models/upsamplers/ltx2_upsampler.py — verbatim copy of the
internal LTX-2 latent upsampler (PixelShuffleND, BlurDownsample,
SpatialRationalResampler, ResBlock, LatentUpsampler, LTX2LatentUpsampler,
upsample_video). Re-exported via models/upsamplers/__init__.py.
* fastvideo/pipelines/basic/ltx2/stages/ltx2_image_conditioning.py —
minimal i2v-conditioning helpers: the four ForwardBatch.extra keys,
apply_ltx2_gaussian_noiser, post_process_ltx2_denoised, and a T2V-only
build_ltx2_image_conditioning that returns None when the request has
no image inputs / no continuation latents (the SR T2V path).
* fastvideo/pipelines/basic/ltx2/stages/ltx2_refine.py — the three
refine stages adapted to public imports: LTX2RefineInitStage halves
the request resolution and stashes the original target;
LTX2UpsampleStage runs the latent upsampler, optionally re-applies
conditioning, and mixes refine-noise scaled by the stage-2 sigma;
LTX2RefineLoRAStage swaps in a refine-specific LoRA (no-op when path
is unset).
* fastvideo/fastvideo_args.py — add the runtime carrier fields
ltx2_refine_{enabled,upsampler_path,transformer_path,lora_path,
num_inference_steps,guidance_scale,add_noise,noise_path,
audio_noise_path}. The typed public surface (ComponentConfig
upsampler_weights/lora_path + pipeline.preset_overrides.refine) keeps
flowing through compat.py:273-279, which expands the dict into these
ltx2_refine_* kwargs at construction time.
Not yet wired: the LTX2 pipeline's create_pipeline_stages still goes
denoise -> audio_decode -> decode without the refine_init / upsample
/ refine_denoise injection. The remaining piece is teaching
LTX2DenoisingStage to accept sigmas_override / num_inference_steps_override
/ force_guidance_scale / initial_audio_latents_key (the internal
version is 725 lines vs public 395; ~330 lines of stage-2 logic still
need bringing across) and to flip the pipeline graph based on
ltx2_refine_enabled. That follow-up unlocks the numerical alignment run.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
Ports the FP4 quantization config + ops from FastVideo-internal
(fastvideo.layers.quantization.fp4_config). flashinfer is imported
lazily inside _require_flashinfer() rather than at module top so
``import fastvideo`` stays cheap on hosts that don't ship flashinfer
— the FP4 kernels themselves still raise a clear ImportError pointing
at ``pip install flashinfer-python`` when invoked.
Behavior is identical to the internal version:
* FP4Config: LTX-2 layer set (48 transformer blocks across attn1/attn2
/audio_to_video_attn/video_to_audio_attn/ffn plus adaln_single.linear),
layer_profile selects between "base" and "refine" stage subsets.
* FP4QuantizeMethod: per-layer quantize_input + apply with stage-aware
routing for refine-only layers (audio_to_video_attn.to_q,
video_to_audio_attn.to_k, video_to_audio_attn.to_v fall through to
dense F.linear during the base stage).
* convert_model_to_fp4(): pre-quantize weight buffers + alpha onto
module attrs at load time so the forward path doesn't pay the
per-step weight-side quantize cost.
Required for Dreamverse integration: dreamverse-server's GPU worker
runs ``pipeline_config.dit_config.quant_config = FP4Config()`` during
LTX-2 model load, which is the dominant inference path.
test_fp4_config covers the lazy-flashinfer contract: import succeeds
without flashinfer; layer_profile round-trips via from_config; calling
into _require_flashinfer raises ImportError with an actionable
message when flashinfer is missing.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
fastvideo/tests/contract/ guards against drift in the two public-facing
adapter shapes that PR 8 locks down:
* test_dynamo_shape.py: mock Dynamo-style handler implementing the
adapter template from docs/design/server_contracts/dynamo.md. It
imports only fastvideo + fastvideo.api, translates a
Dynamo-shape NvCreateVideoRequest dict into a typed
GenerationRequest, runs it through an async generator that matches
endpoint.serve_endpoint(handler.generate, ...), and round-trips the
continuation-state envelope. A source scan asserts the adapter
function body touches no private FastVideo module.
* test_dreamverse_shape.py: the flat init-time kwarg bag the internal
gpu_pool.py passes today lands entirely on typed GeneratorConfig
fields (no leakage into pipeline.experimental) after PR 6's typed
replacement work. Per-request Dreamverse shape round-trips through
legacy_generate_call_to_request -> normalize_generation_request;
output.return_state (PR 7) stays wired through.
Failures in this suite appear at FastVideo CI — before the Dynamo-side
integration or private Dreamverse adapter see the drift.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
docs/design/server_contracts/dynamo.md documents the integration shape
the ai-dynamo/dynamo native backend package consumes. Per the refactor
decision, no Dynamo code lives in FastVideo — the backend package
ships in the Dynamo repo at components/src/dynamo/fastvideo/ with the
layout mirroring components/src/dynamo/sglang/.
Captured here:
* the 8-symbol public surface Dynamo imports (VideoGenerator plus
ContinuationState, GenerationRequest, SamplingConfig, InputConfig,
OutputConfig, VideoEvent, VideoResult) and what is available today
vs. what lands in PR 7.10
* target backend package layout matching the sglang template
* full NvCreateVideoRequest <-> GenerationRequest mapping table and
VideoFinalEvent <-> NvVideosResponse reverse mapping
* worked examples: aggregated handler (works today with generate_video
+ asyncio.to_thread), streaming handler (post-PR 7.10 via
generate_async), health check payload (swapping to
default_health_check_request() after PR 7.10)
* init function sketch, registration (ModelType::Videos fast path),
args adapter
* contract guarantees that let the adapter be written once and not
chase FastVideo drift
* explicit no-import list for the adapter
Also wires the server_contracts/ section into mkdocs.yml under Design.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
docs/design/server_contracts/ lands the contract reference set that PR
8 locks down ahead of the streaming + Dynamo upstreams. This commit
adds the index page plus the OpenAI HTTP reference:
* endpoint catalogue (/v1/videos/generations, list, status, content,
images, models, health)
* VideoGenerationsRequest shape with SGLang-compatible extensions
* merge precedence (body > default_request > hardcoded fallback)
with a pointer to the explicit-path tracking mechanism that makes
it work
* continuation-state roundtrip on the stateless surface
* HTTP error code table
* explicit non-goals: flat legacy kwargs, private Dreamverse fields,
raw tensor payloads — all excluded from this boundary by design
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
* tests/contract/test_generate_async.py: event ordering (Progress ->
Final); exactly-one-final per request; batch expansion emits one
Final per sub-result; VideoFinalEvent carries frames / full
GenerationResult / continuation_state as expected; health-check
request is minimal and round-trips through normalize_generation_
request; public fastvideo.api exports the new symbols; a mock
Dynamo-style async handler wraps generate_async using only the
public import surface.
* tests/api/test_cli_translation.py: the streaming-dispatch test no
longer expects NotImplementedError (PR 7.5 landed the live server);
it now injects a fake run_server and asserts the CLI hands the
typed streaming ServeConfig through unchanged.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
* VideoGenerator.generate_async(request) yields VideoEvent objects:
one VideoProgressEvent at start, one VideoFinalEvent per resulting
GenerationResult (prompt-batch expansion emits one Final per
sub-result). The aggregated code path runs the existing sync
generate() inside asyncio.to_thread; future work threads per-step
progress events from inside the pipeline's denoise loop without
changing the public contract.
* VideoGenerator.default_health_check_request() returns a 256x256 /
8-frame / 1-step typed GenerationRequest. Dynamo uses it to derive
its health_check_payload without knowing FastVideo internals.
* _final_event_from_result packages GenerationResult into a
VideoFinalEvent, reading the encoded MP4 off disk when the pipeline
wrote one so streaming consumers don't re-encode.
Also: streaming_server.run_server validates streaming config before
loading the generator so the existing ValueError fires before the
expensive model-load path.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
Add the typed event dataclasses VideoGenerator.generate_async will
emit, plus a VideoResult alias for the public docs:
* VideoProgressEvent / VideoPartialEvent / VideoFinalEvent
* VideoEvent = union of the three
* VideoResult = GenerationResult
Re-exported from fastvideo.api so the Dynamo backend package, the
streaming server, and the stateless OpenAI server share one set of
names.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
SolitaryThinker