01bf35ef407b4e933f9f8e9dc2c27dcb04963183
749
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Junda Su
|
01bf35ef40 | [refactor] Dreamverse 2/5: launch backend through installed CLI (#1314) | ||
|
Junda Su
|
1125cb789d | [feat] Dreamverse 1/5: package backend as installable module (#1313) | ||
|
|
ed377a99ba |
[docs] dreamverse-integration: D-26 — rebase onto origin/main; PR base flipped
* Records `will/dreamverse-monorepo` rebased directly onto
`origin/main` (
|
||
|
|
83829c5e56 |
[docs] dreamverse-integration: D-22 instrumentation closed; D-25 pipeline bench
* Strikethrough D-22 in open-threads.md (resolved by `bade2c0a` —
`stream_fmp4` now records `av_wav_write_ms`, `av_ffmpeg_spawn_ms`,
`av_first_chunk_ms`, `av_chunk_interval_ms_{min,median,p95,max}`,
`av_chunk_publish_ms_{median,p95}`, `av_chunk_read_ms_{median,p95}`).
* Add D-22-CTL as the new low-priority follow-up for controller-side
WS-send timing (between media event arrival and `ws_send_bytes`).
* Add D-25 entry recording the pipeline benchmark landed in `f98811e0`
and the cold-run baseline numbers captured on B200 NVFP4
(6.99s steady-state, 0.72x realtime without compile, LTX2RefineLoRA
dominates first-run cold start at 40%).
* Bump state.md "Last reconciled" header to mention D-22 + benchmark.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
|
||
|
|
2956ad9fbe |
[test] dreamverse: pipeline benchmark via Python SDK (per-stage timings)
Adds `apps/dreamverse/server/benchmarks/benchmark_pipeline.py` — mirrors how `apps/dreamverse/server/video_generation.py` constructs `GeneratorConfig` and calls `VideoGenerator.generate_video()`, captures per-stage timings via `FASTVIDEO_STAGE_LOGGING=1` (same mechanism as `FastVideo-internal/examples/inference/basic/basic_ltx2_distilled_i2v_two_stage_time.py`). Sweeps three scenarios by default: - cold (compile=False, warmup=False) - compile_cold (compile=True, warmup=False → JIT mid-run) - compile_warm (compile=True, warmup=True → production mode) Reports per scenario: wall_ms_median/p95, peak_gpu_mb, realtime_ratio, plus per-stage means (input_validation_stage, prompt_encoding_stage, ltx2_refine_init_stage, latent_preparation_stage, denoising_stage, ltx2_upsample_stage, ltx2_refine_lora_stage, ltx2_refine_denoising_stage, audio_decoding_stage, decoding_stage). Optional `--output-json` for structured persistence; exit code 1 if any scenario produces realtime_ratio < 1.0 (CI gate). Verified live on GPU 4 (B200, NVFP4, --scenarios cold --runs 2): cold run 1: 43.7s (cold model load + LoRA conversion dominate) cold run 2: 6.99s (steady state, NO compile, NO warmup → 0.72x realtime) per-stage breakdown surfaces LTX2RefineLoRAStage (40%), LTX2DenoisingStage (20%), LTX2TextEncodingStage (11%), DecodingStage (4%), with audio_decoding/upsample/init being sub-1% each. Pairs with `benchmark_av_streaming.py` (D-21 ffmpeg path benchmark): this script profiles the model side, that one profiles the muxing side. Together they fully attribute the segment wall-time budget. Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com> Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com> Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com> Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com> |
||
|
|
bc42ed1068 |
[feat] dreamverse: per-chunk timing instrumentation in stream_fmp4
D-22: extends the AV streaming hot path with per-phase + per-chunk
timing so the 1300ms gap between worker_e2e and main_user_step (per
D-21 root-cause analysis) becomes attributable.
* `av_streaming.stream_fmp4` now records into `timings`:
- `av_wav_write_ms` (audio WAV serialization)
- `av_ffmpeg_spawn_ms` (subprocess.Popen + pipe sizing)
- `av_first_chunk_ms` (cold-start latency to first chunk)
- `av_chunk_interval_ms_min/median/p95/max` (jitter between chunks)
- `av_chunk_publish_ms_median/p95` (publish callback cost)
- `av_chunk_read_ms_median/p95` (ffmpeg stdout read cost)
* `gpu_pool.handle_command` prints a one-line summary per segment so
these show up directly in the deploy log without log scraping:
"[GPU N] AV streamed segment X: encode_total=… wav_write=…
spawn=… first_chunk=… chunk_interval_med=…ms chunk_interval_p95=…
publish_med=… read_med=…"
Verified: `apps/dreamverse/server/av_streaming.py` syntax-checks; new
fields are populated unconditionally and harmless when unused (median
of empty list guarded by `if chunk_intervals_ms:`).
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
|
||
|
|
bc4c79ce10 |
[docs] dreamverse-integration: D-21 chunk-stutter root cause + NVENC path
* Add D-21 to decisions-log.md documenting the chunk-stutter investigation (3 parallel explore agents). Confirmed apps/dreamverse matches FastVideo-internal byte-for-byte on NVFP4 (layer_profile= refine, 48-block superset, e2m1+fp32 alpha+layout_128x4) and on torch.compile coverage (transformer + text_encoder; VAE/audio_vae/ vocoder are eager bf16 in all three repos). Stutter is NOT a migration regression — it's an architectural limit: software libx264 ultrafast on the streaming hot path consumes ~22% of segment wall-time (1300ms gap on a 5s playable segment). * Empirical: benchmark_av_streaming.py reports libx264 ultrafast at 611ms median / 644ms p95 for production-shape input — only 700ms short of the 1300ms gap, which means the rest is IPC + controller WS relay (D-22 follow-up). * B200 has no NVENC silicon — confirmed via direct ffmpeg probe (`OpenEncodeSessionEx unsupported device`). The build infra (install_native_ffmpeg.sh) and deploy infra (--nvenc flag with build+runtime validation) are correct and useful on hosts WITH NVENC (RTX 50-series / T4 / A10 / H100 PCIe), but moot on this dev host. Default codec stays libx264 for backward compat. * Update state.md branch tip and live-services context, README.md Last reconciled, open-threads.md (added D-22 chunk timing instrumentation, D-23 NVENC default flip per-SKU, D-24 B200 pipeline gen+encode architecture follow-ups). Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com> Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com> Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com> Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com> |
||
|
|
7640bc3869 |
[test] dreamverse: AV streaming benchmark for D-21 regression guard
* Add apps/dreamverse/server/benchmarks/benchmark_av_streaming.py.
Drives `av_streaming.stream_fmp4` directly with synthetic 121-frame
1920x1088 video + 5s 24kHz stereo audio (production shape).
* Sweeps codecs (default `libx264 h264_nvenc`); auto-skips encoders
not present in the active ffmpeg.
* Reports per-codec: wall_min/median/p95/max, bytes_median,
chunks_median, realtime_ratio_median (playable / wall, must be
>= 1.0 to avoid FE buffer drain). Exit code 1 if any codec runs
with realtime_ratio < 1.0; useful as a CI gate.
* Captured baseline on this dev host (B200, libx264 ultrafast,
121×1920×1088, 5 runs):
wall_median = 611ms, p95 = 644ms, realtime ratio = 8.25x in
isolation.
Production shows ~1300ms/segment wall time including IPC +
controller WS relay, so isolated ffmpeg encode is ~700ms below
the production gap.
* `h264_nvenc` benchmark on B200 fails with `OpenEncodeSessionEx
unsupported device` (datacenter Blackwell ships without NVENC
silicon — confirmed via direct ffmpeg probe). The benchmark
records the failure cleanly and skips per-codec stats so the
output stays comparable across hosts.
Cross-references the broader root-cause analysis recorded in
.agents/memory/dreamverse-integration/decisions-log.md D-21.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
|
||
|
|
407f9019be |
[feat] dreamverse-deploy: --nvenc/--no-nvenc CLI flag with runtime probe
* Add `--nvenc` / `--no-nvenc` CLI flag (defaults to off — preserves
current libx264 behavior). When `--nvenc`, the deploy exports
FASTVIDEO_VIDEO_CODEC=h264_nvenc to the backend setsid block;
otherwise libx264.
* `DREAMVERSE_NVENC` env var as the env-var counterpart (flag wins).
* Two-phase NVENC validation when --nvenc is requested:
1. Build-time: `ffmpeg -encoders | grep h264_nvenc` confirms the
binary was built with NVENC support; otherwise points at
install_native_ffmpeg.sh.
2. Runtime: minimal `ffmpeg -c:v h264_nvenc` probe against a 64x64
0.2s color frame confirms the GPU silicon actually has NVENC
hardware. On B200 / some H100 SKUs without NVENC, this fails
fast with a clear message and the alternative SKU list (RTX
50-series, T4, A10) instead of cryptic "OpenEncodeSessionEx
unsupported device" surfacing at first segment.
* Update [3/8] banner to print nvenc=true|false alongside warmup +
torch_compile.
* Capture-output-then-pattern-match (instead of `grep -q`) for the
encoder list check to avoid SIGPIPE-via-pipefail false negatives:
`set -euo pipefail` + `ffmpeg -encoders | grep -q h264_nvenc`
causes ffmpeg to receive SIGPIPE when grep exits early on first
match → pipefail propagates → if-branch fires "no NVENC build"
even when the encoder IS present.
* Document the new flag in SKILL.md alongside warmup and
torch-compile flags.
Verified: --nvenc on a B200 host fails fast with the NVENC-silicon
diagnostic and SKU-level alternatives. --no-nvenc and bare invocations
behave identically to the pre-flag version (same libx264 codec
exported, same banner format).
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
|
||
|
|
3858203620 |
[feat] dreamverse: NVENC build support in install_native_ffmpeg.sh
* Add nv-codec-headers prereq install (clones into $INSTALL_PREFIX, no sudo, registers ffnvcodec.pc on pkg-config search path). * Configure ffmpeg with --enable-cuda --enable-nvenc --enable-cuvid --enable-nvdec --extra-cflags=-I$CUDA_PREFIX/include --extra-ldflags=-L$CUDA_PREFIX/lib64. Drop --enable-cuda-nvcc and --enable-libnpp so the build does NOT require --enable-nonfree (we only need hardware encode/decode, not GPU-side filters). * New env knobs: ENABLE_NVENC=0|1 (default 1), CUDA_PREFIX (default /usr/local/cuda), NV_CODEC_REF (default master). Setting ENABLE_NVENC=0 reproduces the previous libx264-only build for hosts without CUDA toolkit. * Sanity check verifies the resulting binary exposes h264_nvenc / hevc_nvenc encoders before exiting 0. * Emitted ffmpeg-env.sh keeps FASTVIDEO_VIDEO_CODEC=libx264 — NVENC is opt-in at deploy time via dreamverse-deploy.sh --nvenc, not a build-time switch. Existing deploys are runtime-unchanged. Verified: rebuilt ffmpeg has libx264, h264_nvenc, hevc_nvenc encoders listed by `ffmpeg -encoders`. On hosts with NVENC silicon (RTX 50, T4, A10, H100 PCIe, etc.), `--nvenc` deploys are now possible. On hosts without NVENC silicon (datacenter Blackwell B200), the build still succeeds but runtime probe in the deploy script catches the mismatch (see follow-up commit). Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com> Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com> Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com> Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com> |
||
|
|
f10095d9b7 |
[fix] dreamverse: 3-pass warmup so user seg=1 doesn't re-compile stage 1
Empirical: with `--warmup --torch-compile`, the FIRST real user segment took ~92s for stage 1 even though warmup had already run. Stage 2 cache-hit (~3s); stage 1 re-compiled from scratch. Root cause: warmup r1 stage 1 compiles BEFORE `ltx2_refine_lora_stage` wraps all 1344 linear modules with LoRA. Once r1 stage 2 applies LoRA, the pre-LoRA module identities are stale and torch.compile guards reject the cached graph. Warmup r2 (seg=2 with continuation) compiles `stage1-LoRA-cont`, but the needed graph for user seg=1 is `stage1-LoRA-nocont` — never compiled by warmup. Add a 3rd warmup pass after r2: r3 = generate_step(seg=1, reset=True) # post-LoRA r3 stage 1 compiles the missing `stage1-LoRA-nocont` graph; r3 stage 2 hits r1's `stage2-LoRA-nocont` cache so the only real work is the missing stage 1 (~96s in practice). Verified end-to-end on GPU4: warmup_segment1_ms = 222665 warmup_segment2_ms = 212443 warmup_segment3_ms = 96432 ← new warmup_total_ms = 531542 (8.85 min, was 7 min) user seg=1: 4.44s (was 92s — 20x faster) user seg=2: 4.87s (unchanged, was already cache-hit) Net trade: +96s of operator-side warmup time, -88s of user-perceived first-segment latency. Worth it for any deploy serving real users. Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com> Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com> Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com> Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com> |
||
|
|
814433a935 |
[fix] dreamverse-deploy: scale /readyz polling budget with flags
Empirical: --warmup + --torch-compile takes ~7 min on B200 (warmup runs two synthetic segments, each triggering torch.compile max-autotune cold start). The supervisor's hardcoded 5-min /readyz poll deadline expired mid-warmup, the supervisor exited with code 5, and the frontend was never launched even though the backend ultimately reached /readyz=200 (BE survived because the backend launch is `setsid` + `disown`'d). Scale READYZ_BUDGET_SECONDS by mode: - both warmup + torch_compile: 900s (15min, observed worst case 7min) - either flag only: 480s (8min) - default (neither): 300s (5min, current behavior) Banner now prints the budget in [3/8] so operators see the expected wait time before the loop starts. Verified: --warmup --torch-compile reaches /readyz=200 at warmup_total_ms=421024, well within the new 900s budget. Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com> Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com> Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com> Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com> |
||
|
|
7bd54bf472 |
[refactor] dreamverse: wrappers exec conda env python directly
* `apps/dreamverse/scripts/dreamverse-server` and
`apps/dreamverse/scripts/dreamverse-mock-server` previously
hardcoded `${REPO_ROOT}/.venv/bin/python`, which on this dev node
is a symlink into `~/miniconda3/envs/fv-main/bin/python3.12`.
Switch both wrappers to read `DREAMVERSE_PYTHON` (default
`$HOME/miniconda3/envs/fv-main/bin/python`) so the cmdline that
shows up in `ps` / `/proc/$PID/exe` accurately names the conda
interpreter that owns the install — no symlink indirection.
* Add a fast-fail guard with a clear hint at the override env var
so a missing/renamed conda env produces an actionable error
instead of an opaque "exec format error".
* Propagate `DREAMVERSE_PYTHON` from the deploy script's prereq
probe into the setsid backend launch block so the wrapper sees
the same interpreter the deploy script just validated, regardless
of whether the operator set the env var as an exported variable
or only as a script-prefix override.
Verified live: redeploy on GPU 4 spawns `/home/william5lin/miniconda3/
envs/fv-main/bin/python main.py --host 0.0.0.0 --port 8009` (no
.venv in cmdline), `/proc/$PID/environ` shows
`DREAMVERSE_PYTHON=/home/william5lin/miniconda3/envs/fv-main/bin/python`
inherited from the deploy script's setsid block.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
|
||
|
|
82efc67655 |
[refactor] dreamverse-deploy: validate conda env python instead of .venv
* The deploy script's prereq probes for `flashinfer-python` previously
ran via `${REPO_ROOT}/.venv/bin/python`, which on this dev node is a
symlink into `~/miniconda3/envs/fv-main/bin/python3.12`. Switch the
probes to use the canonical conda interpreter directly so the
prereq error message accurately names the environment that owns
the install (`run: ... -m pip install flashinfer-python ...`).
* Add `DREAMVERSE_PYTHON` env override for non-default conda env
layouts; default stays `$HOME/miniconda3/envs/fv-main/bin/python`.
* Surface the resolved interpreter in the [1/8] banner alongside
the native-ffmpeg line so misconfigured envs are obvious from the
first lines of stdout.
* Update SKILL.md prereqs and env-overrides table to point at the
conda env path. Note that the launch wrapper at
`apps/dreamverse/scripts/dreamverse-server` still resolves python
via the `.venv` symlink (which points at the same interpreter on
this host); switching the wrapper too is an out-of-scope follow-up.
Verified: deploy runs clean, banner prints
`python: /home/william5lin/miniconda3/envs/fv-main/bin/python`,
backend reaches `/readyz=200`, no behavior change at runtime.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
|
||
|
|
a171d35cfd |
[test] dreamverse: long-running 2-segment Playwright spec for D-20 regression
* Add `apps/dreamverse/web/e2e/long-running-segments.spec.ts` — a Playwright spec that drives a real two-segment session through the FE, captures every WS frame, and asserts segments 1 AND 2 both reach `media_segment_complete` with at least one binary fMP4 chunk per segment. This is the canonical regression guard for D-20: if the public-API audio-kwarg routing block ever silently breaks again, this spec hard-fails with the WS error frame's actual ffmpeg/audio diagnostics rather than an opaque test timeout. * Skipped by default. Opt in with `PLAYWRIGHT_LONG_RUNNING=1`. The spec also asserts `/status.warmup_enabled === true` so a misconfigured deploy fails up-front instead of timing out at ~15 minutes. * Per-test timeout bumped to 900s. Polls every 2s for up to 850s for both segments to complete; raises immediately on any WS error frame; cross-references D-20 in the docstring so future contributors understand what the spec guards against. * Document the spec in dreamverse-deploy SKILL.md with the exact invocation paired with `--warmup --torch-compile`. Verified locally on GPU 4: spec compiles via `playwright test --list`, skips cleanly when `PLAYWRIGHT_LONG_RUNNING` is unset (8 passed + 1 skipped in 5.1s), opt-in path is documented but not run in this commit (~7-9min runtime, requires interactive validation of the emitted media chunks). Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com> Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com> Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com> Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com> |
||
|
|
6c62b1d0e5 |
[docs] dreamverse-integration: D-20 broken-pipe root cause + 4-commit fix
* Add D-20 entry to decisions-log.md documenting the segment-2 BrokenPipe root cause: a TWO-direction silent drop of LTX-2 audio kwargs in public `VideoGenerator`. Inbound, `SamplingParam.update()` rejected `audio_num_frames` / `ltx2_audio_clean_latent` / etc. as unknown fields and `logger.error`'d-and-continued; outbound, the result dict didn't surface `ltx2_audio_latents` from `output_batch.extra`. The two directions hid each other — even after the inbound block was ported, segment 2 still broke until the outbound surface was added. * Document the resolution: 4 commits on `will/dreamverse-monorepo` (`1b686f4e`..`5eaf0a13`) covering ffmpeg toolchain pin, public-API audio kwarg routing + strict `update()`, native-ffmpeg + compile-off deploy defaults, and `--warmup` / `--torch-compile` CLI flags. End- to-end verified on GPU4 with `Cached audio latents shape=(1, 8, 126, 16) for segment 2` and `Segment 2: relayed av chunks=22, bytes=3.8MB`, no BrokenPipeError. * Cross-reference D-11 (Apr 26 cosmetic note about ffmpeg broken pipe on intentional client disconnect) — different code path; both resolved. * Update state.md: `will/dreamverse-monorepo` tip moves to `5eaf0a13` (11 commits ahead of `fbd823df`); refresh live-services PIDs to the post-D-20 deploy (BE 705513, FE 707746, both with native ffmpeg + `ENABLE_TORCH_COMPILE=0`); refresh test-status to include the new `test_extra_overrides_routing.py` (7/7 pass) and the live segment-1→2 audio continuation evidence. * Add D-20-CP follow-up to open-threads.md: cherry-pick `265ce1a6` onto `will/ltx2_sr_port` so the public-API fix lands in PR #1288 rather than waiting for `will/dreamverse-monorepo` to fold in. * Bump README.md "Last reconciled" header to mention D-20. Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com> Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com> Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com> Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com> |
||
|
|
f62c66916e |
[feat] dreamverse-deploy: --warmup / --torch-compile CLI flags
* Add `--warmup` / `--no-warmup` and `--torch-compile` / `--no-torch-compile` CLI flags so callers don't have to remember the `DREAMVERSE_*=true` env-var spelling for the two knobs they flip most often. * Replace the ad-hoc first-arg-is-positional parser with a proper flag-aware loop that accepts flags in any position (before, after, or between the positional GPU / port args). * Flag values win over env vars when both are set; env-only callers still work unchanged. `-h` / `--help` now renders a unified usage. * Update SKILL.md with the new flag table and example invocations alongside the env-var table. Verified end-to-end: `dreamverse-deploy.sh --no-warmup --no-torch-compile 4` reaches `/readyz=200`; `/proc/$BE_PID/environ` shows `ENABLE_TORCH_COMPILE=0` and `FASTVIDEO_ENABLE_STARTUP_WARMUP =false`. Standalone parser test covers 13 flag/env permutations (defaults, single flags, both flags, --no-* overrides, env-only, flag-overrides-env, both-env+both-no-flags, flags interleaved with positional args at every position) plus bad-flag rejection. Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com> Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com> Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com> Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com> |
||
|
|
b0cf3269cb |
[feat] dreamverse-deploy: native ffmpeg + compile-off defaults
* Export `FASTVIDEO_FFMPEG_BIN=$HOME/opt/ffmpeg-native/bin/ffmpeg` and `FASTVIDEO_VIDEO_CODEC=libx264` inside the backend setsid block when the native binary exists. Belt-and-suspenders: even without sourcing `apps/dreamverse/scripts/ffmpeg-env.sh`, deploys pick up the LTO+libx264+native-arch ffmpeg the team playbook mandates. * Detect the native binary up front and warn (or fail-fast under `DREAMVERSE_REQUIRE_NATIVE_FFMPEG=true`) when absent, with a pointer at `apps/dreamverse/scripts/install_native_ffmpeg.sh`. Print the binary path in the [8/8] summary so operators see at a glance which ffmpeg the live worker actually has. * Disable `torch.compile` by default (`ENABLE_TORCH_COMPILE=0`). Max-autotune compile spends ~3-4 minutes on the first segment, which pushes simple e2e cycles past the 5-minute session timeout and makes iterative debugging painful. Subsequent segments save only ~3s vs no-compile, so compile is rarely worth its cold-start cost during development. Re-enable with `DREAMVERSE_TORCH_COMPILE=true` for benchmarking. * Document the new `DREAMVERSE_TORCH_COMPILE` and `DREAMVERSE_REQUIRE_NATIVE_FFMPEG` knobs in SKILL.md. * Gitignore `apps/dreamverse/scripts/ffmpeg-env.sh` — generated by `install_native_ffmpeg.sh` with host-specific absolute paths. Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com> Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com> Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com> Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com> |
||
|
|
dc3a62e0f2 |
[fix] api: route LTX-2 audio kwargs through batch.extra; strict update
* Public `VideoGenerator.generate_video(**kwargs)` was silently dropping
every LTX-2 audio-conditioning kwarg (`audio_num_frames`,
`ltx2_audio_clean_latent`, `ltx2_audio_denoise_mask`,
`video_position_offset_sec`, `ltx2_audio_latents`).
`SamplingParam.update()` `logger.error`'d on unknown fields and
continued, so `apply_audio()` for continuation segments was
effectively a no-op: audio was generated for the default num_frames
duration and `ltx2_audio_latents` was never carried forward across
segments.
* The downstream symptom on Dreamverse was a confusing ffmpeg
"frame writer failed: [Errno 32] Broken pipe" on segment 2: trimmed
audio ended ~1.7s before trimmed video, ffmpeg `-shortest` closed
stdin, the writer thread's next `proc.stdin.write(...)` hit EPIPE.
* Port the FastVideo-internal routing block into public:
- `_BATCH_EXTRA_PASSTHROUGH_KEYS` enumerates the 5 LTX-2 audio kwargs.
- `_generate_video_impl` extracts them from `**kwargs` BEFORE
`sampling_param.update()` and stashes as `kwargs["_extra_overrides"]`.
- `_generate_single_video` pops `_extra_overrides` and copies into
`batch.extra` so `ltx2_denoising` / `ltx2_refine` see them.
- Surface `output_batch.extra["ltx2_audio_latents"]` in the result
dict so `Dreamverse._derive_next_audio_latents` can carry latents
across segments.
* Convert `SamplingParam.update()` to raise `ValueError` on unknown
keys instead of `logger.error`-and-continue. Future contributors who
add a pipeline-specific kwarg without registering it will hit a loud
failure pointing at `_BATCH_EXTRA_PASSTHROUGH_KEYS` rather than
debug a broken-pipe-shaped symptom hours later.
* Add `fastvideo/tests/api/test_extra_overrides_routing.py` (7 tests)
pinning the passthrough-key set, the strict-update contract, and the
error-message hint at the right routing mechanism.
Verified end-to-end on GPU 4 with the dreamverse-deploy skill: segment 2
streams 22 av chunks / 3.8MB clean, no BrokenPipeError, `Cached audio
latents shape=(1, 8, 126, 16)` log line confirms latents now carry.
185/185 fastvideo/tests/api/ tests pass.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
|
||
|
|
eac43304ac |
[fix] dreamverse: pin ffmpeg native build toolchain by uname -m
* The conda fv-main env activates BOTH gcc_linux-64 and gcc_linux-aarch64
triplets on every `conda activate` (alphabetical order makes aarch64
win), so on an x86_64 host the inherited `CC=aarch64-conda-linux-gnu-cc`
defeated the script's `: "${CC:=...}"` deferred default and tripped
x264's compiler probe with "unknown value 'native' for '-march'".
* Replace deferred defaults with unconditional assignment per
`uname -m`. Matches the team playbook's `export CC=...` semantics
verbatim.
* Print the toolchain banner up front so future probe failures are
diagnosable from the first lines of stdout.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
|
||
|
|
12085f1286 |
[refactor] dreamverse: use public health routes via build_app
Wire Dreamverse to the public streaming health router while keeping only the product-specific internal monitor route local. Co-authored-by: Junda Su <junda@berkeley.edu> Co-authored-by: Matthew Noto <matthewnoto@gmail.com> Co-authored-by: XOR-op <xor-op@users.noreply.github.com> Co-authored-by: Zhang Peiyuan <peiyuan_zhang@berkeley.edu> |
||
|
|
cb082d8bbb |
[feat] streaming: promote /healthz /readyz /status to public build_app
Promote the Dreamverse-compatible health, readiness, and status probes into the public streaming package so integrators can compose them through build_app without product-local shims. Co-authored-by: Junda Su <junda@berkeley.edu> Co-authored-by: Matthew Noto <matthewnoto@gmail.com> Co-authored-by: XOR-op <xor-op@users.noreply.github.com> Co-authored-by: Zhang Peiyuan <peiyuan_zhang@berkeley.edu> |
||
|
|
02ab3e45cd |
[feat] dreamverse-deploy skill + docs prereq additions
Adds an executable skill that takes a GPU index and (re)deploys the
migrated apps/dreamverse backend + Next.js frontend cleanly. Also extends
the contributing doc with the additional Phase 0 prereqs Oracle flagged
during the post-migration verification pass.
* .agents/skills/dreamverse-deploy/SKILL.md (new): documents the skill's
contract — args, env overrides, prereqs, what it does and does NOT do,
teardown, and the e2e wrapper command for follow-up Playwright runs.
* .agents/skills/dreamverse-deploy/scripts/dreamverse-deploy.sh (new):
- Validates prereqs upfront (apps/dreamverse, wrapper script, .venv,
flashinfer-python, pnpm, gcc-13/g++-13).
- Kills processes by port pattern AND by GPU UUID via
`nvidia-smi --query-compute-apps`. The earlier naive port-only kill
missed orphan multiproc_executor worker subprocesses, which then OOM'd
the next deploy on the same GPU.
- Sources ~/.env for provider API keys, exports the full env recipe
(CUDA_VISIBLE_DEVICES, FASTVIDEO_*, gcc-13 + NVCC_PREPEND_FLAGS for
flashinfer JIT against gcc-15 conda toolchain on B200/sm_100a).
- Launches backend via apps/dreamverse/scripts/dreamverse-server in a
detached setsid session, polls /readyz up to 5 min.
- Launches frontend via pnpm run dev:devtools (or `next dev --port` for
non-default port) with NEXT_PUBLIC_INCLUDE_DEVTOOLS=1, polls / up to
60s.
- Prints URLs, PIDs, log paths, stop command, e2e command at end.
- Supports `--stop` (port-pattern teardown) and `--stop <GPU>` (also
nukes any process holding that GPU).
* docs/contributing/dreamverse-development.md: Phase 0 prereqs section
extended per Oracle review (D-19 follow-up):
- Adds cerebras-cloud-sdk + openai to the install command (required by
the migrated prompt enhancer; without them, server fails to import
provider clients).
- Documents the gcc-13 / NVCC_PREPEND_FLAGS workaround for B200 hosts
with gcc-15 conda toolchain. Without these, flashinfer JIT compilation
fails at first NVFP4 use.
- Cross-links to the new dreamverse-deploy skill as the recommended
local-deploy path.
Verified: skill ran end-to-end on this dev node (GPU 4, ports 8009/5274),
backend boots /readyz=200 with /proc/$PID/cwd = apps/dreamverse/server,
frontend serves 200 on :5274, all 8 Playwright e2e specs pass against the
migrated stack in 5.1s.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
|
||
|
|
dd9a8f6aae |
[docs] dreamverse-integration: D-19 — migration executed on will/dreamverse-monorepo
Records the actual landing of D-18's Option B+ plan as 5 commits on will/dreamverse-monorepo (08828d96, f3a863ba, 876f7eb3, 1d47ede6, c1fe5d4c). 164 files, +53,294 LOC, 31,725 files under apps/dreamverse/. Significant deviation from integration-plan.md surfaced by Oracle review: the plan's Phase 2 prescription "DELETE generic-merged files (gpu_pool.py, av_streaming.py, worker_ipc.py, mock_server.py, session_init_image.py, session_logger.py) and rewire imports to public fastvideo.entrypoints. streaming.*" was INVALID — public APIs are not drop-ins for the Dreamverse product modules. Public `GpuPool` is an abstract base class with acquire/run/release/shutdown/health semantics; Dreamverse's product GPUPool has totally different shape (per-GPU subprocess workers with slot.user_step / register_stream_queue / acquire(client_id, websocket)). Substituting one for the other made apps/dreamverse/server/main.py:63 fail at boot with `TypeError: GpuPool() takes no arguments`. Fix-up commits (1d47ede6, c1fe5d4c) carried all 7 generic-merged files into apps/dreamverse/server/ as PRODUCT-LOCAL, reverted import rewrites to flat `from gpu_pool import GPUPool` style (matching Dreamverse's existing sys-path-injection convention), and reverted the public-surface deviations introduced in Phase 2. Verification: `git diff fbd823df.. c1fe5d4c -- fastvideo/entrypoints/streaming/` returns empty. Also documented: Phase 0 prerequisites for B200 / sm_100a + gcc-15 conda toolchain. flashinfer JIT requires CUDAHOSTCXX=/usr/bin/g++-13 and NVCC_PREPEND_FLAGS="-ccbin /usr/bin/gcc-13 -allow-unsupported-compiler" to bypass the nvcc gcc-version check. Without these, NVFP4 path fails to compile. E2E evidence (post-fix-up): 8/8 Playwright tests pass in 5.1s against the migrated backend with /proc/$PID/cwd = apps/dreamverse/server confirmed. GPU4 holds 50.9 GiB (NVFP4 model loaded). All routes (/healthz, /readyz, /status, /prompt-system-config, /curated-presets) return 200. * state.md: branch tip table adds will/dreamverse-monorepo @ c1fe5d4c with 5-commit migration summary + e2e evidence reference. * decisions-log.md: full D-19 entry with commit table, Oracle false- positive narrative (initial e2e run hit canonical Dreamverse install via legacy console_script, NOT migrated code; fix-up wrapper scripts resolve this), Phase 0 environment prereqs, e2e evidence dump, implications for the integration-plan.md (Phase 2 prescription invalidated), open follow-ups. * README.md: Last-reconciled bumped with D-19 reference. Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com> Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com> Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com> Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com> |
||
|
SolitaryThinker
|
0e0af68f01 |
[fix] dreamverse-monorepo: entrypoint + audio re-encode + verification
Make the repo-local wrapper scripts the canonical Dreamverse launch path while package=false defers proper console-script packaging. Restore the bounded audio re-encode path from the product backend and document the verification prerequisites used for the migrated server boot probe.\n\nCo-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>\nCo-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>\nCo-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>\nCo-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com> |
||
|
SolitaryThinker
|
aef113a861 |
[fix] dreamverse-monorepo: carry generic-merged files product-local
Source: Dreamverse repo at will/integrate-public-fastvideo @ ec8ef92.\n\nCarry the generic-merged backend modules as Dreamverse product-local code because the public streaming APIs are not drop-in compatible with the product runtime yet. Revert the Phase 2 public compatibility shims so FastVideo's public streaming surface returns to the pre-migration state.\n\nCo-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>\nCo-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>\nCo-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>\nCo-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com> |
||
|
|
9c2da7c2b0 |
[feat] dreamverse-monorepo: Phase 3 — frontend + public assets
Source: Dreamverse repo at will/integrate-public-fastvideo @ ec8ef92. Cross-repo move; original history retained in archived Dreamverse repo. Per .agents/memory/dreamverse-integration/integration-plan.md Phase 3. * Frontend moved to apps/dreamverse/web with Next/Vitest/Playwright config rewired for the monorepo location. * Product docs, scripts, launch helpers, prompt assets, and Dreamverse diagrams/assets moved under apps/dreamverse/. * Public streaming demo config added at examples/serving/streaming_demo.yaml with non-private placeholders. * Launch/smoke scripts now resolve DREAMVERSE_ROOT from apps/dreamverse and run frontend commands from apps/dreamverse/web. * Dreamverse asset allow-list narrowed in .gitignore so copied image/video assets can be staged without broadly tracking generated build output. * Vitest excludes Playwright e2e specs; obsolete page.integration assertions remain skipped until updated for the current migrated shell. Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com> Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com> Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com> Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com> |
||
|
|
86c8389cc9 |
[feat] dreamverse-monorepo: Phase 2 — backend move + import rewires
Source: Dreamverse repo at will/integrate-public-fastvideo @ ec8ef92.
Cross-repo move; original history retained in archived Dreamverse repo.
Per .agents/memory/dreamverse-integration/integration-plan.md Phase 2.
* PRODUCT files moved to apps/dreamverse/server/: main.py, config.py,
video_generation.py, session/controller.py, session/messages.py,
routes/presets.py + 5 product test files + product prompts.
* GENERIC-PENDING shims carried (replaced in Phase 4/5): runtime.py,
utils.py, prompt_safety.py, rewrite_prompt_payload.py,
routes/health.py, prompt_enhancer.py.
* Imports rewritten: gpu_pool/worker_ipc/session_init_image/session_logger
now import from fastvideo.entrypoints.streaming.* public modules.
* Public streaming compatibility aliases added for the migrated product
controller: GPUSlot/get_available_gpus, MediaChunk/MediaComplete,
cleanup_session_init_image, and SessionEventLogger.
* Path rewrites: _APP_ROOT/"apps"/"web" -> _APP_ROOT/"web" in config.py.
* GPU marker: test_realtime_stress.py and test_session_logging.py
marked @pytest.mark.gpu (skip-by-default in CI).
* New: apps/dreamverse/server/tests/test_import_contract.py — AST gate
forbidding fastvideo.{pipelines,models,layers,worker,fastvideo_args}.
Generic-merged files NOT copied (already on public FastVideo):
gpu_pool.py, av_streaming.py, worker_ipc.py, mock_server.py,
session_init_image.py, session_logger.py.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
|
||
|
|
6d925d5333 |
[feat] dreamverse-monorepo: Phase 1 — skeleton + tooling
Establish the apps/dreamverse/ subfolder + monorepo tooling per
.agents/memory/dreamverse-integration/integration-plan.md Phase 1.
No Dreamverse content moved yet — that's Phase 2/3.
* Add apps/dreamverse/{README.md, pyproject.toml, server/__init__.py,
server/tests/__init__.py}.
* Add docs/contributing/dreamverse-development.md with local dev commands.
* Root pyproject.toml: add [tool.uv.workspace] members = ["apps/dreamverse"];
add apps* to setuptools exclude; add gpu pytest marker.
* .pre-commit-config.yaml: exclude ^apps/dreamverse/web/.
* .gitignore: scope Node artifacts + unignore product assets.
* uv.lock: regenerated by `uv lock` to include the new workspace member.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
|
||
|
|
cbac854582 |
[docs] dreamverse-integration: GPU4 smoke validated; flashinfer prereq
Live smoke test on GPU4 confirms public fastvideo serve from will/ltx2_sr_port HEAD
|
||
|
|
996df583b9 |
[docs] dreamverse-integration: align test strategy with B+ + GPU4 hook
Audit found the test strategy in integration-plan.md was partially stale from the prior Option D / split-PR drafting and missing an explicit local-GPU verification path. This commit reconciles testing with the single-mega-PR (D-17) + Option B+ monorepo (D-18) reality and makes the GPU4-on-this-node hook explicit. * New "Test strategy" subsection (added before "File-by-file migration map"): test taxonomy table (unit / integration-with-fakes / live-GPU / FE build / FE Playwright / contract / SSIM), pytest marker scheme (`@pytest.mark.gpu`), pyproject.toml addopts to default-skip GPU tests in ubuntu-latest CI, and a per-phase responsibility matrix (which class runs in which gate per Phase 0..7). * GPU4 local verification hook documented: this dev node has 8x B200; GPU4 currently held by Dreamverse-side server PID 2453227 / port 8009; concrete kill / redeploy / smoke-test / cleanup commands so any phase touching the live-service path (Phase 2/3/4/5) can be validated locally without waiting for Buildkite-Modal. * Phase 2 verification gate: `uv run` command upgraded to `--locked --extra test pytest ... -m 'not gpu'` (matches Phase 1 `--locked` enforcement). Added explicit MANUAL GPU4 QA gate required, not optional: kill PID 2453227, deploy `fastvideo serve` on GPU4 from this branch, hit `/health`, run `pytest -m gpu` against the live deploy, capture output in PR. Rollback expanded to handle GPU4-discovered bugs. * Phase 3 verification gate: Playwright in CI is now explicitly DEFERRED to Phase 4 (matches `ci-dreamverse-frontend.yml` scaffold which already comments out Playwright steps until health routes land). Removed the contradictory "or PR note explains" clause. Added recommended manual GPU4 Playwright `frontend-shell.spec.ts` against the Phase 2 GPU4 deploy. Same `--locked --extra test` upgrade for the backend regression check. * Phase 4 verification gate: Frontend CI Playwright RE-ENABLED at end of this phase (the natural re-baseline once health routes land). Manual GPU4 full E2E for all 3 Playwright specs (backend-health + frontend-shell + preset-prompt-generation) is required to merge. * OSS precedent table: clarified that chainlit's "split CI" is a per-language CI split (frontend vs backend workflows), not a PR-level split — independent of D-17's single-mega-PR decision. Removes any reading-confusion that the testing strategy carries forward from the abandoned split. No claims, references, or commands tied to the abandoned split-PR branches (`will/api_7.10`, `will/api_8`, `will/ltx2_sr_runtime`, `will/ltx2_nvfp4`, `will/ltx2_post_fixes`, `will/agents_cleanup`) remain in the plan doc. All test commands route through `apps/dreamverse/server/` and `apps/dreamverse/web/` per the B+ layout. PR #1288: https://github.com/hao-ai-lab/FastVideo/pull/1288 Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com> Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com> Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com> Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com> |
||
|
|
8b198bbcca |
[docs] dreamverse-integration: D-18 — Option B+ monorepo plan
User decision after reviewing integration-review.md's Option D recommendation. Combines Option D's "generic backend stays generic" principle with Option B's monorepo subfolder layout for the Dreamverse product. Result: single repo (FastVideo), Dreamverse becomes apps/dreamverse/, generic backend stays at fastvideo.entrypoints.streaming.*, Dreamverse repo gets archived after migration. Methodology: 3 parallel exploration agents (FastVideo build/CI surface, Dreamverse product-vs-generic file split, Python+Next.js monorepo precedents). Synthesis authored by ultrabrain category. Oracle review applied as PASS-WITH-FIXES; 10 critical issues addressed (import contract too tight, Phase 2 missing shim list, prompt_enhancer misclassified, frontend CI Playwright/pnpm setup broken, .gitignore product-asset issue, uv.lock missing, backend CI path filters too narrow, cross-repo history claim wrong, Phase 6 too coarse, missing security/CORS risks). * integration-plan.md (new, 1205 lines): executable 7-phase migration plan with file-by-file map, tooling stack (uv workspace + standalone pnpm), pyproject.toml/pre-commit/.gitignore/CI YAML diffs, risk register, verification gates, rollback. Phase 0 lands #1288; Phases 1-7 add skeleton, move backend, move FE, promote generic-pending, retire prompt enhancer fork (DR-1), CI/release cutover (split into 6a-6f), archive Dreamverse repo. * integration-review.md: DEPRECATED banner added at top pointing to integration-plan.md. Body kept for drift-audit and OSS precedent reference (still authoritative). Reading-guide entry updated. * decisions-log.md: D-18 entry capturing strategy decision, rationale (drops cross-repo coordination overhead from D-17 cycle; preserves architectural separation; OSS precedents support shape), why not A/B/C/D (concrete reasons for each), implications (apps/ exclude in setuptools, no cross-repo history preservation, drift items fold into phases), and 4 deferred decisions (DR-2, VPO, history-import method, CORS/write-endpoint security). * README.md: Last-reconciled bumped with D-18 reference. Reading- guide table updated: integration-plan.md is CURRENT; integration-review.md marked DEPRECATED. Verified: pre-commit clean (memory dir excluded from yapf/ruff/mypy; spaces-check passes). Oracle PASS-WITH-FIXES, all 10 critical fixes applied (import contract relaxed to allow fastvideo.{api,configs, entrypoints.streaming,entrypoints.video_generator}; Phase 2 lists 6 explicit shims for generic-pending; Phase 5 explicitly retires the prompt fork; ci-dreamverse-frontend.yml uses pnpm/action-setup BEFORE setup-node; Playwright deferred to Phase 4 until health routes land; .gitignore unignores product assets; Phase 1 commits uv.lock with --locked enforcement; backend CI watches fastvideo/api + streaming + video_generator + configs; cross-repo history disposition documented; Phase 6 split into 6a-6f substeps with independent verification gates; CORS/write-endpoint risks added to register). PR #1288: https://github.com/hao-ai-lab/FastVideo/pull/1288 Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com> Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com> Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com> Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com> |
||
|
|
2f7f0cce5a |
[docs] dreamverse-integration: drift audit + integration tradeoff doc
Comprehensive review against ../Dreamverse and ../FastVideo-internal
plus 4-option integration tradeoff analysis with recommendation.
Driven by user request to (1) verify will/ltx2_sr_port has not drifted
from the integration goal, and (2) analyze whether Dreamverse should
remain a separate repo, become a subfolder, or merge into FastVideo's
namespace entirely.
Methodology: 4 parallel exploration agents mapped Dreamverse repo
structure, FastVideo-internal residual, FastVideo public cross-repo
surfaces, and OSS service-in-monorepo precedents (vLLM, BentoML, Ray
Serve, TGI+ChatUI, Transformers.js, ComfyUI, AUTOMATIC1111). Synthesis
authored by ultrabrain category. Oracle review applied as
PASS-WITH-FIXES; critical issues (broad zero-drift wording, stale Dynamo
mapping citation, broken Ray Serve URL, missing D-8/VPO/D-12-A/D-12-B/
SBS coverage) addressed in this commit.
* integration-review.md (new, 863 lines): Part 1 drift audit, Part 2
integration tradeoffs, Part 3 action items.
Part 1 — Drift audit
- Methodology: typed-public-boundary criterion, intentional refactor
vs drift, evidence sourced from worktree files + memory dir.
- Zero core typed API drift on the integration surface — but the
realtime-runtime contract surface (health routes) IS real drift.
- 8 numbered drift items (Dreamverse README/bootstrap, prompt
enhancer fork, cerebras_ifm gap, health routes, layerwise offload,
upsampler CLI, missing example config, LTX-2 stage equivalence
verification gap).
- 6 deferred/accepted residual items.
- 17-row drift summary table with priority / effort / status /
tracked-where / next-action columns. Includes D-8 (ltx2_image_crf
flow), VPO (video_position_offset_sec semantics — overdue),
D-12-A (GpuPool docstring), D-12-B (run_async migration), SBS
(session/blob lifecycle).
Part 2 — Integration path tradeoffs
- Option A: status quo (Dreamverse separate, depends on fastvideo).
- Option B: Dreamverse as subfolder under FastVideo.
- Option C: full merge into fastvideo.entrypoints.dreamverse.*.
- Option D: hybrid — backend merges, frontend stays separate.
- Comparison matrix on 13 axes.
- 7 OSS precedent rows with citations (vLLM, BentoML, Ray Serve,
TGI, Transformers.js, ComfyUI, AUTOMATIC1111).
- Recommendation: Option D (constrained) — backend merges as
generic FastVideo streaming, frontend stays separate. Mirrors
ComfyUI's late-stage frontend split. Strong precedent: TGI +
ChatUI. Critical librarian finding: NO 1:1 precedent exists for
"Python ML library + Next.js product merged into library
namespace" — argues against Option C.
- Conditions that would change the recommendation enumerated.
Part 3 — Action items: phased migration sketch (phases 0-5) with
concrete file paths, effort estimates, and cross-references to
open-threads.md / decisions-log.md.
* README.md: register integration-review.md in deep-dive reading guide.
Bump Last reconciled to
|
||
|
|
ec3fe68b5d |
[chore] remove STACK.md — split-PR plan abandoned per D-17
The 10-PR / 6-remaining-slice split tracked by STACK.md is no longer the landing strategy. Per [decisions-log.md D-17](.agents/memory/dreamverse-integration/decisions-log.md#d-17), the remaining `will/ltx2_sr_port` content ships as a single mega-PR (#1288) instead of 6 stacked PRs. The bulk-rebase formula and split bookmark table in STACK.md no longer reflect reality, so keeping the file in the repo would mislead future agents/contributors. Already deleted: * PR #1287 (was first slice of split) — closed * origin/will/api_7.10 — remote branch deleted * Local split bookmarks (will/api_7.10, will/api_8, will/ltx2_sr_runtime, will/ltx2_nvfp4, will/ltx2_post_fixes, will/agents_cleanup) — deleted Kept: * CO-AUTHORS.md — still the canonical roster reference (mirrored in .agents/memory/dreamverse-integration/authors.md but the top-level file is the source of truth for the broader stack history). * will/ltx2_sr_port-pre-1286-rebase — local safety backup of the pre-rebase chain, scheduled for deletion after #1288 merges. Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com> Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com> Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com> Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com> |
||
|
|
4077a1649e |
[docs] dreamverse-integration: D-17 — abandon split, single mega-PR #1288
User decision after the post-#1286 rebase + re-slice cycle. Closing #1287 (slice 1-3); opening #1288 on will/ltx2_sr_port covering the full 34-commit chain (71 files, +13,074/-583 LOC) at once. STACK.md model deprecated; split bookmarks no longer maintained. * decisions-log.md: add D-17 with rationale (review-coordination overhead vs structural benefit, layers not actually independent, CI/merge-queue simplicity), implications (STACK.md deprecated, bookmarks stale, runbook protocol replaced), and watch-outs (PR size, fallback to re-split if main moves significantly). Bumps Last updated header. * README.md: switch narrative — single mega-PR #1288 on will/ltx2_sr_port @ |
||
|
|
86a094a418 |
[docs] dreamverse-integration: reconcile post-#1286 merge + 7.10 split
PR #1286 merged at |
||
|
|
2ebb1b28f3 |
[docs] dreamverse-integration: add runbook + fresh-context onboarding
Make the memory dir self-sufficient as an onboarding package — a fresh agent should be able to resume work end-to-end (verify, commit, push, propagate to PR #1286) using only files in this directory. * runbook.md (new): operational how-to. Covers worktree contract, branch topology, verification (pre-commit binary path, pytest, lsp_diagnostics, gh CLI), commit workflow (subject/body conventions, required co-author trailers, the multi-`-m` trailer-parsing trap and its `-F` workaround), push + PR propagation (cherry-pick from ltx2_sr_port to api_7.9 to avoid force-push), memory-dir maintenance table, common pitfalls (pre-commit binary location, stash@{0} not ours, AbsMaxFP8 pre-existing failure, untracked nested clones, live ports, branch-switch by other agents, force-push policy, two known trailerless commits in PR #1286), 8-question self-test, and a "first 60 seconds" copy-paste orientation block. * README.md: replace single "Reading guide" with a two-tier structure. New "Fresh-context onboarding (read in order)" section gives 5 ordered steps for an agent picking up the work for the first time — worktree confirmation via runbook, then state.md, pr-roadmap.md, open-threads.md, runbook.md end-to-end, finishing with the runbook self-test as a context-loaded check. Original table preserved as "Deep-dive reading guide" with a new entry for runbook.md. Bumps "Last reconciled" header to 2026-05-05 / `09647a30` and updates the PR status line (PR #1284 merged; PR #1286 open at `a152cb77`, MERGEABLE). * state.md: bump "Current State" header from 2026-05-03 to 2026-05-05. Branch-tips table refreshed — `will/ltx2_sr_port` now at `09647a30` (was `156103b9`), 32 commits ahead of i2v base; new row for `will/api_7.9` (PR #1286 head `a152cb77`) noting it's an ancestor of the working branch. Added cross-links to runbook.md and authors.md in the front matter, plus a note about other agents sharing the worktree (covers the branch-switch scenario without inventing reconciliation logic). Verified: pre-commit clean (memory dir is yapf/ruff/mypy excluded; only spaces-check runs). All cross-links resolve to siblings in this directory or to top-level CO-AUTHORS.md / STACK.md / AGENTS.md as appropriate. Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com> Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com> Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com> Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com> |
||
|
|
eae529a22d |
[docs] dreamverse-integration: add authors.md + track D-16 router polish
* authors.md (new): dreamverse-integration-scoped mirror of the top-level CO-AUTHORS.md. Documents the 4 human co-authors credited on every commit in the integration scope, verified against PRs #1257/#1258/#1284/#1286. Self-contained so the memory dir is discoverable without traversing to the repo root. Cross-references CO-AUTHORS.md and STACK.md. Includes a 'Known gaps' section flagging that commits a152cb77 / 40e265b8 (the [fix] streaming: router polish pair from earlier in this session) are missing trailers and need an amend + force-push to fix. * README.md: register authors.md in the reading guide table. * decisions-log.md: add D-16 — Streaming router polish round 2 — capturing the 5 second-pass fixes applied on top of D-15's pre-merge polishes (bridge cancellation hygiene, registry UNKNOWN->HEALTHY immediate, httpx hard-fail, URL path/query/fragment/duplicate validation, per-index YAML parser errors, explicit websockets [streaming] dep, +7 test cases). Bumps Last updated header. * open-threads.md: bump Last updated header to mention the second-pass router commit and link to D-16. Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com> Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com> Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com> Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com> |
||
|
SolitaryThinker
|
8596c072ab |
[docs] dreamverse-integration: track D-15 + #13/#14/#15 — PR #1286 arch review
Adds D-15 to decisions-log.md: Oracle review of PR #1286 streaming router. Verdict — keep current shape (Alt A): in-repo at fastvideo/entrypoints/streaming/router/, FastAPI-based, single-primary failover, lazy httpx/websockets imports. Reject Alt B (separate package), Alt C (fold into server), Alt D (delegate to nginx/envoy/HAProxy as the sole answer), Alt E (sticky now), Alt F (round-robin now). Captures the 4 gemini review fixes: * High: select() docstring claimed round-robin but always returns [0] — rewrote with explicit MVP semantics * Medium: per-probe httpx.AsyncClient prevented connection reuse — refactored run_health_check_loop to share one client via _build_default_probe() async context manager * Medium: sequential probes could fall behind interval — now use asyncio.gather for parallel per-cycle probes * Medium: parser duplication — kept manual mapping intentionally because YAML has nested health_check: block while RouterConfig is flat; restructuring is beyond this PR's scope Plus 1 Oracle pre-merge polish: * RouterConfig.__post_init__ validation — empty replicas, non-positive intervals/timeouts, thresholds < 1, non-http(s) URLs, and >1 primary all raise ValueError so misconfigurations surface at config-load instead of confusing runtime failures Plus 1 prep-rebase item carried in: * Migrated FastAPI @app.on_event(startup/shutdown) to a single @contextlib.asynccontextmanager-based _lifespan() handler (deprecated API in newer FastAPI; was the 7.9 caveat in pr-roadmap.md) Adds open-threads.md items #13/#14/#15: * #13: sticky session routing extensibility (forward-compat from D-15) * #14: bridge backpressure note for high-scale deployments * #15: multi-primary semantics if active-active becomes required |
||
|
SolitaryThinker
|
952e0da232 |
[docs] dreamverse-integration: track D-14 + #12 — PR #1284 arch review
Adds D-14 to decisions-log.md: Oracle review of PR #1284 streaming auxiliaries. Verdict — keep current shape (Alt A): single PR, 4 modules under streaming/, mock_server in production module path, concrete PromptSafetyFilter. Reject Alt B (split into 4 PRs), Alt C (move mock to tests/), Alt D (premature observability extraction), Alts E/F (premature Protocol-ization). Captures the 4 gemini review fixes (high: session_logger race vs close; medium: rewrite regex in hot path, safety _ensure_loaded race, streaming extra missing prompt-safety) and 2 pre-merge polishes from Oracle (remove inert RewriteOptions.user_system_prompt_override field, sanitize session_id for filename safety as defense-in-depth). Adds open-threads.md item #12 (Low): when streaming server starts using PromptSafetyFilter, ensure operator-visible logging on SafetyDecision.UNAVAILABLE results — surfaces degraded-safety state per D-14's Watch-Out note. |
||
|
SolitaryThinker
|
0bb7bfae3e |
[docs] dreamverse-integration: reflect 7.5/7.6/7.7 merged + 7.8 opened
Updates following the PR landings during this session: * pr-roadmap.md: move PR 7.5 (#1251 merged 2026-04-26 as |
||
|
SolitaryThinker
|
75b613574e |
[docs] dreamverse-integration: track D-13 + 9 new follow-ups from this thread
Adds D-13 to decisions-log.md mirroring D-12's format: PR #1258 prompt enhancer / LLMProvider abstraction shape review by Oracle. Verdict — keep current shape (Alt A); promote to top-level fastvideo.prompt.* only when a second non-streaming consumer exists. Don't convert Protocol → ABC. Three deferred polishes captured. Adds 9 new items to open-threads.md priority table: * DR-1 (High): Dreamverse must create prompting/_internal_compat.py shim and delete most of the 1933-LOC local prompt_enhancer.py once PR #1258 merges. * DR-2 (Med): Decide cerebras_ifm provider path — public Literal vs. Dreamverse-side custom provider via register_provider(). * D-12-A (Med): GpuPool ABC docstring — mark experimental / server-internal until PR 7.10 cycle. * D-12-B (Med): Replace GpuPool.run() -> Any with run_async() -> AsyncIterator[VideoEvent] in PR 7.10 cycle. * D-13-A (Med): Document streaming/prompt/* as streaming-scoped in user-facing docs; avoid framework-level framing. * D-12-C (Low): Avoid locking PoolAssignment.gpu_id: int as public; prefer worker_id (already exists) or device_ids: list[int] for future multi-GPU-per-worker. * D-13-B (Low): Optional client_factory parameter for httpx pooling if metrics justify. * #9 (Low): PR 8's three commits still have [8/n] Improve API: prefix — regex didn't match single-digit version. Cleanup when PR 8 is opened. * #10 (Low): PR 7.8/7.9 commits have streaming: streaming X duplication. Cleanup when 7.8/7.9 are opened. * #11 (Low): Promote LTX-2 prompt orchestration (locked segments, segment_prompts JSON shape) to public when a second consumer appears. Updates recommended pull order to include the new items. |
||
|
SolitaryThinker
|
8039c86244 |
[docs] dreamverse-integration: track GpuPool architecture decision (D-12)
Adds D-12 to decisions-log.md, documenting the Oracle review of whether GpuPool (PR #1257, just merged) should be folded into VideoGenerator. Decision summary: keep GpuPool separate from VideoGenerator (Alt A) as interim, evolve to Alt C (thin async executor over generate_async) once PR 7.10 lands. Do not pursue Alt B (folding into VideoGenerator). Captures: * Three alternatives evaluated (A: status quo, B: VideoGenerator absorbs pool role, C: async executor over generate_async) * Key finding: MultiprocExecutor and SubprocessGpuPool are orthogonal, not redundant — both spawn subprocesses because CUDA contexts demand process boundaries, but they solve different problems (one-call-fast vs. many-sessions-throughput) * Sticky binding stays in pool, NOT in VideoGenerator (different consumers want different policies — sticky for streaming, lease for HTTP, queue for per-frame real-time) * Risks flagged: GpuPool.run() sync shape, PoolAssignment.gpu_id: int freeze, and the 'is this canonical serving API?' framing question Action items deferred to PR 7.10 cycle: replace run() with run_async() returning AsyncIterator[VideoEvent], clarify worker_id vs gpu_id field naming, mark GpuPool docstring as experimental until 7.10. |
||
|
|
af92021aa3 |
[docs] add STACK.md + CO-AUTHORS.md trackers for the will/ltx2_sr_port stack
* STACK.md tracks the 10-PR split of will/ltx2_sr_port: per-PR slice ranges, tip SHAs (live, not hardcoded), independence map, landing strategy, fast-track candidates, and re-slice commands to run after any rebase or trailer injection. * CO-AUTHORS.md documents the 4 GitHub users credited as co-authors via Co-authored-by trailers on every commit in the stack: - @Davids048 (Junda Su) - @RandNMR73 (Matthew Noto) - @XOR-op - @jzhang38 (Zhang Peiyuan) All 4 verified active in FastVideo-internal git history. Trailer emails use GitHub's <id>+<username>@users.noreply.github.com form for reliable account linkage. Both files are temporary trackers (delete after the full stack merges or as documented at the bottom of each file). Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com> Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com> Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com> Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com> |
||
|
|
4b154835cd |
[docs] .agents/memory: add dreamverse-integration knowledge base
Consolidates the public-API refactor + LTX-2 streaming-server upstream
+ Dreamverse migration + NVFP4 quantization landing into a single agent
memory module. Future agents loading the dreamverse-integration story
can pull a single targeted slice (state, design, PR roadmap, streaming,
cross-repo, quantization, decisions, open threads) rather than wading
through ~200 KB of source documents.
Structure:
.agents/memory/dreamverse-integration/
├── README.md ← index + glossary + reading guide
├── state.md ← branches, commits, live services
├── design.md ← typed schema design
├── pr-roadmap.md ← PRs 0-17 status
├── streaming-server.md ← PRs 7.5-7.10 + Dynamo + build_app
├── cross-repo-surfaces.md ← Dreamverse 3-surface + Dynamo contract
├── quantization.md ← NVFP4 + LinearBase fallback + AbsMaxFP8
├── decisions-log.md ← D-1..D-11 + Q-1..Q-9 status
├── open-threads.md ← 12 active items prioritized
└── source-archive/ ← 7 archived source docs (pre-synthesis)
Source-archive contents (all previously untracked at repo root or in
.agents/exploration/): apirefactor.md (838 lines), PR-plan.md (1145
lines, was 'PR plan.md'), dreamverse_review.md (390 lines),
handoff-nvfp4-launch-demo.md, streaming-server-upstream-plan.md,
dreamverse_integration.md, video-generator-config-api-design.md.
Registered in .agents/memory/index.jsonl as
{name: dreamverse-integration, status: ready, trust: high}.
Also adds CLEANUP.md (temporary tracker) for the multi-phase .agents/
cleanup. Phase 1 (deletes) shipped in the prior commit; Phase 2+
(rewrites, additions, registry consolidation) are tracked there for a
follow-up.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
|
||
|
|
4a2a11f6b0 |
[chore] .agents: drop stale STATUS.md + sync-dashboard SOP + vapor skills
Phase 1 of .agents/ cleanup. All deletions are stale, redundant, or
unused; .agents/ memory + skills + workflows remain functional.
Removed:
* STATUS.md — hand-maintained dashboard, last synced 2026-03-02 with
wrong counts (claimed 8 skills/4 workflows/4 memory; actual 9/5/5)
and references to non-existent snake_case filenames. Strictly
redundant with .agents/{memory,skills}/index.jsonl.
* workflows/sync-dashboard.md — SOP that maintained the deleted
STATUS.md, with obsolete pre-PR-4 file paths.
* skills/index-related-work/, skills/search-related-work/ — vapor
skills operating on the empty .agents/memory/related-work/ registry.
The search skill literally requires 'index has entries' as a
prerequisite but none have ever been added. Re-introduce when the
registry is actually populated.
Updated skills/index.jsonl to drop the two removed entries (was 9
entries; now 7).
.agents/scripts/sync-skills.sh ran post-deletion and pruned 2 stale
.claude/skills/ symlinks pointing at the removed skill directories.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
|
||
|
|
c094262751 |
[fix]: unwrap list-of-generator before torch.randn in LTX-2 latent prep
InputValidationStage normalizes generator to a one-element list when num_videos_per_prompt == 1, but torch.randn only accepts a single torch.Generator. Unwrap the list here to keep the single-sample path working. Batched sampling (>1) currently collapses to the first generator — flagged in a comment for the future batched-inference path. Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com> Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com> Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com> Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com> |
||
|
|
a750c25bdd |
[fix]: avoid model.to() round-trip in Gemma encoder forward
Wrap the device move in an equality guard so Dynamo can DCE it under fullgraph=True (model.to() goes through _parse_to, which returns a non-Tensor torch.device that Dynamo can't trace). The model is already placed on the target device by prepare_for_compile, so the guard is a runtime no-op. Also drop the orig_device save/restore — the encoder should stay resident on the compute device between calls. Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com> Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com> Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com> Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com> |
||
|
|
aca68d9452 |
test(nvfp4): lock LTX-2 wiring + typed transformer_quant flow
Two new contract test files plus a few stale-name doc fixups so the
LTX-2 NVFP4 path can't silently regress.
* ``test_nvfp4_ltx2_wiring.py`` (6 tests): asserts that
- ``LTXSelfAttention``'s ``to_q/to_k/to_v/to_out`` are
``ReplicatedLinear`` (not plain ``nn.Linear``);
- ``NVFP4Config()`` attaches ``NVFP4QuantizeMethod`` to the
quantized subset and the ``layer_prefix`` matches the
``ltx2.blocks.<i>.<sub>.<proj>`` paths in
``NVFP4Config.fp4_layers``;
- non-tagged projections (cross-attn K/V, audio attn, audio FFN)
fall back to ``UnquantizedLinearMethod`` instead of crashing on
the quant_method assert;
- ``BasicAVTransformerBlock`` propagates ``quant_config`` and
``prefix`` correctly to all 4 attention modules + FFN at once.
These are CPU-only and don't need flashinfer.
* ``test_typed_quant_flow.py`` (4 tests): asserts that
- typed ``engine.quantization.transformer_quant: "NVFP4"`` resolves
through ``compat.py`` to a concrete ``NVFP4Config()`` instance;
- omitting the typed surface leaves the ``transformer_quant``
carrier ``None`` so legacy callers that mutate
``pipeline_config.dit_config.quant_config`` directly keep working;
- ``__post_init__._apply_transformer_quant`` pins the carrier onto
``dit_config.quant_config``;
- explicit ``dit_config.quant_config = …`` is preserved (the
explicit setter wins over the typed carrier).
Plus stale ``FP4Config`` → ``NVFP4Config`` doc updates in
``api/compat.py``, ``fastvideo_args.py``, and ``layers/linear.py``
that the previous rename commit missed (comments and docstrings only,
no behavior change).
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
|
||
|
|
b24c4f0d61 |
refactor(quant): rename FP4 → NVFP4 to disambiguate from other FP4 variants
The FP4 implementation is specifically NVIDIA's block-scaled FP4 (e2m1 mantissa, fp32 alpha, ``layout_128x4`` scale layout, group size 16) backed by FlashInfer's ``nvfp4_quantize`` and ``mm_fp4`` kernels. Naming the public surface ``FP4`` would collide with other FP4 variants we may want to support later (OCP-FP4 / MX-FP4 / e3m0). Mechanical rename, no behavior change: * ``fp4_config.py`` → ``nvfp4_config.py`` * ``FP4Config`` → ``NVFP4Config``; ``get_name()`` returns ``"nvfp4"`` * ``FP4QuantizeMethod`` → ``NVFP4QuantizeMethod`` * ``convert_model_to_fp4`` → ``convert_model_to_nvfp4`` * ``QuantizationMethods`` literal: ``"FP4"`` → ``"NVFP4"`` * registered buffer names: ``_fp4_weight``/``_fp4_alpha`` → ``_nvfp4_weight``/``_nvfp4_alpha`` * loader helper ``_maybe_convert_model_to_fp4`` → ``_maybe_convert_model_to_nvfp4`` * test file rename + symbol updates * Dreamverse worker updates its single import + call site Internal-scope torch op namespace ``fastvideo_fp4::*`` and ``_get_ltx2_fp4_stage_profile`` left as-is (purely internal naming that mirrors FastVideo-internal). Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com> Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com> Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com> Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com> |