Adds D-12 to decisions-log.md, documenting the Oracle review of whether
GpuPool (PR #1257, just merged) should be folded into VideoGenerator.
Decision summary: keep GpuPool separate from VideoGenerator (Alt A) as
interim, evolve to Alt C (thin async executor over generate_async) once
PR 7.10 lands. Do not pursue Alt B (folding into VideoGenerator).
Captures:
* Three alternatives evaluated (A: status quo, B: VideoGenerator absorbs
pool role, C: async executor over generate_async)
* Key finding: MultiprocExecutor and SubprocessGpuPool are orthogonal,
not redundant — both spawn subprocesses because CUDA contexts demand
process boundaries, but they solve different problems (one-call-fast
vs. many-sessions-throughput)
* Sticky binding stays in pool, NOT in VideoGenerator (different
consumers want different policies — sticky for streaming, lease for
HTTP, queue for per-frame real-time)
* Risks flagged: GpuPool.run() sync shape, PoolAssignment.gpu_id: int
freeze, and the 'is this canonical serving API?' framing question
Action items deferred to PR 7.10 cycle: replace run() with run_async()
returning AsyncIterator[VideoEvent], clarify worker_id vs gpu_id field
naming, mark GpuPool docstring as experimental until 7.10.
* STACK.md tracks the 10-PR split of will/ltx2_sr_port: per-PR slice
ranges, tip SHAs (live, not hardcoded), independence map, landing
strategy, fast-track candidates, and re-slice commands to run after
any rebase or trailer injection.
* CO-AUTHORS.md documents the 4 GitHub users credited as co-authors
via Co-authored-by trailers on every commit in the stack:
- @Davids048 (Junda Su)
- @RandNMR73 (Matthew Noto)
- @XOR-op
- @jzhang38 (Zhang Peiyuan)
All 4 verified active in FastVideo-internal git history. Trailer
emails use GitHub's <id>+<username>@users.noreply.github.com form
for reliable account linkage.
Both files are temporary trackers (delete after the full stack merges
or as documented at the bottom of each file).
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
Phase 1 of .agents/ cleanup. All deletions are stale, redundant, or
unused; .agents/ memory + skills + workflows remain functional.
Removed:
* STATUS.md — hand-maintained dashboard, last synced 2026-03-02 with
wrong counts (claimed 8 skills/4 workflows/4 memory; actual 9/5/5)
and references to non-existent snake_case filenames. Strictly
redundant with .agents/{memory,skills}/index.jsonl.
* workflows/sync-dashboard.md — SOP that maintained the deleted
STATUS.md, with obsolete pre-PR-4 file paths.
* skills/index-related-work/, skills/search-related-work/ — vapor
skills operating on the empty .agents/memory/related-work/ registry.
The search skill literally requires 'index has entries' as a
prerequisite but none have ever been added. Re-introduce when the
registry is actually populated.
Updated skills/index.jsonl to drop the two removed entries (was 9
entries; now 7).
.agents/scripts/sync-skills.sh ran post-deletion and pruned 2 stale
.claude/skills/ symlinks pointing at the removed skill directories.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
Wrap the device move in an equality guard so Dynamo can DCE it under
fullgraph=True (model.to() goes through _parse_to, which returns a
non-Tensor torch.device that Dynamo can't trace). The model is already
placed on the target device by prepare_for_compile, so the guard is a
runtime no-op. Also drop the orig_device save/restore — the encoder
should stay resident on the compute device between calls.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
Two new contract test files plus a few stale-name doc fixups so the
LTX-2 NVFP4 path can't silently regress.
* ``test_nvfp4_ltx2_wiring.py`` (6 tests): asserts that
- ``LTXSelfAttention``'s ``to_q/to_k/to_v/to_out`` are
``ReplicatedLinear`` (not plain ``nn.Linear``);
- ``NVFP4Config()`` attaches ``NVFP4QuantizeMethod`` to the
quantized subset and the ``layer_prefix`` matches the
``ltx2.blocks.<i>.<sub>.<proj>`` paths in
``NVFP4Config.fp4_layers``;
- non-tagged projections (cross-attn K/V, audio attn, audio FFN)
fall back to ``UnquantizedLinearMethod`` instead of crashing on
the quant_method assert;
- ``BasicAVTransformerBlock`` propagates ``quant_config`` and
``prefix`` correctly to all 4 attention modules + FFN at once.
These are CPU-only and don't need flashinfer.
* ``test_typed_quant_flow.py`` (4 tests): asserts that
- typed ``engine.quantization.transformer_quant: "NVFP4"`` resolves
through ``compat.py`` to a concrete ``NVFP4Config()`` instance;
- omitting the typed surface leaves the ``transformer_quant``
carrier ``None`` so legacy callers that mutate
``pipeline_config.dit_config.quant_config`` directly keep working;
- ``__post_init__._apply_transformer_quant`` pins the carrier onto
``dit_config.quant_config``;
- explicit ``dit_config.quant_config = …`` is preserved (the
explicit setter wins over the typed carrier).
Plus stale ``FP4Config`` → ``NVFP4Config`` doc updates in
``api/compat.py``, ``fastvideo_args.py``, and ``layers/linear.py``
that the previous rename commit missed (comments and docstrings only,
no behavior change).
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
The FP4 implementation is specifically NVIDIA's block-scaled FP4
(e2m1 mantissa, fp32 alpha, ``layout_128x4`` scale layout, group
size 16) backed by FlashInfer's ``nvfp4_quantize`` and ``mm_fp4``
kernels. Naming the public surface ``FP4`` would collide with other
FP4 variants we may want to support later (OCP-FP4 / MX-FP4 / e3m0).
Mechanical rename, no behavior change:
* ``fp4_config.py`` → ``nvfp4_config.py``
* ``FP4Config`` → ``NVFP4Config``; ``get_name()`` returns ``"nvfp4"``
* ``FP4QuantizeMethod`` → ``NVFP4QuantizeMethod``
* ``convert_model_to_fp4`` → ``convert_model_to_nvfp4``
* ``QuantizationMethods`` literal: ``"FP4"`` → ``"NVFP4"``
* registered buffer names: ``_fp4_weight``/``_fp4_alpha`` →
``_nvfp4_weight``/``_nvfp4_alpha``
* loader helper ``_maybe_convert_model_to_fp4`` →
``_maybe_convert_model_to_nvfp4``
* test file rename + symbol updates
* Dreamverse worker updates its single import + call site
Internal-scope torch op namespace ``fastvideo_fp4::*`` and
``_get_ltx2_fp4_stage_profile`` left as-is (purely internal naming
that mirrors FastVideo-internal).
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
The public LTX-2 path was previously running full bf16 even when
callers set ``pipeline_config.dit_config.quant_config = FP4Config()``,
because:
* the DiT used plain ``torch.nn.Linear`` for FP4-eligible projections
(``attn1/attn2.to_q/k/v/out``, ``audio_to_video_attn``,
``video_to_audio_attn``, FFN ``fc_in``/``fc_out``), so
``LinearBase.quant_method.apply`` was never reached;
* the loader did not call ``convert_model_to_fp4`` after weights
loaded, so even matched layers had no ``_fp4_weight*`` buffers;
* ``QuantizationMethods`` did not register ``"FP4"``, so the typed
``engine.quantization.transformer_quant`` surface had no way to
select it.
This commit completes the inference wire-up through the existing
``fastvideo.layers.quantization`` registry — no parallel pathway:
* swap ``nn.Linear`` → ``ReplicatedLinear`` for the LTX-2 attention
``to_q``/``to_k``/``to_v``/``to_out``/``to_gate_compress`` and the
FFN ``fc_in``/``fc_out`` (``GELUApprox.proj`` and ``FeedForward``
``project_out``). Other linears (timestep MLP, caption proj,
patchify, AdaLN scale/shift) stay ``nn.Linear`` to match internal.
* port ``_supports_prequantized_input`` and
``_linear_project_with_optional_prequant`` so the attention forward
quantizes input once and reuses the ``(x_fp4, x_scale, x_global_sf)``
tuple across q/k/v projections — matches internal exactly.
* plumb ``quant_config`` and ``prefix`` through
``BasicAVTransformerBlock`` → ``_init_transformer_blocks`` →
``LTXModel`` → ``LTX2Transformer3DModel`` so each
``ReplicatedLinear`` gets the correct
``ltx2.blocks.<i>.<sub>.<proj>`` prefix that
``FP4Config.get_quant_method`` matches against.
* register ``"FP4"`` in ``QuantizationMethods`` literal +
``get_quantization_config`` registry so typed
``engine.quantization.transformer_quant: "FP4"`` resolves to a
concrete ``FP4Config()`` instance.
* add ``transformer_quant`` carrier on ``FastVideoArgs`` and
``__post_init__._apply_transformer_quant`` to pin the resolved
quant config onto ``dit_config.quant_config`` (without
overwriting an explicit setter on the dit_config).
* add ``_maybe_convert_model_to_fp4`` in ``fsdp_load.py`` that
walks the loaded model once and registers
``_fp4_weight*``/``_fp4_alpha``/``_weight_global_sf`` buffers on
layers whose ``quant_method`` is ``FP4QuantizeMethod``. flashinfer
is imported lazily inside ``convert_model_to_fp4`` so this is a
no-op on hosts without the FP4 kernels.
* restore ``LinearBase`` fallback to ``UnquantizedLinearMethod`` when
``quant_config.get_quant_method`` returns ``None`` — ``FP4Config``
only tags a curated subset of layers, and the previous
``assert quant_method is not None`` would crash any non-tagged
layer that received a quant_config.
Non-FP4 callers are unaffected: ``UnquantizedLinearMethod.apply``
runs the standard ``F.linear`` path, and parameter names
(``to_q.weight``, ``to_out.0.weight`` from ``nn.ModuleList``) are
identical to the previous ``nn.Sequential`` layout.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
Bring the public ``ComposedPipelineBase.post_init`` compile dispatch
to feature parity with FastVideo-internal:
* Compile ``transformer_refine`` alongside ``transformer`` and
``transformer_2`` whenever the DiT compile flag is on. Without this
the LTX-2 stage-2 refine pass silently runs eager while the main
transformer is compiled — different inductor fusion choices vs.
eager execution can produce last-bit divergences in bf16.
* Drive the new per-component compile flags from ``FastVideoArgs``:
text encoder (``enable_torch_compile_text_encoder``), VAE
(``enable_torch_compile_vae``), audio VAE
(``enable_torch_compile_audio_vae``). Each picks its own kwargs
dict (``torch_compile_kwargs_*``) and falls back to the master
``torch_compile_kwargs`` when empty.
* Call ``module.prepare_for_compile()`` on each compiled submodule
before invoking ``torch.compile`` so model-specific external state
(e.g. lazy HF loads) lands outside Dynamo's tracer. ``Gemma3``
implements this hook to materialize the Gemma weights up front.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
Extend the typed inference API so callers can drive torch.compile per
component without falling back to the legacy flat-kwargs pathway.
* ``CompileConfig`` now exposes ``vae_enabled``, ``audio_vae_enabled``,
``dit_kwargs``, ``text_encoder_kwargs``, ``vae_kwargs``,
``audio_vae_kwargs`` alongside the existing ``enabled`` /
``text_encoder_enabled`` switches. The master ``backend``/
``fullgraph``/``mode``/``dynamic``/``extras`` continue to apply to
every compiled submodule unless a per-component kwargs dict is
non-empty (in which case it overrides entirely — matches the
FastVideo-internal precedent).
* Add the matching runtime carrier fields on ``FastVideoArgs``:
``enable_torch_compile_text_encoder/vae/audio_vae`` plus
``torch_compile_kwargs_dit/text_encoder/vae/audio_vae``. These let
the legacy flat-kwargs path round-trip through the typed config.
* ``api/compat.legacy_from_pretrained_to_config`` now lifts the new
flat kwargs into ``CompileConfig``, and
``generator_config_to_fastvideo_args`` writes them back out so
``VideoGenerator.from_pretrained(model_path, ...)`` callers keep
working unchanged.
No behavior change yet — these surfaces are ports of carriers only.
The composed pipeline base will start consuming them in the next
commit.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
Three small parity fixes uncovered while comparing the LTX-2 distilled
path against FastVideo-internal:
* `FastVideoArgs.__post_init__` now calls `_resolve_refine_args()` to
copy the public-facing generic `refine_*` knobs onto their
`ltx2_refine_*` runtime carriers (mirrors internal lines 303-322).
Without this, callers that set `refine_lora_path=...` via the typed
CLI/config surface would silently drop the value, surfacing later as
the "applied to 0 layers" warning.
* Revert the patch-noise sampler in `_randn_ltx2_video_latents` from
`randn_tensor` back to `torch.randn` to bit-match internal under
single-generator inference. `randn_tensor` is identical for a single
`torch.Generator` but diverges for `list[Generator]` (per-sample
seeds), which is the only place the two paths could disagree.
* Classify the 19 previously-unclassified `refine_*` / `ltx2_refine_*`
/ `ltx2_audio_latent_path` / `ltx2_images` / `ltx2_image_crf` /
`ltx2_conditioning_latent_*` / `ltx2_video_conditions` fields in the
schema-parity inventory yaml so the parity test passes.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
Replaces the T2V-only stub with the full internal port:
* resolve_ltx2_images / _resize_and_center_crop / CRF re-encode
* load_ltx2_conditioning_image + load_ltx2_conditioning_video_clip
* _extract_video_latent / _insert_conditioning_latent
* build_ltx2_image_conditioning composes (clean_latent, denoise_mask)
from images + video clips + continuation latents (stage1 vs stage2)
— returns None for plain T2V
ForwardBatch + SamplingParam gain the missing fields the builder
reads (ltx2_images, ltx2_image_crf, ltx2_conditioning_latent_stage1/2,
ltx2_video_conditions). generate_video kwargs flow through
sampling_param.update -> ForwardBatch(**shallow_asdict(...)) so
Dreamverse's ltx2_image_crf=0.0 and continuation latents land on the
batch correctly.
68/68 LTX-2 tests still green.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
The model-name detector loop in get_model_name_for_path tests both
``model_path.lower()`` AND ``pipeline_name`` against each detector,
OR-ing the result. The pipeline_name for the distilled checkpoint is
``ltx2pipeline`` — a string that contains no "distilled" marker — so
the *base* detector's ``and "distilled" not in path`` predicate
returns True against it, even when the absolute model path clearly
contains "distilled".
Result: the resolver matches both LTX-2 base and LTX-2 distilled,
warns "Multiple models matched … Using the first matched: '0'", and
falls back to whichever was registered first. That used to be base
(ltx2_base preset → cfg=3.0 / mod=3.0 / rescale=0.7 / stg=1.0), so
SamplingParam.from_pretrained on a distilled-checkpoint path silently
loaded the *full* LTX-2 sampling defaults — exactly the divergence
the public-vs-internal Dreamverse alignment surfaced.
Reorder so the distilled entry registers first. The registry's
"first matched wins" tiebreak now lands on the more specific preset
when both detectors fire, and SamplingParam.from_pretrained reaches
the internal-aligned distilled defaults (mod=1.0 / rescale=0.0 /
stg=0.0 / cfg_video=1.0 / cfg_audio=1.0).
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
Internal FastVideo declares two LTX-2 SamplingParam variants:
LTX2SamplingParam (mod=3.0/rescale=0.7/stg=1.0 — full LTX-2) and
LTX2DistilledSamplingParam (mod=1.0/rescale=0.0/stg=0.0). Dreamverse
production runs against LTX2-Distilled-Diffusers, so the distilled
defaults are what the runtime should land on.
The public package only has the single SamplingParam class, and its
class-level defaults for these knobs were the *full* LTX-2 values.
Result: when Dreamverse (or any other consumer) called
``SamplingParam.from_pretrained("FastVideo/LTX2-Distilled-Diffusers")``,
the LTX2_DISTILLED preset's defaults dict didn't override these
fields (it didn't list them) so they fell through to the class
defaults — silently picking up *full*-model CFG / modality / rescale /
STG instead of the distilled ones. Streamed video output diverged
from the internal-ui reference.
Switch the class defaults to the distilled values (1.0 / 0.0 / 0.0).
The LTX2_BASE preset already overrides them explicitly to 3.0/0.7/1.0
in its ``defaults`` dict, so users selecting the full-LTX-2 preset
keep the full-model behavior.
The numerical alignment harness still reproduces the same 42.72 dB
PSNR / 52.87% exact pixel match because that test pinned the values
explicitly via legacy kwargs; this fix moves the *default* path on
the public side onto the same numerical track.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
Initial alignment showed PSNR ~12 dB because public's LTX2_BASE preset
sets ``ltx2_modality_scale_video=3.0 / rescale=0.7 / stg_video=1.0``
while internal's default ForwardBatch lands at ``1.0 / 0.0 / 0.0``.
Pinning these explicitly on both runs aligns the denoising mechanics
so the diff measures pipeline parity, not preset divergence.
Result with both sides pinned to mod=1.0/rescale=0.0/stg=0.0:
max_abs_diff = 203
mean_abs_diff = 0.788
rms_diff = 1.864
psnr_db = 42.72 dB
exact pixels = 52.87%
within 1 lvl = 86.94%
within 4 lvls = 97.93%
The ~47% near-but-not-exact bucket is consistent with bf16 reduction
non-determinism on B200 plus the fact that the public LoRA matcher
emitted "applied to 0 layers" for the LTX2-Distilled refine LoRA
(internal side did not). Both runs effectively skip the refine LoRA,
so they share the same effective stage-2 model — the residual diff is
the right ballpark for "same algorithm, different bf16 reduction
schedule".
The typed-API path also sets the same overrides via setattr on the
GenerationRequest so the typed path stays comparable. Once the typed
SamplingConfig grows ltx2-specific fields the setattr workaround can
go.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
LTX-2 SR pipeline now passes module load + stage graph build on the
public side. Each retry of scripts/ltx2_sr_alignment.py surfaced one
more gap; this commit closes them so the pipeline progresses to the
upsample → stage-2 → decode tail.
* harness: PipelineSelection (real class name; PipelineConfig was a typo)
* FastVideoArgs: generic refine_* fields (preferred user-facing API)
* LTX2Pipeline.initialize_pipeline: getattr-with-default for
debug_model_sums / debug_model_detail (public args lack the debug
fields; LTX-2 still flips the env-var toggles when set)
* VAEConfig: use_temporal_scaling_frames default True (LTX-2 latent
prep reads this in the schedule-shift gate)
* LTX2LatentPreparationStage._randn_ltx2_video_latents: switch to
randn_tensor so list-of-generators batches work (matches internal
sampling order)
* ModelRegistry: register LTX2LatentUpsampler in _UPSAMPLERS
* component_loader: register spatial_upsampler / temporal_upsampler
with UpsamplerLoader (was hitting the generic stub loader);
UpsamplerLoader gains LTX-2 fallback when pipeline_config.upsampler_config
isn't a multi-config tuple, plus the 'model.'-prefix-stripping
weight load path the internal version uses
7 retries -> public side now reaches the upsample stage.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
Single script with three modes:
* --label internal --output /tmp/ltx2_sr_internal.pt : runs the legacy
from_pretrained kwargs path and saves frames + metadata. Used against
PYTHONPATH=../FastVideo-internal as the reference run.
* --label public --use-typed-api --output /tmp/ltx2_sr_public.pt : runs
through the new GeneratorConfig + GenerationRequest typed API with
pipeline.preset_overrides["refine"] driving the SR path. Used against
PYTHONPATH=../FastVideo (the public package) for the candidate run.
* --diff --reference R --candidate C : loads both .pt files and reports
shape parity + max_abs / mean_abs / rms / psnr / pixel-bucket
histograms over the uint8 frames.
Pinned alignment fixture: PROMPT + seed + 1088x1920x121x24fps + 8 base
steps + 3 refine steps mirrors basic_ltx2_upscale.py upstream so we
exercise the same code path the user has been generating with.
Deliberately skips FP4 (no quant config) and Dreamverse runtime knobs
to keep the bf16 path clean per request.
The two runs are intentionally invoked in separate processes via
PYTHONPATH overrides because both repos register as the `fastvideo`
package and can't be imported together. The diff script then runs in
either env to load both .pt files and compute the alignment metrics.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
Brings the public LTX2Pipeline into parity with the internal SR-capable
version end-to-end:
* LTX2DenoisingStage gains the stage-2 control surface
(sigmas_override, num_inference_steps_override, force_guidance_scale,
initial_audio_latents_key) plus the i2v conditioning round-trip
(clean_latent + denoise_mask via LTX2_VIDEO_*_KEY) and the audio
refine path. The file is now ~1:1 with the internal stage; only the
i2v_conditioning import path differs (rerouted to the public
ltx2_image_conditioning module).
* LTX2LatentPreparationStage takes vae=, exposes the official LTX-2
noise sampling order via _randn_ltx2_video_latents (patchify -> noise
in token order -> unpatchify), and applies image conditioning when
build_ltx2_image_conditioning returns a state. Matches internal.
* LTX2Pipeline switches its base from ComposedPipelineBase to
LoRAPipeline (refine-LoRA support); create_pipeline_stages adds the
refine_init / upsample / refine_lora / refine_denoising chain when
ltx2_refine_enabled is True; load_modules pulls
spatial_upsampler (and optionally transformer_refine) from the
resolved upsampler path, with model_index.json defaults
(fastvideo_refine_*) feeding the FastVideoArgs.ltx2_refine_*
carriers.
stages/__init__.py re-exports the new refine stages so importers
outside the basic/ltx2 package keep working. 68/68 LTX-2-related tests
still pass after the rewrite (api translation, gpu_pool, contract,
streaming).
Reachability: setting pipeline.preset_overrides["refine"] = {"enabled":
True, ...} and components.upsampler_weights flows through compat.py
into the new ltx2_refine_* args, the pipeline branches, and load_modules
brings in the upsampler. T2V SR is now end-to-end runnable; numerical
alignment harness comes next.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
Lays the runtime-side pieces the LTX2_TWO_STAGE preset will exercise.
Behaviour matches FastVideo-internal 1:1 for the text-to-video SR path;
i2v / continuation conditioning is stubbed with a NotImplementedError
that fires only on those code paths so the public T2V SR run is clean.
* fastvideo/models/upsamplers/ltx2_upsampler.py — verbatim copy of the
internal LTX-2 latent upsampler (PixelShuffleND, BlurDownsample,
SpatialRationalResampler, ResBlock, LatentUpsampler, LTX2LatentUpsampler,
upsample_video). Re-exported via models/upsamplers/__init__.py.
* fastvideo/pipelines/basic/ltx2/stages/ltx2_image_conditioning.py —
minimal i2v-conditioning helpers: the four ForwardBatch.extra keys,
apply_ltx2_gaussian_noiser, post_process_ltx2_denoised, and a T2V-only
build_ltx2_image_conditioning that returns None when the request has
no image inputs / no continuation latents (the SR T2V path).
* fastvideo/pipelines/basic/ltx2/stages/ltx2_refine.py — the three
refine stages adapted to public imports: LTX2RefineInitStage halves
the request resolution and stashes the original target;
LTX2UpsampleStage runs the latent upsampler, optionally re-applies
conditioning, and mixes refine-noise scaled by the stage-2 sigma;
LTX2RefineLoRAStage swaps in a refine-specific LoRA (no-op when path
is unset).
* fastvideo/fastvideo_args.py — add the runtime carrier fields
ltx2_refine_{enabled,upsampler_path,transformer_path,lora_path,
num_inference_steps,guidance_scale,add_noise,noise_path,
audio_noise_path}. The typed public surface (ComponentConfig
upsampler_weights/lora_path + pipeline.preset_overrides.refine) keeps
flowing through compat.py:273-279, which expands the dict into these
ltx2_refine_* kwargs at construction time.
Not yet wired: the LTX2 pipeline's create_pipeline_stages still goes
denoise -> audio_decode -> decode without the refine_init / upsample
/ refine_denoise injection. The remaining piece is teaching
LTX2DenoisingStage to accept sigmas_override / num_inference_steps_override
/ force_guidance_scale / initial_audio_latents_key (the internal
version is 725 lines vs public 395; ~330 lines of stage-2 logic still
need bringing across) and to flip the pipeline graph based on
ltx2_refine_enabled. That follow-up unlocks the numerical alignment run.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
Ports the FP4 quantization config + ops from FastVideo-internal
(fastvideo.layers.quantization.fp4_config). flashinfer is imported
lazily inside _require_flashinfer() rather than at module top so
``import fastvideo`` stays cheap on hosts that don't ship flashinfer
— the FP4 kernels themselves still raise a clear ImportError pointing
at ``pip install flashinfer-python`` when invoked.
Behavior is identical to the internal version:
* FP4Config: LTX-2 layer set (48 transformer blocks across attn1/attn2
/audio_to_video_attn/video_to_audio_attn/ffn plus adaln_single.linear),
layer_profile selects between "base" and "refine" stage subsets.
* FP4QuantizeMethod: per-layer quantize_input + apply with stage-aware
routing for refine-only layers (audio_to_video_attn.to_q,
video_to_audio_attn.to_k, video_to_audio_attn.to_v fall through to
dense F.linear during the base stage).
* convert_model_to_fp4(): pre-quantize weight buffers + alpha onto
module attrs at load time so the forward path doesn't pay the
per-step weight-side quantize cost.
Required for Dreamverse integration: dreamverse-server's GPU worker
runs ``pipeline_config.dit_config.quant_config = FP4Config()`` during
LTX-2 model load, which is the dominant inference path.
test_fp4_config covers the lazy-flashinfer contract: import succeeds
without flashinfer; layer_profile round-trips via from_config; calling
into _require_flashinfer raises ImportError with an actionable
message when flashinfer is missing.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
fastvideo/tests/contract/ guards against drift in the two public-facing
adapter shapes that PR 8 locks down:
* test_dynamo_shape.py: mock Dynamo-style handler implementing the
adapter template from docs/design/server_contracts/dynamo.md. It
imports only fastvideo + fastvideo.api, translates a
Dynamo-shape NvCreateVideoRequest dict into a typed
GenerationRequest, runs it through an async generator that matches
endpoint.serve_endpoint(handler.generate, ...), and round-trips the
continuation-state envelope. A source scan asserts the adapter
function body touches no private FastVideo module.
* test_dreamverse_shape.py: the flat init-time kwarg bag the internal
gpu_pool.py passes today lands entirely on typed GeneratorConfig
fields (no leakage into pipeline.experimental) after PR 6's typed
replacement work. Per-request Dreamverse shape round-trips through
legacy_generate_call_to_request -> normalize_generation_request;
output.return_state (PR 7) stays wired through.
Failures in this suite appear at FastVideo CI — before the Dynamo-side
integration or private Dreamverse adapter see the drift.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
docs/design/server_contracts/dynamo.md documents the integration shape
the ai-dynamo/dynamo native backend package consumes. Per the refactor
decision, no Dynamo code lives in FastVideo — the backend package
ships in the Dynamo repo at components/src/dynamo/fastvideo/ with the
layout mirroring components/src/dynamo/sglang/.
Captured here:
* the 8-symbol public surface Dynamo imports (VideoGenerator plus
ContinuationState, GenerationRequest, SamplingConfig, InputConfig,
OutputConfig, VideoEvent, VideoResult) and what is available today
vs. what lands in PR 7.10
* target backend package layout matching the sglang template
* full NvCreateVideoRequest <-> GenerationRequest mapping table and
VideoFinalEvent <-> NvVideosResponse reverse mapping
* worked examples: aggregated handler (works today with generate_video
+ asyncio.to_thread), streaming handler (post-PR 7.10 via
generate_async), health check payload (swapping to
default_health_check_request() after PR 7.10)
* init function sketch, registration (ModelType::Videos fast path),
args adapter
* contract guarantees that let the adapter be written once and not
chase FastVideo drift
* explicit no-import list for the adapter
Also wires the server_contracts/ section into mkdocs.yml under Design.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
docs/design/server_contracts/ lands the contract reference set that PR
8 locks down ahead of the streaming + Dynamo upstreams. This commit
adds the index page plus the OpenAI HTTP reference:
* endpoint catalogue (/v1/videos/generations, list, status, content,
images, models, health)
* VideoGenerationsRequest shape with SGLang-compatible extensions
* merge precedence (body > default_request > hardcoded fallback)
with a pointer to the explicit-path tracking mechanism that makes
it work
* continuation-state roundtrip on the stateless surface
* HTTP error code table
* explicit non-goals: flat legacy kwargs, private Dreamverse fields,
raw tensor payloads — all excluded from this boundary by design
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
* tests/contract/test_generate_async.py: event ordering (Progress ->
Final); exactly-one-final per request; batch expansion emits one
Final per sub-result; VideoFinalEvent carries frames / full
GenerationResult / continuation_state as expected; health-check
request is minimal and round-trips through normalize_generation_
request; public fastvideo.api exports the new symbols; a mock
Dynamo-style async handler wraps generate_async using only the
public import surface.
* tests/api/test_cli_translation.py: the streaming-dispatch test no
longer expects NotImplementedError (PR 7.5 landed the live server);
it now injects a fake run_server and asserts the CLI hands the
typed streaming ServeConfig through unchanged.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
* VideoGenerator.generate_async(request) yields VideoEvent objects:
one VideoProgressEvent at start, one VideoFinalEvent per resulting
GenerationResult (prompt-batch expansion emits one Final per
sub-result). The aggregated code path runs the existing sync
generate() inside asyncio.to_thread; future work threads per-step
progress events from inside the pipeline's denoise loop without
changing the public contract.
* VideoGenerator.default_health_check_request() returns a 256x256 /
8-frame / 1-step typed GenerationRequest. Dynamo uses it to derive
its health_check_payload without knowing FastVideo internals.
* _final_event_from_result packages GenerationResult into a
VideoFinalEvent, reading the encoded MP4 off disk when the pipeline
wrote one so streaming consumers don't re-encode.
Also: streaming_server.run_server validates streaming config before
loading the generator so the existing ValueError fires before the
expensive model-load path.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
Add the typed event dataclasses VideoGenerator.generate_async will
emit, plus a VideoResult alias for the public docs:
* VideoProgressEvent / VideoPartialEvent / VideoFinalEvent
* VideoEvent = union of the three
* VideoResult = GenerationResult
Re-exported from fastvideo.api so the Dynamo backend package, the
streaming server, and the stateless OpenAI server share one set of
names.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
SolitaryThinker