Two new contract test files plus a few stale-name doc fixups so the
LTX-2 NVFP4 path can't silently regress.
* ``test_nvfp4_ltx2_wiring.py`` (6 tests): asserts that
- ``LTXSelfAttention``'s ``to_q/to_k/to_v/to_out`` are
``ReplicatedLinear`` (not plain ``nn.Linear``);
- ``NVFP4Config()`` attaches ``NVFP4QuantizeMethod`` to the
quantized subset and the ``layer_prefix`` matches the
``ltx2.blocks.<i>.<sub>.<proj>`` paths in
``NVFP4Config.fp4_layers``;
- non-tagged projections (cross-attn K/V, audio attn, audio FFN)
fall back to ``UnquantizedLinearMethod`` instead of crashing on
the quant_method assert;
- ``BasicAVTransformerBlock`` propagates ``quant_config`` and
``prefix`` correctly to all 4 attention modules + FFN at once.
These are CPU-only and don't need flashinfer.
* ``test_typed_quant_flow.py`` (4 tests): asserts that
- typed ``engine.quantization.transformer_quant: "NVFP4"`` resolves
through ``compat.py`` to a concrete ``NVFP4Config()`` instance;
- omitting the typed surface leaves the ``transformer_quant``
carrier ``None`` so legacy callers that mutate
``pipeline_config.dit_config.quant_config`` directly keep working;
- ``__post_init__._apply_transformer_quant`` pins the carrier onto
``dit_config.quant_config``;
- explicit ``dit_config.quant_config = …`` is preserved (the
explicit setter wins over the typed carrier).
Plus stale ``FP4Config`` → ``NVFP4Config`` doc updates in
``api/compat.py``, ``fastvideo_args.py``, and ``layers/linear.py``
that the previous rename commit missed (comments and docstrings only,
no behavior change).
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
The public LTX-2 path was previously running full bf16 even when
callers set ``pipeline_config.dit_config.quant_config = FP4Config()``,
because:
* the DiT used plain ``torch.nn.Linear`` for FP4-eligible projections
(``attn1/attn2.to_q/k/v/out``, ``audio_to_video_attn``,
``video_to_audio_attn``, FFN ``fc_in``/``fc_out``), so
``LinearBase.quant_method.apply`` was never reached;
* the loader did not call ``convert_model_to_fp4`` after weights
loaded, so even matched layers had no ``_fp4_weight*`` buffers;
* ``QuantizationMethods`` did not register ``"FP4"``, so the typed
``engine.quantization.transformer_quant`` surface had no way to
select it.
This commit completes the inference wire-up through the existing
``fastvideo.layers.quantization`` registry — no parallel pathway:
* swap ``nn.Linear`` → ``ReplicatedLinear`` for the LTX-2 attention
``to_q``/``to_k``/``to_v``/``to_out``/``to_gate_compress`` and the
FFN ``fc_in``/``fc_out`` (``GELUApprox.proj`` and ``FeedForward``
``project_out``). Other linears (timestep MLP, caption proj,
patchify, AdaLN scale/shift) stay ``nn.Linear`` to match internal.
* port ``_supports_prequantized_input`` and
``_linear_project_with_optional_prequant`` so the attention forward
quantizes input once and reuses the ``(x_fp4, x_scale, x_global_sf)``
tuple across q/k/v projections — matches internal exactly.
* plumb ``quant_config`` and ``prefix`` through
``BasicAVTransformerBlock`` → ``_init_transformer_blocks`` →
``LTXModel`` → ``LTX2Transformer3DModel`` so each
``ReplicatedLinear`` gets the correct
``ltx2.blocks.<i>.<sub>.<proj>`` prefix that
``FP4Config.get_quant_method`` matches against.
* register ``"FP4"`` in ``QuantizationMethods`` literal +
``get_quantization_config`` registry so typed
``engine.quantization.transformer_quant: "FP4"`` resolves to a
concrete ``FP4Config()`` instance.
* add ``transformer_quant`` carrier on ``FastVideoArgs`` and
``__post_init__._apply_transformer_quant`` to pin the resolved
quant config onto ``dit_config.quant_config`` (without
overwriting an explicit setter on the dit_config).
* add ``_maybe_convert_model_to_fp4`` in ``fsdp_load.py`` that
walks the loaded model once and registers
``_fp4_weight*``/``_fp4_alpha``/``_weight_global_sf`` buffers on
layers whose ``quant_method`` is ``FP4QuantizeMethod``. flashinfer
is imported lazily inside ``convert_model_to_fp4`` so this is a
no-op on hosts without the FP4 kernels.
* restore ``LinearBase`` fallback to ``UnquantizedLinearMethod`` when
``quant_config.get_quant_method`` returns ``None`` — ``FP4Config``
only tags a curated subset of layers, and the previous
``assert quant_method is not None`` would crash any non-tagged
layer that received a quant_config.
Non-FP4 callers are unaffected: ``UnquantizedLinearMethod.apply``
runs the standard ``F.linear`` path, and parameter names
(``to_q.weight``, ``to_out.0.weight`` from ``nn.ModuleList``) are
identical to the previous ``nn.Sequential`` layout.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
Extend the typed inference API so callers can drive torch.compile per
component without falling back to the legacy flat-kwargs pathway.
* ``CompileConfig`` now exposes ``vae_enabled``, ``audio_vae_enabled``,
``dit_kwargs``, ``text_encoder_kwargs``, ``vae_kwargs``,
``audio_vae_kwargs`` alongside the existing ``enabled`` /
``text_encoder_enabled`` switches. The master ``backend``/
``fullgraph``/``mode``/``dynamic``/``extras`` continue to apply to
every compiled submodule unless a per-component kwargs dict is
non-empty (in which case it overrides entirely — matches the
FastVideo-internal precedent).
* Add the matching runtime carrier fields on ``FastVideoArgs``:
``enable_torch_compile_text_encoder/vae/audio_vae`` plus
``torch_compile_kwargs_dit/text_encoder/vae/audio_vae``. These let
the legacy flat-kwargs path round-trip through the typed config.
* ``api/compat.legacy_from_pretrained_to_config`` now lifts the new
flat kwargs into ``CompileConfig``, and
``generator_config_to_fastvideo_args`` writes them back out so
``VideoGenerator.from_pretrained(model_path, ...)`` callers keep
working unchanged.
No behavior change yet — these surfaces are ports of carriers only.
The composed pipeline base will start consuming them in the next
commit.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
Replaces the T2V-only stub with the full internal port:
* resolve_ltx2_images / _resize_and_center_crop / CRF re-encode
* load_ltx2_conditioning_image + load_ltx2_conditioning_video_clip
* _extract_video_latent / _insert_conditioning_latent
* build_ltx2_image_conditioning composes (clean_latent, denoise_mask)
from images + video clips + continuation latents (stage1 vs stage2)
— returns None for plain T2V
ForwardBatch + SamplingParam gain the missing fields the builder
reads (ltx2_images, ltx2_image_crf, ltx2_conditioning_latent_stage1/2,
ltx2_video_conditions). generate_video kwargs flow through
sampling_param.update -> ForwardBatch(**shallow_asdict(...)) so
Dreamverse's ltx2_image_crf=0.0 and continuation latents land on the
batch correctly.
68/68 LTX-2 tests still green.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
Internal FastVideo declares two LTX-2 SamplingParam variants:
LTX2SamplingParam (mod=3.0/rescale=0.7/stg=1.0 — full LTX-2) and
LTX2DistilledSamplingParam (mod=1.0/rescale=0.0/stg=0.0). Dreamverse
production runs against LTX2-Distilled-Diffusers, so the distilled
defaults are what the runtime should land on.
The public package only has the single SamplingParam class, and its
class-level defaults for these knobs were the *full* LTX-2 values.
Result: when Dreamverse (or any other consumer) called
``SamplingParam.from_pretrained("FastVideo/LTX2-Distilled-Diffusers")``,
the LTX2_DISTILLED preset's defaults dict didn't override these
fields (it didn't list them) so they fell through to the class
defaults — silently picking up *full*-model CFG / modality / rescale /
STG instead of the distilled ones. Streamed video output diverged
from the internal-ui reference.
Switch the class defaults to the distilled values (1.0 / 0.0 / 0.0).
The LTX2_BASE preset already overrides them explicitly to 3.0/0.7/1.0
in its ``defaults`` dict, so users selecting the full-LTX-2 preset
keep the full-model behavior.
The numerical alignment harness still reproduces the same 42.72 dB
PSNR / 52.87% exact pixel match because that test pinned the values
explicitly via legacy kwargs; this fix moves the *default* path on
the public side onto the same numerical track.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
Add the typed event dataclasses VideoGenerator.generate_async will
emit, plus a VideoResult alias for the public docs:
* VideoProgressEvent / VideoPartialEvent / VideoFinalEvent
* VideoEvent = union of the three
* VideoResult = GenerationResult
Re-exported from fastvideo.api so the Dynamo backend package, the
streaming server, and the stateless OpenAI server share one set of
names.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
