The public LTX-2 path was previously running full bf16 even when
callers set ``pipeline_config.dit_config.quant_config = FP4Config()``,
because:
* the DiT used plain ``torch.nn.Linear`` for FP4-eligible projections
(``attn1/attn2.to_q/k/v/out``, ``audio_to_video_attn``,
``video_to_audio_attn``, FFN ``fc_in``/``fc_out``), so
``LinearBase.quant_method.apply`` was never reached;
* the loader did not call ``convert_model_to_fp4`` after weights
loaded, so even matched layers had no ``_fp4_weight*`` buffers;
* ``QuantizationMethods`` did not register ``"FP4"``, so the typed
``engine.quantization.transformer_quant`` surface had no way to
select it.
This commit completes the inference wire-up through the existing
``fastvideo.layers.quantization`` registry — no parallel pathway:
* swap ``nn.Linear`` → ``ReplicatedLinear`` for the LTX-2 attention
``to_q``/``to_k``/``to_v``/``to_out``/``to_gate_compress`` and the
FFN ``fc_in``/``fc_out`` (``GELUApprox.proj`` and ``FeedForward``
``project_out``). Other linears (timestep MLP, caption proj,
patchify, AdaLN scale/shift) stay ``nn.Linear`` to match internal.
* port ``_supports_prequantized_input`` and
``_linear_project_with_optional_prequant`` so the attention forward
quantizes input once and reuses the ``(x_fp4, x_scale, x_global_sf)``
tuple across q/k/v projections — matches internal exactly.
* plumb ``quant_config`` and ``prefix`` through
``BasicAVTransformerBlock`` → ``_init_transformer_blocks`` →
``LTXModel`` → ``LTX2Transformer3DModel`` so each
``ReplicatedLinear`` gets the correct
``ltx2.blocks.<i>.<sub>.<proj>`` prefix that
``FP4Config.get_quant_method`` matches against.
* register ``"FP4"`` in ``QuantizationMethods`` literal +
``get_quantization_config`` registry so typed
``engine.quantization.transformer_quant: "FP4"`` resolves to a
concrete ``FP4Config()`` instance.
* add ``transformer_quant`` carrier on ``FastVideoArgs`` and
``__post_init__._apply_transformer_quant`` to pin the resolved
quant config onto ``dit_config.quant_config`` (without
overwriting an explicit setter on the dit_config).
* add ``_maybe_convert_model_to_fp4`` in ``fsdp_load.py`` that
walks the loaded model once and registers
``_fp4_weight*``/``_fp4_alpha``/``_weight_global_sf`` buffers on
layers whose ``quant_method`` is ``FP4QuantizeMethod``. flashinfer
is imported lazily inside ``convert_model_to_fp4`` so this is a
no-op on hosts without the FP4 kernels.
* restore ``LinearBase`` fallback to ``UnquantizedLinearMethod`` when
``quant_config.get_quant_method`` returns ``None`` — ``FP4Config``
only tags a curated subset of layers, and the previous
``assert quant_method is not None`` would crash any non-tagged
layer that received a quant_config.
Non-FP4 callers are unaffected: ``UnquantizedLinearMethod.apply``
runs the standard ``F.linear`` path, and parameter names
(``to_q.weight``, ``to_out.0.weight`` from ``nn.ModuleList``) are
identical to the previous ``nn.Sequential`` layout.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
Extend the typed inference API so callers can drive torch.compile per
component without falling back to the legacy flat-kwargs pathway.
* ``CompileConfig`` now exposes ``vae_enabled``, ``audio_vae_enabled``,
``dit_kwargs``, ``text_encoder_kwargs``, ``vae_kwargs``,
``audio_vae_kwargs`` alongside the existing ``enabled`` /
``text_encoder_enabled`` switches. The master ``backend``/
``fullgraph``/``mode``/``dynamic``/``extras`` continue to apply to
every compiled submodule unless a per-component kwargs dict is
non-empty (in which case it overrides entirely — matches the
FastVideo-internal precedent).
* Add the matching runtime carrier fields on ``FastVideoArgs``:
``enable_torch_compile_text_encoder/vae/audio_vae`` plus
``torch_compile_kwargs_dit/text_encoder/vae/audio_vae``. These let
the legacy flat-kwargs path round-trip through the typed config.
* ``api/compat.legacy_from_pretrained_to_config`` now lifts the new
flat kwargs into ``CompileConfig``, and
``generator_config_to_fastvideo_args`` writes them back out so
``VideoGenerator.from_pretrained(model_path, ...)`` callers keep
working unchanged.
No behavior change yet — these surfaces are ports of carriers only.
The composed pipeline base will start consuming them in the next
commit.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
Three small parity fixes uncovered while comparing the LTX-2 distilled
path against FastVideo-internal:
* `FastVideoArgs.__post_init__` now calls `_resolve_refine_args()` to
copy the public-facing generic `refine_*` knobs onto their
`ltx2_refine_*` runtime carriers (mirrors internal lines 303-322).
Without this, callers that set `refine_lora_path=...` via the typed
CLI/config surface would silently drop the value, surfacing later as
the "applied to 0 layers" warning.
* Revert the patch-noise sampler in `_randn_ltx2_video_latents` from
`randn_tensor` back to `torch.randn` to bit-match internal under
single-generator inference. `randn_tensor` is identical for a single
`torch.Generator` but diverges for `list[Generator]` (per-sample
seeds), which is the only place the two paths could disagree.
* Classify the 19 previously-unclassified `refine_*` / `ltx2_refine_*`
/ `ltx2_audio_latent_path` / `ltx2_images` / `ltx2_image_crf` /
`ltx2_conditioning_latent_*` / `ltx2_video_conditions` fields in the
schema-parity inventory yaml so the parity test passes.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
docs/design/server_contracts/dynamo.md documents the integration shape
the ai-dynamo/dynamo native backend package consumes. Per the refactor
decision, no Dynamo code lives in FastVideo — the backend package
ships in the Dynamo repo at components/src/dynamo/fastvideo/ with the
layout mirroring components/src/dynamo/sglang/.
Captured here:
* the 8-symbol public surface Dynamo imports (VideoGenerator plus
ContinuationState, GenerationRequest, SamplingConfig, InputConfig,
OutputConfig, VideoEvent, VideoResult) and what is available today
vs. what lands in PR 7.10
* target backend package layout matching the sglang template
* full NvCreateVideoRequest <-> GenerationRequest mapping table and
VideoFinalEvent <-> NvVideosResponse reverse mapping
* worked examples: aggregated handler (works today with generate_video
+ asyncio.to_thread), streaming handler (post-PR 7.10 via
generate_async), health check payload (swapping to
default_health_check_request() after PR 7.10)
* init function sketch, registration (ModelType::Videos fast path),
args adapter
* contract guarantees that let the adapter be written once and not
chase FastVideo drift
* explicit no-import list for the adapter
Also wires the server_contracts/ section into mkdocs.yml under Design.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
docs/design/server_contracts/ lands the contract reference set that PR
8 locks down ahead of the streaming + Dynamo upstreams. This commit
adds the index page plus the OpenAI HTTP reference:
* endpoint catalogue (/v1/videos/generations, list, status, content,
images, models, health)
* VideoGenerationsRequest shape with SGLang-compatible extensions
* merge precedence (body > default_request > hardcoded fallback)
with a pointer to the explicit-path tracking mechanism that makes
it work
* continuation-state roundtrip on the stateless surface
* HTTP error code table
* explicit non-goals: flat legacy kwargs, private Dreamverse fields,
raw tensor payloads — all excluded from this boundary by design
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
