Internal FastVideo declares two LTX-2 SamplingParam variants:
LTX2SamplingParam (mod=3.0/rescale=0.7/stg=1.0 — full LTX-2) and
LTX2DistilledSamplingParam (mod=1.0/rescale=0.0/stg=0.0). Dreamverse
production runs against LTX2-Distilled-Diffusers, so the distilled
defaults are what the runtime should land on.
The public package only has the single SamplingParam class, and its
class-level defaults for these knobs were the *full* LTX-2 values.
Result: when Dreamverse (or any other consumer) called
``SamplingParam.from_pretrained("FastVideo/LTX2-Distilled-Diffusers")``,
the LTX2_DISTILLED preset's defaults dict didn't override these
fields (it didn't list them) so they fell through to the class
defaults — silently picking up *full*-model CFG / modality / rescale /
STG instead of the distilled ones. Streamed video output diverged
from the internal-ui reference.
Switch the class defaults to the distilled values (1.0 / 0.0 / 0.0).
The LTX2_BASE preset already overrides them explicitly to 3.0/0.7/1.0
in its ``defaults`` dict, so users selecting the full-LTX-2 preset
keep the full-model behavior.
The numerical alignment harness still reproduces the same 42.72 dB
PSNR / 52.87% exact pixel match because that test pinned the values
explicitly via legacy kwargs; this fix moves the *default* path on
the public side onto the same numerical track.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
Initial alignment showed PSNR ~12 dB because public's LTX2_BASE preset
sets ``ltx2_modality_scale_video=3.0 / rescale=0.7 / stg_video=1.0``
while internal's default ForwardBatch lands at ``1.0 / 0.0 / 0.0``.
Pinning these explicitly on both runs aligns the denoising mechanics
so the diff measures pipeline parity, not preset divergence.
Result with both sides pinned to mod=1.0/rescale=0.0/stg=0.0:
max_abs_diff = 203
mean_abs_diff = 0.788
rms_diff = 1.864
psnr_db = 42.72 dB
exact pixels = 52.87%
within 1 lvl = 86.94%
within 4 lvls = 97.93%
The ~47% near-but-not-exact bucket is consistent with bf16 reduction
non-determinism on B200 plus the fact that the public LoRA matcher
emitted "applied to 0 layers" for the LTX2-Distilled refine LoRA
(internal side did not). Both runs effectively skip the refine LoRA,
so they share the same effective stage-2 model — the residual diff is
the right ballpark for "same algorithm, different bf16 reduction
schedule".
The typed-API path also sets the same overrides via setattr on the
GenerationRequest so the typed path stays comparable. Once the typed
SamplingConfig grows ltx2-specific fields the setattr workaround can
go.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
LTX-2 SR pipeline now passes module load + stage graph build on the
public side. Each retry of scripts/ltx2_sr_alignment.py surfaced one
more gap; this commit closes them so the pipeline progresses to the
upsample → stage-2 → decode tail.
* harness: PipelineSelection (real class name; PipelineConfig was a typo)
* FastVideoArgs: generic refine_* fields (preferred user-facing API)
* LTX2Pipeline.initialize_pipeline: getattr-with-default for
debug_model_sums / debug_model_detail (public args lack the debug
fields; LTX-2 still flips the env-var toggles when set)
* VAEConfig: use_temporal_scaling_frames default True (LTX-2 latent
prep reads this in the schedule-shift gate)
* LTX2LatentPreparationStage._randn_ltx2_video_latents: switch to
randn_tensor so list-of-generators batches work (matches internal
sampling order)
* ModelRegistry: register LTX2LatentUpsampler in _UPSAMPLERS
* component_loader: register spatial_upsampler / temporal_upsampler
with UpsamplerLoader (was hitting the generic stub loader);
UpsamplerLoader gains LTX-2 fallback when pipeline_config.upsampler_config
isn't a multi-config tuple, plus the 'model.'-prefix-stripping
weight load path the internal version uses
7 retries -> public side now reaches the upsample stage.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
Single script with three modes:
* --label internal --output /tmp/ltx2_sr_internal.pt : runs the legacy
from_pretrained kwargs path and saves frames + metadata. Used against
PYTHONPATH=../FastVideo-internal as the reference run.
* --label public --use-typed-api --output /tmp/ltx2_sr_public.pt : runs
through the new GeneratorConfig + GenerationRequest typed API with
pipeline.preset_overrides["refine"] driving the SR path. Used against
PYTHONPATH=../FastVideo (the public package) for the candidate run.
* --diff --reference R --candidate C : loads both .pt files and reports
shape parity + max_abs / mean_abs / rms / psnr / pixel-bucket
histograms over the uint8 frames.
Pinned alignment fixture: PROMPT + seed + 1088x1920x121x24fps + 8 base
steps + 3 refine steps mirrors basic_ltx2_upscale.py upstream so we
exercise the same code path the user has been generating with.
Deliberately skips FP4 (no quant config) and Dreamverse runtime knobs
to keep the bf16 path clean per request.
The two runs are intentionally invoked in separate processes via
PYTHONPATH overrides because both repos register as the `fastvideo`
package and can't be imported together. The diff script then runs in
either env to load both .pt files and compute the alignment metrics.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
Brings the public LTX2Pipeline into parity with the internal SR-capable
version end-to-end:
* LTX2DenoisingStage gains the stage-2 control surface
(sigmas_override, num_inference_steps_override, force_guidance_scale,
initial_audio_latents_key) plus the i2v conditioning round-trip
(clean_latent + denoise_mask via LTX2_VIDEO_*_KEY) and the audio
refine path. The file is now ~1:1 with the internal stage; only the
i2v_conditioning import path differs (rerouted to the public
ltx2_image_conditioning module).
* LTX2LatentPreparationStage takes vae=, exposes the official LTX-2
noise sampling order via _randn_ltx2_video_latents (patchify -> noise
in token order -> unpatchify), and applies image conditioning when
build_ltx2_image_conditioning returns a state. Matches internal.
* LTX2Pipeline switches its base from ComposedPipelineBase to
LoRAPipeline (refine-LoRA support); create_pipeline_stages adds the
refine_init / upsample / refine_lora / refine_denoising chain when
ltx2_refine_enabled is True; load_modules pulls
spatial_upsampler (and optionally transformer_refine) from the
resolved upsampler path, with model_index.json defaults
(fastvideo_refine_*) feeding the FastVideoArgs.ltx2_refine_*
carriers.
stages/__init__.py re-exports the new refine stages so importers
outside the basic/ltx2 package keep working. 68/68 LTX-2-related tests
still pass after the rewrite (api translation, gpu_pool, contract,
streaming).
Reachability: setting pipeline.preset_overrides["refine"] = {"enabled":
True, ...} and components.upsampler_weights flows through compat.py
into the new ltx2_refine_* args, the pipeline branches, and load_modules
brings in the upsampler. T2V SR is now end-to-end runnable; numerical
alignment harness comes next.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
Lays the runtime-side pieces the LTX2_TWO_STAGE preset will exercise.
Behaviour matches FastVideo-internal 1:1 for the text-to-video SR path;
i2v / continuation conditioning is stubbed with a NotImplementedError
that fires only on those code paths so the public T2V SR run is clean.
* fastvideo/models/upsamplers/ltx2_upsampler.py — verbatim copy of the
internal LTX-2 latent upsampler (PixelShuffleND, BlurDownsample,
SpatialRationalResampler, ResBlock, LatentUpsampler, LTX2LatentUpsampler,
upsample_video). Re-exported via models/upsamplers/__init__.py.
* fastvideo/pipelines/basic/ltx2/stages/ltx2_image_conditioning.py —
minimal i2v-conditioning helpers: the four ForwardBatch.extra keys,
apply_ltx2_gaussian_noiser, post_process_ltx2_denoised, and a T2V-only
build_ltx2_image_conditioning that returns None when the request has
no image inputs / no continuation latents (the SR T2V path).
* fastvideo/pipelines/basic/ltx2/stages/ltx2_refine.py — the three
refine stages adapted to public imports: LTX2RefineInitStage halves
the request resolution and stashes the original target;
LTX2UpsampleStage runs the latent upsampler, optionally re-applies
conditioning, and mixes refine-noise scaled by the stage-2 sigma;
LTX2RefineLoRAStage swaps in a refine-specific LoRA (no-op when path
is unset).
* fastvideo/fastvideo_args.py — add the runtime carrier fields
ltx2_refine_{enabled,upsampler_path,transformer_path,lora_path,
num_inference_steps,guidance_scale,add_noise,noise_path,
audio_noise_path}. The typed public surface (ComponentConfig
upsampler_weights/lora_path + pipeline.preset_overrides.refine) keeps
flowing through compat.py:273-279, which expands the dict into these
ltx2_refine_* kwargs at construction time.
Not yet wired: the LTX2 pipeline's create_pipeline_stages still goes
denoise -> audio_decode -> decode without the refine_init / upsample
/ refine_denoise injection. The remaining piece is teaching
LTX2DenoisingStage to accept sigmas_override / num_inference_steps_override
/ force_guidance_scale / initial_audio_latents_key (the internal
version is 725 lines vs public 395; ~330 lines of stage-2 logic still
need bringing across) and to flip the pipeline graph based on
ltx2_refine_enabled. That follow-up unlocks the numerical alignment run.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
Ports the FP4 quantization config + ops from FastVideo-internal
(fastvideo.layers.quantization.fp4_config). flashinfer is imported
lazily inside _require_flashinfer() rather than at module top so
``import fastvideo`` stays cheap on hosts that don't ship flashinfer
— the FP4 kernels themselves still raise a clear ImportError pointing
at ``pip install flashinfer-python`` when invoked.
Behavior is identical to the internal version:
* FP4Config: LTX-2 layer set (48 transformer blocks across attn1/attn2
/audio_to_video_attn/video_to_audio_attn/ffn plus adaln_single.linear),
layer_profile selects between "base" and "refine" stage subsets.
* FP4QuantizeMethod: per-layer quantize_input + apply with stage-aware
routing for refine-only layers (audio_to_video_attn.to_q,
video_to_audio_attn.to_k, video_to_audio_attn.to_v fall through to
dense F.linear during the base stage).
* convert_model_to_fp4(): pre-quantize weight buffers + alpha onto
module attrs at load time so the forward path doesn't pay the
per-step weight-side quantize cost.
Required for Dreamverse integration: dreamverse-server's GPU worker
runs ``pipeline_config.dit_config.quant_config = FP4Config()`` during
LTX-2 model load, which is the dominant inference path.
test_fp4_config covers the lazy-flashinfer contract: import succeeds
without flashinfer; layer_profile round-trips via from_config; calling
into _require_flashinfer raises ImportError with an actionable
message when flashinfer is missing.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
fastvideo/tests/contract/ guards against drift in the two public-facing
adapter shapes that PR 8 locks down:
* test_dynamo_shape.py: mock Dynamo-style handler implementing the
adapter template from docs/design/server_contracts/dynamo.md. It
imports only fastvideo + fastvideo.api, translates a
Dynamo-shape NvCreateVideoRequest dict into a typed
GenerationRequest, runs it through an async generator that matches
endpoint.serve_endpoint(handler.generate, ...), and round-trips the
continuation-state envelope. A source scan asserts the adapter
function body touches no private FastVideo module.
* test_dreamverse_shape.py: the flat init-time kwarg bag the internal
gpu_pool.py passes today lands entirely on typed GeneratorConfig
fields (no leakage into pipeline.experimental) after PR 6's typed
replacement work. Per-request Dreamverse shape round-trips through
legacy_generate_call_to_request -> normalize_generation_request;
output.return_state (PR 7) stays wired through.
Failures in this suite appear at FastVideo CI — before the Dynamo-side
integration or private Dreamverse adapter see the drift.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
docs/design/server_contracts/dynamo.md documents the integration shape
the ai-dynamo/dynamo native backend package consumes. Per the refactor
decision, no Dynamo code lives in FastVideo — the backend package
ships in the Dynamo repo at components/src/dynamo/fastvideo/ with the
layout mirroring components/src/dynamo/sglang/.
Captured here:
* the 8-symbol public surface Dynamo imports (VideoGenerator plus
ContinuationState, GenerationRequest, SamplingConfig, InputConfig,
OutputConfig, VideoEvent, VideoResult) and what is available today
vs. what lands in PR 7.10
* target backend package layout matching the sglang template
* full NvCreateVideoRequest <-> GenerationRequest mapping table and
VideoFinalEvent <-> NvVideosResponse reverse mapping
* worked examples: aggregated handler (works today with generate_video
+ asyncio.to_thread), streaming handler (post-PR 7.10 via
generate_async), health check payload (swapping to
default_health_check_request() after PR 7.10)
* init function sketch, registration (ModelType::Videos fast path),
args adapter
* contract guarantees that let the adapter be written once and not
chase FastVideo drift
* explicit no-import list for the adapter
Also wires the server_contracts/ section into mkdocs.yml under Design.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
docs/design/server_contracts/ lands the contract reference set that PR
8 locks down ahead of the streaming + Dynamo upstreams. This commit
adds the index page plus the OpenAI HTTP reference:
* endpoint catalogue (/v1/videos/generations, list, status, content,
images, models, health)
* VideoGenerationsRequest shape with SGLang-compatible extensions
* merge precedence (body > default_request > hardcoded fallback)
with a pointer to the explicit-path tracking mechanism that makes
it work
* continuation-state roundtrip on the stateless surface
* HTTP error code table
* explicit non-goals: flat legacy kwargs, private Dreamverse fields,
raw tensor payloads — all excluded from this boundary by design
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
* tests/contract/test_generate_async.py: event ordering (Progress ->
Final); exactly-one-final per request; batch expansion emits one
Final per sub-result; VideoFinalEvent carries frames / full
GenerationResult / continuation_state as expected; health-check
request is minimal and round-trips through normalize_generation_
request; public fastvideo.api exports the new symbols; a mock
Dynamo-style async handler wraps generate_async using only the
public import surface.
* tests/api/test_cli_translation.py: the streaming-dispatch test no
longer expects NotImplementedError (PR 7.5 landed the live server);
it now injects a fake run_server and asserts the CLI hands the
typed streaming ServeConfig through unchanged.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
* VideoGenerator.generate_async(request) yields VideoEvent objects:
one VideoProgressEvent at start, one VideoFinalEvent per resulting
GenerationResult (prompt-batch expansion emits one Final per
sub-result). The aggregated code path runs the existing sync
generate() inside asyncio.to_thread; future work threads per-step
progress events from inside the pipeline's denoise loop without
changing the public contract.
* VideoGenerator.default_health_check_request() returns a 256x256 /
8-frame / 1-step typed GenerationRequest. Dynamo uses it to derive
its health_check_payload without knowing FastVideo internals.
* _final_event_from_result packages GenerationResult into a
VideoFinalEvent, reading the encoded MP4 off disk when the pipeline
wrote one so streaming consumers don't re-encode.
Also: streaming_server.run_server validates streaming config before
loading the generator so the existing ValueError fires before the
expensive model-load path.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
Add the typed event dataclasses VideoGenerator.generate_async will
emit, plus a VideoResult alias for the public docs:
* VideoProgressEvent / VideoPartialEvent / VideoFinalEvent
* VideoEvent = union of the three
* VideoResult = GenerationResult
Re-exported from fastvideo.api so the Dynamo backend package, the
streaming server, and the stateless OpenAI server share one set of
names.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
