Makes the 'Full A/V' LTX-2.3 deliverable reachable from the user-facing entrypoint, not just the
engine. A model advertising TEXT_TO_VIDEO_SOUND (LTX-2.3) now auto-issues a T2VS request, so generate()
/ generate_video() return BOTH modalities in one joint pass:
- VideoGenerator stores the resident instance + a supports_av flag (from card.capabilities); generate()
gains want_audio (None=auto-by-capability, True/False to force) and routes T2V vs T2VS+{video,audio}.
- _result saves the stereo waveform as a sibling .wav at the vocoder's REAL rate (24000) — read off the
built audio_vae adapter (TorchLTX2AudioVAE.sample_rate = Vocoder.output_sample_rate), since the
AudioArtifact default rate is a placeholder. Populates GenerationResult.audio/.audio_sample_rate and
extra['audio_path']. scipy IEEE-float WAV; [channels,samples] auto-transposed.
- Rewrote v2_basic_ltx2_3_distilled.py: registry routes to build_ltx2_3_card (its own joint T2VS A/V
card, not the LTX-2 base/2-stage card); the example prints both the mp4 and the wav.
- GPU-verified via the convenience API: ev.mp4 + ev.wav (24000 Hz, stereo 61920x2, nonzero, std 0.043).
CPU mini green (202 passed, 2 skipped); engine/program/toy paths untouched (test_ltx2_av pins 44100).
Per review: dispatch should be a directly-mapped HF-string -> card registry (like fastvideo), shared
by every entrypoint (VideoGenerator + a future CLI / server), not buried in the generator.
- New v2/registry.py mirrors fastvideo's fastvideo/registry.py hybrid resolution: (1) exact HF repo id
in an explicit ModelEntry registry [PRIMARY — correct per-model card/capabilities, and the only way to
split same-architecture capability variants like Wan2.1 T2V vs the i2v 'InP' 1.3B], (2) short repo-name
match, (3) architecture inference [FALLBACK — local paths / unregistered repos]. resolve(model_path[,
root]) is the single shared entry point.
- video_generator.py: moved _read_arch_signature/_select_builders into the registry; from_config now
calls resolve() — registered ids resolve with no config read, else arch inference on a cheap *.json
snapshot. Reconciles the earlier 'no brittle table' refactor with the 'map hf string -> card' ask: one
clean registry + a fallback, not three coupled structures.
- Verified: registry resolves exact-id / short-name / arch-fallback / unregistered correctly; CPU suite
green (202+2); wan21 GPU smoke generates via the new resolve path.
- LTX-2 base (Davids048) and LTX-2.3-Distilled both generate real video (inter-frame motion 4.5 / 6.6)
via the single-stage base card — moved to Working (7 models now verified).
- Environment: the aarch64 venv was rebuilt for x86 (torch 2.11.0+cu128) + re-validated (CPU suite green,
wan21 + LTX-2 base/2.3 generate).
- Remaining Wan+LTX-2 ports documented with concrete mechanisms: Wan2.2-i2v (SigLIP image_encoder +
VAE-encode first-frame + concat-mask -> larger-in_channels i2v DiT), TurboWan (RCMScheduler consistency
loop), Lucy-Edit (Wan v2v via VideoVAEEncodingStage), FastWan (VSA, env-blocked: no nvcc).
- v2_basic_ltx2_3_distilled.py: LTX-2.3-Distilled routes to the single-stage base card (no
spatial_upsampler) via the arch dispatch; pass few steps for the distilled schedule.
- V2_PORTING_STATUS.md: A14B moved to Working (CPU expert offload, 60GB peak); LTX-2 base + 2.3 added
(code-complete, GPU re-verify pending); Environment-status note on the mid-session aarch64->x86 host
reschedule that blocks GPU re-verify.
Within the bounded Wan+LTX-2 scope:
- Wan2.2-A14B MoE now GENERATES on a single 80GB GPU via CPU offload (TorchWanDiT offload_group):
the two 14B experts live on CPU and only the active one is swapped onto the GPU at the boundary-
timestep transition (a single swap, not per-step). GPU-verified: 60GB peak (vs 79GB OOM), produced
wan22_a14b_lion.mp4 (17x480x832, std 54.7, motion 8.36). Single-expert Wan stays resident.
- LTX-2 base (single-stage) port: build_ltx2_base_card + build_ltx2_base_program reuse the LTX-2
adapters at FULL latent res with a request-driven many-step flow-match (LTX2DenoiseLoop full_res/
request_steps/base_flow_sigmas; distilled base/refine path preserved via False defaults). The SAME
single-stage card serves LTX-2.3-Distilled (also single-stage: no spatial_upsampler) — dispatched by
the new has_spatial_upsampler discriminator in _select_builders. v2_basic_ltx2.py added; VideoGenerator
gains shutdown() for API parity.
- Fixes: from_config 'os' scoping (shadowed module import); LTX-2 upsampler per_channel_statistics
source (the AE's .decoder, not the top-level module).
Verification status: A14B offload, the upsampler, and the arch-dispatch refactor were GPU-verified
earlier this session. LTX-2 base/2.3 are CPU-verified (cards/programs build, dispatch routes, schedule
correct); their GPU smoke tests were pending when the box was rescheduled aarch64->x86 mid-session,
which broke the aarch64 venv (numpy/torch unrunnable on x86) — GPU re-verify blocked on the env.
Two reviewer asks + the next Wan port, all within the bounded Wan+LTX-2 scope:
1. Architecture-driven dispatch (replaces the HF-id table + substring fallback): from_config reads the
checkpoint's pipeline/transformer/VAE class names (+ z_dim, transformer_2) and picks the v2 card via
_select_builders — mirroring fastvideo's get_pipeline_config_cls_from_name. Resolves local paths /
renamed repos / new distilled variants of a known arch with no table edits, and cleanly REJECTS
FastWan (detected by WanDMDPipeline) with a precise message instead of a confusing load crash.
2. Real LTX-2 spatial upsampler (was a nearest-neighbor np.repeat stand-in): new 'upsampler' component
kind -> TorchLTX2Upsampler wraps the real LTX2LatentUpsampler and applies the repo's upsample_video
(un_normalize via the VAE decoder's per_channel_statistics -> learned 2x upsample -> normalize). CPU
keeps ToyUpsampler (np.repeat) via the factory terminal, so the program calls
component('spatial_upsampler').upsample(...) on both backends with no device branch. GPU-verified:
9x512x768, std 70.5, motion 6.51.
3. Wan2.2-T2V-A14B MoE card (build_wan22_a14b_card): two WanTransformer3DModel experts +
BoundaryTimestepRouting @0.875. GPU-verified that both experts denoise; OOMs in VAE decode on one
80GB GPU (~70GB resident) — upstream offloads the DiT for MoE; documented as offload-blocked.
CPU suite green (202 passed, 2 skipped). FastWan root-caused (non-strict load of VSA gate_compress +
VSA not built); roadmap (V2_PORTING_STATUS.md) updated with the bounded scope + per-model status.
- VideoGenerator gains the convenience surface most basic examples use: from_pretrained(model,
num_gpus/*_cpu_offload/...) and generate_video(prompt, sampling_param=, **kwargs) -> result, on top
of the typed from_config/generate. Accepts SamplingParam.
- v2 examples matching the upstream convenience-API examples for the verified models: v2_basic.py
(Wan2.1) and v2_basic_self_forcing_causal.py (SF-causal).
- V2_PORTING_STATUS.md: honest per-family roadmap. Working: wan21, wan_causal, ltx2-distilled. Each
further model needs per-model work (FastWan: WanDMD to_gate_compress param mapping; TurboWan: RCM
consistency sampler; Wan2.2: MoE card; LTX2 base/i2v; new families: new cards/adapters; gated Flux2 /
local GEN3C / audio StableAudio / interactive MatrixGame blocked in this env).
SolitaryThinker