[docs] v2: LTX-2.3 example + roadmap (A14B offload working, base/2.3 ported, env status)

- v2_basic_ltx2_3_distilled.py: LTX-2.3-Distilled routes to the single-stage base card (no
  spatial_upsampler) via the arch dispatch; pass few steps for the distilled schedule.
- V2_PORTING_STATUS.md: A14B moved to Working (CPU expert offload, 60GB peak); LTX-2 base + 2.3 added
  (code-complete, GPU re-verify pending); Environment-status note on the mid-session aarch64->x86 host
  reschedule that blocks GPU re-verify.
This commit is contained in:
SolitaryThinker
2026-06-18 09:41:17 +00:00
parent 8b343db604
commit 0a2c985b56
2 changed files with 43 additions and 4 deletions
+16 -4
View File
@@ -18,6 +18,15 @@ some are blocked by the environment (gated weights, local-only weights, non-vide
The v2 bring-up runs **single-GPU, resident, on the `TORCH_SDPA` backend** (no fastvideo-kernel / VSA /
FP4), so distilled models that depend on VSA/SLA/FP4 kernels or non-flow-match samplers won't match.
(Wan2.2-A14B MoE is the exception to "resident": its two 14B experts are CPU-offloaded and swapped onto
the GPU one at a time at the boundary-timestep transition, so it fits a single 80GB GPU.)
**Environment status (current):** the box was rescheduled from an aarch64 host to an x86_64 host
mid-session (`ipp1a1…` → `ipp2-0493`; `uname -m` = x86_64). The `.venv` is aarch64 wheels
(numpy/torch/fastvideo/CUDA) which cannot execute on x86 (numpy `_core` is aarch64; `uv` hits a
missing-aarch64-loader binfmt error). GPU verification is blocked until the box returns to aarch64 (the
existing venv then works as-is) or the venv is rebuilt for x86. Every model marked *verified* above was
GPU-confirmed **before** the flip; the ‡ models are code-complete, pending GPU re-verify.
## Working today (verified, real video) — committed on `v2`
| Official example(s) | Model | v2 card | v2 example |
@@ -26,19 +35,22 @@ FP4), so distilled models that depend on VSA/SLA/FP4 kernels or non-flow-match s
| `basic_self_forcing_causal.py` | `wlsaidhi/SFWan2.1-T2V-1.3B-Diffusers` | wan_causal | `v2_basic_self_forcing_causal.py`, `v2_basic_new_api.py` |
| `basic_ltx2_distilled.py`, `basic_ltx2_distilled_fast_profile.py` | `FastVideo/LTX2-Distilled-Diffusers` | ltx2 — real `LTX2LatentUpsampler` between base/refine (un_normalize→learned 2× upsample→normalize) | `v2_basic_new_api.py` |
| `basic_wan2_2_ti2v.py` (T2V branch) | `Wan-AI/Wan2.2-TI2V-5B-Diffusers` | wan2.2-ti2v | `v2_basic_wan2_2_ti2v.py` |
| `basic_wan2_2.py` | `Wan-AI/Wan2.2-T2V-A14B-Diffusers` (MoE) | wan2.2-a14b + **CPU expert offload** (one 14B expert resident at a time, swapped at the boundary → 60GB peak) | `v2_basic_wan2_2.py` |
| `basic_ltx2.py` ‡ | `Davids048/LTX2-Base-Diffusers` | ltx2 base (single-stage, request-driven steps) | `v2_basic_ltx2.py` |
| `basic_ltx2_3_distilled.py` ‡ | `FastVideo/LTX-2.3-Distilled-Diffusers` | ltx2 base card (single-stage; pass few steps) | `v2_basic_ltx2_3_distilled.py` |
\* `basic_mps.py` (Apple MPS) and `basic_ray.py` (ray executor) are device/executor variants of the
same Wan2.1 model — the model itself is covered by wan21; those runtime modes are out of v2 scope.
‡ Code-complete + CPU-verified (cards/programs build, dispatch routes, schedule correct); GPU re-verify
pending the env (see **Environment status** below).
## Needs per-model work (architecture/sampler matches partially)
| Official example(s) | Model | What v2 needs |
|---|---|---|
| `basic_dmd.py`, `basic_dmd_new_api.py` | FastWan2.1-T2V-1.3B (WanDMDPipeline) | **Blocked (env):** loading needs a *non-strict* load to skip the VSA-only `to_gate_compress` (root cause: the checkpoint key mis-maps under the generic Wan path, and `TransformerLoader` is strict except Cosmos2.5 — `component_loader.py:1005`). Even loaded, the DMD model is trained for VSA, which is **not built here (no nvcc)**, so SDPA output would be degraded. Needs the fastvideo-kernel VSA build + a non-strict WanDMD load. |
| `basic_turbodiffusion*.py` | TurboWan2.1/2.2 (RCM) | **New sampler:** Reparameterized Consistency Model — v2 has no consistency loop (flow-match Euler won't match). |
| `basic_wan2_2.py` | Wan2.2-T2V-A14B (MoE) | **Card + MoE routing DONE + GPU-verified** (`build_wan22_a14b_card`: two `WanTransformer3DModel` experts + `BoundaryTimestepRouting` @0.875; both experts denoised correctly) but **OOMs in VAE decode on one 80GB GPU** (~70GB resident: 2×14B + UMT5). Upstream itself runs A14B with `num_gpus=2` + `dit_cpu_offload=True` ("DiT need to be offloaded for MoE"). Needs the resident path to offload an expert / free UMT5. `v2_basic_wan2_2.py` added. |
| `basic_wan2_2_i2v.py`, `basic_wan2_2_Fun.py` | Wan2.2-I2V / Fun | I2V needs image conditioning (+ A14B offload); Fun needs a control input. |
| `basic_ltx2.py` | `Davids048/LTX2-Base-Diffusers` | Base (non-distilled): base sigma schedule + more steps (the v2 ltx2 card hardcodes the distilled 8+3). |
| `basic_ltx2_3_distilled_i2v*.py` | LTX-2.3-Distilled I2V | Image conditioning on the ltx2 program (encode + condition). |
| `basic_wan2_2_i2v.py`, `basic_wan2_2_Fun.py` | Wan2.2-I2V / Fun | I2V needs an image-encode node + conditioning (the Wan adapter already accepts `encoder_hidden_states_image`; reuse the A14B offload); Fun needs a control input. |
| `basic_ltx2_3_distilled_i2v*.py` | LTX-2.3-Distilled I2V | Image conditioning on the ltx2 single-stage program (encode the input image + condition the denoise). |
| `basic_lucy_edit.py` | `decart-ai/Lucy-Edit-Dev` | Wan-family video-edit: input-video conditioning. |
## Out of current scope — new families (kept as an upstream reference map only)
@@ -0,0 +1,27 @@
"""v2 port of basic_ltx2_3_distilled.py — LTX-2.3 Distilled (single-stage) through the v2 VideoGenerator.
LTX-2.3-Distilled is single-stage (no spatial upsampler, unlike LTX-2.0 distilled's two-stage), so the
architecture-driven dispatch (v2/video_generator.py:_select_builders, via has_spatial_upsampler=False)
routes it to the SAME single-stage card as LTX-2 base (build_ltx2_base_card). Being distilled, it wants
FEW steps — pass a small num_inference_steps for the few-step schedule.
NOTE: the v2 single-stage loop currently builds a request-driven *linspace* flow-match schedule
(base_flow_sigmas); the distilled-tuned few-step schedule is a follow-up. GPU re-verify of this example
is pending the env (the box was rescheduled aarch64->x86 mid-session — see V2_PORTING_STATUS.md).
"""
from v2 import VideoGenerator
PROMPT = "a hummingbird hovering beside a red flower, wings blurred, macro, cinematic, highly detailed"
def main() -> None:
generator = VideoGenerator.from_pretrained("FastVideo/LTX-2.3-Distilled-Diffusers", num_gpus=1)
video = generator.generate_video(
prompt=PROMPT, output_path="v2_video_samples_ltx2_3", output_video_name="ltx2_3_hummingbird",
save_video=True, num_frames=9, height=512, width=768, num_inference_steps=8)
print(f"Output: {video.video_path}")
generator.shutdown()
if __name__ == "__main__":
main()