The official kandinsky-5 repo zeros the visual_cond block for I2V while the
diffusers port copies the image latent into it. A same-seed A/B on the
Pro-distilled Diffusers export shows the copy is required (zeroing smears
faces mid-video), so keep the diffusers semantics and record the evidence.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
get_model_info again skips model_index.json resolution entirely when
override_pipeline_cls_name is set, so overrides keep working for local
checkpoints without a diffusers model_index (and trigger no downloads).
Replace the nine copy-pasted kandinsky5 detectors with a parameterized
factory (identical matching semantics) and add a dedicated I2V Lite distilled
entry + preset so those checkpoints get distilled sampling defaults
(guidance 1.0 / 16 steps) instead of the sft fallback's 5.0 / 50.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Match the official kandinskylab/kandinsky-5 RNG order for I2V seed parity:
image encoding now runs after latent preparation and places the conditioning
latent itself, so the initial noise is the seeded generator's first draw
(upstream leaves the image-latent sample unseeded; drawing it second from the
same generator keeps FastVideo deterministic). The denoising stage now sets
the forward context with NABLA attention metadata per step — which also
routes the dense path through LocalAttention instead of the no-context SDPA
fallback. Resize tensor conditioning images to the requested resolution like
the PIL path. Collapse Kandinsky5DecodingStage to a channels-first permute +
super().forward(), regaining the base class's MPS handling.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
text_encoder_cpu_offload means params rest on CPU between calls but compute
on GPU: FSDP-wrapped encoders (T5/llama/clip) already get this via
CPUOffloadPolicy streaming, but encoders without _fsdp_shard_conditions
(reason1/Qwen2.5-VL) were parked on CPU and, after the encoder-device input
routing added for kandinsky5, silently ran their 7B forward on CPU (crashing
outright with flash-attn installed). Move plain CPU-parked encoders to the
target device for the forward and back afterwards, mirroring the
image-encoder/VAE offload pattern; FSDP (DTensor) encoders keep the existing
input routing. Revert the examples to text_encoder_cpu_offload=True now that
the default works. Verified e2e: T2V/I2V outputs are metric-identical to
GPU-resident runs, with ~12GB lower VRAM during denoising.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Validate NABLA resolutions (divisible by 128) in latent preparation instead
of crashing mid-denoise with a cryptic reshape error after all encoding work.
Encode the I2V conditioning image with a generator-seeded sample instead of
mode() for diffusers seed parity. Restore base DecodingStage behavior the
override dropped (pipeline.add_module on lazy VAE reload,
return_trajectory_decoded) and record trajectory latents in the denoise loop.
Always normalize PIL conditioning images deterministically; keep the
diffusers-style range heuristic only for raw tensors and log the assumption.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Denoising keyed cond_frames off transformer.visual_cond, but official T2V
checkpoints also ship visual_cond=true, so latent frame 0 was never stepped
and decoded as pure noise in every T2V video; key off batch.image_latent
instead. Restore the byte-exact upstream prompt template (typos included):
the checkpoints were trained with it, and ENCODE_START_IDX=129 matches the
upstream template while the corrected wording moves user content to 127,
silently dropping the first user-prompt tokens. Verified against the
checkpoint tokenizer and by end-to-end T2V/I2V generation.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Aryan Kumar