The pixel path (vae + source_image) can slow sampling badly on VRAM-tight
setups. The encode itself is not the cost — px_cache already means it runs
once per target resolution — the problem is *when* it runs.
vae.encode reaches load_models_gpu(..., memory_required=), which calls
free_memory() with an empty keep_loaded, so it partially unloads whatever is
already resident. Firing from inside the DIFFUSION_MODEL wrapper makes that
victim the diffusion model the sampler is mid-run on, and nothing re-expands
it (sampler_helpers loads once, before the loop). Every remaining step then
streams weights from CPU. Setups with headroom for the model and the VAE at
once never see it, which is why reports are inconsistent.
New optional target_latent input: wire the same latent that feeds
KSampler.latent_image and patch() knows the target grid up front, priming
px_cache at node-execution time. That restores the ordinary
VAEEncode -> KSampler order where the sampler evicts the VAE, not the reverse.
- target_latent is declared last in optional, so no existing socket index
moves and workflows saved before this reload with their links intact.
- Unwired, behavior is unchanged; the node prints a note instead.
- A target_latent that disagrees with the sampled resolution misses the cache
and re-encodes at the real size — correct, just not fast — and warns.
- The pixel-path announce moved off `if not px_cache`, which is wrong once
the cache is pre-primed, to an explicit one-shot flag.
- Shipped workflow wires EmptySD3LatentImage into target_latent.
tests/test_pre_encode.py pins the observable behavior: encode-call count at
patch() return vs after driving the wrapper, two-ref priming, the mismatch
fallback, and target_latent being inert on the latent path. It installs a
minimal comfy stub when the real one is not importable, so unlike the
existing test it also runs standalone outside a ComfyUI checkout.
Worth noting for anyone benchmarking: this removes a per-step penalty, so it
only shows up over many steps. A 1-step run is almost entirely fixed overhead
and will show no difference — the source encode is a fixed cost that moves
rather than disappears.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
