From 799bf4ef63e49fe3ab6220809d07979a08b89110 Mon Sep 17 00:00:00 2001 From: Jason Hoku Date: Wed, 4 Nov 2026 19:50:58 -0800 Subject: [PATCH] docs: text-encoder batching measured on MOSS-Image: slower and not image-identical, default stays off (Roadmap 29) te_batch_check on the 5060 Ti (A1/A2 unbatched, B te_batch_size 8): encoder outputs match (relL2 5e-5, shapes equal) but B ran at 0.66x the per-prompt speed (padding to the longest prompt in each batch of 8 on the w4a8 Qwen3.5-9B encoder) and the sampler amplified the float noise into different images (mean abs diff 4-52/255 against a 0.000 A1-A2 floor; the un-batched leftovers were bit-identical). Co-Authored-By: Claude Fable 5.1 --- README.md | 6 +++--- Roadmap.md | 3 +-- web/conf_builder/conf-builder-config-management.js | 2 +- 3 files changed, 5 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index b5aa1cb..f44a5d9 100644 --- a/README.md +++ b/README.md @@ -194,7 +194,7 @@ If you'd rather build the graph yourself: * **LoRA Compatibility Detection:** Automatically detects and skips incompatible LoRAs with detailed error reporting, preventing log spam. * **Graceful Interruption:** Cancel button now stops ALL remaining jobs (not just current generation) and saves all completed work including pending remote VAE decodes. * **VAE Batching:** Includes a `vae_batch_size` input to batch decode images, significantly speeding up large grid runs. -* **Text-Encoder Batching:** The `te_batch_size` Run Setting encodes several prompts per text-encoder pass (MOSS-Image, Krea 2, Qwen-Image), cutting pre-encode time on prompt-heavy grids. Off by default. +* **Text-Encoder Batching:** The `te_batch_size` Run Setting encodes several prompts per text-encoder pass (MOSS-Image, Krea 2, Qwen-Image), intended to cut pre-encode time on prompt-heavy grids. Off by default, and the live check argues for leaving it off: Measured 2026-11-04 on a 5060 Ti with MOSS-Image (Qwen3.5-9B w4a8): a batch of 8 was 1.5x SLOWER per prompt than one-by-one (padding to the longest prompt in the batch) and the images were not identical to the unbatched run (the encoder outputs match to relL2 5e-5, which the sampler amplifies into visibly different images), so leave it at 1 unless you measure a gain on your encoder. * **Live Dashboard Updates:** Configure `flush_batch_every` to update the dashboard incrementally (e.g., every 4 images) instead of waiting for the entire batch to complete. * **Remote VAE Support:** Offload VAE decoding to remote servers (HuggingFace endpoints or local) for 20-30% faster generation and lower VRAM usage. * **VAE Selection & Iteration:** Test multiple VAEs per config — add individual VAEs from a searchable dropdown or expand entire VAE folders. Each VAE becomes a dimension in the Cartesian grid. @@ -417,7 +417,7 @@ This suite consists of three nodes found under the `sampling/testing` category: - `12-24`: For 24GB+ VRAM - `-1`: Decode all images at once (fastest, but risky) -* **`te_batch_size`** (Run Setting "Text Encoder Batch Size"): How many prompts to encode per text-encoder forward. `1` (default) = off. +* **`te_batch_size`** (Run Setting "Text Encoder Batch Size"): How many prompts to encode per text-encoder forward. `1` (default) = off. Measured 2026-11-04 on a 5060 Ti with MOSS-Image (Qwen3.5-9B w4a8): a batch of 8 was 1.5x SLOWER per prompt than one-by-one (padding to the longest prompt in the batch) and the images were not identical to the unbatched run (the encoder outputs match to relL2 5e-5, which the sampler amplifies into visibly different images), so leave it at 1 unless you measure a gain on your encoder. - Only MOSS-Image, Krea 2 and Qwen-Image encoders batch; every other encoder ignores it. - Weighted prompts and `AND` / `CAT` / `AVG()` / `BREAK` / `COUPLE` prompts are still encoded one at a time. - The first batch is self-checked against a single-prompt encode; on any mismatch the run falls back to one-by-one automatically. A CUDA OOM halves the batch and retries. @@ -1300,7 +1300,7 @@ Create a consistent quality stack using the Config Builder: add 3 LoRAs manually 2. Use `add_random_seeds_to_gens: 2-3` to evaluate model consistency. ### For MOSS-Image / Krea 2 / Qwen-Image grids with many prompts -1. Try `te_batch_size: 8` (Run Settings > Text Encoder Batch Size) to cut pre-encode time; lower it if you see an OOM retry message. +1. `te_batch_size` (Run Settings > Text Encoder Batch Size) is NOT a free win: on MOSS-Image a batch of 8 measured 1.5x slower and not image-identical (see Performance Settings). Try it only on an encoder you have measured; lower it if you see an OOM retry message. ### For Memory-Constrained Systems 1. Lower `vae_batch_size` to 1-2. diff --git a/Roadmap.md b/Roadmap.md index 7d43cc2..753ec7c 100644 --- a/Roadmap.md +++ b/Roadmap.md @@ -180,8 +180,7 @@ README is 2000+ words; most users won't read it. Efficiency Nodes grew via commu ## Open follow-ups (11/4/26) -#### **29. Text-Encoder Batching: live check, then flip the default** -Verify `te_batch_size` on the 5060 Ti with the real MOSS-Image, Krea 2 and Qwen-Image encoders, then decide whether the default of 1 (off) should change. +#### ~~**29. Text-Encoder Batching: live check, then flip the default**~~ DONE (11/4/26): checked on the 5060 Ti with MOSS-Image via docs/tools/te_batch_check: encoder outputs match (relL2 5e-5, shapes equal) but a batch of 8 was 0.66x the per-prompt speed (padding to the longest prompt) and the sampler turned the float noise into visibly different images (mean abs diff 4-52/255 vs a 0.000 A1-A2 floor). The default stays 1 (off); documented in README. Krea 2 / Qwen-Image not measured. #### **30. Distributed companion parity** **Done (wave 3, `feat/dist-parity`):** `job_order`, the preview pass and image metrics now work in distributed mode (master-owned plan, `populate_from_plan`, `dist_worker` on every item). DONE (feat/dist-open): rejects made mid-run are re-checked at claim time, claim affinity, Start At Job in distributed mode, LTX rendered on the master (never on workers). Open: live two-machine verification of the whole set. diff --git a/web/conf_builder/conf-builder-config-management.js b/web/conf_builder/conf-builder-config-management.js index 7a07eb3..012f89d 100644 --- a/web/conf_builder/conf-builder-config-management.js +++ b/web/conf_builder/conf-builder-config-management.js @@ -7312,7 +7312,7 @@ export function renderRunSettingsSection(node, container) { _addRunSetting(content, { stateKey: "vae_batch_size", label: "VAE Batch Size", kind: "int", min: -1, max: 64, defaultValue: 1, tooltip: "How many images to encode/decode per VAE pass. Lower = less VRAM. -1 = process all at once. Default: 4." }); _addRunSetting(content, { stateKey: "te_batch_size", label: "Text Encoder Batch Size", kind: "int", min: 1, max: 32, defaultValue: 1, - tooltip: "Encode this many prompts per text-encoder pass (MOSS-Image, Krea 2, Qwen-Image only; other encoders ignore it). 1 = off. 8 is a good value once verified; weighted/AND/CAT/AVG/BREAK/COUPLE prompts are still encoded one by one." }); + tooltip: "Encode this many prompts per text-encoder pass (MOSS-Image, Krea 2, Qwen-Image only; other encoders ignore it). 1 = off (recommended: on MOSS-Image a batch of 8 measured 1.5x slower and not image-identical). Weighted/AND/CAT/AVG/BREAK/COUPLE prompts are still encoded one by one." }); _addRunSetting(content, { stateKey: "add_random_seeds_to_gens", label: "Additional Random Seeds", kind: "int", min: 0, max: 100, defaultValue: 0, tooltip: "Generate this many extra images per config using additional random seeds. 0 = disabled." });