ttulttul-ComfyUI-Modal/docs/LEARNINGS.md

Learnings

2026-08-28: A matching protocol number cannot make a stale Vast worker safe

• A newly rented Vast instance can still report the same old runtime fingerprint as a destroyed instance because COMFY_MODAL_VAST_IMAGE is an immutable registry digest; destroying capacity does not rebuild that image.

• The worker staging CLI gained arguments and commands while the remote protocol remained at version 1. Allowing fingerprint drift whenever the protocol matched therefore admitted a worker that could pass readiness but fail model staging.

• Vast readiness now treats every source fingerprint mismatch as a typed drift error. Queue-time acquisition destroys the stale lease, coordinates a single local uv run python scripts/build_vast_worker_image.py --push, streams bounded subprocess output through the existing per-environment setup status, adopts the returned digest in the current process, and retries with fresh capacity.

• Automatic publication must remain fail-closed: missing uv, Docker, registry authentication, a failed build/push, a missing digest, or a second mismatched worker cancels preparation with the manual build command and configuration steps.

• Vast can report a contract as successful and start the worker/watchdog internally while its advertised direct port mapping continues to refuse connections. Preserve both the direct and ssh*.vast.ai proxy endpoints in lease state, try the proxy during every failed direct readiness probe, and persist whichever endpoint first returns valid worker metadata so later operations do not fall back to the dead mapping.

• A custom-node deployment can contain a stale .venv whose interpreter symlink and pyvenv.cfg describe different Python installations. Automatic image publication must invoke the build script directly with sys.executable so neither project discovery nor a presumed virtualenv location can select that copied environment, and failure summaries must skip traceback footers such as <no Python frame> in favor of the substantive error line.

• A long-running ComfyUI process can retain fingerprint A while hot-deployed source on disk has fingerprint B. Pass A into the build subprocess, have it report and compare B before exporting a Docker context, and refuse to publish or rent until ComfyUI restarts; adopting B's digest while retaining A in runtime configuration guarantees a second drift failure.

• The pinned vLLM wheel is large enough that pip's default five resumptions and 15-second socket timeout can be exhausted by an unstable connection during an SSH/Vast worker build. Give the accelerator-wheel layer an extended timeout and bounded resume budget while leaving earlier package layers unchanged so Docker can reuse their successful cache entries.

2026-08-28

• A DOM widget can declare width: 100% and still overflow a resized legacy ComfyUI node when the widget object's cached width field remains fixed. Keep that field undefined outside Vue-node mode so ComfyUI recalculates the widget allocation from the node on every layout, and let the panel stretch within that allocation.
• macOS keyring status -25308 is a recoverable interaction state, not a missing Cloudflare credential. Preserve it as a credential-safe browser error code and expose an explicit recovery action; /usr/bin/security unlock-keychain -u delegates password entry to the system-owned dialog without placing the login password in ComfyUI memory or process arguments.
• A semantic source-tree hash can select a reusable custom-node archive without being the archive's content identity. Object storage verification must use the SHA-256 of the final ZIP bytes; otherwise every safe worker-side upload rejects the payload even though the archive path correctly represents its source tree.
• A local sync-index hit is evidence that the hot layer contains an asset, not evidence that a newly enabled or previously failed backing cache contains it. Schedule the same idempotent R2 write-back on indexed reuse, and traverse entries behind an indexed custom-node manifest when write-back is enabled so existing installations can backfill without clearing their remote volumes.
• A storage helper that runs inside the fingerprinted SSH image cannot execute during queue-time write-back until that image exists. Give the runtime manager an image-only, lifecycle-locked preparation path and call it before the first Docker materializer; later worker startup can reuse the image without racing another build.
• Presigned transfer errors often embed credential-bearing URLs, but suppressing the entire error makes authorization, request, and provider failures indistinguishable. Walk the exception chain on the worker and emit only a fixed category plus a validated numeric HTTP status, never exception text, response bodies, headers, or URLs.
• Intentional queue-preparation cancellation is a control-flow outcome even though its exception derives from RuntimeError. Provider boundaries that translate ordinary runtime failures into validation errors must pass the cancellation type through first, allowing the queue handler to log at INFO, emit execution_interrupted, and return its existing cancelled response without a misleading provider-failure traceback.

2026-08-24

• Prompt assets should be prepared in parallel across concrete execution environments, not across arbitrary components. One bounded worker per environment allows independent Vast and SSH destinations to download concurrently while preserving a single request cache, deterministic component order, and sync-engine safety within each destination.
• A provider-wide setup message is not enough for a multi-environment Configurator: lifecycle events must retain the planned environment identity before a concrete lease exists, then switch to the concrete identity only after readiness. An empty per-environment state must not be interpreted as ready for capacity that still needs acquisition.
• A custom node cannot require per-model registration as its ordinary asset workflow. Treat ComfyUI Manager catalogs, standard Hub cache paths, Hugging Face Git checkouts, browser download provenance, and embedded safetensors URLs as untrusted location hints; require the Hub's exact content digest and size to match the local file, then persist the verified mapping automatically. Preserve provenance-bearing symlink paths until discovery, and use streamed SSH for genuinely originless files.
• Hugging Face Xet is an origin transport, not a global reverse-SHA search service. Preserve a verified mapping from local content SHA-256 to repository, exact commit, filename, and size; let the Vast worker download through huggingface_hub, verify the local content contract again, and atomically publish at the existing remote asset path. Keep unauthorized or unavailable Hub sources recoverable through a streamed SSH fallback, and never place HF_TOKEN in a workflow, registry, launch environment, command line, or log.
• A storage backend can appear to stream over SSH while still buffering the entire input before subprocess.run. Passing a regular file as OpenSSH stdin gives bounded-memory kernel streaming; the remote atomic writer must also copy stdin in chunks rather than calling read() with no bound.
• A configuration node's ordinary output is the wrong transport for queue-time planning UX because ComfyUI does not execute the node until its normal scheduler reaches it. Keep planning authoritative in the queue route, carry the serialized configurator node id in replayable websocket status, and render a non-serialized DOM widget directly on that node. Populate the final assignment table from the accepted queue response and retain it after prompt cleanup so setup progress and placement remain inspectable out of band.

2026-08-23

• Remote capacity acquisition happens before the rewritten prompt enters ComfyUI's native queue, so a one-shot frontend event is not enough to represent liveness. Register prompt preparation in a server-side bridge that augments get_current_queue, get_current_queue_volatile, and get_tasks_remaining, deduplicate it when the native prompt arrives, and publish queue updates on both registration and cleanup.
• Vast can report a failed private-image pull as actual_status=loading even though cur_state and intended_status are both stopped. Preserve those provider fields, surface bounded status messages during every readiness poll, and treat the stopped/stopped combination as terminal instead of waiting for the full startup timeout.

2026-08-22

• Current free VRAM is conservative only if the planner distinguishes unknown GPU consumers from its own idle warm workers. Compare normal placement with an optimistic total-capacity placement; recycle only the labeled idle worker selected by the cost model, then re-probe actual free memory before admission. A running worker is idle only when docker top shows neither an execution relay nor a model-staging process.
• A queue route can return the planner's exact per-host rejection while ComfyUI's synthetic pre-execution error replaces it with a generic ModalQueueError. Preserve the response payload on PromptExecutionError and use its server message for both the node overlay and the synthetic execution_error event.

2026-08-21

• vLLM engine shutdown can finish before CUDA free-memory accounting reflects all released allocations. A post-eviction admission check must ask ComfyUI to release memory again, empty allocator caches, and poll with a bounded deadline. If the threshold still cannot be recovered, classify the container as dirty, retire it, and retry once with a one-use worker affinity; when this follows auto eager-to-throughput promotion, carry throughput intent into the fresh worker so recovery does not silently undo the optimization.

2026-08-20

• GPU memory snapshots are a per-profile optimization, not an unconditional model-loading accelerator. Persist direct and snapshot warmup measurements with the content-addressed loader profile, sample both variants, and select the lower median; workers with no loader state should start directly.
• Speculative warmup must be coalesced with real dispatch using the complete worker identity: deployed app, GPU, affinity, loader profile, and snapshot variant. Scheduling-only dedupe still allows a real request to race the warmup into a second cold container.
• A future LLM component needs semantic warmup, not merely an allocated GPU. Resolve and stage the fixed profile on CPU, load the resident engine, run bounded representative text and vision requests, then commit content-addressed compiler-cache manifests before downstream execution reaches the LLM.
• Modal Volume-backed compiler caches need explicit lifecycle operations. Reload the cache mount before importing vLLM/Triton, persist Torch extensions and Numba beside vLLM, TorchInductor, Triton, and CUDA caches, and explicitly commit representative warmup products plus every genuinely new specialization.
• vLLM 0.27.1's jit_post_compile_hook cannot distinguish a persistent Triton cache hit from a real compilation: Triton invokes it after both paths, so its “JIT compilation” warning actually means only a process-local cache miss. Install Triton's compilation listener before the spawned EngineCore starts, log its real cache_hit value and stage timings, and signal the Modal parent only on cache_hit=False. Commit from that signal instead of deduplicating by model profile, because a later request can introduce a new shape for an already-seen profile. Search only the executable subgraph dependency closure for ModalLLM; serialized boundary metadata can contain historical LLM nodes that do not execute in the current worker.
• A parent-process monkey-patch does not cross vLLM's multiprocessing spawn boundary. Patch the EngineCoreProc.run_engine_core target before CoreEngineProcManager constructs its Process, using a top-level importable wrapper so spawn can unpickle it; the wrapper must install the Triton listener before calling vLLM's original entrypoint. Publish the live EngineCore pid separately from compile-miss events, otherwise an absent child listener and a perfect persistent-cache hit both look like an unchanged zero-byte signal. Apply the same before/after miss checkpoint to representative prewarm commits as ordinary inference commits.
• LLM speculative prewarm discovery must not recursively scan an entire serialized proxy payload. Those payloads can embed later components and planning metadata, which made an image-only #251 worker load a 51 GiB LLM and add 92 seconds to the critical path. Iterate each subgraph fragment, resolve its declared execute targets' dependency closure, and inspect only those nodes; the separately assigned LLM worker can still prewarm concurrently.
• Parallel loader prewarms are safe only when the process-global _patched_folder_paths_absolute_lookup() monkey-patch is reference-counted. Each overlapping context must keep the shared patch installed until the last loader exits; otherwise the first completed CLIP load can restore ComfyUI's original lookup while UNET is still active and make an existing /storage/assets/... file fail as "not found." A concurrency test must exercise the real folder lookup context rather than mocking _execute_subgraph_prompt.
• Cache-friendly proxy payloads cannot be rehydrated from a node-id-only registry when multiple prompts are queued. A later prompt overwrites the active prompt's execution payload before its proxy runs. Carry the queue prompt id through a hidden ComfyUI metadata input, index transient contexts by (prompt_id, node_id), and consume the exact context at execution while leaving volatile run fields out of the visible cache surface.
• A deployment is not performance-ready merely because its runtime protocol responds. Seed the current important loader or LLM profile immediately after automatic deployment and make the first real dispatch join that seed future.

2026-08-19

• A terminal local SaveVideo can accidentally turn a tiny encoded remote artifact into a multi-gigabyte boundary result. One observed 481-frame 768x480 video serialized 2,127,790,080 bytes of float32 RGB frames and about 5,130,752 bytes of float32 stereo audio so the local node could recreate a 2.7 MB MP4. Do not change generic VIDEO transport to a lossy encoded representation because nonterminal consumers may need exact tensors. Instead, auto-place a terminal SaveVideo fed entirely by remote values into the remote component and let artifact capture materialize its encoded output locally.

• A split component can correctly isolate ModalLLM through worker affinity and still defeat warm reuse if the whole component's Comfy loader snapshot profile is stamped onto that LLM phase. Snapshot identity is also part of the parameterized Modal class identity: omit Comfy loader profiles from LLM-affinity payloads while preserving the profile on the surrounding Comfy phases.

• A Modal class-parameter affinity key can still be silently defeated after the first successful protocol check if the cached-current branch returns the parameterless compatibility-probe instance. Treat protocol probes as validation-only handles: both the initial-compatible and cached-compatible paths must perform a fresh parameterized lookup before execution, and regression-test both branches.

• Modal class construction can allocate container lifecycle work before a method is called. Checking the compatibility cache inside a function that has already constructed a parameterless probe is therefore too late: consult the cache first, and construct no probe at all for a known-current runtime. An empty cache after local restart still needs identity validation, so expose the same protocol and fingerprint from the CPU-only ModelStager; retain the parameterless GPU probe only as compatibility fallback for older deployments. Otherwise validation can spend the spare GPU container and queue the intended affinity-bound speculative worker behind it.

• Canvas decoration refactors must exercise an active node-progress render through onDrawForeground, not only verify palette source strings. Moving shadowColor into a palette object left one bare progress-panel reference that parsed successfully but raised only when the panel was painted; use the current palette color with a neutral fallback for the local-warning path.

• Deliberate mixed-fanout cuts can still form a cycle in the coarse component graph when an earlier value bypasses the cut and feeds a later remote node. SCC merging must preserve graph correctness, so the repair belongs after that merge: force ordered split-proxy phases at annotated local-materializer/remote-session boundaries, carry the local materializers through the phase rewrite, and isolate LLM versus ordinary Comfy phases with distinct reusable worker-affinity groups.

• ComfyUI's V3 loader assigns RELATIVE_PYTHON_MODULE only to nodes returned during startup. Any internal proxy or bridge class inserted into NODE_CLASS_MAPPINGS later must copy that loader-owned identity at the registration boundary, including cache-hit reinsertions; otherwise /object_info emits python_module: null and the frontend can abort before filtering development-only nodes.

• A reusable worker-pool affinity key is ineffective if it appears only in diagnostics. Modal class parameters define the independently autoscaled instance identity, so the stable slot key must be passed as a RemoteEngine parameter on every real invocation and keepalive pulse; prompt-scoped session identity still belongs only in the payload.

• Modal's scaledown_window is an idle upper bound, not a reservation that guarantees the same loaded GPU worker survives a long local phase. For a known remote → local → remote gap, combine an extended per-call class option with a bounded lightweight heartbeat, and stop that heartbeat as soon as the next remote component dispatches.

• ComfyUI gives ready output and async nodes priority, which can put an independent local preview or save branch on the remote critical path even when its result cannot affect later remote work. Insert an async gate only on branches whose full downstream closure contains no remote proxy; wait for the nearest remote continuation to begin dispatch, then release the local branch while that remote call is in flight. Any branch that transitively re-enters remote execution must retain its original dependency ordering.

• A dispatch gate cannot help when the planner has already co-located the producer and downstream remote consumer merely because the producer also has a local preview tap. Split that mixed fanout into a durable producer bridge, a downstream remote proxy, and an asynchronous local bridge materializer. This both prevents the local download from blocking the remote continuation and keeps unrelated heavyweight models out of the same GPU process.

• Streaming a boundary preview to the browser is not a general substitute for executing its local branch. A durable local materializer lets ordinary preview, save, and other local-only nodes receive the real ComfyUI value while the remote consumer uses the same bridge independently; suppress the duplicate synthetic preview stream for that boundary.

• Cache-friendly proxy inputs are only a stable cache surface; execution rehydrates the full run-scoped payload from the proxy context registry. Any rewrite-time control fields added after initial registration must update and re-register that full payload as well as the embedded prompt value, or the live proxy silently executes without them. A parallel-preview dispatch group must remain part of the cache surface so its signalling proxy executes on every prompt instead of being skipped by ComfyUI cache reuse.

• Modal Dict operator assignment is always synchronous, even inside an async helper. When a cache path becomes asynchronous for ComfyUI compatibility, use await dictionary.put.aio(key, value) for remote writes and retain operator assignment only as a local-dictionary compatibility fallback; otherwise Modal 1.3.3 and newer warn while the network call blocks the active event loop.

• Hugging Face Xet acceleration is a transport capability of huggingface_hub, not a separate model-download path. Pin and import-validate hf-xet in the CPU staging image, keep passing the environment token through the existing Hub calls, and let the Hub select Xet for repositories that use it. Adaptive concurrency is the safe default for the 16 GiB stager; high-performance mode raises buffers for machines with at least 64 GiB of RAM and must remain an explicit operator choice.

• Qwen3.5 exposes a hard per-request thinking switch through the chat template's enable_thinking argument, but not a backend-neutral reasoning-token budget. Carry an enable_reasoning boolean through the shared generation settings into both Transformers and vLLM chat-template calls. When it is false, use a direct-response parser for that request as well; leaving the Qwen reasoning parser active would misclassify a delimiter-free direct answer as unfinished reasoning.

• vLLM's async engine is only useful as a resident ComfyUI backend if its owning asyncio loop outlives each node call. Keep one loop thread and one AsyncLLM per resident model, consume cumulative request outputs for token count/TTFT/rate, and abort unfinished requests when ComfyUI cancellation interrupts the callback. Recreating the loop with asyncio.run() would strand the engine's background output task after every request.

• LLM startup progress mixes measurable and opaque phases. Hugging Face snapshot_download(tqdm_class=...) can expose aggregate file counts, and a local safetensor index can expose the total shard count; vLLM's weight load, KV-cache initialization, multimodal profiling, and kernel warmup still happen inside engine startup without a stable callback. Show truthful stage text and an indeterminate pulse for that interval instead of inventing percentages, then switch to engine-derived token metrics during decoding.

• Reasoning separation belongs at the model-output boundary, not in the ComfyUI node or one inference engine. Transformers generation returns token IDs, while the resident vLLM integration consumes cumulative AsyncLLM.generate outputs containing text and token IDs; vLLM's separately parsed reasoning field is part of its OpenAI-serving response path. Preserve model boundary tokens, select the parser in the immutable profile, and decode reasoning and final content independently. Prefer a native engine field if one appears in a future backend contract.

• Qwen3.5 begins generation in reasoning state and may reach max_new_tokens before emitting </think>. In that case, classify the whole partial continuation as reasoning and return an empty primary response; treating delimiter-free output as a final answer leaks chain-of-thought into the response channel.

• A painless Hugging Face model-ID input still needs an immutable deployment boundary. Resolve the requested repository to an exact commit on CPU, derive a content-addressed schema-v2 profile from model metadata plus a versioned compatibility policy, and reject unknown architectures or quantization before downloading tens of gigabytes. Qwen3.5 block-FP8 and ModelOpt NVFP4 can share the current vLLM adapter; Muse-Glimmer needs the released Transformers 5.15 implementation.

• vLLM 0.27.1 pins Torch 2.13.0, so the remote image must move as a unit to Torch 2.13.0, torchvision 0.28.0, Transformers 5.15.0, NumPy 2.3.5, and vLLM 0.27.1. The installed PyTorch wheel reports torch.version.cuda == "13.0"; validate the observed compiled-toolkit value rather than deriving it from the wheel-index label. For ComfyUI co-residency, pass kv_cache_memory_bytes explicitly (currently 12 GiB) and start in eager mode; the default GPU utilization fraction is inappropriate on a large GPU shared with image/video models.

• A GPU-specific dependency branch made the Modal LLM node work on B300 but fail with No module named 'vllm' when the same workflow selected RTX PRO 6000. Backend availability must not depend on GPU price or capacity. All selectable Modal GPUs are Turing compute capability 7.5 or newer, so use one CUDA 13.0/Torch 2.13/vLLM 0.27.1 stack from T4 through B300, install vLLM after Torch, and make every image build import and version-check vLLM. Let the separate measured-VRAM admission guard reject models that do not fit.

• Installing custom-node requirements after the base runtime can silently downgrade shared packages: the RTX PRO 6000 canary changed NumPy 2.3.5 to 1.26.4 and OpenCV 4.13 to 4.11 before vLLM then selected newer unpinned replacements. Reapply deployment-owned runtime pins after custom-node requirements and make the final accelerator build handshake assert the actual NumPy and OpenCV distribution versions as well as importing them.

• A clean RTX PRO 6000 deployment is the necessary regression gate for the missing-vLLM incident. The 2026-08-19 canary validated the reconciled image at Torch 2.13.0+cu130, CUDA 13.0, vLLM 0.27.1, NumPy 2.3.5, and OpenCV 4.13.0.92, then passed real generated-profile inference for the exact Blackfrost ModelOpt NVFP4 revision. An import-only handshake would not have proved that the model-specific backend and kernels actually worked.

• Generated LLM profile dispatch has two model-reference channels: the workflow/subgraph payload used for CPU discovery and the separately serialized direct-node kwargs sent to the GPU worker. Rewrite both to the immutable hf-... profile ID after staging; rewriting only the subgraph lets a successful multi-gigabyte download end in an Unknown Modal LLM profile error before model load. Normalize Hub download failures into plain actionable exceptions before they cross Modal's serialization boundary, especially for gated repositories.

• Modal image layer order determines whether ordinary source or secret changes force multi-gigabyte dependency rebuilds. Install the pinned ComfyUI and custom-node packages, then the Torch stack, then its ABI-matched vLLM wheel before applying deployment-specific environment values such as the runtime fingerprint and before mounting source trees. This keeps runtime identity strict, prevents later layers from invalidating vLLM, and allows code-only deployments to reuse the expensive dependency image.

• vLLM 0.27.1 can load a 29 GiB Qwen3.5 FP8 checkpoint on B300 and then fail during startup profiling because its optional FlashInfer top-k/top-p sampler attempts to JIT-compile a kernel with nvcc. Set VLLM_USE_FLASHINFER_SAMPLER=0 in the worker image to select vLLM's native sampler without disabling model-specific FlashInfer attention paths or adding a full CUDA compiler toolchain to the runtime.

• vLLM's kernel warmup imports FlashInfer communications code even for a single-GPU engine with custom all-reduce disabled. The FlashInfer build bundled with vLLM 0.27.1 evaluates an array.array[int] annotation during import, which raises TypeError on Python 3.11 after the checkpoint has already loaded. Use the supported Python 3.13 runtime rather than patching an installed dependency; Python 3.13 makes array.array subscriptable and matches this project's local interpreter.

• Disabling the FlashInfer sampler is not enough for Qwen3.5 on B300: automatic attention selection chooses FlashInfer's TRT-LLM Blackwell path, which attempts another nvcc JIT on the first real multimodal prefill. With the policy's BF16 KV cache, select vLLM's documented TRITON_ATTN full-attention backend explicitly while retaining the model-specific FlashInfer GDN kernel. Translate vLLM RuntimeError failures into built-in exceptions at the node boundary so Modal clients do not need the remote vLLM package merely to deserialize an error.

• Blackwell overrides a requested FlashAttention 3 backend to version 4, and Qwen3.5's first real prefill then asserts that SM100 head dimension 256 does not support seqused_q/seqused_k. Use the Triton full-attention backend instead. Also catch vLLM's specific EngineDeadError: it does not inherit RuntimeError, and allowing it across Modal RPC makes clients without vLLM fail while deserializing the private exception class.

• Live B300 multimodal canaries passed for the exact requested Orcarouter FP8, Muse-Glimmer, and Blackfrost ModelOpt NVFP4 revisions. The model-selectable co-residency canary also retained the generated Orcarouter profile across a real Flux VAE encode and reported an LLM cache hit afterward on the same single-container B300 worker.

• Transformers 5.15.0 raises its safetensors floor to 0.8.0. Keeping the older 0.7.0 ComfyUI pin makes Modal's image dependency resolver fail before deployment, so the backend upgrade must include safetensors 0.8.0 in the same runtime fingerprint.

2026-08-18

• Same-worker LLM inference needs a pre-GPU staging boundary, not a lazy model download inside the node. Discover fixed revision-pinned profiles in the serialized remote payload, download them through a single-container CPU class, commit the shared Volume, and force the GPU worker to reload that revision before model construction. Otherwise first use spends expensive GPU time waiting on network I/O and a warm mount may not see the completed snapshot.
• A deployment-owned node cannot rely on the optional custom-node archive to register itself inside the headless worker. Register the shipped ModalLLM class explicitly after each ComfyUI node-registry initialization path while preserving any already-imported custom registration; this keeps the production node available even when custom-node sync is disabled.
• Co-resident Transformers models are visible to CUDA free-memory accounting but invisible to ComfyUI's managed-model eviction list. Keep them in a separate serialized LRU, ask ComfyUI to release idle managed models before a cold LLM load, enforce a real free-VRAM reserve, and expose both resident sets in telemetry. Co-residency increases GPU utilization but should not be described as concurrent kernel execution.
• A local ComfyUI .env file should remain outside the Modal image and synced custom-node bundle. Attach an existing named Modal Secret to the deployed RemoteEngine class so all contained key-value pairs become worker environment variables, and include only the collection name in the runtime fingerprint so changing COMFY_MODAL_SECRET_NAME replaces a deployment that references the old collection.

2026-08-15

• ComfyUI OUTPUT_IS_LIST is a strict runtime container contract, not only planning metadata. If a proxy advertises a scheduler-list IMAGE but returns a raw [1,H,W,C] tensor, ComfyUI extends that tensor as the output list and passes [H,W,C] downstream; PreviewImage then treats height as batch size and saves H arrays shaped [W,C]. Normalize every scheduler-list proxy result to a Python list even when singleton execution skipped mapped aggregation.
• Multi-item mapped tests do not cover the singleton contract. Their aggregation path already wraps a joined tensor, so regression coverage also needs an ordinary remote result with one IMAGE and must verify that ComfyUI list merging preserves the complete [1,H,W,C] tensor as one scheduler item.

2026-08-14

• A costly local re-entry is a graph property, not just a single-edge pattern. The planner can identify every affected local node by intersecting two local-only closures: nodes reachable downstream from any remote node and nodes that can reach any remote node upstream. This catches both one-node and multi-node remote-to-local-to-remote paths while excluding local preview branches that never re-enter remote execution.
• Planner advice must not reuse execution-failure colors. A local node between remote regions should keep its native theme and show only a neutral ! badge whose hover text explains the possible placement mistake; reserve red node styling for actual execution failures.

2026-08-13

• A ComfyUI requirements-file change affects the remote runtime fingerprint but does not install the new dependency by itself when the Modal image uses an explicit pinned support set. Reconcile new import-time core requirements with that pinned set: otherwise a built-in extra such as nodes_math.py can fail on simpleeval, then be misreported later as an unsynced custom node because its V3 class never registered.
• Iteration-rate labels can be derived entirely from successive streamed value/max samples in the frontend, but refocus replay must use the backend event timestamps rather than replay arrival time or buffered events will produce artificially high it/s values. Track and smooth rates independently for ordinary node progress and every mapped worker lane.
• A fast prompt can emit execution_success and clear its frontend state before the queue route's successful HTTP response is processed. Queue-response UI reconciliation must treat a null prompt state as an already-terminal success and avoid recreating ready/waiting visuals or converting completion into a frontend TypeError.
• Artifact-only remote workflows need two execution anchors: the remote component planner must keep a terminal side-effect node in execute_node_ids even when it has no outputs, and the rewritten local prompt must expose a real ComfyUI output path. A synthetic completion token emitted after artifact materialization and consumed by an internal V3 output sink satisfies both requirements without adding a user-visible workflow output.
• V3 dynamic inputs use their complete schema path in queued prompt JSON. An autogrow template named component_ inside an input group named components must be linked as components.component_0; a flattened component_0 key looks plausible in unit assertions but fails ComfyUI validation as a missing required input. Regression tests should run get_finalized_class_inputs against the live prompt inputs to catch namespace mistakes.

2026-08-07

• A workflow-level GPU choice has to survive beyond the queue request. ComfyUI executes rewritten proxies asynchronously after the HTTP handler returns, so the selected GPU must be stamped into every ordinary, mapped, split-phase, and proactive-warmup payload; consulting process settings later can silently deploy or invoke the wrong target.
• GPU choice is part of the immutable Modal runtime identity. Request-specific fingerprint checks and cloud-module construction must use the payload-selected settings together, otherwise a newly selected GPU can reuse a locally cached deployment decision or rebuild the image with the startup environment's GPU.
• ComfyUI's root graph extra mapping is the right persistence boundary for workflow-wide extension settings. Values stored there are included by normal graph serialization without adding a synthetic node or a prompt widget.

2026-06-26

• Frontend canvas animation should be driven by visuals that actually change over time, not by the mere presence of progress or cache state. Static Modal progress can redraw on websocket updates; only pulsing phases, setup placeholder lanes, and fade-outs need a throttled rAF loop, and prompt cleanup should cancel that loop once the last animated state is gone.
• The README should stay a user guide and operational reference, not a running release-note log. Remote runtime details are easiest to maintain when grouped by stable user-facing concepts such as deployment, snapshots, warmup, mapped execution, session bridges, cancellation, and sync behavior; commit-by-commit implementation notes belong in this file instead.

2026-06-22

• ComfyUI's Dependency cycle detected message can originate entirely in the local scheduler after Modal-Sync rewrites remote nodes into proxy phases, even when the Modal worker and individual remote sampler components finish successfully. The local queue path now logs compact rewritten-prompt diagnostics, including proxy dependency edges, component stages, payload summaries, and any static cycle path, so the next UI-only cycle error can be traced from comfy.log without guessing which remote phase was responsible.
• Split-proxy phase ordering cannot consider only the remote component's internal links. A local feedback path such as remote sampler A -> VAEDecode -> BetterGrok -> remote sampler B is still a dependency from A to B when B's boundary input re-enters the remote island, and ignoring that path can make B produce shared model bridge outputs that A then depends on, creating a proxy-level cycle that the original workflow UI did not show.
• Local preview branches are not the same as local feedback. If a remote output also feeds later remote work, and the local branch downstream from that output never reaches another remote node, the planner can absorb that local branch as a non-returning tap and keep the remote sampler chain in one component instead of splitting at every preview.
• Absorbing a preview tap means closing over its non-transportable boundary in both directions. A local VAEDecode preview branch cannot be moved remote while leaving its VAELoader local, and once that loader is remote, any remaining local VAEEncode or VAEDecode consumers of the same VAE must join the component too; otherwise the planner just moves the illegal VAE boundary from remote input to remote output.
• Preview taps should suppress split proxies only when the final component has no local re-entry dependency. If a component output feeds a local node such as BetterGrok and that local node feeds back into the same remote component, a single proxy creates a synthetic proxy -> local -> same proxy dependency cycle, so the component still has to split into ordered phases.
• Durable session bridge replay is acceptable for cheap loader/encoder recovery paths, but replaying a sampler bridge is usually a planning or affinity failure. When a downstream phase loses the live or warm bridge value for a sampler-producing component, the runtime now fails loudly instead of silently rerunning that sampler phase.
• Durable bridge identity and durable recovery payload are separate concerns. The bridge key must continue to include the complete canonical producer-input signature, but producer inputs should be persisted only when the selected recovery path consumes them: direct serialized outputs and literal single-node loader plans need none, linked loader subgraph plans need only their surviving boundary inputs, and full producer replay needs the complete set.
• Modal ExecutionError is used for both lookup/deployment failures and real remote runtime failures. Auto-deploy recovery must not key only on broad strings like not found, and both the stale-handle retry layer and the outer deployed-invocation catch need the same check; if the message contains a remote traceback or RemoteSubgraphExecutionError, surface the remote failure directly instead of importing the cloud deploy module and replacing the real error with an existing-app guard.
• Refocus replay cannot depend only on buffered Modal websocket events. Native ComfyUI completion events such as execution_success may be missed while the browser is backgrounded, so the frontend now reconciles temporary Modal UI state against ComfyUI's /queue and /history/{prompt_id} endpoints before leaving old node colors or the global pill visible.

2026-05-20

• The Modal status pill should be driven by actual remote work, not by the custom node route being installed. Since the frontend patches every prompt through /modal/queue_prompt, the backend must fast-path prompts with zero Run on Modal nodes without emitting modal_status, and the frontend should ignore any empty Modal status event as a defensive guard.
• A visibility/focus redraw is not enough if the browser missed Modal websocket events while backgrounded. The local ComfyUI side needs to retain recent client-scoped modal_status and modal_progress payloads, and the frontend should replay that buffer on refocus before redrawing the status pill and node overlays.
• Programmatic context-menu changes to properties.is_modal_remote also have to update the live Run on Modal widget value. Otherwise the saved workflow state and node border can be correct while the visible toggle still appears disabled.
• ComfyUI restores saved widget values after nodeCreated, so the inverse mismatch is also possible: the visible Run on Modal widget can be disabled while an older properties.is_modal_remote value remains true. Reconcile the property in loadedGraphNode/afterConfigureGraph, treat widgets_values_named["Run on Modal"] as authoritative at queue time, and keep the backend guard so stale metadata cannot launch Modal.
• Graph-wide Modal enablement should operate on live eligible nodes, not just serialized workflow JSON. Scanning the root graph plus nested subgraphs lets the action update both node properties and visible widgets in one pass.
• Once Modal has several node context actions, putting them under one Modal submenu keeps labels short and prevents repeated prefixes from cluttering the main node menu.

2026-04-23

• Nested workflow-marker resolution has to stop at the first matching composed prompt-id suffix, not union every suffix. If a subgraph node 195:27 is marked remote and the prompt also contains an unrelated root node 27, collecting both matches makes the real nested node look local during boundary validation and proxy rewrite.
• Frontend interruption cleanup has to key off prompt_id, not the native execution event's node fields. ComfyUI can emit execution_interrupted without a representative remote node id, and if Modal UI teardown depends on resolving that node the cancelled prompt stays visually stuck in ready/active state even though interrupt propagation worked.
• The global Modal status pill cannot infer liveness from STATE_COMPLETE remote node states. If a prompt-level cleanup event is delayed or missing, treating completed remote nodes as still "executing" leaves stale waiting/executing/finalizing pill entries behind. The pill needs an explicit "does this prompt still have live remote work?" reconciliation pass when remote nodes complete.

2026-04-22

• Prompt-scoped Modal class parameters are the wrong place to carry remote session identity. The remote_session.session_id belongs in the payload for data isolation, but @app.cls affinity should use reusable worker-pool slots like worker-pool:slot:N; otherwise every prompt asks Modal for a brand-new parameterized class instance and cross-prompt warm-container reuse never happens.
• The local custom_nodes/ tree is effectively process-static for one ComfyUI lifetime. Since adding or removing custom nodes still requires a ComfyUI restart before node availability changes, the queue-time sync engine should resolve custom_nodes/ once per process and then reuse that cached decision instead of rehashing or rescanning the tree on every workflow run.
• Modal image builds snapshot local directories and fail if included files change during the build. ComfyUI writes under .cache/, including .cache/strings, while a local server is active, so the ComfyUI image include filter must treat that cache as volatile build input and exclude it.
• Warm bridge restoration is only half of the mapped seed optimization. Once every exported static bridge output has already been restored into the target remote session without replay, the seed payload should skip prompt_executor_execute entirely and synthesize its bridge-ref outputs directly from session state; otherwise the system still wastes seconds rerunning loader and encoder nodes after the expensive work was already avoided.
• Modal function snapshots are initialization snapshots, not mid-request checkpoints. The practical way to hide first-request model loads is to detect root literal loader nodes in the rewritten prompt, then execute them as synthetic one-node subgraphs during proactive warmup so the worker-local loader cache is hot before the real payload arrives.

2026-04-21

• Modal @app.cls class parameters are sensitive to postponed annotations. If a Modal entry module uses from __future__ import annotations, then a declaration like snapshot_profile_key: str = modal.parameter(...) reaches Modal as the literal string "str" at import time and class construction can crash before any request runs. Keep real annotations on the Modal entry classes, or quote only the non-Modal forward references instead.
• Durable bridge refs still are not enough on their own for fast repeat mapped runs. If the worker already has the live remote-only value in memory, the cheapest path is to cache that value by bridge_key and restore it directly into the next prompt-scoped session instead of replaying the whole static producer subgraph just to recreate the same MODEL or CONDITIONING.
• MODEL needs a different durable restore path from CONDITIONING. The practical shape is a self-contained node rehydration plan: if the producer node's inputs are all literals, the bridge record can rebuild just that loader node output and skip replaying the whole static phase. Pairing that with warm loader-cache coverage for checkpoint loaders turns repeat model restoration into a single-node rebuild instead of a multi-node prompt replay.
• Replay optimization needs its own instrumentation. A single mapped-phase summary that breaks bridge resolution down into live-session hits, warm bridge-cache hits, durable-record lookup time, replay time, session-restore writes, and loader-cache hit/miss deltas makes it obvious whether a repeat run really skipped work or just hid it behind another internal phase.
• Remote custom-node reuse is mostly an idempotence problem, not a raw import-speed problem. Once one extracted bundle root has already been initialized on a warm worker, repeating init_external_custom_nodes() for the same root mostly adds log noise and risk, so the runtime should treat that root as already loaded and make the reuse explicit in logs.
• The local streamed Modal bridge should stop waiting once it receives the terminal result envelope. In real runs, the underlying remote_gen(...) iterator can take tens of extra seconds to unwind after the remote payload has already finished, and that tail latency blocks mapped cleanup and makes one lane look like a straggler even though inference is done.
• Queue-time warmup estimates are not enough for mapped fan-out because the real list length is often only known when ModalMapInput executes. Registering the local map node as a warmup trigger lets Modal-Sync top up exact container demand one node earlier than the mapped proxy, and a short bounded head-start on those in-flight warmup slots is enough to overlap some cold boot time instead of discovering it only at the first seed call.
• custom_nodes/ behaves much better as a manifest plus per-package archives than as one whole-tree ZIP. The overall manifest still gives one deterministic tree digest for volume reload and runtime cache identity, but hashing and packaging each top-level custom-node package separately means one edited package no longer forces a full rebuild and reupload of every other unchanged package.
• Split custom-node archives only pay off if the local pipeline is parallel too. Building each per-package ZIP and uploading it independently in worker threads lets the local machine overlap the mostly I/O-bound file reads, compression, and Modal volume transfers instead of serializing those stages back into one long queue.
• Parallel local packaging needs a stricter policy on the Modal volume side than on the filesystem side. Metadata probes like VolumeListFiles can rate-limit well before the actual uploads do, so the safe pattern is to keep a small fixed concurrency for volume SDK calls and publish one shared exponential backoff window across all local workers when any of them gets ResourceExhaustedError.

2026-04-20

• Early local delivery for hybrid remote components is much simpler if the queue-time rewrite stops pretending the whole island is one proxy. Splitting the island into a static proxy and a mapped proxy lets ComfyUI unblock ordinary downstream local nodes using its existing scheduler semantics instead of inventing partial-completion behavior for one proxy node.
• The bridge between those split proxies cannot be a normal transport value when the shared upstream output is something like MODEL. The workable shape is an opaque session-ref payload: the static proxy returns a JSON-safe reference, the remote runtime stores the live object in a prompt-scoped session, and the mapped proxy resolves that ref back into the live object when it runs.
• A pure in-memory session ref is not enough once Modal can reroute later split phases onto a fresh container. The durable fix is a hybrid bridge: keep the fast path in the in-memory session store, but also persist replay metadata for each session-backed output in a separate Modal Dict so a later phase can rebuild the live remote-only value into its current session instead of failing on a dead ref.
• Session-backed split proxies can be local-cache-safe after all, but only if their exported bridge outputs are durable replay refs rather than prompt-scoped session ids. Once the proxy cache surface strips prompt_id, remote_session, and clear_remote_session, ComfyUI can reuse cached proxy outputs across prompt runs and the remote runtime can rehydrate any still-needed remote-only state lazily from the durable bridge metadata.
• Cache-safe stripping for session-backed proxies needs one more escape hatch in real ComfyUI runs: hidden unique_id is not guaranteed to come back on every execution path. Embedding a stable proxy-cache context id in the sanitized payload makes execution-payload rehydration robust even when the hidden field is absent.
• Every synthesized split-phase payload has to carry remote_session, not just the top-level hybrid payload. If a static or mapped phase builder drops that field, the remote worker only discovers the mistake at boundary-output collection time when it tries to export a session-backed output.
• Rewriting a component into multiple proxies means the old queue-time dependency metadata is stale. Execution stages, dependency edges, and partial-target remapping all need to be derived from the final rewritten proxy graph, not from the pre-rewrite coarse component plan.
• Prompt-scoped remote state is only useful if both halves of the split can reach the same worker. The current implementation now passes a reusable worker-pool affinity key into deployed RemoteEngine lookup, while the prompt-scoped remote_session.session_id stays inside the payload for data isolation.
• Online deployed testing confirmed the prompt-scoped session-affinity path for the tested hybrid workflow: the mapped proxy successfully reused the static proxy's remote-only state on the same worker instead of failing with a missing session/value-ref.
• The cheapest durable regression for this feature is a proxy-level scheduling test, not a full ComfyUI integration harness. If the rewrite produces separate static and mapped proxies, and a local downstream consumer can run off the static proxy result while the mapped proxy task is still awaiting completion, that already protects the core early-unblock behavior.
• It is still worth keeping one checked-in workflow-shaped artifact for this feature family. A JSON fixture that carries both the saved workflow metadata and the queued prompt shape catches rewrite drift that pure unit tests can miss, especially around proxy ids, downstream rewiring, and shared session ids.
• Session observability is most useful when the store and the runtimes both log. Store-level logs explain create/reuse/resolve/cleanup semantics, while runtime-level logs tie those session ids back to concrete component executions and clear_remote_session boundaries.
• Modal's own modal container logs path is just a thin wrapper over the SDK's AppGetLogs stream, so local observability does not need a separate sidecar. The remote worker can surface its MODAL_TASK_ID in the existing streamed payload channel, and the local ComfyUI side can mirror that task's container logs directly into local stderr with a CLI fallback when the Python SDK is unavailable.
• In practice, the local Python SDK log stream can emit Modal client task-context warnings when called from ComfyUI's own worker threads. Preferring the modal container logs subprocess path avoids that noise while still preserving an SDK fallback for environments that only have the Python package installed.
• The hybrid split-proxy pattern should remain tied to a correctness requirement such as mapped execution or local re-entry. Splitting an ordinary component merely because it has several execute branches turns shared remote-only state into avoidable bridge traffic and can cost far more than any early-completion benefit.

2026-04-19

• Early local unblocking is currently limited by the proxy shape, not just by transportable boundary types. If queue-time rewrite packs a remote branch into one hybrid mapped_subgraph proxy, static outputs that are already finished still cannot drive downstream local nodes until the mapped phase also finishes, because the remote runtime only merges and returns those outputs once at the end. Fixing that needs either partial-completion semantics for one proxy node or a shared remote session/cache that lets separate proxies reuse non-transportable state like MODEL.
• Mapped remote execution cannot hoist invariant upstream work safely by caching {prompt_id}:{node_id} outputs or by replaying per-item subgraph RPCs. The queue-time rewrite has to split the component into explicit static and mapped phase prompts, expose static-to-mapped boundaries, and let one runtime invocation execute both phases so non-transportable objects like MODEL stay in-process while PromptExecutor still decides reuse from effective inputs and dependencies.
• The frontend cannot treat mapped execution like a prompt-wide single active node. Per-lane progress is the durable source of truth there: ordinary non-mapped runs can still use activeNodeId, but mapped remote styling should derive purple/active state from whichever nodes currently hold live lane or numeric progress, and lane ownership needs to move when one worker advances from node A to node B.
• The test bootstrap should not hard-code ~/git/ComfyUI as its only source of ComfyUI imports. Reading COMFYUI_ROOT first makes it easy to validate the extension against a disposable checkout in /tmp while preserving the old local-default behavior.
• A usable uv test environment for this repo needs more than pytest. Importing current ComfyUI APIs during tests also requires aiohttp, Pillow, torch, safetensors, numpy, and tqdm, so those belong in the dedicated test dependency group instead of being installed ad hoc on one machine.
• On this host, uv sync worked reliably with UV_PROJECT_ENVIRONMENT and UV_CACHE_DIR=/tmp/uv-cache, but uv run ... crashed inside the installed uv binary with a macOS system-configuration panic. The environment itself was still valid, and invoking /tmp/.../bin/python -m pytest against that uv-managed environment was a workable fallback.

2026-04-18

• The README had drifted into release-note territory, especially in the old Important distinction: block. For this project, the maintainable structure is: short overview, explicit local-vs-remote execution sections, concise operational caveats, and a separate learnings log for historical implementation details.
• ComfyUI registry setup is mostly metadata, not code. The repo needs a real [tool.comfy] block in pyproject.toml, a stable immutable node-pack id, repository URLs, and an optional GitHub publish workflow; choosing a software license should stay explicit instead of being guessed during packaging work.
• comfy node pack is not git-aware by default; it uses .gitignore as its practical packaging filter. For a clean registry artifact, this repo needs an allowlist-style .gitignore so untracked local files like logs, caches, virtualenvs, and scratch docs cannot leak into node.zip.
• Remote per-node phase events are not enough to mirror sampler progress in the local UI. The Modal worker also needs a headless equivalent of ComfyUI's global progress hook so ProgressBar updates feed progress_state, and the local relay should forward those numeric updates as a separate high-frequency UI event rather than overloading coarse execution-phase messages.
• ComfyUI prompt cancellation does not need a separate frontend API hook to reach Modal. The proxy node can watch ComfyUI's existing processing_interrupted() flag while it waits on the remote call, send one targeted remote interrupt for the active (prompt_id, component_id), and then re-raise ComfyUI's own InterruptProcessingException so the local executor reports a normal execution_interrupted.
• That cancellation watch cannot live only in the outer proxy wrapper. The blocking Modal request bridge, especially the streamed remote_gen(...) path, also has to poll processing_interrupted() itself and set the per-request cancellation event before it asks Modal to stop; otherwise a locally cancelled prompt can unwind while the remote container keeps running.
• On the Modal side, targeted interrupt delivery cannot depend on a second RPC into the same GPU execution class. The earlier interrupt_payload lane only worked by reserving extra class concurrency, which fights horizontal scale-out. The better design is a shared control store that running executions poll while the GPU class itself stays at @modal.concurrent(max_inputs=1). Older allow_concurrent_inputs class options now hard-fail on current Modal SDKs.
• To let ComfyUI overlap multiple ready remote components, the proxy node itself has to be async. Merely increasing Modal-side concurrency is not enough if the local proxy still blocks inside a synchronous execute(), because ComfyUI's scheduler only uses its pending-task overlap path for async nodes.
• There are two separate local bottlenecks to remove for remote overlap: the proxy node must await remote work instead of blocking the scheduler, and the helper that submits blocking Modal SDK calls must use more than one worker thread or the local process will still serialize every remote invocation even after the proxy becomes async.
• Horizontal Modal scale-out needs a different concurrency model from interrupt delivery. If one GPU container should run only one heavy Comfy workflow at a time, cancellation can no longer depend on a second RPC lane into the same class; the clean fix is a shared control store such as a Modal Dict, with each execution polling its own cancel flag while max_inputs stays at 1.
• Data-parallel batching fits this architecture best as a queue-time marker plus a local scheduler, not as a magical runtime pass-through node. A ModalMapInput marker can stay inside the rewritten remote component, the proxy can split one boundary input locally into per-item subgraph calls, and the remote runtime can stay unaware because each item is still an ordinary subgraph execution.
• Transport support and component-boundary desirability are separate policies. Inexpensive scalar values can safely define remote-to-remote component boundaries, but serializable tensor and media values such as LATENT, IMAGE, MASK, SIGMAS, AUDIO, and VIDEO should force co-location just like non-transportable runtime objects; otherwise Modal storage and cross-container restoration dominate fast downstream nodes.
• ModalMapInput changes the partitioning rule in one important way: the whole downstream remote region reachable from that marker should stay inside one mapped component, even across otherwise transportable edges. Without that exception, the marker would only map the first transport-aware slice of the workflow instead of the downstream region the user intended.
• That first transport-aware split is still only a coarse partition. Even when the original prompt is a DAG, the quotient between coarse groups can become cyclic after you merge only selected edges. Before rewriting proxies, the backend has to collapse strongly connected coarse groups back together or ComfyUI can later see a synthetic dependency cycle.
• Streaming intermediate remote UI outputs back into local ComfyUI is safest when it reuses ComfyUI's native websocket event shapes. Forwarding remote executed payloads and preview-image frames through the local PromptServer gives the browser real mid-run previews without pretending those inner remote node values are now available as local graph inputs.
• A downstream local PreviewImage node is a different problem from a remote node that already emits UI output. If a remote boundary IMAGE feeds a local preview node, the useful fix is to remember that direct preview consumer during prompt rewrite, stream the boundary tensor as soon as the remote source node finishes, and synthesize the local preview node's executed UI payload on the local side. That improves preview UX without trying to make ComfyUI execute arbitrary downstream local nodes incrementally.
• Queue-time remote-boundary fixes should come from the backend's actual rewrite logic, not from parsing error text in the browser. A small dry-run analysis route can reuse the same prompt-id resolution and upstream auto-expansion rules, then return workflow-node paths for the frontend to mark directly from a context-menu action, including nested subgraph nodes.
• For ComfyUI node right-click menus, patch getExtraMenuOptions at node-definition time in beforeRegisterNodeDef, not per-instance in nodeCreated. The prototype-level hook matches how working extensions inject menu entries and avoids menu items silently disappearing even when other per-node UI changes still show up.

2026-04-17

• The workflow metadata already rides along in extra_data.extra_pnginfo.workflow, so the backend can discover node.properties.is_modal_remote without any extra frontend metadata channel.
• That workflow metadata is not always flat. ComfyUI can embed saved node records inside nested subgraph fragments, so Modal marker discovery has to recurse through workflow JSON instead of only scanning the top-level workflow["nodes"] list.
• Nested saved subgraph ids are not necessarily queued prompt ids. Before rewrite-time graph traversal, nested Modal markers need to be resolved against the queued prompt id space: sometimes that means a composed id like 24:23, and when ComfyUI does not expose the inner node directly it means falling back to the nearest ancestor prompt node. Without that resolution step, the backend will crash trying to rewrite a saved workflow id that does not exist in the actual prompt.
• A single wildcard-output proxy node is not enough for server-side graph rewriting. ComfyUI validates downstream link indices and output arity against the replacement class, so the backend has to generate proxy node classes that mirror the original node’s output signature before validation runs.
• Keeping the sync engine content-addressable makes prompt rewrites cheap after the first upload. The route can block on sync safely as long as repeated requests collapse to a hash marker check.
• Hashing and zipping the entire custom_nodes/ tree inside the queue route is too expensive for COMFY_MODAL_EXECUTION_MODE=local. Local mode should skip custom-node bundle sync by default, with an explicit override when that path needs to be exercised.
• When rewriting a prompt node into a proxy, never reuse the same input dict for both prompt_node["inputs"] and original_node_data["inputs"]. Inserting original_node_data back into the live prompt inputs creates a recursive structure that breaks ComfyUI cache signature hashing with a RecursionError.
• Queue-time validation is necessary for remote nodes that still receive local Comfy runtime objects. Even if a node is marked remote, inputs like MODEL or CONDITIONING cannot be serialized across the current boundary, so the prompt should be rejected before execution rather than failing inside the serializer.
• Per-node proxying is not enough for real ComfyUI sampler graphs. Connected remote-marked nodes must be partitioned into a single remote component so internal edges like MODEL, CLIP, and CONDITIONING stay inside the remote subgraph and only the true local/remote boundary has to be transport-safe.
• Manually marking every upstream dependency is too brittle for arbitrary custom nodes. A better default is to auto-expand the remote node set upstream across any non-transportable edge, so a user can mark the downstream custom node they care about and let the backend absorb the required loaders/builders into the same remote island automatically.
• The local fallback runtime cannot call PromptExecutor.execute() inline from inside a proxy node, because ComfyUI is already running inside an active asyncio loop. Running remote subgraph execution in a worker thread avoids nested-event-loop failures while still reusing ComfyUI’s executor.
• Modal objects defined in-process are not directly callable from an arbitrary long-running app like ComfyUI unless they are inside a hydrated app.run() session or looked up from a deployed app with modal.Cls.from_name(...). In practice the correct default for this project is deployed lookup only, because ephemeral app.run() fallback rebuilds too much state between workflow executions and undermines the scale-to-zero cost model.
• A deployed-only strategy is still correct, but the custom node should bootstrap that deployment itself on first use. The pragmatic flow is: try Cls.from_name(...), and if the app is missing, programmatically call app.deploy(...) from the stable cloud entry module, then retry the lookup.
• ComfyUI loads custom node packs from directories under synthetic module names derived from filesystem paths, and for directory-based packs that module key is not guaranteed to be a valid Python import path. Modal container startup requires a real importable module name for @app.cls services, so the cloud entrypoint has to live in a separate, stable module such as comfyui_modal_sync_cloud.py rather than on the custom-node package module itself.
• When the local side imports that stable cloud entrypoint for first-run auto-deploy, a failed import can poison sys.modules with a half-initialized module. The loader has to evict cached modules that do not expose a real deployable app, and it has to remove failed imports from sys.modules, or later auto-deploy attempts will keep reusing the broken stub.
• Shipping the ComfyUI source tree into the Modal image is not enough by itself. The container also needs ComfyUI's import-time Python dependencies such as psutil, aiohttp, einops, torchsde, torchvision, and related runtime packages, or the remote worker fails before it can execute any graph nodes.
• Core ComfyUI imports extend beyond the obvious tensor stack. In practice the Modal image also needs startup/runtime packages such as alembic, pydantic-settings, spandrel, and kornia, because built-in nodes and config-parsing paths import them before a workflow gets very far.
• Quantized ComfyUI models add another hard dependency: if comfy-kitchen is missing in the Modal image, comfy.quant_ops.get_layout_class(...) falls back to None and FP8/FP4 model loading eventually fails with 'NoneType' object has no attribute 'Params'.
• A captured "working workflow" exported after a remote run is not necessarily a pristine authoring graph. It can contain internal rewrite artifacts like ModalUniversalExecutor, so checked-in workflow JSON should be treated explicitly as a smoke fixture unless it has been normalized.
• When performance work starts, timestamped logging matters more than adding yet more ad hoc messages. Giving the extension its own formatter with both wall-clock time and relativeCreated makes it much easier to see whether the delay is in hashing, sync, deployment, image build, or actual execution.
• Once the slow path is visible, the first easy win is persistent local caching of digests and bundle artifacts. For this project, caching file hashes by (path, size, mtime_ns) and caching digest-keyed custom_nodes archives under the local storage root avoids re-reading huge model files and re-zipping the same custom_nodes tree on every execution.
• After the local side is fast, the next blind spot is inside the Modal container. The cloud runtime needs its own phase-timed logs around volume reload, runtime initialization, PromptExecutor execution, and output collection; otherwise the remote call just looks like one giant wall-clock block and there is no way to tell scheduling delay from real inference time.
• A dedicated Python logger is not always enough for those cloud timings. In practice, Modal container logs may still omit named-logger INFO lines during ComfyUI startup, so the timing tracer should mirror its phase lines directly to container stdout to guarantee they appear in captured remote logs.
• ComfyUI root discovery for deploy-time image building cannot rely only on ~/git/ComfyUI. When the extension is installed under ComfyUI/custom_nodes/ComfyUI-Modal, the most reliable default is to derive the ComfyUI root from that install path and only fall back to home-directory heuristics.
• If the remote image also contains ComfyUI's top-level custom_nodes/ tree, the container will import the full custom node set twice: once from the image and once from the extracted sync bundle. The image should ship ComfyUI core without custom_nodes/, and the synced bundle should be the single source of external nodes.
• ComfyUI still assumes that its default custom_nodes directory exists even if you do not want to populate it from the image. If that directory is missing, nodes.init_external_custom_nodes() can crash on os.listdir(...), so the remote runtime should create an empty placeholder directory before external-node initialization.
• Warm-container optimization is most effective when expensive but graph-agnostic work moves into @modal.enter. For this project that includes built-in ComfyUI runtime initialization and execution-module imports, so snapshots and scaledown-window reuse can amortize that startup cost across requests.
• On GPU-backed functions that only use CPU memory snapshots by default, @modal.enter(snap=True) must stay CUDA-safe. Snapshot-phase prewarm should only do CPU-only work there, and the heavier ComfyUI runtime initialization should move to @modal.enter(snap=False) unless GPU snapshots are explicitly enabled.
• If GPU memory snapshots are enabled, the snapshot-phase hook can safely pre-initialize more of ComfyUI core and capture that state. This repo now uses that split so the same code path benefits from either CPU-only snapshots or GPU snapshots without accidentally initializing CUDA in the wrong phase.
• Once warm-container runtime init is amortized, the next bottleneck is usually the graph's own loader nodes. For this repo, UNETLoader, CLIPLoader, and VAELoader are worth caching inside the warm container by resolved loader inputs, and cached outputs should be cloned when possible so later requests can reuse loaded runtime objects safely.
• Loader caches are only part of the story. Local ComfyUI keeps one long-lived PromptExecutor in the prompt worker, so ordinary node outputs stay cached across runs as long as their signatures do not change. A remote worker that creates a fresh PromptExecutor per request throws that benefit away, so it needs a warm-container executor pool keyed by cache scope to preserve ComfyUI-style node-output reuse between runs.
• End-to-end prompt_executor_execute timing is still too coarse once the denoising loop is only a small fraction of the total. A headless prompt server can observe executing and executed events from PromptExecutor and emit per-node spans, which exposes whether the real time is in model loaders, encoders, or the sampler itself.
• For repeat ComfyUI workflows, the highest-leverage lifecycle knob is usually not min_containers=1 but a longer scaledown_window with min_containers=0. That lets a recently used container keep its fully initialized ComfyUI process warm for a while without forcing permanent idle billing between totally unrelated runs.
• Floating pip install torch torchvision in the Modal image is too unstable once ComfyUI tracks very new PyTorch releases. The remote image should pin the same known-good CUDA wheel family as the local ComfyUI environment, and for this repo that means the explicit PyTorch CUDA 12.8 index with torch==2.10.0, torchvision==0.25.0, and torchaudio==2.10.0.
• The frontend can reuse ComfyUI's native executing, execution_error, and execution_success websocket events for remote component visuals as long as the queue route also publishes the representative-node-to-original-node mapping after rewrite. That keeps the UI in sync with actual proxy execution instead of inventing a separate polling channel.
• When filtering the ComfyUI checkout for the Modal image, only exclude top-level runtime asset directories like models/ and output/. A blanket "ignore any path part named models" rule will also delete real Python packages such as comfy/ldm/models, causing import failures inside the container.
• Importing nodes.py alone is not enough inside the Modal container. The remote worker also has to run nodes.init_extra_nodes(...) once so built-in comfy_extras nodes such as EmptyFlux2LatentImage exist in NODE_CLASS_MAPPINGS, and then run init_external_custom_nodes() for each extracted custom-node bundle root it exposes.
• A bare PromptExecutor(_NullPromptServer()) is not equivalent to real ComfyUI startup. The headless worker must pass the same cache configuration that main.py uses, or execution eventually crashes when the executor polls caches with self.cache_args["ram"].
• If /modal/queue_prompt does expensive preparatory work before it returns, expose it through a server-side queue bridge and use a synthetic status event only to refresh ComfyUI's queue view. Do not dispatch native execution_start or executing events for preparation: ComfyUI starts its execution watchdog from those events and can periodically report a healthy multi-minute asset download as a stalled node execution.
• A server-side queue bridge alone does not make a queue-time preparation visible in ComfyUI's client job store. api.queuePrompt() must return the client-generated prompt id before long remote setup completes, and the Jobs API interruption path must translate cancellation of that synthetic running entry into the preparation's prompt-scoped cancellation event.
• Synthetic queue tuples must carry the same extra_data.create_time field as native PromptQueue entries. The modern queue panel validates /api/jobs strictly; one preparation job without create_time causes the complete response to be rejected and rendered as 0 active even while both /queue and /api/jobs contain the prompt.
• For remote node-level UI feedback, a streamed Modal generator method is cleaner than a side-channel file or volume poller. Yielding progress envelopes directly from the remote execution method lets the local client forward active-node updates into ComfyUI's websocket stream in real time without extra storage commits, reloads, or cleanup logic.
• Some custom-node packs register HTTP routes or prompt handlers at import time through PromptServer.instance. A headless remote worker still needs to provide a minimal PromptServer.instance shim with routes, app, supports, and no-op send methods, or those packs fail during external custom-node initialization before any workflow execution begins.
• For good remote UX, one generic "executing" node state is not enough. The frontend needs separate visual states for "setup", "ready in Modal", "currently executing remotely", "finished for this run", and "workflow fully done", with whole-workflow success clearing the temporary remote states back to the normal idle outline.
• The global Modal activity badge needs the same cleanup discipline as the per-node overlays. If a prompt entry survives in badge state after the prompt itself is gone, the UI can get stuck showing an orange "setup running" badge even though the workflow finished; pruning orphaned prompt ids fixes that.
• One orange badge state is not expressive enough. The frontend can use queue-time modal_status setup for "Syncing graph with Modal", then switch to a separate waiting state when the proxy begins executing but before the first streamed remote-node event arrives, which makes the remote lifecycle much easier to understand from the UI.
• Browser backgrounding can make the badge look stale even when the underlying prompt state already advanced. Deriving the global pill phase from live per-prompt node state, and refreshing it again on visibilitychange / focus, makes the pill recover promptly when the user returns to the tab.
• Subgraph-expanded prompt ids need UI aggregation just as much as backend marker resolution. If a streamed remote event targets 24:23, the visible outer subgraph node 24 should derive its phase from all descendant remote ids under that prefix; otherwise the outer node can stay stuck in setup even though the internal remote nodes are already running or complete.
• Warm-container caches and loader reuse can conflict with modal.Volume.reload(). If a request truly needs a fresh volume reload after uploading new assets, the worker may have to clear warm loader caches, drop reusable prompt executors, unload ComfyUI models, and retry the reload once to get past Modal's "open files preventing the operation" error.
• The preparation status event must use ComfyUI's inner status payload, and the queue-list initialization notification remains useful when keyed by prompt id. Detailed setup progress belongs in the extension's Configurator panel and global activity badge until real node execution begins.
• Queue-time setup failures are not node execution failures. Preserve the server response on PromptExecutionError and update extension-owned error state, but do not synthesize an execution_error; otherwise ComfyUI creates a misleading node crash report and can retain it across a later healthy run.
• Synced asset paths cannot stay as abstract /assets/... markers inside the Modal worker. Remote execution needs those markers materialized to real absolute files under the mounted storage root, and ComfyUI's folder_paths helpers need an absolute-path fast path so loader nodes can still resolve them.
• A local filesystem mirror is not enough in COMFY_MODAL_EXECUTION_MODE=remote. Queue-time sync must upload assets into the actual named Modal volume via the local SDK, and the remote worker must call vol.reload() before execution so warm containers can see newly uploaded files.
• Once warm-container caches keep models and executors alive, unconditional vol.reload() becomes wrong: Modal can reject reloads while files from the mounted volume are still open. The queue side already knows whether a request actually uploaded anything new, so that fact should travel in the payload and only requests with fresh uploads should reload the mounted volume.
• Fast cancel-and-restart sequences can still race the container's file-handle cleanup even after warm caches are cleared. For the specific Modal "open files preventing the operation" reload failure, the remote worker needs a generous bounded retry window after cache/model cleanup rather than a single immediate retry.
• That queue-time requires_volume_reload bit is not enough once one logical request fans out into many per-item Modal calls. Mapped execution also needs a stable per-request reload marker so each warm container can reload the mounted volume once for that asset snapshot and skip the duplicate reloads from later mapped items.
• The reload marker also has to be request-wide across remote components, not minted separately per component payload. Otherwise a single workflow that rewrites into several remote chunks can still force repeated vol.reload() calls on the same warm container as it advances from one chunk to the next.
• Even after bounded retries, vol.reload() can still fail on a warm container that already has the right hashed assets visible on disk. In that case the worker should verify the payload's referenced mounted-volume paths and proceed when they are already present, rather than treating a redundant reload failure as fatal.
• ModalMapInput cannot blindly repeat every execute target in its coarse remote component. If a mapped branch shares a non-transportable upstream dependency with an unmapped remote sibling, the payload needs to separate one-time execute nodes from per-item execute nodes so the sibling branch still runs once while the mapped branch fans out per item.
• Mapped Modal fanout needs stable local worker lanes, not just a semaphore over one task per item. The UI can only render sensible concurrent progress if each in-flight mapped run carries a durable lane id; otherwise parallel item updates all target one node-progress slot and the bar flickers incoherently.
• Hybrid mapped components also need stream-level UI filtering. A warm PromptExecutor can still emit cached executed or preview UI events for static sibling nodes during per-item mapped runs, so the local relay has to drop any streamed UI event whose node id is outside the current mapped/static payload's execute and boundary-output node set.
• Modal proxy nodes that sit on a mapped boundary must opt into ComfyUI INPUT_IS_LIST semantics themselves. Otherwise ComfyUI can auto-map the proxy once per item before Modal-Sync sees the batch, which makes the entire hybrid component rerun per item instead of letting the internal mapped scheduler split only the intended branch.
• Once a proxy opts into INPUT_IS_LIST, ComfyUI also wraps the proxy's hidden payload input itself. The proxy has to unwrap singleton lists on original_node_data as well as ordinary kwargs, or it will reject valid rewritten payloads with a misleading "must be a mapping" error before remote execution even starts.
• Rewritten remote prompts need a last-mile canonicalization pass before execution. If a socket index leaks through as ["node", [0]], a scalar input arrives as [4], or an execute node id arrives as ["12"], ComfyUI's executor eventually crashes trying to treat the list like an integer or node id; normalizing those singleton wrappers on both the local and Modal-side runners avoids that failure mode.
• Once the easy wrapper shapes are normalized, the next debugging step should be structured worker-side diagnostics, not more blind guesses. Logging the normalized execute-node ids, boundary outputs, suspicious remaining wrapped prompt inputs, and the raw ComfyUI execution_error payload makes the surviving malformed field visible on the very next remote failure.
• Hybrid mapped components cannot hand the whole coarse component prompt to every sub-run. Even when a static sibling should execute once, PromptExecutor still validates the full prompt, so static and per-item sub-runs must prune the rewritten prompt to the dependency closure of their own execute_node_ids or the unrelated branch's batched inputs can still fail validation.
• That trim pass also has to tolerate stale execute ids that survive into a sub-run payload after rewrite. Dependency resolution should drop any execute target missing from the current trimmed prompt instead of assuming every listed node id still exists and crashing with KeyError.
• Prompt canonicalization also has to run on hydrated boundary inputs, not just on the stored subgraph prompt. Otherwise a proxied scalar boundary value like [4] can be spliced back into an INT socket unchanged and fail later inside PromptExecutor with int(list).
• A generic PromptExecutor failure string is not enough for debugging rewritten remote prompts. The worker should surface execution_error context like node_id, node_type, and current_inputs, and it should preflight primitive widget sockets for raw list literals before execution so the first visible error names the offending node/input/value instead of only reporting int(list).
• Modal's public Python SDK does not expose a direct "kill this @app.cls container" call from the local client. If we want crashy warm GPU workers to disappear immediately, the practical design is to have the local side opt into crash teardown and the remote worker schedule os._exit(1) on itself after surfacing the exception.
• Content-addressed sync should trust the deterministic payload path, not only the sidecar marker. If a crash happens after uploading a hash-named asset but before writing its .done marker, the next run should backfill the marker and reuse the existing payload instead of trying the same upload again and failing with FileExistsError.
• The streamed Modal generator path needs the same tensor-aware serialization rules as ordinary node input/output transport. If a progress or result envelope leaks a raw torch.Tensor, Modal's JSON encoding fails before the local client ever sees it, so the stream layer should normalize event payloads through the shared serializer too.
• Modal volume metadata calls can become a bottleneck on their own. Repeated VolumeListFiles checks for the same marker path are worth memoizing for the life of the queue request, and transient ResourceExhaustedError responses should be retried with backoff instead of surfacing immediately as hard prompt failures.
• Modal volumes are a poor fit for high-cardinality existence checks. For content-addressed uploads, the fast path should be a separate Modal Dict keyed by digest, with the volume reserved for the payload bytes themselves. That changes unchanged syncs from "many VolumeListFiles calls" into "a few metadata reads," and it avoids rebuilding a marker-file protocol on top of the volume API.
• A digest index is not enough by itself if the metadata store can outlive the underlying volume contents. The sync index needs a volume-epoch guard that is validated against the volume itself, otherwise a fresh or recreated volume can incorrectly reuse stale dict records for payloads that are no longer present.
• Modal first-run deploy needs a readiness phase, not just a deploy phase. app.deploy() returning does not guarantee that Cls.from_name(...) will succeed immediately from every caller, so concurrent warmup and payload threads should share one thread-safe "deploying -> lookup-ready" state and wait for lookup readiness instead of treating early lookup misses as grounds for a second deploy.
• Modal volume SDK calls should not run directly on the aiohttp request event loop. Running them in a worker thread avoids AsyncUsageWarning, and a missing path from Volume.listdir() should be treated as an ordinary "not uploaded yet" cache miss instead of a hard failure.
• The Modal container can pick up a non-package utils module before ComfyUI initializes. Preloading ComfyUI's own top-level utils package from the bundled source tree prevents utils.install_util import failures during init_extra_nodes(...).
• If the goal is "no idle billing, but no minute-long rebuild every run", the right Modal pattern is a deployed app with min_containers=0, not ephemeral app.run(). That lets Modal reuse the built image and deployed objects while still scaling compute down to zero between executions.
• Modal hardware selection belongs in deployment configuration, not in request payloads. For this repo, the remote class should read a configurable GPU string from environment at deploy time, and changing it later requires a redeploy or app recreation because existing deployed app hardware does not update in place.
• Modal memory snapshots only help once the app is deployed. They are a good default for CPU-side initialization work, while GPU memory snapshots should remain opt-in because Modal still labels them alpha and they may require additional runtime refactors.
• If the goal is faster cold restarts for heavyweight diffusion loaders, request-time warmup is not enough. The loader work has to move into @modal.enter(snap=True) behind a stable per-model-set snapshot profile, otherwise fresh workers will still pay one UNETLoader/CLIPLoader miss each even when GPU snapshots are enabled.
• That snapshot profile must also survive any split-payload rewriting. If static/mapped phase payload builders drop snapshot_profile_key, mapped worker lanes will look up the default RemoteEngine class and all the snapshot work will appear to do nothing in logs.
• Preserving snapshot_profile_key in split-phase helpers is still not enough if the key only exists as a transient local variable during deployed-class lookup. Once a coarse payload synthesizes a loader snapshot profile, that key has to be written back onto the payload itself so later mapped/static descendants inherit it instead of recomputing from a loader-free subgraph and falling back to snapshot_profile=None.
• Even that runtime-side coarse-payload writeback can still be too late when the queue rewrite has already materialized split proxy payloads. In that case the rewrite step itself must stamp the derived snapshot_profile_key onto the split payloads before they are inserted into the rewritten prompt.
• A payload carrying snapshot_profile_key is not sufficient by itself if the underlying snapshot-profile record has never been stored. Remote lookup should opportunistically store the record whenever it can still reconstruct the loader prewarm plan set, even if the payload already arrived with the key prefilled.
• GPU snapshots are not worth enabling blindly for generic no-profile workers. A snapshot_profile=None container that snapshots only “ComfyUI core initialized” state can still be unstable after restore and does not buy the model-loaded cold-start win we care about, so the safer policy is to reserve GPU snapshot prewarm for real loader-profile:... workers only.
• The same rule should apply to proactive warmup. If GPU snapshots are enabled but _build_prompt_warmup_request() cannot derive a loader profile, the scheduler should skip the ::warmup lookup entirely instead of instantiating a generic snapshot_profile=None class that cannot create a useful model-loaded snapshot.
• A seeded-lane scheduler should not impose a global seed barrier. Once each lane has its own stable bound worker, the highest-throughput policy is to let each lane await only its own seed:* phase and start mapped items immediately, otherwise one slow cold lane wastes GPU time on every other already-ready lane.
• Prompt completion also should not wait for every seeded lane task to return. If faster lanes already drained the mapped item queue, the scheduler can return immediately, clean up only the lanes that actually seeded, and let any extra slow lane finish booting in the background so it becomes a warm worker for the next prompt instead of blocking the current one.
• Modal snapshot identity should follow immutable cold-start state, not scheduler lane identity. If lane affinity is folded into the @app.cls parameters, Modal builds and caches one GPU snapshot per slot even when every lane loads the exact same ComfyUI runtime and model profile. Keeping only snapshot_profile_key and GPU-snapshot mode in the class parameters lets all lanes share the same expensive cold-start snapshot while lane assignment stays a local scheduling concern.
• Deployed Modal class behavior that matters at snap=True should not be inferred from environment inside the remote container. The container does not automatically inherit the local ComfyUI launch env, so flags like GPU-snapshot mode need to be carried as deploy-time/class parameters if the snapshot hook logic depends on them.
• Native ComfyUI video is already decoded and uniformly sampled before it reaches a multimodal processor. Passing that frame list to Transformers without a top-level do_sample_frames=False chat-template option makes SmolVLM try to sample it again and fail because an in-memory frame list has no source-video metadata; Transformers 5.1 does not honor that option inside the newer processor_kwargs compatibility wrapper.
• ComfyUI's loaded_models() registry is not a reliable assertion for standalone VAE residency: a real VAELoader plus VAEEncode can run successfully while that registry remains empty. A same-worker co-residency proof should instead show that VAE execution succeeds between two LLM calls and that the second LLM call is a resident cache hit, proving the image workload did not evict the LLM.
• A remote proxy cannot treat every incoming list literal as just another scalar boundary input. Ordinary remote components also have to preserve ComfyUI's implicit zipped batch semantics across the boundary, or primitive widget sockets like seed: INT end up receiving whole lists such as [10, 11, 12, 13] instead of per-item values.
• That implicit batching path also needs the same hybrid static-vs-mapped split as explicit ModalMapInput. If two remote execute nodes share a non-transportable upstream object like MODEL but only one of them depends on the multi-item boundary input, the sibling execute node still has to run once, not once per batch item.
• Suppressed per-item progress streams should not overwrite display_node_id with the component representative. Lane ids are enough to separate concurrent runs, so progress bars can stay attached to the actual executing node instead of collapsing onto the first node in the remote island.
• Placeholder lane-start updates are counterproductive for mapped progress. If the local scheduler paints a lane on the component representative node before any real node_progress event arrives, that ghost bar can outlive the later real-node updates and make the first node in the island look like it is still sampling.
• Aggregate batch-completion updates are a different kind of signal from real node progress. Reusing the same per-node rendering path for both makes ordinary remote nodes look like they emit sampler-style progress when they do not, so the frontend should treat aggregate mapped progress as global-only UI state.
• Aggregate mapped progress is still useful, just not as a fake node bar. The right presentation is UI that is explicitly batch-oriented: a shaded global status pill for overall completion and a small 3/16-style badge on the representative remote node.
• Prompt-wide UI state must not be rebuilt from one component's latest status event. Once the backend has published the full remote-component membership for a prompt, later per-component modal_status events should merge into that prompt metadata rather than shrinking it to whichever component happened to report most recently.
• ComfyUI progress_state payloads are not all true numeric progress. Upstream nodes can report trivial running states like value=0, max=1, and forwarding those verbatim makes loaders and text encoders sprout fake progress bars. The remote worker should treat only states with a meaningful progress range as node-local progress.
• Queue-time Modal setup is too long-lived to hide behind one generic "Syncing graph with Modal" label. The websocket status payload should be able to carry human-readable setup detail like packaging the custom-nodes ZIP, naming the asset currently uploading, and a short finalizing phase while the local side is still receiving remote outputs.
• Request-wide reload markers are useful for deduping vol.reload() across components, but the reload trigger itself should still be component-specific. If any uploaded asset in the workflow marks every component requires_volume_reload=True, warm containers end up retrying reload for components that never reference the fresh file.
• Reload diagnostics need to speak in terms of mounted-volume paths the payload actually references. Logging both the newly uploaded relevant paths and the broader referenced mounted-volume paths makes it much easier to see whether reload is being blocked by files the component truly needs or by stale warm-container handles from previous work.
• Ready/green is a prompt-acceptance state, not a per-component execution state. If the frontend waits for each proxy node's first executing event before marking nodes ready, most of the remote graph stays orange even after the rewritten prompt has already been accepted by Modal. The queue-success path should flip the whole remote prompt to ready immediately, while purple remains reserved for the currently active node.
• Websocket setup events are not guaranteed to land before the HTTP queue response resolves. If the queue-success path relies on prompt metadata that only arrived over websocket, backend-expanded upstream nodes can remain orange because the frontend only knows about the originally toggled workflow nodes. The accepted queue response should echo the resolved remote-node ids and component map so ready-state painting is deterministic.
• Component membership needs a reverse lookup as well as a representative lookup. Mapped progress and native ComfyUI execution events often arrive keyed to an ordinary node inside the remote island, and if the frontend can only expand representative ids it leaves the rest of that component stuck orange even though the island is already running.
• Streamed Modal node-progress events need to preserve the real executing node id even when ComfyUI also reports a display node id. Using only the display id collapses fan-out progress onto the mapped-input receiver or another proxy node, which hides the actual executing node and breaks lane cleanup when the mapped worker finishes.
• Queue-time rewrite has enough graph structure to estimate remote concurrency before execution starts. A stage decomposition of the remote component DAG is cheap to compute during prompt rewrite and useful to carry in extra_data.modal.
• Queue-time planning is still too early to launch Modal warmup containers because the prompt may only be waiting in ComfyUI's queue behind another remote workflow. Store the staged component plan, but tie actual warmup top-ups to runtime signals such as exact mapped fan-out once the proxy execution path is active.
• In-node progress bars are easy to lose underneath ComfyUI widgets. Rendering the remote progress UI in a small temporary panel below the node keeps sampler progress, parallel lanes, and batch badges readable without competing with text boxes or other widget chrome.
• Remote subgraph execution has to apply ComfyUI's primitive widget coercion before it calls PromptExecutor or manual local node execution. ComfyUI's validator converts literal INT, FLOAT, BOOLEAN, and STRING inputs in-place, and skipping that step remotely leaves values like steps=18.0 alive long enough to crash custom nodes that assume true Python integers.
• ComfyUI's CacheKeySetInputSignature is reusable for a distributed node-output cache, but the raw key object is an in-memory frozenset/tuple structure rather than a persistence-ready string. The workable approach is to canonicalize that structure into stable JSON, hash it into an NC_... key for Modal Dict, and only persist entries whose outputs are both transport-safe and below a strict size ceiling.
• Remote boundary injection can splice literal tensors back into CacheKeySetInputSignature objects for otherwise cacheable nodes like KSampler. Those tensors should contribute a stable content digest to the distributed cache key instead of poisoning the whole signature with ComfyUI's in-memory Unhashable marker.
• For remote distributed caching, cache_key_set.get_data_key(node_id) is not always reusable as-is. CacheKeySetInputSignature may already have converted boundary tensors into Unhashable, so the remote cache layer has to rebuild the same ancestry-based input signature from dynprompt and is_changed_cache before canonicalizing it for persistence.
• Restoring persisted outputs before PromptExecutor.execute() is too early. execute() immediately rebuilds the live cache-key set with its own set_prompt() call, so a distributed cache restore must run inside or immediately after that phase if the restored entries are supposed to participate in the actual execution walk.
• Mapped progress lanes have to preserve the scheduler-assigned lane id all the way through the remote worker. If the worker hardcodes every item's node progress to lane 0, concurrent mapped items overwrite each other in the frontend and upstream sampler bars appear to flicker or disappear entirely.
• Static mapped sub-runs cannot reuse the same "suppress status chatter" rule as per-item mapped lanes for numeric node progress. Those static phases usually do not have a lane id at all, so the local relay has to keep lane-less node_progress events for execute/boundary nodes or upstream samplers like KSampler will run remotely without ever showing a bar in the UI.
• Once those lane-less static progress events are forwarded, they also need their own explicit clear path. Suppressed static streams do not forward the terminal execution_success status event into the frontend, so relying on phase completion to clear ordinary progress leaves the last 100% sampler bar stuck onscreen.
• Distributed node-cache restore and persist should not be treated as unrelated phases. If one prompt run restores NC_<hash> into the live PromptExecutor cache and then persists the same node/key again unconditionally, Modal Dict churns on pure cache hits for no benefit; the persist pass should recognize restored hits and skip rewriting the same key.
• A distributed node-cache size cap that is generous enough for medium images can still be too expensive in practice. In these logs, repeatedly serializing and writing ~25 MiB cache entries dominated the post-execution phase, so the default cap should bias toward smaller transport-safe artifacts unless there is a clear reason to keep large intermediates.
• Once the persist phase knows a node is a restored-hit, that check has to happen before entry serialization and size estimation. Otherwise the code still burns CPU serializing outputs it already knows it is going to skip writing.
• Distributed node-cache restore also benefits from fan-out/fan-in structure. The old per-node await key -> await Dict get -> deserialize loop made total restore time roughly the sum of all independent Dict reads, while batching those lookups concurrently preserves prompt-order hydration but trims the network-bound portion substantially.
• Distributed node-cache restore must be boundary-aware. A downstream cache hit can make ComfyUI skip upstream dependency execution, so if that upstream node is also a boundary or session-bridge output, restoring only the downstream hit leaves no local cache entry to collect. Skip those downstream hits unless the required boundary ancestors are already local or restored too.
• Remote headless PromptServer shims need to track ComfyUI's UI-side helper surface, not only send_sync. Core API nodes like GetImageSize call PromptServer.instance.send_progress_text() during execution, and a missing no-op method is enough to fail otherwise valid headless remote runs.
• Implicit mapped fan-out cannot treat every Python list as a batch axis. ComfyUI semantic payloads like CONDITIONING are also list-backed, and splitting them per item corrupts the conditioning structure into fragments like ["cond", {...}] -> "cond" that later explode in sampler_helpers.convert_cond. Only explicitly batchable transport types and list-wrapped primitive sockets should participate in that automatic zipping path.
• Primitive widget sockets are not always scalar once a graph is running. Custom nodes can legally emit list(INT) or similar values into an INT input and rely on ComfyUI's normal over-list execution behavior, so remote validation should only reject list values that are still raw widget literals, not boundary-fed or upstream-fed runtime values.
• Singleton-list cleanup on the remote boundary needs the same nuance. It is correct to unwrap transport-added wrappers like [4] -> 4 and [[["node", [0]]]] -> ["node", 0], but wrong to flatten single-entry semantic payloads like [[tensor, meta]] for CONDITIONING; that shape change is enough to turn convert_cond() into a tensor indexing crash.
• Outer implicit batching cannot blindly fan out just because a boundary input is a list. If that split boundary lands on a node with INPUT_IS_LIST=True, the node needs the whole list in one call and ComfyUI should own any downstream over-list execution inside one PromptExecutor run; replaying the whole remote component per item duplicates scalar sibling work and turns single outputs into repeated lists.
• Local ComfyUI caching for remote proxies is extremely sensitive to transient payload fields. It was not enough to strip only prompt_id: cache-safe proxy payloads also had to drop run-scoped extra_data, volume-reload bookkeeping, and prompt-scoped session fields from original_node_data. Session-backed split proxies became cache-safe only after their exported bridge outputs switched from raw prompt-scoped session refs to durable replay refs backed by replay metadata in Modal Dict.
• Distributed cache keys for later split phases cannot rely on the prepared prompt alone. Once a non-transportable boundary input has been injected, the live prompt only contains a Python runtime object like ModelPatcher, and rebuilding CacheKeySetInputSignature from that prompt reintroduces the same unhashable-object problem. The fix is to carry a stable provenance fingerprint for each non-transportable boundary input, derived from the original upstream prompt subgraph, and hash that provenance instead of the live object identity.
• Live combined logs confirmed the current split-proxy weak point: worker affinity is best-effort, not a hard lease. One prompt-scoped session can be created on container A and a later phase can still land on container B, producing a RemoteSessionStateError even though neither worker is corrupted. That class of failure should not trigger container teardown; it needs stronger routing semantics.
• ModalMapInput cannot be treated as a remote-only marker. If the marker stays local and only the downstream sampler is remote-marked, rewrite still has to recognize that boundary as mapped work. Otherwise the remote component falls back to an ordinary subgraph, the boundary io_type degrades to *, and batched primitive lists like list(INT) seeds stop itemizing before they hit the sampler.
• A wildcard mapped boundary can still carry recoverable type information at runtime. If the value is a list of LATENT-like mappings or every target socket declares the same concrete input type, implicit batching should use that effective type instead of treating io_type="*" as an unconditional broadcast and handing a raw Python list to nodes like KSampler.latent_image.
• Modal background pollers can outlive the request-scoped gRPC client by a few milliseconds during asyncio.run() teardown. Threads that keep calling Modal SDK objects like Dict.contains() in that window must treat modal.exception.ClientClosed as a normal shutdown signal, otherwise successful runs end with noisy unhandled thread exceptions after the real work is already done.
• Split mapped proxies need their own seedable copy of the static producer phase. If the mapped proxy only carries remote-session bridge refs, fresh mapped workers have no way to populate those refs except replaying the durable bridge record payload item-by-item. Carrying a bridge-only static_phase on the mapped proxy lets the local scheduler prime each mapped lane once on its bound worker before sending any per-item sampler calls there.
• Seeded mapped lanes must not inherit clear_remote_session=True. If the seed payload clears the prompt-scoped session as soon as it finishes, the first mapped item on that lane throws away the whole point of seeding and has to fall back to warm-bridge restore immediately. Session teardown belongs only in the explicit final cleanup payload for that lane.
• Durable bridge replay metadata should carry directly restorable payloads for the remote-only types that can be serialized safely. CONDITIONING is the first useful case: its nested tensors fit the container RAM budget, so fresh workers can deserialize the conditioning straight back into the current remote session and skip producer replay for that bridge ref entirely.
• Nontransportable mapped state needs a different batching rule from raw semantic payloads. A plain CONDITIONING Python list usually encodes one conditioning structure and must stay broadcast, but a Python list whose elements are all remote session refs encodes per-item mapped remote values. Those ref lists need to split item-by-item even when the declared io_type is MODEL or CONDITIONING.
• Implicit mapped fan-out also needs its own session-lifetime rule. If every synthesized ::item:* or ::static payload inherits clear_remote_session=True, the first sibling to finish can tear down shared remote state while later siblings are still queued. Cleanup has to move into one explicit final payload that runs after all mapped siblings complete.
• Refocus replay has to treat progress-only prompt state as live UI state until queue/history reconciliation proves otherwise. If a cancelled prompt's lane progress is replayed after the terminal marker expires, the replay timestamp should become that prompt's startedAt; otherwise stale progress looks fresh and can turn nodes purple again.
• Some saved Flux/Flux2 diffusion models can use RMSNorm keys ending in .weight while current ComfyUI detection expects .scale. The Modal worker should add non-destructive in-memory .scale aliases before load_diffusion_model_state_dict so synced saved models can still be classified without rewriting the safetensors file.
• The global Modal status pill must treat cancellation state as prompt-scoped and supersedable. Because cancelling has high display priority, a cancelled prompt with no live remote work can otherwise keep showing Cancelling Modal workflow over a newer prompt's setup or running state.
• Frontend mapped-progress lanes cannot be keyed only by laneId at prompt scope. Different remote components legitimately reuse local worker lane numbers like 0 and 1, so the UI has to namespace lane ownership by component representative or concurrent components will clear each other's purple bars and appear to flicker between active and ready states.
• Streamed remote progress should suppress native proxy executing churn, but not the proxy's terminal completion. If the frontend drops every native phase update after streamed progress starts, a single-node remote component can stay stuck purple because it only ever receives streamed finalizing and never consumes the completion event that should turn it green.
• The in-process "already auto-deployed" cache cannot be treated as authoritative once operators can delete the Modal app out-of-band. A later app not found lookup has to invalidate that cache entry and redeploy instead of assuming the earlier successful deploy still exists remotely.
• With deployed modal.Cls handles, "app missing" is not only a lookup-time failure. A stale handle can also survive lookup and then fail later during hydrate() or the first remote() call. The invoke path has to treat that the same way as a missing-app lookup, auto-deploy, and retry before surfacing an error locally.
• Mapped lane observability needs a distinct setup state, not just "first real progress wins." If the backend emits a dedicated setup_only lane progress event as soon as a worker starts seeding, the frontend can render a translucent pulsing placeholder bar while the lane is provisioning and then seamlessly replace it with real node progress once execution begins.
• Remote node UI needs two separate non-executing signals to stay legible under Modal fan-out: "ready but not in the active component" and "reused from the persisted node cache." The former should be derived locally from component membership and active_node_id, while the latter needs an explicit backend event because cached nodes often never emit ordinary execution progress at all.
• Once mapped fan-out is allowed to return before every spare lane finishes seeding, late failures from those detached lanes must not flow back through the main prompt future. They are no longer part of the critical path, so the scheduler has to ignore them after all outputs are already complete.
• Combined first-run and second-run performance logs are now clear enough to prioritize work. In this workflow, uploads and volume reload were already cheap, while the recurring non-sampling bottleneck was static-phase model loading (UNETLoader + CLIPLoader at roughly 24-25s total per run). The second run also showed that local proxy caching can skip the static proxy dispatch while still leaving a large replay cost inside the mapped phase, which makes two no-brainer optimizations obvious: reuse heavy loader outputs on the same worker, and avoid replaying the whole static producer branch just to reconstruct session-backed bridge values.
• Modal deployment updates are too permissive for GPU-type changes. Because a named deployed app can be updated in place while its resource expectations and synced storage assumptions may have come from an older COMFY_MODAL_GPU, the cloud entrypoint should fail local app construction when that app name already exists and require the operator to stop/delete the old app before building the new one.
• ComfyUI's cache setup indexes NODE_CLASS_MAPPINGS[class_type] before node execution starts, so missing custom-node registrations can surface as a raw KeyError from get_immediate_node_signature. Remote workers should preflight prompt class types after bundle extraction, retry external custom-node import once, and raise an explicit missing-class error if the registry is still incomplete.
• A successful custom-node sync only proves the package files reached Modal storage; it does not prove ComfyUI imported the package. When a synced class remains unregistered, the remote error should include whether the extracted bundle contains Python files mentioning that class and which top-level package they came from, so the next debugging step is import failure logs instead of another sync toggle check.
• Stale deployed Modal apps are not limited to missing-app failures. The local client needs an explicit app protocol handshake with RemoteEngine, and an unversioned or older remote app should be stopped and replaced before payload execution so local code changes are not silently tested against an old image.
• Remote execution and asset sync must derive their mode from the same settings snapshot. If sync uses LocalMirrorVolume but invocation reads a newer raw COMFY_MODAL_EXECUTION_MODE=remote value, the logs misleadingly show custom-node sync success while the Modal worker cannot see the bundle; queue rewrite should fail before dispatch in that split-brain state.
• Modal volume duplicate-file errors are benign for content-addressed upload paths. If /custom_nodes/entries/.../<sha>_custom_nodes_bundle.zip already exists, the digest in the path is the identity check, so the sync engine should mark the upload as present instead of failing the run.
• ComfyUI's execution-error current_inputs display stringifies complex values like CONDITIONING, so a quoted tensor dump in the error payload is not proof that transport serialization converted the conditioning to text. If the real failure is a missing sampler input, preflight required node inputs before PromptExecutor so the error points at the malformed prompt shape directly.
• Modal proxy execution needs prompt-level admission control, not just per-call awaiting. ComfyUI can have multiple async proxy executions in flight, so a second prompt can otherwise dispatch Modal work while the first prompt's remote components are still running. Gating by prompt_id preserves within-prompt component parallelism while making later prompts wait for the active prompt's remote calls to drain.
• Prompt cancellation creates two separate state problems. The local proxy gate must be cancellable while it is waiting behind another prompt, otherwise a cancelled waiter can later acquire the gate and confuse subsequent prompts. The frontend also needs a terminal prompt marker so late Modal status/progress events from a cancelled remote worker are ignored instead of recreating the global status pill after cleanup.
• Queue-time Modal UI updates are not execution-start signals. When a user submits a new Modal prompt while an older Modal prompt is still active, the frontend must register the new prompt metadata but suppress setup/ready node coloring until a real ComfyUI execution or Modal progress event arrives for that prompt. Otherwise a queued prompt visually appears to have started before ComfyUI has actually advanced past the current remote workflow.
• Node progress needs a short completion lifecycle. Clearing progress immediately on executed makes completion visually abrupt, but leaving the progress payload around keeps the node looking active. Marking progress as fading lets the node switch to complete styling while the bar fades out and is removed after a bounded delay.
• Local proxy execution is not proof that a Modal worker is running the graph. The frontend should keep the global pill in an orange waiting state while ComfyUI is executing the proxy and Modal is still provisioning capacity, and only switch to green running once streamed remote feedback includes an active remote node or actual node-progress data.
• A Modal cancellation flag can be written before the remote method body starts if the user cancels while Modal is still provisioning capacity. Remote execution registration must not clear that key on entry, and the local proxy still needs a bounded post-interrupt grace period so ComfyUI is released even when remote code is slow to observe the interrupt.
• ComfyUI's interrupt flag is cooperative, so some remote workflows keep running after the shared Modal cancellation flag is consumed. The remote interrupt monitor should schedule a short, cancellable worker-process exit when it observes cancellation, and cancel that exit only if the prompt unwinds before the timeout.
• A split mapped proxy can carry an embedded static_phase that is executed later for mapped lane seeding. Any original boundary inputs required by that embedded phase must still be wired onto the mapped proxy node, even if they are not part of the mapped payload's top-level boundary_inputs; otherwise the remote worker reaches lane seeding with no hydrated value for names like remote_input_0.
• Remote execution streams do not reliably produce explicit completion events for every node, especially quick nodes or nodes without numeric progress. The local stream bridge still has the remote subgraph prompt, so it should send dependency ancestors for each active/progress node and let the frontend mark those ancestors complete before painting the current node active.
• Coarse component-level execution status is allowed to say "these nodes are ready", but it must not downgrade same-prompt nodes that were already marked complete. Some prompt input structures also wrap links inside nested lists or dicts, so graph ancestry helpers need recursive link discovery instead of checking only top-level [node_id, output_index] values.
• Syncing a custom-node package's Python files is not enough for remote execution. If the package carries a requirements.txt, those requirements have to be part of the Modal image build, because subprocess-heavy nodes like PiD import their vendored code inside the remote container and fail late with ordinary ModuleNotFoundError errors such as missing omegaconf.
• The remote planner should never execute a workflow node merely because it is downstream of a marked remote node. Unmarked preview/UI branches are not enumerable across custom nodes, so they must stay in the local prompt and consume proxy boundary outputs instead of being absorbed into remote component_node_ids or execute_node_ids.
• Component membership is useful as visible debugging state, not only hidden scheduler metadata. Prompt-local component labels can be derived from the backend component list and drawn on each Modal node, which lets users see which remote nodes are assigned together without exposing long representative node ids.
• Worker affinity cannot be the only protection for sampler-produced bridge values. If a downstream phase lands on a fresh worker, serialized LATENT bridge records let the worker restore sampler output directly while still preserving the invariant that sampler producer components are never replayed as a fallback.
• A MODEL bridge can be produced inside a sampler component without being sampler output itself, for example UNETLoader -> LoraLoaderModelOnly -> KSampler. Durable rehydration should trim to the requested model output's dependency closure and reject the plan only if that closure contains sampler work, instead of treating the whole producer component as replay-only.
• CLIP bridges have the same failure mode as linked MODEL bridges when a sampler component exports shared text-encoder state. Durable bridge rehydration should cover loader-style remote-only outputs such as MODEL, CLIP, and VAE, not only model patchers.
• Durable subgraph rehydration plans can themselves depend on earlier bridge refs. They must execute inside the downstream target remote_session, otherwise nested bridge resolution has no session to restore into and fails before it can rebuild the requested loader-style output.
• Dispatch-time feedback is separate from real remote execution. When Modal is provisioning or all warm workers are occupied, the local proxy can wait tens of seconds before the first streamed active-node event arrives, so the local dispatch path should emit a lightweight starting status immediately and let later streamed progress replace that temporary orange state.
• Prompt-wide active-node state is not a safe completion signal under parallel remote execution. If two components stream sampler progress at the same time, alternating events must not fade or complete the previous active node unless both nodes belong to the same remote component.
• Modal UI preflight and dispatch waiting need separate states. Orange setup should mean graph/package/upload work, while yellow starting should mean a component has been locally dispatched and is waiting for Modal to stream real node execution. The global pill should also ignore stale active-node ids unless that node still has active state or live progress.
• ComfyUI targeted interrupts only call nodes.interrupt_processing() when the prompt id still appears in the native running queue. Modal proxy calls can be blocked on remote work after that native check stops seeing the prompt, so active Modal prompt ids need to be exposed back through the local prompt queue for /interrupt {"prompt_id": ...} to propagate cancellation instead of logging "not currently running".
• Cancellation can also arrive before a Modal payload reaches the registered remote-call wrapper, especially during first-run auto-deploy or deployed-app readiness polling. The outer local waiter must write the shared cancel flag and enforce the same local release grace during deploy/provisioning, and the worker must refuse to start the remote payload if cancellation was already observed.
• Users need immediate cancellation feedback even when Modal cleanup is still cooperative. The frontend should intercept ComfyUI /interrupt requests, mark the prompt as cancelling, and ignore nonterminal late progress until the native terminal event clears the UI.
• Prompt-level Modal cancellation has to release both the remote work and the local prompt admission gate. If a user cancels during Modal deploy/readiness, ComfyUI can mark the prompt interrupted while the background Modal call still owns the local gate, leaving the next prompt stuck behind an already-cancelled prompt id.
• ModalMapInput does not remove the need for ordered split proxies. If a mapped remote component has a local re-entry dependency, such as an early sampler preview feeding a local text/image node that returns into a later mapped sampler, the static side must still split into ordered phases before the mapped phase or the rewritten prompt can create a false dependency cycle.
• Implicit mapped cancellation has to stop the local item scheduler, not only the remote payload that is currently running. If a sampler lane observes a prompt interrupt, queued sibling items and final cleanup payloads must be suppressed immediately or a quick cancel/restart can dispatch stale mapped sampler work after the UI already reports cancellation.
• ModalMapInput should accept scalar primitive values as a one-item map. This keeps the marker useful as a no-op adapter when an upstream control value like INT is not batched, while preserving fan-out behavior for lists, tensor batches, and LATENT mappings.
• The implicit mapped dispatcher needs an explicit ordinary-subgraph bypass. When a batch-like boundary targets an INPUT_IS_LIST node, the helper correctly decides not to fan out, but re-entering the public dispatcher without disabling implicit mapping just rediscovers the same split and recurses forever.
• Mapped CONDITIONING outputs need explicit item-boundary metadata. A single ComfyUI conditioning value is already list-shaped, so shape alone cannot distinguish "one conditioning payload to broadcast" from "ordered conditioning payloads from a mapped CLIP run"; downstream implicit mapping must split only the latter.
• Remote session bridge resolution must recurse through list-shaped outputs. Over-list Comfy execution can produce one output slot that contains several bridge refs, and resolving only top-level refs leaves downstream samplers seeing unresolved refs or repeated first-item state.
• Rebuilt remote node-cache signatures must use stricter prompt-link detection than general prompt rewriting. Runtime values such as CONDITIONING can contain nested two-item lists, and treating those data lists as [node_id, output_index] edges makes first-run distributed cache restore fail before the target node executes.
• Remote session bridge refs need bridge-keyed live cache entries, not only (node_id, output_index) entries. A mapped CLIP output can produce multiple bridge refs for the same node output slot, and target-session lookup by node/output alone makes downstream mapped sampler items reuse the first conditioning vector.
• Bridge refs should not fall back to a source session's plain (node_id, output_index) slot. Mapped same-node outputs overwrite that ordinary live slot as later lanes finish, so a downstream bridge ref must resolve through its bridge_key cache or durable bridge resolver to preserve item identity across repeated sampler stages.
• Comfy can wrap a proxy output value in an outer list before handing it to an INPUT_IS_LIST-aware local/debug node or another proxy. For mapped remote session refs this creates [[bridge_ref_a, bridge_ref_b]]; implicit mapped splitting must unwrap exactly that singleton ref-list wrapper before deciding whether nontransportable CONDITIONING should broadcast.
• Distributed remote node-cache signatures also need bridge identity, not only boundary source provenance. By the time PromptExecutor prepares its cache keys, bridge refs have already been resolved to live runtime objects, so the remote worker must carry the unresolved bridge_key into boundary cache metadata or mapped sampler items with different conditioning refs can collide and restore the first item's cached output.
• Modal app deletion is not a complete reset for Modal-Sync. Persistent state also lives in named Modal Dicts for interrupts, node output cache, durable session bridges, sync index, and snapshot profiles, plus the shared Modal Volume for uploaded assets and custom-node bundles. Expose Dict deletion separately from Volume deletion because clearing caches is a debugging reset, while deleting the Volume forces a full reupload.
• Local CPU count is not a safe default for Modal GPU fan-out. Remote calls need an explicit in-flight budget shared by ordinary component dispatch, mapped lanes, and proactive warmup; Modal execution and container startup also need separate deadlines. Queue-time hashing, packaging, and upload work must run off ComfyUI's event loop, and deterministic node failures should preserve a healthy warm worker while poisoned CUDA/runtime failures retire it.
• A protocol integer only detects explicit wire-contract changes; it cannot distinguish two deployments built from different source, dependency, or runtime settings. The client and worker need the same deterministic fingerprint over remote Python source, bundled ComfyUI source, custom-node requirements, exact package pins, and deployment-shaping options. Cache compatibility by that fingerprint and bake it into a fixed Python image so a same-protocol stale worker is replaced before it can run payloads.
• Base64 is a costly default for tensor-heavy RPC payloads. A 12 MiB tensor payload measured about 16 MiB and 66.6 ms to encode through safetensors plus base64 JSON, while a versioned raw-attachment envelope stayed near 12 MiB and encoded in 31.8 ms (14.5 ms decode) in the same local check. Keep legacy JSON reads for rolling compatibility, deduplicate asset decisions across the whole queued prompt, and bound streamed progress memory by coalescing stale progress without dropping terminal events.
• Client retries need a logical invocation identity that survives Modal handle recovery. Hash the immutable payload and serialized inputs before dispatch, persist running/completed/failed attempts in a shared Dict, replay completed bytes, and keep large results out of Dict values by storing content-addressed objects on the mounted Volume. Apply the same object-store threshold to both durable bridge outputs and their producer inputs; offloading only the output still leaves tensor-heavy replay metadata large enough to overwhelm the metadata store.
• Live cloud canaries should be explicit, billable opt-ins and should avoid model assets so they diagnose deployment mechanics rather than checkpoint availability. A dependency-light worker payload can verify fingerprint handshake, raw tensor attachments, durable result replay, and shared cancellation. For concurrency, timestamps alone are flaky under cold starts; a shared Dict barrier proves that two one-input workers were actually active together and fails directly when autoscaling or local dispatch silently serializes them.
• A Modal image build context should be an explicit ComfyUI runtime allowlist, not an expanding collection of exclusions. On the development checkout that motivated this change, the old filter admitted 5,480,145,171 bytes across 756 files because an unexpected top-level directory held .pth files; the runtime allowlist admitted 25,026,233 bytes across 497 files, a 99.54% byte reduction, while retaining ComfyUI's source packages and top-level Python modules.
• Custom-node source and package-owned model artifacts need different content-addressed lifecycles. The measured tree contained 182,825,396 bytes of ordinary package files but 3,062,086,445 bytes in two model artifacts; keeping those artifacts outside per-package ZIPs means a code-only edit no longer recompresses or reuploads them, while directory exclusions continue to discard accidentally synchronized .venv trees entirely.
• Modal Volume persistence belongs to the stream-owning request lifecycle, not a daemon compute thread that can outlive its RPC client during shutdown or preemption. Defer bridge-object writes from the worker, merge them with any oversized final result, perform one volume commit on the consumer thread, and only then publish completed invocation metadata so retries cannot observe a durable ref before its bytes are committed.
• A stale mounted-volume snapshot is not a reason to call Volume.reload() while a warm ComfyUI worker holds mmap-backed model tensors. Modal rejects that operation with there are open files preventing the operation; durable object misses should instead fetch the committed path through Volume.read_file(), validate its recorded size and SHA256, and leave both the mount and warm loader state untouched.
• The same stale-mount rule applies to content-addressed model assets. A successful sync-index lookup proves the file exists in the committed Volume, not that an already-warm worker can see it. Missing runtime assets should stream through Volume.read_file() into an ephemeral read-through cache, while the worker still verifies referenced-path visibility even when queue-time sync uploaded nothing.
• Pinning opencv-python-headless in the core Modal runtime does not guarantee that cv2 stays headless. Custom-node requirements can install opencv-python directly, and dependencies such as RMBG's transparent-background can pull it transitively. The Modal image therefore needs libgl1 and libglib2.0-0 so GUI-enabled OpenCV remains importable for custom nodes such as RMBG and Inspire Pack.
• Modal can retry a generator input after an internal delivery failure even while the original container's daemon compute thread is still running. A durable invocation guard should prevent duplicate GPU work, but a raw already running error turns that protection into a prompt failure. On stream abandonment, cancel the original compute and mark its record failed without committing pending Volume objects; overlapping delivery should briefly wait for the record to become completed or retryable before deciding whether to replay or start the next attempt.
• A persistent compiler cache Volume must be reloaded before runtime initialization, never during request-time warmup. Triton's cuda_utils native extension remains visible in /proc/<pid>/maps after its ordinary file descriptor closes, so Modal correctly rejects later reloads as busy even though its SDK cannot annotate the responsible path from /proc/<pid>/fd. Fresh speculative containers receive the latest cache during their enter hook; resident containers should keep their current snapshot and compile any missing entry instead of reloading mapped code.
• A fresh uv-created ComfyUI virtual environment may intentionally have no pip module, so startup dependency repair cannot assume sys.executable -m pip works. For remote mode, detect the missing Modal SDK before importing the rest of the extension, prefer uv pip install --python sys.executable with the exact supported pin, fall back to the interpreter's pip module when available, and retry the import in the same process after invalidating import caches.
• A fixed default Modal app name makes independent ComfyUI installations collide inside one Modal workspace and also shares every app-derived Dict namespace. Persist 64 random bits in the effective ComfyUI user directory, publish the fully written identity atomically so concurrent first starts agree, encode it with unpadded URL-safe Base64 for the app suffix, preserve explicit name overrides, and inject the resolved app name into the remote image so workers never generate a second identity.
• Current ComfyUI imports comfy_aimdo unconditionally from its core memory-management modules, so omitting it from the Modal image prevents the worker from reaching node initialization. Keep both comfy-aimdo and comfy-kitchen aligned with the exact pins in the supported ComfyUI checkout; adding only the newly missing package can leave a stale kitchen ABI or API behind.
• ComfyUI changed args.cache_ram from a scalar opt-in threshold to a list of zero, one, or two active/inactive thresholds and made RAM-pressure caching the default. Remote PromptExecutor construction must distinguish the legacy scalar and current list shapes, reproduce ComfyUI's system-RAM-derived defaults, and pass ram_inactive; comparing the list directly with zero fails only after a healthy worker reaches graph execution.
• Modal SDK 1.4 exposes deployed-app stopping through modal.experimental.stop_app, not App.stop. Falling back to modal app stop without --yes blocks on an invisible confirmation prompt when ComfyUI has captured the subprocess streams, making the local 120-second subprocess guard look like an app-build timeout. Prefer the SDK API with the active Modal environment, pass --yes and --env to the CLI fallback, and translate a real CLI timeout into an intentional stop failure.
• Current ComfyUI cache get and set methods are coroutines even though older releases exposed synchronous methods. Treating a returned cache coroutine as a cache entry creates false local hits, persists empty output records, and eventually fails on .outputs. Await cache operations in async restore/persist phases, use current get_local only at synchronous progress and boundary-collection points, and accept an awaitable PromptExecutor.execute result for forward compatibility.
• Current ComfyUI serializes reusable subgraphs as root instances whose type references a separate entry in workflow.definitions.subgraphs; executable prompt ids combine the instance and definition-node ids, such as 105:14. Recursing through raw workflow JSON sees the marked definition node only as bare 14, so marker resolution must expand each definition through every matching instance path, including nested reusable definitions, before comparing workflow paths with queued prompt ids.
• Current ComfyUI AUDIO values are tensor-backed mappings and already fit the generic transport, while VIDEO is a VideoInput runtime object that does not. Transport VIDEO explicitly as versioned components containing frames, optional audio and alpha, metadata, an exact fractional frame rate, and bit depth, then reconstruct VideoFromComponents on the receiving side so nodes such as remote CreateVideo can feed a local SaveVideo.
• ComfyUI's NOISE socket is a strategy object rather than a realized tensor: built-in RandomNoise returns an object that retains a seed and only generates a tensor after receiving the sampler's latent shape and batch indices. Classifying NOISE as generically transportable can split two remote nodes across an artificial executor boundary and then fail while serializing the strategy object. Keep NOISE non-transportable so connected remote producers and consumers remain in one component unless a future versioned noise codec explicitly supports the concrete implementation.
• Compatibility wrappers around ComfyUI callables must transparently forward positional and keyword arguments. ComfyUI added disable_dynamic to load_diffusion_model_state_dict, and an older fixed-signature wrapper caused every GPU-snapshot container to fail its UNET loader prewarm and be restarted by Modal before it could serve the request.
• Modal GPU selection and the image's PyTorch CUDA wheel selection are one compatibility decision. B300 requires CUDA 13.1 or newer, so B300 and upgradeable B200+ specifications must select the CUDA 13.2 wheel index instead of inheriting the CUDA 12.8 default; count suffixes such as B300:2 should not affect that choice. Record the resolved build in both logs and the runtime fingerprint so an incompatible image cannot masquerade as the expected deployment.
• A runtime-fingerprint method on a GPU-backed Modal class is not a metadata-only probe: invoking it can provision that class's configured GPU. Resolve a GPU-specific persistent app name before lookup so a B300 workflow never contacts an A100 class, while retaining the stable instance-derived Volume and Dict names for shared assets and caches.
• Global execution UI must retain the selected GPU as prompt-scoped state rather than reread the live workflow selection on every redraw. Users can change the workflow GPU while an older prompt is still running, and replayed backend status events need to carry the authoritative target so the pill cannot relabel active work.
• The workflow/backend GPU default and the legacy base-app GPU identity are separate compatibility concerns. Changing the default to RTX-PRO-6000 must not reassign an existing base app that may already contain an A100-backed class; retain A100 at the base app name and give the RTX default its own -gpu-rtx-pro-6000 deployment.
• A ComfyUI LATENT is not guaranteed to contain one plain tensor under samples. Multimodal nodes such as MiniMax H3 use comfy.nested_tensor.NestedTensor to retain ordered video and audio tensors, so boundary transport must encode that wrapper explicitly and reconstruct it after transferring each tensor member.
• Remote status maps use executable prompt paths such as 105:14, while an inner node opened in the subgraph canvas still has the local id 14. Node decoration must resolve workflowNodePath(node) before reading phase, progress, cache, component, or badge state; otherwise only the outer subgraph instance can reflect descendant execution.
• Files written by remote ComfyUI output nodes live on the Modal worker's ephemeral filesystem and must be captured before the invocation returns. Snapshot the effective folder_paths output tree immediately before execution, bundle new or replaced regular files into the durable invocation result, and materialize them locally with validated relative paths, content digests, atomic no-overwrite publication, and stable retry naming. Returning the encoded artifact is substantially smaller than returning its decoded IMAGE or VIDEO tensor.
• An IMAGE tensor's leading dimension is not always an independently mappable batch. CreateVideo.images interprets it as one ordered frame sequence, so implicit boundary fan-out must preserve that complete tensor; splitting a 124-frame input runs CreateVideo 124 times and returns a list that local SaveVideo cannot consume.
• Modal container task ids can be reused for many sequential payloads. Immediately stopping the local modal container logs -f watcher whenever one payload releases it makes the next watcher replay that container's complete history. Retain the watcher through a bounded idle grace period so nearby task reuse shares one continuous log cursor, then stop it if the task remains idle.
• A transport-safe type is not automatically a reason to materialize a component boundary locally. If every consumer of an IMAGE, AUDIO, LATENT, MASK, SIGMAS, VIDEO, or scalar output is another remote component, the planner should mark that output as a durable session bridge and let only its small reference pass through local ComfyUI. Any local consumer must keep the ordinary materialized boundary because local nodes do not understand bridge refs.
• Cross-container remote bridges need direct serialization for every supported transport type, not only LATENT and CONDITIONING. Otherwise the reference avoids local transfer but a fresh downstream worker can still replay the producer subgraph. Keep small metadata and payloads in Modal Dict, offload values above the configured inline threshold to content-addressed Volume objects, commit before publishing completion, and use the warm bridge cache or Volume.read_file() for restore.
• A CUDA-compatible Torch layer can still be invalidated by a later companion-package layer. The B300 image installed CUDA 13.2 PyTorch and TorchVision, then installed the ordinary PyPI TorchAudio wheel compiled for CUDA 13.0; every GPU-snapshot worker died at import and Modal kept replacing it. Keep image dependency layers explicit and fingerprinted, install the official CPU-only TorchAudio wheel with --no-deps when no matching CUDA 13.2 wheel exists, and run a build-time import plus torch.version.cuda assertion so dependency mismatches fail deployment rather than creating a remote crash loop. Copy COMFY_MODAL_GPU into the image environment too, or container-side module imports misleadingly fall back to A100 even though the image and class were built for B300.
• A mapped LATENT batch can contain different spatial shapes, so torch.cat(..., dim=0) cannot always produce one ComfyUI value. The proxy must advertise mapped boundaries as list outputs: wrap a compatible concatenated batch as one scheduler item, but leave heterogeneous latent mappings as several items so ComfyUI maps ordinary downstream nodes such as VAEDecode over them. Dynamic proxy cache identities must include output list metadata or a mapped proxy can silently reuse a scalar-output class.
• Scheduler-list metadata must propagate at queue-rewrite time through every later remote component whose outputs depend on that input. Runtime implicit fan-out happens after the dynamic proxy class is registered, so annotating mapped outputs only inside the remote dispatcher is too late: ComfyUI will otherwise pass the returned Python list to a scalar local consumer such as VAEDecode as one invalid LATENT value.
• A committed Modal Volume object can be newer than the mounted view in another warm container, and the stale view may be a truncated regular file rather than a clean ENOENT. Durable reads must apply size and digest validation to the mounted bytes and fall back to Volume.read_file() on either absence or integrity failure. Durable writes should also handle short writes explicitly and validate the staged file before atomically publishing it.
• Remote-session cleanup is lifecycle bookkeeping, not an artifact-capture policy. Final components commonly carry clear_remote_session=True, so using that flag to suppress the pre-execution output snapshot silently drops exactly the files produced by terminal save nodes.
• Import-time custom-node compatibility often extends beyond routes and websocket sends. VideoHelperSuite captures PromptServer.instance.prompt_queue during import and expects queue collections, flags, a monotonic server number, and basic queue inspection methods; the headless shim needs that inert interface even when no server queue worker exists remotely.
• Hydrating boundary values directly into a remote prompt makes the graph executable but also changes what ComfyUI exposes through the hidden PROMPT input. Output nodes commonly JSON-serialize that graph into file metadata, so an injected IMAGE or AUDIO tensor fails before the first frame is written. Preserve a separate JSON-safe prompt before hydration and return it only from DynamicPrompt.get_original_prompt(); use thread-local capture so concurrent remote components keep their own metadata graphs while the executor continues using the hydrated graph internally.
• ComfyUI Nodes 2.0 does not call a node's LiteGraph onDrawForeground hook because it renders nodes as Vue-backed .lg-node DOM elements above the graph canvas. Transient extension highlighting therefore needs a renderer-neutral visual-state/palette function plus separate canvas and DOM adapters. Observe late Vue mounts and route every redraw request through both adapters so workflow loads, renderer switches, pulses, terminal cleanup, and toggle changes stay synchronized.
• Subgraph execution highlighting must be derived separately from properties.is_modal_remote: setting a container's placement flag would change semantics, while gating decoration on that flag hides already-aggregated descendant state. Index each marked leaf under every composed-path prefix for recursive idle highlighting, aggregate prompt phases for those same prefixes, and let both renderers accept derived container state. For conflicting descendants, retain phase counts for the tooltip and use a deterministic operational priority for the single border colour.
• Modal's active-container list is workspace-wide and reports lifecycle timing but not prompt ownership. For prompt-time UI monitoring, query TaskList only while the global Modal workflow state is active, filter results against every GPU-specific app name derived from this ComfyUI instance, and present pending versus started containers without implying that unrelated Modal apps belong to the workflow. Poll quickly during assignment changes, slow down once the signature stabilizes, back off on API failures, and stop as soon as the prompt UI becomes terminal.
• V3 dynamic-input nodes expose a group socket in raw INPUT_TYPES() but queue expanded prompt paths such as images.image0 and images.image1. Remote prevalidation must call ComfyUI's get_finalized_class_inputs(raw_schema, live_inputs) before required-input, primitive-coercion, or shape checks; otherwise valid io.Autogrow nodes fail on a group key like images that is intentionally absent from executable prompt JSON.
• Modal's private async _Client.from_env() is a process-global singleton guarded by an asyncio lock, so calling it from independently created ComfyUI request or log-watcher loops can fail with a cross-loop lock error. Low-level SDK operations should be wrapped with Modal's synchronizer and invoked from a worker thread, keeping both client creation and RPC awaits on the SDK's managed loop; the shared client must not be closed after each status poll or log stream.
• Modal does not expose authoritative per-prompt billing in real time, but the active-container poll can support a clearly labeled GPU-only estimate by integrating each running container's published per-second rate over the active prompt window. Filter cost attribution to the prompt's selected GPU because the task list is instance-wide, retain disappeared-container intervals in the accumulated total, reset on prompt changes, and disclose the pricing snapshot date plus excluded CPU, memory, storage, credits, reservations, and post-prompt warm time.
• Modal's authoritative granular billing report bottoms out at hourly resolution, excludes the partial current hour, and may arrive after a collection delay. Query one completed UTC hour behind a short buffer, cache it by environment, GPU-specific app name, and interval, and label the result as app-wide metered cost before credits and reservations so it is not confused with the prompt-scoped live GPU estimate.
• A Modal app redeployment can produce multiple billing rows with the same app name and environment but different historical app IDs in one hour. Billing ambiguity must therefore be detected from distinct environment names alone; retain and sum every matching app ID within the selected environment.
• Viewer-stable canvas text must not reserve graph-space width beside a zoom-scaled progress meter. At low zoom, a fixed-screen-size iterations-per-second column can consume almost the entire bar; keep the fill full width, draw it first, then paint the rate as a tightly padded opaque overlay so progress can flow underneath without sacrificing legibility.
• ComfyUI core dependency pins can become startup-critical before the remote worker reaches a payload. ComfyUI's INT8 attention import now calls comfy_kitchen.int8_attention_is_available(), so pairing current ComfyUI with comfy-kitchen==0.2.26 makes every GPU-snapshot container fail during @modal.enter(snap=True) and crash-loop. Keep both exact support packages aligned with the copied ComfyUI checkout (comfy-kitchen==0.2.31 and comfy-aimdo==0.4.13 for the current revision), and let the changed runtime fingerprint force an image rebuild.
• Current ComfyUI built-in replacements register through PromptServer.instance.node_replace_manager during nodes.init_extra_nodes(...). A remote headless PromptServer shim must instantiate the real server.NodeReplaceManager; supplying only routes, queue state, and websocket no-ops makes nodes_replacements.py fail to import and silently drops compatibility mappings for renamed nodes.
• ComfyUI's Settings -> Secrets API belongs to Comfy Cloud and does not provide a local custom-node backend with a retrievable secret store. A local endpoint node should therefore treat injected environment values as authoritative and put one-time generated credentials in the operating-system vault, failing closed when no secure keyring backend exists instead of writing plaintext into workflow or settings data.
• Modal hosted-model endpoints use OpenAI Chat Completions at /v1/chat/completions, but they require a proxy-token pair (wk- and ws-) rather than the ordinary Modal API token pair (ak- and as-). Use the separate Modal-Key and Modal-Secret headers, restrict credential-bearing URLs to HTTPS modal.direct origins, and disable redirects so a pasted endpoint cannot exfiltrate workspace credentials.
• The modal workspace proxy-tokens create --json command is available in current Modal clients but not in this project's pinned remote-runtime SDK. Credential bootstrap can preserve the remote runtime pin by falling back to uvx --from 'modal>=1.5.4' modal ... only for the local CLI operation, parsing the documented JSON keys and never logging the one-time secret.
• In an RBAC-enabled Modal workspace, workspace proxy-tokens create produces a valid but scoped token that a Direct endpoint reports as Webhook token not found until it is explicitly allowed into the endpoint's environment. Persist the one-time secret first, then run the idempotent workspace proxy-tokens allow operation for node-managed vault credentials; cache successful authorization in-process to avoid adding CLI latency to every inference call, and leave environment-injected administrator credentials unchanged.
• Modal Server deliberately returns HTTP 503 with an empty body while a scale-to-zero endpoint has no ready replica; the request triggers scaling but is not queued. Endpoint clients must inspect status before decoding JSON, retry only the explicit 503 cold-start response, and share one deadline across model discovery and inference so cold-start polling cannot silently exceed the node's total timeout.
• A newly added optional ComfyUI string widget can reach execution as an empty string in an existing workflow even when its schema declares a non-empty default. Normalize the Modal environment at the resolver boundary and treat blank as main, so credential authorization remains compatible with workflows saved before the advanced field existed.
• Opt-in live Modal tests must preserve the operator-supplied app, GPU, ComfyUI root, and container limits. A blanket autouse fixture that resets those variables can silently deploy an expensive canary to the default GPU; exempt live-marked tests, use a default-only app assignment at import time, and name each live deployment explicitly.
• For a freshly auto-deployed LLM app, stage pinned Hugging Face snapshots through the CPU-only class immediately after deployment and before the first RemoteEngine lookup. A runtime-version method belongs to the GPU class and can allocate the B300, defeating CPU-first staging if it is called first.
• Transformers can load a SmolVLM2 snapshot successfully and still fail while constructing its processor because num2words is a model-specific optional dependency rather than a core Transformers dependency. Pin it in the Modal runtime image and cover model construction with a live canary, not just registry and staging unit tests.
• Reasoning-enabled Qwen models can consume a small max_new_tokens allowance entirely before emitting </think>, correctly leaving the separate response output empty. Live inference canaries need enough combined reasoning-and-response budget, and vLLM canaries should assert positive streamed TTFT and token-rate telemetry rather than treating any generated token as proof that the async progress contract worked.
• The Modal marker already provides the correct local/remote placement contract for an LLM node. Once the node's explicit local rejection is removed, an unmarked LLM remains a normal local scheduler node even between two remote regions; this preserves the existing transport and warning behavior without a second backend selector widget.
• Apple-local generated profiles must include their execution target in the content digest so CUDA and MLX compatibility decisions can never share an identity. Preserve the historical Modal digest shape by omitting the default modal target, or existing immutable profile ids and saved payloads change unnecessarily.
• Sampled ComfyUI video is already reduced to timestamped PIL frames by the shared input layer. Passing that ordered frame list to MLX-VLM as multiple image inputs avoids backend-specific video-container handling while preserving the bounded visual evidence described in the prompt.
• MLX-VLM exposes both Qwen's hard enable_thinking switch and a thinking-budget criterion that can force its closing reasoning token. Pass the shared per-request reasoning toggle to MLX, select the matching request parser, and cap enabled Apple-local thinking at half of max_new_tokens; otherwise small models can consume the entire allowance without a final response or a direct answer can be misclassified as unfinished reasoning.
• Generated LLM profile identity contains backend and runtime policy, so using the profile id as the weight directory duplicates an unchanged Hugging Face revision whenever execution tuning changes. Key new weight storage by repository plus exact revision and validate legacy reuse through the completion marker. Do not rename an old directory out from under already-deployed code that still resolves the historical path; reuse it in place until those deployments have rolled forward.
• A persistent weight Volume does not preserve vLLM's ephemeral compilation products. Mount a separate compilation-cache Volume and point vLLM, TorchInductor, Triton, and CUDA caches into a GPU-and-runtime namespace; keeping that mount separate also prevents ordinary asset reloads from colliding with open JIT cache files.
• Cold/warm LLM benchmarking needs more than two consecutive calls. Repeat a cold-container cycle with the same persistent caches to distinguish first compilation from cached compilation, then run an immediate resident repeat inside each cycle; record exact input, source revision, TTFT, load time, generation time, token count, and token rate so variable output length cannot masquerade as an engine speedup.
• A benchmark app's configured base name is not necessarily its deployed name: GPU selection adds a suffix. Stop the exact deployed app and poll until Modal reports both stopped and zero tasks, or a supposed cold cycle can silently reuse the resident engine. On RTX PRO 6000, persistent AOT caching cut Qwen3.8 NVFP4 compilation from 138.0 to 22.4 seconds in the best observed cached container, while CUDA graphs improved warm decode from 17.2 to about 38.7 tokens per second at the cost of longer cold initialization.
• mlx-dspark's Qwen3.5/Qwen3.8 loading path is deliberately text-only and strips the vision tower, even when the source checkpoint is multimodal. Local auto-selection must therefore stay request-aware: use registered DSpark/DFlash acceleration only without image or video inputs, preserve MLX-VLM for media, and reject forced mlx-dspark media requests. Pin the separately resolved drafter commit just like the target instead of allowing a floating Hub download to bypass the immutable-profile contract, and stage only its config plus safetensors because drafter repositories can also contain multi-gigabyte GGUF alternatives that the MLX adapter does not use.
• vLLM cannot turn an already constructed eager engine into a CUDA-graph engine in place. Container-local auto mode must observe ComfyUI prompt ids at the RemoteEngine boundary, keep repeated components and retries from one workflow in eager mode, and on the second distinct workflow retire the eager resident exactly once before rebuilding for throughput. Counting only LLM calls would incorrectly promote workflows with multiple LLM nodes and would miss an image-only first workflow that already proved the container is active.
• A local node between two remote sampler regions can force an ordered phase split, but losing the first worker must not replay the first sampler merely to reconstruct RandomNoise or KSamplerSelect strategy objects. Build literal NOISE and SAMPLER producers directly on the downstream worker, and evaluate the complete execute-node dependency closure before permitting any fallback producer replay; checking only a terminal node such as ImageFromBatch misses its expensive sampler ancestors.
• A streamed remote method timer can conceal several distinct post-compute waits: publishing the worker result into the bounded buffer, committing durable objects, writing completed invocation metadata, suspending at the Modal generator yield, receiving the result locally, and closing the stream. Log each boundary with the same component, prompt, invocation, task, byte count, queue depth, and elapsed time before changing storage or transport policy; a single gap between output bundling and method completion cannot identify which subsystem stalled.
• Speculative worker preparation should follow the dependency DAG, not merely the textual order of proxy nodes. Attaching only the nearest reachable component with a different worker-affinity group keeps lookahead bounded, and triggering it on the current stream's first event prevents the speculative request from racing ahead of the critical container. Warmup dedupe must include the affinity key; prompt-wide slot numbers alone collapse distinct LLM and Comfy pools into the same slot.
• V3 nodes do not receive hidden extra_pnginfo and unique_id values through execution kwargs; ComfyUI stores them on the execution-specific cloned class as cls.hidden. Dynamic proxies must read that holder first and use list-normalized kwargs only as a legacy/direct-call fallback. A regression must execute a class produced by PREPARE_CLASS_CLONE, because calling the V3 node directly with synthetic hidden keyword arguments models the wrong runtime convention and can miss cross-prompt dispatch-group deadlocks.
• Generic remote execution needs a provider-neutral environment identity before transport code is added. Persist only an SSH destination and scheduling metadata, keep credentials in the ordinary SSH agent/configuration boundary, distinguish unknown cost from explicit zero cost, and reject unprobed or incompatible hosts before comparing estimated price. Arbitrary-host hardware is discoverable, but its amortized or leased per-second cost is not and must remain an explicit user input.
• Reusing the Modal execution kernel on SSH hosts is safest when the transport stays deliberately small: a fingerprinted OCI image, a persistent Unix-socket worker, and a length-prefixed binary relay through ssh docker exec. The shared payload/event/result formats then preserve progress, cancellation, sessions, custom-node loading, tensor serialization, and output capture without opening a remote service port.
• ComfyUI frontend settings do not define a string-valued button control type; unknown strings fall back to a text input and ignore an onClick property. Settings-page actions must use the supported function-valued custom renderer and return the interactive element.
• Remote output parity is a controller responsibility. The shared stream consumer returns the worker's raw result envelope; a non-Modal executor must still call the common artifact materializer before deserializing node outputs, or save/preview nodes appear to finish remotely while generated files never arrive locally.
• A local content-addressed index for an SSH named volume needs the same epoch-sentinel rule as a Modal Volume. Host or volume recreation can otherwise leave valid-looking local records pointing to missing remote assets; mark only durable remote backends as epoch-scoped so ordinary test doubles and local mirrors retain their existing behavior.
• Hardware, Docker, and GPU capacity can be probed generically, but arbitrary-host pricing has no equivalent kernel API. Accept a reserved Docker daemon cost label from infrastructure automation, keep a UI override, and preserve unknown as distinct from explicit zero so the optimizer never invents economics.
• SSH aliases often carry interactive local or remote forwards. Machine-control sessions must set ClearAllForwardings=yes, or every probe/build/relay can collide with unrelated listening ports even though authentication and the remote command itself are valid.
• Reapplying deployment-owned pins after custom-node requirements is too late when a conflicting requirement has no wheel for the worker Python version: pip can fail while attempting a source build before the repair layer runs. Filter requirements for packages owned by the immutable Python/CUDA stack, retain all other custom-node dependencies in the fingerprint, and let import validation expose genuine incompatibility.
• A selected BuildKit docker-container driver does not automatically place a successful docker build result in the host image store. Pass --load so the fingerprinted image is available to the subsequent docker run; otherwise the build succeeds into cache and Docker tries to pull the private local tag from a registry.
• Running python -m remote.ssh_worker imports remote/__init__.py before the worker module. Eagerly importing remote.modal_app there breaks because its parent-relative imports assume the node pack has a parent package, while the OCI source root deliberately exposes remote at top level. Keep those convenience exports lazy and regression-test the exact subprocess entrypoint used by the container.
• The SSH runtime fingerprint hashed top-level ComfyUI Python modules while its Docker context omitted them, producing an image with package directories but no nodes.py. Keep the image selection aligned with the fingerprint selection: include every top-level .py file plus the selected package trees, and exclude unrelated top-level assets explicitly.
• Cost optimization is safe only after component resource admission. Ordinary loader nodes often have no explicit VRAM declaration, so resolve their local model files before placement, budget the largest primary weight set plus additive LoRA/adapter/ControlNet weights, apply runtime headroom, and compare against probed free VRAM and available RAM. Otherwise an explicit zero-cost 24 GiB host can beat a large Modal GPU for a 66 GiB BF16 model and begin a doomed upload before failing at execution.
• A raw Hugging Face model ID cannot be passed directly to an SSH worker just because the Modal dispatcher knows how to resolve it. Run the provider-neutral resolver and snapshot stager inside the selected SSH container, persist both generated manifest and weights in its named volume, stream CPU-stage progress to ComfyUI, and rewrite the payload to the immutable profile ID before invoking the GPU execution kernel.
• SSH LLM staging happens too late to influence an already chosen environment. Resolve Hugging Face metadata before scheduler admission and apply the generated profile's artifact size and estimated VRAM; otherwise a zero-cost 24 GiB host begins downloading a 52 GiB checkpoint before discovering the placement was impossible.
• Cancelling an asyncio task that is awaiting to_thread does not stop the blocking Docker staging process, and terminating the local docker exec client does not reliably kill its container child. Track active staging invocations and stop the managed worker on cancellation; its named volume preserves partial Hugging Face files for a later compatible run.
• A Hugging Face GGUF repository can contain hundreds of gigabytes of mutually exclusive quant files and no Transformers tokenizer files. Curated GGUF staging must pin the chosen weight and any matching multimodal projector by exact filename, download those artifacts rather than *.gguf, and independently pin a tokenizer source without allowing its safetensors. The deployed vLLM 0.27.1 image has no GGUF loader, so SSH workers use the official digest-pinned CUDA 13 llama.cpp server as a private loopback backend; copying Python 3.13 from its slim build stage into the newer Ubuntu runtime preserves both the existing worker and llama.cpp's required glibc. Image requests use its OpenAI-compatible multimodal chat endpoint while text-only requests retain the token-reporting completion endpoint.
• Replacing a provider image's entrypoint does not replace its Docker health check. The llama-server base probes HTTP port 8080, while the managed SSH worker initially serves only its Unix socket; explicitly emit HEALTHCHECK NONE in the derived image so an otherwise ready worker is not trapped in an unhealthy startup state.
• Copying an executable from a provider image does not make its sibling shared libraries discoverable after the derived image overrides LD_LIBRARY_PATH. Preserve /app in the SSH image library path and also prefix the configured llama-server binary directory in the spawned process environment; validate with ldd or an actual server launch because image construction alone does not reveal the missing dynamic dependency.
• Provider-aware placement must happen before remote-session bridging. Cut explicit backend boundaries during component formation, preserve an acyclic coarse graph when later expensive-edge merges would wrap around that cut, and materialize cross-environment values through ComfyUI; otherwise session co-location silently overwrites the scheduler's provider decisions. Modal speculative prewarm and local-gap keepalive metadata must carry and match both provider and environment identity, because recursively embedded SSH payloads can otherwise stage self-hosted-only models on Modal.
• Vast marketplace capacity is not equivalent to an ephemeral per-call GPU. Quote the configured idle-retention extension as part of incremental cost, give an existing owned lease credit for an already-paid deadline, and rent only after the provider-neutral scheduler selects the quote. Persist the exact profile and runtime fingerprints so changed workflow constraints or worker source cannot silently reuse incompatible capacity.
• A long-lived marketplace controller needs two independent cleanup paths. The local registry can reconcile and reap exact owned labels, while an in-container watchdog uses Vast's injected instance-scoped key and a controller-written busy/deadline snapshot. The watchdog must fail closed on missing, malformed, stale-busy, or identity-mismatched state; otherwise a transient controller failure can either leak an expensive instance indefinitely or destroy active work.
• Vast SSH launch mode replaces the image entrypoint and runs onstart only after its own SSH initialization. Put worker and watchdog supervision in onstart, export the runtime environment for later SSH sessions, use a dedicated known_hosts file with no inherited forwards, and require a digest-pinned worker fingerprint before syncing assets.
• Worker image construction must resolve a real ComfyUI source checkout even when this repository is run standalone rather than installed under custom_nodes. Support an explicit builder path and the locally used Latest_ComfyUI checkout name; otherwise a credential-ready live canary fails before marketplace work with a misleading deep image-context error.
• A node execution-location badge needs both queue-time and runtime identity. Return the planner's provider/environment assignment keyed to rewritten proxy representatives so every original member node can render immediately, then replace the Modal environment label with the exact task/container ID learned from the remote log stream. For SSH, derive the display hostname from the configured target instead of exposing the scheduler's environment key. Gate the row on the count of distinct environments actually assigned to that prompt, not the number configured, so a one-environment plan stays visually quiet.
• Nodes 2.0 node bodies can cover legacy canvas foreground drawings, so execution-location UI needs a DOM-layer equivalent rather than relying only on onDrawForeground. Attach the exact runtime identity to streamed status events as well as numeric progress; otherwise loader and other non-numeric phases know the active node but cannot display its Modal task ID.

2026-08-23: Vast instances need a direct-container backend

• Deterministic SSH worker names need lifecycle serialization per environment and worker slot. Parallel components can both observe an absent container and race through docker run; the loser should verify and adopt the correctly labeled, fingerprint-matching winner rather than failing the workflow with Docker status 125. Exact-name replacement must still pass the environment ownership-label check before removal.
• ComfyUI Nodes 2.0 converts legacy extension context menus into a Vue menu model whose submenu entries are action-only leaves. A third menu level is silently discarded, so extension-owned provider and GPU selectors must put their choices directly in the first submenu and use a disabled heading to show the current selection.
• Vast worker images must be built explicitly for linux/amd64. Without Docker's --platform linux/amd64, an Apple Silicon host selects arm64 base-image stages and rejects the intentionally x86_64 CUDA/vLLM wheels before the image can be pushed.
• Provider-aware DAG cuts are valid only for serializable or safely reproducible edges. For shared MODEL, CLIP, VAE, NOISE, and SAMPLER outputs around an SSH-only LLM phase, clone a non-sampling loader dependency closure into each downstream component and rewire it before final partitioning. Fall back to hard affinity only when the producer cannot be reproduced safely; otherwise one GGUF node can pull an unrelated 61.73 GiB Comfy model onto a 52.64 GiB-RAM SSH host.
• A raw Hugging Face repository ID is valid only for the Transformers-compatible model resolver, which requires config.json and safetensors. GGUF repositories must be selected through a curated profile that pins the exact weight, optional projector, tokenizer source, and llama.cpp backend; otherwise prompt validation fails before provider placement.
• Vast instances are already containers and do not support Docker-in-Docker, so they cannot be modeled as ordinary SshDockerController hosts. Reuse the provider-neutral planner, framing protocol, worker kernel, serialization, progress, cancellation, and artifact paths, but add direct SSH runtime and filesystem storage adapters.
• Treat the disconnected Vast configuration node as a local v3 output sink. This preserves it in the queued prompt without wiring it into user data flow, and it must remain in the local-only class set during automatic placement.
• The durable execution-plan payload already carries safe metadata for both capacity and storage configurations. Configurator UI should split those concepts at render time: capacity configurations belong in assignment and readiness rows, while storage configurations belong in a distinct summary beneath the plan. That preserves the provider-neutral payload, avoids portraying storage as schedulable capacity, and lets provider-specific cards expose useful policy such as an R2 bucket, jurisdiction, write-back mode, and key prefix without surfacing credential references.
• R2's account-level REST metrics do not provide a per-bucket total, while its bucket analytics use a separate Cloudflare API authorization surface. A controller that retains only bucket-scoped S3 credentials can still compute exact current bucket payload bytes and object count through paginated ListObjectsV2; cache that aggregate briefly so the Configurator can show useful usage without enumerating the bucket on every workflow submission.
• ComfyUI's v3 execution path shallow-clones a node class and then rebuilds it with a metaclass that rejects class mutation. Adding Python's ABC to an intermediate v3 node base changes that clone to ABCMeta, whose construction writes __abstractmethods__ and collides with ComfyUI's lock. Extension-owned v3 node bases must stay concrete (using an explicit NotImplementedError for provider hooks) unless ComfyUI's class-lock implementation becomes ABC-aware.
• A storage refresh control must update both the visible card and the Configurator's retained plan state; otherwise a later status event or remount immediately restores stale usage. Resolve the opaque credential reference from the live storage node, send it only to the same-origin controller, bypass and replace the server usage cache, then persist the safe aggregate response in the prompt-local frontend state.
• Revalidate every offer response locally even when the marketplace query contains identical hard filters. Rank compatible offers deterministically by total hourly price, then reliability, DLPerf, download rate, and offer ID.
• Keep idle retention separate from minimum offer duration. Retention is a billed lease-lifecycle decision; offer duration only describes how long the marketplace host claims it will remain available.
• The offline Vast simulator should exercise offer disappearance between search and rent, loading-to-running polling, state changes, and destruction while never recording authorization or per-instance API keys.
• Vast marketplace lookup belongs before ordered scheduling: derive and deduplicate the effective memory-adjusted profiles, prefetch their searches with bounded concurrency, then retain sequential provider selection and acquisition so a newly rented lease can reduce later components' incremental cost. A process-wide TTL cache must include successful empty searches, isolate credentials and API base URLs in its keys, stay bounded, and expose a force-refresh path for offer-disappearance races.
• A provider-local session bridge cannot replace an entire transportable OUTPUT_IS_LIST value with one mapping token. ComfyUI interprets the proxy output as scheduler items, so the mapping can be extended into its keys or later resolve back to the whole list where a scalar item is required. Materialize transportable list outputs through ComfyUI; retain session bridging for scalar/nontransportable boundaries until list bridges carry one independently replayable reference per item.
• Vast marketplace selection works best when workflow-authored preferences are distinguishable from runtime requirements. Represent unconstrained selectors as None, omit them from the API payload, and then raise only VRAM/RAM to the planner's inferred component floors. Literal zero filters are not the same contract as an explicit Any, and a mandatory default such as 100 TFLOPS can eliminate otherwise suitable H200 capacity because marketplace compute scores are not comparable across GPU architectures.
• Vast reports memory as an integer offer field whose displayed capacity can differ from marketed GPU names. Keep the workflow field explicitly GiB-based and do not silently treat a marketed “96 GB” label as proof that the offer meets a literal 96 GiB threshold.
• A disconnected queue-time configuration node can still expose placement results without becoming part of the remote graph. Request ComfyUI's hidden rewritten prompt, stamp only credential-free lease metadata into proxy payloads, filter by the configuration node's hidden unique ID, and fingerprint the generated Markdown so cached local output changes with the actual selected GPU type.
• Provider selection must compare equivalent compute rates. Price a Vast candidate from its marketplace hourly rate over predicted execution time, and retain the full execution-plus-idle-retention amount only as a separate lifecycle billing estimate; folding 24 hours of optional retention into placement cost makes otherwise-cheaper Vast hardware lose every automatic comparison.

2026-08-23: Workflow capacity needs one compiled configuration graph

• ComfyUI plans and rewrites a prompt before ordinary node execution, so a runtime REMOTE_CONFIGURATION Python value cannot itself configure queue-time placement. Use the same pure builders from both provider-node execution and a queue compiler that follows the single Remote Execution Configurator input branch.
• Keep configuration identity, capacity-slot identity, and concrete environment identity separate. A Vast profile can yield several leases, a Modal GPU pool can yield several containers, and one SSH host can expose several worker slots without changing the workflow configuration that contributed them.
• Capacity counts are concurrent ceilings rather than total-component limits. Plan the component DAG in stages, retain per-slot availability, reuse slots across sequential stages, and let waiting compete with another environment's incremental cost.
• Finish the complete side-effect-free assignment plan before acquiring Vast capacity. Search and quote every connected profile first, then rent only the selected (configuration, slot) pairs so a later validation failure cannot leave an unnecessary lease behind.
• Workflow-declared SSH hosts should carry a validated credential-free snapshot into the queued proxy payload. Requiring the executor to look the host up again in the installation-wide registry would make the graph non-authoritative and would silently reintroduce the original SSH asymmetry.
• Queue-time v3 Autogrow inputs retain their full dynamic path, such as configurations.configuration_0; only the execution engine reconstructs that into configurations={"configuration_0": ...}. A queue compiler must accept the flattened serialized path (and can tolerate the reconstructed form), and its regression fixture must use the actual queue representation.
• Provider-neutral planning also needs provider-neutral setup telemetry. Do not attach the legacy workflow Modal GPU or poll Modal containers and billing before a configurator-backed plan selects Modal; Vast-only and SSH-only workflows must retain a null Modal GPU through synthetic setup and provider-stamped runtime events.

2026-08-24: Pre-capacity plans need a durable UI path

• A queue response cannot be the first carrier of a workflow plan when the same request performs capacity acquisition. If Vast rental or SSH initialization stalls, that response never arrives; publish the side-effect-free scheduler result before acquisition and retain it in prompt-scoped browser state.
• Durable backend events do not restore a freshly loaded page unless the frontend requests them after the graph and its DOM panels mount. Refocus-only replay misses an already-visible initial load, leaving the Configurator blank until a new prompt emits a live event; trigger the same replay-and-reconcile pass after graph configuration.
• A single Configurator status field cannot represent parallel remote environments: interleaved Vast, Modal, and SSH events continually replace one another. Stamp setup telemetry with the same concrete environment identity used by runtime progress, retain one lane per planned environment, and prevent delayed events from an older prompt from reclaiming a panel already owned by a newer prompt.
• ComfyUI DOM widgets may mount after their prompt events arrive or remain temporarily detached during canvas virtualization. Store plan and status independently of the element, replay the newest matching prompt after mount, and set a mounted guard only after addDOMWidget succeeds so a transient mount failure remains retryable.
• Python-only source filters are unsafe for copied runtime packages. ComfyUI keeps tokenizer vocabularies, model configurations, SentencePiece models, and other import-adjacent data beneath its Python package directories; copy every file in the curated runtime directories and hash the same complete set so a data-only update rebuilds the worker instead of failing later when a local path is mistaken for a Hugging Face repository ID.
• A persisted Vast lease registry describes controller intent, not live SSH health. Worker readiness loops must log changed probe diagnostics as they occur and include instance/environment identity at the service boundary; otherwise a provider closing the connection during key exchange appears as a silent 15-minute initialization wait.
• Provider running plus an assigned SSH endpoint is not sufficient evidence that rented capacity is reusable. If bounded runtime initialization fails, mark the lease draining before cleanup, destroy the owned instance, and preserve the original failure even if destruction itself fails; otherwise every later plan can select the same poisoned registry entry.
• A module-local panel map cannot prove that a ComfyUI node has no existing DOM widget after extension reload. Before retrying a mount whose node flag and current module state disagree, remove the named widget through node.removeWidget (and detach its element) so the frontend store is unregistered and the node never accumulates duplicate panels.
• A websocket setup-error event can arrive before the failed queue request returns. If that event ends synthetic execution without forwarding its error_message, ComfyUI records a generic queue failure before the HTTP catch can preserve the backend detail. Use the same backend message for the synthetic error, global status, and configurator terminal state.
• Vast lifecycle fields are not individually authoritative: fall back from an absent actual_status to cur_state and intended_status. When all lifecycle fields, the provider message, and the SSH host remain absent continuously, use a shorter bounded grace period instead of presenting unknown for the entire readiness timeout; observable loading and download states must retain the normal window.
• Vast may close an individual SSH connection even after the same instance has passed worker readiness and immediately accepts a subsequent probe. Treat explicit connection closures, resets, and handshake failures as transient transport events with bounded jittered retries, while leaving authentication and remote-command failures immediate. A retried streamed upload must reopen its source and verify the received byte count before atomic publication; otherwise a severed stream can promote a truncated temporary file into a trusted content-addressed path.
• ComfyUI's /interrupt flag only reaches work already executing through its prompt worker. Remote planning and asset synchronization happen inside the queue HTTP request's to_thread, so cancellation needs its own prompt-scoped event, chunk-level hashing checks, and subprocess termination for active SSH transfers; changing the status pill alone does not stop network or disk work.
• A controller must not inject its expected runtime fingerprint into a rented worker's environment, because runtime-info then reports the expectation rather than the image's actual embedded identity. Preserve the fingerprint baked by the image builder and compare it to the controller expectation before trusting the worker.
• Queue-time tooling can evolve before a costly worker image is rebuilt. Ship small controller-owned helpers, such as the verified Hugging Face materializer, as deterministic content-addressed zipapps over SSH; this preserves current validation behavior without turning a missing Python module into a 50+ GB fallback upload.
• A full repository-source fingerprint changes for controller-only edits even when the configured Vast worker image and its wire protocol remain usable. Persist the image reference on new lease records, allow legacy records one compatibility adoption, and reuse only the same configured image with the exact worker protocol; otherwise a cancellation or UI fix can rent another expensive GPU without improving remote execution.
• Vast's v0 show-instance endpoint can report a deleted contract as HTTP 200 with instances: null rather than HTTP 404. Normalize both shapes to the same not-found exception so registry reconciliation removes stale leases and continues to the next compatible live rental instead of treating the nullable field as a fatal schema error.
• A workflow-authored SSH snapshot must retain the GPU capabilities established during planning. Stripping them before queueing makes Docker launch without a device request even though the host has a healthy GPU. Re-probe legacy capability-less payloads, pass the selected GPU UUID explicitly, and fail closed when GPU or NVIDIA-runtime capability cannot be established rather than silently starting a CPU worker.
• Durable Configurator replay can restore the last setup event after ComfyUI has already failed or interrupted the prompt. History reconciliation must inspect the prompt's terminal status and messages, render that terminal outcome before clearing prompt-scoped memory, and explicitly replace stale submission text when neither queue nor history reports active execution.
• Deployment-owned package pins must advance with the ComfyUI source copied into a remote worker. ComfyUI now imports ColorPrimaries from PyAV and requires av>=17, while the worker's old av==16.1.0 pin masked that requirement and failed during headless node registration. Pin the verified current PyAV release and execute the exact ComfyUI import during image construction so API drift is caught before a workflow starts.

2026-08-24: Shared object storage should be a backing cache, not a mounted filesystem

• A provider-neutral cache must preserve each execution environment's native storage as the hot layer. Resolve the local sync index first, adopt a content-addressed file already present on the remote volume, use authoritative origin downloads such as Hugging Face next, consult shared object storage only after those sources, and retain the existing controller upload as a reliable fallback.
• Do not distribute permanent object-store credentials to rented or arbitrary workers. Keep the S3-compatible client on the controller, issue expiring exact-object GET/PUT or multipart-part URLs over protected stdin, validate their HTTPS origin remotely, and verify byte size plus SHA-256 before atomically publishing downloaded content.
• Content-addressing makes one R2 bucket safely reusable across Vast instances, SSH hosts, and installations without a mutable filename catalog. Write-back can run outside queue-time critical latency, while a synchronous mode remains useful when short-lived capacity may disappear before background population completes.
• Capacity and storage belong in the same workflow-level configurator but have different planner semantics. Model them as sibling remote-execution configuration types, filter scheduler inputs to capacity configurations, and keep storage configurations available to the transport layer; otherwise a storage node can accidentally count as a fallback provider or contribute nonexistent concurrency.
• Cloudflare OAuth is a bootstrap credential, not an R2 S3 credential. Cloudflare's live OAuth scope catalog exposes account-settings.read and workers-r2.write, but no user- or account-API-token creation scope; access-service-token.write controls unrelated Zero Trust credentials. Request only the two API scopes for R2 provisioning: Cloudflare can reject an explicit openid request even when the dashboard presents it as an identity option, and no identity token is needed here. Use Authorization Code + S256 PKCE to select the account and provision the bucket, then require a one-time user-created Object Read & Write R2 token restricted to that bucket. Validate its S3 access before storing it in the OS credential vault behind an opaque workflow reference, and continue giving workers only exact-object presigned URLs.
• R2 presigned URLs retain the authorization constraints of the API token that signed them. A client-IP-filtered token can therefore pass controller-side validation and self-hosted transfers while every Vast worker receives 403 AccessDenied from its different egress address. Keep direct-worker tokens bucket-scoped but unrestricted by source IP unless all dynamic worker addresses are covered, and reduce R2 XML failures to a bounded symbolic code before they cross the remote-helper stderr boundary. Before asset synchronization, validate the controller normally and issue a short-lived ranged GET for a random absent sentinel from each worker: the expected 404 proves effective authorization from its real egress, while GET preserves the structured denial code that an empty HEAD response may lose.
• ComfyUI's origin-only middleware rejects even top-level OAuth callback GETs when the browser labels them Sec-Fetch-Site: cross-site, returning an empty 403 before a custom-node route runs. A loopback OAuth integration must install a pre-middleware that handles only its exact callback GET ahead of that guard. Keep the exemption safe with random single-use state plus PKCE, and retain the standard cross-site rejection for every other path and method.
• Marketplace network capacity is not geographically uniform. A host can accept inbound SSH and reach ordinary HTTPS sites while DNS interception or regional filtering makes Hugging Face's CloudFront endpoints unreachable; expose explicit country exclusions in the workflow profile, send them as geolocation.notin, and revalidate returned offers locally rather than treating advertised bandwidth as proof of origin reachability.
• Vast may report an offer-disappearance race as a non-404 HTTP response containing provider code 3603 and no_such_ask. Classify the provider code as well as HTTP 404 so the lease manager invalidates the stale ask, force-refreshes marketplace results, and tries another candidate.
• Configurator terminal rendering must update retained per-environment state, not only the currently mounted DOM rows. Otherwise a later plan render, graph remount, or replay reconstructs the last healthy setup meter after the workflow has already failed; terminal transitions should also clear numeric progress so a failed lane cannot look active.
• Retrying SSH at both the command layer and an enclosing readiness loop multiplies a dead endpoint into bursts of nearly identical connections and warnings. Let each readiness interval make one transport attempt while keeping bounded command-layer retries for work against an already-ready host, especially streamed uploads that can safely reopen their source.
• Vast can publish a stale ssh_host/ssh_port proxy that closes every handshake while its console command uses a working public_ipaddr plus the ports["22/tcp"][].HostPort mapping. Prefer that console-equivalent direct mapping when it is complete and valid, with the proxy retained only as a fallback for instances that omit direct-port metadata.
• A Vast contract can disappear after returning a usable SSH endpoint. Revalidate the exact provider contract immediately before each worker-readiness connection; otherwise a deleted direct endpoint looks like a 15-minute SSH authentication failure. Remove the stale registry record, exclude its offer, and cold-start one bounded replacement so transient host loss does not fail the workflow or return it to the same vanished capacity.
• Asset synchronization is not worker readiness. An SSH environment can finish custom-node and prompt-asset preparation while its fingerprinted image still needs to build and its container still needs to start. End queue-time setup with a provider-specific intermediate state, then thread the execution-time runtime lifecycle through the same environment-scoped status stream; reserve Ready for a worker that has passed its actual readiness probe.
• Importing Triton successfully during image construction does not prove the runtime can execute Triton-backed nodes. Triton lazily compiles native launchers on first use, so slim worker images must retain a C/C++ compiler toolchain, expose deterministic CC/CXX paths, and validate the compiler binaries while building the image instead of discovering their absence inside a billable workflow.

2026-08-25: Queue admission must distinguish occupied capacity from undersized capacity

• Live free VRAM is the correct admission signal for an otherwise idle SSH host, but it is misleading when the same ComfyUI queue already has a workflow assigned to that host. Preserve each queued prompt's provider assignment in queue metadata, identify the exact SSH environments owned by earlier prompts, and admit later serialized work against those hosts' reserve-adjusted total capacity. Do not generalize this to every busy GPU: unrelated workloads still need the conservative live-free-memory check, and physically undersized hardware must remain incompatible.
• A persistent managed worker can retain model allocations after its originating ComfyUI process has exited, so a new process has no queue metadata proving where the VRAM went. Configurator-backed planning needs the same ownership-checked idle-worker reclamation as legacy planning: only when live capacity fails but reserve-adjusted nameplate capacity succeeds, inspect the extension-owned worker, remove it only if no execution or staging process is active, re-probe, and then rebuild candidate states.
• Progress telemetry should retain provider identity independently of the optional execution-location row. A single-environment plan suppresses that row, and mapped worker lanes can outlive aggregate progress, so each aggregate or lane event needs a provider fallback from its planner assignment before the canvas can reliably label every progress bar with the correct backend icon.
• A provider-local bridge is usable by downstream work on the same Vast or SSH worker, but its durable record is not automatically readable by local ComfyUI. When the same transportable output also feeds a non-returning local branch, return the real value through ComfyUI instead of inserting the Modal-backed local bridge materializer; otherwise a Vast workflow can fail after expensive inference by attempting to authenticate to Modal solely to render a local preview.
• Dockerfile instruction order does not protect a large SSH worker when the selected docker-container BuildKit instance has no retained cache. Give the heavyweight dependency stack its own dependency-only fingerprint and local image tag, then build the frequently changing source image from that explicit base with Docker's persistent default builder. This turns an ordinary runtime-source change into one roughly 52 MiB COPY overlay even when anonymous BuildKit cache usage is zero, while requirements, base-image references, and an explicit dependency-layer schema version still invalidate the 13 GiB base when necessary.
• Fingerprinted worker tags need an ownership-scoped garbage-collection policy: a busy development host accumulated 23 obsolete 10–13 GiB comfy-remote images and filled a 1.8 TiB filesystem even though Docker reported most image storage as reclaimable. Before rebuilding, remove only superseded comfy-remote and comfy-remote-deps tags carrying this extension's labels, without forcing deletion; Docker then protects any image still referenced by a running or stopped container while unrelated repositories remain untouched.
• Component-level staging progress cannot use the component representative as its UI node: representatives are selected for graph execution and may be an unrelated dependency such as VAELoader. Carry the fixed model reference through every staging event, recover the matching ModalLLM node IDs from the nested payload, and emit to those nodes. Also report the pre-download state machine explicitly; a single Inspecting label otherwise conceals snapshot checks, lease contention, and Hugging Face's pre-transfer preparation.
• A useful execution plan is target-centric rather than component-centric. Attach a credential-free hardware snapshot to every assignment while provider data is still available, group components beneath each concrete environment in the Configurator, and distinguish known HBM/VRAM and system RAM from dynamic provider-managed capacity instead of inventing precision.
• File age is not process liveness. A multi-hour model download can legitimately outlive a stale-lock threshold, so a snapshot lease must publish host, PID, process-start identity, and a random ownership token, heartbeat while work is live, reclaim a provably dead same-host owner immediately, and remove the lease only while its token still matches.
• Stopping an SSH client does not prove the remote process stopped. Give every staging invocation a controller-issued owner ID, publish its exact remote PID and start identity, validate both the owner record and /proc/<pid>/cmdline before signalling, and keep cancellation separate from worker-container recycling. Terminate the local transport too, retry briefly for the owner-publication race, and apply a bounded no-progress deadline to SSH, Vast, and Modal staging streams.
• Queue-time model inspection should be authoritative for the queued execution. Serialize the validated, credential-free profile and its security-scan result into only the payload subtrees that use it; remote stagers can persist the generated manifest and begin snapshot checks without another mutable Hub metadata request. Bump the worker protocol when adding this payload/CLI contract so an older image cannot silently reject it.
• Model admission needs storage as well as GPU and RAM capacity. Under the per-snapshot lease, subtract bytes already present for a resumable snapshot from the profile's artifact size, require that remainder plus a configurable post-download reserve to fit on the actual mounted filesystem, and fail before opening network transfers.

2026-08-28: Vast setup failure needs bounded instance replacement

• Destroying a rental after its SSH-readiness timeout is cleanup, not recovery. Treat provider-readiness timeouts and worker-initialization failures as instance-scoped: mark the lease draining, terminate it through the Vast API, exclude its marketplace offer, and cold-start one replacement during the same acquisition.
• Preserve a strict initial-plus-one-replacement bound. If the replacement also fails, report the attempted instance IDs, whether cleanup was confirmed, and the final provider/setup cause so the workflow receives an actionable error without creating an open-ended billable rental loop.
• A persisted loading lease must consume only the remainder of its original readiness budget. Giving it a new full startup timeout after every controller restart can retain broken capacity indefinitely.
• Vast can delete a contract between an ownership check and the destroy request. A not-found response from destruction confirms that capacity is already absent and should remove the stale registry record rather than turn successful cleanup into another failure.
• Vast SSH and Jupyter runtypes replace the image entrypoint, inject access, and then run onstart. A custom base image can still leave /root or the injected key path incompatible with sshd StrictModes; normalize root ownership plus .ssh mode 0700 and authorized_keys mode 0600 in onstart. This repairs new rentals without rebuilding the heavyweight worker image.
• Workflow plans are not a complete cloud-resource inventory: cancellation or a controller restart can leave valid idle leases that no longer appear in the current plan. Put provider-owned live capacity in a separately refreshable Configurator section, target destructive controls by immutable provider ID, and repeat the ownership check on the server. A deliberate kill path must be able to override an active-work guard only after an explicit warning; routine reaping and other callers should retain the guard.
• Modal exposes active execution containers as task IDs, while its app-level stop operation can terminate unrelated warm capacity. Verify the requested task against the installation's managed app names, then call the exact SDK ContainerStop operation on Modal's synchronized event loop instead of stopping the whole deployed app.
• A click-through DOM widget cannot become interactive merely by attaching button listeners. When the panel root uses pointer-events: none so the canvas remains draggable, opt each intended control back in with pointer-events: auto, stop its pointer and wheel events at the panel boundary, and provide pressed/loading state so canvas interaction cannot swallow a destructive or refresh action invisibly.
• Vast setup changes identity when a planned configuration such as vast:380 becomes a concrete lease such as vast:380:49052528. Because readiness telemetry can arrive before the post-acquisition plan, promote the provisional environment lane as soon as that concrete ID appears and transfer its retained state; treating the concrete ID as unrelated briefly renders the same capacity twice.
• A successful manual Vast kill invalidates the whole scheduler result, not just the assignment that named that lease: removing capacity can change every placement under a fresh cost and availability calculation. Clear the retained Configurator plan and environment telemetry, and reject delayed plan or status events for that prompt so a websocket race or remount cannot resurrect assumptions that are no longer valid.