ttulttul-ComfyUI-Modal/docs/LEARNINGS.md

Learnings

2026-04-20

• Early local delivery for hybrid remote components is much simpler if the queue-time rewrite stops pretending the whole island is one proxy. Splitting the island into a static proxy and a mapped proxy lets ComfyUI unblock ordinary downstream local nodes using its existing scheduler semantics instead of inventing partial-completion behavior for one proxy node.
• The bridge between those split proxies cannot be a normal transport value when the shared upstream output is something like MODEL. The workable shape is an opaque session-ref payload: the static proxy returns a JSON-safe reference, the remote runtime stores the live object in a prompt-scoped session, and the mapped proxy resolves that ref back into the live object when it runs.
• Rewriting a component into multiple proxies means the old queue-time dependency metadata is stale. Execution stages, dependency edges, and partial-target remapping all need to be derived from the final rewritten proxy graph, not from the pre-rewrite coarse component plan.
• Prompt-scoped remote state is only useful if both halves of the split can reach the same worker. The current implementation now passes a session-affinity key into deployed RemoteEngine lookup, but that still needs live Modal verification under real scale-out before it should be treated as fully proven.
• Online deployed testing confirmed the prompt-scoped session-affinity path for the tested hybrid workflow: the mapped proxy successfully reused the static proxy's remote-only state on the same worker instead of failing with a missing session/value-ref.
• The cheapest durable regression for this feature is a proxy-level scheduling test, not a full ComfyUI integration harness. If the rewrite produces separate static and mapped proxies, and a local downstream consumer can run off the static proxy result while the mapped proxy task is still awaiting completion, that already protects the core early-unblock behavior.
• It is still worth keeping one checked-in workflow-shaped artifact for this feature family. A JSON fixture that carries both the saved workflow metadata and the queued prompt shape catches rewrite drift that pure unit tests can miss, especially around proxy ids, downstream rewiring, and shared session ids.
• Session observability is most useful when the store and the runtimes both log. Store-level logs explain create/reuse/resolve/cleanup semantics, while runtime-level logs tie those session ids back to concrete component executions and clear_remote_session boundaries.
• Modal's own modal container logs path is just a thin wrapper over the SDK's AppGetLogs stream, so local observability does not need a separate sidecar. The remote worker can surface its MODAL_TASK_ID in the existing streamed payload channel, and the local ComfyUI side can mirror that task's container logs directly into local stderr with a CLI fallback when the Python SDK is unavailable.
• In practice, the local Python SDK log stream can emit Modal client task-context warnings when called from ComfyUI's own worker threads. Preferring the modal container logs subprocess path avoids that noise while still preserving an SDK fallback for environments that only have the Python package installed.
• The hybrid split-proxy pattern is useful beyond ModalMapInput. If a coarse remote component has several execute branches that only share a remote-only upstream value like MODEL, rewriting that component into ordered ordinary subgraph proxies gives ComfyUI real early completion points without needing partial-result semantics inside one proxy.

2026-04-19

• Early local unblocking is currently limited by the proxy shape, not just by transportable boundary types. If queue-time rewrite packs a remote branch into one hybrid mapped_subgraph proxy, static outputs that are already finished still cannot drive downstream local nodes until the mapped phase also finishes, because the remote runtime only merges and returns those outputs once at the end. Fixing that needs either partial-completion semantics for one proxy node or a shared remote session/cache that lets separate proxies reuse non-transportable state like MODEL.
• Mapped remote execution cannot hoist invariant upstream work safely by caching {prompt_id}:{node_id} outputs or by replaying per-item subgraph RPCs. The queue-time rewrite has to split the component into explicit static and mapped phase prompts, expose static-to-mapped boundaries, and let one runtime invocation execute both phases so non-transportable objects like MODEL stay in-process while PromptExecutor still decides reuse from effective inputs and dependencies.
• The frontend cannot treat mapped execution like a prompt-wide single active node. Per-lane progress is the durable source of truth there: ordinary non-mapped runs can still use activeNodeId, but mapped remote styling should derive purple/active state from whichever nodes currently hold live lane or numeric progress, and lane ownership needs to move when one worker advances from node A to node B.
• The test bootstrap should not hard-code ~/git/ComfyUI as its only source of ComfyUI imports. Reading COMFYUI_ROOT first makes it easy to validate the extension against a disposable checkout in /tmp while preserving the old local-default behavior.
• A usable uv test environment for this repo needs more than pytest. Importing current ComfyUI APIs during tests also requires aiohttp, Pillow, torch, safetensors, numpy, and tqdm, so those belong in the dedicated test dependency group instead of being installed ad hoc on one machine.
• On this host, uv sync worked reliably with UV_PROJECT_ENVIRONMENT and UV_CACHE_DIR=/tmp/uv-cache, but uv run ... crashed inside the installed uv binary with a macOS system-configuration panic. The environment itself was still valid, and invoking /tmp/.../bin/python -m pytest against that uv-managed environment was a workable fallback.

2026-04-18

• The README had drifted into release-note territory, especially in the old Important distinction: block. For this project, the maintainable structure is: short overview, explicit local-vs-remote execution sections, concise operational caveats, and a separate learnings log for historical implementation details.
• ComfyUI registry setup is mostly metadata, not code. The repo needs a real [tool.comfy] block in pyproject.toml, a stable immutable node-pack id, repository URLs, and an optional GitHub publish workflow; choosing a software license should stay explicit instead of being guessed during packaging work.
• comfy node pack is not git-aware by default; it uses .gitignore as its practical packaging filter. For a clean registry artifact, this repo needs an allowlist-style .gitignore so untracked local files like logs, caches, virtualenvs, and scratch docs cannot leak into node.zip.
• Remote per-node phase events are not enough to mirror sampler progress in the local UI. The Modal worker also needs a headless equivalent of ComfyUI's global progress hook so ProgressBar updates feed progress_state, and the local relay should forward those numeric updates as a separate high-frequency UI event rather than overloading coarse execution-phase messages.
• ComfyUI prompt cancellation does not need a separate frontend API hook to reach Modal. The proxy node can watch ComfyUI's existing processing_interrupted() flag while it waits on the remote call, send one targeted remote interrupt for the active (prompt_id, component_id), and then re-raise ComfyUI's own InterruptProcessingException so the local executor reports a normal execution_interrupted.
• That cancellation watch cannot live only in the outer proxy wrapper. The blocking Modal request bridge, especially the streamed remote_gen(...) path, also has to poll processing_interrupted() itself and set the per-request cancellation event before it asks Modal to stop; otherwise a locally cancelled prompt can unwind while the remote container keeps running.
• On the Modal side, targeted interrupt delivery cannot depend on a second RPC into the same GPU execution class. The earlier interrupt_payload lane only worked by reserving extra class concurrency, which fights horizontal scale-out. The better design is a shared control store that running executions poll while the GPU class itself stays at @modal.concurrent(max_inputs=1). Older allow_concurrent_inputs class options now hard-fail on current Modal SDKs.
• To let ComfyUI overlap multiple ready remote components, the proxy node itself has to be async. Merely increasing Modal-side concurrency is not enough if the local proxy still blocks inside a synchronous execute(), because ComfyUI's scheduler only uses its pending-task overlap path for async nodes.
• There are two separate local bottlenecks to remove for remote overlap: the proxy node must await remote work instead of blocking the scheduler, and the helper that submits blocking Modal SDK calls must use more than one worker thread or the local process will still serialize every remote invocation even after the proxy becomes async.
• Horizontal Modal scale-out needs a different concurrency model from interrupt delivery. If one GPU container should run only one heavy Comfy workflow at a time, cancellation can no longer depend on a second RPC lane into the same class; the clean fix is a shared control store such as a Modal Dict, with each execution polling its own cancel flag while max_inputs stays at 1.
• Data-parallel batching fits this architecture best as a queue-time marker plus a local scheduler, not as a magical runtime pass-through node. A ModalMapInput marker can stay inside the rewritten remote component, the proxy can split one boundary input locally into per-item subgraph calls, and the remote runtime can stay unaware because each item is still an ordinary subgraph execution.
• A single undirected "remote island" is too coarse once transportable values can cross remote-to-remote boundaries. The rewrite step should only force nodes together across non-transportable edges like MODEL or CONDITIONING, then topologically order the remaining transport-aware components from upstream to downstream so later proxies can safely wire themselves against earlier proxies.
• ModalMapInput changes the partitioning rule in one important way: the whole downstream remote region reachable from that marker should stay inside one mapped component, even across otherwise transportable edges. Without that exception, the marker would only map the first transport-aware slice of the workflow instead of the downstream region the user intended.
• That first transport-aware split is still only a coarse partition. Even when the original prompt is a DAG, the quotient between coarse groups can become cyclic after you merge only selected edges. Before rewriting proxies, the backend has to collapse strongly connected coarse groups back together or ComfyUI can later see a synthetic dependency cycle.
• Streaming intermediate remote UI outputs back into local ComfyUI is safest when it reuses ComfyUI's native websocket event shapes. Forwarding remote executed payloads and preview-image frames through the local PromptServer gives the browser real mid-run previews without pretending those inner remote node values are now available as local graph inputs.
• A downstream local PreviewImage node is a different problem from a remote node that already emits UI output. If a remote boundary IMAGE feeds a local preview node, the useful fix is to remember that direct preview consumer during prompt rewrite, stream the boundary tensor as soon as the remote source node finishes, and synthesize the local preview node's executed UI payload on the local side. That improves preview UX without trying to make ComfyUI execute arbitrary downstream local nodes incrementally.
• Queue-time remote-boundary fixes should come from the backend's actual rewrite logic, not from parsing error text in the browser. A small dry-run analysis route can reuse the same prompt-id resolution and upstream auto-expansion rules, then return workflow-node paths for the frontend to mark directly from a context-menu action, including nested subgraph nodes.
• For ComfyUI node right-click menus, patch getExtraMenuOptions at node-definition time in beforeRegisterNodeDef, not per-instance in nodeCreated. The prototype-level hook matches how working extensions inject menu entries and avoids menu items silently disappearing even when other per-node UI changes still show up.

2026-04-17

• The workflow metadata already rides along in extra_data.extra_pnginfo.workflow, so the backend can discover node.properties.is_modal_remote without any extra frontend metadata channel.
• That workflow metadata is not always flat. ComfyUI can embed saved node records inside nested subgraph fragments, so Modal marker discovery has to recurse through workflow JSON instead of only scanning the top-level workflow["nodes"] list.
• Nested saved subgraph ids are not necessarily queued prompt ids. Before rewrite-time graph traversal, nested Modal markers need to be resolved against the queued prompt id space: sometimes that means a composed id like 24:23, and when ComfyUI does not expose the inner node directly it means falling back to the nearest ancestor prompt node. Without that resolution step, the backend will crash trying to rewrite a saved workflow id that does not exist in the actual prompt.
• A single wildcard-output proxy node is not enough for server-side graph rewriting. ComfyUI validates downstream link indices and output arity against the replacement class, so the backend has to generate proxy node classes that mirror the original node’s output signature before validation runs.
• Keeping the sync engine content-addressable makes prompt rewrites cheap after the first upload. The route can block on sync safely as long as repeated requests collapse to a hash marker check.
• Hashing and zipping the entire custom_nodes/ tree inside the queue route is too expensive for COMFY_MODAL_EXECUTION_MODE=local. Local mode should skip custom-node bundle sync by default, with an explicit override when that path needs to be exercised.
• When rewriting a prompt node into a proxy, never reuse the same input dict for both prompt_node["inputs"] and original_node_data["inputs"]. Inserting original_node_data back into the live prompt inputs creates a recursive structure that breaks ComfyUI cache signature hashing with a RecursionError.
• Queue-time validation is necessary for remote nodes that still receive local Comfy runtime objects. Even if a node is marked remote, inputs like MODEL or CONDITIONING cannot be serialized across the current boundary, so the prompt should be rejected before execution rather than failing inside the serializer.
• Per-node proxying is not enough for real ComfyUI sampler graphs. Connected remote-marked nodes must be partitioned into a single remote component so internal edges like MODEL, CLIP, and CONDITIONING stay inside the remote subgraph and only the true local/remote boundary has to be transport-safe.
• Manually marking every upstream dependency is too brittle for arbitrary custom nodes. A better default is to auto-expand the remote node set upstream across any non-transportable edge, so a user can mark the downstream custom node they care about and let the backend absorb the required loaders/builders into the same remote island automatically.
• The local fallback runtime cannot call PromptExecutor.execute() inline from inside a proxy node, because ComfyUI is already running inside an active asyncio loop. Running remote subgraph execution in a worker thread avoids nested-event-loop failures while still reusing ComfyUI’s executor.
• Modal objects defined in-process are not directly callable from an arbitrary long-running app like ComfyUI unless they are inside a hydrated app.run() session or looked up from a deployed app with modal.Cls.from_name(...). In practice the correct default for this project is deployed lookup only, because ephemeral app.run() fallback rebuilds too much state between workflow executions and undermines the scale-to-zero cost model.
• A deployed-only strategy is still correct, but the custom node should bootstrap that deployment itself on first use. The pragmatic flow is: try Cls.from_name(...), and if the app is missing, programmatically call app.deploy(...) from the stable cloud entry module, then retry the lookup.
• ComfyUI loads custom node packs from directories under synthetic module names derived from filesystem paths, and for directory-based packs that module key is not guaranteed to be a valid Python import path. Modal container startup requires a real importable module name for @app.cls services, so the cloud entrypoint has to live in a separate, stable module such as comfyui_modal_sync_cloud.py rather than on the custom-node package module itself.
• When the local side imports that stable cloud entrypoint for first-run auto-deploy, a failed import can poison sys.modules with a half-initialized module. The loader has to evict cached modules that do not expose a real deployable app, and it has to remove failed imports from sys.modules, or later auto-deploy attempts will keep reusing the broken stub.
• Shipping the ComfyUI source tree into the Modal image is not enough by itself. The container also needs ComfyUI's import-time Python dependencies such as psutil, aiohttp, einops, torchsde, torchvision, and related runtime packages, or the remote worker fails before it can execute any graph nodes.
• Core ComfyUI imports extend beyond the obvious tensor stack. In practice the Modal image also needs startup/runtime packages such as alembic, pydantic-settings, spandrel, and kornia, because built-in nodes and config-parsing paths import them before a workflow gets very far.
• Quantized ComfyUI models add another hard dependency: if comfy-kitchen is missing in the Modal image, comfy.quant_ops.get_layout_class(...) falls back to None and FP8/FP4 model loading eventually fails with 'NoneType' object has no attribute 'Params'.
• A captured "working workflow" exported after a remote run is not necessarily a pristine authoring graph. It can contain internal rewrite artifacts like ModalUniversalExecutor, so checked-in workflow JSON should be treated explicitly as a smoke fixture unless it has been normalized.
• When performance work starts, timestamped logging matters more than adding yet more ad hoc messages. Giving the extension its own formatter with both wall-clock time and relativeCreated makes it much easier to see whether the delay is in hashing, sync, deployment, image build, or actual execution.
• Once the slow path is visible, the first easy win is persistent local caching of digests and bundle artifacts. For this project, caching file hashes by (path, size, mtime_ns) and caching digest-keyed custom_nodes archives under the local storage root avoids re-reading huge model files and re-zipping the same custom_nodes tree on every execution.
• After the local side is fast, the next blind spot is inside the Modal container. The cloud runtime needs its own phase-timed logs around volume reload, runtime initialization, PromptExecutor execution, and output collection; otherwise the remote call just looks like one giant wall-clock block and there is no way to tell scheduling delay from real inference time.
• A dedicated Python logger is not always enough for those cloud timings. In practice, Modal container logs may still omit named-logger INFO lines during ComfyUI startup, so the timing tracer should mirror its phase lines directly to container stdout to guarantee they appear in captured remote logs.
• ComfyUI root discovery for deploy-time image building cannot rely only on ~/git/ComfyUI. When the extension is installed under ComfyUI/custom_nodes/ComfyUI-Modal, the most reliable default is to derive the ComfyUI root from that install path and only fall back to home-directory heuristics.
• If the remote image also contains ComfyUI's top-level custom_nodes/ tree, the container will import the full custom node set twice: once from the image and once from the extracted sync bundle. The image should ship ComfyUI core without custom_nodes/, and the synced bundle should be the single source of external nodes.
• ComfyUI still assumes that its default custom_nodes directory exists even if you do not want to populate it from the image. If that directory is missing, nodes.init_external_custom_nodes() can crash on os.listdir(...), so the remote runtime should create an empty placeholder directory before external-node initialization.
• Warm-container optimization is most effective when expensive but graph-agnostic work moves into @modal.enter. For this project that includes built-in ComfyUI runtime initialization and execution-module imports, so snapshots and scaledown-window reuse can amortize that startup cost across requests.
• On GPU-backed functions that only use CPU memory snapshots by default, @modal.enter(snap=True) must stay CUDA-safe. Snapshot-phase prewarm should only do CPU-only work there, and the heavier ComfyUI runtime initialization should move to @modal.enter(snap=False) unless GPU snapshots are explicitly enabled.
• If GPU memory snapshots are enabled, the snapshot-phase hook can safely pre-initialize more of ComfyUI core and capture that state. This repo now uses that split so the same code path benefits from either CPU-only snapshots or GPU snapshots without accidentally initializing CUDA in the wrong phase.
• Once warm-container runtime init is amortized, the next bottleneck is usually the graph's own loader nodes. For this repo, UNETLoader, CLIPLoader, and VAELoader are worth caching inside the warm container by resolved loader inputs, and cached outputs should be cloned when possible so later requests can reuse loaded runtime objects safely.
• Loader caches are only part of the story. Local ComfyUI keeps one long-lived PromptExecutor in the prompt worker, so ordinary node outputs stay cached across runs as long as their signatures do not change. A remote worker that creates a fresh PromptExecutor per request throws that benefit away, so it needs a warm-container executor pool keyed by cache scope to preserve ComfyUI-style node-output reuse between runs.
• End-to-end prompt_executor_execute timing is still too coarse once the denoising loop is only a small fraction of the total. A headless prompt server can observe executing and executed events from PromptExecutor and emit per-node spans, which exposes whether the real time is in model loaders, encoders, or the sampler itself.
• For repeat ComfyUI workflows, the highest-leverage lifecycle knob is usually not min_containers=1 but a longer scaledown_window with min_containers=0. That lets a recently used container keep its fully initialized ComfyUI process warm for a while without forcing permanent idle billing between totally unrelated runs.
• Floating pip install torch torchvision in the Modal image is too unstable once ComfyUI tracks very new PyTorch releases. The remote image should pin the same known-good CUDA wheel family as the local ComfyUI environment, and for this repo that means the explicit PyTorch CUDA 12.8 index with torch==2.10.0, torchvision==0.25.0, and torchaudio==2.10.0.
• The frontend can reuse ComfyUI's native executing, execution_error, and execution_success websocket events for remote component visuals as long as the queue route also publishes the representative-node-to-original-node mapping after rewrite. That keeps the UI in sync with actual proxy execution instead of inventing a separate polling channel.
• When filtering the ComfyUI checkout for the Modal image, only exclude top-level runtime asset directories like models/ and output/. A blanket "ignore any path part named models" rule will also delete real Python packages such as comfy/ldm/models, causing import failures inside the container.
• Importing nodes.py alone is not enough inside the Modal container. The remote worker also has to run nodes.init_extra_nodes(...) once so built-in comfy_extras nodes such as EmptyFlux2LatentImage exist in NODE_CLASS_MAPPINGS, and then run init_external_custom_nodes() for each extracted custom-node bundle root it exposes.
• A bare PromptExecutor(_NullPromptServer()) is not equivalent to real ComfyUI startup. The headless worker must pass the same cache configuration that main.py uses, or execution eventually crashes when the executor polls caches with self.cache_args["ram"].
• If /modal/queue_prompt does expensive preparatory work before it returns, the frontend can keep ComfyUI's own queue/execution indicators alive by dispatching temporary synthetic status, execution_start, and executing events, then letting the real websocket events take over once the prompt is actually queued.
• For remote node-level UI feedback, a streamed Modal generator method is cleaner than a side-channel file or volume poller. Yielding progress envelopes directly from the remote execution method lets the local client forward active-node updates into ComfyUI's websocket stream in real time without extra storage commits, reloads, or cleanup logic.
• Some custom-node packs register HTTP routes or prompt handlers at import time through PromptServer.instance. A headless remote worker still needs to provide a minimal PromptServer.instance shim with routes, app, supports, and no-op send methods, or those packs fail during external custom-node initialization before any workflow execution begins.
• For good remote UX, one generic "executing" node state is not enough. The frontend needs separate visual states for "setup", "ready in Modal", "currently executing remotely", "finished for this run", and "workflow fully done", with whole-workflow success clearing the temporary remote states back to the normal idle outline.
• The global Modal activity badge needs the same cleanup discipline as the per-node overlays. If a prompt entry survives in badge state after the prompt itself is gone, the UI can get stuck showing an orange "setup running" badge even though the workflow finished; pruning orphaned prompt ids fixes that.
• One orange badge state is not expressive enough. The frontend can use queue-time modal_status setup for "Syncing graph with Modal", then switch to a separate waiting state when the proxy begins executing but before the first streamed remote-node event arrives, which makes the remote lifecycle much easier to understand from the UI.
• Browser backgrounding can make the badge look stale even when the underlying prompt state already advanced. Deriving the global pill phase from live per-prompt node state, and refreshing it again on visibilitychange / focus, makes the pill recover promptly when the user returns to the tab.
• Subgraph-expanded prompt ids need UI aggregation just as much as backend marker resolution. If a streamed remote event targets 24:23, the visible outer subgraph node 24 should derive its phase from all descendant remote ids under that prefix; otherwise the outer node can stay stuck in setup even though the internal remote nodes are already running or complete.
• Warm-container caches and loader reuse can conflict with modal.Volume.reload(). If a request truly needs a fresh volume reload after uploading new assets, the worker may have to clear warm loader caches, drop reusable prompt executors, unload ComfyUI models, and retry the reload once to get past Modal's "open files preventing the operation" error.
• The synthetic frontend events have to match ComfyUI's websocket adapter shapes exactly, not just semantically. In practice that means status detail must be the inner status payload, executing detail should be the display node id string, and queue-list initialization state benefits from a matching "Waiting for a machine" notification payload keyed by prompt id.
• Even with correctly-shaped synthetic events, ComfyUI's built-in queue UI may still not present an obvious "workflow running" state during long pre-queue Modal setup. The extension should own a separate global activity badge so users have explicit feedback while the remote runtime is still being prepared.
• Synced asset paths cannot stay as abstract /assets/... markers inside the Modal worker. Remote execution needs those markers materialized to real absolute files under the mounted storage root, and ComfyUI's folder_paths helpers need an absolute-path fast path so loader nodes can still resolve them.
• A local filesystem mirror is not enough in COMFY_MODAL_EXECUTION_MODE=remote. Queue-time sync must upload assets into the actual named Modal volume via the local SDK, and the remote worker must call vol.reload() before execution so warm containers can see newly uploaded files.
• Once warm-container caches keep models and executors alive, unconditional vol.reload() becomes wrong: Modal can reject reloads while files from the mounted volume are still open. The queue side already knows whether a request actually uploaded anything new, so that fact should travel in the payload and only requests with fresh uploads should reload the mounted volume.
• Fast cancel-and-restart sequences can still race the container's file-handle cleanup even after warm caches are cleared. For the specific Modal "open files preventing the operation" reload failure, the remote worker needs a generous bounded retry window after cache/model cleanup rather than a single immediate retry.
• That queue-time requires_volume_reload bit is not enough once one logical request fans out into many per-item Modal calls. Mapped execution also needs a stable per-request reload marker so each warm container can reload the mounted volume once for that asset snapshot and skip the duplicate reloads from later mapped items.
• The reload marker also has to be request-wide across remote components, not minted separately per component payload. Otherwise a single workflow that rewrites into several remote chunks can still force repeated vol.reload() calls on the same warm container as it advances from one chunk to the next.
• Even after bounded retries, vol.reload() can still fail on a warm container that already has the right hashed assets visible on disk. In that case the worker should verify the payload's referenced mounted-volume paths and proceed when they are already present, rather than treating a redundant reload failure as fatal.
• ModalMapInput cannot blindly repeat every execute target in its coarse remote component. If a mapped branch shares a non-transportable upstream dependency with an unmapped remote sibling, the payload needs to separate one-time execute nodes from per-item execute nodes so the sibling branch still runs once while the mapped branch fans out per item.
• Mapped Modal fanout needs stable local worker lanes, not just a semaphore over one task per item. The UI can only render sensible concurrent progress if each in-flight mapped run carries a durable lane id; otherwise parallel item updates all target one node-progress slot and the bar flickers incoherently.
• Hybrid mapped components also need stream-level UI filtering. A warm PromptExecutor can still emit cached executed or preview UI events for static sibling nodes during per-item mapped runs, so the local relay has to drop any streamed UI event whose node id is outside the current mapped/static payload's execute and boundary-output node set.
• Modal proxy nodes that sit on a mapped boundary must opt into ComfyUI INPUT_IS_LIST semantics themselves. Otherwise ComfyUI can auto-map the proxy once per item before Modal-Sync sees the batch, which makes the entire hybrid component rerun per item instead of letting the internal mapped scheduler split only the intended branch.
• Once a proxy opts into INPUT_IS_LIST, ComfyUI also wraps the proxy's hidden payload input itself. The proxy has to unwrap singleton lists on original_node_data as well as ordinary kwargs, or it will reject valid rewritten payloads with a misleading "must be a mapping" error before remote execution even starts.
• Rewritten remote prompts need a last-mile canonicalization pass before execution. If a socket index leaks through as ["node", [0]], a scalar input arrives as [4], or an execute node id arrives as ["12"], ComfyUI's executor eventually crashes trying to treat the list like an integer or node id; normalizing those singleton wrappers on both the local and Modal-side runners avoids that failure mode.
• Once the easy wrapper shapes are normalized, the next debugging step should be structured worker-side diagnostics, not more blind guesses. Logging the normalized execute-node ids, boundary outputs, suspicious remaining wrapped prompt inputs, and the raw ComfyUI execution_error payload makes the surviving malformed field visible on the very next remote failure.
• Hybrid mapped components cannot hand the whole coarse component prompt to every sub-run. Even when a static sibling should execute once, PromptExecutor still validates the full prompt, so static and per-item sub-runs must prune the rewritten prompt to the dependency closure of their own execute_node_ids or the unrelated branch's batched inputs can still fail validation.
• That trim pass also has to tolerate stale execute ids that survive into a sub-run payload after rewrite. Dependency resolution should drop any execute target missing from the current trimmed prompt instead of assuming every listed node id still exists and crashing with KeyError.
• Prompt canonicalization also has to run on hydrated boundary inputs, not just on the stored subgraph prompt. Otherwise a proxied scalar boundary value like [4] can be spliced back into an INT socket unchanged and fail later inside PromptExecutor with int(list).
• A generic PromptExecutor failure string is not enough for debugging rewritten remote prompts. The worker should surface execution_error context like node_id, node_type, and current_inputs, and it should preflight primitive widget sockets for raw list literals before execution so the first visible error names the offending node/input/value instead of only reporting int(list).
• Modal's public Python SDK does not expose a direct "kill this @app.cls container" call from the local client. If we want crashy warm GPU workers to disappear immediately, the practical design is to have the local side opt into crash teardown and the remote worker schedule os._exit(1) on itself after surfacing the exception.
• Content-addressed sync should trust the deterministic payload path, not only the sidecar marker. If a crash happens after uploading a hash-named asset but before writing its .done marker, the next run should backfill the marker and reuse the existing payload instead of trying the same upload again and failing with FileExistsError.
• The streamed Modal generator path needs the same tensor-aware serialization rules as ordinary node input/output transport. If a progress or result envelope leaks a raw torch.Tensor, Modal's JSON encoding fails before the local client ever sees it, so the stream layer should normalize event payloads through the shared serializer too.
• Modal volume metadata calls can become a bottleneck on their own. Repeated VolumeListFiles checks for the same marker path are worth memoizing for the life of the queue request, and transient ResourceExhaustedError responses should be retried with backoff instead of surfacing immediately as hard prompt failures.
• Modal volume SDK calls should not run directly on the aiohttp request event loop. Running them in a worker thread avoids AsyncUsageWarning, and a missing path from Volume.listdir() should be treated as an ordinary "not uploaded yet" cache miss instead of a hard failure.
• The Modal container can pick up a non-package utils module before ComfyUI initializes. Preloading ComfyUI's own top-level utils package from the bundled source tree prevents utils.install_util import failures during init_extra_nodes(...).
• If the goal is "no idle billing, but no minute-long rebuild every run", the right Modal pattern is a deployed app with min_containers=0, not ephemeral app.run(). That lets Modal reuse the built image and deployed objects while still scaling compute down to zero between executions.
• Modal hardware selection belongs in deployment configuration, not in request payloads. For this repo, the remote class should read a configurable GPU string from environment at deploy time, and changing it later requires a redeploy or app recreation because existing deployed app hardware does not update in place.
• Modal memory snapshots only help once the app is deployed. They are a good default for CPU-side initialization work, while GPU memory snapshots should remain opt-in because Modal still labels them alpha and they may require additional runtime refactors.
• A remote proxy cannot treat every incoming list literal as just another scalar boundary input. Ordinary remote components also have to preserve ComfyUI's implicit zipped batch semantics across the boundary, or primitive widget sockets like seed: INT end up receiving whole lists such as [10, 11, 12, 13] instead of per-item values.
• That implicit batching path also needs the same hybrid static-vs-mapped split as explicit ModalMapInput. If two remote execute nodes share a non-transportable upstream object like MODEL but only one of them depends on the multi-item boundary input, the sibling execute node still has to run once, not once per batch item.
• Suppressed per-item progress streams should not overwrite display_node_id with the component representative. Lane ids are enough to separate concurrent runs, so progress bars can stay attached to the actual executing node instead of collapsing onto the first node in the remote island.
• Placeholder lane-start updates are counterproductive for mapped progress. If the local scheduler paints a lane on the component representative node before any real node_progress event arrives, that ghost bar can outlive the later real-node updates and make the first node in the island look like it is still sampling.
• Aggregate batch-completion updates are a different kind of signal from real node progress. Reusing the same per-node rendering path for both makes ordinary remote nodes look like they emit sampler-style progress when they do not, so the frontend should treat aggregate mapped progress as global-only UI state.
• Aggregate mapped progress is still useful, just not as a fake node bar. The right presentation is UI that is explicitly batch-oriented: a shaded global status pill for overall completion and a small 3/16-style badge on the representative remote node.
• Prompt-wide UI state must not be rebuilt from one component's latest status event. Once the backend has published the full remote-component membership for a prompt, later per-component modal_status events should merge into that prompt metadata rather than shrinking it to whichever component happened to report most recently.
• ComfyUI progress_state payloads are not all true numeric progress. Upstream nodes can report trivial running states like value=0, max=1, and forwarding those verbatim makes loaders and text encoders sprout fake progress bars. The remote worker should treat only states with a meaningful progress range as node-local progress.
• Queue-time Modal setup is too long-lived to hide behind one generic "Syncing graph with Modal" label. The websocket status payload should be able to carry human-readable setup detail like packaging the custom-nodes ZIP, naming the asset currently uploading, and a short finalizing phase while the local side is still receiving remote outputs.
• Request-wide reload markers are useful for deduping vol.reload() across components, but the reload trigger itself should still be component-specific. If any uploaded asset in the workflow marks every component requires_volume_reload=True, warm containers end up retrying reload for components that never reference the fresh file.
• Reload diagnostics need to speak in terms of mounted-volume paths the payload actually references. Logging both the newly uploaded relevant paths and the broader referenced mounted-volume paths makes it much easier to see whether reload is being blocked by files the component truly needs or by stale warm-container handles from previous work.
• Ready/green is a prompt-acceptance state, not a per-component execution state. If the frontend waits for each proxy node's first executing event before marking nodes ready, most of the remote graph stays orange even after the rewritten prompt has already been accepted by Modal. The queue-success path should flip the whole remote prompt to ready immediately, while purple remains reserved for the currently active node.
• Websocket setup events are not guaranteed to land before the HTTP queue response resolves. If the queue-success path relies on prompt metadata that only arrived over websocket, backend-expanded upstream nodes can remain orange because the frontend only knows about the originally toggled workflow nodes. The accepted queue response should echo the resolved remote-node ids and component map so ready-state painting is deterministic.
• Component membership needs a reverse lookup as well as a representative lookup. Mapped progress and native ComfyUI execution events often arrive keyed to an ordinary node inside the remote island, and if the frontend can only expand representative ids it leaves the rest of that component stuck orange even though the island is already running.
• Streamed Modal node-progress events need to preserve the real executing node id even when ComfyUI also reports a display node id. Using only the display id collapses fan-out progress onto the mapped-input receiver or another proxy node, which hides the actual executing node and breaks lane cleanup when the mapped worker finishes.
• Queue-time rewrite has enough graph structure to estimate remote concurrency before execution starts. A stage decomposition of the remote component DAG is cheap to compute during prompt rewrite and is good enough to start warming containers while the prompt is still entering ComfyUI's queue.
• Queue-time planning still cannot know exact mapped fan-out if the batch size depends on runtime data. The practical compromise is to store the staged component plan in extra_data.modal, warm conservatively from rewrite-time structure, and then let mapped execution top that target up as soon as the exact item count is known.
• In-node progress bars are easy to lose underneath ComfyUI widgets. Rendering the remote progress UI in a small temporary panel below the node keeps sampler progress, parallel lanes, and batch badges readable without competing with text boxes or other widget chrome.
• Remote subgraph execution has to apply ComfyUI's primitive widget coercion before it calls PromptExecutor or manual local node execution. ComfyUI's validator converts literal INT, FLOAT, BOOLEAN, and STRING inputs in-place, and skipping that step remotely leaves values like steps=18.0 alive long enough to crash custom nodes that assume true Python integers.
• ComfyUI's CacheKeySetInputSignature is reusable for a distributed node-output cache, but the raw key object is an in-memory frozenset/tuple structure rather than a persistence-ready string. The workable approach is to canonicalize that structure into stable JSON, hash it into an NC_... key for Modal Dict, and only persist entries whose outputs are both transport-safe and below a strict size ceiling.
• Remote boundary injection can splice literal tensors back into CacheKeySetInputSignature objects for otherwise cacheable nodes like KSampler. Those tensors should contribute a stable content digest to the distributed cache key instead of poisoning the whole signature with ComfyUI's in-memory Unhashable marker.
• For remote distributed caching, cache_key_set.get_data_key(node_id) is not always reusable as-is. CacheKeySetInputSignature may already have converted boundary tensors into Unhashable, so the remote cache layer has to rebuild the same ancestry-based input signature from dynprompt and is_changed_cache before canonicalizing it for persistence.
• Restoring persisted outputs before PromptExecutor.execute() is too early. execute() immediately rebuilds the live cache-key set with its own set_prompt() call, so a distributed cache restore must run inside or immediately after that phase if the restored entries are supposed to participate in the actual execution walk.
• Mapped progress lanes have to preserve the scheduler-assigned lane id all the way through the remote worker. If the worker hardcodes every item's node progress to lane 0, concurrent mapped items overwrite each other in the frontend and upstream sampler bars appear to flicker or disappear entirely.
• Static mapped sub-runs cannot reuse the same "suppress status chatter" rule as per-item mapped lanes for numeric node progress. Those static phases usually do not have a lane id at all, so the local relay has to keep lane-less node_progress events for execute/boundary nodes or upstream samplers like KSampler will run remotely without ever showing a bar in the UI.
• Once those lane-less static progress events are forwarded, they also need their own explicit clear path. Suppressed static streams do not forward the terminal execution_success status event into the frontend, so relying on phase completion to clear ordinary progress leaves the last 100% sampler bar stuck onscreen.
• Distributed node-cache restore and persist should not be treated as unrelated phases. If one prompt run restores NC_<hash> into the live PromptExecutor cache and then persists the same node/key again unconditionally, Modal Dict churns on pure cache hits for no benefit; the persist pass should recognize restored hits and skip rewriting the same key.
• A distributed node-cache size cap that is generous enough for medium images can still be too expensive in practice. In these logs, repeatedly serializing and writing ~25 MiB cache entries dominated the post-execution phase, so the default cap should bias toward smaller transport-safe artifacts unless there is a clear reason to keep large intermediates.
• Once the persist phase knows a node is a restored-hit, that check has to happen before entry serialization and size estimation. Otherwise the code still burns CPU serializing outputs it already knows it is going to skip writing.
• Distributed node-cache restore also benefits from fan-out/fan-in structure. The old per-node await key -> await Dict get -> deserialize loop made total restore time roughly the sum of all independent Dict reads, while batching those lookups concurrently preserves prompt-order hydration but trims the network-bound portion substantially.
• Implicit mapped fan-out cannot treat every Python list as a batch axis. ComfyUI semantic payloads like CONDITIONING are also list-backed, and splitting them per item corrupts the conditioning structure into fragments like ["cond", {...}] -> "cond" that later explode in sampler_helpers.convert_cond. Only explicitly batchable transport types and list-wrapped primitive sockets should participate in that automatic zipping path.
• Primitive widget sockets are not always scalar once a graph is running. Custom nodes can legally emit list(INT) or similar values into an INT input and rely on ComfyUI's normal over-list execution behavior, so remote validation should only reject list values that are still raw widget literals, not boundary-fed or upstream-fed runtime values.
• Singleton-list cleanup on the remote boundary needs the same nuance. It is correct to unwrap transport-added wrappers like [4] -> 4 and [[["node", [0]]]] -> ["node", 0], but wrong to flatten single-entry semantic payloads like [[tensor, meta]] for CONDITIONING; that shape change is enough to turn convert_cond() into a tensor indexing crash.