Jedrzej KosinskiandAmp
866b7e32a3
ComfyUI's V3 ``build_nested_inputs`` (comfy_api/latest/_io.py L1722) rebuilds
AUTOGROW / DYNAMIC_COMBO runtime kwargs into nested dicts before they reach
the proxy_node's ``execute()``. For an AUTOGROW input named
``reference_images``, the value arrives as ``{"image1": <tensor>,
"image2": <tensor>, ...}``; for a DYNAMIC_COMBO branch with a nested
AUTOGROW (e.g. Wan2ReferenceVideoApi's ``model`` DC with nested
``video1..video3`` + ``image1..image5``), the value arrives as
``{"<dc-name>": branch_key, "video_refs": {"video1": <VideoInput>},
"reference_images": {"image1": <tensor>}, ...}``.
Before this change ``_encode_one`` only encoded values where the **top-
level** value was itself a torch.Tensor / VideoInput / audio dict /
model3d input. A non-audio dict (the AG / DC runtime shape) failed all
four duck-type checks and was returned unchanged, so per-slot tensors /
VideoInputs survived past encoding and either crashed at
``json.dumps`` time (no custom JSONEncoder for torch.Tensor) or
violated the server's contract (comfy_rnp_server's
``_ordered_wan_videoedit_image_envelopes`` and the WAN2 / HappyHorse /
Grok / Bria / Luma AG providers shipped over the last 5 PRs all
require each slot value to already be an RNP envelope before they
reach the upload step).
The same gap also defeated the externalization cap: ``_encode_inputs``
only ran ``maybe_externalize`` when the *top-level* encoded value
was itself an envelope, so per-slot envelopes inside an AG dict would
ship inline regardless of ``max_inline_bytes``.
Fix is structural — match the runtime data model:
1. ``_encode_one`` recurses into plain (non-envelope, non-audio) dict
values, calling itself for each entry. Envelope dicts pass through
so we don't double-encode; audio dicts still route to
``encode_audio_input`` because their dict shape (waveform +
sample_rate) is a heavy leaf, not a container. The recursion
passes the parent's ``accepted_encodings`` downward — single-image
AG slots and VIDEO / AUDIO / MODEL_3D leaves don't depend on it,
and gating per-child policy on descriptor metadata adds plumbing
for no concrete provider need today.
2. New ``_externalize_nested`` helper walks the encoded tree the same
way: envelope leaves go through ``serialization.maybe_externalize``;
plain dicts recurse; everything else passes through. ``_encode_inputs``
replaces the old "if is_envelope: externalize" guard with one call
to the recursive walker, so per-slot envelopes inside AG / DC dicts
get the same presigned-PUT treatment top-level envelopes get.
Lists are intentionally NOT traversed — AG / DC runtime values always
materialize as dicts, never lists, and recursing into arbitrary lists
would touch opaque payloads. Recursion depth is bounded by the
upstream Autogrow max-slot cap (100) and the maximum DC branch
nesting depth observed in any provider today (1), so an in-place
``async def`` walk is well-bounded.
Tests:
* New ``notes/run_encode_nested_dicts.py`` standalone smoke test
exercises 9 scenarios: top-level AG-of-IMAGE, top-level AG-of-VIDEO,
DC-wrapping-AG with branch key untouched, AUDIO dict still routes to
encode_audio_input, pre-encoded envelope passes through unchanged,
scalar config dict passes through unchanged, ``_externalize_nested``
walks per-slot envelopes (cap=3 forces every slot to externalize),
scalar siblings preserved alongside externalized envelopes, full
encoded payload is JSON-serializable (regression guard for the
exact bug fixed), and single-tensor / single-video / single-audio
top-level inputs still encode unchanged (no leaf-path regression).
* Loads ``_encode_one`` / ``_encode_inputs`` / ``_externalize_nested``
out of proxy_node.py via AST extraction so it runs without
importing the full comfy_api / comfy_api_nodes packages, same
pattern as ``run_image_max_batch.py``.
Unblocks the next batch of comfy-rnp-server providers that need
AG-of-VIDEO or DC-wrapping-AG-of-VIDEO support (WanReferenceVideoApi
``character1..character3`` and Wan2ReferenceVideoApi
``video1..video3`` + ``image1..image5``).
Co-authored-by: Amp <amp@ampcode.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019e367d-c5b5-758d-9387-e44477ef4cb5
comfy-remote-nodes
ComfyUI custom-node package that fetches node descriptors from an RNP/1 (Remote Node Protocol v1) server and exposes them as native V3 nodes inside ComfyUI.
The reference server lives at
Comfy-Org/comfy-rnp-server
(private). This client is intentionally provider-agnostic: any server
that speaks RNP/1 can publish nodes through it.
Install
Clone into your ComfyUI custom_nodes/ directory:
cd ComfyUI/custom_nodes
git clone https://github.com/Comfy-Org/comfy-remote-nodes.git
(The directory name on disk doesn't matter — ComfyUI loads the package
via spec_from_file_location, so comfy-remote-nodes/ is fine.)
Configure
Point the extension at one RNP server URL via env var before starting ComfyUI:
export RNP_SERVER_URL=http://127.0.0.1:9190
python main.py
If RNP_SERVER_URL is unset, the extension registers zero nodes and
logs a single line — ComfyUI starts normally with no remote nodes.
What works today
- Manifest + capability handshake
object_infofetch and dynamic V3 class generation (with ETag conditional GETs)- Sync
executeand asyncexecute_async+ poll/cancel - Image / mask / audio / video envelope encode + decode
- Presigned upload helper for >8 MiB inputs
- Tier 3 (recent):
TASK_LOSTdetection with idempotent resubmission — survives an RNP server restart mid-task without losing the workflowSERVER_BUSY/MAINTENANCE503 envelopes treated as rate limits withRetry-Afterhonoring and friendly per-second progress text ("Server busy, retrying in 30s...")- Out-of-band
url_fetchfor URI-only output envelopes, driven by the descriptor'sremote.url_fetchpolicy - Per-poll auth refresh so long-running tasks can rotate tokens
What does not (yet)
validate_inputs/fingerprint_inputs/check_lazy_status— the server returns501 NOT_IMPLEMENTED; the client falls through to ComfyUI's defaults- Multi-server config file (
comfy_remote_nodes.yamlor similar) — today's setup is one server per ComfyUI process via env var - Streaming execute (server-sent events for incremental previews)