main
14
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Jedrzej KosinskiandAmp
|
866b7e32a3 |
fix(encoder): recurse into nested AUTOGROW / DYNAMIC_COMBO runtime dicts
ComfyUI's V3 ``build_nested_inputs`` (comfy_api/latest/_io.py L1722) rebuilds
AUTOGROW / DYNAMIC_COMBO runtime kwargs into nested dicts before they reach
the proxy_node's ``execute()``. For an AUTOGROW input named
``reference_images``, the value arrives as ``{"image1": <tensor>,
"image2": <tensor>, ...}``; for a DYNAMIC_COMBO branch with a nested
AUTOGROW (e.g. Wan2ReferenceVideoApi's ``model`` DC with nested
``video1..video3`` + ``image1..image5``), the value arrives as
``{"<dc-name>": branch_key, "video_refs": {"video1": <VideoInput>},
"reference_images": {"image1": <tensor>}, ...}``.
Before this change ``_encode_one`` only encoded values where the **top-
level** value was itself a torch.Tensor / VideoInput / audio dict /
model3d input. A non-audio dict (the AG / DC runtime shape) failed all
four duck-type checks and was returned unchanged, so per-slot tensors /
VideoInputs survived past encoding and either crashed at
``json.dumps`` time (no custom JSONEncoder for torch.Tensor) or
violated the server's contract (comfy_rnp_server's
``_ordered_wan_videoedit_image_envelopes`` and the WAN2 / HappyHorse /
Grok / Bria / Luma AG providers shipped over the last 5 PRs all
require each slot value to already be an RNP envelope before they
reach the upload step).
The same gap also defeated the externalization cap: ``_encode_inputs``
only ran ``maybe_externalize`` when the *top-level* encoded value
was itself an envelope, so per-slot envelopes inside an AG dict would
ship inline regardless of ``max_inline_bytes``.
Fix is structural — match the runtime data model:
1. ``_encode_one`` recurses into plain (non-envelope, non-audio) dict
values, calling itself for each entry. Envelope dicts pass through
so we don't double-encode; audio dicts still route to
``encode_audio_input`` because their dict shape (waveform +
sample_rate) is a heavy leaf, not a container. The recursion
passes the parent's ``accepted_encodings`` downward — single-image
AG slots and VIDEO / AUDIO / MODEL_3D leaves don't depend on it,
and gating per-child policy on descriptor metadata adds plumbing
for no concrete provider need today.
2. New ``_externalize_nested`` helper walks the encoded tree the same
way: envelope leaves go through ``serialization.maybe_externalize``;
plain dicts recurse; everything else passes through. ``_encode_inputs``
replaces the old "if is_envelope: externalize" guard with one call
to the recursive walker, so per-slot envelopes inside AG / DC dicts
get the same presigned-PUT treatment top-level envelopes get.
Lists are intentionally NOT traversed — AG / DC runtime values always
materialize as dicts, never lists, and recursing into arbitrary lists
would touch opaque payloads. Recursion depth is bounded by the
upstream Autogrow max-slot cap (100) and the maximum DC branch
nesting depth observed in any provider today (1), so an in-place
``async def`` walk is well-bounded.
Tests:
* New ``notes/run_encode_nested_dicts.py`` standalone smoke test
exercises 9 scenarios: top-level AG-of-IMAGE, top-level AG-of-VIDEO,
DC-wrapping-AG with branch key untouched, AUDIO dict still routes to
encode_audio_input, pre-encoded envelope passes through unchanged,
scalar config dict passes through unchanged, ``_externalize_nested``
walks per-slot envelopes (cap=3 forces every slot to externalize),
scalar siblings preserved alongside externalized envelopes, full
encoded payload is JSON-serializable (regression guard for the
exact bug fixed), and single-tensor / single-video / single-audio
top-level inputs still encode unchanged (no leaf-path regression).
* Loads ``_encode_one`` / ``_encode_inputs`` / ``_externalize_nested``
out of proxy_node.py via AST extraction so it runs without
importing the full comfy_api / comfy_api_nodes packages, same
pattern as ``run_image_max_batch.py``.
Unblocks the next batch of comfy-rnp-server providers that need
AG-of-VIDEO or DC-wrapping-AG-of-VIDEO support (WanReferenceVideoApi
``character1..character3`` and Wan2ReferenceVideoApi
``video1..video3`` + ``image1..image5``).
Co-authored-by: Amp <amp@ampcode.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019e367d-c5b5-758d-9387-e44477ef4cb5
|
||
|
Jedrzej KosinskiandAmp
|
197e7ba406 |
Map "MODEL_3D" input type to MultiType socket (mirror upstream union)
Follow-up to PR #20: the `MODEL_3D` input wire type now resolves to an `IO.MultiType.Input(types=[IO.File3DGLB, IO.File3DOBJ, IO.File3DFBX, IO.File3DAny])` socket — symmetric counterpart of the existing `"MODEL_3D"` entry in `_OUTPUT_CLASSES` that maps to `IO.File3DGLB.Output`. Previously the descriptor wire type `"MODEL_3D"` fell through to the opaque `IO.Custom("MODEL_3D").Input(...)` socket, which is a single-comfytype socket the frontend only allows connections from identically-tagged outputs. Real upstream 3D sources expose different comfytypes: - `Load3D` outputs `IO.File3DAny` (comfytype `FILE_3D`). - Meshy / Hunyuan3D / Tripo / Rodin partner nodes output `IO.File3DGLB` (comfytype `FILE_3D_GLB`) via the existing `_OUTPUT_CLASSES["MODEL_3D"]` mapping. - Meshy's dual `FILE_3D_FBX` output socket emits `FILE_3D_FBX`. None of these connect to `IO.Custom("MODEL_3D")`. The upstream Tencent Hunyuan3D MODEL_3D-input nodes (`TencentModelTo3DUVNode` / `Tencent3DTextureEditNode` / `Tencent3DPartNode` / `TencentSmartTopologyNode`) all use `IO.MultiType.Input(types=[...])` to accept the union; mirroring that union here keeps the proxy socket compatible with every existing 3D-emitting node. `get_io_type()` on `IO.MultiType.Input` returns the comma-joined inner type IDs (e.g. `"FILE_3D_GLB,FILE_3D_OBJ,FILE_3D_FBX,FILE_3D"`) — matches the convention SxS workflows already use for their `SaveGLB` connector inputs. Per-node format validation (e.g. 3DTextureEdit's FBX-only enforcement, 3DPart's FBX-only enforcement) is done server-side in each provider's `execute()` against the inline envelope's `format` extra; this socket-level filter is purely about frontend connection validity. Unblocks the paired comfy-rnp-server PR (`TencentModelTo3DUVNode_RNP` + `Tencent3DPartNode_RNP`) which uses the `["MODEL_3D", {}]` wire shape. Co-authored-by: Amp <amp@ampcode.com> Amp-Thread-ID: https://ampcode.com/threads/T-019e362f-db40-7338-8ca4-99f617d89812 |
||
|
Jedrzej KosinskiandAmp
|
01f71bd806 |
Add MODEL_3D input encoder to close the encode-side gap
Adds the symmetric client-side encoder for `File3D` values so a remote node descriptor can declare a `MODEL_3D` input and consume a 3D mesh from an upstream local node (e.g. `Load3D`, `TencentTextToModelNode`, `MeshyTextToModelNode`). Mirrors the existing AUDIO encoder (`encode_audio_input`, `is_audio_input`) and VIDEO encoder (`encode_video_input`, `is_video_input` — PR #19) added at `comfy_remote_nodes/serialization.py`: - `encode_model3d_input(file3d)` reads the primary mesh bytes via `File3D.get_data()` (returns a `BytesIO`, per `comfy_api.latest._util.geometry_types.File3D`) and emits a `{type: "model_3d", encoding: "glb_inline", data: b64, format: ...}` envelope. `encoding="glb_inline"` is the generic single-file inline encoding the server-side decoder (`decode_model3d_envelope`) already dispatches on; the decoder hands the bytes to `File3D(BytesIO(...), format)` regardless of whether the actual format is glb / obj / fbx / etc., so the encoding name is a logical "inline single-file 3D" tag and the `format` extra picks the file extension at decode time. - `is_model3d_input(value)` duck-types on the `format` string attribute + `get_data` callable. Both are present on the upstream `File3D` class and the `BundledFile3D` subclass built lazily in `_bundled_file3d_class()`. Avoids importing `File3D` at module load (which would pull torch via the `comfy_api` tree). - `_encode_one` in `proxy_node.py` dispatches AUDIO / VIDEO before MODEL_3D as defense-in-depth (the `VideoInput` shape — `save_to` + `get_duration` — does not expose a `format` string property, so VIDEO and MODEL_3D values don't collide either way). Multi-file bundles (`BundledFile3D`-style values where every companion file matters — e.g. an OBJ with sibling .mtl + texture PNGs) are intentionally not supported on the encode path in this PR: the first server consumer is the Tencent Hunyuan3D MODEL_3D-input node family (`TencentModelTo3DUVNode` / `Tencent3DPartNode` / `Tencent3DTextureEditNode` / `TencentSmartTopologyNode`), and the upstream upload helper `upload_3d_model_to_comfyapi(cls, model_3d, file_format)` only uploads a single file (`model_3d.get_data()`). Bundle encode (`bundle_inline`) lands when a partner node actually needs the companion bytes. This unblocks 4 Tencent Hunyuan3D nodes (out of 6 in the module) that were deferred to a follow-up PR — see matrix coverage TODO #6. First server consumers ship in a paired comfy-rnp-server PR (TencentModelTo3DUVNode_RNP + Tencent3DPartNode_RNP). Co-authored-by: Amp <amp@ampcode.com> Amp-Thread-ID: https://ampcode.com/threads/T-019e362f-db40-7338-8ca4-99f617d89812 |
||
|
Jedrzej KosinskiandAmp
|
04ca70a853 |
Add VIDEO input encoder to close the encode-side gap
The proxy_node has accepted VIDEO outputs since PR #13 (server-side `make_video_envelope` + client-side `decode_video_envelope`), but the inbound encoding direction has been a no-op: `_encode_one` in `proxy_node.py` only dispatched AUDIO + IMAGE + MASK tensors, so any remote node declaring an `IO.Video.Input` would receive the raw `VideoInput` object as a non-JSON-serialisable Python value and the upstream serialisation would either drop it or fail. `serialization.py:468` made the gap explicit with a placeholder comment ("encode lands when a remote node accepts VIDEO inputs"), and several server-side providers shipped against the `VIDEO_MP4_BASE64` capability + `input_serialization={"video": "mp4_base64"}` declarations on the assumption that the client encoder would land soon — most notably `GrokVideoEditProvider` and `GrokVideoExtendProvider`, whose end-to-end VIDEO upload path was broken until now. This change wires the encode direction: * `serialization.py` gains `encode_video_input(video)` — mirrors the AUDIO encoder above. Re-encodes the `VideoInput` to an in-memory mp4/H.264 byte buffer via the upstream `comfy_api_nodes/util/conversions.video_to_base64_string` (same call shape: `video.save_to(buf, format=MP4, codec=H264)`) and base64-encodes the result. Populates `duration_s` from `video.get_duration()` so server-side providers (e.g. `_validate_video_duration_envelope` in `grok.py`) can range-check without demuxing the MP4. * `serialization.py` gains `is_video_input(value)` — duck-types on the `save_to` + `get_duration` method pair (both defined on `comfy_api.latest._input.VideoInput` and present on every concrete subclass). Avoids importing `VideoInput` at module load (which would pull torch via the `comfy_api` tree). Both helpers added to `__all__`. * `proxy_node.py:_encode_one` gains a VIDEO branch right after the AUDIO branch (and before the tensor-rank dispatch), mirroring the AUDIO pattern at the same call site. * `proxy_node.py` `_inputs_to_envelopes` docstring updates the duck-typing-rules list to mention the new VIDEO branch. Symmetric to the (still-open) MODEL_3D INPUT gap noted at `serialization.py:484`. AUDIO INPUT was already wired (ElevenLabs nodes exercise it end-to-end); IMAGE / MASK have always been wired; VIDEO INPUT closes today's last input-direction envelope gap for the existing capability vocabulary. Verified by importing `serialization` with a stubbed `comfy_remote_nodes` package: `is_video_input` returns False for None / dict / audio-shaped dict and True for an object exposing both `save_to` + `get_duration`. `encode_video_input` + the proxy_node integration are exercised end-to-end by the server-side `Wan2VideoContinuationApi_RNP` provider landing in the matching comfy-rnp-server PR (it consumes `first_clip: VIDEO` and uploads the resolved bytes to Comfy storage), and also unblocks the shipped-but-broken Grok VideoEdit/VideoExtend providers. Co-authored-by: Amp <amp@ampcode.com> Amp-Thread-ID: https://ampcode.com/threads/T-019e35d1-3c68-74ce-9918-fe9cacd74276 |
||
|
Jedrzej KosinskiandAmp
|
37b77f837a |
Parse DYNAMIC_COMBO + AUTOGROW inputs into real V3 IO objects
`proxy_node._parse_input_spec` previously had explicit branches only
for the primitive V3 input types (STRING / INT / FLOAT / BOOLEAN /
COMBO / IMAGE / VIDEO / AUDIO / MASK); every other `io_type` string —
including the two V3 dynamic input types `DYNAMIC_COMBO` and
`AUTOGROW` — fell through to the opaque `IO.Custom(io_type).Input`
bucket. Server-side runtime validation already worked (rules-
extraction at `_rules_from_input_spec` walks `__branches__` and
`__template__`), but on the frontend every Topaz / Magnific / Vidu /
Luma / Wan / Bria / Grok / ElevenLabs / HitPaw provider that declares
a `DynamicCombo` rendered as one opaque socket instead of the
branch-selector dropdown with per-branch sub-widgets the V3 IO is
designed to produce.
Refactor the parser so the construction logic is shared between two
entry points (top-level descriptor inputs use the 2-tuple
`[io_type, options]` form; DynamicCombo branch sub-inputs use the
3-tuple `[name, io_type, options]` form) and add real handlers for
both dynamic types:
* `_parse_input_spec(name, spec, optional)` — top-level entry, same
signature as before. Strips the `[io_type, options]` head and
delegates.
* `_parse_named_input_spec(spec, optional)` — branch-sub-input entry.
Strips the leading name and delegates.
* `_build_input(name, io_type, options, optional)` — shared
constructor dispatch. All primitive io_types route through the same
bag of kwargs; the new `DYNAMIC_COMBO` branch builds
`IO.DynamicCombo.Input(name, options=[IO.DynamicCombo.Option(key,
inputs=[...]), ...])` by recursing into `_parse_named_input_spec`
for each branch sub-input; the new `AUTOGROW` branch builds
`IO.Autogrow.Input(name, template=IO.Autogrow.TemplatePrefix(...))`
(or `TemplateNames`) by recursing into `_parse_input_spec` for the
template entry.
The flattened RNP wire dialect for `AUTOGROW` (`template` is a flat
`[io_type, options]` entry directly, with `prefix` / `min` / `max` or
`names` / `min` as siblings of `template`) is what every server
provider already emits (e.g. `grok.py:_grok_v2_model_subinputs`); the
parser mirrors that shape rather than the upstream
`Autogrow.Input.as_dict` nested form.
Guard rails:
* Empty branch input lists are valid (Bria moderation's `"false"`
branch carries `"inputs": []`).
* The branch selector's id is the outer input id (`"moderation"`,
`"inputs"`, etc.) — not always `"model"`.
* Branch sub-inputs are treated as required at schema-construction
time; the wire dialect doesn't express per-branch optionality, and
server-side `__branches__` enforcement covers runtime presence.
* AUTOGROW template inputs that are themselves dynamic
(`DynamicCombo.Input` / `Autogrow.Input`) are rejected at parse
time, mirroring the upstream `_AutogrowTemplate.__init__` assert.
* Malformed `DYNAMIC_COMBO` / `AUTOGROW` specs return `None` (caller
logs + skips the descriptor) instead of degrading to a single
opaque socket — the opaque fallthrough is reserved for genuinely
unknown io_type strings, not for known-but-broken dynamic specs.
* `advanced` only routes to primitive `WidgetInput` constructors; the
dynamic constructors don't accept it.
No capability gating in this PR. The two dynamic io_type strings are
already V3-canonical so a legacy client without the parser fix sees
the same opaque-socket degradation it sees today (just rendered as
`Custom("DYNAMIC_COMBO")` instead of the real DynamicCombo input).
Adding `Capability.DYNAMIC_COMBO` / `Capability.AUTOGROW` would
require pairing the cap with real descriptor-load-time enforcement
(server-side filtering of `/object_info` or client-side rejection
based on `required_capabilities`); neither exists yet, so adding
caps now would just be inert metadata. The cap pair is the right
follow-up alongside the enforcement plumbing.
Smoke test at `notes/run_dynamic_combo_parser.py` — 8 checks covering
Bria's 2-branch DynamicCombo (with empty branch), Grok V2's 3-branch
DynamicCombo with nested AUTOGROW(IMAGE), ElevenLabs' branch mixing
custom-IO (voiceN) + primitive STRING, a synthetic nested
DynamicCombo-in-DynamicCombo (defensive recursion coverage), the
malformed-spec rejection path, a regression check that
`_collect_local_validate` output is byte-identical before and after,
the AUTOGROW-rejects-dynamic-template assertion, and a source-level
check that the new helpers + branches exist.
Co-authored-by: Amp <amp@ampcode.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019e331c-c202-746b-96ec-86ce0b9c54a5
|
||
|
Jedrzej KosinskiandAmp
|
58d93daf2a |
feat(model_3d): add client-side MODEL_3D decoder + capability advertisement
Closes the loop on the just-merged comfy-rnp-server PR #37 which added the four Tripo / Rodin 3D providers and the ``Capability.MODEL_3D_GLB_INLINE`` token. Without this client-side half the providers were unreachable from any RNP client — descriptors would fail negotiation because the proxy_node didn't advertise the token. Client changes (all inside ``ComfyUI/custom_nodes/comfy_remote_nodes``): * ``protocol.py`` — mirror the new ``Capability.MODEL_3D_GLB_INLINE`` constant and add ``"model_3d"`` to ``HEAVY_TYPES`` so ``is_envelope`` recognises inbound 3D-model envelopes. The vendored ``protocol`` module is the single source of truth for the proxy_node and serialization layers; both already consume ``is_envelope`` / ``decode_envelope_data`` from here. * ``serialization.py`` — add ``decode_model3d_envelope`` that hands raw GLB bytes to ``comfy_api.latest._util.File3D`` via ``BytesIO``, mirroring the existing ``decode_video_envelope`` shape. Only ``glb_inline`` is accepted today; any other encoding surfaces ``RnpProtocolError(code=INTERNAL)`` as defence-in-depth so a future encoding never silently downgrades to garbage bytes. Registered in the ``_DECODERS`` dispatch table and re-exported via ``__all__``. * ``proxy_node.py`` — add ``"MODEL_3D" -> IO.File3DGLB.Output`` to ``_OUTPUT_CLASSES`` so descriptors with ``output=["MODEL_3D"]`` parse to a real socket. The descriptor io_type string is the logical name ``"MODEL_3D"`` (consistent across vendors) — the resulting V3 Output carries the upstream ``"FILE_3D_GLB"`` comfytype tag so frontend-side socket matching works against any GLB-aware downstream node (preview3d / SaveGLB / Load3D). * ``client.py`` — advertise ``Capability.MODEL_3D_GLB_INLINE`` in ``CLIENT_CAPABILITIES`` so servers gate provider descriptors on the presence of the token. Legacy clients that don't advertise it still see NEGOTIATION_FAILED at descriptor-load time (safe degradation), not a parse error at execute. Smoke test (``notes/run_model3d_decoder.py``) covers: 1. ``HEAVY_TYPES`` includes ``"model_3d"``. 2. ``is_envelope`` recognises a MODEL_3D envelope. 3. ``decode_model3d_envelope`` round-trips a minimal valid GLB (binary glTF 2.0, magic ``b"glTF"``). 4. ``decode_envelope`` dispatches ``type=model_3d`` to the new decoder. 5. Unsupported encoding raises ``RnpProtocolError``. 6. End-to-end against the server's ``make_model3d_envelope`` helper — server packs, client decodes, bytes match. 7. ``proxy_node._OUTPUT_CLASSES`` contract: ``"MODEL_3D" -> IO.File3DGLB.Output``. 8. ``client.CLIENT_CAPABILITIES`` advertises ``Capability.MODEL_3D_GLB_INLINE``. No server-side changes. The 4 just-shipped Tripo / Rodin providers (``TripoTextToModelApi_RNP``, ``TripoImageToModelApi_RNP``, ``TripoMultiviewToModelApi_RNP``, ``Rodin3DRegularApi_RNP``) are now reachable end-to-end from ComfyUI. Co-authored-by: Amp <amp@ampcode.com> Amp-Thread-ID: https://ampcode.com/threads/T-019e3314-4336-71d2-a333-6d1e33a9986e |
||
|
Jedrzej KosinskiandAmp
|
35b8a3b17f |
feat(serialization): tighten batch dispatch + accept response-side batch URIs
Three follow-ups identified in the PR-2 / PR-3 review pass: 1. ``serialization.maybe_externalize`` now keys batch dispatch off the ``encoding`` string (``png_base64_batch``) instead of the presence of a ``frames`` field. The old ``isinstance(frames, list) and frames`` heuristic would silently take the batch branch on a malformed envelope that mixed both fields and externalise garbage under a non-batch encoding. 2. An empty / missing ``frames`` list on a ``png_base64_batch`` envelope now raises ``RnpProtocolError(INTERNAL)`` rather than silently no-op-ing through to the wire (where the server's ``resolve_image_envelope_pngs`` rejects it after the round trip anyway). Mirrors the server-side ``make_image_batch_envelope`` contract: an empty batch is a hard wire bug. 3. ``proxy_node._deserialize_output`` now accepts the externalised ``png_base64_batch_uri`` shape on the response path. No server emits this shape today (future-proofing only — the symmetric counterpart of the request-side externalisation), but the decoder owns the symmetry: any envelope shape the client may externalise on the way out, the client must also accept on the way in. Per-frame URIs are fetched in parallel via ``asyncio.gather`` so a 9-frame batch import is one round-trip rather than N. Frame ordering is preserved (gather returns positional results) — the IMAGE tensor's batch axis carries semantic order. Fetched bytes are re-stamped as inline ``frames`` so the existing ``_decode_image_batch_envelope`` path handles them uniformly. notes/run_batch_uri_followups.py — new standalone smoke test that exercises all three changes (encoding-keyed dispatch, empty-frames hard-error, response-side batch URI parallel fetch + ordering). Amp-Thread-ID: https://ampcode.com/threads/T-019e2ce6-9403-720c-ae5b-1597b8068be7 Co-authored-by: Amp <amp@ampcode.com> |
||
|
KosinkadinkandAmp
|
6ab23718a4 |
fix(local-validate): traverse AUTOGROW templates and DYNAMIC_COMBO branches
Pre-PR-4 the descriptor-driven `image_max_batch` cap fired only on
top-level IMAGE inputs (Flux2 `images`, LTXV i2v `image`).
Providers whose cap rides a nested template — GPT Image V2 (16),
Grok ImageEditV2 (3 / 1 by branch), Grok VideoReference (7) — silently
fell through to the server's defense-in-depth path, defeating the
"no paid round trip" guarantee PR-3 promised.
## Client
* `_collect_local_validate` now recurses via the new
`_rules_from_input_spec` helper. AUTOGROW specs surface their
template's rules under a private `__template__` key; DYNAMIC_COMBO
specs surface a `__branches__` map of `{branch_key: {sub_name: rules}}`.
* `_enforce_local_validate` walks `__template__` against each
Autogrow slot value (`{"image0": <env>, "image1": <env>, ...}`) and
`__branches__` against the DynamicCombo runtime dict, dispatching
on `value[<name>]` to pick the active branch's sub-input rules.
* `image_max_batch` checking lifted into a dedicated
`_check_image_max_batch` helper so the recursive enforcer keeps a
single rule-application code path.
* Failure messages now carry a `parent.slot[.sub]` label so the
surfaced `INPUT_INVALID` says
`input 'model.images.image0' accepts at most 3 image frames…`
for nested cases instead of just the parent name.
## Docs
* No descriptor or matrix changes — the server already published the
`local_validate.image_max_batch` rules on the nested templates in
PR-3 ( eat(descriptor): declare image_max_batch local_validate caps,
comfy-rnp-server#15). The wire shape is unchanged; this PR just
makes the client honour what the server already declared.
## Verification
* Extended `notes/run_image_max_batch.py` with two new fixtures
(`_autogrow_descriptor` and `_dyncombo_with_autogrow_descriptor`)
and 9 new sub-cases covering: AUTOGROW slot over-cap fires,
AUTOGROW None-slot tolerance, AUTOGROW all-under-cap passes,
DYNAMIC_COMBO branch-A over-cap fires, DYNAMIC_COMBO branch-A
under-cap passes, DYNAMIC_COMBO branch-B (cap=1) fires the
single-frame copy, DYNAMIC_COMBO unknown branch is a defensive
no-op, plus collected-rules shape assertions.
* End-to-end check against live registered descriptors confirmed all
four nested-cap providers fire client-side: GrokImageEditNodeV2_RNP
(`model.images.image0` cap=3 and cap=1 branches),
GrokVideoReferenceNode_RNP (`model.reference_images.reference0`
cap=7), OpenAIGPTImageNodeV2_RNP (`model.images.image0` cap=16).
## Lockstep
Client-only fix. Server already publishes the rules; no paired
server PR required. The companion paper trail is comfy-rnp-server#15
(declarative caps) and the parent code-review note from the merged
PR-2/PR-3 wave that flagged this gap as the blocking follow-up.
Amp-Thread-ID: T-019e29b4-d606-7278-8584-52910661c981
Co-authored-by: Amp <amp@ampcode.com>
|
||
|
KosinkadinkandAmp
|
37a4961d6d |
feat(local-validate): fail-fast on over-cap image batches
Wire ``proxy_node._encode_inputs`` to consult each input's
``descriptor.input.<section>.<name>[1].local_validate.image_max_batch``
rule. When set, the encoder asserts ``tensor.shape[0] <= cap`` for
IMAGE inputs (rank-4 ``[B,H,W,C]`` tensors) and raises
``RnpProtocolError(code=INPUT_INVALID, user_facing=True)`` *before*
any HTTP round trip — so a workflow with N>cap reference frames
surfaces the per-provider error in the standard friendly-message
dialog instead of paying for a doomed upstream call.
## Server
(See companion PR ``Comfy-Org/comfy-rnp-server#15`` —
``feat(descriptor): declare image_max_batch local_validate caps``.)
The server-side per-call caps inside each provider's ``execute()``
stay in place as defense-in-depth; non-compliant clients still fail.
## Client
- ``proxy_node._collect_local_validate(descriptor)`` walks
``descriptor.input.{required,optional}`` and returns
``{name: rules}`` for every input that carries a non-empty
``local_validate`` block. Inputs nested inside ``AUTOGROW`` /
``DYNAMIC_COMBO`` aren't surfaced — the parser doesn't yet flatten
them into top-level inputs, so the cap declared on those templates
documents the future contract rather than firing today.
- ``proxy_node._enforce_local_validate(name, value, rules,
*, node_id)`` applies each supported rule to ``value``. Today only
``image_max_batch`` is implemented: rank-4 / rank-3-channels-last
tensors with ``shape[0] > cap`` raise ``RnpProtocolError`` with
code ``INPUT_INVALID`` and ``user_facing=True``. Friendly per-
provider message echoes the upstream wording (Flux2's "The
current maximum number of supported images is N" for cap≥2;
a "expects a single image frame" copy for cap=1 like LTXV i2v).
- ``_encode_inputs`` runs the validator before the encoder so the
raise happens before any tensor → PNG conversion or
``maybe_externalize`` upload — the cheapest possible fail point.
- Lookup is defensive at every layer: missing ``local_validate`` →
no-op, missing ``image_max_batch`` sub-key → no-op, non-tensor
input → no-op, non-image rank → no-op. Older descriptors emit
no rule and the encoder behaves identically to before.
## Docs
(Matrix prose updated on the server side — the canonical
``notes/remote-nodes-port-matrix.md`` lives in the rnp-server repo.)
## Verification
- ``notes/run_image_max_batch.py`` — new standalone smoke test
(loads the helper functions via AST extraction so the test runs
in a bare venv with just ``torch`` installed; doesn't drag in
the ComfyUI package). Asserts:
1. ``_collect_local_validate`` extracts the rule from a Flux2-
shaped descriptor.
2. A 3-frame tensor against ``cap=2`` raises ``RnpProtocolError``
with ``code=INPUT_INVALID``, ``user_facing=True``, and the
``node_id`` + counts mentioned in the message.
3. A 2-frame tensor against ``cap=2`` encodes cleanly.
4. ``cap=1`` surfaces the single-frame copy.
5. Descriptor without ``local_validate`` is a no-op.
6. ``local_validate`` block missing the ``image_max_batch`` key
is a no-op.
Run with: ``python notes/run_image_max_batch.py`` — all
assertions pass on PASS.
## Lockstep
Paired with server PR ``Comfy-Org/comfy-rnp-server#15``
(``feat/image-max-batch-rule``). Both branches were cut off
``origin/main`` before PR-2 lands; the orchestrator will rebase PR-3
onto PR-2 before merging.
Co-authored-by: Amp <amp@ampcode.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019e2adb-6ddf-72ad-88e1-8b30c57e873f
|
||
|
Jedrzej Kosinski
|
6a82114ded |
feat(serialization): input-side image:png_base64_batch envelope (#7)
feat: input-side image:png_base64_batch envelope |
||
|
KosinkadinkandAmp
|
aad9f0ba43 |
feat(serialization): input-side image:png_base64_batch envelope
Closes the silent `tensor[0]` truncation bug for multi-image inputs:
when a descriptor advertises `"png_base64_batch"` for an IMAGE input
and the inbound tensor has `shape[0] > 1`, the encoder now emits a
multi-frame envelope (`frames=[…]`) instead of dropping `tensor[1:]`.
Lockstep server change ships in `comfy-rnp-server` PR (paired branch
`feat/input-batch-envelope`).
## Client
- `serialization.py`: extend `encode_image_tensor` with an
`accepts_batch=False` keyword. When True AND `tensor.shape[0] > 1`,
emit `{"type":"image","encoding":"png_base64_batch","frames":[…],
"shape":[B,H,W,C],…}` — the same wire shape the existing
`_decode_image_batch_envelope` consumes on the response path, so
both directions round-trip symmetrically. Default-False keeps the
legacy single-frame contract for descriptors that haven't migrated.
- `proxy_node.py`: plumb the per-input `input_serialization`
allow-list from `descriptor.remote.input_serialization` through
`build_node_class` → `_execute_remote` → `_encode_inputs` →
`_encode_one`. The wire shape `{input_name: <str | list[str]>}`
normalizes to `dict[str, list[str]]` once at descriptor parse time
so the encoder can do a single `in` check per input. `_encode_one`
passes `accepts_batch=("png_base64_batch" in encodings)` to the
image encoder.
- `IMAGE_PNG_BASE64_BATCH` is already in `client.CLIENT_CAPABILITIES`
(was added when the output-side decoder shipped) so no wire-header
change.
## Verification
- Both modified files parse cleanly via `ast.parse`.
- `serialization._decode_image_batch_envelope` (output-side decoder
already in this file) consumes the same `frames=[…]` shape the
new encoder emits — round-trips symmetrically.
## Lockstep
- Paired server PR: `comfy-rnp-server` branch
`feat/input-batch-envelope` — adds `"png_base64_batch"` to
`DEFAULT_INPUT_SERIALIZATIONS["image"]`, the shared
`comfy_rnp_protocol.envelopes.resolve_image_envelope_pngs(envelope)`
helper, and migrates Flux2Pro/Max, OpenAI ChatNode + GPTImageV2,
Grok ImageEditV2 + VideoReference, and LTXV ImageToVideo
providers to consume the multi-frame envelope. Both PRs must
merge together; no back-compat between unmatched versions.
Co-authored-by: Amp <amp@ampcode.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019e29b4-d606-7278-8584-52910661c981
|
||
|
KosinkadinkandAmp
|
d67d42cd59 |
feat(custom-io): opaque pass-through custom IO bucket
Track B (opaque-custom-io) end-to-end on the proxy_node side. Adds a
generic ``IO.Custom(io_type)`` fallthrough in the parser so any
io_type string the client doesn't explicitly recognise becomes an
opaque socket with the original io_type preserved (so node-to-node
connections only chain when the strings match — an
``OPENAI_INPUT_FILES`` output cannot connect to a
``GEMINI_INPUT_FILES`` input). Pairs with the matching
``feat/opaque-custom-io`` PR on ``Comfy-Org/comfy-rnp-server`` (new
``Capability.IO_OPAQUE`` + the four deferred OpenAI / Gemini helper
node providers); both PRs must merge for the new path to light up
end-to-end.
## Client
- ``protocol.py``: mirror ``Capability.IO_OPAQUE = "io:opaque"``
from the server's ``comfy_rnp_protocol/constants.py``.
- ``client.py``: advertise ``Capability.IO_OPAQUE`` in
``CLIENT_CAPABILITIES`` so servers gating descriptors on the
token (every helper-config descriptor declares
``required_capabilities: [Capability.IO_OPAQUE]``) hand them to
this client.
- ``proxy_node._parse_input_spec``: after the explicit STRING / INT
/ FLOAT / BOOLEAN / COMBO / IMAGE / VIDEO / AUDIO / MASK branches,
any non-empty string io_type falls through to
``IO.Custom(io_type).Input(name, optional=…, tooltip=…)``. The
V3 ``@comfytype`` decorator binds the original io_type string to
the synthesised ComfyType, so the frontend's connection-validity
check enforces strict string matching between sockets.
- ``proxy_node._parse_outputs``: mirror branch — when
``_OUTPUT_CLASSES.get(io_type)`` is None, build
``IO.Custom(io_type).Output(...)`` instead of failing the whole
descriptor. Only a non-string / empty io_type still warns and
skips.
- Encoder dispatch: opaque values flow through ``_encode_one``
unchanged (already JSON-serialisable dicts/lists/strings — no
envelope wrapping). Verified against the existing fallthrough at
the end of ``_encode_one`` (returns ``value`` for non-tensor /
non-audio inputs). Output deserialisation already passes
non-envelope values through unchanged in
``_deserialize_output``.
## Docs
The proxy_node module-level docstring is updated to reflect the
new fallthrough (previously documented only the strict-skip
behaviour for unknown io_types).
## Verification
- Static AST parse of all four touched modules
(``proxy_node.py``, ``client.py``, ``serialization.py``,
``protocol.py``).
- ``Capability.IO_OPAQUE`` resolves to ``"io:opaque"`` (matches the
server's wire shape).
- ``IO.Custom("OPENAI_INPUT_FILES").Input("files", optional=True)``
and ``.Output()`` both expose ``io_type ==
"OPENAI_INPUT_FILES"``; ``IO.Custom("GEMINI_INPUT_FILES")``
similarly exposes its own io_type, so the two sockets cannot
cross-chain.
## Lockstep
Pairs with ``feat/opaque-custom-io`` on
``Comfy-Org/comfy-rnp-server`` (the new ``Capability.IO_OPAQUE``
constant lives there as the source of truth; this repo's
``protocol.py`` is the byte-compatible mirror). The server PR also
ports the four deferred helper nodes that exercise the new path
end-to-end (``OpenAIInputFiles_RNP``, ``OpenAIChatConfig_RNP``,
``GeminiInputFiles_RNP``, plus the chat node's ``files=`` /
``advanced_options=`` inputs). Until that PR merges, this client
parser fallthrough lights up only against the existing Recraft
custom IO descriptors (``RECRAFT_COLOR``, ``RECRAFT_CONTROLS``,
``RECRAFT_V3_STYLE``) — which previously caused the whole
descriptor to be skipped because the parser had no fallthrough.
Co-authored-by: Amp <amp@ampcode.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019e29b4-d606-7278-8584-52910661c981
|
||
|
KosinkadinkandAmp
|
94f995bb05 |
feat(async-polling): advertise ASYNC_EXECUTE capability and derive poll cap from duration
Final pieces of Track A (async-polling mode) on the client side. The proxy_node already routes between ``/execute`` and ``/execute_async`` on the descriptor's ``remote.execution.mode``; this commit makes the contract honest in two ways: - ``client.py``: advertise the new ``Capability.ASYNC_EXECUTE`` token so the server can gate ``async_polling`` descriptors on its presence. An older client that only knows ``/execute`` is now visible to the server and won't be silently routed to a long-poll provider whose lifecycle it can't drive. - ``protocol.py`` + ``proxy_node.py``: drop the ``DEFAULT_MAX_POLL_ATTEMPTS`` wire-default in favour of duration-based ``DEFAULT_SOFT_TIMEOUT_S`` / ``DEFAULT_HARD_TIMEOUT_S`` / ``DEFAULT_POLL_GRACE_S``. The poll-loop cap is now a *derived* value (``ceil((hard_timeout_s + grace) / poll_interval_s)``) so changing the poll cadence can never silently shrink or stretch the contract. Industry precedent: OAuth2 device flow §3.5, Kubernetes ``--timeout``, gRPC deadlines all express bounds as durations. Lockstep with the server side of Track A (comfy-rnp-server feat/async-polling-mode), which now enforces ``mode`` dispatch on ``/execute`` vs ``/execute_async`` and migrates existing providers (BFL/Bria/LTXV/Grok video/OpenAIChatNode/Stability UpscaleCreative → ``async_polling``; DALL-E/Ideogram/Recraft/sync Stability → ``request_response``). Co-authored-by: Amp <amp@ampcode.com> Amp-Thread-ID: https://ampcode.com/threads/T-019e28eb-31ae-76a1-bb45-c0922f43bcc3 |
||
|
Kosinkadink
|
cf55bd58ae |
Initial RNP/1 client implementation with Tier 3 features
ComfyUI custom-node package that fetches node descriptors from an
RNP/1 (Remote Node Protocol v1) server and exposes them as native V3
nodes inside ComfyUI.
Includes:
- Manifest + capability handshake, ETag-conditional object_info fetch
- Sync execute_remote and async _run_async_polling flows via
comfy_api_nodes.util.client.poll_op_raw
- Image / mask / audio / video envelope encode + decode
- Presigned upload helper for >8 MiB inputs
- Tier 3:
* TASK_LOST detection + idempotent resume — survives an RNP server
restart mid-task
* SERVER_BUSY / MAINTENANCE 503 envelopes treated as rate limits
with Retry-After honoring and friendly per-second progress text
* Out-of-band url_fetch for URI-only output envelopes per the
descriptor's remote.url_fetch policy
* Per-poll auth refresh so long-running tasks pick up rotated tokens
Pairs with Comfy-Org/comfy-rnp-server (private) as the reference
server implementation.
|