feat/client-model-3d-decoder
7
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Jedrzej KosinskiandAmp
|
58d93daf2a |
feat(model_3d): add client-side MODEL_3D decoder + capability advertisement
Closes the loop on the just-merged comfy-rnp-server PR #37 which added the four Tripo / Rodin 3D providers and the ``Capability.MODEL_3D_GLB_INLINE`` token. Without this client-side half the providers were unreachable from any RNP client — descriptors would fail negotiation because the proxy_node didn't advertise the token. Client changes (all inside ``ComfyUI/custom_nodes/comfy_remote_nodes``): * ``protocol.py`` — mirror the new ``Capability.MODEL_3D_GLB_INLINE`` constant and add ``"model_3d"`` to ``HEAVY_TYPES`` so ``is_envelope`` recognises inbound 3D-model envelopes. The vendored ``protocol`` module is the single source of truth for the proxy_node and serialization layers; both already consume ``is_envelope`` / ``decode_envelope_data`` from here. * ``serialization.py`` — add ``decode_model3d_envelope`` that hands raw GLB bytes to ``comfy_api.latest._util.File3D`` via ``BytesIO``, mirroring the existing ``decode_video_envelope`` shape. Only ``glb_inline`` is accepted today; any other encoding surfaces ``RnpProtocolError(code=INTERNAL)`` as defence-in-depth so a future encoding never silently downgrades to garbage bytes. Registered in the ``_DECODERS`` dispatch table and re-exported via ``__all__``. * ``proxy_node.py`` — add ``"MODEL_3D" -> IO.File3DGLB.Output`` to ``_OUTPUT_CLASSES`` so descriptors with ``output=["MODEL_3D"]`` parse to a real socket. The descriptor io_type string is the logical name ``"MODEL_3D"`` (consistent across vendors) — the resulting V3 Output carries the upstream ``"FILE_3D_GLB"`` comfytype tag so frontend-side socket matching works against any GLB-aware downstream node (preview3d / SaveGLB / Load3D). * ``client.py`` — advertise ``Capability.MODEL_3D_GLB_INLINE`` in ``CLIENT_CAPABILITIES`` so servers gate provider descriptors on the presence of the token. Legacy clients that don't advertise it still see NEGOTIATION_FAILED at descriptor-load time (safe degradation), not a parse error at execute. Smoke test (``notes/run_model3d_decoder.py``) covers: 1. ``HEAVY_TYPES`` includes ``"model_3d"``. 2. ``is_envelope`` recognises a MODEL_3D envelope. 3. ``decode_model3d_envelope`` round-trips a minimal valid GLB (binary glTF 2.0, magic ``b"glTF"``). 4. ``decode_envelope`` dispatches ``type=model_3d`` to the new decoder. 5. Unsupported encoding raises ``RnpProtocolError``. 6. End-to-end against the server's ``make_model3d_envelope`` helper — server packs, client decodes, bytes match. 7. ``proxy_node._OUTPUT_CLASSES`` contract: ``"MODEL_3D" -> IO.File3DGLB.Output``. 8. ``client.CLIENT_CAPABILITIES`` advertises ``Capability.MODEL_3D_GLB_INLINE``. No server-side changes. The 4 just-shipped Tripo / Rodin providers (``TripoTextToModelApi_RNP``, ``TripoImageToModelApi_RNP``, ``TripoMultiviewToModelApi_RNP``, ``Rodin3DRegularApi_RNP``) are now reachable end-to-end from ComfyUI. Co-authored-by: Amp <amp@ampcode.com> Amp-Thread-ID: https://ampcode.com/threads/T-019e3314-4336-71d2-a333-6d1e33a9986e |
||
|
Jedrzej KosinskiandAmp
|
cd3e051956 |
feat(audio): add Capability.AUDIO_OPUS_BASE64 + decode_audio_envelope branch
Pairs with comfy-rnp-server feat/audio-opus-base64. Lets the proxy_node accept and decode `audio:opus_base64` envelopes (OGG/ Opus container as base64) so audio-out partner-node providers whose upstream returns OGG/Opus bytes (e.g. ElevenLabs `output_format=opus_48000_192`) can ship an honest envelope instead of label-laundering the bytes as `mp3_base64`. protocol.py: - `Capability.AUDIO_OPUS_BASE64 = "audio:opus_base64"`. client.py: - `CLIENT_CAPABILITIES` now includes `Capability.AUDIO_OPUS_BASE64`, matching the AUDIO sweep precedent (mp3 + wav already in the list). serialization.py: - `decode_audio_envelope` allow-list extended with `opus_base64`. PyAV/FFmpeg already decode OGG/Opus natively in `audio_bytes_to_audio_input`, so the decoder branch is a one-line allow-list change — same path as MP3/WAV, just a different container. The wire-format hint is now honest rather than label- laundering, since the inbound envelope's `encoding` matches the actual container. The capability negotiation is symmetric to the existing `image:png_base64_batch` / `audio:mp3_base64` negotiation: servers gate emission on the inbound advertise list, and the per-format fallback (mp3) is required while opus is opportunistic. Amp-Thread-ID: https://ampcode.com/threads/T-019e2e33-da42-74c2-a6cf-8c7942cb8180 Co-authored-by: Amp <amp@ampcode.com> |
||
|
Jedrzej KosinskiandAmp
|
35b8a3b17f |
feat(serialization): tighten batch dispatch + accept response-side batch URIs
Three follow-ups identified in the PR-2 / PR-3 review pass: 1. ``serialization.maybe_externalize`` now keys batch dispatch off the ``encoding`` string (``png_base64_batch``) instead of the presence of a ``frames`` field. The old ``isinstance(frames, list) and frames`` heuristic would silently take the batch branch on a malformed envelope that mixed both fields and externalise garbage under a non-batch encoding. 2. An empty / missing ``frames`` list on a ``png_base64_batch`` envelope now raises ``RnpProtocolError(INTERNAL)`` rather than silently no-op-ing through to the wire (where the server's ``resolve_image_envelope_pngs`` rejects it after the round trip anyway). Mirrors the server-side ``make_image_batch_envelope`` contract: an empty batch is a hard wire bug. 3. ``proxy_node._deserialize_output`` now accepts the externalised ``png_base64_batch_uri`` shape on the response path. No server emits this shape today (future-proofing only — the symmetric counterpart of the request-side externalisation), but the decoder owns the symmetry: any envelope shape the client may externalise on the way out, the client must also accept on the way in. Per-frame URIs are fetched in parallel via ``asyncio.gather`` so a 9-frame batch import is one round-trip rather than N. Frame ordering is preserved (gather returns positional results) — the IMAGE tensor's batch axis carries semantic order. Fetched bytes are re-stamped as inline ``frames`` so the existing ``_decode_image_batch_envelope`` path handles them uniformly. notes/run_batch_uri_followups.py — new standalone smoke test that exercises all three changes (encoding-keyed dispatch, empty-frames hard-error, response-side batch URI parallel fetch + ordering). Amp-Thread-ID: https://ampcode.com/threads/T-019e2ce6-9403-720c-ae5b-1597b8068be7 Co-authored-by: Amp <amp@ampcode.com> |
||
|
KosinkadinkandAmp
|
6433a46c93 |
feat(serialization): externalize oversize image batch envelopes via per-frame presigned URIs
The pre-existing `serialization.maybe_externalize` only inspected the
singleton `data` field, so a multi-frame `frames=[…]` IMAGE envelope
whose summed base64 payload crossed the 8 MiB
`DEFAULT_MAX_INLINE_PAYLOAD_BYTES` cap silently rode the wire as a
30+ MiB POST body — defeating the entire reason the cap exists. A
9× 2048² Flux2 reference batch hits ~200 MB base64 inline; that's
the worst-case the new path is designed for.
This commit adds the **client side** of the lockstep fix: detect
oversize multi-frame batches (sum of per-frame raw bytes), upload
each frame as an individual presigned PUT in parallel via
`asyncio.gather`, and rewrite the envelope to the new
`image:png_base64_batch_uri` encoding with a parallel `uris` list.
Server side closes the loop with a matching encoding-string
dispatch in `resolve_image_envelope_pngs`.
Per-frame URIs (NOT a tar/zip blob) compose with the existing
per-frame fetch helpers and let the server fetch in parallel.
## Server
- See lockstep PR on `Comfy-Org/comfy-rnp-server` branch
`feat/batch-envelope-externalization` for the
`Capability.IMAGE_PNG_BASE64_BATCH_URI` token, the
`_resolve_png_base64_batch_uri` per-frame fetch helper, and the
encoding-string dispatch in `resolve_image_envelope_pngs`.
## Client
- `protocol.py`: mirror new `Capability.IMAGE_PNG_BASE64_BATCH_URI =
"image:png_base64_batch_uri"` token next to
`IMAGE_PNG_BASE64_BATCH`. Same comment block as the server-side
copy so the two stay byte-comparable.
- `client.py`: append `Capability.IMAGE_PNG_BASE64_BATCH_URI` to
`CLIENT_CAPABILITIES` so the server knows this client can emit
the externalised batch shape. Comment notes the opportunistic
posture (any input that already accepts `png_base64_batch`
transparently accepts the externalised variant — no descriptor
opt-in needed).
- `serialization.py`:
- `maybe_externalize` now branches on payload shape: if
`envelope["frames"]` is a populated list, dispatch to the new
`_maybe_externalize_batch` helper; otherwise fall through to
the existing singleton-`data` path. Docstring expanded to
describe both shapes and the encoding-flip-on-spill behaviour.
- New `_maybe_externalize_batch` helper sums per-frame raw bytes
(`len(b64) * 3 // 4`), short-circuits when the batch fits under
the cap, and otherwise uploads every frame in parallel via
`asyncio.gather` (preserves frame ordering — critical because
Flux2 maps `uris[k]` → `input_image_{k+1}` 1:1) before
rewriting the envelope to `{type, encoding:
png_base64_batch_uri, uris: [...], shape, dtype, color_space}`
with `frames` stripped.
- `import asyncio` added at module top so `asyncio.gather` is
available without per-call imports.
## Docs
- The pilot side-by-side workflow `notes/sxs_flux2_pro_multi_ref_xl.json`
(9× 2048² references) and the closed-gap entry in
`notes/remote-nodes-vocabulary-gaps.md` live in the server
worktree (this client repo has no `notes/` directory). Both ship
in the paired server PR.
## Verification
- AST parse of `protocol.py`, `serialization.py`, `client.py`
succeeds.
- In-process round-trip test (with a fake `comfy_api_nodes.util.client`
shim and a stubbed `upload_asset`):
* Under-cap batch envelope (1× 5 MiB raw, cap=8 MiB) → passes
through unchanged (object identity preserved).
* Over-cap batch envelope (3× 5 MiB raw, cap=8 MiB) → triggers
3 PUTs via the stub, returns
`{type: image, encoding: png_base64_batch_uri,
uris: ['…/asset/1','…/asset/2','…/asset/3'], shape: [3,1,1,3],
dtype: uint8, color_space: srgb}` with `frames` stripped.
* Singleton `data` envelope (10 MiB raw, cap=8 MiB) → still gets
the legacy `data → uri` swap (existing path unaffected).
* `Capability.IMAGE_PNG_BASE64_BATCH_URI in CLIENT_CAPABILITIES`
holds.
## Lockstep
- Paired server PR: https://github.com/Comfy-Org/comfy-rnp-server/pull/14
(`feat/batch-envelope-externalization`). Server PR was opened
first because the wire shape needs a server resolver to be
meaningful — the orchestrator merges them in lockstep order.
- Capability gating posture: `IMAGE_PNG_BASE64_BATCH_URI` is
opportunistic — this client emits it whenever a batch envelope it
produced overflows the inline cap, regardless of whether the
descriptor's `input_serialization` lists it. Server accepts on
any image input that already accepts `png_base64_batch` via the
shared `resolve_image_envelope_pngs` dispatch.
- Output-side decoder: server providers emit `png_base64_batch` (or
single-frame `png_base64`) on responses today; the URI shape is
consumed on input only, so the client decoder (`_DECODERS` /
`_decode_image_batch_envelope`) is unchanged. The existing
`decode_image_envelope` already dispatches on the `encoding`
string (not field presence), so adding a `png_base64_batch_uri`
decode branch is a drop-in if a future provider starts emitting
the URI shape on outputs. Logged as a known follow-up.
Amp-Thread-ID: T-019e29b4-d606-7278-8584-52910661c981
Co-authored-by: Amp <amp@ampcode.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019e2adb-6ddf-72ad-88e1-8b30c57e873f
Co-authored-by: Amp <amp@ampcode.com>
|
||
|
KosinkadinkandAmp
|
aad9f0ba43 |
feat(serialization): input-side image:png_base64_batch envelope
Closes the silent `tensor[0]` truncation bug for multi-image inputs:
when a descriptor advertises `"png_base64_batch"` for an IMAGE input
and the inbound tensor has `shape[0] > 1`, the encoder now emits a
multi-frame envelope (`frames=[…]`) instead of dropping `tensor[1:]`.
Lockstep server change ships in `comfy-rnp-server` PR (paired branch
`feat/input-batch-envelope`).
## Client
- `serialization.py`: extend `encode_image_tensor` with an
`accepts_batch=False` keyword. When True AND `tensor.shape[0] > 1`,
emit `{"type":"image","encoding":"png_base64_batch","frames":[…],
"shape":[B,H,W,C],…}` — the same wire shape the existing
`_decode_image_batch_envelope` consumes on the response path, so
both directions round-trip symmetrically. Default-False keeps the
legacy single-frame contract for descriptors that haven't migrated.
- `proxy_node.py`: plumb the per-input `input_serialization`
allow-list from `descriptor.remote.input_serialization` through
`build_node_class` → `_execute_remote` → `_encode_inputs` →
`_encode_one`. The wire shape `{input_name: <str | list[str]>}`
normalizes to `dict[str, list[str]]` once at descriptor parse time
so the encoder can do a single `in` check per input. `_encode_one`
passes `accepts_batch=("png_base64_batch" in encodings)` to the
image encoder.
- `IMAGE_PNG_BASE64_BATCH` is already in `client.CLIENT_CAPABILITIES`
(was added when the output-side decoder shipped) so no wire-header
change.
## Verification
- Both modified files parse cleanly via `ast.parse`.
- `serialization._decode_image_batch_envelope` (output-side decoder
already in this file) consumes the same `frames=[…]` shape the
new encoder emits — round-trips symmetrically.
## Lockstep
- Paired server PR: `comfy-rnp-server` branch
`feat/input-batch-envelope` — adds `"png_base64_batch"` to
`DEFAULT_INPUT_SERIALIZATIONS["image"]`, the shared
`comfy_rnp_protocol.envelopes.resolve_image_envelope_pngs(envelope)`
helper, and migrates Flux2Pro/Max, OpenAI ChatNode + GPTImageV2,
Grok ImageEditV2 + VideoReference, and LTXV ImageToVideo
providers to consume the multi-frame envelope. Both PRs must
merge together; no back-compat between unmatched versions.
Co-authored-by: Amp <amp@ampcode.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019e29b4-d606-7278-8584-52910661c981
|
||
|
Jedrzej KosinskiandAmp
|
ed63c5ae49 |
feat(serialization): decode image:png_base64_batch envelope
Mirrors the server-side change in comfy-rnp-server PR #7. Adds the client-side half of the multi-image envelope contract: - Capability.IMAGE_PNG_BASE64_BATCH (\image:png_base64_batch\) added to protocol.Capability, mirroring comfy_rnp_protocol/constants.py. - decode_image_envelope() now dispatches png_base64_batch to a new _decode_image_batch_envelope() helper that stacks each PNG frame into a [B, H, W, C] tensor — the native ComfyUI image-batch shape — so providers whose upstream returns n>1 surface as a real batched tensor in the workflow runtime instead of being silently truncated to the first frame. - Frame-count integrity check: if the envelope declares shape[0]=B but \ rames\ has a different length, fail hard rather than drop frames silently. Matches the SVG decoder's \count\ check. - Per-frame H/W mismatch is also a hard error — the server-side encoder enforces uniform sizing already, so a mismatch on the wire indicates a wire bug worth surfacing. - IMAGE_PNG_BASE64_BATCH added to client.CLIENT_CAPABILITIES so the server's per-request negotiator (\client_advertises_capability\ in comfy_rnp_protocol.capabilities) lights up the batch path. Single-frame envelopes (\png_base64\ / \jpeg_base64\ / \webp_base64\) remain unchanged — back-compat with every already-merged provider. Verified end-to-end with a synthetic 2-frame envelope (white + black 32x64 PNGs) plus the mismatch / empty-frames error paths. Co-authored-by: Amp <amp@ampcode.com> Amp-Thread-ID: https://ampcode.com/threads/T-019e24d1-250c-750a-ab00-2eb5e9cb2500 |
||
|
Kosinkadink
|
cf55bd58ae |
Initial RNP/1 client implementation with Tier 3 features
ComfyUI custom-node package that fetches node descriptors from an
RNP/1 (Remote Node Protocol v1) server and exposes them as native V3
nodes inside ComfyUI.
Includes:
- Manifest + capability handshake, ETag-conditional object_info fetch
- Sync execute_remote and async _run_async_polling flows via
comfy_api_nodes.util.client.poll_op_raw
- Image / mask / audio / video envelope encode + decode
- Presigned upload helper for >8 MiB inputs
- Tier 3:
* TASK_LOST detection + idempotent resume — survives an RNP server
restart mid-task
* SERVER_BUSY / MAINTENANCE 503 envelopes treated as rate limits
with Retry-After honoring and friendly per-second progress text
* Out-of-band url_fetch for URI-only output envelopes per the
descriptor's remote.url_fetch policy
* Per-poll auth refresh so long-running tasks pick up rotated tokens
Pairs with Comfy-Org/comfy-rnp-server (private) as the reference
server implementation.
|