feat/client-model-3d-decoder
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Jedrzej KosinskiandAmp
|
58d93daf2a |
feat(model_3d): add client-side MODEL_3D decoder + capability advertisement
Closes the loop on the just-merged comfy-rnp-server PR #37 which added the four Tripo / Rodin 3D providers and the ``Capability.MODEL_3D_GLB_INLINE`` token. Without this client-side half the providers were unreachable from any RNP client — descriptors would fail negotiation because the proxy_node didn't advertise the token. Client changes (all inside ``ComfyUI/custom_nodes/comfy_remote_nodes``): * ``protocol.py`` — mirror the new ``Capability.MODEL_3D_GLB_INLINE`` constant and add ``"model_3d"`` to ``HEAVY_TYPES`` so ``is_envelope`` recognises inbound 3D-model envelopes. The vendored ``protocol`` module is the single source of truth for the proxy_node and serialization layers; both already consume ``is_envelope`` / ``decode_envelope_data`` from here. * ``serialization.py`` — add ``decode_model3d_envelope`` that hands raw GLB bytes to ``comfy_api.latest._util.File3D`` via ``BytesIO``, mirroring the existing ``decode_video_envelope`` shape. Only ``glb_inline`` is accepted today; any other encoding surfaces ``RnpProtocolError(code=INTERNAL)`` as defence-in-depth so a future encoding never silently downgrades to garbage bytes. Registered in the ``_DECODERS`` dispatch table and re-exported via ``__all__``. * ``proxy_node.py`` — add ``"MODEL_3D" -> IO.File3DGLB.Output`` to ``_OUTPUT_CLASSES`` so descriptors with ``output=["MODEL_3D"]`` parse to a real socket. The descriptor io_type string is the logical name ``"MODEL_3D"`` (consistent across vendors) — the resulting V3 Output carries the upstream ``"FILE_3D_GLB"`` comfytype tag so frontend-side socket matching works against any GLB-aware downstream node (preview3d / SaveGLB / Load3D). * ``client.py`` — advertise ``Capability.MODEL_3D_GLB_INLINE`` in ``CLIENT_CAPABILITIES`` so servers gate provider descriptors on the presence of the token. Legacy clients that don't advertise it still see NEGOTIATION_FAILED at descriptor-load time (safe degradation), not a parse error at execute. Smoke test (``notes/run_model3d_decoder.py``) covers: 1. ``HEAVY_TYPES`` includes ``"model_3d"``. 2. ``is_envelope`` recognises a MODEL_3D envelope. 3. ``decode_model3d_envelope`` round-trips a minimal valid GLB (binary glTF 2.0, magic ``b"glTF"``). 4. ``decode_envelope`` dispatches ``type=model_3d`` to the new decoder. 5. Unsupported encoding raises ``RnpProtocolError``. 6. End-to-end against the server's ``make_model3d_envelope`` helper — server packs, client decodes, bytes match. 7. ``proxy_node._OUTPUT_CLASSES`` contract: ``"MODEL_3D" -> IO.File3DGLB.Output``. 8. ``client.CLIENT_CAPABILITIES`` advertises ``Capability.MODEL_3D_GLB_INLINE``. No server-side changes. The 4 just-shipped Tripo / Rodin providers (``TripoTextToModelApi_RNP``, ``TripoImageToModelApi_RNP``, ``TripoMultiviewToModelApi_RNP``, ``Rodin3DRegularApi_RNP``) are now reachable end-to-end from ComfyUI. Co-authored-by: Amp <amp@ampcode.com> Amp-Thread-ID: https://ampcode.com/threads/T-019e3314-4336-71d2-a333-6d1e33a9986e |
||
|
Jedrzej KosinskiandAmp
|
35b8a3b17f |
feat(serialization): tighten batch dispatch + accept response-side batch URIs
Three follow-ups identified in the PR-2 / PR-3 review pass: 1. ``serialization.maybe_externalize`` now keys batch dispatch off the ``encoding`` string (``png_base64_batch``) instead of the presence of a ``frames`` field. The old ``isinstance(frames, list) and frames`` heuristic would silently take the batch branch on a malformed envelope that mixed both fields and externalise garbage under a non-batch encoding. 2. An empty / missing ``frames`` list on a ``png_base64_batch`` envelope now raises ``RnpProtocolError(INTERNAL)`` rather than silently no-op-ing through to the wire (where the server's ``resolve_image_envelope_pngs`` rejects it after the round trip anyway). Mirrors the server-side ``make_image_batch_envelope`` contract: an empty batch is a hard wire bug. 3. ``proxy_node._deserialize_output`` now accepts the externalised ``png_base64_batch_uri`` shape on the response path. No server emits this shape today (future-proofing only — the symmetric counterpart of the request-side externalisation), but the decoder owns the symmetry: any envelope shape the client may externalise on the way out, the client must also accept on the way in. Per-frame URIs are fetched in parallel via ``asyncio.gather`` so a 9-frame batch import is one round-trip rather than N. Frame ordering is preserved (gather returns positional results) — the IMAGE tensor's batch axis carries semantic order. Fetched bytes are re-stamped as inline ``frames`` so the existing ``_decode_image_batch_envelope`` path handles them uniformly. notes/run_batch_uri_followups.py — new standalone smoke test that exercises all three changes (encoding-keyed dispatch, empty-frames hard-error, response-side batch URI parallel fetch + ordering). Amp-Thread-ID: https://ampcode.com/threads/T-019e2ce6-9403-720c-ae5b-1597b8068be7 Co-authored-by: Amp <amp@ampcode.com> |
||
|
KosinkadinkandAmp
|
6ab23718a4 |
fix(local-validate): traverse AUTOGROW templates and DYNAMIC_COMBO branches
Pre-PR-4 the descriptor-driven `image_max_batch` cap fired only on
top-level IMAGE inputs (Flux2 `images`, LTXV i2v `image`).
Providers whose cap rides a nested template — GPT Image V2 (16),
Grok ImageEditV2 (3 / 1 by branch), Grok VideoReference (7) — silently
fell through to the server's defense-in-depth path, defeating the
"no paid round trip" guarantee PR-3 promised.
## Client
* `_collect_local_validate` now recurses via the new
`_rules_from_input_spec` helper. AUTOGROW specs surface their
template's rules under a private `__template__` key; DYNAMIC_COMBO
specs surface a `__branches__` map of `{branch_key: {sub_name: rules}}`.
* `_enforce_local_validate` walks `__template__` against each
Autogrow slot value (`{"image0": <env>, "image1": <env>, ...}`) and
`__branches__` against the DynamicCombo runtime dict, dispatching
on `value[<name>]` to pick the active branch's sub-input rules.
* `image_max_batch` checking lifted into a dedicated
`_check_image_max_batch` helper so the recursive enforcer keeps a
single rule-application code path.
* Failure messages now carry a `parent.slot[.sub]` label so the
surfaced `INPUT_INVALID` says
`input 'model.images.image0' accepts at most 3 image frames…`
for nested cases instead of just the parent name.
## Docs
* No descriptor or matrix changes — the server already published the
`local_validate.image_max_batch` rules on the nested templates in
PR-3 ( eat(descriptor): declare image_max_batch local_validate caps,
comfy-rnp-server#15). The wire shape is unchanged; this PR just
makes the client honour what the server already declared.
## Verification
* Extended `notes/run_image_max_batch.py` with two new fixtures
(`_autogrow_descriptor` and `_dyncombo_with_autogrow_descriptor`)
and 9 new sub-cases covering: AUTOGROW slot over-cap fires,
AUTOGROW None-slot tolerance, AUTOGROW all-under-cap passes,
DYNAMIC_COMBO branch-A over-cap fires, DYNAMIC_COMBO branch-A
under-cap passes, DYNAMIC_COMBO branch-B (cap=1) fires the
single-frame copy, DYNAMIC_COMBO unknown branch is a defensive
no-op, plus collected-rules shape assertions.
* End-to-end check against live registered descriptors confirmed all
four nested-cap providers fire client-side: GrokImageEditNodeV2_RNP
(`model.images.image0` cap=3 and cap=1 branches),
GrokVideoReferenceNode_RNP (`model.reference_images.reference0`
cap=7), OpenAIGPTImageNodeV2_RNP (`model.images.image0` cap=16).
## Lockstep
Client-only fix. Server already publishes the rules; no paired
server PR required. The companion paper trail is comfy-rnp-server#15
(declarative caps) and the parent code-review note from the merged
PR-2/PR-3 wave that flagged this gap as the blocking follow-up.
Amp-Thread-ID: T-019e29b4-d606-7278-8584-52910661c981
Co-authored-by: Amp <amp@ampcode.com>
|
||
|
KosinkadinkandAmp
|
37a4961d6d |
feat(local-validate): fail-fast on over-cap image batches
Wire ``proxy_node._encode_inputs`` to consult each input's
``descriptor.input.<section>.<name>[1].local_validate.image_max_batch``
rule. When set, the encoder asserts ``tensor.shape[0] <= cap`` for
IMAGE inputs (rank-4 ``[B,H,W,C]`` tensors) and raises
``RnpProtocolError(code=INPUT_INVALID, user_facing=True)`` *before*
any HTTP round trip — so a workflow with N>cap reference frames
surfaces the per-provider error in the standard friendly-message
dialog instead of paying for a doomed upstream call.
## Server
(See companion PR ``Comfy-Org/comfy-rnp-server#15`` —
``feat(descriptor): declare image_max_batch local_validate caps``.)
The server-side per-call caps inside each provider's ``execute()``
stay in place as defense-in-depth; non-compliant clients still fail.
## Client
- ``proxy_node._collect_local_validate(descriptor)`` walks
``descriptor.input.{required,optional}`` and returns
``{name: rules}`` for every input that carries a non-empty
``local_validate`` block. Inputs nested inside ``AUTOGROW`` /
``DYNAMIC_COMBO`` aren't surfaced — the parser doesn't yet flatten
them into top-level inputs, so the cap declared on those templates
documents the future contract rather than firing today.
- ``proxy_node._enforce_local_validate(name, value, rules,
*, node_id)`` applies each supported rule to ``value``. Today only
``image_max_batch`` is implemented: rank-4 / rank-3-channels-last
tensors with ``shape[0] > cap`` raise ``RnpProtocolError`` with
code ``INPUT_INVALID`` and ``user_facing=True``. Friendly per-
provider message echoes the upstream wording (Flux2's "The
current maximum number of supported images is N" for cap≥2;
a "expects a single image frame" copy for cap=1 like LTXV i2v).
- ``_encode_inputs`` runs the validator before the encoder so the
raise happens before any tensor → PNG conversion or
``maybe_externalize`` upload — the cheapest possible fail point.
- Lookup is defensive at every layer: missing ``local_validate`` →
no-op, missing ``image_max_batch`` sub-key → no-op, non-tensor
input → no-op, non-image rank → no-op. Older descriptors emit
no rule and the encoder behaves identically to before.
## Docs
(Matrix prose updated on the server side — the canonical
``notes/remote-nodes-port-matrix.md`` lives in the rnp-server repo.)
## Verification
- ``notes/run_image_max_batch.py`` — new standalone smoke test
(loads the helper functions via AST extraction so the test runs
in a bare venv with just ``torch`` installed; doesn't drag in
the ComfyUI package). Asserts:
1. ``_collect_local_validate`` extracts the rule from a Flux2-
shaped descriptor.
2. A 3-frame tensor against ``cap=2`` raises ``RnpProtocolError``
with ``code=INPUT_INVALID``, ``user_facing=True``, and the
``node_id`` + counts mentioned in the message.
3. A 2-frame tensor against ``cap=2`` encodes cleanly.
4. ``cap=1`` surfaces the single-frame copy.
5. Descriptor without ``local_validate`` is a no-op.
6. ``local_validate`` block missing the ``image_max_batch`` key
is a no-op.
Run with: ``python notes/run_image_max_batch.py`` — all
assertions pass on PASS.
## Lockstep
Paired with server PR ``Comfy-Org/comfy-rnp-server#15``
(``feat/image-max-batch-rule``). Both branches were cut off
``origin/main`` before PR-2 lands; the orchestrator will rebase PR-3
onto PR-2 before merging.
Co-authored-by: Amp <amp@ampcode.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019e2adb-6ddf-72ad-88e1-8b30c57e873f
|