Compare commits

...
Author SHA1 Message Date
Jedrzej Kosinski 0946b83b21 Merge pull request #22 from Comfy-Org/feat/encode-nested-autogrow-dicts
fix(encoder): recurse into nested AUTOGROW / DYNAMIC_COMBO runtime dicts
2026-05-17 08:22:55 -07:00
Jedrzej KosinskiandAmp 866b7e32a3 fix(encoder): recurse into nested AUTOGROW / DYNAMIC_COMBO runtime dicts
ComfyUI's V3 ``build_nested_inputs`` (comfy_api/latest/_io.py L1722) rebuilds
AUTOGROW / DYNAMIC_COMBO runtime kwargs into nested dicts before they reach
the proxy_node's ``execute()``. For an AUTOGROW input named
``reference_images``, the value arrives as ``{"image1": <tensor>,
"image2": <tensor>, ...}``; for a DYNAMIC_COMBO branch with a nested
AUTOGROW (e.g. Wan2ReferenceVideoApi's ``model`` DC with nested
``video1..video3`` + ``image1..image5``), the value arrives as
``{"<dc-name>": branch_key, "video_refs": {"video1": <VideoInput>},
"reference_images": {"image1": <tensor>}, ...}``.

Before this change ``_encode_one`` only encoded values where the **top-
level** value was itself a torch.Tensor / VideoInput / audio dict /
model3d input. A non-audio dict (the AG / DC runtime shape) failed all
four duck-type checks and was returned unchanged, so per-slot tensors /
VideoInputs survived past encoding and either crashed at
``json.dumps`` time (no custom JSONEncoder for torch.Tensor) or
violated the server's contract (comfy_rnp_server's
``_ordered_wan_videoedit_image_envelopes`` and the WAN2 / HappyHorse /
Grok / Bria / Luma AG providers shipped over the last 5 PRs all
require each slot value to already be an RNP envelope before they
reach the upload step).

The same gap also defeated the externalization cap: ``_encode_inputs``
only ran ``maybe_externalize`` when the *top-level* encoded value
was itself an envelope, so per-slot envelopes inside an AG dict would
ship inline regardless of ``max_inline_bytes``.

Fix is structural — match the runtime data model:

1. ``_encode_one`` recurses into plain (non-envelope, non-audio) dict
   values, calling itself for each entry. Envelope dicts pass through
   so we don't double-encode; audio dicts still route to
   ``encode_audio_input`` because their dict shape (waveform +
   sample_rate) is a heavy leaf, not a container. The recursion
   passes the parent's ``accepted_encodings`` downward — single-image
   AG slots and VIDEO / AUDIO / MODEL_3D leaves don't depend on it,
   and gating per-child policy on descriptor metadata adds plumbing
   for no concrete provider need today.

2. New ``_externalize_nested`` helper walks the encoded tree the same
   way: envelope leaves go through ``serialization.maybe_externalize``;
   plain dicts recurse; everything else passes through. ``_encode_inputs``
   replaces the old "if is_envelope: externalize" guard with one call
   to the recursive walker, so per-slot envelopes inside AG / DC dicts
   get the same presigned-PUT treatment top-level envelopes get.

Lists are intentionally NOT traversed — AG / DC runtime values always
materialize as dicts, never lists, and recursing into arbitrary lists
would touch opaque payloads. Recursion depth is bounded by the
upstream Autogrow max-slot cap (100) and the maximum DC branch
nesting depth observed in any provider today (1), so an in-place
``async def`` walk is well-bounded.

Tests:

* New ``notes/run_encode_nested_dicts.py`` standalone smoke test
  exercises 9 scenarios: top-level AG-of-IMAGE, top-level AG-of-VIDEO,
  DC-wrapping-AG with branch key untouched, AUDIO dict still routes to
  encode_audio_input, pre-encoded envelope passes through unchanged,
  scalar config dict passes through unchanged, ``_externalize_nested``
  walks per-slot envelopes (cap=3 forces every slot to externalize),
  scalar siblings preserved alongside externalized envelopes, full
  encoded payload is JSON-serializable (regression guard for the
  exact bug fixed), and single-tensor / single-video / single-audio
  top-level inputs still encode unchanged (no leaf-path regression).
* Loads ``_encode_one`` / ``_encode_inputs`` / ``_externalize_nested``
  out of proxy_node.py via AST extraction so it runs without
  importing the full comfy_api / comfy_api_nodes packages, same
  pattern as ``run_image_max_batch.py``.

Unblocks the next batch of comfy-rnp-server providers that need
AG-of-VIDEO or DC-wrapping-AG-of-VIDEO support (WanReferenceVideoApi
``character1..character3`` and Wan2ReferenceVideoApi
``video1..video3`` + ``image1..image5``).

Co-authored-by: Amp <amp@ampcode.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019e367d-c5b5-758d-9387-e44477ef4cb5
2026-05-17 08:22:05 -07:00
Jedrzej Kosinski 1d70ef223b Merge pull request #21 from Comfy-Org/feat/model3d-input-multitype-mapping
Map MODEL_3D input type to MultiType socket (mirror upstream union)
2026-05-17 06:57:47 -07:00
Jedrzej KosinskiandAmp 197e7ba406 Map "MODEL_3D" input type to MultiType socket (mirror upstream union)
Follow-up to PR #20: the `MODEL_3D` input wire type now resolves to
an `IO.MultiType.Input(types=[IO.File3DGLB, IO.File3DOBJ,
IO.File3DFBX, IO.File3DAny])` socket — symmetric counterpart of the
existing `"MODEL_3D"` entry in `_OUTPUT_CLASSES` that maps to
`IO.File3DGLB.Output`.

Previously the descriptor wire type `"MODEL_3D"` fell through to the
opaque `IO.Custom("MODEL_3D").Input(...)` socket, which is a
single-comfytype socket the frontend only allows connections from
identically-tagged outputs. Real upstream 3D sources expose
different comfytypes:

- `Load3D` outputs `IO.File3DAny` (comfytype `FILE_3D`).
- Meshy / Hunyuan3D / Tripo / Rodin partner nodes output
  `IO.File3DGLB` (comfytype `FILE_3D_GLB`) via the existing
  `_OUTPUT_CLASSES["MODEL_3D"]` mapping.
- Meshy's dual `FILE_3D_FBX` output socket emits `FILE_3D_FBX`.

None of these connect to `IO.Custom("MODEL_3D")`. The upstream
Tencent Hunyuan3D MODEL_3D-input nodes (`TencentModelTo3DUVNode` /
`Tencent3DTextureEditNode` / `Tencent3DPartNode` /
`TencentSmartTopologyNode`) all use `IO.MultiType.Input(types=[...])`
to accept the union; mirroring that union here keeps the proxy
socket compatible with every existing 3D-emitting node.

`get_io_type()` on `IO.MultiType.Input` returns the comma-joined
inner type IDs (e.g. `"FILE_3D_GLB,FILE_3D_OBJ,FILE_3D_FBX,FILE_3D"`)
— matches the convention SxS workflows already use for their
`SaveGLB` connector inputs.

Per-node format validation (e.g. 3DTextureEdit's FBX-only
enforcement, 3DPart's FBX-only enforcement) is done server-side in
each provider's `execute()` against the inline envelope's `format`
extra; this socket-level filter is purely about frontend connection
validity.

Unblocks the paired comfy-rnp-server PR (`TencentModelTo3DUVNode_RNP`
+ `Tencent3DPartNode_RNP`) which uses the `["MODEL_3D", {}]` wire
shape.

Co-authored-by: Amp <amp@ampcode.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019e362f-db40-7338-8ca4-99f617d89812
2026-05-17 06:57:14 -07:00
Jedrzej Kosinski 95a3073d79 Merge pull request #20 from Comfy-Org/feat/model3d-input-encoder
Add MODEL_3D input encoder to close the encode-side gap
2026-05-17 06:53:57 -07:00
Jedrzej KosinskiandAmp 01f71bd806 Add MODEL_3D input encoder to close the encode-side gap
Adds the symmetric client-side encoder for `File3D` values so a remote
node descriptor can declare a `MODEL_3D` input and consume a 3D mesh
from an upstream local node (e.g. `Load3D`, `TencentTextToModelNode`,
`MeshyTextToModelNode`).

Mirrors the existing AUDIO encoder (`encode_audio_input`,
`is_audio_input`) and VIDEO encoder (`encode_video_input`,
`is_video_input` — PR #19) added at
`comfy_remote_nodes/serialization.py`:

- `encode_model3d_input(file3d)` reads the primary mesh bytes via
  `File3D.get_data()` (returns a `BytesIO`, per
  `comfy_api.latest._util.geometry_types.File3D`) and emits a
  `{type: "model_3d", encoding: "glb_inline", data: b64, format: ...}`
  envelope. `encoding="glb_inline"` is the generic single-file inline
  encoding the server-side decoder (`decode_model3d_envelope`)
  already dispatches on; the decoder hands the bytes to
  `File3D(BytesIO(...), format)` regardless of whether the actual
  format is glb / obj / fbx / etc., so the encoding name is a
  logical "inline single-file 3D" tag and the `format` extra picks
  the file extension at decode time.
- `is_model3d_input(value)` duck-types on the `format` string
  attribute + `get_data` callable. Both are present on the upstream
  `File3D` class and the `BundledFile3D` subclass built lazily in
  `_bundled_file3d_class()`. Avoids importing `File3D` at module load
  (which would pull torch via the `comfy_api` tree).
- `_encode_one` in `proxy_node.py` dispatches AUDIO / VIDEO before
  MODEL_3D as defense-in-depth (the `VideoInput` shape — `save_to`
  + `get_duration` — does not expose a `format` string property, so
  VIDEO and MODEL_3D values don't collide either way).

Multi-file bundles (`BundledFile3D`-style values where every
companion file matters — e.g. an OBJ with sibling .mtl + texture
PNGs) are intentionally not supported on the encode path in this
PR: the first server consumer is the Tencent Hunyuan3D
MODEL_3D-input node family (`TencentModelTo3DUVNode` /
`Tencent3DPartNode` / `Tencent3DTextureEditNode` /
`TencentSmartTopologyNode`), and the upstream upload helper
`upload_3d_model_to_comfyapi(cls, model_3d, file_format)` only
uploads a single file (`model_3d.get_data()`). Bundle encode
(`bundle_inline`) lands when a partner node actually needs the
companion bytes.

This unblocks 4 Tencent Hunyuan3D nodes (out of 6 in the module)
that were deferred to a follow-up PR — see matrix coverage TODO #6.
First server consumers ship in a paired comfy-rnp-server PR
(TencentModelTo3DUVNode_RNP + Tencent3DPartNode_RNP).

Co-authored-by: Amp <amp@ampcode.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019e362f-db40-7338-8ca4-99f617d89812
2026-05-17 06:53:14 -07:00
Jedrzej Kosinski 569212f1e7 Merge pull request #19 from Comfy-Org/feat/video-input-encoder
Add VIDEO input encoder to close the encode-side gap
2026-05-17 05:08:38 -07:00
Jedrzej KosinskiandAmp 04ca70a853 Add VIDEO input encoder to close the encode-side gap
The proxy_node has accepted VIDEO outputs since PR #13 (server-side
`make_video_envelope` + client-side `decode_video_envelope`), but
the inbound encoding direction has been a no-op: `_encode_one` in
`proxy_node.py` only dispatched AUDIO + IMAGE + MASK tensors, so any
remote node declaring an `IO.Video.Input` would receive the raw
`VideoInput` object as a non-JSON-serialisable Python value and the
upstream serialisation would either drop it or fail.

`serialization.py:468` made the gap explicit with a placeholder
comment ("encode lands when a remote node accepts VIDEO inputs"),
and several server-side providers shipped against the
`VIDEO_MP4_BASE64` capability + `input_serialization={"video":
"mp4_base64"}` declarations on the assumption that the client
encoder would land soon — most notably `GrokVideoEditProvider`
and `GrokVideoExtendProvider`, whose end-to-end VIDEO upload path
was broken until now.

This change wires the encode direction:

* `serialization.py` gains `encode_video_input(video)` — mirrors
  the AUDIO encoder above. Re-encodes the `VideoInput` to an
  in-memory mp4/H.264 byte buffer via the upstream
  `comfy_api_nodes/util/conversions.video_to_base64_string` (same
  call shape: `video.save_to(buf, format=MP4, codec=H264)`) and
  base64-encodes the result. Populates `duration_s` from
  `video.get_duration()` so server-side providers (e.g.
  `_validate_video_duration_envelope` in `grok.py`) can range-check
  without demuxing the MP4.

* `serialization.py` gains `is_video_input(value)` — duck-types on
  the `save_to` + `get_duration` method pair (both defined on
  `comfy_api.latest._input.VideoInput` and present on every
  concrete subclass). Avoids importing `VideoInput` at module
  load (which would pull torch via the `comfy_api` tree). Both
  helpers added to `__all__`.

* `proxy_node.py:_encode_one` gains a VIDEO branch right after the
  AUDIO branch (and before the tensor-rank dispatch), mirroring the
  AUDIO pattern at the same call site.

* `proxy_node.py` `_inputs_to_envelopes` docstring updates the
  duck-typing-rules list to mention the new VIDEO branch.

Symmetric to the (still-open) MODEL_3D INPUT gap noted at
`serialization.py:484`. AUDIO INPUT was already wired (ElevenLabs
nodes exercise it end-to-end); IMAGE / MASK have always been
wired; VIDEO INPUT closes today's last input-direction envelope
gap for the existing capability vocabulary.

Verified by importing `serialization` with a stubbed
`comfy_remote_nodes` package: `is_video_input` returns False for
None / dict / audio-shaped dict and True for an object exposing
both `save_to` + `get_duration`. `encode_video_input` + the
proxy_node integration are exercised end-to-end by the server-side
`Wan2VideoContinuationApi_RNP` provider landing in the matching
comfy-rnp-server PR (it consumes `first_clip: VIDEO` and uploads
the resolved bytes to Comfy storage), and also unblocks the
shipped-but-broken Grok VideoEdit/VideoExtend providers.

Co-authored-by: Amp <amp@ampcode.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019e35d1-3c68-74ce-9918-fe9cacd74276
2026-05-17 05:07:57 -07:00
Jedrzej Kosinski ed5823e8e7 Merge pull request #18 from Comfy-Org/revert/task-handle-decoder
Revert "Add client-side cross-node task_handle decoder" (PR #17)
2026-05-16 22:08:11 -07:00
3 changed files with 603 additions and 9 deletions
+360
View File
@@ -0,0 +1,360 @@
"""Smoke test for nested dict recursion in ``_encode_one`` /
``_encode_inputs`` / ``_externalize_nested``.
Background: ComfyUI's V3 ``build_nested_inputs`` rebuilds AUTOGROW and
DYNAMIC_COMBO runtime kwargs into nested dicts (e.g. ``{"reference_images":
{"image1": <tensor>, "image2": <tensor>}}`` for a top-level AG, or
``{"model": {"branch": "wan2.7-r2v", "video_refs": {"video1":
<VideoInput>}}}`` for AG-inside-DC). Before this fix, ``_encode_one``
returned a non-tensor non-VideoInput non-audio dict unchanged, which
meant per-slot tensors / VideoInputs survived past encoding and would
either crash at ``json.dumps`` time or violate the server's contract
(comfy_rnp_server's ``_ordered_wan_videoedit_image_envelopes`` etc.
expect each slot value to already be an RNP envelope).
Asserts:
1. Top-level AG-of-IMAGE dict: each ``image<N>`` tensor becomes an
image envelope; the wrapping dict structure is preserved.
2. Top-level AG-of-VIDEO dict: each ``video<N>`` VideoInput becomes a
video envelope.
3. DC-wrapping-AG: top-level dict carries a branch key (string) AND a
nested AG dict of tensors / VideoInputs; encoder recurses one level
deeper and encodes the leaves while leaving the branch key untouched.
4. AUDIO input is NOT swallowed by the dict-recursion branch: the
``{"waveform": ..., "sample_rate": ...}`` shape still routes to
``encode_audio_input``.
5. Already-encoded envelope dicts pass through unchanged (no double-
encoding) — protects against re-running the encoder on a value that
the caller pre-encoded.
6. Plain scalar config dicts (``{"width": 1024, "height": 576}``) pass
through unchanged.
7. ``_externalize_nested`` walks the same shape: a nested AG dict of
oversize envelopes uploads each envelope via the stubbed externalize
path and the envelope's ``data`` field is swapped for ``uri``;
non-envelope siblings are untouched; the wrapping dict structure is
preserved.
8. JSON-serializability post-condition: ``json.dumps`` round-trips the
full encoded payload (regression guard for the exact bug fixed).
9. Single-tensor / single-VideoInput / single-audio top-level inputs
still encode the same way they did before (no regression for the
existing leaf path).
Run with any python that has ``torch`` available (uses the ComfyUI
venv on this workstation):
python notes/run_encode_nested_dicts.py
The test loads ``_encode_one`` / ``_encode_inputs`` / ``_externalize_nested``
out of ``proxy_node.py`` via AST extraction so we don't pay the cost
of importing ComfyUI's ``comfy_api`` / ``comfy_api_nodes`` packages
(which those helpers don't touch). Same standalone pattern as
``run_image_max_batch.py``.
"""
from __future__ import annotations
import ast
import asyncio
import importlib.util
import json
import os
import sys
import types
import torch
# ---------------------------------------------------------------------------
# Locate the worktree under test (this file's parent's parent).
# ---------------------------------------------------------------------------
_HERE = os.path.dirname(os.path.abspath(__file__))
_CLIENT_DIR = os.path.dirname(_HERE)
def _load_protocol() -> types.ModuleType:
"""Load ``protocol.py`` standalone (no relative imports)."""
path = os.path.join(_CLIENT_DIR, "protocol.py")
spec = importlib.util.spec_from_file_location("rnp_smoke_protocol", path)
mod = importlib.util.module_from_spec(spec)
sys.modules[spec.name] = mod
spec.loader.exec_module(mod)
return mod
class _FakeVideoInput:
"""Duck-typed VideoInput stand-in: ``save_to`` + ``get_duration``
are the only attrs ``serialization.is_video_input`` checks for."""
def __init__(self, tag: str, duration: float = 4.0) -> None:
self.tag = tag
self._duration = duration
def save_to(self, *args, **kwargs): # pragma: no cover — never called by stub
return None
def get_duration(self) -> float:
return self._duration
def _build_serialization_stub(protocol: types.ModuleType) -> types.ModuleType:
"""Minimal ``serialization`` shim covering exactly the surface the
helpers under test use."""
mod = types.ModuleType("rnp_smoke_serialization")
def _is_torch_tensor(value): # noqa: ANN001
return isinstance(value, torch.Tensor)
def is_audio_input(value): # noqa: ANN001
return (
isinstance(value, dict)
and {"waveform", "sample_rate"} <= set(value.keys())
)
def is_video_input(value): # noqa: ANN001
return (
callable(getattr(value, "save_to", None))
and callable(getattr(value, "get_duration", None))
)
def is_model3d_input(value): # noqa: ANN001
return False # unused in this test
def encode_image_tensor(tensor, *, accepts_batch=False): # noqa: ANN001, ARG001
return {
"type": "image",
"encoding": "png_base64",
"data": f"img:{int(tensor.shape[0])}",
}
def encode_mask_tensor(tensor): # noqa: ANN001, ARG001
return {"type": "mask", "encoding": "png_base64", "data": "mask"}
def encode_audio_input(value): # noqa: ANN001, ARG001
return {"type": "audio", "encoding": "mp3_base64", "data": "aud"}
def encode_video_input(value): # noqa: ANN001
return {
"type": "video",
"encoding": "mp4_base64",
"data": f"vid:{getattr(value, 'tag', '?')}",
}
def encode_model3d_input(value): # noqa: ANN001, ARG001 # pragma: no cover
return {"type": "model_3d", "encoding": "glb_base64", "data": "m3d"}
HEAVY_TYPES = {"image", "mask", "audio", "video", "model_3d"}
def is_envelope(value): # noqa: ANN001
return (
isinstance(value, dict)
and isinstance(value.get("type"), str)
and value.get("type") in HEAVY_TYPES
and isinstance(value.get("encoding"), str)
)
# Cap-aware ``maybe_externalize`` stub: swap ``data`` for ``uri`` when
# the inline payload would exceed ``max_inline_bytes``. Lets us prove
# that ``_externalize_nested`` reaches every per-slot envelope.
upload_log: list[str] = []
async def maybe_externalize(
envelope, # noqa: ANN001
*, server_url=None, max_inline_bytes=None, auth_headers=None, # noqa: ANN001, ARG001
):
if not server_url or max_inline_bytes is None:
return envelope
data = envelope.get("data")
if not isinstance(data, str):
return envelope
if len(data) <= max_inline_bytes:
return envelope
upload_log.append(f"{envelope.get('type')}:{data}")
out = {k: v for k, v in envelope.items() if k != "data"}
out["uri"] = f"https://upload/{envelope.get('type')}/{len(upload_log)}"
return out
mod._is_torch_tensor = _is_torch_tensor
mod.is_audio_input = is_audio_input
mod.is_video_input = is_video_input
mod.is_model3d_input = is_model3d_input
mod.encode_image_tensor = encode_image_tensor
mod.encode_mask_tensor = encode_mask_tensor
mod.encode_audio_input = encode_audio_input
mod.encode_video_input = encode_video_input
mod.encode_model3d_input = encode_model3d_input
mod.is_envelope = is_envelope
mod.maybe_externalize = maybe_externalize
mod._upload_log = upload_log # exposed for assertions
return mod
def _build_proxy_node_stub(
protocol: types.ModuleType, serialization: types.ModuleType,
) -> types.ModuleType:
"""Extract the helpers under test from ``proxy_node.py``."""
mod = types.ModuleType("rnp_smoke_proxy_node")
mod.__dict__.update({
"Any": object,
"log": types.SimpleNamespace(warning=lambda *a, **kw: None),
"serialization": serialization,
"RnpProtocolError": protocol.RnpProtocolError,
"ErrorCode": protocol.ErrorCode,
})
src_path = os.path.join(_CLIENT_DIR, "proxy_node.py")
with open(src_path, "r", encoding="utf-8") as fh:
src = fh.read()
tree = ast.parse(src)
wanted = {
"_enforce_local_validate",
"_check_image_max_batch",
"_encode_one",
"_encode_inputs",
"_externalize_nested",
}
nodes = [
n for n in tree.body
if isinstance(n, (ast.AsyncFunctionDef, ast.FunctionDef))
and n.name in wanted
]
snippet = ast.Module(body=nodes, type_ignores=[])
code = compile(snippet, src_path, "exec")
exec(code, mod.__dict__)
missing = wanted - set(mod.__dict__)
if missing:
raise RuntimeError(f"proxy_node helpers missing from extract: {missing}")
return mod
# ---------------------------------------------------------------------------
# Tests
# ---------------------------------------------------------------------------
async def main() -> int:
protocol = _load_protocol()
serialization = _build_serialization_stub(protocol)
proxy_node = _build_proxy_node_stub(protocol, serialization)
img1 = torch.zeros((1, 64, 64, 3), dtype=torch.float32)
img2 = torch.zeros((1, 32, 32, 3), dtype=torch.float32)
vid1 = _FakeVideoInput("v1")
vid2 = _FakeVideoInput("v2")
audio = {
"waveform": torch.zeros((1, 1, 1000), dtype=torch.float32),
"sample_rate": 16000,
}
# ---- 1. Top-level AG-of-IMAGE dict encodes per-slot
out = await proxy_node._encode_inputs(
{"reference_images": {"image1": img1, "image2": img2}},
)
refs = out["reference_images"]
assert isinstance(refs, dict) and set(refs.keys()) == {"image1", "image2"}, refs
assert refs["image1"]["type"] == "image" and refs["image1"]["data"] == "img:1"
assert refs["image2"]["type"] == "image" and refs["image2"]["data"] == "img:1"
print("ok: top-level AG-of-IMAGE encoded per slot")
# ---- 2. Top-level AG-of-VIDEO dict encodes per-slot
out = await proxy_node._encode_inputs(
{"video_refs": {"video1": vid1, "video2": vid2}},
)
vrefs = out["video_refs"]
assert isinstance(vrefs, dict) and set(vrefs.keys()) == {"video1", "video2"}, vrefs
assert vrefs["video1"]["type"] == "video" and vrefs["video1"]["data"] == "vid:v1"
assert vrefs["video2"]["type"] == "video" and vrefs["video2"]["data"] == "vid:v2"
print("ok: top-level AG-of-VIDEO encoded per slot")
# ---- 3. DC-wrapping-AG: branch key untouched, nested AG dict encoded
dc_value = {
"model": "wan2.7-r2v",
"video_refs": {"video1": vid1},
"reference_images": {"image1": img1, "image2": img2},
"duration": 5,
}
out = await proxy_node._encode_inputs({"model": dc_value})
enc = out["model"]
assert enc["model"] == "wan2.7-r2v", enc
assert enc["duration"] == 5, enc
assert enc["video_refs"]["video1"]["type"] == "video"
assert enc["reference_images"]["image1"]["type"] == "image"
assert enc["reference_images"]["image2"]["type"] == "image"
print("ok: DC-wrapping-AG encoded leaves at every level")
# ---- 4. AUDIO dict is encoded, NOT recursed into
out = await proxy_node._encode_inputs({"audio": audio})
assert out["audio"] == {"type": "audio", "encoding": "mp3_base64", "data": "aud"}
print("ok: AUDIO dict still routes to encode_audio_input")
# ---- 5. Already-encoded envelope passes through (no double-encode)
pre_encoded = {"type": "image", "encoding": "png_base64", "data": "preimg"}
out = await proxy_node._encode_inputs({"images": pre_encoded})
assert out["images"] == pre_encoded, out
print("ok: pre-encoded envelope passed through unchanged")
# ---- 6. Plain scalar config dict passes through unchanged
out = await proxy_node._encode_inputs(
{"config": {"width": 1024, "height": 576, "name": "x"}},
)
assert out["config"] == {"width": 1024, "height": 576, "name": "x"}, out
print("ok: scalar config dict passed through unchanged")
# ---- 7. _externalize_nested walks per-slot envelopes
serialization._upload_log.clear()
big_img = torch.zeros((1, 64, 64, 3), dtype=torch.float32)
# encode_image_tensor stub emits ``data="img:1"`` (5 chars). Cap at 3
# to force every slot to externalize.
out = await proxy_node._encode_inputs(
{"reference_images": {"image1": big_img, "image2": big_img}},
server_url="https://fake.server",
max_inline_bytes=3,
)
refs = out["reference_images"]
assert refs["image1"].get("uri", "").startswith("https://upload/image/"), refs
assert refs["image2"].get("uri", "").startswith("https://upload/image/"), refs
assert "data" not in refs["image1"] and "data" not in refs["image2"]
assert len(serialization._upload_log) == 2, serialization._upload_log
print("ok: nested AG envelopes externalized per slot")
# ---- 7b. _externalize_nested leaves non-envelope siblings alone
serialization._upload_log.clear()
out = await proxy_node._encode_inputs(
{"model": {
"model": "wan2.7-r2v",
"duration": 5,
"reference_images": {"image1": big_img},
}},
server_url="https://fake.server",
max_inline_bytes=3,
)
enc = out["model"]
assert enc["model"] == "wan2.7-r2v" and enc["duration"] == 5, enc
assert enc["reference_images"]["image1"]["uri"].startswith("https://upload/image/")
print("ok: scalar siblings preserved alongside externalized envelopes")
# ---- 8. JSON-serializability post-condition
payload = await proxy_node._encode_inputs({
"reference_images": {"image1": img1, "image2": img2},
"video_refs": {"video1": vid1},
"audio": audio,
"config": {"width": 1024},
"scalar": "hello",
"n": 42,
})
json.dumps(payload) # would raise TypeError pre-fix
print("ok: full encoded payload is JSON-serializable")
# ---- 9. Leaf-path regression: single-tensor / single-video / single-audio
out = await proxy_node._encode_inputs({"image": img1})
assert out["image"]["type"] == "image", out
out = await proxy_node._encode_inputs({"video": vid1})
assert out["video"]["type"] == "video", out
out = await proxy_node._encode_inputs({"audio": audio})
assert out["audio"]["type"] == "audio", out
out = await proxy_node._encode_inputs({"prompt": "hi", "seed": 7})
assert out == {"prompt": "hi", "seed": 7}, out
print("ok: top-level leaf path unchanged")
print("ALL OK")
return 0
if __name__ == "__main__":
raise SystemExit(asyncio.run(main()))
+101 -7
View File
@@ -695,6 +695,30 @@ def _build_input(name: str, io_type: Any, options: dict[str, Any], optional: boo
return IO.Audio.Input(name, **primitive_common)
if io_type == "MASK":
return IO.Mask.Input(name, **primitive_common)
if io_type == "MODEL_3D":
# 3D MODEL input: symmetric counterpart of the ``"MODEL_3D"``
# entry in ``_OUTPUT_CLASSES`` above. The Tencent Hunyuan3D
# MODEL_3D-input nodes (``TencentModelTo3DUVNode`` /
# ``Tencent3DTextureEditNode`` / ``Tencent3DPartNode`` /
# ``TencentSmartTopologyNode``) all declare their model_3d
# input as ``IO.MultiType.Input(types=[IO.File3DGLB,
# IO.File3DOBJ, IO.File3DFBX, IO.File3DAny])`` so the socket
# accepts any upstream 3D source: ``Load3D``'s ``File3DAny``
# output, Meshy / Hunyuan3D / Tripo / Rodin's per-format
# outputs (``FILE_3D_GLB`` / ``FILE_3D_OBJ`` / ``FILE_3D_FBX``),
# etc. Mirror that union here so the proxy socket accepts the
# same set of comfytypes upstream does. Per-node format
# validation (e.g. 3DTextureEdit's FBX-only enforcement) is
# done server-side in the provider's ``execute()`` against
# the inline envelope's ``format`` extra; this socket-level
# filter is purely about frontend connection validity.
return IO.MultiType.Input(
name,
types=[
IO.File3DGLB, IO.File3DOBJ, IO.File3DFBX, IO.File3DAny,
],
**primitive_common,
)
if io_type == "DYNAMIC_COMBO":
# Wire shape: ``["DYNAMIC_COMBO", {"options": [{"key": str,
# "inputs": [[name, io_type, opts], ...]}, ...], "tooltip": ...}]``.
@@ -1546,6 +1570,8 @@ async def _encode_inputs(
* ``torch.Tensor`` rank 3 with last-dim != 3/4 → mask envelope (B,H,W).
Rank-2 (H,W) tensors are also treated as masks.
* ``dict`` with ``waveform`` + ``sample_rate`` → audio envelope.
* ``VideoInput`` subclass (duck-typed via ``save_to`` +
``get_duration``) → mp4-base64 video envelope.
* Everything else passes through as-is (already JSON-serializable).
When ``max_inline_bytes`` is set, any envelope whose inline payload
@@ -1574,17 +1600,65 @@ async def _encode_inputs(
name, value, validate_map.get(name) or {}, node_id=node_id,
)
encoded = _encode_one(name, value, serialization_map.get(name))
# Only envelope-shaped values can be externalized; scalars pass
# straight through ``maybe_externalize`` unchanged.
if serialization.is_envelope(encoded):
encoded = await serialization.maybe_externalize(
encoded,
# Externalize any envelope payloads, walking into nested dicts so
# AUTOGROW slot values and DYNAMIC_COMBO sub-inputs whose encoded
# form is a per-slot envelope are uploaded the same way a
# top-level singleton envelope is. ``_externalize_nested`` is a
# no-op on plain scalars and on dicts that contain no envelopes.
encoded = await _externalize_nested(
encoded,
server_url=server_url,
max_inline_bytes=max_inline_bytes,
auth_headers=auth_headers,
)
out[name] = encoded
return out
async def _externalize_nested(
value: Any,
*,
server_url: str | None,
max_inline_bytes: int | None,
auth_headers: dict[str, str] | None,
) -> Any:
"""Recursively externalize any RNP envelopes inside ``value``.
Mirrors :func:`_encode_one`'s dict-recursion: ComfyUI hands the
proxy_node nested AUTOGROW / DYNAMIC_COMBO runtime dicts
(e.g. ``{"reference_images": {"image1": <env>, "image2": <env>}}``
or ``{"model": {"branch": "wan2.7-r2v",
"video_refs": {"video1": <env>}}}``); without this walk the
per-slot envelopes would ship inline regardless of
``max_inline_bytes`` because :func:`_encode_inputs` only inspected
the top-level value.
Stops at envelope leaves (delegates the per-envelope decision to
:func:`serialization.maybe_externalize`) and at plain scalars (leaves
them unchanged). Lists are intentionally left alone — AG/DC runtime
shapes always materialize as dicts, never lists, and recursing into
arbitrary lists would touch opaque payloads. Recursion is bounded
by the AUTOGROW max slot count (100) and the maximum DC branch
nesting depth (1 today), so an in-place ``async def`` walk is fine.
"""
if serialization.is_envelope(value):
return await serialization.maybe_externalize(
value,
server_url=server_url,
max_inline_bytes=max_inline_bytes,
auth_headers=auth_headers,
)
if isinstance(value, dict):
out: dict[str, Any] = {}
for k, v in value.items():
out[k] = await _externalize_nested(
v,
server_url=server_url,
max_inline_bytes=max_inline_bytes,
auth_headers=auth_headers,
)
out[name] = encoded
return out
return out
return value
def _enforce_local_validate(
@@ -1692,8 +1766,28 @@ def _encode_one(
value: Any,
accepted_encodings: list[str] | None = None,
) -> Any:
# ComfyUI's V3 ``build_nested_inputs`` rebuilds AUTOGROW / DYNAMIC_COMBO
# runtime kwargs into nested dicts (``{"reference_images": {"image1":
# <tensor>, ...}}`` for top-level AG; ``{"<dc-name>": branch_key,
# "<nested-ag>": {"video1": <VideoInput>, ...}}`` for AG-inside-DC).
# Recurse into plain dicts so heavy leaves anywhere in that tree get
# encoded — without this an AG-of-IMAGE slot would ship the raw
# torch.Tensor inside a dict and crash at ``json.dumps`` time. Stops
# at envelope dicts (already-encoded payloads) so we don't double-
# encode wire-shape values that happen to be dicts.
if serialization.is_envelope(value):
return value
if isinstance(value, dict) and not serialization.is_audio_input(value):
return {
k: _encode_one(f"{name}.{k}", v, accepted_encodings)
for k, v in value.items()
}
if serialization.is_audio_input(value):
return serialization.encode_audio_input(value)
if serialization.is_video_input(value):
return serialization.encode_video_input(value)
if serialization.is_model3d_input(value):
return serialization.encode_model3d_input(value)
if not serialization._is_torch_tensor(value):
return value
rank = value.dim()
+142 -2
View File
@@ -465,9 +465,44 @@ def decode_audio_envelope(envelope: dict[str, Any]) -> dict[str, Any]:
# ---------------------------------------------------------------------------
# Video decode (encode lands when a remote node accepts VIDEO inputs)
# Video encode / decode
# ---------------------------------------------------------------------------
def encode_video_input(video: Any) -> dict[str, Any]:
"""Encode a ComfyUI VIDEO (``VideoInput`` subclass) as an mp4-base64
video envelope.
Mirrors the AUDIO encoder above: re-encodes the Video object to an
in-memory mp4/H.264 byte buffer (matches upstream
``comfy_api_nodes/util/conversions.py`` ``video_to_base64_string``)
and base64-encodes it. Populates ``duration_s`` metadata when the
Video object exposes ``get_duration()`` so server-side providers
(e.g. ``GrokVideoExtendProvider._validate_video_duration_envelope``)
can range-check without demuxing the MP4.
"""
from comfy_api_nodes.util.conversions import video_to_base64_string
b64 = video_to_base64_string(video)
extra: dict[str, Any] = {}
duration_s: float | None = None
try:
getter = getattr(video, "get_duration", None)
if callable(getter):
d = getter()
if d is not None:
duration_s = float(d)
except Exception:
duration_s = None
if duration_s is not None:
extra["duration_s"] = duration_s
return {
"type": "video",
"encoding": "mp4_base64",
"data": b64,
**extra,
}
def decode_video_envelope(envelope: dict[str, Any]) -> Any:
"""Decode a video envelope into a ComfyUI Video object (mp4 inline)."""
encoding = envelope.get("encoding")
@@ -481,9 +516,69 @@ def decode_video_envelope(envelope: dict[str, Any]) -> Any:
# ---------------------------------------------------------------------------
# 3D model decode (encode lands when a remote node accepts MODEL_3D inputs)
# 3D model encode / decode
# ---------------------------------------------------------------------------
def encode_model3d_input(model_3d: Any) -> dict[str, Any]:
"""Encode a ComfyUI ``File3D`` (or compatible) as an inline
single-file 3D-model envelope.
Mirrors the AUDIO / VIDEO encoders above: reads the primary mesh
bytes (via the ``File3D`` API: ``get_data().read()`` returns a
``BytesIO`` payload; falling back to ``read_bytes()`` for older
sources) and base64-encodes them. The envelope's ``encoding`` is
``"glb_inline"`` (the generic single-file inline encoding the
server-side decoder dispatches on at :func:`decode_model3d_envelope`
— the decoder hands the bytes to ``File3D(BytesIO(...), format)``
regardless of whether the actual format is glb / obj / fbx / etc.,
so the ``encoding`` name is a logical "inline single-file 3D" tag
and the ``format`` extra picks the file extension).
Multi-file bundles (``BundledFile3D``-style values where every
companion file matters — e.g. an OBJ with a sibling .mtl + texture
PNGs) are not yet supported on the encode path: this PR ships the
minimal wire shape needed for the Tencent Hunyuan3D MODEL_3D-input
nodes which all upload a single file via ``upload_3d_model_to_comfyapi``
upstream. Bundle encode (``bundle_inline``) lands when a partner
node actually needs the companion bytes.
"""
file_format = ""
fmt_attr = getattr(model_3d, "format", None)
if isinstance(fmt_attr, str):
file_format = fmt_attr.lower()
# ``File3D.get_data()`` returns a ``BytesIO`` (per
# ``comfy_api.latest._util.geometry_types.File3D``). Other File3D-
# compatible classes may instead expose ``read_bytes`` returning
# raw bytes; fall back to that for compatibility.
raw: bytes
get_data = getattr(model_3d, "get_data", None)
if callable(get_data):
buf = get_data()
if hasattr(buf, "seek"):
buf.seek(0)
raw = buf.read() if hasattr(buf, "read") else bytes(buf)
else:
read_bytes = getattr(model_3d, "read_bytes", None)
if callable(read_bytes):
raw = read_bytes()
else:
raise RnpProtocolError(
f"unsupported 3D model value: {type(model_3d).__name__!r} "
f"has no get_data() or read_bytes()",
code=ErrorCode.INTERNAL,
)
envelope: dict[str, Any] = {
"type": "model_3d",
"encoding": "glb_inline",
"data": base64.b64encode(raw).decode("ascii"),
}
if file_format:
envelope["format"] = file_format
return envelope
# ---------------------------------------------------------------------------
# Multi-file 3D bundle: a ``File3D``-compatible class that carries the
# primary mesh plus N companion files (textures / .mtl / .bin / etc.).
@@ -837,6 +932,47 @@ def is_audio_input(value: Any) -> bool:
)
def is_video_input(value: Any) -> bool:
"""Best-effort check for a ComfyUI ``VideoInput`` subclass.
Duck-types on the ``save_to`` + ``get_duration`` method pair: both
are defined on the ``VideoInput`` abstract base in
``comfy_api.latest._input`` and present on every concrete subclass
(``VideoFromFile`` / ``VideoFromComponents``). Avoids importing
``VideoInput`` at module load (which would pull torch via the
``comfy_api`` tree).
"""
if isinstance(value, dict):
return False
return (
callable(getattr(value, "save_to", None))
and callable(getattr(value, "get_duration", None))
)
def is_model3d_input(value: Any) -> bool:
"""Best-effort check for a ComfyUI ``File3D`` (or compatible).
Duck-types on the ``format`` string attribute + ``get_data``
callable: both are present on the ``File3D`` class in
``comfy_api.latest._util.geometry_types`` and the
``BundledFile3D`` subclass built lazily in
:func:`_bundled_file3d_class`. Avoids importing ``File3D`` at
module load (which would pull torch via the ``comfy_api`` tree).
The ``VideoInput`` shape (``save_to`` + ``get_duration``) does not
expose a ``format`` string property, so VIDEO and MODEL_3D values
don't collide; caller (:func:`_encode_one`) still dispatches
AUDIO / VIDEO before MODEL_3D as defense-in-depth.
"""
if isinstance(value, dict):
return False
return (
isinstance(getattr(value, "format", None), str)
and callable(getattr(value, "get_data", None))
)
def _is_torch_tensor(value: Any) -> bool:
try:
import torch
@@ -853,7 +989,9 @@ __all__ = [
"decode_mask_envelope",
"encode_audio_input",
"decode_audio_envelope",
"encode_video_input",
"decode_video_envelope",
"encode_model3d_input",
"decode_model3d_envelope",
"_bundled_file3d_class",
"decode_envelope",
@@ -861,4 +999,6 @@ __all__ = [
"is_image_tensor",
"is_mask_tensor",
"is_audio_input",
"is_video_input",
"is_model3d_input",
]