159 lines
7.6 KiB
Markdown
159 lines
7.6 KiB
Markdown
# ComfyUI Claude CLI (Vision + Text) — Node Pack
|
||
|
||
A small collection of ComfyUI custom nodes built around calling the local **`claude` CLI** (Claude Code) for vision+text analysis of video frames, plus the batch-manipulation utilities needed to feed it cleanly.
|
||
|
||
The headline node runs Claude against one or many images per call, reusing a single session across many invocations to avoid the ~30k-token cold-start penalty. The utility nodes fix common friction points from other packs (VHS `SelectImages` out-of-bounds crashes, iteration/indexing patterns, batch splitting for pair-wise or grid-wise analysis).
|
||
|
||
**Repo:** <https://github.com/RandyHaylor/comfyui-claude-cli-vision-text-node>
|
||
|
||
---
|
||
|
||
## Installation
|
||
|
||
1. Install [Claude Code](https://docs.claude.com/en/docs/claude-code/overview) and run `claude login` once.
|
||
2. Clone this repo into your ComfyUI custom-nodes directory:
|
||
|
||
```bash
|
||
cd <ComfyUI>/custom_nodes
|
||
git clone https://github.com/RandyHaylor/comfyui-claude-cli-vision-text-node.git
|
||
```
|
||
|
||
3. Restart ComfyUI.
|
||
4. Hard-refresh the ComfyUI browser tab (`Ctrl+Shift+R`) so the frontend picks up the new node schemas.
|
||
|
||
No extra `pip install` required. Uses only `numpy`, `torch`, and `Pillow` — already in ComfyUI's environment.
|
||
|
||
---
|
||
|
||
## Nodes
|
||
|
||
### 1. Claude CLI (Vision+Text) — `ClaudeCliVisionText`
|
||
|
||
Calls `claude -p` once per invocation against an optional image input. Uses the user's existing `claude login` OAuth session by default; no `ANTHROPIC_API_KEY` needed.
|
||
|
||
**Inputs:**
|
||
- `system_prompt`, `user_prompt` — headers rendered above each field via JS widget.
|
||
- `model` — dropdown: `haiku` / `sonnet` / `opus` / `claude-haiku-4-5` / `claude-sonnet-4-6` / `claude-opus-4-7`.
|
||
- `agent` — dropdown: `general-purpose` / `Explore` / `Plan` / `code-reviewer` / `debugger` / `data-scientist`. `general-purpose` omits `--agent` for compatibility with older CLIs.
|
||
- `reuse_session` + `session_id` — session UUID auto-populated on node creation; "Regenerate Session UUID" button with a cost warning reminds users that new sessions cost ~30k tokens to cold-start.
|
||
- `send_all_frames` — when `images` is a multi-frame batch, attach all frames to one CLI call (vs. just frame 0).
|
||
- `timeout_seconds` — subprocess timeout (default 120).
|
||
- `on_error` — `empty` / `raise` / `retry_once` (default `retry_once`).
|
||
- `is_enable` — quick disable toggle.
|
||
- Optional `images` — IMAGE tensor (batch or single); text-only when unwired.
|
||
- Optional `api_key` — enables `--bare` headless mode with `ANTHROPIC_API_KEY` env (session reuse disabled in bare mode).
|
||
|
||
**Outputs:**
|
||
- `response` — parsed `.result` from the CLI's JSON envelope.
|
||
- `session_id` — the effective UUID actually used (copy into widget to reuse).
|
||
- `stderr` — CLI stderr for debugging.
|
||
- `session_usage` — cumulative token and cost summary for the session, parsed from `~/.claude/projects/**/<session_id>.jsonl`. Knows when to advise starting a new session.
|
||
|
||
**Features:**
|
||
- Strict UUID validation on `session_id` (non-UUID strings raise cleanly).
|
||
- Normalizes round-trip junk values (`"null"`, `"undefined"`, `""`, whitespace) on optional widgets so a stale widget state doesn't accidentally trip `--bare` mode.
|
||
- Auth-failure detection: non-zero exits with `"not logged in"` in stderr raise a ComfyUI-visible "Run `claude login`" error.
|
||
- Per-session debug log at `<ComfyUI>/output/claude-sessions/<session_id>.debug.log` on CLI failures.
|
||
- Downsamples images to 1568 px long edge before attaching (safe across Haiku/Sonnet/Opus vision limits).
|
||
- Image attachments deleted from disk after each call (PNGs are transient; the Anthropic cache is server-side).
|
||
|
||
---
|
||
|
||
### 2. Image Batch Current Index — `ImageBatchCurrentIndex`
|
||
|
||
Drives ComfyUI's per-item iteration while emitting the matching frame index as a parallel integer list. Put this (or any `OUTPUT_IS_LIST=True` node) between a batch source and a per-frame consumer when you need an integer index alongside the image.
|
||
|
||
**Inputs:** `images` (IMAGE batch, or list of batches), `index_offset` (INT, default 0 — add `+1` for pair-consecutive workflows).
|
||
|
||
**Outputs:**
|
||
- `image` — list of single-frame tensors.
|
||
- `index` — list of INTs, one per image.
|
||
|
||
Downstream nodes that take a single IMAGE and a single INT will be called N times, once per `(frame, index)` pair.
|
||
|
||
---
|
||
|
||
### 3. Select Images (Bounds Safe) — `SelectImagesBoundsSafe`
|
||
|
||
Drop-in replacement for VHS `SelectImages` with a real `err_if_bad_index` toggle. Fixes the VHS bug where `err_if_missing=False` at parse time still passes out-of-bounds indices through to the tensor step, crashing with `IndexError`.
|
||
|
||
**Inputs:** `image` (IMAGE batch), `indexes` (STRING, VHS syntax: `"3"`, `"0,2,5"`, `"2:8"`, `"0:10:2"`, `"-1"`, mixed `"0,2:4,-1"`), `err_if_bad_index` (default False), `err_if_empty` (default False), `fallback_on_empty` (`first_frame` / `all_frames` / `blank`).
|
||
|
||
**Outputs:** `image`, `selected_count`, `skipped_count`.
|
||
|
||
---
|
||
|
||
### 4. Indexed 2x2 Grid from Batch — `IndexedGrid2x2FromBatch`
|
||
|
||
Collapses a multi-node "select + label + grid-composite" subgraph into one node. Produces a batch of 2×2 grid images from an N-frame batch, with each cell labeled with its original frame index.
|
||
|
||
Example — N=8, stride=3 →
|
||
- Grid 1: frames 0,1,2,3
|
||
- Grid 2: frames 3,4,5,6 (stride=3 → 1-frame overlap)
|
||
- Grid 3: frames 4,5,6,7 (final grid backfilled from the end)
|
||
|
||
When N<4, outputs a single 2×2 grid with the missing cells black-filled (real frames are never repeated).
|
||
|
||
**Inputs:** `images`, `stride` (default 3), `label_prefix` (e.g. `"frame "`), `label_font_size` (default 16), `label_bar_height` (default 24).
|
||
|
||
**Outputs:**
|
||
- `grids` — IMAGE batch of M grids.
|
||
- `grid_count` — INT.
|
||
- `index_groups_debug` — STRING summary like `"0,1,2,3 | 3,4,5,6 | 4,5,6,7"`.
|
||
|
||
---
|
||
|
||
### 5. Split Image Batch to List of Image Batches — `SplitImageBatchToListOfImageBatches`
|
||
|
||
Takes an IMAGE batch and emits a **list of smaller batches** (sub-batches), so ComfyUI iterates downstream once per sub-batch, each call receiving a multi-frame tensor of up to `sub_batch_size`. Perfect for feeding Claude pairs or quads of frames per CLI call (with `send_all_frames=True`).
|
||
|
||
**Inputs:**
|
||
- `images` — IMAGE batch.
|
||
- `sub_batch_size` — INT, frames per sub-batch (default 2).
|
||
- `stride` — INT, frames advanced per sub-batch. `0` means "same as sub_batch_size" (non-overlapping partition). `1` = sliding window, any other value = custom.
|
||
- `last_sub_batch_policy` — `keep_remainder` (default) / `drop_remainder` / `pad_with_last`.
|
||
|
||
**Outputs (first three are scheduler-iterated lists):**
|
||
- `sub_batches` — list of IMAGE sub-batches.
|
||
- `start_index` — list of INTs, the original-batch index where each sub-batch begins.
|
||
- `end_index` — list of INTs, inclusive end index.
|
||
- `sub_batch_count` — single INT.
|
||
|
||
The node displays a bottom hint: `"stride 0: stride = sub-batch size"`.
|
||
|
||
---
|
||
|
||
## Typical workflow patterns
|
||
|
||
**Per-frame analysis with session reuse (cheapest):**
|
||
```
|
||
VHS_LoadVideo → ImageBatchCurrentIndex → ClaudeCliVisionText (reuse_session=True)
|
||
```
|
||
N scheduler calls, one frame each. Same session_id across all runs.
|
||
|
||
**Pair-wise analysis (cross-frame reasoning per pair):**
|
||
```
|
||
VHS_LoadVideo → SplitImageBatchToListOfImageBatches (sub_batch_size=2)
|
||
→ ClaudeCliVisionText (send_all_frames=True, reuse_session=True)
|
||
```
|
||
N/2 calls, two frames per call.
|
||
|
||
**2×2 grid montage per call:**
|
||
```
|
||
VHS_LoadVideo → IndexedGrid2x2FromBatch (stride=3)
|
||
→ ClaudeCliVisionText (send_all_frames=True)
|
||
```
|
||
One call per grid image with all four labeled frames in a single view.
|
||
|
||
---
|
||
|
||
## License
|
||
|
||
MIT.
|
||
|
||
---
|
||
|
||
## Contributing
|
||
|
||
Issues and PRs welcome at <https://github.com/RandyHaylor/comfyui-claude-cli-vision-text-node>.
|