feat: make library builder reruns incremental

Adding assets to a catalogued library used to mean re-describing every asset:
the checkpoint is deleted on success, and its key hashed the file list, so any
change invalidated it. Cataloguing five new assets cost the same as the whole
library.

The manifest itself is now the durable record of what is described. With the
new update_existing input (default on), a rerun loads the existing manifest,
reuses those descriptions, and only sends genuinely new files to Claude.
Entries whose file no longer exists are dropped. Turning it off re-describes
everything.

The checkpoint key no longer hashes the file list either, so adding an asset
mid-recovery does not discard an interrupted run's progress.

Not applied to organise-into-a-copy, where manifest paths are post-rename and
cannot be matched against a scan of the untouched source.

Verified with a stubbed CLI across four reruns: fresh library describes all 6,
an unchanged rerun describes 0, adding 3 describes only those 3, and deleting
2 describes 0 while dropping both from the manifest.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

README.md

@@ -130,9 +130,23 @@ blow up in turns and cost on libraries of 100+ assets, with nothing produced at
 batch now only reads its own assets and replies with JSON; the node assembles and writes the
 manifest itself, so no file-writing tool is ever handed to Claude for this.
 
-A hidden checkpoint (`.sf_library_builder_checkpoint.json`, removed automatically on success)
-tracks which assets are already catalogued. If a run fails or times out partway, rerunning with
-the same settings **resumes** — finished batches are skipped, not re-paid for.
+**Reruns are incremental.** With `update_existing` on (the default), the node reads the manifest
+already in the library and only catalogues files it doesn't yet describe. So the normal workflow
+is: drop new assets into the folder, run the same node against the same path, and only the new
+ones cost anything. Entries whose file has since been deleted are dropped automatically. Turn
+`update_existing` off to re-describe everything from scratch.
+
+*(Not available in `organise into a copy` mode, where the manifest holds post-rename paths that
+can't be matched against the untouched source folder.)*
+
+Separately, a hidden checkpoint (`.sf_library_builder_checkpoint.json`, removed automatically on
+success) covers a run that dies partway. Rerunning **resumes** — finished batches are skipped,
+not re-paid for.
+
+If a batch's reply comes back **cut off** — the model ran past its reply length limit partway
+through describing a wordy set of assets — that batch is automatically **retried as two smaller
+halves**, repeatedly if needed, rather than failing the run. You'll see it in the log as
+`Batch 7/7: the reply was cut off… Retrying as two batches of 5 and 6`.
 
 #### Using SF Claude Asset Director
 

py/sf_claude_library_builder.py

@@ -55,16 +55,25 @@ def chunk(items, size):
         yield items[i : i + size]
 
 
+class BatchParseError(RuntimeError):
+    """Claude's reply was not usable JSON.
+
+    Nearly always a reply that ran past its length limit and stopped partway,
+    which a smaller batch fixes — so the caller retries by splitting rather than
+    giving up.
+    """
+
+
 def extract_json(text):
     """The JSON object out of a reply that may be fenced or prefaced."""
     start, end = text.find("{"), text.rfind("}")
     if start == -1 or end <= start:
-        raise RuntimeError(f"Claude did not return JSON. It said: {text.strip()[:300]}")
+        raise BatchParseError("the reply stopped before any complete JSON object")
     body = text[start : end + 1]
     try:
         return json.loads(body)
     except json.JSONDecodeError as e:
-        raise RuntimeError(f"Claude's JSON could not be parsed ({e}): {body[:300]}")
+        raise BatchParseError(f"the reply was cut off or malformed ({e})")
 
 
 def canonicalize(claude_path, expected):
@@ -80,21 +89,47 @@ def canonicalize(claude_path, expected):
     return cleaned
 
 
-def checkpoint_key(mode, manifest_name, system_prompt, project_notes, all_files):
-    """A fingerprint of this exact job, so a checkpoint only resumes a matching rerun."""
+def checkpoint_key(mode, manifest_name, system_prompt, project_notes):
+    """A fingerprint of this job's settings, so a checkpoint only resumes a
+    matching rerun.
+
+    Deliberately excludes the file list: adding or removing an asset should not
+    throw away a half-finished run, and entries are keyed by path anyway, so
+    stale ones are filtered out when the manifest is assembled.
+    """
     payload = json.dumps(
         {
             "mode": mode,
             "manifest_name": manifest_name,
             "system_prompt": system_prompt,
             "project_notes": project_notes,
-            "files": all_files,
         },
         sort_keys=True,
     )
     return hashlib.sha256(payload.encode("utf-8")).hexdigest()[:16]
 
 
+def load_existing_manifest(path):
+    """A previously written manifest, as (entries by file, project, art_style).
+
+    This is what makes a rerun incremental: anything already described here is
+    not sent to Claude again.
+    """
+    if not os.path.isfile(path):
+        return {}, None, None
+    try:
+        with open(path, encoding="utf-8") as f:
+            data = json.load(f)
+    except (OSError, json.JSONDecodeError):
+        return {}, None, None
+
+    entries = {}
+    for entry in data.get("assets") or []:
+        if isinstance(entry, dict) and entry.get("file"):
+            entries[entry["file"]] = entry
+    return entries, data.get("project"), data.get("art_style")
+
+
 def load_checkpoint(root, key):
     path = os.path.join(root, CHECKPOINT_NAME)
     if not os.path.isfile(path):
@@ -267,6 +302,18 @@ class SFClaudeLibraryBuilder:
                         ),
                     },
                 ),
+                "update_existing": (
+                    "BOOLEAN",
+                    {
+                        "default": True,
+                        "tooltip": (
+                            "On: reuse descriptions already in the manifest and only "
+                            "catalogue files that are new, so adding assets costs only "
+                            "those assets. Entries whose file has since been deleted are "
+                            "dropped. Off: describe every asset again from scratch."
+                        ),
+                    },
+                ),
                 "batch_size": (
                     "INT",
                     {
@@ -324,6 +371,10 @@ once, and tracks progress in a hidden checkpoint file. If a run fails partway,
 rerunning with the same settings resumes from where it stopped instead of
 re-paying for batches already done.
 
+With **update_existing** on (the default), rerunning against a library that
+already has a manifest only catalogues files that are new — so adding assets
+costs just those assets. Entries whose file has since been deleted are dropped.
+
 Defaults to **manifest only**, which touches nothing: you get the JSON as text to
 read and save yourself. Switch `write_manifest` on to have it saved into the
 library, or change `mode` to let Claude reorganise and rename the files too.
@@ -340,6 +391,7 @@ Budget about 5 seconds per asset — it opens every image.
         system_prompt,
         manifest_name,
         write_manifest,
+        update_existing,
         batch_size,
         model,
         timeout_seconds,
@@ -368,21 +420,51 @@ Budget about 5 seconds per asset — it opens every image.
         if not assets:
             raise ValueError(f"No image files found in {root}")
 
-        key = checkpoint_key(mode, manifest_name, system_prompt, project_notes, assets)
-        checkpoint = load_checkpoint(root, key)
+        manifest_root = out_dir if mode == MODE_COPY else root
+        manifest_path = os.path.join(manifest_root, manifest_name)
+
         done_by_file = {}
         project = art_style = None
+
+        # An existing manifest is the durable record of what is already described,
+        # so a rerun only pays for assets that are genuinely new. Skipped for the
+        # copy mode, where the manifest holds post-rename paths that cannot be
+        # matched against a scan of the untouched source folder.
+        reused = 0
+        if update_existing and mode != MODE_COPY:
+            existing, prev_project, prev_style = load_existing_manifest(manifest_path)
+            if existing:
+                done_by_file.update(existing)
+                project, art_style = prev_project, prev_style
+                reused = sum(1 for f in assets if f in existing)
+                gone = len(existing) - reused
+                print(
+                    f"[SF Claude Library Builder] Found {manifest_name}: reusing "
+                    f"{reused} existing description(s)"
+                    + (f", dropping {gone} for deleted file(s)" if gone else "")
+                    + "."
+                )
+
+        # A checkpoint from an interrupted run is newer than the manifest, so it wins.
+        key = checkpoint_key(mode, manifest_name, system_prompt, project_notes)
+        checkpoint = load_checkpoint(root, key)
         if checkpoint:
-            project = checkpoint.get("project")
-            art_style = checkpoint.get("art_style")
             for entry in checkpoint.get("done_assets", []):
                 if entry.get("file"):
                     done_by_file[entry["file"]] = entry
+            project = checkpoint.get("project") or project
+            art_style = checkpoint.get("art_style") or art_style
             print(
-                f"[SF Claude Library Builder] Resuming — {len(done_by_file)}/{len(assets)} "
-                f"asset(s) already catalogued from a previous run."
+                f"[SF Claude Library Builder] Resuming an interrupted run — "
+                f"{len(done_by_file)}/{len(assets)} asset(s) already catalogued."
             )
 
+        todo = sum(1 for f in assets if f not in done_by_file)
+        if todo == 0:
+            print("[SF Claude Library Builder] Nothing new to catalogue.")
+        else:
+            print(f"[SF Claude Library Builder] {todo} asset(s) to catalogue.")
+
         # Claude only ever reads images and replies with JSON; the node writes the
         # manifest itself, and the Write tool is never offered. Bash is needed
         # only to move/rename files in the organise modes.
@@ -392,35 +474,34 @@ Budget about 5 seconds per asset — it opens every image.
         total_cost = 0.0
         started = time.time()
 
-        for batch_num, batch_files in enumerate(batches, start=1):
-            remaining = [f for f in batch_files if f not in done_by_file]
-            if not remaining:
-                continue  # fully covered by the checkpoint already
-
-            established = (project, art_style) if project and art_style else None
-            instruction = build_batch_instruction(
-                mode, remaining, len(assets), project_notes, established, out_dir
+        def fail(message):
+            """Stop the run, leaving finished batches safely checkpointed."""
+            save_checkpoint(root, key, project, art_style, list(done_by_file.values()))
+            raise RuntimeError(
+                f"{message} {len(done_by_file)}/{len(assets)} asset(s) are saved in a "
+                f"checkpoint — rerun with the same settings to resume rather than "
+                f"start over."
             )
 
-            command = [binary, "-p", instruction, "--output-format", "json", "--allowedTools", tools]
+        def run_batch(files, label):
+            """One Claude call. Returns (parsed reply, cost, turns)."""
+            established = (project, art_style) if project and art_style else None
+            command = [
+                binary,
+                "-p",
+                build_batch_instruction(
+                    mode, files, len(assets), project_notes, established, out_dir
+                ),
+                "--output-format",
+                "json",
+                "--allowedTools",
+                tools,
+            ]
             if system_prompt.strip():
                 command += ["--append-system-prompt", system_prompt.strip()]
             if model != "default":
                 command += ["--model", model]
 
-            print(
-                f"[SF Claude Library Builder] Batch {batch_num}/{len(batches)} "
-                f"({len(remaining)} asset(s))..."
-            )
-
-            def _fail(message):
-                save_checkpoint(root, key, project, art_style, list(done_by_file.values()))
-                raise RuntimeError(
-                    f"{message} {len(done_by_file)}/{len(assets)} asset(s) are saved in a "
-                    f"checkpoint — rerun with the same settings to resume rather than "
-                    f"start over."
-                )
-
             try:
                 completed = subprocess.run(
                     command,
@@ -431,39 +512,55 @@ Budget about 5 seconds per asset — it opens every image.
                     stdin=subprocess.DEVNULL,
                 )
             except subprocess.TimeoutExpired:
-                _fail(
-                    f"Batch {batch_num}/{len(batches)} did not finish within "
-                    f"{timeout_seconds}s."
-                )
+                fail(f"Batch {label} did not finish within {timeout_seconds}s.")
 
             if completed.returncode != 0:
                 detail = (completed.stderr or completed.stdout or "").strip()[:500]
-                _fail(f"Batch {batch_num}/{len(batches)} failed (exit {completed.returncode}). {detail}")
+                fail(f"Batch {label} failed (exit {completed.returncode}). {detail}")
 
             try:
                 payload = json.loads(completed.stdout)
             except json.JSONDecodeError:
-                _fail(
-                    f"Batch {batch_num}/{len(batches)}: unreadable response: "
-                    f"{completed.stdout.strip()[:300]}"
-                )
+                fail(f"Batch {label}: unreadable response: {completed.stdout.strip()[:300]}")
 
             if payload.get("is_error"):
-                _fail(
-                    f"Batch {batch_num}/{len(batches)} reported an error: "
-                    f"{payload.get('result', 'unknown')}."
-                )
-
-            cost = payload.get("total_cost_usd") or 0.0
-            total_cost += cost
-            turns = payload.get("num_turns", "?")
+                fail(f"Batch {label} reported an error: {payload.get('result', 'unknown')}.")
 
+            # May raise BatchParseError, which catalogue() retries by splitting.
             data = extract_json(payload.get("result", ""))
+            return data, payload.get("total_cost_usd") or 0.0, payload.get("num_turns", "?")
+
+        def catalogue(files, label):
+            """Catalogue `files`, halving and retrying if the reply comes back cut off.
+
+            Each piece that succeeds is merged and checkpointed immediately, so a
+            later failure never costs work already paid for.
+            """
+            nonlocal project, art_style, total_cost
+
+            try:
+                data, cost, turns = run_batch(files, label)
+            except BatchParseError as exc:
+                if len(files) == 1:
+                    fail(
+                        f"Batch {label}: {exc}. This is a single asset "
+                        f"({files[0]}), so it cannot be split any further."
+                    )
+                half = len(files) // 2
+                print(
+                    f"[SF Claude Library Builder] Batch {label}: {exc}. "
+                    f"Retrying as two batches of {half} and {len(files) - half}."
+                )
+                catalogue(files[:half], f"{label}a")
+                catalogue(files[half:], f"{label}b")
+                return
+
+            total_cost += cost
             if project is None:
                 project = data.get("project", "Untitled project")
                 art_style = data.get("art_style", "")
 
-            expected = set(remaining)
+            expected = set(files)
             for entry in data.get("assets") or []:
                 raw = entry.get("file")
                 if not raw:
@@ -474,11 +571,29 @@ Budget about 5 seconds per asset — it opens every image.
             save_checkpoint(root, key, project, art_style, list(done_by_file.values()))
 
             print(
-                f"[SF Claude Library Builder] Batch {batch_num}/{len(batches)} done: "
-                f"{turns} turns, ${cost:.2f}. Running total: "
-                f"{len(done_by_file)}/{len(assets)} asset(s), ${total_cost:.2f}."
+                f"[SF Claude Library Builder] Batch {label} done: {turns} turns, "
+                f"${cost:.2f}. Running total: {len(done_by_file)}/{len(assets)} "
+                f"asset(s), ${total_cost:.2f}."
             )
 
+            skipped = [f for f in files if f not in done_by_file]
+            if skipped:
+                print(
+                    f"[SF Claude Library Builder] Batch {label}: {len(skipped)} file(s) "
+                    f"came back undescribed; a rerun will pick them up."
+                )
+
+        for batch_num, batch_files in enumerate(batches, start=1):
+            remaining = [f for f in batch_files if f not in done_by_file]
+            if not remaining:
+                continue  # fully covered by the checkpoint already
+
+            print(
+                f"[SF Claude Library Builder] Batch {batch_num}/{len(batches)} "
+                f"({len(remaining)} asset(s))..."
+            )
+            catalogue(remaining, f"{batch_num}/{len(batches)}")
+
         elapsed = time.time() - started
 
         # Assemble the final manifest in the library's original scan order.
@@ -492,8 +607,6 @@ Budget about 5 seconds per asset — it opens every image.
         }
         manifest_text = json.dumps(manifest_data, indent=2)
 
-        manifest_root = out_dir if mode == MODE_COPY else root
-        manifest_path = os.path.join(manifest_root, manifest_name)
         if write_manifest:
             os.makedirs(manifest_root, exist_ok=True)
             with open(manifest_path, "w", encoding="utf-8") as f:
@@ -505,11 +618,13 @@ Budget about 5 seconds per asset — it opens every image.
         for entry in final_assets:
             counts[entry.get("category", "?")] = counts.get(entry.get("category", "?"), 0) + 1
 
+        newly_done = max(0, len(final_assets) - reused)
         report = "\n".join(
             [
                 f"mode:        {mode}",
                 f"scanned:     {len(assets)} file(s) in {root}",
-                f"catalogued:  {len(final_assets)} asset(s) across {len(batches)} batch(es)",
+                f"in manifest: {len(final_assets)} asset(s)",
+                f"reused:      {reused} already described, {newly_done} newly catalogued",
                 "categories:  " + (", ".join(f"{k} x{v}" for k, v in sorted(counts.items())) or "none"),
                 f"art style:   {manifest_data['art_style'] or '(not set)'}",
                 "manifest:    " + (manifest_path if write_manifest else "returned as text, not saved"),

pyproject.toml

@@ -1,7 +1,7 @@
 [project]
 name = "comfyui-sf-nodes"
 description = "Custom nodes for ComfyUI including LLM Chat with support for Claude, Gemini, and OpenAI models, WaveSpeed AI image/video generation, VertexAI integration, and workflow utilities"
-version = "1.3.1"
+version = "1.4.0"
 license = { file = "LICENSE" }
 dependencies = [
     "requests>=2.31.0",