MASTER_SPEC.md lived outside version control (in the parent directory), so it had
no history and the README's ../MASTER_SPEC.md link was broken on GitHub. It is now
tracked at the repository root and the links are fixed.
Content updated from 'Version 1 / Draft' to 'Version 2 / Maintained'. Chapter and
section numbers are deliberately unchanged so the wiki's existing § references stay
valid; superseded chapters are marked in place rather than deleted:
- New 'Document Status & Version History' section with a per-release change table.
- §3: three new design principles capturing the V2 shift — 3.13 Fidelity,
3.14 Automation Without Gates, 3.15 Ask Only What Matters.
- §8: pipeline reduced to three stages; stages 4-5 retained as historical.
- §17 Resolver: now the final stage. Adds modifier preservation, the config-gated
deduplication, the opt-in semantic fallback with its four normative guarantees,
prompt translation, and the output-preserving performance rules.
- §18/§19: Category Splitter and Prompt Builder marked superseded; the category
list stays normative as traceability metadata; new §19.7 specifies the flat prompt.
- §20: eight nodes -> five, with real inputs/outputs and the KB Editor constraints.
- §23: current config document, plus a new §23.4 recording which fields are
deliberately not exposed and why.
- §15: new §15.13 specifying automatic KB construction with no approval gate.
- §13: new §13.13 fidelity rules (lists, relations, conjunctions, explicit content).
- §9/§11: Category Map and Prompt Output data model + schemas marked removed;
Knowledge Base Manifest schema added.
- Appendix B: documents SC0015-SC0022.
- §30: V1 and V2 marked delivered; records that the automatic KB builder shipped
deliberately without the originally specified manual-validation gate, that
localization was dropped, and that the Entity Graph was not pursued.
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
The docs had drifted badly: docs/nodes.md still documented the Category Splitter
and Prompt Builder (removed in V1.1) and the Knowledge Base Loader (removed in
V2.1), gave the Resolver the wrong outputs, and the consistency test actively
pinned the retired RESOLVED_TAGS/CATEGORY_MAP types.
- README: rewritten around the V2 philosophy shift — 'know more, ask less'.
Documents the automatic Knowledge Base growth, the opt-in KB-bounded semantic
fallback and the fidelity work (know more), and the configuration
centralization, node consolidation (eight nodes -> five) and hidden
non-user-facing options (ask less), plus what did not change: determinism.
Corrects the tagline (flat prompt, not category-separated) and the node tables.
- docs/nodes.md: rewritten for the five shipped nodes with their real inputs and
outputs, plus a full Configuration input reference and the settings that are
deliberately not exposed.
- docs/definition_of_done.md: V1 record marked complete; evidence re-pointed at
the code as it stands (no compiler/splitter, compiler/builder).
- docs/README.md: MASTER_SPEC reframed as the original design spec (code wins);
index the in-repo docs.
- examples/README.md: pipeline diagram redrawn for the current graph.
- tests/test_docs_consistency.py: stop requiring the retired inter-stage types and
add a guard so removed nodes cannot reappear in the reference.
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Bump version 2.0.0 -> 2.1.0 and document the node consolidation (Knowledge Base
Loader folded into Configuration) and configuration cleanup (trimmed inputs,
plain-language tooltips). Update the release-readiness checks.
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Remove settings the user should not have to touch, and make every tooltip
plain-language:
- knowledge_base path: removed; the Knowledge Base ships in the package, so the
path is fixed rather than a prompted input.
- prompt_target: removed (reserved; to be replaced in V3).
- prompt_separator: removed; always ',' this version. Both keep their compiler
defaults, just no longer exposed as node inputs.
- analyzer_system_prompt (Configuration) and system_prompt (Scene Analyzer):
removed. A user-authored system prompt is not UX-friendly; the built-in analyzer
prompt is always used.
- Rewrite all node tooltips in plain language reflecting the current nodes.
Example workflow and node tests updated; docs/nodes.md refreshed.
BREAKING CHANGE: the Configuration node's knowledge_base, prompt_target,
prompt_separator, and analyzer_system_prompt inputs and the Scene Analyzer's
system_prompt input are removed.
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
The Knowledge Base directory was entered in two places — the Configuration node's
knowledge_base path and the Knowledge Base Loader node's path — a duplication with
no benefit. Remove the Knowledge Base Loader node and have the Configuration node
load the Knowledge Base itself, emitting it on a new knowledge_base output wired
into the Resolver.
- New nodes/kb_loading.py holds the shared, process-wide load-once cache and
package-relative path resolution (extracted from the deleted loader node).
- Configuration node: gains a knowledge_base_reload input and a KNOWLEDGE_BASE
output; RETURN is now (config, knowledge_base, warnings, errors). A KB load
failure surfaces on errors while the config is still emitted (the Analyzer and
Validator do not need the KB).
- Remove SceneCompilerKnowledgeBaseLoader from the node registrations.
- Rebuild the example workflow to wire Configuration.knowledge_base -> Resolver.
- Docs (README, docs/nodes.md) updated; tests moved to test_kb_loading.py.
BREAKING CHANGE: the Knowledge Base Loader node is removed. Workflows that used it
must load the Knowledge Base via the Configuration node's knowledge_base output.
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Bump version 1.1.0 -> 2.0.0, finalize the V2 changelog section (add analyzer
config centralization, example-workflow fix, and relational fidelity entries;
note the breaking removals), and update the release-readiness checks.
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* feat(nodes): centralize duplicated settings on Configuration + add tooltips
The Scene Analyzer built its own Config from local model_name/temperature/
timeout/system_prompt widgets and ignored the Configuration node entirely, so
the analyzer model was set in two disconnected places (bug report #4).
- Scene Analyzer: add an optional `config` (COMPILER_CONFIG) input and drop the
duplicated model/temperature/timeout widgets. Settings now come from the
Configuration node; without one, defaults apply with a 300 s cold-load-friendly
timeout. A non-empty `system_prompt` widget still overrides the config.
- Knowledge Base Loader: accept an optional `config`; when wired, its KB path and
version take precedence over the node's own widgets.
- Add tooltips to every input on all six nodes.
Docs (docs/nodes.md) updated for the Analyzer/KB Loader input changes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(examples): refresh workflow widget values; guard against widget drift
The shipped example workflow carried stale positional widget values from before
V2 removed prompt_builder.trim_empty_outputs from the Configuration node. ComfyUI
stores widgets positionally, so every later value shifted: debug_level received a
leftover boolean ("false") and, in older saved workflows, the prompt separator
received a boolean ("true") instead of ",".
- Rebuild the Configuration node's 21 widget values in the current order.
- Update the Analyzer node to the current shape (natural_language + system_prompt;
config now arrives via a wired COMPILER_CONFIG input) and wire the Configuration
node into it.
- Add the KB Loader's version widget value.
- New regression test: every example-workflow node's widgets_values length must
match the node's current widget count, so this drift cannot recur silently.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
The Analyzer collapsed relational phrasing to the bare noun ("a bunch of money
in her hand" -> just "money", losing the held relationship) and could drop the
later half of "X and Y" enumerations, especially on NSFW input. The reference
Knowledge Base already covers the relational tags (holding money, hand on own
hip, in mouth, 400+ holding_* tags), so the loss was entirely at the analysis
step.
- System prompt: add "Relations, positions, and holding", "Conjunctions and
enumerations", and "Completeness (including explicit content)" sections, and
cross-reference them from "Objects". Held/worn/positioned objects now also
yield the relational concept (holding <object>); every conjunct survives;
explicit content gets the same full fidelity.
- Regression tests: relational (holding money, hand on own hip) and conjunction
(holding sword + holding shield) cases prove the reference KB resolves these
concepts end to end, so the improved Analyzer output compiles.
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
The Scene Analyzer built its own Config from local model_name/temperature/
timeout/system_prompt widgets and ignored the Configuration node entirely, so
the analyzer model was set in two disconnected places (bug report #4).
- Scene Analyzer: add an optional `config` (COMPILER_CONFIG) input and drop the
duplicated model/temperature/timeout widgets. Settings now come from the
Configuration node; without one, defaults apply with a 300 s cold-load-friendly
timeout. A non-empty `system_prompt` widget still overrides the config.
- Knowledge Base Loader: accept an optional `config`; when wired, its KB path and
version take precedence over the node's own widgets.
- Add tooltips to every input on all six nodes.
Docs (docs/nodes.md) updated for the Analyzer/KB Loader input changes.
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* perf: KB load-once cache, compiled index, lazy loading, parallel resolution (#38)
Closes#130, #131, #132, #133, #134.
- Loader node routes through a process-wide KnowledgeBaseLoader cache so the KB
loads once per (path, version) and is reused across run() calls.
- Normalized lookup table compiled once per KB (per NSFW gating) and reused across
resolves instead of rebuilt per resolve_scene. normalize_concept moved to
knowledge_base (re-exported from the resolver).
- Incremental per-category lazy file loading behind KnowledgeBaseLoader.
- Optional parallel per-concept resolution (thread pool), merged in exact
discovery order; deterministic and threshold-gated (default sequential).
- Performance regression guard (generous ceilings + relative ratio). Refreshed
docs/benchmarks.md (resolution 57.6us -> 17.8us).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(changelog): note V2 performance improvements (#38)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* feat(kb-editor): web KB editor with live validation & atomic saves (#33)
Closes#123, #124, #125, #126.
- Backend API + transport-independent handlers under /scene-compiler/kb,
registered from the package init and guarded so it no-ops without ComfyUI.
- Static self-contained web page (list/search/filter/create/edit/delete) with
live validation and save blocking on errors.
- Validation reuses kb_validation (new validate_entry / validate_cross_entry) as
the single source of truth, against the candidate and the whole-KB state.
- Atomic (temp+fsync+rename), format-safe saves: only the edited entry changes,
key order and trailing newline preserved, curated-vs-generated split respected.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(kb-editor): associate form labels with inputs; await load (SonarCloud a11y)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(kb-editor): label sidebar controls; async IIFE for init (SonarCloud)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Closes#114, #115, #116, #117.
- Deterministic, offline, dependency-free char-ngram embedding backend + config
section (semantic.enabled default false, min_similarity, backend).
- Committed embedding index (data/kb_embedding_index.json) mapping vectors to real
KB entry ids only; built by scripts/build_embedding_index.py, loaded once/cached.
- Resolver falls back to the nearest entry on a deterministic miss when enabled,
accepting >= min_similarity and emitting SC0020; NSFW-gated neighbours are
skipped. Disabled => byte-identical to before.
- Guarantee tests: deterministic lookup always wins (exact hit never queries the
index), fallback emits only KB tags (no invention), identical input => identical
output. SC0020 registered.
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Closes#111, #112, #113.
- Resolver recovers leading modifiers dropped by the head-noun reduction as
<modifier> <head-noun> compounds (open white shirt -> white shirt + open
shirt). KB-only, deterministic, no bare-word invention; goldens unchanged.
- Analyzer system prompt instructs full list transcription; a deterministic
post-parse check warns SC0021 when a tag-list yields fewer concepts than items.
- Data-driven fidelity regression suite locks both failure modes.
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
remove_duplicate_tags now gates the Resolver's post-expansion dedup (SC0007);
trim_empty_outputs was inert since the V1.1 category removal and is dropped
from config, schema, node, and default_config.
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* feat!: collapse pipeline — Resolver emits a flat prompt, drop categories (v1.1)
BREAKING CHANGE: the Category Splitter and Prompt Builder stages and nodes are
removed. The Resolver is now the translator from Scene JSON to prompt: it resolves
concepts to Knowledge Base tags and joins them into one flat comma-separated
prompt string, in resolution order (which naturally leads with character concepts).
Resolver node outputs are now (prompt, warnings, errors, json) — the json being
the traceable resolved tags so a translation problem is inspectable. Node count
drops from 8 to 6. The CategoryMap/PromptOutput models and their schemas are
removed; the `category` field remains on entries and resolved tags for
traceability only. Debug Viewer simplified to scene/warnings/errors.
Goldens, example scenes/workflow, integration/performance/repo-layout/release
tests, benchmark script, and the docs (README + wiki) are updated to the flat
prompt. Version bumped to 1.1.0.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* build: drop removed splitter/builder packages from pyproject
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Audit follow-ups touching shipped code:
- Configuration node now exposes resolver_include_nsfw (the NSFW gate we added
was unreachable from ComfyUI) and its analyzer_timeout default is aligned to
300s to match the Scene Analyzer node.
- README: note the large Danbooru-derived KB, clarify the determinism scope
(compiler core deterministic; LLM analysis is the one non-deterministic step),
add the V4 (consolidation) roadmap entry, and an AI-assistance disclaimer.
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Observed SC0011: the model emitted objects as [{"id":0,"concept":"book"}]
instead of plain concept strings, failing Scene validation. The system prompt
never showed a populated objects array (the example has objects: []), so the
model reused the character/interaction {id, ...} shape.
Add an Objects section with concrete examples (book, umbrella) and state
explicitly that the scene-level arrays (objects/environment/camera/lighting) hold
plain concept strings, never {id, concept} objects. Prevention layer on top of
the retry temperature escalation (recovery).
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(analyzer): escalate temperature on retries to self-recover bad JSON
Ollama is non-deterministic even at temperature 0 (no fixed seed, CUDA/batch
variance), so the analyzer occasionally emits non-schema JSON that fails parsing
(SC0011 -> null Scene). The retry loop reused the prompt unchanged at the same
temperature, so all internal retries reproduced the same failure -- only a fresh
manual re-run recovered.
Escalate the sampling temperature on retry attempts only (base -> +0.2 per
retry, capped at 1.0); the prompt is still byte-for-byte unchanged (§12.9) and
the first attempt keeps the configured temperature, so the happy path stays
deterministic while a malformed run can now self-recover.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test(analyzer): use pytest.approx for temperature assertions (SonarCloud S1244)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Local Ollama cold-loads the model into VRAM on the first call, so 60s often
times out (SC0013 -> no Scene) for longer prompts. Bump the ComfyUI node input
default to 300s. The library/spec default (AnalyzerConfig.timeout, MASTER_SPEC
§23.2) is left at 60 unchanged.
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix: recover concept detail lost in resolver and harden analyzer parsing
Three shippability fixes for the v1 pipeline:
P2 (main): the Illustrious Resolver did exact-key KB lookup only, so any
compound concept the analyzer emits ("white summer dress", "black business
suit") missed and was dropped silently as SC0001. Add a deterministic
head-noun fallback (§17.2): on a miss, drop leading modifiers and look up the
longest remaining suffix ("white summer dress" -> "dress"). No tag/concept is
invented. Emits SC0019 (Concept Reduced) so the reduction is visible.
P1a: the analyzer required the entire model response to be a single JSON
document, so any surrounding prose ("Sure! Here is the scene:") failed parsing
and, with the prompt reused unchanged, every retry failed identically ->
empty output. Extract the first balanced top-level {...} object from
surrounding text (string-aware brace scan); the object bytes are unchanged and
must still validate, so this is unwrapping, not JSON repair.
P1b: the node's "raw" output was to_raw_json(result.data) -- a re-serialization
of the parsed Scene, so on failure it was empty and the model's actual text was
invisible. Capture the last raw model text in analyzer metadata and surface it
as the node's raw output.
Regenerated goldens: unknown_concepts "hologram jacket" now resolves to
"jacket" via head-noun fallback (intended).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat: add entry rating + resolver.include_nsfw gating
Foundations for the Danbooru-vocab KB: a per-entry `rating` (general|explicit,
default general) and a `resolver.include_nsfw` config flag (default false). The
resolver omits explicit-rated entries from lookup and expansion unless enabled,
so a gated concept is reported as unknown (SC0001). Schema, model, config, and
example config updated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat: generate KB from embedded Danbooru vocab (188 -> 16.4k tags)
Replace the coverage ceiling of the 188-entry hand KB by importing a large,
validated Danbooru tag vocabulary. This is the fix for the core "detail loss"
symptom: compound concepts the analyzer emits ("red dress", "sunflower field",
"train station platform") are now real tags that resolve instead of being
dropped.
- scripts/build_vocab.py: one-time dev step that filters the Raffle node's
categorized tag dump (drops artist/character/copyright/attribution categories
and subjective adjectives, keeps only id-pattern-valid tags) into a committed,
self-contained snapshot at data/danbooru_vocab.txt (16.4k tags).
- scripts/generate_kb_from_vocab.py: additive, non-destructive generator. Writes
knowledge_base/gen_<category>.json, mapping the Raffle categories to the 19
canonical categories and tagging NSFW categories rating=explicit. Skips any tag
already defined as a hand-curated id/alias, so the curated canonicalization
(female + aliases) and expansions (school_uniform -> blazer, pleated_skirt) are
preserved and always win.
Result: 16.4k additive entries + 188 curated. Plain natural-language concepts
now resolve broadly via curated aliases + head-noun fallback with no analyzer
prompt change; NSFW gated behind resolver.include_nsfw; golden and dataset tests
stay green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* style: ruff format build_vocab.py
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(security): drop untrusted CLI path in build_vocab (SonarCloud S8707)
The --source CLI argument fed an untrusted path to the file system (taint).
The script is a one-time dev step; hard-code the source path constant instead.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Two ComfyUI issues found while testing.
1. Empty output with no error: the Knowledge Base Loader resolved a relative
`path` against ComfyUI's working directory, not this package, so the default
`knowledge_base/` loaded 0 entries (an empty KB is valid) and every concept
became unknown -> empty prompts. Relative paths now resolve against the
package root, and the loader warns when the Knowledge Base is empty.
2. Add a `raw` string output (a JSON dump of the node's data) to the Scene
Analyzer, Scene Validator, Resolver, Category Splitter, and Prompt Builder
nodes, plus a summary `raw` on the Knowledge Base Loader, for inspecting the
pipeline in the graph. Adds a shared to_raw_json helper.
Tests: package-relative path resolution (independent of cwd), empty-KB warning,
raw output content, and updated node metadata/return-arity expectations.
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(nodes): handle a failed upstream stage without crashing
When a stage fails it passes None down its data output. Downstream nodes called
`.to_json()` / iterated the None and raised an unhelpful AttributeError (found in
ComfyUI: a failed Scene Analyzer crashed the Scene Validator on
`None.to_json()`).
Each consuming node now short-circuits on a missing required input: the
Validator, Resolver, and Category Splitter return None plus an error string that
points at the upstream node's errors output; the Prompt Builder returns empty
strings. Adds a shared upstream_failure_message helper and regression tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(analyzer): unwrap Markdown code fences in model responses
Most local models (e.g. gemma) return valid Scene JSON wrapped in a ```json
... ``` code fence despite the system prompt forbidding markdown. Strict
json.loads then rejected it (SC0011), and at temperature 0 every retry returned
the identical fenced output, so the Analyzer always failed and produced a None
scene.
The response parser now unwraps a leading/trailing code fence before parsing.
This is not the heuristic JSON repair forbidden by MASTER_SPEC §12.9: the JSON
inside is left byte-for-byte unchanged; only the surrounding ``` markers are
removed. Malformed JSON inside a fence is still rejected.
Verified end-to-end against a live Ollama (gemma): the fenced response now parses
and the full pipeline produces prompt outputs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Finalize packaging and release readiness (MASTER_SPEC Phase 8, §7.8, §28).
- pyproject.toml: bump version to 1.0.0; ship the JSON Schemas as package data
and enable include-package-data.
- MANIFEST.in: include the data directories (schemas, knowledge_base, prompts,
config, examples, docs) in the source distribution.
- CHANGELOG.md: 1.0.0 release notes.
- docs/definition_of_done.md: the §7.8 Definition-of-Done checklist mapped to
evidence, with the ComfyUI visual check and the tag/release step flagged as the
remaining manual steps.
- tests/test_release_readiness.py: assert the version, changelog, DoD doc, and
that every schema / Knowledge Base / config / prompt data file is present and
declared for packaging.
Verified: `python -m build --sdist` produces comfyui_scene_compiler-1.0.0 with
all 13 Knowledge Base files, 10 schemas, the system prompt, and the default
config bundled. Tagging v1.0.0 and publishing the release are intentionally left
to a human after the ComfyUI visual check.
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* docs: verify and complete project documentation (#31)
Align the docs with the shipped V1 implementation (MASTER_SPEC §7.6, Phase 7).
- README: fix the Category Splitter output (it emits a Category Map; the
per-category strings are produced by the Prompt Builder).
- docs/nodes.md: developer node reference with the exact inputs, outputs, and
inter-stage data types for all eight nodes.
- CONTRIBUTING.md: link the Knowledge Base contribution guide and validation
command, the golden-regeneration note, and the node/benchmark references.
- tests/test_docs_consistency.py: guards that README and docs/nodes.md mention
every registered node and cover the inter-stage types, and that CONTRIBUTING
links the Knowledge Base guide -- so docs cannot drift from the code.
The wiki ComfyUI-Nodes page is updated to match (Category Splitter emits a
Category Map) in the wiki repository.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* chore: retrigger CI (runner backlog)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Ship ready-to-use examples (MASTER_SPEC §24, Phase 7).
- examples/workflows/scene_compiler_pipeline.json: a ComfyUI workflow wiring
Configuration + Knowledge Base Loader + Analyzer -> Validator -> Resolver ->
Category Splitter -> Prompt Builder.
- examples/scenes/*.scene.json: standalone example Scene JSON documents
(single character; two characters with an interaction and an expansion).
- examples/scenes/*.prompts.json: the exact prompt outputs each scene compiles to
with the shipped Knowledge Base and default config.
- examples/README.md: how to load the workflow and use the scene examples.
tests/test_examples.py asserts each example Scene JSON compiles to its documented
outputs, and that the workflow references only registered Scene Compiler nodes
and includes the full pipeline. The workflow JSON still wants a ComfyUI visual
check.
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Add performance benchmarks for the non-Analyzer pipeline (MASTER_SPEC §26.6,
NFR-004) plus tests.
- scripts/benchmark.py: measures Knowledge Base load time, single-scene
resolution time, and full pipeline time (median over iterations) and records a
baseline to docs/benchmarks.md.
- docs/benchmarks.md: recorded baseline numbers (indicative, machine-dependent).
- tests/test_performance.py: a benchmark smoke test (functions run, all metrics
positive, a generous non-flaky bound), a check that the recorded baseline
exists, and two tests confirming the Knowledge Base is loaded once and reused
(the loader caches; the pipeline never reloads per compilation).
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Add golden tests for the ten reference scenes (MASTER_SPEC §26.7, §26.8),
guarding determinism across changes.
- tests/regression/golden_scenes.py: the ten reference scenes (single character,
two characters, interaction, indoor, outdoor, fantasy, modern, unknown
concepts, ambiguous concepts, complex multi-character) plus the shared no-LLM
pipeline runner.
- tests/regression/golden/*.json: the golden prompt outputs for each scene.
- scripts/regenerate_goldens.py: the deliberate, documented way to regenerate the
goldens when a change is intended to alter output (never automatic).
- tests/regression/test_golden.py: asserts exact equality against the goldens and
determinism across repeated runs.
- tests/regression/golden/README.md: documents regeneration and the regression
convention (every fixed bug adds a permanent test that is never removed).
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Add tests/integration/test_pipeline.py exercising the no-LLM pipeline end to end
(MASTER_SPEC §26.2): Scene JSON -> Validator -> Resolver -> Category Splitter ->
Prompt Builder against the shipped Knowledge Base. The Analyzer is bypassed with
fixed Scene JSON inputs, so no test needs a running Ollama.
- Covers the reference scenes: single character, two characters, character
interaction, indoor, and outdoor.
- Asserts the expected per-category prompt strings, no errors, and no
unknown-concept warnings for core concepts.
- Asserts deterministic outputs across repeated runs, and that expansion
(school_uniform -> blazer + pleated_skirt) flows end to end.
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Add nodes/configuration_node.py: ConfigurationNode, a thin adapter that
centralizes compiler configuration (MASTER_SPEC §20, §23).
- Exposes all §23.2 options as node inputs (analyzer model/temperature/
max_retries/timeout + optional system_prompt; resolver knowledge_base/
strict_mode/allow_aliases/expansion_enabled/max_expansion_depth; validator
allow_unknown_fields; prompt_builder target/separator/trim_empty_outputs/
remove_duplicate_tags; debug enabled/level).
- Builds a Config via Config.from_json and outputs (config, errors) as a
COMPILER_CONFIG. The Validator, Resolver, and Prompt Builder nodes already
accept this via their optional config input, so behaviour changes without
editing the workflow. No compiler logic in the node.
- Registered as SceneCompilerConfiguration ("Configuration").
Tests: metadata/inputs, defaults reproduce Config(), inputs flow into the config,
system-prompt override + omission, all debug levels valid, and registration.
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* feat(nodes): add the Knowledge Base Loader ComfyUI node (#25)
Add nodes/knowledge_base_loader_node.py: KnowledgeBaseLoaderNode, a thin adapter
that delegates to load_knowledge_base (MASTER_SPEC §20).
- Input path (STRING, default knowledge_base/) plus an optional reload (INT):
bumping reload changes the inputs so ComfyUI re-executes the node -- an
explicit manual reload with no automatic file watching.
- Outputs (knowledge_base, warnings, errors): the KNOWLEDGE_BASE for the Resolver
node, or None with the surfaced SC0004 message and findings on failure. No
compiler logic in the node.
- Registered as SceneCompilerKnowledgeBaseLoader ("Knowledge Base Loader").
Tests: metadata/inputs, successful load, SC0004 surfacing on an invalid KB,
reload input accepted, and registration.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* ci: run push only on main and add concurrency guard
Feature-branch CI now runs via the pull_request event only (push runs on main),
eliminating duplicate runs per PR; a concurrency group cancels superseded runs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Add nodes/debug_viewer_node.py: DebugViewerNode, a read-only inspector
(MASTER_SPEC §20), plus render_debug_report in nodes/adapters.py.
- Optional inputs for each intermediate state: scene (SCENE), resolved_tags
(RESOLVED_TAGS), category_map (CATEGORY_MAP), warnings and errors (STRING).
- Renders whichever are connected into a single `report` STRING via each value's
to_json(); absent inputs are skipped. Terminal viewer (OUTPUT_NODE); read-only,
never modifies the data passing through. No compiler logic in the node.
- Registered as SceneCompilerDebugViewer ("Debug Viewer").
Tests: metadata/inputs, empty report, rendering each state (with empty sections
omitted), valid serialization, read-only guarantee, and registration.
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Add nodes/prompt_builder_node.py: PromptBuilderNode, a thin adapter that
delegates to build_prompts (MASTER_SPEC §20, §19).
- Input category_map (CATEGORY_MAP); optional config (COMPILER_CONFIG) defaulting
to Config().
- Emits 21 STRING outputs: one per canonical category (in order) plus the
reserved `negative` and `scene` outputs (always empty in V1). Values are keyed
by output name so wiring stays stable. No compiler logic in the node.
- Registered as SceneCompilerPromptBuilder ("Prompt Builder").
Tests: 21-output metadata, comma formatting + ordering + empty-category strings,
reserved-empty outputs, custom separator via config, and registration.
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Add nodes/category_splitter_node.py: CategorySplitterNode, a thin adapter that
delegates to split_into_categories (MASTER_SPEC §20).
- Input resolved_tags (RESOLVED_TAGS); outputs (category_map, warnings, errors):
the CATEGORY_MAP (per-category grouping, correct assignment + order) plus
surfaced diagnostics. No compiler logic in the node.
- The per-category prompt strings are produced downstream by the Prompt Builder
node (§19); this node produces the intermediate Category Map that feeds it.
- Registered as SceneCompilerCategorySplitter ("Category Splitter").
Tests: metadata/inputs, real split into a CategoryMap, delegation + SC0008
surfacing (monkeypatched), and registration.
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Add nodes/resolver_node.py: ResolverNode, a thin adapter that delegates to
resolve_scene (MASTER_SPEC §20).
- Required inputs scene (SCENE model) and knowledge_base (KNOWLEDGE_BASE, from
the loader node); optional config (COMPILER_CONFIG) defaulting to Config().
- Outputs (resolved_tags, warnings, errors): RESOLVED_TAGS plus surfaced
diagnostics. No compiler logic in the node.
- Registered as SceneCompilerResolver ("Resolver").
Tests: metadata/inputs, real resolution over a small in-memory Knowledge Base,
delegation + message surfacing (monkeypatched), default config, registration.
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Add nodes/scene_validator_node.py: SceneValidatorNode, a thin adapter that
delegates to validate_scene (MASTER_SPEC §20).
- Required input scene (SCENE model, serialized via to_json); optional config
(COMPILER_CONFIG) defaulting to Config() when unconnected.
- Outputs (scene, warnings, errors): the validated SCENE plus warnings/errors
passed through unchanged. No compiler logic in the node.
- Registered as SceneCompilerValidator ("Scene Validator").
Tests: node metadata/inputs, valid scene pass-through, delegation + message
surfacing + config threading (monkeypatched), default config when absent, and
ComfyUI registration.
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Expose the Scene Analyzer as a ComfyUI node (MASTER_SPEC §20) plus the node
registration plumbing and a shared output adapter.
- nodes/scene_analyzer_node.py: SceneAnalyzerNode, a thin adapter. Inputs are
Natural Language, Model Name, Temperature, Timeout, and an optional System
Prompt override; it builds a Config, constructs OllamaBackend.from_config, and
delegates to analyze(). Outputs (scene, warnings, errors): the SCENE model and
the warnings/errors formatted for display. No compiler logic in the node.
- nodes/adapters.py: format_messages() renders diagnostics as CODE: description
lines.
- Registration: nodes/__init__.py aggregates NODE_CLASS_MAPPINGS /
NODE_DISPLAY_NAME_MAPPINGS; the root __init__.py re-exports them.
ComfyUI compatibility: the node package uses relative imports internally so our
`nodes` package is never resolved as ComfyUI's own top-level `nodes` module, and
the entry point puts the repo root on sys.path so `compiler`/`schemas` resolve
when ComfyUI loads the package by file path.
Tests (no live Ollama): INPUT_TYPES/RETURN metadata, delegation to analyze
(monkeypatched) with warnings/errors surfaced, empty-message strings, system
prompt override passthrough, and ComfyUI registration.
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Add compiler/analyzer/scene_analyzer.py: analyze(description, backend, config,
logger=None) -> CompilerResult, per MASTER_SPEC §12.9, §12.10.
- Composes the prompt once (resolved system prompt + user description) and reuses
it unchanged on every attempt; JSON is never repaired heuristically.
- Bounded retries: total attempts = max_retries + 1 (1 initial + up to N
retries, default 3). A bad response (parse/schema failure SC0011, or an
unexpected backend response SC0018) is retried; the last such error is returned
once attempts are exhausted.
- Terminal failures return immediately: connection failure -> Fatal SC0012;
timeout -> SC0013 (no retry).
- Attaches telemetry to the result metadata (model, attempts, retries, duration,
validation status) and logs the outcome.
Tests (mocked backend): first-attempt success, succeed-on-retry, prompt reused
unchanged, prompt includes system prompt + description, exhaust-retries -> SC0011
with the right call count, zero-retries single attempt, connection -> SC0012 (no
retry), timeout -> SC0013 (no retry), and telemetry.
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Add compiler/analyzer/response_parser.py: parse_scene_response(text, logger=None)
-> CompilerResult, per MASTER_SPEC §12.2, §12.8.
- Strictly json.loads the model response; JSON is never repaired heuristically
(§12.9), so malformed or markdown-fenced output fails here (feeding the retry
path in #18).
- Validates the parsed document against the Scene schema; any failure (malformed
JSON, non-object, missing field, wrong type) is reported as SC0011 and stops
the compilation -- invalid responses are never silently accepted.
- Handles both object and string concept forms via the Scene model; confidence
stays metadata-only.
- Stamps a default schema_version onto compiler-owned metadata (the model emits
`"metadata": {}`) before validation.
Tests: valid response -> Scene, metadata stamping, object/string concepts +
confidence, and rejection (SC0011) of malformed JSON, non-object JSON, missing
fields, wrong types, and markdown-fenced (unrepaired) output.
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Ship the official Analyzer system prompt and make it loadable/overridable, per
MASTER_SPEC §13.
- prompts/analyzer_system_prompt.md: the official prompt assembled from §13.1-
§13.12 (deterministic semantic parser; extract explicit concepts only; never
generate tags or invent information; ignore subjective/artistic language;
explicit camera/lighting/environment only; independent characters with
scene-level interactions; preserve unknown concepts; return valid JSON only,
no markdown) plus the §12.2 Scene JSON output shape and example.
- compiler/analyzer/system_prompt.py: load_default_system_prompt() reads the
shipped default; resolve_system_prompt(config) returns config.analyzer.
system_prompt when set, otherwise the default.
- Add optional analyzer.system_prompt to the config model and schema (nullable).
Tests: default file loads, a parametrized drift guard asserting every required
§13 directive is present, output-shape keys present, config override and default
resolution, and config round-trip of the override.
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* feat(analyzer): add the Ollama Analyzer backend (#15)
Add compiler/analyzer/backend.py per MASTER_SPEC §12.3, §12.10.
- AnalyzerBackend Protocol keeps the LLM backend swappable with no downstream
coupling; the module imports no downstream compiler stage.
- OllamaBackend talks to a local Ollama /api/generate endpoint using only the
standard library (urllib). The HTTP call is an injectable transport, so the
backend is fully unit-testable without a live Ollama.
- Sends the configured model, temperature (default 0.0), and the user prompt
unmodified, applying the configured timeout. BackendResult exposes the response
text, model name, and request duration for logging.
- Failure modes map to codes: connection failure -> BackendUnavailableError
(SC0012, Fatal); timeout -> BackendTimeoutError (SC0013); HTTP error /
unexpected response -> BackendResponseError (SC0018, new extension).
Adds SC0018 (Analyzer Unexpected Response) to the message-code registry.
Tests (no live Ollama): interface conformance, from_config, success with
telemetry + unmodified prompt, and timeout/connection/URLError/HTTP/malformed
response mapping.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test(analyzer): use pytest.approx and single throwing call in backend tests
Avoid float-equality on temperature (SonarCloud reliability) and construct the
backend outside pytest.raises so each block has a single throwing invocation.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Add compiler/builder/prompt_builder.py: build_prompts(category_map, config,
logger=None) -> CompilerResult (data = tuple[PromptOutput]), per MASTER_SPEC §19.
- Emits one output per canonical category (in order) plus reserved `negative`
and `scene`; V1 always emits all 21 outputs for stable node sockets.
- Tags joined with config.prompt_builder.separator (default comma, no surrounding
whitespace, no trailing separator). Empty categories yield an empty string,
never null/placeholder (§19.4).
- Formatting only: no tag generation, removal, reordering, dedup, or escaping
(§19.1, §19.5). Output names are the lowercase canonical category ids and
`negative`/`scene`, mapping to the §19.2/§19.3 display names.
Tests: output set (19 + 2 reserved), comma join, order preservation, empty
strings, reserved-empty, custom separator, no-trailing-separator, no escaping,
determinism.
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Add compiler/splitter/category_splitter.py: split_into_categories(tags,
logger=None) -> CompilerResult (data = CategoryMap), per MASTER_SPEC §18.
- Each Resolved Tag is placed in the category recorded on it by the Resolver
(its Knowledge Base Entry category); tags are never modified, generated, or
removed (original objects kept verbatim).
- Tags within a category preserve Resolver order; category ordering follows the
shared canonical 19-category order; empty categories are omitted (§18.4).
- Defensive: a tag whose category is outside the 19 stops with SC0008.
Tests: assignment, within-category ordering, canonical category ordering,
empty-category omission, tag identity preserved, empty input, and SC0008.
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Add compiler/resolver/illustrious_resolver.py: resolve_scene(scene,
knowledge_base, config, logger=None) -> CompilerResult, per MASTER_SPEC §17. No
LLM; no concept or tag is invented.
- normalize_concept (§17.3): NFKC + lowercase, unifying underscores/spaces so
natural concepts, snake_case ids, and spaced aliases share one lookup key. The
resolver builds its own normalized index over the Knowledge Base.
- Per-concept pipeline in scene discovery order (§17.8): normalize -> direct
alias resolution (no chains; skipped when allow_aliases is false) -> canonical
lookup -> recursive expansion -> tag generation.
- Expansion (§17.5): own tags first, then depth-first in list order per expand
target, inserted immediately after; capped by max_expansion_depth; skipped when
expansion_enabled is false; defensive path-based cycle detection -> SC0003.
- Deduplication after expansion keeps the first occurrence (SC0007).
- Unknown concept -> SC0001 warning (continue); deprecated -> SC0006 (continue);
invalid category -> SC0008 (stop); circular expansion -> SC0003 (stop).
- Every ResolvedTag keeps its entry's category and full traceability
(source_concept = originating scene concept, knowledge_base_entry = generating
entry). Verbose logging of each decision.
Tests (no Ollama): normalization, alias resolution + disable, expansion order,
nested/depth-capped/disabled expansion, traceability, dedup, unknown/deprecated,
circular/invalid-category stops, category preservation, and determinism.
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Add the Scene Validator per MASTER_SPEC §14 and a central message-code registry.
- compiler/common/message_codes.py: single source of truth for message codes
(Appendix B SC0001-SC0014 plus documented extensions SC0015 Unexpected Field
Removed, SC0016 Empty Concept Removed, SC0017 Interaction Dropped) with a
message() factory that pins each code's severity and title.
- compiler/validator/scene_validator.py: validate_scene(scene_json, config,
logger=None) -> CompilerResult.
- Strict schema validation classifies failures: missing required -> SC0009,
wrong type/malformed -> SC0002 (Errors that stop the compilation).
- Recoverable normalization: unexpected fields -> SC0015 warning + removed
(suppressed when config.validator.allow_unknown_fields is true); empty/
whitespace concepts -> SC0016 warning + removed after trimming; interactions
whose participants reference a non-existent character -> SC0017 warning +
dropped.
- Never creates concepts, generates tags, or does KB lookup. Output is a
Scene model wrapped in a CompilerResult; order is deterministic.
New warning codes (SC0015-SC0017) are extensions to be folded into Appendix B.
Tests: valid input, hard errors (SC0009/SC0002) that stop, recoverable warnings
(SC0015/SC0016/SC0017), allow_unknown_fields suppression, determinism, and the
code registry.
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Populate the 13 domain files with a starter Knowledge Base (188 entries)
covering the reference test scenarios (single/two characters, interactions,
indoor/outdoor, fantasy, modern) per MASTER_SPEC §16.
- appearance.json: character identity, general appearance, hair, eyes, face.
- clothing.json: garments plus worn accessories.
- anatomy.json (body), expressions.json, poses.json, actions.json,
interactions.json, objects.json, environments.json, camera.json, lighting.json.
- quality.json / style.json: reserved, empty in V1 (not auto-generated).
- Synonym aliases (girl/woman/lady -> female, boy/man -> male, etc.).
- Exactly two deterministic expansions: school_uniform -> blazer + pleated_skirt;
twin_braids -> braid. No speculative expansions.
- Exactly one valid category per entry; no subjective concepts.
The whole dataset validates with zero findings via the KB validation tooling
(#9), so the loader (#7) accepts it. A test loads the shipped dataset, asserts
clean load, checks core concepts across every reference scenario, verifies
synonym aliases resolve, confirms the deterministic expansions and their
targets, and guards against subjective concepts.
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Implement Knowledge Base loading and indexing per MASTER_SPEC §15.11, §27.3 in
compiler/common/knowledge_base.py.
- KnowledgeBaseEntry: frozen model of a canonical concept.
- KnowledgeBase: immutable, with id->entry and alias->id indexes and
get / resolve_alias / lookup accessors (read-only mappings).
- load_knowledge_base: reads every *.json (array of entries) from a directory,
delegates full validation to validate_knowledge_base (#9), and on any finding
raises Fatal SC0004 KnowledgeBaseError with the findings attached -- indexes
are built only after validation passes, so a failed load leaves no partial
state.
- KnowledgeBaseLoader: caches the Knowledge Base after first get(); reload() is
explicit and side-effect-free on failure (the previous KB is kept). No
automatic file watching. Optional structured logger.
Tests: multi-file load, id/alias lookups, expand/deprecated preservation, empty
dir, invalid KB / duplicate id / invalid JSON -> SC0004, caching identity,
reload picks up changes, reload failure keeps the previous KB, and immutability.
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
