* perf: KB load-once cache, compiled index, lazy loading, parallel resolution (#38)
Closes#130, #131, #132, #133, #134.
- Loader node routes through a process-wide KnowledgeBaseLoader cache so the KB
loads once per (path, version) and is reused across run() calls.
- Normalized lookup table compiled once per KB (per NSFW gating) and reused across
resolves instead of rebuilt per resolve_scene. normalize_concept moved to
knowledge_base (re-exported from the resolver).
- Incremental per-category lazy file loading behind KnowledgeBaseLoader.
- Optional parallel per-concept resolution (thread pool), merged in exact
discovery order; deterministic and threshold-gated (default sequential).
- Performance regression guard (generous ceilings + relative ratio). Refreshed
docs/benchmarks.md (resolution 57.6us -> 17.8us).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(changelog): note V2 performance improvements (#38)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Closes#114, #115, #116, #117.
- Deterministic, offline, dependency-free char-ngram embedding backend + config
section (semantic.enabled default false, min_similarity, backend).
- Committed embedding index (data/kb_embedding_index.json) mapping vectors to real
KB entry ids only; built by scripts/build_embedding_index.py, loaded once/cached.
- Resolver falls back to the nearest entry on a deterministic miss when enabled,
accepting >= min_similarity and emitting SC0020; NSFW-gated neighbours are
skipped. Disabled => byte-identical to before.
- Guarantee tests: deterministic lookup always wins (exact hit never queries the
index), fallback emits only KB tags (no invention), identical input => identical
output. SC0020 registered.
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Closes#111, #112, #113.
- Resolver recovers leading modifiers dropped by the head-noun reduction as
<modifier> <head-noun> compounds (open white shirt -> white shirt + open
shirt). KB-only, deterministic, no bare-word invention; goldens unchanged.
- Analyzer system prompt instructs full list transcription; a deterministic
post-parse check warns SC0021 when a tag-list yields fewer concepts than items.
- Data-driven fidelity regression suite locks both failure modes.
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
remove_duplicate_tags now gates the Resolver's post-expansion dedup (SC0007);
trim_empty_outputs was inert since the V1.1 category removal and is dropped
from config, schema, node, and default_config.
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* feat!: collapse pipeline — Resolver emits a flat prompt, drop categories (v1.1)
BREAKING CHANGE: the Category Splitter and Prompt Builder stages and nodes are
removed. The Resolver is now the translator from Scene JSON to prompt: it resolves
concepts to Knowledge Base tags and joins them into one flat comma-separated
prompt string, in resolution order (which naturally leads with character concepts).
Resolver node outputs are now (prompt, warnings, errors, json) — the json being
the traceable resolved tags so a translation problem is inspectable. Node count
drops from 8 to 6. The CategoryMap/PromptOutput models and their schemas are
removed; the `category` field remains on entries and resolved tags for
traceability only. Debug Viewer simplified to scene/warnings/errors.
Goldens, example scenes/workflow, integration/performance/repo-layout/release
tests, benchmark script, and the docs (README + wiki) are updated to the flat
prompt. Version bumped to 1.1.0.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* build: drop removed splitter/builder packages from pyproject
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix: recover concept detail lost in resolver and harden analyzer parsing
Three shippability fixes for the v1 pipeline:
P2 (main): the Illustrious Resolver did exact-key KB lookup only, so any
compound concept the analyzer emits ("white summer dress", "black business
suit") missed and was dropped silently as SC0001. Add a deterministic
head-noun fallback (§17.2): on a miss, drop leading modifiers and look up the
longest remaining suffix ("white summer dress" -> "dress"). No tag/concept is
invented. Emits SC0019 (Concept Reduced) so the reduction is visible.
P1a: the analyzer required the entire model response to be a single JSON
document, so any surrounding prose ("Sure! Here is the scene:") failed parsing
and, with the prompt reused unchanged, every retry failed identically ->
empty output. Extract the first balanced top-level {...} object from
surrounding text (string-aware brace scan); the object bytes are unchanged and
must still validate, so this is unwrapping, not JSON repair.
P1b: the node's "raw" output was to_raw_json(result.data) -- a re-serialization
of the parsed Scene, so on failure it was empty and the model's actual text was
invisible. Capture the last raw model text in analyzer metadata and surface it
as the node's raw output.
Regenerated goldens: unknown_concepts "hologram jacket" now resolves to
"jacket" via head-noun fallback (intended).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat: add entry rating + resolver.include_nsfw gating
Foundations for the Danbooru-vocab KB: a per-entry `rating` (general|explicit,
default general) and a `resolver.include_nsfw` config flag (default false). The
resolver omits explicit-rated entries from lookup and expansion unless enabled,
so a gated concept is reported as unknown (SC0001). Schema, model, config, and
example config updated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat: generate KB from embedded Danbooru vocab (188 -> 16.4k tags)
Replace the coverage ceiling of the 188-entry hand KB by importing a large,
validated Danbooru tag vocabulary. This is the fix for the core "detail loss"
symptom: compound concepts the analyzer emits ("red dress", "sunflower field",
"train station platform") are now real tags that resolve instead of being
dropped.
- scripts/build_vocab.py: one-time dev step that filters the Raffle node's
categorized tag dump (drops artist/character/copyright/attribution categories
and subjective adjectives, keeps only id-pattern-valid tags) into a committed,
self-contained snapshot at data/danbooru_vocab.txt (16.4k tags).
- scripts/generate_kb_from_vocab.py: additive, non-destructive generator. Writes
knowledge_base/gen_<category>.json, mapping the Raffle categories to the 19
canonical categories and tagging NSFW categories rating=explicit. Skips any tag
already defined as a hand-curated id/alias, so the curated canonicalization
(female + aliases) and expansions (school_uniform -> blazer, pleated_skirt) are
preserved and always win.
Result: 16.4k additive entries + 188 curated. Plain natural-language concepts
now resolve broadly via curated aliases + head-noun fallback with no analyzer
prompt change; NSFW gated behind resolver.include_nsfw; golden and dataset tests
stay green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* style: ruff format build_vocab.py
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(security): drop untrusted CLI path in build_vocab (SonarCloud S8707)
The --source CLI argument fed an untrusted path to the file system (taint).
The script is a one-time dev step; hard-code the source path constant instead.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Two ComfyUI issues found while testing.
1. Empty output with no error: the Knowledge Base Loader resolved a relative
`path` against ComfyUI's working directory, not this package, so the default
`knowledge_base/` loaded 0 entries (an empty KB is valid) and every concept
became unknown -> empty prompts. Relative paths now resolve against the
package root, and the loader warns when the Knowledge Base is empty.
2. Add a `raw` string output (a JSON dump of the node's data) to the Scene
Analyzer, Scene Validator, Resolver, Category Splitter, and Prompt Builder
nodes, plus a summary `raw` on the Knowledge Base Loader, for inspecting the
pipeline in the graph. Adds a shared to_raw_json helper.
Tests: package-relative path resolution (independent of cwd), empty-KB warning,
raw output content, and updated node metadata/return-arity expectations.
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Add nodes/resolver_node.py: ResolverNode, a thin adapter that delegates to
resolve_scene (MASTER_SPEC §20).
- Required inputs scene (SCENE model) and knowledge_base (KNOWLEDGE_BASE, from
the loader node); optional config (COMPILER_CONFIG) defaulting to Config().
- Outputs (resolved_tags, warnings, errors): RESOLVED_TAGS plus surfaced
diagnostics. No compiler logic in the node.
- Registered as SceneCompilerResolver ("Resolver").
Tests: metadata/inputs, real resolution over a small in-memory Knowledge Base,
delegation + message surfacing (monkeypatched), default config, registration.
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Add compiler/resolver/illustrious_resolver.py: resolve_scene(scene,
knowledge_base, config, logger=None) -> CompilerResult, per MASTER_SPEC §17. No
LLM; no concept or tag is invented.
- normalize_concept (§17.3): NFKC + lowercase, unifying underscores/spaces so
natural concepts, snake_case ids, and spaced aliases share one lookup key. The
resolver builds its own normalized index over the Knowledge Base.
- Per-concept pipeline in scene discovery order (§17.8): normalize -> direct
alias resolution (no chains; skipped when allow_aliases is false) -> canonical
lookup -> recursive expansion -> tag generation.
- Expansion (§17.5): own tags first, then depth-first in list order per expand
target, inserted immediately after; capped by max_expansion_depth; skipped when
expansion_enabled is false; defensive path-based cycle detection -> SC0003.
- Deduplication after expansion keeps the first occurrence (SC0007).
- Unknown concept -> SC0001 warning (continue); deprecated -> SC0006 (continue);
invalid category -> SC0008 (stop); circular expansion -> SC0003 (stop).
- Every ResolvedTag keeps its entry's category and full traceability
(source_concept = originating scene concept, knowledge_base_entry = generating
entry). Verbose logging of each decision.
Tests (no Ollama): normalization, alias resolution + disable, expansion order,
nested/depth-capped/disabled expansion, traceability, dedup, unknown/deprecated,
circular/invalid-category stops, category preservation, and determinism.
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Scaffold the Phase 0 foundation per MASTER_SPEC §24:
- Directory layout: nodes/, compiler/{analyzer,validator,resolver,splitter,
builder,common}/, knowledge_base/, schemas/, prompts/, config/,
tests/{analyzer,resolver,validator,integration,regression}/, examples/,
scripts/, docs/.
- pyproject.toml targeting Python 3.11+, jsonschema as the single runtime dep,
ruff + pytest as dev deps, explicit package list, and ruff/pytest config.
- ruff for linting and formatting; pytest with repo root on pythonpath.
- GitHub Actions CI running install, ruff lint, ruff format check, and pytest
on Python 3.11 and 3.12 for pushes and pull requests.
- docs/ skeleton and CONTRIBUTING.md referencing the wiki.
- Layout test asserting the §24 structure and package importability.
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
