* fix: recover concept detail lost in resolver and harden analyzer parsing
Three shippability fixes for the v1 pipeline:
P2 (main): the Illustrious Resolver did exact-key KB lookup only, so any
compound concept the analyzer emits ("white summer dress", "black business
suit") missed and was dropped silently as SC0001. Add a deterministic
head-noun fallback (§17.2): on a miss, drop leading modifiers and look up the
longest remaining suffix ("white summer dress" -> "dress"). No tag/concept is
invented. Emits SC0019 (Concept Reduced) so the reduction is visible.
P1a: the analyzer required the entire model response to be a single JSON
document, so any surrounding prose ("Sure! Here is the scene:") failed parsing
and, with the prompt reused unchanged, every retry failed identically ->
empty output. Extract the first balanced top-level {...} object from
surrounding text (string-aware brace scan); the object bytes are unchanged and
must still validate, so this is unwrapping, not JSON repair.
P1b: the node's "raw" output was to_raw_json(result.data) -- a re-serialization
of the parsed Scene, so on failure it was empty and the model's actual text was
invisible. Capture the last raw model text in analyzer metadata and surface it
as the node's raw output.
Regenerated goldens: unknown_concepts "hologram jacket" now resolves to
"jacket" via head-noun fallback (intended).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat: add entry rating + resolver.include_nsfw gating
Foundations for the Danbooru-vocab KB: a per-entry `rating` (general|explicit,
default general) and a `resolver.include_nsfw` config flag (default false). The
resolver omits explicit-rated entries from lookup and expansion unless enabled,
so a gated concept is reported as unknown (SC0001). Schema, model, config, and
example config updated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat: generate KB from embedded Danbooru vocab (188 -> 16.4k tags)
Replace the coverage ceiling of the 188-entry hand KB by importing a large,
validated Danbooru tag vocabulary. This is the fix for the core "detail loss"
symptom: compound concepts the analyzer emits ("red dress", "sunflower field",
"train station platform") are now real tags that resolve instead of being
dropped.
- scripts/build_vocab.py: one-time dev step that filters the Raffle node's
categorized tag dump (drops artist/character/copyright/attribution categories
and subjective adjectives, keeps only id-pattern-valid tags) into a committed,
self-contained snapshot at data/danbooru_vocab.txt (16.4k tags).
- scripts/generate_kb_from_vocab.py: additive, non-destructive generator. Writes
knowledge_base/gen_<category>.json, mapping the Raffle categories to the 19
canonical categories and tagging NSFW categories rating=explicit. Skips any tag
already defined as a hand-curated id/alias, so the curated canonicalization
(female + aliases) and expansions (school_uniform -> blazer, pleated_skirt) are
preserved and always win.
Result: 16.4k additive entries + 188 curated. Plain natural-language concepts
now resolve broadly via curated aliases + head-noun fallback with no analyzer
prompt change; NSFW gated behind resolver.include_nsfw; golden and dataset tests
stay green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* style: ruff format build_vocab.py
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(security): drop untrusted CLI path in build_vocab (SonarCloud S8707)
The --source CLI argument fed an untrusted path to the file system (taint).
The script is a one-time dev step; hard-code the source path constant instead.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Populate the 13 domain files with a starter Knowledge Base (188 entries)
covering the reference test scenarios (single/two characters, interactions,
indoor/outdoor, fantasy, modern) per MASTER_SPEC §16.
- appearance.json: character identity, general appearance, hair, eyes, face.
- clothing.json: garments plus worn accessories.
- anatomy.json (body), expressions.json, poses.json, actions.json,
interactions.json, objects.json, environments.json, camera.json, lighting.json.
- quality.json / style.json: reserved, empty in V1 (not auto-generated).
- Synonym aliases (girl/woman/lady -> female, boy/man -> male, etc.).
- Exactly two deterministic expansions: school_uniform -> blazer + pleated_skirt;
twin_braids -> braid. No speculative expansions.
- Exactly one valid category per entry; no subjective concepts.
The whole dataset validates with zero findings via the KB validation tooling
(#9), so the loader (#7) accepts it. A test loads the shipped dataset, asserts
clean load, checks core concepts across every reference scenario, verifies
synonym aliases resolve, confirms the deterministic expansions and their
targets, and guards against subjective concepts.
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Formalize the Knowledge Base on-disk format and tighten the entry schema per
MASTER_SPEC §15.3-§15.10, §18.2.
- compiler/common/categories.py: single source of truth for the 19 canonical
categories (lowercase identifiers of the §18.2 list) plus is_valid_category.
- knowledge_base_entry.schema.json (v1.1): category restricted to the 19; tags
require at least one item; expand entries must be snake_case canonical ids;
aliases uniqueItems.
- knowledge_base/README.md: by-domain file layout, the "each file is a JSON array
of entries" format, and contributor field rules (id/alias global uniqueness,
tag order, expansion targets, one-tag minimum).
- examples/knowledge_base.example.json: worked file demonstrating an alias and an
expansion.
Cross-entry rules (global uniqueness, expand targets exist) are deferred to the
KB validation tooling (#9); this issue defines per-entry structure only.
Tests: 19-category constant and schema-enum consistency, entry validation
(category/tags/id/expand/unknown-field), and the worked example (array shape,
per-entry validity, alias+expansion present, expand targets resolve).
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Scaffold the Phase 0 foundation per MASTER_SPEC §24:
- Directory layout: nodes/, compiler/{analyzer,validator,resolver,splitter,
builder,common}/, knowledge_base/, schemas/, prompts/, config/,
tests/{analyzer,resolver,validator,integration,regression}/, examples/,
scripts/, docs/.
- pyproject.toml targeting Python 3.11+, jsonschema as the single runtime dep,
ruff + pytest as dev deps, explicit package list, and ruff/pytest config.
- ruff for linting and formatting; pytest with repo root on pythonpath.
- GitHub Actions CI running install, ruff lint, ruff format check, and pytest
on Python 3.11 and 3.12 for pushes and pull requests.
- docs/ skeleton and CONTRIBUTING.md referencing the wiki.
- Layout test asserting the §24 structure and package importability.
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
