Closes#111, #112, #113.
- Resolver recovers leading modifiers dropped by the head-noun reduction as
<modifier> <head-noun> compounds (open white shirt -> white shirt + open
shirt). KB-only, deterministic, no bare-word invention; goldens unchanged.
- Analyzer system prompt instructs full list transcription; a deterministic
post-parse check warns SC0021 when a tag-list yields fewer concepts than items.
- Data-driven fidelity regression suite locks both failure modes.
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(analyzer): escalate temperature on retries to self-recover bad JSON
Ollama is non-deterministic even at temperature 0 (no fixed seed, CUDA/batch
variance), so the analyzer occasionally emits non-schema JSON that fails parsing
(SC0011 -> null Scene). The retry loop reused the prompt unchanged at the same
temperature, so all internal retries reproduced the same failure -- only a fresh
manual re-run recovered.
Escalate the sampling temperature on retry attempts only (base -> +0.2 per
retry, capped at 1.0); the prompt is still byte-for-byte unchanged (§12.9) and
the first attempt keeps the configured temperature, so the happy path stays
deterministic while a malformed run can now self-recover.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test(analyzer): use pytest.approx for temperature assertions (SonarCloud S1244)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix: recover concept detail lost in resolver and harden analyzer parsing
Three shippability fixes for the v1 pipeline:
P2 (main): the Illustrious Resolver did exact-key KB lookup only, so any
compound concept the analyzer emits ("white summer dress", "black business
suit") missed and was dropped silently as SC0001. Add a deterministic
head-noun fallback (§17.2): on a miss, drop leading modifiers and look up the
longest remaining suffix ("white summer dress" -> "dress"). No tag/concept is
invented. Emits SC0019 (Concept Reduced) so the reduction is visible.
P1a: the analyzer required the entire model response to be a single JSON
document, so any surrounding prose ("Sure! Here is the scene:") failed parsing
and, with the prompt reused unchanged, every retry failed identically ->
empty output. Extract the first balanced top-level {...} object from
surrounding text (string-aware brace scan); the object bytes are unchanged and
must still validate, so this is unwrapping, not JSON repair.
P1b: the node's "raw" output was to_raw_json(result.data) -- a re-serialization
of the parsed Scene, so on failure it was empty and the model's actual text was
invisible. Capture the last raw model text in analyzer metadata and surface it
as the node's raw output.
Regenerated goldens: unknown_concepts "hologram jacket" now resolves to
"jacket" via head-noun fallback (intended).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat: add entry rating + resolver.include_nsfw gating
Foundations for the Danbooru-vocab KB: a per-entry `rating` (general|explicit,
default general) and a `resolver.include_nsfw` config flag (default false). The
resolver omits explicit-rated entries from lookup and expansion unless enabled,
so a gated concept is reported as unknown (SC0001). Schema, model, config, and
example config updated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat: generate KB from embedded Danbooru vocab (188 -> 16.4k tags)
Replace the coverage ceiling of the 188-entry hand KB by importing a large,
validated Danbooru tag vocabulary. This is the fix for the core "detail loss"
symptom: compound concepts the analyzer emits ("red dress", "sunflower field",
"train station platform") are now real tags that resolve instead of being
dropped.
- scripts/build_vocab.py: one-time dev step that filters the Raffle node's
categorized tag dump (drops artist/character/copyright/attribution categories
and subjective adjectives, keeps only id-pattern-valid tags) into a committed,
self-contained snapshot at data/danbooru_vocab.txt (16.4k tags).
- scripts/generate_kb_from_vocab.py: additive, non-destructive generator. Writes
knowledge_base/gen_<category>.json, mapping the Raffle categories to the 19
canonical categories and tagging NSFW categories rating=explicit. Skips any tag
already defined as a hand-curated id/alias, so the curated canonicalization
(female + aliases) and expansions (school_uniform -> blazer, pleated_skirt) are
preserved and always win.
Result: 16.4k additive entries + 188 curated. Plain natural-language concepts
now resolve broadly via curated aliases + head-noun fallback with no analyzer
prompt change; NSFW gated behind resolver.include_nsfw; golden and dataset tests
stay green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* style: ruff format build_vocab.py
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(security): drop untrusted CLI path in build_vocab (SonarCloud S8707)
The --source CLI argument fed an untrusted path to the file system (taint).
The script is a one-time dev step; hard-code the source path constant instead.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(nodes): handle a failed upstream stage without crashing
When a stage fails it passes None down its data output. Downstream nodes called
`.to_json()` / iterated the None and raised an unhelpful AttributeError (found in
ComfyUI: a failed Scene Analyzer crashed the Scene Validator on
`None.to_json()`).
Each consuming node now short-circuits on a missing required input: the
Validator, Resolver, and Category Splitter return None plus an error string that
points at the upstream node's errors output; the Prompt Builder returns empty
strings. Adds a shared upstream_failure_message helper and regression tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(analyzer): unwrap Markdown code fences in model responses
Most local models (e.g. gemma) return valid Scene JSON wrapped in a ```json
... ``` code fence despite the system prompt forbidding markdown. Strict
json.loads then rejected it (SC0011), and at temperature 0 every retry returned
the identical fenced output, so the Analyzer always failed and produced a None
scene.
The response parser now unwraps a leading/trailing code fence before parsing.
This is not the heuristic JSON repair forbidden by MASTER_SPEC §12.9: the JSON
inside is left byte-for-byte unchanged; only the surrounding ``` markers are
removed. Malformed JSON inside a fence is still rejected.
Verified end-to-end against a live Ollama (gemma): the fenced response now parses
and the full pipeline produces prompt outputs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Add compiler/analyzer/scene_analyzer.py: analyze(description, backend, config,
logger=None) -> CompilerResult, per MASTER_SPEC §12.9, §12.10.
- Composes the prompt once (resolved system prompt + user description) and reuses
it unchanged on every attempt; JSON is never repaired heuristically.
- Bounded retries: total attempts = max_retries + 1 (1 initial + up to N
retries, default 3). A bad response (parse/schema failure SC0011, or an
unexpected backend response SC0018) is retried; the last such error is returned
once attempts are exhausted.
- Terminal failures return immediately: connection failure -> Fatal SC0012;
timeout -> SC0013 (no retry).
- Attaches telemetry to the result metadata (model, attempts, retries, duration,
validation status) and logs the outcome.
Tests (mocked backend): first-attempt success, succeed-on-retry, prompt reused
unchanged, prompt includes system prompt + description, exhaust-retries -> SC0011
with the right call count, zero-retries single attempt, connection -> SC0012 (no
retry), timeout -> SC0013 (no retry), and telemetry.
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Add compiler/analyzer/response_parser.py: parse_scene_response(text, logger=None)
-> CompilerResult, per MASTER_SPEC §12.2, §12.8.
- Strictly json.loads the model response; JSON is never repaired heuristically
(§12.9), so malformed or markdown-fenced output fails here (feeding the retry
path in #18).
- Validates the parsed document against the Scene schema; any failure (malformed
JSON, non-object, missing field, wrong type) is reported as SC0011 and stops
the compilation -- invalid responses are never silently accepted.
- Handles both object and string concept forms via the Scene model; confidence
stays metadata-only.
- Stamps a default schema_version onto compiler-owned metadata (the model emits
`"metadata": {}`) before validation.
Tests: valid response -> Scene, metadata stamping, object/string concepts +
confidence, and rejection (SC0011) of malformed JSON, non-object JSON, missing
fields, wrong types, and markdown-fenced (unrepaired) output.
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Ship the official Analyzer system prompt and make it loadable/overridable, per
MASTER_SPEC §13.
- prompts/analyzer_system_prompt.md: the official prompt assembled from §13.1-
§13.12 (deterministic semantic parser; extract explicit concepts only; never
generate tags or invent information; ignore subjective/artistic language;
explicit camera/lighting/environment only; independent characters with
scene-level interactions; preserve unknown concepts; return valid JSON only,
no markdown) plus the §12.2 Scene JSON output shape and example.
- compiler/analyzer/system_prompt.py: load_default_system_prompt() reads the
shipped default; resolve_system_prompt(config) returns config.analyzer.
system_prompt when set, otherwise the default.
- Add optional analyzer.system_prompt to the config model and schema (nullable).
Tests: default file loads, a parametrized drift guard asserting every required
§13 directive is present, output-shape keys present, config override and default
resolution, and config round-trip of the override.
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* feat(analyzer): add the Ollama Analyzer backend (#15)
Add compiler/analyzer/backend.py per MASTER_SPEC §12.3, §12.10.
- AnalyzerBackend Protocol keeps the LLM backend swappable with no downstream
coupling; the module imports no downstream compiler stage.
- OllamaBackend talks to a local Ollama /api/generate endpoint using only the
standard library (urllib). The HTTP call is an injectable transport, so the
backend is fully unit-testable without a live Ollama.
- Sends the configured model, temperature (default 0.0), and the user prompt
unmodified, applying the configured timeout. BackendResult exposes the response
text, model name, and request duration for logging.
- Failure modes map to codes: connection failure -> BackendUnavailableError
(SC0012, Fatal); timeout -> BackendTimeoutError (SC0013); HTTP error /
unexpected response -> BackendResponseError (SC0018, new extension).
Adds SC0018 (Analyzer Unexpected Response) to the message-code registry.
Tests (no live Ollama): interface conformance, from_config, success with
telemetry + unmodified prompt, and timeout/connection/URLError/HTTP/malformed
response mapping.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test(analyzer): use pytest.approx and single throwing call in backend tests
Avoid float-equality on temperature (SonarCloud reliability) and construct the
backend outside pytest.raises so each block has a single throwing invocation.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Scaffold the Phase 0 foundation per MASTER_SPEC §24:
- Directory layout: nodes/, compiler/{analyzer,validator,resolver,splitter,
builder,common}/, knowledge_base/, schemas/, prompts/, config/,
tests/{analyzer,resolver,validator,integration,regression}/, examples/,
scripts/, docs/.
- pyproject.toml targeting Python 3.11+, jsonschema as the single runtime dep,
ruff + pytest as dev deps, explicit package list, and ruff/pytest config.
- ruff for linting and formatting; pytest with repo root on pythonpath.
- GitHub Actions CI running install, ruff lint, ruff format check, and pytest
on Python 3.11 and 3.12 for pushes and pull requests.
- docs/ skeleton and CONTRIBUTING.md referencing the wiki.
- Layout test asserting the §24 structure and package importability.
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
