50e7b0a8a4
* feat(kb): automatic KB builder, ingestion & validation tooling (#35) Closes #118, #119, #120, #121, #122. - Ingest committed Danbooru alias/implication snapshots during generation: synonyms reach canonicals (long black stockings -> thighhighs), implications become expand relations. Curated-wins preserved. - Automatic candidate validator (structural rules reused from kb_validation + confidence heuristic, no approval gate; SC0022). - Duplicate/conflict detection tool (scripts/detect_kb_conflicts.py) + reusable detect_conflicts; clean reference KB reports zero. - KB coverage benchmark harness (scripts/coverage_benchmark.py) + JSON summary. - Single fully-automated, CI-runnable build pipeline (scripts/build_knowledge_base.py) that fails fast on rejections/conflicts and never prompts. Docs added. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(kb): use pytest.approx for confidence float assertions (SonarCloud reliability) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: kunail0804 <kunail0804@gmail.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2.2 KiB
2.2 KiB
Rebuilding the Knowledge Base
The generated Knowledge Base is produced by a single, fully-automated pipeline
with zero human input (epic #35). It regenerates the gen_*.json files and the
manifest.json deterministically from the committed source snapshots under
data/.
One command
python scripts/build_knowledge_base.py
The pipeline runs:
- build vocab (dev-only) — rebuilds
data/danbooru_vocab.txtfrom an external Danbooru/Raffle export. Skipped automatically when that source is absent (e.g. in CI), so the committed snapshot is used and no network access is needed. - generate candidates from the vocab snapshot.
- ingest aliases/implications from
data/danbooru_aliases.txtanddata/danbooru_implications.txt(synonyms → canonical aliases, implications →expand). - auto-validate every candidate (structural rules reused from
compiler/common/kb_validation.py+ a confidence heuristic — no approval gate). - conflict-scan the whole Knowledge Base (duplicate ids/aliases, ambiguous tag ownership, contradictory expansions).
- write
gen_*.jsonand stampmanifest.json— only if steps 4–5 are clean.
On any rejected candidate or detected conflict the pipeline writes a structured JSON log to stderr and exits non-zero without writing — it never prompts. The curated-wins additive merge is preserved end-to-end: a candidate colliding with a curated id/alias is rejected.
Source snapshots (committed)
| File | Built by | Contents |
|---|---|---|
data/danbooru_vocab.txt |
scripts/build_vocab.py |
raffle_category<TAB>tag |
data/danbooru_aliases.txt |
scripts/build_aliases.py |
alias<TAB>canonical_id |
data/danbooru_implications.txt |
scripts/build_aliases.py |
antecedent_id<TAB>consequent_id |
The build_* scripts read an external export from a hard-coded path (edit the
constant before running) and are occasional dev steps, not part of runtime or CI.
Related tools
python scripts/detect_kb_conflicts.py knowledge_base/— standalone conflict scan (CI-gateable).python scripts/coverage_benchmark.py— Knowledge Base coverage report.python scripts/validate_knowledge_base.py knowledge_base/— full KB validation.