87d9525176
Closes #114, #115, #116, #117. - Deterministic, offline, dependency-free char-ngram embedding backend + config section (semantic.enabled default false, min_similarity, backend). - Committed embedding index (data/kb_embedding_index.json) mapping vectors to real KB entry ids only; built by scripts/build_embedding_index.py, loaded once/cached. - Resolver falls back to the nearest entry on a deterministic miss when enabled, accepting >= min_similarity and emitting SC0020; NSFW-gated neighbours are skipped. Disabled => byte-identical to before. - Guarantee tests: deterministic lookup always wins (exact hit never queries the index), fallback emits only KB tags (no invention), identical input => identical output. SC0020 registered. Co-authored-by: kunail0804 <kunail0804@gmail.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2.4 KiB
2.4 KiB
Rebuilding the Knowledge Base
The generated Knowledge Base is produced by a single, fully-automated pipeline
with zero human input (epic #35). It regenerates the gen_*.json files and the
manifest.json deterministically from the committed source snapshots under
data/.
One command
python scripts/build_knowledge_base.py
The pipeline runs:
- build vocab (dev-only) — rebuilds
data/danbooru_vocab.txtfrom an external Danbooru/Raffle export. Skipped automatically when that source is absent (e.g. in CI), so the committed snapshot is used and no network access is needed. - generate candidates from the vocab snapshot.
- ingest aliases/implications from
data/danbooru_aliases.txtanddata/danbooru_implications.txt(synonyms → canonical aliases, implications →expand). - auto-validate every candidate (structural rules reused from
compiler/common/kb_validation.py+ a confidence heuristic — no approval gate). - conflict-scan the whole Knowledge Base (duplicate ids/aliases, ambiguous tag ownership, contradictory expansions).
- write
gen_*.jsonand stampmanifest.json— only if steps 4–5 are clean.
On any rejected candidate or detected conflict the pipeline writes a structured JSON log to stderr and exits non-zero without writing — it never prompts. The curated-wins additive merge is preserved end-to-end: a candidate colliding with a curated id/alias is rejected.
Source snapshots (committed)
| File | Built by | Contents |
|---|---|---|
data/danbooru_vocab.txt |
scripts/build_vocab.py |
raffle_category<TAB>tag |
data/danbooru_aliases.txt |
scripts/build_aliases.py |
alias<TAB>canonical_id |
data/danbooru_implications.txt |
scripts/build_aliases.py |
antecedent_id<TAB>consequent_id |
The build_* scripts read an external export from a hard-coded path (edit the
constant before running) and are occasional dev steps, not part of runtime or CI.
Related tools
python scripts/build_embedding_index.py— rebuild the committed semantic-fallback embedding index (data/kb_embedding_index.json) from the current KB.python scripts/detect_kb_conflicts.py knowledge_base/— standalone conflict scan (CI-gateable).python scripts/coverage_benchmark.py— Knowledge Base coverage report.python scripts/validate_knowledge_base.py knowledge_base/— full KB validation.