# ComfyUI Scene Compiler **A deterministic prompt compiler for ComfyUI.** Turn a plain-language scene description into a clean, valid Illustrious tag prompt — without an LLM inventing anything. > **Status:** Version 2 released (`v2.1.0`). The deterministic Version 1 pipeline > shipped in v1.0.0 / v1.1.0. Full documentation lives in the [Wiki](../../wiki); > see the [Roadmap](../../wiki/Roadmap) and [issues](../../issues) for what's next. --- ## The idea Most prompt-generation workflows ask a language model to write Danbooru tags directly: ``` Natural Language → LLM → Prompt ``` LLMs are bad at this. They invent clothing, poses, lighting, and environments that were never described, emit invalid tags, and return different output for identical input. Scene Compiler takes a different position: > **The language model understands the scene. The compiler generates the prompt.** Prompt generation is treated as a **compilation** problem, not a text-generation problem. Language understanding happens once, up front; everything after it is deterministic and traceable. ``` Natural Language → Scene Analyzer → Scene JSON → Scene Validator → Validated Scene JSON → Resolver → Prompt ``` Only the **Scene Analyzer** uses an LLM, and only to understand language — never to produce tags. Every generated tag is looked up in a data-only **Knowledge Base** built from a real Danbooru tag vocabulary, so tags are always valid and never invented. The Resolver both resolves concepts to tags and joins them into a single flat prompt string, in resolution order. Every tag stays traceable: ``` Natural Language → Scene JSON → Knowledge Base Entry → Resolved Tag → Prompt ``` --- ## What Version 2 changed Version 1 proved the compiler works. Version 2 is about making it **know more and ask less** — without giving up a single guarantee. ### Know more The V1 compiler was honest but narrow: a concept missing from the Knowledge Base was simply dropped, and the Knowledge Base only grew by hand. - **The Knowledge Base grows itself.** A fully automated pipeline regenerates it from committed Danbooru snapshots — generate candidates, ingest aliases and implications, auto-validate, scan for conflicts, write. There is **no approval gate**: an automation that stops to ask a human is an automation that doesn't run. - **Near-misses are caught, not dropped.** An **opt-in** semantic (nearest-neighbour) fallback resolves concepts that miss exact lookup. It is off by default, deterministic when enabled, and can only return an entry that *already exists* in the Knowledge Base — so it can never invent a tag. Deterministic lookup always wins. - **Detail stops leaking.** The compiler used to silently discard parts of what you wrote. Now modifiers survive (`open white shirt` → `white shirt` **and** `open shirt`), every item of a list is transcribed, and relationships are kept rather than flattened to a noun (`money in her hand` → `holding money`, not just `money`). The same fidelity applies to explicit content — nothing is quietly softened or skipped. ### Ask less V1 exposed almost every internal as a node or a widget. That is honest, but it pushes the compiler's internal shape onto the user. - **Configuration lives in one place.** All settings are on the **Configuration** node; the stage nodes take a single optional `config` connection. The analyzer model used to be set in two disconnected places — now it's set once. - **Fewer nodes.** Stages that carried no real decisions were folded away: the Category Splitter and Prompt Builder collapsed into the Resolver (V1.1), and the Knowledge Base Loader collapsed into the Configuration node (V2.1). Eight nodes became five. - **Don't ask what the system already knows.** The Knowledge Base ships inside the package, so its path is no longer a question. Options that aren't ready or aren't really user-facing (prompt target, separator, hand-written system prompts) are not exposed — they keep working defaults instead. - **Plain language.** Every input has a tooltip written for a user, not for a compiler engineer. ### What did not change Determinism. Every V2 addition is opt-in, bounded to the existing Knowledge Base, and reproducible: the same Scene JSON, Knowledge Base, and configuration always compile to the same prompt. Knowledge stays data, never code. Nothing is invented. --- ## Key principles - **Determinism first** — a given Scene JSON + Knowledge Base + config always produces the same output, with no random seeds. (The one non-deterministic step is the LLM understanding the language up front; everything after it is reproducible.) - **No hallucinations** — if it isn't in the description, it isn't in the output. Unknown concepts are reported, never guessed. - **Knowledge is data** — every concept→tag mapping lives in the Knowledge Base, never in code. - **Fidelity** — described detail must survive to the prompt, or be reported. Silence is a bug. - **Traceability** — every tag can be explained back to the sentence it came from. - **Automation without gates** — the pipeline runs end to end with zero required human input. - **Nodes are interfaces** — ComfyUI nodes are thin wrappers; all logic lives in the compiler package, and settings live on one node. Version 2 targets the **Illustrious** model family. The architecture is model-independent, so future versions can add Pony, Flux, NoobAI, and others without changing the compiler core. --- ## Nodes Scene Compiler ships as a ComfyUI custom-node package of **five** nodes. It is a toolbox, not a fixed workflow — every node can be used independently. ### Pipeline nodes | Node | Inputs | Outputs | |---|---|---| | **Scene Analyzer** | `natural_language`, `config` *(optional)* | `scene`, `warnings`, `errors`, `raw` | | **Scene Validator** | `scene`, `config` *(optional)* | `scene`, `warnings`, `errors`, `raw` | | **Resolver** | `scene`, `knowledge_base`, `config` *(optional)* | `prompt`, `warnings`, `errors`, `json` | The Resolver is the final stage: it resolves concepts to Knowledge Base tags and joins them into one flat `prompt` string. Its `json` output carries the traceable resolved tags so a translation problem is inspectable. ### Support nodes | Node | Inputs | Outputs | |---|---|---| | **Configuration** | all compiler settings (see below) | `config`, `knowledge_base`, `warnings`, `errors` | | **Debug Viewer** | `scene`, `warnings`, `errors` *(all optional)* | `report` | - **Configuration** is the single place for compiler settings *and* the Knowledge Base loader: it reads the Knowledge Base that ships with the package and emits it on its `knowledge_base` output, which you wire into the Resolver. Bump its `knowledge_base_reload` counter after editing the Knowledge Base to re-read it. - **Debug Viewer** is read-only; connect any intermediate state to inspect it. A typical graph: ``` Configuration ─ config ──────────► Scene Analyzer / Scene Validator / Resolver └ knowledge_base ───► Resolver Scene Analyzer → Scene Validator → Resolver → (your image workflow) ``` ### Knowledge Base Editor (optional web tool) Not a node. Version 2 adds an optional browser editor for curated Knowledge Base entries, served by ComfyUI at `http://127.0.0.1:8188/scene-compiler/kb`. It offers create/edit/delete with live validation and atomic, format-safe saves. It is the *only* manual surface in the project and is entirely off the compile critical path — the compiler never depends on it. --- ## Installation > Install manually from Git today; a ComfyUI Manager listing is planned. **Via ComfyUI Manager** (recommended once published): search for *Scene Compiler*. **Manual:** ```bash cd ComfyUI/custom_nodes git clone https://github.com/kunail0804/ComfyUI-Scene-Compiler.git ``` Then restart ComfyUI. The Scene Analyzer requires a local [Ollama](https://ollama.com) instance for language understanding; the rest of the pipeline runs fully offline. --- ## Quick start 1. Add a **Configuration** node (it also loads the Knowledge Base). 2. Add a **Scene Analyzer** and type a description, e.g. *"A blonde girl wearing a white dress hugs a young man while walking in the rain."* 3. Wire **Scene Analyzer → Scene Validator → Resolver**, and connect Configuration's `config` to each stage and its `knowledge_base` to the Resolver. 4. Send the **Resolver**'s `prompt` output into your image-generation workflow (e.g. EasyIllustrious). 5. Drop a **Debug Viewer** on any connection to inspect the intermediate state. A ready-made graph is in [`examples/workflows/scene_compiler_pipeline.json`](examples/workflows/scene_compiler_pipeline.json). Given the example above, the compiler extracts `female`, `blonde hair`, `white dress`, `male`, the interaction `hug`, and environment `rain` — and nothing else. It does not add eye colour, quality tags, or lighting that were never described. --- ## Documentation Full documentation lives in the **[Wiki](../../wiki)**: - [Overview & Vision](../../wiki/Overview-and-Vision) - [Architecture & Pipeline](../../wiki/Architecture-and-Pipeline) - [Data Model](../../wiki/Data-Model) - [Scene JSON & Schemas](../../wiki/Scene-JSON-and-Schemas) - [Scene Analyzer](../../wiki/Scene-Analyzer) - [Knowledge Base](../../wiki/Knowledge-Base) - [Resolver & Prompt](../../wiki/Resolver-and-Prompt) - [ComfyUI Nodes](../../wiki/ComfyUI-Nodes) - [Configuration, Errors & Logging](../../wiki/Configuration-Errors-and-Logging) - [Development, Testing & Contributing](../../wiki/Development-Testing-and-Contributing) - [Roadmap](../../wiki/Roadmap) - [Glossary & Reference](../../wiki/Glossary-and-Reference) In-repo notes live in [`docs/`](docs/) — see [`docs/nodes.md`](docs/nodes.md) for the node reference and [`docs/knowledge_base_build.md`](docs/knowledge_base_build.md) for how the Knowledge Base is regenerated. [`MASTER_SPEC.md`](MASTER_SPEC.md) is the full normative specification, maintained through Version 2. Chapters superseded by later versions are marked in place rather than deleted, so the original design intent stays readable. --- ## Roadmap - **Version 1 — Deterministic Compiler Foundation** *(done — v1.0.0 / v1.1.0)*: the complete deterministic pipeline for Illustrious, plus Ollama integration, Knowledge Base, Debug Viewer, and regression tests. - **Version 2 — Semantic Resolution** *(done — v2.0.0 / v2.1.0)*: opt-in semantic/embedding fallback, automatic Knowledge Base builder, Knowledge Base versioning and editor, concept fidelity, performance, and the node/configuration consolidation — determinism preserved. (Analyzer localization was dropped as unnecessary.) - **Version 3 — Extensible Compiler Platform**: multi-model Resolvers, a plugin system, multiple Analyzer backends, and a standalone Compiler SDK. - **Version 4 — Consolidation**: a single unified node that runs the whole pipeline, continuing the V2 consolidation. The granular nodes stay available for debugging. See the [Roadmap](../../wiki/Roadmap) for details. --- ## Contributing Contributions are welcome — especially Knowledge Base entries. Please read [CONTRIBUTING.md](CONTRIBUTING.md) and the [Development, Testing & Contributing](../../wiki/Development-Testing-and-Contributing) guide first. Core rule: **never hardcode knowledge, and never break determinism.** ## License See [LICENSE](LICENSE). --- ## A note on AI assistance This project — its code, tests, and documentation — is built with heavy help from AI coding assistants, and is developed and maintained by a solo author. AI makes mistakes: despite testing and review, parts of this project and its docs may be inaccurate, out of date, or simply wrong. Please treat it as a good-faith best effort rather than authoritative truth — **if the docs and the code disagree, the code wins** — and open an issue when you spot something off.