CyberShadowX-ComfyUI-Smart-Metadata-Reader/docs/spec.md

ComfyUI Smart Metadata Reader Specification

1. Project Positioning

ComfyUI-Smart-Metadata-Reader is an image-loader style ComfyUI custom node package. It is not only a parser library.

The primary node, Smart Metadata Reader, must behave like a practical image loader and metadata reader inside ComfyUI:

• Let the user select an image from ComfyUI's input directory.
• Let the user upload an image through the node UI.
• Load and output the image and mask.
• Read image metadata from PNG, JPEG, and WEBP files.
• Extract common generation values and prompt text from the metadata.
• Output common ports that can be wired directly into other ComfyUI nodes.

The core technical difference from ordinary prompt readers is graph-aware ComfyUI parsing. For ComfyUI images, the node must prefer the backend execution prompt JSON, select the final sampler that generated the image, and trace backward through the sampler's positive and negative conditioning inputs to recover the real text that entered generation.

The package may expose internal parser modules for testing and reuse, but the user-facing deliverable is the ComfyUI node.

2. Node Split

2.1 Smart Metadata Reader

Smart Metadata Reader is the primary node and must be implemented first.

Suggested category:

metadata/image

Required inputs:

image: COMBO[STRING], image_upload=True
parameter_index: INT, default 0
prefer_cached_text: BOOLEAN, default True
include_raw_json: BOOLEAN, default True
max_depth: INT, default 40

Required outputs:

image: IMAGE
mask: MASK
positive: STRING
negative: STRING
seed: INT
steps: INT
cfg: FLOAT
width: INT
height: INT
model_name: STRING
filename: STRING
setting: STRING

The main node must be useful on its own. If development time is limited, this node takes priority over every debug or advanced node.

2.2 Smart Metadata Debug

Smart Metadata Debug is an enhanced node that can be added after the main node is stable.

Suggested category:

metadata/debug

Enhanced outputs:

vae_name: STRING
sampler_name: STRING
scheduler: STRING
loras_json: STRING
raw_prompt_json: STRING
raw_workflow_json: STRING
partial_result_json: STRING
debug_trace: STRING
confidence: FLOAT
status_message: STRING

The debug node should expose parser internals without making the primary node difficult to use.

3. UI Phases

3.1 Phase 1: Python-Only MVP

Phase 1 must deliver a working ComfyUI image loader node with:

• Image selection and upload input.
• Image and mask output.
• Metadata reading.
• ComfyUI prompt and workflow parsing.
• A1111/Forge parameters fallback.
• Graph traversal for positive and negative prompt extraction.
• Common numeric and string outputs.
• A human-readable setting output.
• Basic UI payload feedback when practical.

Phase 1 does not require a full SD Prompt Reader style frontend layout with three embedded large text boxes. The node may rely on output ports and simple ComfyUI UI payload feedback.

3.2 Phase 2: Frontend UI Enhancement

Phase 2 adds a ComfyUI JavaScript extension through:

WEB_DIRECTORY = "./web/js"

The frontend enhancement should display these read-only areas inside the node:

• positive text box.
• negative text box.
• setting text box.
• Image preview.
• Image dimensions.
• OK, PARTIAL, or FAILED status.

JavaScript is only for presentation. Parser logic must remain in Python.

3.3 Phase 3: Compatibility Expansion

Phase 3 adds:

• More custom node adapters.
• Stronger A1111/Forge parameters fallback.
• More precise multi-sampler, refiner, and hires-fix handling.
• The Smart Metadata Debug node if it was not completed earlier.
• More machine-readable debug outputs.

4. Metadata Priority

For ComfyUI images, parsing priority is:

1. Use backend execution prompt JSON as the primary source because it is closest to the actual generation graph.
2. Use workflow JSON as supplemental data for UI cache, widgets_values, ShowText cache, preview context, and fallback.
3. Use A1111/Forge parameters when ComfyUI prompt data is absent or unusable.

The parser must not treat stale workflow widget text as the true execution prompt when execution prompt JSON provides a better source. If workflow fallback is used for a prompt segment, the trace and status must say so.

5. Prompt Parsing Principles

positive must be the merged text that truly entered the selected sampler's positive conditioning.

negative must be the merged text that truly entered the selected sampler's negative conditioning.

Hard rules:

• Do not hard-code node IDs.
• Do not hard-code a single test workflow shape.
• Do not assume a fixed node order.
• Parse by graph traversal, class_type, inputs, and link references such as [node_id, output_index].
• Preserve all text that enters final conditioning.
• Do not remove LoRA trigger words.
• Do not remove character tags.
• Do not remove style tags.
• Do not remove NSFW terms.
• Do not remove quality tags.
• Do not remove translated text.
• Do not remove user-written prompt text.
• Do not rewrite words.
• Do not optimize prompts.
• Do not perform content moderation.

Allowed formatting cleanup:

• Remove empty segments.
• Trim obvious leading and trailing whitespace.
• Collapse clearly repeated commas produced by joining segments.
• Collapse excessive blank lines.
• Normalize obvious spacing around separators.

Formatting cleanup must never delete meaningful tokens.

6. LoRA Rules

LoRA loader nodes and LoRA trigger words are separate concerns.

LoRA loader nodes such as LoraLoader, LoraLoaderModelOnly, and compatible custom loaders should be extracted to loras_json with:

• Node ID.
• class_type.
• LoRA file name.
• strength_model.
• strength_clip.
• Connection path.

Example:

[
  {
    "node_id": "17",
    "class_type": "LoraLoader",
    "lora_name": "character_lora.safetensors",
    "strength_model": 0.8,
    "strength_clip": 1.0,
    "path": "KSampler 72.model <- LoraLoader 17"
  }
]

If LoRA trigger words, character tags, or style tags enter CLIPTextEncode or final conditioning through text nodes, they must remain in positive or negative. The parser must not filter text because it looks like a LoRA trigger word.

7. Final Sampler Selection

Sampler selection must prefer the sampler that generated the loaded image.

Selection order:

1. Start from final image output nodes when possible, including SaveImage, PreviewImage, SaveAnimatedWEBP, and common image save nodes.
2. Walk backward through image, images, samples, latent, and VAE decode chains.
3. Select the nearest valid sampler upstream of the final output chain.
4. If no final output chain can be determined, choose a valid sampler through fallback heuristics.
5. If fallback heuristics are used, setting, status_message, and debug_trace must clearly state the reason and confidence.

Initial sampler candidates:

KSampler
KSamplerAdvanced
SamplerCustom
SamplerCustomAdvanced

Additional sampler types should be added through adapters.

The parser must not pretend fallback selection is fully certain.

8. Conditioning Trace Rules

Conditioning resolution begins at:

selected_sampler.inputs.positive
selected_sampler.inputs.negative

The resolver must follow links backward through passthrough and branch nodes, including:

ControlNetApply
ControlNetApplyAdvanced
FaceDetailer
DetailerForEach
Impact Pack detailer nodes
InpaintModelConditioning
ConditioningCombine
ConditioningConcat
ConditioningSet

The resolver should also support adapter-driven matching for custom conditioning nodes that expose fields such as:

conditioning
positive
negative
positive_conditioning
negative_conditioning
base_positive
base_negative

For branch nodes such as ConditioningCombine and ConditioningConcat, every actual branch that reaches the final sampler must be resolved and merged. debug_trace must show which prompt segment came from which node.

Unknown passthrough nodes must produce a partial result instead of fabricated prompt text.

9. Text Adapters

Initial text adapters must support:

CLIPTextEncode
CLIPTextEncodeSDXL
StringFunction|pysssss
ShowText|pysssss
DeepTranslatorTextNode
Primitive/String/Text-like nodes

Common text fields:

text
text_0
text_a
text_b
text_c
string
value
prompt
result

Adapter rules:

• CLIPTextEncode: resolve inputs.text; if it is a link, continue upstream.
• CLIPTextEncodeSDXL: resolve all prompt text fields that enter conditioning, such as text, text_g, and text_l.
• StringFunction|pysssss: for append-like actions, resolve text_a, text_b, and text_c in graph order and merge the segments.
• ShowText|pysssss: prefer inputs.text_0 or workflow/cache display results. Only follow upstream when no cache exists. When upstream is used, reduce confidence and record the reason in trace.
• DeepTranslatorTextNode: resolve inputs.text for Phase 1; future adapters may prefer cached translated output when reliable metadata is available.
• Primitive/String/Text-like nodes: read common text fields without assuming exact node names.

ShowText|pysssss must avoid treating upstream GeminiChatNode prompt templates or system instructions as final image prompt text when cached displayed text is available.

10. Prompt Merge Strategy

Prompt segments should be represented internally with:

• Text value.
• Source node ID.
• Source class type.
• Source input field.
• Trace path.
• Confidence contribution.

Merging must follow actual graph path order where known. Branch nodes should preserve branch order from graph inputs.

The merger may clean separators and empty segments, but it must not classify, censor, or rewrite prompt content.

11. Setting Field

setting is human-readable text, not JSON.

Recommended format:

Filename: 001.png
Source: ComfyUI prompt/workflow
Model: model.safetensors
VAE: vae.safetensors
LoRA:
- character_lora.safetensors (model 0.80, clip 1.00)
Seed: 123456789
Steps: 30
CFG: 7.0
Sampler: euler
Scheduler: normal
Size: 1200 x 1600
Status: OK
Confidence: 0.94

For partial results:

Status: PARTIAL
Confidence: 0.52
Unresolved:
- Node 88 FooBarConditioning input positive_conditioning

Machine-readable data belongs in debug outputs such as loras_json, partial_result_json, and debug_trace.

12. Debug Trace

debug_trace should explain how the result was reached.

Example:

SAMPLER_SELECTION:
SaveImage 91.images <- VAEDecode 89.samples <- KSampler 72.samples
Selected KSampler 72 because it is upstream of final SaveImage.

POSITIVE_TRACE:
KSampler 72.positive
 -> ControlNetApply 65.conditioning
 -> ConditioningCombine 61.conditioning_1
 -> CLIPTextEncode 40.text
 -> StringFunction|pysssss 22 [append]
    text_a: direct string
    text_b: ShowText|pysssss 143.text_0 cache
    text_c: DeepTranslatorTextNode 24.text

NEGATIVE_TRACE:
KSampler 72.negative
 -> CLIPTextEncode 41.text
 -> StringFunction|pysssss 48.text_a

LORA_TRACE:
KSampler 72.model <- LoraLoader 33 <- LoraLoader 28 <- CheckpointLoaderSimple 4

When fallback or workflow cache is used, trace must say so.

13. Failure and Partial Results

The parser must not guess when it cannot determine a value.

Failure behavior:

• Return all fields that were resolved.
• Return empty or sentinel values for fields that were not resolved.
• Set status to PARTIAL or FAILED.
• Lower confidence.
• Include unresolved node IDs, class types, and input fields in trace or partial result.

Examples:

Status: PARTIAL
Unresolved:
- Node 88 UnknownConditioningRouter input positive
Reason: no adapter matched this conditioning passthrough node

14. A1111 and Forge Fallback

When ComfyUI prompt metadata is missing, invalid, or unusable, the parser should attempt A1111/Forge parameters parsing.

Phase 1 should parse common fields:

• Positive prompt.
• Negative prompt.
• Seed.
• Steps.
• CFG scale.
• Sampler.
• Size.
• Model name when present.

parameter_index selects among multiple parameter sets when the metadata contains more than one set.

15. Package Architecture

Planned layout:

ComfyUI-Smart-Metadata-Reader/
  __init__.py
  nodes.py
  requirements.txt
  README.md
  pyproject.toml
  docs/
    spec.md
    implementation_plan.md
  smart_metadata_reader/
    __init__.py
    metadata_reader.py
    image_io.py
    models.py
    graph.py
    sampler_selector.py
    resolver.py
    prompt_merge.py
    lora_extractor.py
    settings_formatter.py
    a1111_parser.py
    trace.py
    adapters/
      __init__.py
      base.py
      clip_text.py
      text_sources.py
      pysssss.py
      conditioning.py
      samplers.py
      loaders.py
  web/
    js/
      smart_metadata_reader.js
  tests/
    fixtures/
    test_metadata_reader.py
    test_graph.py
    test_sampler_selector.py
    test_conditioning_resolver.py
    test_text_adapters.py
    test_lora_extractor.py
    test_node_id_independence.py
    test_node_contract.py

Parser modules must be testable without launching ComfyUI.

16. Test Requirements

Tests must cover:

• Simple KSampler -> CLIPTextEncode.
• CLIPTextEncode.text connected to upstream text node.
• StringFunction|pysssss three-part append.
• ShowText|pysssss text_0 cache priority.
• DeepTranslatorTextNode.
• ControlNetApply passthrough.
• ConditioningCombine multi-branch merge.
• Multiple CLIPTextEncode nodes entering positive conditioning.
• LoRA trigger words retained in positive prompt.
• Multiple LoRA loader nodes exported to loras_json.
• Multiple samplers with final output sampler selection.
• Randomized node IDs with unchanged parse result.
• Unknown passthrough node returns partial result instead of invented prompt text.

17. Relationship to comfyui-prompt-reader-node

This project may use receyuki/comfyui-prompt-reader-node as a user experience reference:

• Image selection and upload.
• Positive prompt area.
• Negative prompt area.
• Setting area.
• Image preview.
• Common output ports.

This project must not copy its implementation and must not depend on its submodule. The parser must be owned by this project and designed around final sampler and conditioning graph traversal.