fix(inspect): bound inspection work, not graph size (#244)

* test(inspect): add synthetic inspection parity fixtures and legacy characterization

* test(inspect): pin test-time parity vectors and keep fixtures fully synthetic

* fix(inspect): bound resolver work instead of graph size

* fix(inspect): bound shared-node and stripped-text work in the inspection profile

Keep one memo per resolution whose entries carry their unfolded visit cost;
a memo hit charges that cost to the current candidate's allowance only.
Cache node validation and charge input keys, stripped characters and copied
provenance entries to a whole-resolution work budget before doing the work.
The default profile is unchanged.

* fix(inspect): keep memoized depth honest across inspection candidates

A memo entry now records its height below the reusing depth; a hit raises
depth-limit exactly where a fresh walk would, so each candidate's outcome
matches its anchored result and does not depend on trace or node order.
The default profile is unchanged.

* test(inspect): pin memo height propagation

Three samplers share one chain, each later one hanging below a memo hit.
The third candidate must report depth-limit in both trace orders and when
anchored; removing the memo-hit height propagation fails this test. The
provenance test now asserts a plain subset.

* fix(inspect): tolerate ComfyUI node metadata quirks in projections

Projections now take a kind. Each prompt node keeps exactly class_type and
inputs; _meta, is_changed and other node keys are validated and dropped, so
"is_changed": [NaN] no longer marks the whole prompt graph malformed. A
nonfinite value inside inputs, anywhere in a workflow, or in a retained
carrier member projects to null and is never reported as a number. The
carrier applies the prompt/workflow kinds to its members. Exact numbers are
unchanged (Decimal spelling, unsafe integers as strings) and duplicate keys
are still malformed. The 500-node gate is replaced by a 20,000-node backstop
(workflow count includes subgraph definitions). Raw records are untouched.

The f14 inspection vector's facts become known; default vectors are unchanged.

* fix(inspect): budget metadata per record so large workflows never hide prompts

PNG inspection no longer counts every text chunk against one shared 2 MiB
budget. Role records (prompt, parameters, the pair execution record) keep a
2 MiB budget together, encoded and decoded, and exceeding it still fails the
whole read. A workflow has its own 4 MiB budget: a chunk over it is skipped
by seek, and a compressed one stops inflating at budget + 1. Only the
workflow is then reported as limited (record format "limited", no raw text,
workflow_graph limited); prompts, facts and the prompt graph are unaffected.

Each text chunk's keyword is read first (at most 80 bytes). Chunks that are
not retained are skipped by length without being read or checksummed, so
memory stays bounded. Byte-identical duplicates of a retained record are
accepted and do not consume budget; differing duplicates still raise
"Duplicate prompt metadata", and a duplicate of a limited workflow stays
limited. The chunk-count limit is unchanged.

InspectionRead gains limited_records (empty by default). The reader is only
used by the inspection routes, so ingestion, scan and import are unchanged.

Vectors: f04-inspection (2.5 MB workflow, within the new workflow budget)
and f15-inspection (identical duplicate chunks) now read and resolve, with
the same outcome and facts as f05-inspection. Default vectors are unchanged.

* fix(inspect): keep prompt evidence longest under the response budget

* fix(inspect): bound identical duplicate chunk reads

* fix(inspect): fail closed when role duplicates exceed their budget

* fix(inspect): use the bounded inspection profile end to end

Inspection routes resolve with INSPECTION_PROFILE and expose reason_code
and node on the resolution; scan, live monitor and admin import keep the
default profile (pinned by spies over the real resolver on f03).

JS facts drop the entries and workflow node-count gates; per-lookup visit
sets and the server's node ceiling bound the work. The validator accepts
the textless `limited` workflow format, follows the workflow-first
response omission order, and checks the closed reason-code vocabulary.
The file inspector labels limited records and disables their workflow
download.

vectors.json: 12 inspection vectors gain their reason codes; f03 facts
become known and f01 facts ambiguous. Default vectors are unchanged. A
new parity test asserts vectors.json facts through the workspace JS.

* chore(tools): add read-only inspection corpus tool

tools/inspection_corpus.py runs the inspection route pipeline and the
workspace JS facts of a chosen code tree over a media folder, read-only,
and reports the #240 invariants H1 to H8 with aggregate counts. Rows hold
statuses and SHA-256 digests of resolved role texts, never file names or
prompt text unless --with-paths is given. Outputs inside the root or a code
tree are refused.

The facts-over-vectors test now fails instead of skipping when CI is set
and Node.js is missing.

* docs(testing): document bounded inspection work

* fix(tools): fail the corpus run when facts or inspection did not run

Rows whose facts were never computed (no or exiting Node.js) now fail H3,
and a new harness line fails on any tool exception or missing facts, so a
run where nothing was measured can no longer exit 0. Unreadable files stay
inspector outcomes. A missing --node binary is refused, an exited harness
is reaped, H8 has a 50 ms floor for tiny baselines, and JSONL rows are
sorted by id.

* docs(inspect): correct ceiling wording and bandit annotations

Describe the two 20,000-node ceilings (prompt entries, workflow UI plus
subgraph nodes) in TESTING.md, and use bare nosec ids so Bandit stops
warning about unparsed test names.

* test(inspect): make parity observation tuples explicit
This commit is contained in:
Vito
2026-10-19 09:54:29 -07:00
committed by GitHub
parent 44b6ed2420
commit 048d599386
46 changed files with 6186 additions and 192 deletions
+32 -4
View File
@@ -194,10 +194,38 @@ The inspector does not establish historical file identity, infer run membership,
edit image tags, export all EXIF, or load/execute a workflow. Saved vocabulary is
labelled by its recipe or positive-prompt scope. Limited raw evidence is omitted
atomically, with JSON exports unavailable and the rooted original still downloadable.
The inspection API uses one bounded read (2 MiB aggregate metadata, 8 MiB non-PNG
header, 10,000 PNG chunks, three-second ffprobe, 500 graph nodes/semantic expansions,
64 link/JSON-container levels and an 8 MiB serialized response); retry does not raise
these limits. Installing the Python inspection/source-scope changes requires a
The inspection API uses one bounded read with per-record budgets: role records
(`prompt`, `parameters`, the pair execution record) share 2 MiB and exceeding it
fails the read; the `workflow` record has its own 4 MiB and past it only the workflow
becomes record format `limited` (no raw text, graph `limited`, roles and facts
unaffected); other text chunks are skipped unread; byte-identical duplicates are
accepted within a bounded read budget, while differing duplicates (and any duplicate
role record over that budget) still fail. Other read bounds are unchanged: 8 MiB
non-PNG header, 10,000 PNG chunks and a three-second ffprobe. Resolution under
`INSPECTION_PROFILE` bounds work, not graph size: 2,000 visits per candidate chain,
20,000 in total, 64 link/JSON-container levels, 100,000 resolved characters, and a
two 20,000-node ceilings: one over the top-level entries of the API prompt, and one
over the workflow graph's UI nodes plus subgraph-definition nodes (past it the
workflow graph and its facts become limited). The serialized
response stays under 8 MiB by omitting, in order, the workflow graph, then the prompt
graph, then all raw records atomically. Retry does not raise these limits. Ingestion
(scan, live monitor, admin import) and queue-time run counting keep the default
profile, including its 500-node gate.
`tools/inspection_corpus.py --root DIR --out JSONL [--workers N] [--baseline JSONL]
[--summary TXT] [--code TREE]` measures a media folder read-only against the #240
invariants H1 to H8. It runs the inspection route pipeline and the workspace JS facts
of `--code` (default: its own tree), opens media only for reading, follows no
symlinks, imports no database, and refuses `--out`/`--summary` inside the root or a
code tree. Its rows hold statuses, sizes, timings and SHA-256 digests of resolved
role texts, never file names or prompt text unless `--with-paths` is given; digests
of short prompts can still be guessed, so the JSONL stays local and only the
aggregate summary goes into a PR body. Run it on the base commit first and pass that
JSONL as `--baseline` for H1 and H8 (2x the baseline p99, never below 50 ms). It
exits 1 when an invariant fails, including the `harness` line: a tool exception or
facts that were not computed (no Node.js) fail the run. Unreadable or unsupported
files are inspector outcomes, not harness errors. Never point it at a live install's folders from an
automated job. Installing the Python inspection/source-scope changes requires a
maintainer ComfyUI restart. Automated tests use disposable hosts and never restart
the running personal ComfyUI.
+2
View File
@@ -82,6 +82,8 @@ def inspect_current_file(
"state": "unresolved",
"sampler_id": None,
"reason": "Media metadata could not be read",
"reason_code": None,
"node": None,
},
}
else:
+15
View File
@@ -153,6 +153,21 @@ test('full inspector downloads actual bounded evidence',async({page,request})=>{
await panel.getByText('File & provenance',{exact:true}).click();await expect(panel.getByRole('link',{name:'Download original',exact:true})).toHaveAttribute('href',/root=0&root_scope=[a-f0-9]{64}/);
});
test('full inspector labels a limited workflow record without hiding prompt facts',async({page,request})=>{
// The server marks a workflow over its 4 MiB read budget `limited`: no raw text, a limited graph.
await page.route('**/images/inspect/inspector/complete.png?*',async route=>{const response=await route.fetch();const data=await response.json();data.metadata.record_formats.workflow='limited';delete data.metadata.raw.workflow;data.metadata.workflow_graph={status:'limited',value:null};await route.fulfill({response,json:data});});
await fullFixture(page,request);const panel=inspector(page);
await expect(panel.locator('.generation-facts dd')).toHaveText(['9007199254740993','28','6.5','dpmpp_2m','karras','fixture-model.safetensors']);
await expect(panel.getByRole('button',{name:'Send prompt pair',exact:true})).toBeVisible();
await panel.getByText('Workflow & raw metadata',{exact:true}).click();
await expect(panel.getByText('Record exceeds inspection limits; text is unavailable',{exact:true})).toBeVisible();await expect(panel.locator('pre code')).toHaveCount(1);
const workflow=panel.getByRole('button',{name:'Download workflow JSON',exact:true});await expect(workflow).toHaveAttribute('aria-disabled','true');
await expect(workflow).toHaveAccessibleDescription('Workflow exceeds inspection limits. Download the original file to inspect it externally.');
let downloads=0;const count=()=>downloads++;page.on('download',count);await workflow.focus();await page.keyboard.press('Enter');expect(downloads).toBe(0);page.off('download',count);
const metadata=JSON.parse((await downloadText(page,panel.getByRole('button',{name:'Download metadata JSON',exact:true}))).text);
expect(metadata.metadata.record_formats.workflow).toBe('limited');expect(metadata.metadata.raw.workflow).toBeUndefined();expect(metadata.metadata.representations.workflow).toBe('Record exceeds inspection limits; text is unavailable');
});
test('full inspector rejects stale shared action state',async({page,request})=>{
let reads=0,stale=false;await page.route('**/images/inspect/**',route=>{reads++;return stale?route.fulfill({status:409,json:{success:false,code:'source-changed',error:'Source changed'}}):route.continue();});
await fullFixture(page,request);const panel=inspector(page);await expect(panel.getByRole('button',{name:'Send prompt pair',exact:true})).toBeVisible();const before=reads;
+32
View File
@@ -12,3 +12,35 @@ ffmpeg -f lavfi -i testsrc=size=32x32:rate=4:duration=1 \
```
Browser tests use the committed file; FFmpeg is not required to run them.
# Inspection parity fixtures
`inspection_parity/` holds small synthetic PNGs (`f01b.png` ... `f17.png`, each
under 32 KiB) and `vectors.json`, used by the #240 inspection parity tests. Every
graph, prompt text, model name and filename prefix in them is original synthetic
content written for this repository. Nothing comes from user output. They are
dedicated to the public domain under CC0-1.0.
`tests/inspection_parity.py` builds every fixture from code. It writes the
`prompt` and `workflow` records the way core ComfyUI `SaveImage` does (Pillow
`PngInfo.add_text`, `compress_level=4`, a 64x64 constant colour image), and
splices extra raw `tEXt` chunks with real CRC32 for the duplicate-chunk shapes
`f15` and `f16`. The larger shapes `f01`, `f04` and `f05` are built at test
time instead of committed. Reproduce the committed files with:
```sh
python3 -m tests.inspection_parity --write tests/fixtures/inspection_parity
```
`tests/test_inspection_parity_fixtures.py` compares the decoded text chunks with
the builders (not bytes, because zlib output differs between platforms).
`vectors.json` holds an `inspection` and a `default` vector for every fixture,
committed or built at test time (test-time fixtures are built into a temporary
directory). `inspection` is the workspace inspector path, with the facts the
workspace shows. `default` pins only the resolver's default profile, run on the
role records decoded from the PNG text chunks. Ingestion's own readers are covered
by their own suites. The expectations start as the observed baseline. A change
that alters an outcome edits the vector in the same commit.
`tests/js/legacy-metadata-characterization.test.js` pins what
the legacy viewers show on the same files.
Binary file not shown.

After

Width:  |  Height:  |  Size: 6.0 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 27 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 32 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.1 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.0 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.1 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.2 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.5 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.1 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.3 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.2 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.3 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.0 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1020 B

Binary file not shown.

After

Width:  |  Height:  |  Size: 1008 B

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.7 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 5.0 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 2.2 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 2.2 KiB

File diff suppressed because it is too large Load Diff
+920
View File
@@ -0,0 +1,920 @@
"""Deterministic synthetic ComfyUI-shaped fixtures for inspection parity (#240).
Every graph here is original synthetic content (CC0): neutral texts, generic model
file names and invented filename prefixes. Nothing is copied from user outputs.
The shapes reproduce the parity audit's fixture numbering (``f01``..``f17``).
``build(name)`` returns ``{"prompt", "workflow", "extra_chunks"}``. ``write_png``
writes the two records the way core ``SaveImage`` does (Pillow ``PngInfo.add_text``,
``compress_level=4``, a 64x64 constant colour image), then splices any extra raw
``tEXt`` chunks, with real CRC32, before ``IEND``.
Regenerate the committed files with::
python3 -m tests.inspection_parity --write tests/fixtures/inspection_parity
"""
import argparse
import io
import json
import struct
import tempfile
import zlib
from pathlib import Path
from PIL import Image
from PIL.PngImagePlugin import PngInfo
POS = "a red fox in snow, detailed fur"
NEG = "blurry, lowres"
FIXTURE_DIR = Path(__file__).resolve().parent / "fixtures" / "inspection_parity"
VECTORS_PATH = FIXTURE_DIR / "vectors.json"
MAX_FIXTURE_BYTES = 32 * 1024
_SIGNATURE = b"\x89PNG\r\n\x1a\n"
ABSENT = "<absent>"
FACT_FIELDS = ("seed", "steps", "cfg", "sampler", "scheduler", "model")
EXPECT_KEYS = (
"metadata_status",
"state",
"reason",
"reason_code",
"selection",
"positive",
"negative",
"candidate_count",
"facts",
)
# Small enough to commit (each file stays under MAX_FIXTURE_BYTES).
COMMITTED = (
"f01b",
"f02",
"f03",
"f06a",
"f06b",
"f06c",
"f06d",
"f06e",
"f07a",
"f07b",
"f08",
"f09",
"f10",
"f11",
"f12",
"f14",
"f15",
"f16",
"f17",
)
# Built in a temporary directory at test time (too large to commit).
TEST_TIME = ("f01", "f04", "f05")
def _ui(prompt, extra_nodes=0, subgraphs=0, big_widget=None):
"""A UI workflow: one node per prompt node plus inert reroutes/notes/primitives."""
nodes = [
{"id": index + 1, "type": node["class_type"], "mode": 0}
for index, node in enumerate(prompt.values())
]
base = len(nodes)
for offset in range(extra_nodes):
kind = ("Reroute", "Note", "PrimitiveNode")[offset % 3]
nodes.append({"id": base + offset + 1, "type": kind, "mode": 0})
if big_widget is not None:
nodes.append(
{"id": 999999, "type": "Note", "mode": 0, "widgets_values": [big_widget]}
)
workflow = {"last_node_id": len(nodes), "nodes": nodes, "links": [], "version": 0.4}
if subgraphs:
workflow["definitions"] = {
"subgraphs": [
{"id": f"sg-{index}", "nodes": [], "links": []}
for index in range(subgraphs)
]
}
return workflow
def _enc(text):
return {"class_type": "CLIPTextEncode", "inputs": {"text": text, "clip": ["4", 1]}}
def _ksampler_graph(positive, negative, sampler="KSampler", latent=None, extra=None):
graph = {
"4": {
"class_type": "CheckpointLoaderSimple",
"inputs": {"ckpt_name": "sdxl_base.safetensors"},
},
"5": latent
or {
"class_type": "EmptyLatentImage",
"inputs": {"width": 1024, "height": 1024, "batch_size": 1},
},
"6": positive,
"7": negative,
"3": {
"class_type": sampler,
"inputs": {
"seed": 42,
"steps": 25,
"cfg": 6.5,
"sampler_name": "euler",
"scheduler": "normal",
"denoise": 1.0,
"model": ["4", 0],
"positive": ["6", 0],
"negative": ["7", 0],
"latent_image": ["5", 0],
},
},
"8": {
"class_type": "VAEDecode",
"inputs": {"samples": ["3", 0], "vae": ["4", 2]},
},
"9": {
"class_type": "SaveImage",
"inputs": {"filename_prefix": "ComfyUI", "images": ["8", 0]},
},
}
if sampler == "KSamplerAdvanced":
inputs = graph["3"]["inputs"]
inputs["noise_seed"] = inputs.pop("seed")
inputs.pop("denoise")
inputs.update(
add_noise="enable",
start_at_step=0,
end_at_step=10000,
return_with_leftover_noise="disable",
)
graph.update(extra or {})
return graph
def _basic(prompt, workflow=None, extra_chunks=()):
return {
"prompt": prompt,
"workflow": workflow,
"extra_chunks": [tuple(chunk) for chunk in extra_chunks],
}
def _flux2_branches(branches, gates=4):
"""Subgraph-expanded flux2 shape: N SaveImage branches plus a preview-only detailer.
Each branch is SamplerCustomAdvanced <- BasicGuider <- (easy ifElse /
ReferenceLatent) x gates <- FluxGuidance <- CLIPTextEncode <- JoinStringMulti.
Every SaveImage shares one JoinStringMulti ``filename_prefix``. A FaceDetailer
with a different prompt feeds only a PreviewImage.
"""
graph = {
"11": {
"class_type": "VAELoader",
"inputs": {"vae_name": "vae.safetensors"},
},
"12": {
"class_type": "CLIPLoader",
"inputs": {"clip_name": "text_encoder.safetensors", "type": "flux2"},
},
"13": {
"class_type": "UNETLoader",
"inputs": {
"unet_name": "flux-2-klein-9b-fp8.safetensors",
"weight_dtype": "default",
},
},
"14": {"class_type": "LoadImage", "inputs": {"image": "reference.png"}},
"21": {
"class_type": "StringConstantMultiline",
"inputs": {"string": POS + ",", "strip_newlines": False},
},
"22": {
"class_type": "StringConstantMultiline",
"inputs": {"string": "soft winter light,", "strip_newlines": False},
},
"31": {"class_type": "PrimitiveString", "inputs": {"value": "Project"}},
"32": {"class_type": "PrimitiveString", "inputs": {"value": "fixture"}},
"33": {"class_type": "PrimitiveString", "inputs": {"value": "IMG"}},
"34": {
"class_type": "JoinStringMulti",
"inputs": {
"inputcount": 2,
"delimiter": "_",
"return_list": False,
"string_1": ["32", 0],
"string_2": ["33", 0],
},
},
"35": {
"class_type": "JoinStringMulti",
"inputs": {
"inputcount": 4,
"delimiter": "/",
"return_list": False,
"string_1": ["31", 0],
"string_2": ["32", 0],
"string_3": ["34", 0],
"string_4": ["34", 0],
},
},
"41": {
"class_type": "PrimitiveStringMultiline",
"inputs": {"value": "Detailer pass: refine the face"},
},
"42": {
"class_type": "CLIPTextEncode",
"inputs": {"text": ["41", 0], "clip": ["12", 0]},
},
"43": {
"class_type": "ConditioningZeroOut",
"inputs": {"conditioning": ["42", 0]},
},
"44": {
"class_type": "FaceDetailer",
"inputs": {
"image": ["14", 0],
"model": ["13", 0],
"clip": ["12", 0],
"vae": ["11", 0],
"positive": ["42", 0],
"negative": ["43", 0],
"seed": 271828182845,
"steps": 20,
"cfg": 1,
"sampler_name": "euler",
"scheduler": "simple",
},
},
"45": {"class_type": "PreviewImage", "inputs": {"images": ["44", 0]}},
}
for branch in range(branches):
scope = f"{1300 + branch}"
def sub(local, scope=scope):
return f"{scope}:{local}"
graph[f"{1100 + branch}"] = {
"class_type": "StringConstantMultiline",
"inputs": {
"string": f"Scene {branch}: unique shot description",
"strip_newlines": False,
},
}
graph[f"{1200 + branch}"] = {
"class_type": "JoinStringMulti",
"inputs": {
"inputcount": 3,
"delimiter": " ",
"return_list": False,
"string_1": ["21", 0],
"string_2": [f"{1100 + branch}", 0],
"string_3": ["22", 0],
},
}
graph[sub(1)] = {
"class_type": "CLIPTextEncode",
"inputs": {"text": [f"{1200 + branch}", 0], "clip": ["12", 0]},
}
graph[sub(2)] = {
"class_type": "FluxGuidance",
"inputs": {"guidance": 3.5, "conditioning": [sub(1), 0]},
}
previous = sub(2)
for gate in range(gates):
graph[sub(10 + gate)] = {
"class_type": "ImageExists",
"inputs": {"image": ["14", 0]},
}
graph[sub(20 + gate)] = {
"class_type": "VAEEncode",
"inputs": {"pixels": [sub(10 + gate), 0], "vae": ["11", 0]},
}
graph[sub(30 + gate)] = {
"class_type": "ReferenceLatent",
"inputs": {
"conditioning": [previous, 0],
"latent": [sub(20 + gate), 0],
},
}
graph[sub(40 + gate)] = {
"class_type": "easy ifElse",
"inputs": {
"boolean": [sub(10 + gate), 1],
"on_true": [sub(30 + gate), 0],
"on_false": [previous, 0],
},
}
previous = sub(40 + gate)
graph[sub(3)] = {
"class_type": "BasicGuider",
"inputs": {"model": ["13", 0], "conditioning": [previous, 0]},
}
graph[sub(4)] = {
"class_type": "RandomNoise",
"inputs": {"noise_seed": 1000 + branch},
}
graph[sub(5)] = {
"class_type": "KSamplerSelect",
"inputs": {"sampler_name": "euler"},
}
graph[sub(6)] = {
"class_type": "Flux2Scheduler",
"inputs": {"steps": 20, "width": 1024, "height": 1024},
}
graph[sub(7)] = {
"class_type": "EmptyFlux2LatentImage",
"inputs": {"width": 1024, "height": 1024, "batch_size": 1},
}
graph[sub(8)] = {
"class_type": "SamplerCustomAdvanced",
"inputs": {
"noise": [sub(4), 0],
"guider": [sub(3), 0],
"sampler": [sub(5), 0],
"sigmas": [sub(6), 0],
"latent_image": [sub(7), 0],
},
}
graph[sub(9)] = {
"class_type": "VAEDecode",
"inputs": {"samples": [sub(8), 0], "vae": ["11", 0]},
}
graph[f"{1400 + branch}"] = {
"class_type": "SaveImage",
"inputs": {"filename_prefix": ["35", 0], "images": [sub(9), 0]},
}
return graph
def _top_level(graph):
return {key: node for key, node in graph.items() if ":" not in key}
def _f01():
graph = _flux2_branches(24)
return _basic(graph, _ui(_top_level(graph), 450, subgraphs=24))
def _f01b():
graph = _flux2_branches(1)
return _basic(graph, _ui(_top_level(graph), subgraphs=1))
def _f02():
graph = _ksampler_graph(_enc(POS), _enc(NEG))
return _basic(graph, _ui(graph, 600))
def _f03():
side = {
f"{100 + index}": {"class_type": "PrimitiveInt", "inputs": {"value": 1}}
for index in range(499)
}
return _basic(_ksampler_graph(_enc(POS), _enc(NEG), extra=side))
def _f04():
graph = _ksampler_graph(_enc(POS), _enc(NEG))
return _basic(graph, _ui(graph, big_widget="A" * 2_500_000))
def _f05():
graph = _ksampler_graph(_enc(POS), _enc(NEG))
return _basic(graph, _ui(graph, big_widget='"' * 900_000))
def _f06a():
sdxl = {
"class_type": "CLIPTextEncodeSDXL",
"inputs": {
"text_g": POS,
"text_l": POS,
"width": 1024,
"height": 1024,
"crop_w": 0,
"crop_h": 0,
"target_width": 1024,
"target_height": 1024,
"clip": ["4", 1],
},
}
return _basic(_ksampler_graph(sdxl, _enc(NEG)))
def _f06b():
flux = {
"class_type": "CLIPTextEncodeFlux",
"inputs": {"clip_l": POS, "t5xxl": POS, "guidance": 3.5, "clip": ["4", 1]},
}
return _basic(_ksampler_graph(flux, _enc(NEG)))
def _f06c():
guidance = {
"class_type": "FluxGuidance",
"inputs": {"guidance": 3.5, "conditioning": ["10", 0]},
}
return _basic(_ksampler_graph(guidance, _enc(NEG), extra={"10": _enc(POS)}))
def _f06d():
combine = {
"class_type": "ConditioningCombine",
"inputs": {"conditioning_1": ["10", 0], "conditioning_2": ["11", 0]},
}
extra = {"10": _enc(POS), "11": _enc("a second subject")}
return _basic(_ksampler_graph(combine, _enc(NEG), extra=extra))
def _f06e():
def apply():
return {
"class_type": "ControlNetApplyAdvanced",
"inputs": {
"positive": ["10", 0],
"negative": ["11", 0],
"control_net": ["12", 0],
"image": ["13", 0],
"strength": 1.0,
"start_percent": 0,
"end_percent": 1,
},
}
extra = {
"10": _enc(POS),
"11": _enc(NEG),
"12": {
"class_type": "ControlNetLoader",
"inputs": {"control_net_name": "control.safetensors"},
},
"13": {"class_type": "LoadImage", "inputs": {"image": "control.png"}},
}
graph = _ksampler_graph(apply(), apply(), extra=extra)
graph["3"]["inputs"]["negative"] = ["7", 1]
return _basic(graph)
def _f07a():
return _basic(_ksampler_graph(_enc(POS), _enc(NEG), "KSamplerAdvanced"))
def _f07b():
graph = _ksampler_graph(_enc(POS), _enc(NEG))
graph["3"] = {
"class_type": "SamplerCustomAdvanced",
"inputs": {
"noise": ["20", 0],
"guider": ["21", 0],
"sampler": ["22", 0],
"sigmas": ["23", 0],
"latent_image": ["5", 0],
},
}
graph.update(
{
"20": {"class_type": "RandomNoise", "inputs": {"noise_seed": 42}},
"21": {
"class_type": "CFGGuider",
"inputs": {
"model": ["4", 0],
"positive": ["6", 0],
"negative": ["7", 0],
"cfg": 6.5,
},
},
"22": {"class_type": "KSamplerSelect", "inputs": {"sampler_name": "euler"}},
"23": {
"class_type": "BasicScheduler",
"inputs": {
"model": ["4", 0],
"scheduler": "normal",
"steps": 25,
"denoise": 1.0,
},
},
}
)
return _basic(graph)
def _f08():
extra = {
"30": {
"class_type": "StringConcatenate",
"inputs": {"string_a": ["31", 0], "string_b": ", winter", "delimiter": ""},
},
"31": {"class_type": "PrimitiveStringMultiline", "inputs": {"value": POS}},
}
return _basic(_ksampler_graph(_enc(["30", 0]), _enc(NEG), extra=extra))
def _f09():
graph = _ksampler_graph(_enc(POS), _enc(NEG))
graph["40"] = {
"class_type": "LatentUpscaleBy",
"inputs": {
"upscale_method": "nearest-exact",
"scale_by": 1.5,
"samples": ["3", 0],
},
}
graph["41"] = {
"class_type": "KSampler",
"inputs": {**graph["3"]["inputs"], "latent_image": ["40", 0], "denoise": 0.5},
}
graph["8"]["inputs"]["samples"] = ["41", 0]
return _basic(graph)
def _f10():
latent = {
"class_type": "VAEEncode",
"inputs": {"pixels": ["50", 0], "vae": ["4", 2]},
}
extra = {"50": {"class_type": "LoadImage", "inputs": {"image": "input.png"}}}
return _basic(_ksampler_graph(_enc(POS), _enc(NEG), latent=latent, extra=extra))
def _f11():
latent = {
"class_type": "EmptyHunyuanLatentVideo",
"inputs": {"width": 640, "height": 384, "length": 17, "batch_size": 1},
}
return _basic(_ksampler_graph(_enc(POS), _enc(NEG), latent=latent))
def _f12():
def link(name, link_id):
return {"name": name, "link": link_id}
workflow = {
"nodes": [
{
"id": 3,
"type": "KSampler",
"mode": 0,
"inputs": [
link("model", 1),
link("positive", 2),
link("negative", 3),
link("latent_image", 4),
],
"widgets_values": [42, "fixed", 25, 6.5, "euler", "normal", 1.0],
},
{
"id": 4,
"type": "CheckpointLoaderSimple",
"mode": 0,
"inputs": [],
"widgets_values": ["sdxl_base.safetensors"],
},
{
"id": 6,
"type": "CLIPTextEncode",
"mode": 0,
"inputs": [],
"widgets_values": [POS],
},
{
"id": 7,
"type": "CLIPTextEncode",
"mode": 0,
"inputs": [],
"widgets_values": [NEG],
},
{
"id": 5,
"type": "EmptyLatentImage",
"mode": 0,
"inputs": [],
"widgets_values": [1024, 1024, 1],
},
],
"links": [
[1, 4, 0, 3, 0, "MODEL"],
[2, 6, 0, 3, 1, "CONDITIONING"],
[3, 7, 0, 3, 2, "CONDITIONING"],
[4, 5, 0, 3, 3, "LATENT"],
],
}
return _basic(None, workflow)
def _f14():
"""ComfyUI writes ``_meta`` titles and ``"is_changed": [NaN]`` outside inputs."""
graph = _ksampler_graph(_enc(POS), _enc(NEG))
for node in graph.values():
node["_meta"] = {"title": node["class_type"]}
graph["4"]["is_changed"] = [float("nan")]
return _basic(graph, _ui(graph))
def _f15():
"""Byte-identical duplicate ``prompt``/``workflow`` chunks (four copies each)."""
graph = _ksampler_graph(_enc(POS), _enc(NEG))
workflow = _ui(graph)
prompt_text, workflow_text = json.dumps(graph), json.dumps(workflow)
extra = [("prompt", prompt_text), ("workflow", workflow_text)] * 3
return _basic(graph, workflow, extra)
def _f16():
"""A second ``prompt`` chunk that differs from the first."""
graph = _ksampler_graph(_enc(POS), _enc(NEG))
other = _ksampler_graph(_enc("a grey wolf in fog"), _enc(NEG))
return _basic(graph, _ui(graph), [("prompt", json.dumps(other))])
def _f17():
"""SamplerCustomAdvanced + BasicGuider only: no positive/negative inputs."""
graph = {
"10": {
"class_type": "UNETLoader",
"inputs": {"unet_name": "flux1-dev.safetensors", "weight_dtype": "default"},
},
"11": {
"class_type": "DualCLIPLoader",
"inputs": {
"clip_name1": "clip_l.safetensors",
"clip_name2": "t5xxl.safetensors",
"type": "flux",
},
},
"12": {"class_type": "VAELoader", "inputs": {"vae_name": "ae.safetensors"}},
"6": {
"class_type": "CLIPTextEncode",
"inputs": {"text": POS, "clip": ["11", 0]},
},
"26": {
"class_type": "FluxGuidance",
"inputs": {"guidance": 3.5, "conditioning": ["6", 0]},
},
"22": {
"class_type": "BasicGuider",
"inputs": {"model": ["10", 0], "conditioning": ["26", 0]},
},
"25": {"class_type": "RandomNoise", "inputs": {"noise_seed": 42}},
"16": {"class_type": "KSamplerSelect", "inputs": {"sampler_name": "euler"}},
"17": {
"class_type": "BasicScheduler",
"inputs": {
"model": ["10", 0],
"scheduler": "simple",
"steps": 20,
"denoise": 1.0,
},
},
"27": {
"class_type": "EmptySD3LatentImage",
"inputs": {"width": 1024, "height": 1024, "batch_size": 1},
},
"13": {
"class_type": "SamplerCustomAdvanced",
"inputs": {
"noise": ["25", 0],
"guider": ["22", 0],
"sampler": ["16", 0],
"sigmas": ["17", 0],
"latent_image": ["27", 0],
},
},
"8": {
"class_type": "VAEDecode",
"inputs": {"samples": ["13", 0], "vae": ["12", 0]},
},
"9": {
"class_type": "SaveImage",
"inputs": {"filename_prefix": "ComfyUI", "images": ["8", 0]},
},
}
return _basic(graph, _ui(graph))
_BUILDERS = {
"f01": _f01,
"f01b": _f01b,
"f02": _f02,
"f03": _f03,
"f04": _f04,
"f05": _f05,
"f06a": _f06a,
"f06b": _f06b,
"f06c": _f06c,
"f06d": _f06d,
"f06e": _f06e,
"f07a": _f07a,
"f07b": _f07b,
"f08": _f08,
"f09": _f09,
"f10": _f10,
"f11": _f11,
"f12": _f12,
"f14": _f14,
"f15": _f15,
"f16": _f16,
"f17": _f17,
}
def build(name):
"""Return a fresh ``{"prompt", "workflow", "extra_chunks"}`` for one fixture."""
try:
builder = _BUILDERS[name]
except KeyError:
raise KeyError(f"Unknown inspection parity fixture: {name}") from None
return builder()
def fixture_path(name, directory=FIXTURE_DIR):
return Path(directory) / f"{name}.png"
def expected_text_chunks(built):
"""The decoded ``(keyword, text)`` chunks ``write_png`` emits, in file order."""
chunks = []
if built["prompt"] is not None:
chunks.append(("prompt", json.dumps(built["prompt"])))
if built["workflow"] is not None:
chunks.append(("workflow", json.dumps(built["workflow"])))
return chunks + [tuple(chunk) for chunk in built["extra_chunks"]]
def _raw_chunk(kind, data):
return (
struct.pack(">I", len(data))
+ kind
+ data
+ struct.pack(">I", zlib.crc32(kind + data) & 0xFFFFFFFF)
)
def write_png(path, built):
"""Write ``built`` as core SaveImage does, then splice extra raw text chunks."""
info = PngInfo()
if built["prompt"] is not None:
info.add_text("prompt", json.dumps(built["prompt"]))
if built["workflow"] is not None:
info.add_text("workflow", json.dumps(built["workflow"]))
buffer = io.BytesIO()
with Image.new("RGB", (64, 64), (40, 80, 120)) as image:
image.save(buffer, format="PNG", pnginfo=info, compress_level=4)
data = buffer.getvalue()
iend = data.rindex(b"IEND") - 4
extra = b"".join(
_raw_chunk(b"tEXt", key.encode("latin-1") + b"\0" + text.encode("latin-1"))
for key, text in built["extra_chunks"]
)
with open(path, "wb") as stream:
stream.write(data[:iend] + extra + data[iend:])
def write_all(directory, names=COMMITTED):
directory = Path(directory)
directory.mkdir(parents=True, exist_ok=True)
for name in names:
write_png(fixture_path(name, directory), build(name))
return [fixture_path(name, directory) for name in names]
def read_text_chunks(path):
"""Decode every tEXt/zTXt/iTXt chunk as ``(keyword, text)``, duplicates kept."""
with open(path, "rb") as stream:
data = stream.read()
if not data.startswith(_SIGNATURE):
raise ValueError(f"Not a PNG: {path}")
chunks, offset = [], len(_SIGNATURE)
while offset < len(data):
length, kind = struct.unpack(">I4s", data[offset : offset + 8])
body = data[offset + 8 : offset + 8 + length]
offset += 12 + length
if kind == b"tEXt":
key, _, text = body.partition(b"\0")
chunks.append((key.decode("latin-1"), text.decode("latin-1")))
elif kind == b"zTXt":
key, _, rest = body.partition(b"\0")
chunks.append(
(key.decode("latin-1"), zlib.decompress(rest[1:]).decode("latin-1"))
)
elif kind == b"iTXt":
key, _, rest = body.partition(b"\0")
flag, fields = rest[0], rest[2:].split(b"\0", 2)
text = zlib.decompress(fields[2]) if flag else fields[2]
chunks.append((key.decode("latin-1"), text.decode("utf-8")))
elif kind == b"IEND":
break
return chunks
def load_vectors(path=VECTORS_PATH):
with open(path, encoding="utf-8") as stream:
return json.load(stream)
def _role_text(status, text):
if status == "resolved":
return text
return ABSENT if status == "absent" else None
def _observe_inspection(read):
from utils.prompt_inspector import inspect_prompt_evidence
from utils.prompt_pair_resolver import INSPECTION_PROFILE
if read is None: # inspect_current_file's unreadable-media evidence
return "unresolved", "Media metadata could not be read", None, None, {}
evidence = inspect_prompt_evidence(read.metadata, profile=INSPECTION_PROFILE)
resolution, prompts = evidence["resolution"], evidence["prompts"]
roles = [
_role_text(prompts[role]["status"], prompts[role]["text"])
for role in ("positive", "negative")
]
positive, negative = roles
return resolution["state"], resolution["reason"], positive, negative, resolution
def _observe_default(path):
from utils.prompt_pair_resolver import DEFAULT_PROFILE, resolve_prompt_pair
# Last chunk wins, as Pillow's ``image.text`` (the metadata extractor) does.
records = dict(read_text_chunks(path))
canonical = {
key: value for key, value in records.items() if key in ("prompt", "parameters")
}
if isinstance(canonical.get("prompt"), str):
canonical["prompt"] = json.loads(canonical["prompt"])
resolved = resolve_prompt_pair(canonical, profile=DEFAULT_PROFILE)
state = resolved["state"]
roles = [
(
resolved[role]["text"]
if state == "resolved" and isinstance(resolved[role], dict)
else (ABSENT if state == "absent" else None)
)
for role in ("positive", "negative")
]
resolved = {**resolved, "candidate_count": len(resolved.get("candidates") or [])}
positive, negative = roles
return "available", state, resolved.get("reason", ""), positive, negative, resolved
def _observe_path(vector, path):
from utils.prompt_inspector import MetadataReadError, read_inspection_file
if vector["profile"] != "inspection":
return _observe_default(path)
try:
read, status = read_inspection_file(path), "available"
except MetadataReadError:
read, status = None, "unreadable"
state, reason, positive, negative, resolution = _observe_inspection(read)
return status, state, reason, positive, negative, resolution
def observe(vector, directory=FIXTURE_DIR):
"""Today's outcome for one vector, shaped like ``vector["expect"]`` minus facts.
``inspection`` is the workspace inspector path (``read_inspection_file`` then
``inspect_prompt_evidence``) with ``INSPECTION_PROFILE``. ``default`` pins
the resolver's ``DEFAULT_PROFILE`` only: ``resolve_prompt_pair`` on the role
records decoded from the PNG text chunks. Ingestion's own readers are
covered by their own suites.
Committed fixtures are read from ``directory``, which must contain them.
Test-time fixtures are built into a temporary directory; the file is
written and closed before it is read, and closed again before cleanup.
"""
name = vector["fixture"]
if name in TEST_TIME:
with tempfile.TemporaryDirectory() as temporary:
path = fixture_path(name, temporary)
write_png(path, build(name))
observed = _observe_path(vector, path)
else:
path = fixture_path(name, directory)
if not path.is_file():
raise FileNotFoundError(path)
observed = _observe_path(vector, path)
status, state, reason, positive, negative, resolution = observed
return {
"metadata_status": status,
"state": state,
"reason": reason,
"reason_code": resolution.get("reason_code"),
"selection": resolution.get("selection"),
"positive": positive,
"negative": negative,
"candidate_count": resolution.get("candidate_count", 0),
}
def main(argv=None):
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
parser.add_argument("--write", required=True, help="output directory")
args = parser.parse_args(argv)
for written in write_all(args.write):
print(written)
if __name__ == "__main__":
main()
+26 -4
View File
@@ -380,11 +380,33 @@ test("conservative facts bound cycles fan-out visits and link depth", () => {
custom.s = { class_type: "SamplerCustomAdvanced", inputs: { noise: ["noise", 0] } };
custom.noise = { class_type: "RandomNoise", inputs: { noise_seed: ["p0", 0] } };
assert.equal(factsFor(custom).seed.status, "unresolved", "the provider itself consumes one of the 64 allowed links");
});
// Spec section 10: graph size is not a facts bound. Only the per-lookup visit
// set (MAX_VISITS) and the projection's 20,000-node ceiling (`limited`) bound work.
test("facts resolve on a parsed 600-node graph with one KSampler", () => {
const large = factGraph();
for (let index = 0; index < 498; index++) large[`unused${index}`] = { class_type: "PrimitiveInt", inputs: { value: index } };
assert.equal(factsFor(large).seed.value, "9007199254740993");
large.extra = { class_type: "PrimitiveInt", inputs: { value: 1 } };
assert.equal(factsFor(large).seed.status, "unresolved");
for (let index = 0; index < 598; index++) large[`unused${index}`] = { class_type: "PrimitiveInt", inputs: { value: index } };
assert.equal(Object.keys(large).length, 600);
const facts = factsFor(large);
for (const [field, value] of [["seed", "9007199254740993"], ["steps", "28"], ["cfg", "6.5000000000000000001"]]) {
assert.equal(facts[field].status, "known", field);
assert.equal(facts[field].value, value);
assert.equal(facts[field].reason, null);
}
});
test("workflow facts are not gated by UI node count", () => {
const nodes = [
{ id: 1, type: "CheckpointLoaderSimple", widgets_values: ["wide.safetensors"] },
{ id: 2, type: "KSampler", widgets_values: [42, "fixed", 20, "7", "euler", "normal"], inputs: [{ name: "model", link: 1 }] },
];
for (let index = 0; index < 598; index++) nodes.push({ id: `note${index}`, type: "Note", widgets_values: [`note ${index}`] });
const workflow = { nodes, links: [[1, 1, 0, 2, 0, "MODEL"]] };
const facts = factsFor(null, { prompt_graph: absentFactGraph(), workflow_graph: parsedFactGraph(workflow) });
assert.deepEqual(factFields.map(field => facts[field].status), Array(6).fill("known"));
assert.deepEqual(factFields.map(field => facts[field].value), ["42", "20", "7", "euler", "normal", "wide.safetensors"]);
assert.ok(Object.values(facts).every(fact => fact.source === "workflow" && fact.reason === null));
});
// Same shape as the workflow attached to #75: PromptManager nodes are saved as
@@ -0,0 +1,82 @@
// Run with: node --test tests/js/
// Characterization, not a specification: pins what the legacy viewers
// (web/metadata.html, web/js/gallery.js) show today on the synthetic inspection
// parity fixtures, so the #240 workspace work cannot change them by accident.
// Some answers are known to be wrong (f01b shows the preview-only detailer text
// as both roles; f16 shows the last of two conflicting prompt chunks). They are
// recorded as-is and intentionally unchanged (spec section 11).
const test = require("node:test");
const assert = require("node:assert/strict");
const fs = require("node:fs");
const path = require("node:path");
const { parsePngTextChunks } = require("../../web/js/png-metadata.js");
const { extractGenerationParams } = require("../../web/js/comfy-metadata.js");
const FIXTURES = path.join(__dirname, "..", "fixtures", "inspection_parity");
const DETAILER = "Detailer pass: refine the face";
// Copy of the legacy pages' inline glue (NaN cleanup and field order): web/metadata.html
// extractComfyUIData (line 107) and web/js/gallery.js extractComfyUIData (line 1643).
// The spec only promises extractGenerationParams is unchanged, so the glue is copied here.
function legacyComfyData(metadata) {
const parse = text => JSON.parse(text.replace(/:\s*NaN\b/g, ": null").replace(/\bNaN\b/g, "null"));
const first = fields => {
for (const field of fields) {
if (!metadata[field]) continue;
try { return parse(metadata[field]); } catch { /* the pages try the next field */ }
}
return null;
};
return { workflow: first(["workflow", "Workflow", "comfy", "ComfyUI"]), prompt: first(["prompt", "Prompt", "parameters", "Parameters"]) };
}
function legacyParams(name) {
const bytes = fs.readFileSync(path.join(FIXTURES, `${name}.png`));
const buffer = bytes.buffer.slice(bytes.byteOffset, bytes.byteOffset + bytes.byteLength);
return extractGenerationParams(legacyComfyData(parsePngTextChunks(buffer) || {}));
}
const EXPECTED = {
f01b: { positivePrompt: "Detailer pass: refine the face", negativePrompt: "Detailer pass: refine the face", checkpoint: "flux-2-klein-9b-fp8.safetensors", steps: 20, cfgScale: 1, sampler: "euler", seed: 271828182845 },
f02: { positivePrompt: "a red fox in snow, detailed fur", negativePrompt: "blurry, lowres", checkpoint: "sdxl_base.safetensors", steps: 25, cfgScale: 6.5, sampler: "euler", seed: 42 },
f03: { positivePrompt: "a red fox in snow, detailed fur", negativePrompt: "blurry, lowres", checkpoint: "sdxl_base.safetensors", steps: 25, cfgScale: 6.5, sampler: "euler", seed: 42 },
f06a: { positivePrompt: "No prompt found", negativePrompt: "blurry, lowres", checkpoint: "sdxl_base.safetensors", steps: 25, cfgScale: 6.5, sampler: "euler", seed: 42 },
f06b: { positivePrompt: "No prompt found", negativePrompt: "blurry, lowres", checkpoint: "sdxl_base.safetensors", steps: 25, cfgScale: 6.5, sampler: "euler", seed: 42 },
f06c: { positivePrompt: "a red fox in snow, detailed fur", negativePrompt: "blurry, lowres", checkpoint: "sdxl_base.safetensors", steps: 25, cfgScale: 6.5, sampler: "euler", seed: 42 },
f06d: { positivePrompt: "a red fox in snow, detailed fur", negativePrompt: "blurry, lowres", checkpoint: "sdxl_base.safetensors", steps: 25, cfgScale: 6.5, sampler: "euler", seed: 42 },
f06e: { positivePrompt: "a red fox in snow, detailed fur", negativePrompt: "blurry, lowres", checkpoint: "sdxl_base.safetensors", steps: 25, cfgScale: 6.5, sampler: "euler", seed: 42 },
f07a: { positivePrompt: "a red fox in snow, detailed fur", negativePrompt: "blurry, lowres", checkpoint: "sdxl_base.safetensors", steps: 25, cfgScale: 6.5, sampler: "euler", seed: 42 },
f07b: { positivePrompt: "a red fox in snow, detailed fur", negativePrompt: "blurry, lowres", checkpoint: "sdxl_base.safetensors", steps: 25, cfgScale: 6.5, sampler: "euler", seed: 42 },
f08: { positivePrompt: "a red fox in snow, detailed fur, winter", negativePrompt: "blurry, lowres", checkpoint: "sdxl_base.safetensors", steps: 25, cfgScale: 6.5, sampler: "euler", seed: 42 },
f09: { positivePrompt: "a red fox in snow, detailed fur", negativePrompt: "blurry, lowres", checkpoint: "sdxl_base.safetensors", steps: 25, cfgScale: 6.5, sampler: "euler", seed: 42 },
f10: { positivePrompt: "a red fox in snow, detailed fur", negativePrompt: "blurry, lowres", checkpoint: "sdxl_base.safetensors", steps: 25, cfgScale: 6.5, sampler: "euler", seed: 42 },
f11: { positivePrompt: "a red fox in snow, detailed fur", negativePrompt: "blurry, lowres", checkpoint: "sdxl_base.safetensors", steps: 25, cfgScale: 6.5, sampler: "euler", seed: 42 },
f12: { positivePrompt: "a red fox in snow, detailed fur", negativePrompt: "blurry, lowres", checkpoint: "sdxl_base.safetensors", steps: 25, cfgScale: 6.5, sampler: "euler", seed: 42 },
f14: { positivePrompt: "a red fox in snow, detailed fur", negativePrompt: "blurry, lowres", checkpoint: "sdxl_base.safetensors", steps: 25, cfgScale: 6.5, sampler: "euler", seed: 42 },
f15: { positivePrompt: "a red fox in snow, detailed fur", negativePrompt: "blurry, lowres", checkpoint: "sdxl_base.safetensors", steps: 25, cfgScale: 6.5, sampler: "euler", seed: 42 },
f16: { positivePrompt: "a grey wolf in fog", negativePrompt: "blurry, lowres", checkpoint: "sdxl_base.safetensors", steps: 25, cfgScale: 6.5, sampler: "euler", seed: 42 },
f17: { positivePrompt: "No prompt found", negativePrompt: "No negative prompt found", checkpoint: "flux1-dev.safetensors", steps: "Unknown", cfgScale: "Unknown", sampler: "euler", seed: "Unknown" },
};
test("legacy extractGenerationParams characterization covers every committed fixture", () => {
const committed = fs.readdirSync(FIXTURES).filter(file => file.endsWith(".png")).map(file => file.slice(0, -4)).sort();
assert.deepEqual(Object.keys(EXPECTED).sort(), committed);
});
for (const [name, expected] of Object.entries(EXPECTED)) {
test(`legacy extractGenerationParams characterization: ${name}`, () => {
assert.deepEqual(legacyParams(name), expected);
});
}
test("legacy extractGenerationParams characterization keeps f01b's detailer text as both roles", () => {
const params = legacyParams("f01b");
assert.equal(params.positivePrompt, DETAILER);
assert.equal(params.negativePrompt, DETAILER);
});
test("legacy extractGenerationParams characterization replaces NaN before parsing", () => {
const params = legacyParams("f14");
assert.equal(params.positivePrompt, "a red fox in snow, detailed fur");
assert.equal(params.seed, 42);
});
@@ -0,0 +1,25 @@
// Facts for encoded inspection responses, through the workspace's own path:
// projectInspectionPayload validates the wire, then extractGenerationFacts reads
// the bounded projection. Used by tests/test_inspection_parity_facts.py, which
// encodes each parity vector's file with the real route pipeline.
// stdin: [{name, body}] (body is the exact response JSON text).
// stdout: {name: {field: {status, value}}}.
import { createRequire } from 'node:module';
import { fileURLToPath, pathToFileURL } from 'node:url';
import path from 'node:path';
const here = path.dirname(fileURLToPath(import.meta.url));
const web = path.join(here, '..', '..', '..', 'web', 'js');
const require = createRequire(import.meta.url);
const { extractGenerationFacts } = require(path.join(web, 'comfy-metadata.js'));
const { projectInspectionPayload } = await import(pathToFileURL(path.join(web, 'workspace', 'inspection-state.mjs')).href);
const chunks = [];
for await (const chunk of process.stdin) chunks.push(chunk);
const output = {};
for (const { name, body } of JSON.parse(Buffer.concat(chunks).toString('utf8'))) {
const identity = { kind: 'rooted-file', key: `0:${name}.png`, root_index: 0, path: `${name}.png`, root_scope: null };
const record = projectInspectionPayload(JSON.parse(body), identity, null, extractGenerationFacts);
output[name] = Object.fromEntries(Object.entries(record.generation).map(([field, fact]) => [field, { status: fact.status, value: fact.value }]));
}
process.stdout.write(JSON.stringify(output));
+11
View File
@@ -54,3 +54,14 @@ test('reported location metadata export carries exact occurrence without assigni
source.identity={kind:'reported-location',key:'b'.repeat(64),path:'folder/a.png',occurrence:{run_id:'22222222-2222-2222-2222-222222222222',output_node_id:'𐀀',image_ordinal:255},history_epoch:'11111111-1111-1111-1111-111111111111',current_scope:'c'.repeat(64),reference:{folder_type:'temp',subfolder:'folder',filename:'a.png',path_style:'posix',root_key:null}};
const value=JSON.parse(inspectionDownloads(source).metadata.bodyText);assert.deepEqual(value.identity,source.identity);assert.equal(value.identity.root_index,undefined);assert.equal(value.context_id,undefined);assert.equal(value.file_identity,undefined);assert.deepEqual(value.prompts,source.prompts);
});
test('limited workflow record exports its representation and disables workflow download',async()=>{
const {inspectionDownloads}=await load(),source=record();
source.metadata.record_formats.workflow='limited';delete source.metadata.raw.workflow;source.metadata.workflow_graph={status:'limited',value:null};
const downloads=inspectionDownloads(source),value=JSON.parse(downloads.metadata.bodyText);
assert.equal(value.metadata.record_formats.workflow,'limited');
assert.equal(value.metadata.representations.workflow,'Record exceeds inspection limits; text is unavailable');
assert.equal(value.metadata.raw.workflow,undefined);
assert.equal(downloads.workflow.available,false);assert.equal(downloads.workflow.bodyText,null);
assert.equal(downloads.workflow.reason,'Workflow exceeds inspection limits. Download the original file to inspect it externally.');
});
+82 -1
View File
@@ -136,7 +136,9 @@ test('record rejects contradictory omission provenance before extracting facts',
data => { data.metadata.record_formats.workflow = 'original-text'; data.metadata.workflow_graph = { status: 'parsed', value: { nodes: [] } }; },
data => { Object.assign(data.metadata, { status: 'limited', omitted_reason: 'response-size-limit', raw: {}, prompt_graph: { status: 'limited', value: null } }); data.metadata.record_formats.workflow = 'invalid'; data.metadata.workflow_graph = { status: 'malformed', value: null }; },
data => { data.metadata.record_formats.workflow = 'invalid'; data.metadata.raw.workflow = 'null'; data.metadata.workflow_graph = { status: 'limited', value: null }; },
data => { data.metadata.record_formats.workflow = 'invalid'; data.metadata.workflow_graph = { status: 'limited', value: null }; },
// The response budget omits the workflow graph before the prompt graph, so an
// omitted invalid prompt graph means the workflow graph was omitted too.
data => { data.metadata.record_formats.prompt = 'invalid'; data.metadata.raw = {}; data.metadata.prompt_graph = { status: 'limited', value: null }; data.metadata.record_formats.workflow = 'original-text'; data.metadata.raw.workflow = '{"nodes":[]}'; data.metadata.workflow_graph = { status: 'parsed', value: { nodes: [] } }; },
]) {
const data = envelope(); mutate(data); let extractions = 0;
assert.throws(() => toRecord(data, image(), () => { extractions++; return extractGenerationFacts({}); }), TypeError);
@@ -144,6 +146,85 @@ test('record rejects contradictory omission provenance before extracting facts',
}
});
function limitedWorkflow(data = envelope()) {
data.metadata.record_formats.workflow = 'limited';
data.metadata.workflow_graph = { status: 'limited', value: null };
return data;
}
test('record accepts limited workflow format without raw and with a limited graph', () => {
const toRecord = contract('toInspectionRecord');
const record = toRecord(limitedWorkflow(), image(), extractGenerationFacts);
assert.deepEqual(record.metadata.record_formats, { prompt: 'reserialized-json', workflow: 'limited' });
assert.deepEqual(record.metadata.raw, { prompt: '{"rounded":"raw copy"}' });
assert.deepEqual(record.metadata.workflow_graph, { status: 'limited', value: null });
assert.equal(record.metadata.status, 'available');
assert.equal(record.generation.seed.value, '9007199254740993', 'a limited workflow never hides prompt-graph facts');
const workflowOnly = limitedWorkflow();
Object.assign(workflowOnly.metadata, { raw: {}, record_formats: { workflow: 'limited' }, prompt_graph: { status: 'absent', value: null } });
const facts = toRecord(workflowOnly, image(), extractGenerationFacts).generation;
assert.ok(Object.values(facts).every(fact => fact.status === 'unresolved' && fact.reason === 'projection-limited'));
const omitted = limitedWorkflow();
Object.assign(omitted.metadata, { raw: {}, parameters: null, status: 'limited', omitted_reason: 'response-size-limit', prompt_graph: { status: 'limited', value: null } });
assert.deepEqual(toRecord(omitted, image(), extractGenerationFacts).metadata.record_formats, { prompt: 'reserialized-json', workflow: 'limited' });
});
test('record rejects limited format with raw present', () => {
const toRecord = contract('toInspectionRecord');
for (const mutate of [
data => { data.metadata.raw.workflow = '{"nodes":[]}'; },
data => { data.metadata.workflow_graph = { status: 'parsed', value: { nodes: [] } }; },
data => { data.metadata.workflow_graph = { status: 'malformed', value: null }; },
data => { data.metadata.workflow_graph = { status: 'absent', value: null }; },
// Role records fail closed on the server: they are never limited.
data => { data.metadata.record_formats.prompt = 'limited'; delete data.metadata.raw.prompt; data.metadata.prompt_graph = { status: 'limited', value: null }; },
data => { data.metadata.record_formats.parameters = 'limited'; },
]) {
const data = limitedWorkflow(); mutate(data); let extractions = 0;
assert.throws(() => toRecord(data, image(), () => { extractions++; return extractGenerationFacts({}); }), TypeError);
assert.equal(extractions, 0);
}
});
test('record accepts the workflow-first response omission of an invalid workflow record', () => {
const data = envelope();
data.metadata.record_formats.workflow = 'invalid';
data.metadata.workflow_graph = { status: 'limited', value: null };
const record = contract('toInspectionRecord')(data, image(), extractGenerationFacts);
assert.deepEqual(record.metadata.prompt_graph.status, 'parsed', 'prompt evidence outlives workflow evidence');
assert.equal(record.generation.seed.value, '9007199254740993');
});
test('record accepts closed reason codes and rejects contradictory ones', () => {
const toRecord = contract('toInspectionRecord');
const codes = ['work-limit', 'depth-limit', 'text-limit', 'graph-ceiling', 'unsupported-node', 'runtime-value', 'composite', 'no-sampler', 'malformed', 'multistage-differs'];
for (const reason_code of codes) {
const data = envelope(); Object.assign(data.resolution, { state: 'unresolved', reason_code, node: null });
assert.doesNotThrow(() => toRecord(data, image(), extractGenerationFacts), reason_code);
}
const node = envelope(); Object.assign(node.resolution, { state: 'unresolved', reason_code: 'unsupported-node', node: { id: '7', class_type: 'LoRAfromText' } });
assert.doesNotThrow(() => toRecord(node, image(), extractGenerationFacts));
node.resolution.node.class_type = null; // the server names a non-string class as null
assert.doesNotThrow(() => toRecord(node, image(), extractGenerationFacts));
for (const state of ['resolved', 'absent', 'ambiguous', 'unresolved']) {
const data = envelope(); Object.assign(data.resolution, { state, reason_code: null, node: null });
assert.doesNotThrow(() => toRecord(data, image(), extractGenerationFacts), state);
}
for (const change of [
{ state: 'unresolved', reason_code: 'guess' }, { state: 'unresolved', reason_code: 1 },
{ state: 'resolved', reason_code: 'work-limit' }, { state: 'absent', reason_code: 'no-sampler' },
{ state: 'unresolved', reason_code: null, node: { id: '7', class_type: 'X' } },
{ state: 'unresolved', reason_code: 'unsupported-node', node: { id: 7, class_type: 'X' } },
{ state: 'unresolved', reason_code: 'unsupported-node', node: { id: '7' } },
{ state: 'unresolved', reason_code: 'unsupported-node', node: { id: '7', class_type: 7 } },
{ state: 'unresolved', reason_code: 'unsupported-node', node: 'X' },
]) {
const data = envelope(); Object.assign(data.resolution, change); let extractions = 0;
assert.throws(() => toRecord(data, image(), () => { extractions++; return extractGenerationFacts({}); }), TypeError, JSON.stringify(change));
assert.equal(extractions, 0);
}
});
test('record rejects root path media and invalid versioned fields with retryable TypeError', () => {
const toRecord = contract('toInspectionRecord');
for (const mutate of [
+107 -9
View File
@@ -7,8 +7,10 @@ from PIL import Image, PngImagePlugin
try:
from test_api_images import ImageAPITestCase, make_png
import inspection_parity as parity
except ImportError:
from tests.test_api_images import ImageAPITestCase, make_png
from tests import inspection_parity as parity
from utils.pair_execution import EXECUTION_KEY, record_pair_execution
from database.pair_ingestion import ingest_resolved_pair
@@ -311,7 +313,9 @@ class TestPromptInspector(ImageAPITestCase):
cases = [base[:-1], base[:30]]
index = base.index(b"tEXt")
broken = bytearray(base)
broken[index + 8] ^= 1
# Corrupt the retained record's body: its CRC is checked. (A corrupt
# keyword names an unretained chunk, which is skipped unchecked.)
broken[index + 4 + len(b"prompt\0") + 1] ^= 1
cases.append(bytes(broken))
for kind, payload in (
(b"tEXt", b"prompt\0{}"),
@@ -713,17 +717,93 @@ class TestPromptInspector(ImageAPITestCase):
self.assertLessEqual(len(await response.read()), 8 * 1024 * 1024)
self.assertNotIn(str(self.comfy_dir), json.dumps(result))
async def test_workflow_raw_uses_same_aggregate_budget_and_read(self):
from utils.prompt_inspector import MAX_METADATA_BYTES
async def test_workflow_has_its_own_budget_and_never_hides_prompts(self):
from utils.prompt_inspector import ROLE_METADATA_BYTES, WORKFLOW_METADATA_BYTES
path = self.output_dir / "image.png"
png(path, prompt=json.dumps(graph()), workflow="x" * MAX_METADATA_BYTES)
# Over the role budget, within the workflow budget: fully readable.
workflow = "x" * ROLE_METADATA_BYTES
png(path, prompt=json.dumps(graph()), workflow=workflow)
response, result = await self.inspect()
self.assertEqual(response.status, 200)
self.assertEqual(result.get("metadata", {}).get("status"), "unreadable")
self.assertEqual(result["prompts"]["positive"]["status"], "unresolved")
self.assertEqual(result["metadata"]["raw"], {})
self.assertEqual(result["metadata"]["record_formats"], {})
self.assertEqual(result["metadata"]["status"], "available")
self.assertEqual(result["prompts"]["positive"], self.side("portrait"))
self.assertEqual(result["metadata"]["raw"]["workflow"], workflow)
self.assertEqual(result["metadata"]["workflow_graph"]["status"], "malformed")
# Over the workflow budget: only the workflow is limited.
png(
path,
prompt=json.dumps(graph()),
workflow="x" * (WORKFLOW_METADATA_BYTES + 1),
)
response, result = await self.inspect()
self.assertEqual(response.status, 200)
self.assertEqual(result["metadata"]["status"], "available")
self.assertEqual(result["prompts"]["positive"], self.side("portrait"))
self.assertEqual(result["prompts"]["negative"], self.side("blurry"))
self.assertEqual(
result["metadata"]["record_formats"],
{"prompt": "original-text", "workflow": "limited"},
)
self.assertEqual(set(result["metadata"]["raw"]), {"prompt"})
self.assertEqual(
result["metadata"]["workflow_graph"], {"status": "limited", "value": None}
)
self.assertEqual(result["metadata"]["prompt_graph"]["status"], "parsed")
def parity_fixture(self, name):
path = self.output_dir / f"{name}.png"
parity.write_png(path, parity.build(name))
return path
async def test_inspect_route_resolves_506_node_graph(self):
# f03: 506 executable nodes. The default (ingestion) profile's 500-node
# gate reports "Resolution limit reached"; the inspection profile bounds
# work instead of graph size, so the route resolves the pair.
self.parity_fixture("f03")
response, result = await self.inspect("f03.png")
self.assertEqual(response.status, 200)
self.assertEqual(
result["prompts"],
{"positive": self.side(parity.POS), "negative": self.side(parity.NEG)},
)
self.assertEqual(result["resolution"]["state"], "resolved")
self.assertIsNone(result["resolution"]["reason_code"])
self.assertIsNone(result["resolution"]["node"])
self.assertEqual(result["metadata"]["prompt_graph"]["status"], "parsed")
async def test_inspect_route_reports_no_sampler_instead_of_absent(self):
# f17: a BasicGuider-only prompt graph is a record, never "not recorded".
self.parity_fixture("f17")
response, result = await self.inspect("f17.png")
self.assertEqual(response.status, 200)
self.assertEqual(
{role: side["status"] for role, side in result["prompts"].items()},
{"positive": "unresolved", "negative": "unresolved"},
)
self.assertEqual(result["resolution"]["state"], "unresolved")
self.assertEqual(result["resolution"]["reason_code"], "no-sampler")
self.assertEqual(
result["resolution"]["reason"],
"No supported sampler pair found (saw SamplerCustomAdvanced)",
)
async def test_inspect_route_exposes_reason_code_and_blocking_node(self):
self.parity_fixture("f06a")
_, result = await self.inspect("f06a.png")
resolution = result["resolution"]
self.assertEqual(resolution["state"], "unresolved")
self.assertEqual(resolution["reason_code"], "unsupported-node")
self.assertEqual(set(resolution["node"]), {"id", "class_type"})
self.assertIsInstance(resolution["node"]["class_type"], str)
png(self.output_dir / "image.png", prompt="{broken")
_, result = await self.inspect()
self.assertEqual(result["resolution"]["reason_code"], "malformed")
make_png(self.output_dir / "image.png")
_, result = await self.inspect()
self.assertEqual(result["resolution"]["state"], "absent")
self.assertIsNone(result["resolution"]["reason_code"])
self.assertIsNone(result["resolution"]["node"])
async def test_workflow_only_keeps_existing_role_uncertainty(self):
path = png(self.output_dir / "image.png", workflow={"nodes": []})
@@ -988,7 +1068,6 @@ class TestPromptInspector(ImageAPITestCase):
source = json.dumps(graph())
for suffix, expected in (
(',"duplicate":1,"duplicate":2', "malformed"),
(',"nonfinite":NaN', "malformed"),
(',"depth":' + "[" * 64 + "0" + "]" * 64, "limited"),
):
_, result = await self.video_inspect(
@@ -1002,6 +1081,25 @@ class TestPromptInspector(ImageAPITestCase):
self.assertIsNone(result["metadata"]["prompt_graph"]["value"])
self.assertEqual(result["metadata"]["workflow_graph"]["status"], expected)
async def test_video_carrier_nonfinite_outside_graph_inputs_is_tolerated(self):
# ComfyUI quirks: NaN in an unrelated carrier field, in ``is_changed`` and
# in ``_meta`` are validated and dropped; they never reach the projection.
source = graph()
source["3"]["is_changed"] = ["NAN_MARKER"]
source["3"]["_meta"] = {"title": "K", "weight": "NAN_MARKER"}
text = json.dumps(source).replace('"NAN_MARKER"', "NaN")
_, result = await self.video_inspect(
'{"prompt":' + text + ',"workflow":{"nodes":[]},"nonfinite":NaN}'
)
self.assertEqual(result["prompts"]["positive"], self.side("portrait"))
prompt_graph = result["metadata"]["prompt_graph"]
self.assertEqual(prompt_graph["status"], "parsed")
self.assertEqual(set(prompt_graph["value"]["3"]), {"class_type", "inputs"})
self.assertEqual(
result["metadata"]["workflow_graph"],
{"status": "parsed", "value": {"nodes": []}},
)
async def test_video_numeric_role_text_never_becomes_sendable(self):
source = graph(positive=123)
_, result = await self.video_inspect(json.dumps({"prompt": source}))
+84
View File
@@ -0,0 +1,84 @@
"""Ingestion keeps the default resolver profile; only inspection routes moved (D1).
The live monitor and the admin scan write pairs and links to the user's
database, so their resolver semantics stay byte-for-byte today's. f03 (506
executable nodes) separates the profiles: the default profile's 500-node gate
leaves it unresolved, while the inspection profile resolves it.
"""
import logging
import unittest
from unittest.mock import Mock, patch
from utils import prompt_pair_resolver as resolver
try:
from tests import inspection_parity as parity
except ImportError: # pragma: no cover - discover -s tests puts tests/ first
import inspection_parity as parity
GATED = "Resolution limit reached"
def _f03_metadata():
return {"prompt": parity.build("f03")["prompt"]}
class _Spy:
"""Wrap the real resolver, recording each call's profile and outcome."""
def __init__(self):
self.calls = []
def __call__(self, metadata, **kwargs):
result = resolver.resolve_prompt_pair(metadata, **kwargs)
self.calls.append((kwargs.get("profile", resolver.DEFAULT_PROFILE), result))
return result
def assert_default_and_gated(self, case):
case.assertTrue(self.calls, "the ingestion path resolved the metadata")
for profile, result in self.calls:
case.assertIs(profile, resolver.DEFAULT_PROFILE)
case.assertEqual((result["state"], result["reason"]), ("unresolved", GATED))
class TestIngestionKeepsDefaultProfile(unittest.TestCase):
def test_f03_separates_the_profiles(self):
metadata = _f03_metadata()
default = resolver.resolve_prompt_pair(metadata)
self.assertEqual((default["state"], default["reason"]), ("unresolved", GATED))
inspection = resolver.resolve_prompt_pair(
metadata, profile=resolver.INSPECTION_PROFILE
)
self.assertEqual(inspection["state"], "resolved")
def test_live_monitor_lookup_and_linking_use_the_default_profile(self):
from utils.image_monitor import ImageGenerationHandler
handler = ImageGenerationHandler.__new__(ImageGenerationHandler)
handler.db_manager = Mock()
handler.db_manager.get_prompt_by_hash.return_value = None
handler.logger = logging.getLogger("test.ingestion-profile")
spy, metadata = _Spy(), _f03_metadata()
with patch("utils.image_monitor.resolve_prompt_pair", spy):
handler._find_prompt_from_metadata(metadata)
handler.link_image_to_prompt("/out/f03.png", {"id": 1}, metadata)
self.assertEqual(len(spy.calls), 2)
spy.assert_default_and_gated(self)
handler.db_manager.link_image_to_prompt.assert_called_once()
def test_admin_scan_ingestion_uses_the_default_profile(self):
from py.api import admin
mixin = admin.AdminRoutesMixin.__new__(admin.AdminRoutesMixin)
mixin.db = Mock()
spy = _Spy()
with patch.object(admin, "resolve_prompt_pair", spy):
status = mixin._ingest_scanned_file("/out/f03.png", _f03_metadata())
self.assertEqual(status, "unresolved")
spy.assert_default_and_gated(self)
mixin.db.assert_not_called()
if __name__ == "__main__":
unittest.main()
+367
View File
@@ -0,0 +1,367 @@
"""The read-only corpus tool (``tools/inspection_corpus.py``) on synthetic fixtures.
The committed parity fixtures are copied into a disposable root. The tool must
reproduce the counts their ``vectors.json`` expectations imply, leave the root
byte-for-byte and mtime-for-mtime unchanged, refuse outputs inside the root or the
code tree, keep file names and prompt text out of its output, and compute H1
against a baseline file.
"""
import contextlib
import hashlib
import io
import json
import os
import shutil
import struct
import tempfile
import unittest
import unittest.mock
import zlib
from collections import Counter
from pathlib import Path
from tools import inspection_corpus as corpus
try:
from tests import inspection_parity as parity
from tests.open_handles import assert_closed
except ImportError: # pragma: no cover - discover -s tests puts tests/ first
import inspection_parity as parity
from open_handles import assert_closed
FIXTURES = Path(parity.__file__).resolve().parent / "fixtures" / "inspection_parity"
def _snapshot(root: Path) -> dict:
state = {}
for directory, subdirectories, names in os.walk(root):
for name in subdirectories + names:
path = Path(directory, name)
info = path.lstat()
digest = (
hashlib.sha256(path.read_bytes()).hexdigest()
if path.is_file()
else None
)
state[path.relative_to(root).as_posix()] = (
info.st_mtime_ns,
info.st_size,
digest,
)
return state
def _expected():
names = {path.stem for path in FIXTURES.glob("*.png")}
return [
vector
for vector in parity.load_vectors()
if vector["profile"] == "inspection"
and vector["anchor"] is None
and vector["fixture"] in names
]
def _chunk(kind: bytes, data: bytes) -> bytes:
crc = zlib.crc32(kind + data) & 0xFFFFFFFF
return struct.pack(">I", len(data)) + kind + data + struct.pack(">I", crc)
def _png(*texts: bytes) -> bytes:
header = _chunk(b"IHDR", struct.pack(">IIBBBBB", 1, 1, 8, 0, 0, 0, 0))
body = b"".join(_chunk(b"tEXt", text) for text in texts)
return corpus.PNG_SIGNATURE + header + body + _chunk(b"IEND", b"")
class CorpusToolTest(unittest.TestCase):
def setUp(self):
temporary = tempfile.TemporaryDirectory()
self.addCleanup(temporary.cleanup)
self.addCleanup(assert_closed, self, temporary.name)
self.base = Path(temporary.name)
self.root = self.base / "root"
shutil.copytree(FIXTURES, self.root / "nested")
(self.root / "nested" / "vectors.json").unlink()
self.node = shutil.which("node")
if self.node is None and os.environ.get("CI"):
self.fail("Node.js is required in CI for corpus facts")
def _run(self, *extra: str) -> tuple:
output = io.StringIO()
with contextlib.redirect_stdout(output), contextlib.redirect_stderr(output):
code = corpus.main(["--root", str(self.root), "--workers", "1", *extra])
return code, output.getvalue()
def _rows(self, path: Path) -> list:
return [json.loads(line) for line in path.read_text("utf-8").splitlines()]
def test_corpus_tool_aggregates_synthetic_directory_read_only(self):
before = _snapshot(self.root)
out, summary = self.base / "out.jsonl", self.base / "summary.txt"
code, printed = self._run("--out", str(out), "--summary", str(summary))
expected_code = corpus.EXIT_OK if self.node else corpus.EXIT_INVARIANT_FAILED
self.assertEqual(code, expected_code, printed)
self.assertEqual(_snapshot(self.root), before)
expected = _expected()
rows = self._rows(out)
self.assertEqual(len(rows), len(expected))
report = corpus.build_report(rows, None, corpus._STATE.code.cap)
states = Counter(vector["expect"]["state"] for vector in expected)
self.assertEqual(report["states"], states)
codes = Counter(
str(vector["expect"]["reason_code"] or "null") for vector in expected
)
self.assertEqual(report["reason_codes"], codes)
self.assertEqual(report["selection_kinds"], Counter({"null": len(expected)}))
self.assertEqual(report["harness_errors"], Counter())
# An unreadable file is an inspector outcome, not a harness error.
self.assertEqual(report["metadata_status"]["unreadable"], 1)
if self.node is not None:
self.assertEqual(report["invariants"]["harness"]["status"], "PASS")
for name in ("H2", "H3", "H4", "H5", "H6", "H8"):
self.assertEqual(report["invariants"][name]["status"], "PASS", name)
self.assertEqual(report["invariants"]["H1"]["status"], "N/A")
if self.node is not None:
known = sum(
1
for vector in expected
if sum(
fact["status"] == "known"
for fact in (vector["expect"]["facts"] or {}).values()
)
>= corpus.FACTS_KNOWN_TARGET
)
self.assertEqual(report["facts_known_at_least_4"], known)
self.assertEqual(report["facts_errors"], Counter())
self.assertEqual(summary.read_text("utf-8"), printed)
self.assertIn("\nstates: ", printed)
text = out.read_text("utf-8") + printed
for fixture in FIXTURES.glob("*.png"):
self.assertNotIn(fixture.name, text)
def test_outputs_inside_root_or_code_tree_are_refused(self):
before = _snapshot(self.root)
inside = self.root / "nested" / "out.jsonl"
code_tree = corpus.TOOL_TREE / "corpus-out.jsonl"
cases = [
["--out", str(inside)],
["--out", str(self.base / "ok.jsonl"), "--summary", str(code_tree)],
[
"--out",
str(self.base / "same.txt"),
"--summary",
str(self.base / "same.txt"),
],
["--out", str(self.base)],
]
link = self.base / "link"
try:
link.symlink_to(self.root, target_is_directory=True)
cases.append(["--out", str(link / "via-link.jsonl")])
except OSError: # pragma: no cover - Windows without symlink privilege
pass
for arguments in cases:
with self.subTest(arguments=arguments):
code, printed = self._run(*arguments)
self.assertEqual(code, corpus.EXIT_REFUSED)
self.assertIn("inspection_corpus:", printed)
self.assertEqual(_snapshot(self.root), before)
self.assertFalse(code_tree.exists())
self.assertFalse((self.base / "ok.jsonl").exists())
def test_h1_is_computed_against_a_baseline_file(self):
first = self.base / "first.jsonl"
self._run("--out", str(first), "--with-paths")
rows = self._rows(first)
self.assertTrue(all(row["path"].startswith("nested/") for row in rows))
resolved = [row for row in rows if row["state"] == "resolved"]
unresolved = [row for row in rows if row["state"] == "unresolved"]
changed = {resolved[0]["id"], unresolved[0]["id"]}
baseline = []
for row in rows:
if row["id"] == resolved[0]["id"]:
row = {
**row,
"prompts": {**row["prompts"], "positive": {"sha256": "0"}},
}
if row["id"] == unresolved[0]["id"]:
row = {**row, "state": "resolved"}
baseline.append(row)
baseline_path = self.base / "baseline.jsonl"
baseline_path.write_text("".join(json.dumps(r) + "\n" for r in baseline))
code, printed = self._run(
"--out", str(self.base / "second.jsonl"), "--baseline", str(baseline_path)
)
self.assertEqual(code, corpus.EXIT_INVARIANT_FAILED, printed)
h1 = corpus.invariants(
self._rows(self.base / "second.jsonl"), baseline, corpus._STATE.code.cap
)["H1"]
self.assertEqual(h1["status"], "FAIL")
self.assertEqual(h1["count"], len(changed))
self.assertEqual(
h1["breakdown"],
{
"resolved:texts-differ": 1,
f"unresolved:{unresolved[0]['reason_code']}": 1,
},
)
self.assertIn(" H1 FAIL 2", printed)
clean = corpus.invariants(rows, rows, corpus._STATE.code.cap)["H1"]
self.assertEqual((clean["status"], clean["count"]), ("PASS", 0))
missing = corpus.invariants([], resolved, corpus._STATE.code.cap)["H1"]
self.assertEqual(missing["breakdown"], {"missing": len(resolved)})
def test_parallel_workers_match_the_single_process_rows(self):
single, parallel = self.base / "single.jsonl", self.base / "parallel.jsonl"
self._run("--out", str(single))
self._run("--out", str(parallel), "--workers", "2")
def stable(path):
rows = self._rows(path)
return sorted(
({k: v for k, v in r.items() if k != "ms"} for r in rows),
key=lambda r: r["id"],
)
self.assertEqual(stable(parallel), stable(single))
ids = [row["id"] for row in self._rows(parallel)]
self.assertEqual(ids, sorted(ids))
def test_symlinks_and_other_files_are_not_read(self):
outside = self.base / "outside.png"
shutil.copy(FIXTURES / "f02.png", outside)
(self.root / "notes.txt").write_text("not media")
try:
(self.root / "linked.png").symlink_to(outside)
except OSError: # pragma: no cover - Windows without symlink privilege
self.skipTest("symlinks unavailable")
extensions = frozenset({".png"})
found = [rel for _, rel in corpus.media_files(self.root, extensions)]
self.assertEqual(len(found), len(_expected()))
self.assertNotIn("linked.png", found)
def test_missing_root_baseline_or_code_tree_is_refused(self):
out = str(self.base / "x.jsonl")
for arguments in (
["--root", str(self.base / "absent"), "--out", out],
["--root", str(self.root), "--out", out, "--baseline", out + ".none"],
["--root", str(self.root), "--out", out, "--code", str(self.base)],
):
with self.subTest(arguments=arguments):
with contextlib.redirect_stderr(io.StringIO()):
self.assertEqual(corpus.main(arguments), corpus.EXIT_REFUSED)
def test_without_node_facts_fail_h3_and_the_run(self):
out = self.base / "out.jsonl"
code, printed = self._run("--out", str(out), "--node", "")
self.assertEqual(code, corpus.EXIT_INVARIANT_FAILED)
rows = self._rows(out)
self.assertTrue(all(row["facts_error"] == "node-unavailable" for row in rows))
self.assertIn(f"facts errors: node-unavailable {len(rows)}", printed)
self.assertIn(f" H3 FAIL {len(rows)}", printed)
self.assertIn(f" harness FAIL {len(rows)}", printed)
def test_a_node_that_exits_fails_the_run(self):
false = shutil.which("false")
if false is None: # pragma: no cover - Windows has no false binary
self.skipTest("no false binary")
code, printed = self._run("--out", str(self.base / "o.jsonl"), "--node", false)
self.assertEqual(code, corpus.EXIT_INVARIANT_FAILED)
self.assertIn("node-exited", printed)
self.assertIsNone(corpus._STATE.node)
def test_a_missing_node_binary_is_refused(self):
missing = str(self.base / "no-such-node")
code, printed = self._run(
"--out", str(self.base / "o.jsonl"), "--node", missing
)
self.assertEqual(code, corpus.EXIT_REFUSED)
self.assertIn("--node", printed)
def test_harness_exceptions_fail_the_run(self):
out = self.base / "out.jsonl"
with unittest.mock.patch.object(
corpus, "_observe", side_effect=RuntimeError("boom")
):
code, printed = self._run("--out", str(out))
self.assertEqual(code, corpus.EXIT_INVARIANT_FAILED)
rows = self._rows(out)
self.assertEqual({row["error"] for row in rows}, {"RuntimeError"})
self.assertIn(f" harness FAIL {len(rows)}", printed)
self.assertNotIn("boom", out.read_text("utf-8") + printed)
class CorpusHelpersTest(unittest.TestCase):
def test_malformed_cause_names_only_quirks_outside_inputs(self):
node = '{"1": {"class_type": "K", "inputs": {"seed": %s}%s}}'
cases = {
node % ("1", ', "is_changed": [NaN]'): "non-input",
node % ("NaN", ""): "input-nonfinite",
'{"1": {"inputs": {}}}': "shape",
"[1]": "shape",
'{"1": {}, "1": {}}': "unparseable",
"{": "unparseable",
}
for text, cause in cases.items():
with self.subTest(text=text):
self.assertEqual(corpus.malformed_cause(text), cause)
def test_png_duplicate_scan_compares_record_bytes(self):
directory = tempfile.TemporaryDirectory()
self.addCleanup(directory.cleanup)
self.addCleanup(assert_closed, self, directory.name)
path = Path(directory.name, "f.png")
cases = (
((b"prompt\0{}", b"prompt\0{}"), True),
((b"prompt\0{}", b"prompt\0[]"), False),
((b"prompt\0{}", b"other\0x", b"other\0y"), None),
)
for texts, expected in cases:
with self.subTest(texts=texts):
path.write_bytes(_png(*texts))
self.assertIs(corpus.png_duplicates_identical(path), expected)
path.write_bytes(b"not a png")
self.assertIsNone(corpus.png_duplicates_identical(path))
def test_percentile_is_nearest_rank(self):
self.assertIsNone(corpus.percentile([], 0.5))
self.assertEqual(corpus.percentile([3, 1, 2], 0.5), 2)
self.assertEqual(corpus.percentile(list(range(1, 101)), 0.99), 99)
def test_h8_fails_on_slow_files_or_responses_over_the_cap(self):
fast = [{"id": "a", "ext": ".png", "ms": 1.0}]
self.assertEqual(corpus.invariants(fast, None, 8)["H8"]["status"], "PASS")
for rows in (
[{"id": "a", "ext": ".png", "ms": corpus.MAX_MS}],
[{"id": "a", "ext": ".png", "ms": corpus.H8_FLOOR_MS + 1}],
[{"id": "a", "ext": ".png", "ms": 1.0, "over_cap": True}],
):
with self.subTest(rows=rows):
result = corpus.invariants(rows, fast, 8)["H8"]
self.assertEqual(result["status"], "FAIL")
# A tiny baseline p99 cannot make H8 flap below the floor.
noisy = [{"id": "a", "ext": ".png", "ms": 3.0}]
self.assertEqual(corpus.invariants(noisy, fast, 8)["H8"]["status"], "PASS")
def test_h5_excludes_negative_only_absence(self):
def row(positive, state):
return {
"ext": ".png",
"records": ["prompt"],
"state": state,
"prompts": {"positive": {"status": positive}},
}
rows = [row("resolved", "absent"), row("absent", "absent"), row("absent", "x")]
self.assertEqual(corpus.invariants(rows, None, 8)["H5"]["count"], 2)
no_record = {**row("absent", "absent"), "records": ["workflow"]}
self.assertEqual(corpus.invariants([no_record], None, 8)["H5"]["count"], 0)
if __name__ == "__main__":
unittest.main()
+112
View File
@@ -0,0 +1,112 @@
"""Shared parity vectors: the workspace's JS facts match ``vectors.json``.
Each ``inspection`` vector's file goes through the real current-file route
pipeline (``inspect_current_file`` then ``encode_inspection_response``). The
exact response text is handed to Node, where the workspace's own
``projectInspectionPayload`` validates it and ``extractGenerationFacts`` reads
it. Python and JS therefore assert the same files with the same expectations.
"""
import json
import os
import shutil
import subprocess # nosec B404
import tempfile
import unittest
from pathlib import Path
from py.api.inspection_service import inspect_current_file
from utils.inspection_response import encode_inspection_response
try:
from tests import inspection_parity as parity
from tests.open_handles import assert_closed
except ImportError: # pragma: no cover - discover -s tests puts tests/ first
import inspection_parity as parity
from open_handles import assert_closed
HARNESS = (
Path(__file__).resolve().parent / "js" / "support" / "inspection-parity-facts.mjs"
)
LIBRARY = {
"status": "none",
"scope": "saved-library",
"positive_id": None,
"pair_id": None,
"title": None,
"category": None,
"tags": [],
"metadata_scope": None,
}
NODE_TIMEOUT_SECONDS = 120
def _encoded(name: str, path: Path) -> str:
observed = inspect_current_file(
path, source_label="parity", saved=None, library=LIBRARY
)
body = encode_inspection_response(
image={"root_index": 0, "path": f"{name}.png", "filename": f"{name}.png"},
root_scope="a" * 64,
file=observed["file"],
prompts=observed["prompts"],
resolution=observed["resolution"],
inspection=observed["inspection"],
library=observed["library"],
)
return body.decode("utf-8")
def _responses(vectors, directory: Path) -> list:
responses = []
for vector in vectors:
name = vector["fixture"]
if name in parity.TEST_TIME:
path = parity.fixture_path(name, directory)
parity.write_png(path, parity.build(name))
else:
path = parity.fixture_path(name)
responses.append({"name": vector["name"], "body": _encoded(name, path)})
return responses
class TestParityFacts(unittest.TestCase):
def setUp(self):
temporary = tempfile.TemporaryDirectory()
self.addCleanup(temporary.cleanup)
self.addCleanup(assert_closed, self, temporary.name)
self.directory = Path(temporary.name)
def test_vectors_json_facts_match_workspace_js(self):
node = shutil.which("node")
if node is None: # pragma: no cover - CI runners and dev hosts ship Node
if os.environ.get("CI"):
self.fail("Node.js is required in CI for the facts vectors")
self.skipTest("Node.js is not installed")
vectors = [v for v in parity.load_vectors() if v["profile"] == "inspection"]
self.assertTrue(vectors)
payload = json.dumps(_responses(vectors, self.directory))
# Fixed local Node harness, no shell.
completed = subprocess.run( # nosec B603
[node, str(HARNESS)],
input=payload.encode("utf-8"),
capture_output=True,
timeout=NODE_TIMEOUT_SECONDS,
check=False,
)
self.assertEqual(completed.returncode, 0, completed.stderr.decode("utf-8"))
observed = json.loads(completed.stdout)
for vector in vectors:
with self.subTest(vector=vector["name"]):
self.assertEqual(observed[vector["name"]], vector["expect"]["facts"])
def test_default_vectors_carry_no_facts(self):
# Facts are a workspace (inspection) concern; default vectors pin the
# ingestion resolver only.
for vector in parity.load_vectors():
if vector["profile"] == "default":
self.assertIsNone(vector["expect"]["facts"], vector["name"])
if __name__ == "__main__":
unittest.main()
+148
View File
@@ -0,0 +1,148 @@
"""Committed synthetic inspection parity fixtures stay reproducible and referenced."""
import json
import tempfile
import unittest
from pathlib import Path
try:
from tests import inspection_parity as parity
except ImportError: # pragma: no cover - discover -s tests puts tests/ first
import inspection_parity as parity
class InspectionParityFixtureTests(unittest.TestCase):
def test_committed_fixtures_match_builders(self):
# Decoded text chunks, not bytes: zlib output differs between platforms.
for name in parity.COMMITTED:
with self.subTest(fixture=name):
path = parity.fixture_path(name)
self.assertTrue(path.is_file(), f"missing committed fixture {path}")
self.assertLess(path.stat().st_size, parity.MAX_FIXTURE_BYTES)
self.assertEqual(
parity.read_text_chunks(path),
parity.expected_text_chunks(parity.build(name)),
)
def test_no_uncommitted_files_in_fixture_directory(self):
# Only fixture data counts; OS litter (.DS_Store, Thumbs.db) is ignored.
expected = {f"{name}.png" for name in parity.COMMITTED} | {"vectors.json"}
present = {
path.name
for path in parity.FIXTURE_DIR.iterdir()
if path.suffix in (".png", ".json")
}
self.assertEqual(present, expected)
def test_writer_round_trips_extra_raw_chunks(self):
for name in ("f15", "f16"):
built = parity.build(name)
with tempfile.TemporaryDirectory() as directory:
path = Path(directory) / f"{name}.png"
parity.write_png(path, built)
chunks = parity.read_text_chunks(path)
with self.subTest(fixture=name):
self.assertEqual(chunks, parity.expected_text_chunks(built))
self.assertGreater(len(chunks), 2)
def test_duplicate_fixtures_have_the_intended_shape(self):
identical = parity.read_text_chunks(parity.fixture_path("f15"))
self.assertEqual(len(identical), 8)
self.assertEqual(len(set(identical)), 2)
conflicting = [
text
for key, text in parity.read_text_chunks(parity.fixture_path("f16"))
if key == "prompt"
]
self.assertEqual(len(conflicting), 2)
self.assertNotEqual(conflicting[0], conflicting[1])
def test_nan_fixture_keeps_comfyui_spelling_outside_inputs(self):
prompt = dict(parity.read_text_chunks(parity.fixture_path("f14")))["prompt"]
self.assertIn('"is_changed": [NaN]', prompt)
self.assertIn('"_meta": {"title": "CheckpointLoaderSimple"}', prompt)
for node in json.loads(prompt).values():
for value in node["inputs"].values():
self.assertEqual(value, value, "NaN stays outside inputs")
def test_builders_are_deterministic_and_fresh(self):
for name in parity.COMMITTED + parity.TEST_TIME:
with self.subTest(fixture=name):
first, second = parity.build(name), parity.build(name)
self.assertEqual(
parity.expected_text_chunks(first),
parity.expected_text_chunks(second),
)
if first["prompt"] is not None:
first["prompt"].clear()
self.assertTrue(parity.build(name)["prompt"])
with self.assertRaises(KeyError):
parity.build("f99")
def test_test_time_shapes_exceed_commit_size(self):
sizes = {
name: sum(
len(text) for _, text in parity.expected_text_chunks(parity.build(name))
)
for name in parity.TEST_TIME
}
self.assertGreater(sizes["f01"], 3 * parity.MAX_FIXTURE_BYTES)
prompt = parity.build("f01")["prompt"]
self.assertEqual(len(prompt), 688)
self.assertEqual(sum(":" in node_id for node_id in prompt), 600)
self.assertGreater(sizes["f04"], 2_500_000)
self.assertGreater(sizes["f05"], 900_000)
def test_vectors_reference_existing_fixtures(self):
vectors = parity.load_vectors()
self.assertTrue(vectors)
names = [vector["name"] for vector in vectors]
self.assertEqual(len(names), len(set(names)), "vector names are unique")
known = set(parity.COMMITTED) | set(parity.TEST_TIME)
for vector in vectors:
with self.subTest(vector=vector["name"]):
self.assertIn(vector["fixture"], known)
if vector["fixture"] in parity.COMMITTED:
self.assertTrue(parity.fixture_path(vector["fixture"]).is_file())
self.assertIn(vector["profile"], ("default", "inspection"))
self.assertEqual(set(vector["expect"]), set(parity.EXPECT_KEYS))
facts = vector["expect"]["facts"]
if vector["profile"] == "default":
# Facts are a property of the inspection response only.
self.assertIsNone(facts)
else:
self.assertEqual(set(facts), set(parity.FACT_FIELDS))
for fact in facts.values():
self.assertEqual(set(fact), {"status", "value"})
for profile in ("inspection", "default"):
covered = {
vector["fixture"] for vector in vectors if vector["profile"] == profile
}
self.assertEqual(
covered, set(parity.COMMITTED) | set(parity.TEST_TIME), profile
)
def test_vectors_match_current_resolution(self):
"""Baseline vectors pin today's outcomes; behavior changes edit them."""
for vector in parity.load_vectors():
with self.subTest(vector=vector["name"]):
expected = dict(vector["expect"])
expected.pop("facts")
self.assertEqual(parity.observe(vector), expected)
def test_observe_builds_test_time_fixtures_instead_of_reading_missing_files(self):
# A missing file would otherwise read as "unreadable" and hide a stale vector.
for name in parity.TEST_TIME:
with self.subTest(fixture=name):
self.assertFalse(parity.fixture_path(name).exists())
observed = parity.observe(
{"fixture": name, "profile": "default", "anchor": None}
)
self.assertEqual(observed["metadata_status"], "available")
self.assertNotEqual(observed["state"], "absent")
with self.assertRaises(FileNotFoundError):
parity.observe({"fixture": "f02", "profile": "inspection"}, "missing-dir")
if __name__ == "__main__":
unittest.main()
+767 -9
View File
@@ -2,11 +2,15 @@
import io
import json
import struct
import tempfile
import unittest
import zlib
from pathlib import Path
from unittest.mock import MagicMock, patch
from PIL import Image
from utils.inspection_response import encode_inspection_response
from utils.prompt_inspector import (
InspectionRead,
@@ -18,6 +22,13 @@ from utils.prompt_inspector import (
)
from utils.pair_execution import EXECUTION_KEY
try:
from tests import inspection_parity as parity
from tests.open_handles import assert_closed
except ImportError: # pragma: no cover - discover -s tests puts tests/ first
import inspection_parity as parity
from open_handles import assert_closed
CAP = 8 * 1024 * 1024
IMAGE = {"root_index": 0, "path": "image.png", "filename": "image.png"}
PROMPTS = {
@@ -134,15 +145,23 @@ class TestInspectionResponse(unittest.TestCase):
self.assertEqual(result["metadata"]["raw"], read.metadata)
self.assertEqual(result["prompts"], PROMPTS)
def test_graph_node_limit_counts_nodes_not_json_fields(self):
for count, status in ((500, "parsed"), (501, "limited")):
def test_graph_ceiling_is_limited(self):
# f27: the 20,000-node backstop, built at test time. Prompt nodes count
# top-level API nodes; workflow nodes include subgraph definitions.
for count, status in ((20_000, "parsed"), (20_001, "limited")):
prompt = graph_text(count)
half = count // 2
workflow = json.dumps(
{
"nodes": [
{"id": i, "type": "Primitive", "widgets_values": [1, True]}
for i in range(count)
]
for i in range(count - half)
],
"definitions": {
"subgraphs": [
{"id": "sg", "nodes": [{"id": i} for i in range(half)]}
]
},
}
)
for read in (
@@ -162,12 +181,237 @@ class TestInspectionResponse(unittest.TestCase):
):
result = json.loads(encode(read))
for key in ("prompt_graph", "workflow_graph"):
self.assertEqual(result["metadata"][key]["status"], status)
if status == "limited":
self.assertIsNone(result["metadata"][key]["value"])
with self.subTest(count=count, key=key):
self.assertEqual(result["metadata"][key]["status"], status)
if status == "limited":
self.assertIsNone(result["metadata"][key]["value"])
self.assertEqual(result["metadata"]["raw"], read.metadata)
self.assertEqual(result["prompts"], PROMPTS)
def test_prompt_graph_with_600_executable_nodes_is_parsed(self):
graph = parity.build("f03")["prompt"]
for index in range(100):
graph[f"{700 + index}"] = {
"class_type": "PrimitiveInt",
"inputs": {"value": index},
}
self.assertGreater(len(graph), 600)
text = json.dumps(graph)
result = json.loads(encode(inspection({"prompt": text})))
projected = result["metadata"]["prompt_graph"]
self.assertEqual(projected["status"], "parsed")
self.assertEqual(len(projected["value"]), len(graph))
self.assertEqual(projected["value"]["3"]["inputs"]["seed"], 42)
self.assertEqual(projected["value"]["3"]["inputs"]["cfg"], "6.5")
self.assertEqual(result["metadata"]["raw"]["prompt"], text)
def test_workflow_with_513_ui_nodes_is_parsed_and_downloadable(self):
path = parity.fixture_path("f02")
chunks = dict(parity.read_text_chunks(path))
self.assertGreater(len(json.loads(chunks["workflow"])["nodes"]), 513)
result = json.loads(encode(read_inspection_file(path)))
metadata = result["metadata"]
self.assertEqual(metadata["workflow_graph"]["status"], "parsed")
self.assertEqual(
metadata["workflow_graph"]["value"]["nodes"],
json.loads(chunks["workflow"])["nodes"],
)
self.assertEqual(metadata["prompt_graph"]["status"], "parsed")
# The download is the original record text, byte for byte.
self.assertEqual(metadata["raw"]["workflow"], chunks["workflow"])
self.assertEqual(metadata["record_formats"]["workflow"], "original-text")
def test_nan_is_changed_and_meta_keep_prompt_graph_parsed(self):
path = parity.fixture_path("f14")
chunks = dict(parity.read_text_chunks(path))
self.assertIn("NaN", chunks["prompt"])
source = parity.build("f14")["prompt"]
result = json.loads(encode(read_inspection_file(path)))
metadata = result["metadata"]
graph = metadata["prompt_graph"]
self.assertEqual(graph["status"], "parsed")
self.assertEqual(set(graph["value"]), set(source))
for node_id, node in graph["value"].items():
self.assertEqual(set(node), {"class_type", "inputs"})
self.assertEqual(node["class_type"], source[node_id]["class_type"])
inputs = graph["value"]["3"]["inputs"]
expected = dict(source["3"]["inputs"], cfg="6.5", denoise="1.0")
self.assertEqual(inputs, expected)
self.assertEqual(graph["value"]["4"]["inputs"], source["4"]["inputs"])
self.assertNotIn("_meta", json.dumps(graph))
self.assertNotIn("is_changed", json.dumps(graph))
# Raw records are never modified: the NaN stays in the download.
self.assertEqual(metadata["raw"]["prompt"], chunks["prompt"])
self.assertEqual(metadata["workflow_graph"]["status"], "parsed")
def test_nonfinite_inside_inputs_projects_null(self):
prompt = (
'{"3":{"class_type":"KSampler","inputs":{"cfg":NaN,"seed":1,'
'"denoise":-Infinity,"list":[Infinity,0.5]},"is_changed":[NaN],'
'"_meta":{"title":"K","weight":Infinity}},"9":{"value":NaN}}'
)
workflow = (
'{"nodes":[{"id":1,"widgets_values":[NaN,1.5,"x"],'
'"extra":{"v":Infinity}}],"extra":{"ds":{"scale":-Infinity}}}'
)
for read in (
inspection({"prompt": prompt, "workflow": workflow}),
inspection(
{"prompt": prompt, "workflow": workflow},
formats={
"prompt": "reserialized-json",
"workflow": "reserialized-json",
},
carrier='{"prompt":'
+ prompt
+ ',"workflow":'
+ json.dumps(workflow)
+ ',"other":{"x":NaN}}',
),
):
result = json.loads(encode(read))
graph = result["metadata"]["prompt_graph"]
self.assertEqual(graph["status"], "parsed")
self.assertEqual(
graph["value"],
{
"3": {
"class_type": "KSampler",
"inputs": {
"cfg": None,
"seed": 1,
"denoise": None,
"list": [None, "0.5"],
},
},
"9": {},
},
)
flow = result["metadata"]["workflow_graph"]
self.assertEqual(flow["status"], "parsed")
self.assertEqual(
flow["value"],
{
"nodes": [
{
"id": 1,
"widgets_values": [None, "1.5", "x"],
"extra": {"v": None},
}
],
"extra": {"ds": {"scale": None}},
},
)
self.assertEqual(result["metadata"]["raw"], read.metadata)
self.assertEqual(result["prompts"], PROMPTS)
def test_prompt_projection_keeps_odd_node_shapes_and_rejects_non_objects(self):
prompt = '{"1":[NaN,2],"2":"text","3":{"inputs":{"a":1}},"4":{"_meta":{}}}'
expected = {"1": [None, 2], "2": "text", "3": {"inputs": {"a": 1}}, "4": {}}
carrier_cases = (
('{"prompt":' + prompt + "}", "parsed"),
('{"prompt":[1],"workflow":{"nodes":[]}}', "malformed"),
('{"prompt":' + json.dumps(prompt) + "}", "parsed"),
("[]", "malformed"),
)
result = json.loads(encode(inspection({"prompt": prompt})))
self.assertEqual(
result["metadata"]["prompt_graph"], {"status": "parsed", "value": expected}
)
for carrier, status in carrier_cases:
with self.subTest(carrier=carrier):
result = json.loads(
encode(
inspection(
{"prompt": prompt},
formats={"prompt": "reserialized-json"},
carrier=carrier,
)
)
)
graph = result["metadata"]["prompt_graph"]
self.assertEqual(graph["status"], status)
self.assertEqual(
graph["value"], expected if status == "parsed" else None
)
def test_exact_numbers_survive_tolerance(self):
prompt = (
'{"3":{"class_type":"KSampler","inputs":{"seed":1152921504606846976,'
'"steps":1e3,"cfg":0.1,"denoise":1e0,"exact":6.5000000000000000001,'
'"negative_seed":-1152921504606846976,"safe":9007199254740991},'
'"is_changed":[NaN]}}'
)
workflow = (
'{"nodes":[{"id":1,"widgets_values":[1152921504606846976,1e3,0.1,NaN]}]}'
)
result = json.loads(
encode(inspection({"prompt": prompt, "workflow": workflow}))
)
for key in ("prompt_graph", "workflow_graph"):
self.assertEqual(result["metadata"][key]["status"], "parsed")
inputs = result["metadata"]["prompt_graph"]["value"]["3"]["inputs"]
self.assertEqual(
inputs,
{
"seed": str(2**60),
"steps": "1E+3",
"cfg": "0.1",
"denoise": "1",
"exact": "6.5000000000000000001",
"negative_seed": str(-(2**60)),
"safe": 9007199254740991,
},
)
self.assertEqual(inputs["seed"], "1152921504606846976")
widgets = result["metadata"]["workflow_graph"]["value"]["nodes"][0]
self.assertEqual(
widgets["widgets_values"], ["1152921504606846976", "1E+3", "0.1", None]
)
def test_duplicate_json_keys_still_malformed(self):
for prompt, workflow in (
('{"1":{},"1":{}}', '{"nodes":[],"nodes":[]}'),
(
'{"1":{"class_type":"A","inputs":{},"_meta":{"t":1,"t":2}}}',
'{"nodes":[{"id":1,"extra":{"a":NaN,"a":NaN}}]}',
),
(
'{"1":{"class_type":"A","inputs":{"x":NaN,"x":1}}}',
'{"nodes":[{"id":1,"id":2}]}',
),
):
with self.subTest(prompt=prompt):
result = json.loads(
encode(inspection({"prompt": prompt, "workflow": workflow}))
)
for key in ("prompt_graph", "workflow_graph"):
self.assertEqual(
result["metadata"][key], {"status": "malformed", "value": None}
)
self.assertEqual(
result["metadata"]["raw"], {"prompt": prompt, "workflow": workflow}
)
valid = '{"1":{"class_type":"A","inputs":{}}}'
for carrier in (
'{"prompt":' + valid + ',"prompt":' + valid + "}",
'{"prompt":' + valid + ',"other":{"a":1,"a":2}}',
):
with self.subTest(carrier=carrier):
result = json.loads(
encode(
inspection(
{"prompt": valid},
formats={"prompt": "reserialized-json"},
carrier=carrier,
)
)
)
self.assertEqual(
result["metadata"]["prompt_graph"],
{"status": "malformed", "value": None},
)
def test_strict_projection_depth_is_separate_from_semantic_link_depth(self):
for depth in (64, 65):
# API map root + node + inputs + nested array containers.
@@ -238,8 +482,6 @@ class TestInspectionResponse(unittest.TestCase):
def test_strict_malformed_records_preserve_original_raw_and_roles(self):
for text in (
'{"1":{},"1":{}}',
'{"1":{"value":NaN}}',
'{"1":{"value":Infinity}}',
"{broken",
"[]",
"42",
@@ -318,6 +560,42 @@ class TestInspectionResponse(unittest.TestCase):
result["metadata"]["omitted_reason"], "response-size-limit"
)
def test_response_budget_drops_workflow_graph_before_prompt_graph(self):
import utils.inspection_response as response
with tempfile.TemporaryDirectory() as directory:
path = parity.fixture_path("f05", directory)
parity.write_png(path, parity.build("f05"))
read = read_inspection_file(path)
def metadata_at(cap, read):
with patch.object(response, "MAX_RESPONSE_BYTES", cap):
body = encode(read)
self.assertLessEqual(len(body), cap)
return json.loads(body)["metadata"]
# Step 1: the workflow graph alone is dropped; prompt evidence stays.
step1 = metadata_at(4_500_000, read)
self.assertEqual(step1["workflow_graph"], {"status": "limited", "value": None})
self.assertEqual(step1["prompt_graph"]["status"], "parsed")
self.assertEqual(step1["status"], "available")
self.assertIn("workflow", step1["raw"])
# Step 2: a large prompt graph goes too, while raw records stay.
large = inspection({"prompt": graph_text(20000)})
step2 = metadata_at(3_000_000, large)
self.assertEqual(step2["prompt_graph"], {"status": "limited", "value": None})
self.assertEqual(step2["workflow_graph"]["status"], "absent")
self.assertEqual(step2["status"], "available")
self.assertIn("prompt", step2["raw"])
# Step 3: raw is omitted atomically.
step3 = metadata_at(3_000_000, read)
self.assertEqual(step3["prompt_graph"]["status"], "limited")
self.assertEqual(step3["workflow_graph"]["status"], "limited")
self.assertEqual(step3["raw"], {})
self.assertEqual(step3["omitted_reason"], "response-size-limit")
def test_response_byte_boundary_is_inclusive(self):
prompts = {
role: {
@@ -505,3 +783,483 @@ class TestInspectionResponse(unittest.TestCase):
self.assertEqual(body["library"]["status"], "limited")
self.assertNotIn("image", body)
self.assertNotIn("root_scope", body)
MIB = 1024 * 1024
def _text_chunk(kind, keyword, body):
return parity._raw_chunk(kind, keyword.encode("latin-1") + b"\0" + body)
def _png_with_chunks(path, chunks):
"""A tiny real PNG with raw metadata chunks spliced before IEND, then closed."""
buffer = io.BytesIO()
with Image.new("RGB", (4, 4), (40, 80, 120)) as image:
image.save(buffer, format="PNG")
data = buffer.getvalue()
iend = data.rindex(b"IEND") - 4
with open(path, "wb") as stream:
stream.write(data[:iend])
for chunk in chunks:
stream.write(chunk)
stream.write(data[iend:])
return path
class TestPerRecordBudgets(unittest.TestCase):
"""Role records keep the 2 MiB budget; a workflow has its own 4 MiB budget."""
def setUp(self):
temporary = tempfile.TemporaryDirectory()
self.addCleanup(temporary.cleanup)
self.addCleanup(assert_closed, self, temporary.name)
self.directory = Path(temporary.name)
def prompt_chunk(self):
prompt = json.dumps(parity.build("f04")["prompt"]).encode("latin-1")
return _text_chunk(b"tEXt", "prompt", prompt)
def assert_prompts_resolved(self, read):
from utils.prompt_inspector import inspect_prompt_evidence
from utils.prompt_pair_resolver import INSPECTION_PROFILE
evidence = inspect_prompt_evidence(read.metadata, profile=INSPECTION_PROFILE)
self.assertEqual(evidence["resolution"]["state"], "resolved")
self.assertEqual(evidence["prompts"]["positive"]["text"], parity.POS)
self.assertEqual(evidence["prompts"]["negative"]["text"], parity.NEG)
def test_budget_constants(self):
from utils import prompt_inspector
self.assertEqual(prompt_inspector.ROLE_METADATA_BYTES, 2 * MIB)
self.assertEqual(prompt_inspector.WORKFLOW_METADATA_BYTES, 4 * MIB)
self.assertEqual(prompt_inspector.MAX_KEYWORD_BYTES, 80)
self.assertEqual(MAX_METADATA_BYTES, prompt_inspector.ROLE_METADATA_BYTES)
default = inspection({"prompt": "{}"})
self.assertEqual(default.limited_records, frozenset())
def test_f04_large_workflow_within_budget_no_longer_hides_prompts(self):
# f04's 2.5 MB workflow is over the old shared 2 MiB budget but within
# the 4 MiB workflow budget: everything is readable now.
path = self.directory / "f04.png"
parity.write_png(path, parity.build("f04"))
read = read_inspection_file(path)
self.assert_prompts_resolved(read)
metadata = json.loads(encode(read))["metadata"]
self.assertEqual(metadata["status"], "available")
self.assertEqual(metadata["record_formats"]["workflow"], "original-text")
self.assertEqual(metadata["workflow_graph"]["status"], "parsed")
self.assertEqual(metadata["prompt_graph"]["status"], "parsed")
def test_oversize_workflow_chunk_keeps_prompt_evidence(self):
from utils.prompt_inspector import WORKFLOW_METADATA_BYTES
built = parity.build("f04") # f04's shape, its note grown past budget
built["workflow"]["nodes"][-1]["widgets_values"] = [
"A" * (WORKFLOW_METADATA_BYTES + 1)
]
path = self.directory / "f04-oversize.png"
parity.write_png(path, built)
read = read_inspection_file(path)
self.assertNotIn("workflow", read.metadata)
self.assertEqual(read.limited_records, frozenset({"workflow"}))
self.assert_prompts_resolved(read)
metadata = json.loads(encode(read))["metadata"]
self.assertEqual(metadata["status"], "available")
self.assertEqual(
metadata["workflow_graph"], {"status": "limited", "value": None}
)
self.assertEqual(metadata["prompt_graph"]["status"], "parsed")
self.assertEqual(metadata["record_formats"]["workflow"], "limited")
self.assertEqual(metadata["record_formats"]["prompt"], "original-text")
self.assertNotIn("workflow", metadata["raw"])
self.assertIn("prompt", metadata["raw"])
def test_oversize_workflow_is_skipped_by_seek_not_read(self):
from utils.prompt_inspector import WORKFLOW_METADATA_BYTES
workflow = b"A" * (WORKFLOW_METADATA_BYTES + 1)
path = _png_with_chunks(
self.directory / "seek.png",
[_text_chunk(b"tEXt", "workflow", workflow), self.prompt_chunk()],
)
reads = self.tracked_reads(path)
self.assertLess(sum(reads), 64 * 1024)
def tracked_reads(self, path):
original_open = type(path).open
reads = []
def tracked_open(item, *args, **kwargs):
stream = original_open(item, *args, **kwargs)
original_read = stream.read
def read(size=-1):
value = original_read(size)
reads.append(len(value))
return value
stream.read = read
return stream
with patch.object(type(path), "open", tracked_open):
read = read_inspection_file(path)
self.assert_prompts_resolved(read)
return reads
def test_oversize_compressed_workflow_stops_inflating(self):
from utils import prompt_inspector
workflow = json.dumps({"nodes": [], "note": "A" * (6 * MIB)}).encode()
compressed = zlib.compress(workflow, 9)
self.assertLess(len(compressed), prompt_inspector.WORKFLOW_METADATA_BYTES)
produced = []
real = zlib.decompressobj
class Tracked:
def __init__(self):
self.inner = real()
def decompress(self, data, max_length=0):
self.assert_bounded(max_length)
value = self.inner.decompress(data, max_length)
produced.append(len(value))
return value
@staticmethod
def assert_bounded(max_length):
if not 0 < max_length <= prompt_inspector.WORKFLOW_METADATA_BYTES + 1:
raise AssertionError(f"unbounded inflate: {max_length}")
def __getattr__(self, name):
return getattr(self.inner, name)
for kind, body in (
(b"zTXt", b"\0" + compressed),
(b"iTXt", b"\1\0\0\0" + compressed),
):
with self.subTest(kind=kind):
produced.clear()
path = _png_with_chunks(
self.directory / "compressed.png",
[self.prompt_chunk(), _text_chunk(kind, "workflow", body)],
)
with patch.object(prompt_inspector.zlib, "decompressobj", Tracked):
read = read_inspection_file(path)
self.assertTrue(produced)
self.assertLessEqual(
max(produced), prompt_inspector.WORKFLOW_METADATA_BYTES + 1
)
self.assertEqual(read.limited_records, frozenset({"workflow"}))
self.assertEqual(read.record_formats["workflow"], "limited")
self.assertNotIn("workflow", read.metadata)
self.assert_prompts_resolved(read)
def test_compressed_workflow_over_budget_is_skipped(self):
from utils.prompt_inspector import WORKFLOW_METADATA_BYTES
body = b"\0" + bytes(range(256)) * (WORKFLOW_METADATA_BYTES // 256 + 1)
path = _png_with_chunks(
self.directory / "compressed-length.png",
[_text_chunk(b"zTXt", "workflow", body), self.prompt_chunk()],
)
read = read_inspection_file(path)
self.assertEqual(read.limited_records, frozenset({"workflow"}))
self.assert_prompts_resolved(read)
def test_role_records_over_budget_still_unreadable(self):
from utils.prompt_inspector import ROLE_METADATA_BYTES
prompt = self.prompt_chunk()
cases = {
"prompt-text": [
_text_chunk(b"tEXt", "prompt", b"x" * (ROLE_METADATA_BYTES + 1))
],
"prompt-inflate": [
_text_chunk(
b"zTXt",
"prompt",
b"\0" + zlib.compress(b"x" * (ROLE_METADATA_BYTES + 1)),
)
],
"roles-together": [
prompt,
_text_chunk(b"tEXt", "parameters", b"x" * ROLE_METADATA_BYTES),
],
"execution-record": [
prompt,
_text_chunk(b"tEXt", EXECUTION_KEY, b"x" * (ROLE_METADATA_BYTES + 1)),
],
}
for name, chunks in cases.items():
with self.subTest(case=name):
path = _png_with_chunks(self.directory / f"{name}.png", chunks)
with self.assertRaisesRegex(
MetadataReadError, "exceeds inspection limits"
):
read_inspection_file(path)
def test_role_budget_is_not_shared_with_workflow(self):
from utils.prompt_inspector import ROLE_METADATA_BYTES
path = _png_with_chunks(
self.directory / "shared.png",
[
self.prompt_chunk(),
_text_chunk(b"tEXt", "parameters", b"p" * (ROLE_METADATA_BYTES - 4096)),
_text_chunk(b"tEXt", "workflow", b"w" * (3 * MIB)),
],
)
read = read_inspection_file(path)
self.assertEqual(read.limited_records, frozenset())
self.assertEqual(len(read.metadata["workflow"]), 3 * MIB)
self.assert_prompts_resolved(read)
def test_identical_duplicate_chunks_are_accepted(self):
path = parity.fixture_path("f15")
chunks = dict(parity.read_text_chunks(path))
read = read_inspection_file(path)
self.assertEqual(read.metadata["prompt"], chunks["prompt"])
self.assertEqual(read.metadata["workflow"], chunks["workflow"])
self.assert_prompts_resolved(read)
metadata = json.loads(encode(read))["metadata"]
self.assertEqual(metadata["prompt_graph"]["status"], "parsed")
self.assertEqual(metadata["workflow_graph"]["status"], "parsed")
def test_conflicting_duplicate_chunks_fail(self):
with self.assertRaisesRegex(MetadataReadError, "Duplicate prompt metadata"):
read_inspection_file(parity.fixture_path("f16"))
prompt = json.dumps(parity.build("f04")["prompt"]).encode("latin-1")
cases = {
# Same decoded text, different bytes on disk: not byte-identical.
"kind": [
self.prompt_chunk(),
_text_chunk(b"zTXt", "prompt", b"\0" + zlib.compress(prompt)),
],
"length": [
self.prompt_chunk(),
_text_chunk(b"tEXt", "prompt", prompt + b" "),
],
"same-length": [
self.prompt_chunk(),
_text_chunk(b"tEXt", "prompt", prompt.replace(b"red fox", b"red cat")),
],
"workflow": [
self.prompt_chunk(),
_text_chunk(b"tEXt", "workflow", b'{"nodes": []}'),
_text_chunk(b"tEXt", "workflow", b'{"nodes": [1]}'),
],
}
for name, chunks in cases.items():
with self.subTest(case=name):
path = _png_with_chunks(self.directory / f"{name}.png", chunks)
with self.assertRaisesRegex(
MetadataReadError, "Duplicate prompt metadata"
):
read_inspection_file(path)
def test_duplicate_of_limited_workflow_stays_limited(self):
from utils.prompt_inspector import WORKFLOW_METADATA_BYTES
oversize = _text_chunk(
b"tEXt", "workflow", b"A" * (WORKFLOW_METADATA_BYTES + 1)
)
path = _png_with_chunks(
self.directory / "limited-twice.png",
[oversize, self.prompt_chunk(), oversize],
)
read = read_inspection_file(path)
self.assertEqual(read.limited_records, frozenset({"workflow"}))
self.assertNotIn("workflow", read.metadata)
self.assert_prompts_resolved(read)
def test_identical_duplicates_do_not_consume_role_budget(self):
from utils.prompt_inspector import ROLE_METADATA_BYTES
parameters = _text_chunk(
b"tEXt", "parameters", b"p" * (ROLE_METADATA_BYTES * 3 // 4)
)
path = _png_with_chunks(
self.directory / "repeat.png",
[self.prompt_chunk(), parameters, parameters],
)
read = read_inspection_file(path)
self.assertEqual(len(read.metadata["parameters"]), ROLE_METADATA_BYTES * 3 // 4)
def test_workflow_duplicates_are_limited_without_touching_role_records(self):
from utils import prompt_inspector
self.assertEqual(prompt_inspector.DUPLICATE_WORKFLOW_BYTES, 8 * MIB)
self.assertEqual(prompt_inspector.DUPLICATE_ROLE_BYTES, 4 * MIB)
workflow = _text_chunk(b"tEXt", "workflow", b"{}" + b" " * 1000)
size = len(workflow) - 12
path = _png_with_chunks(
self.directory / "wf.png", [self.prompt_chunk()] + [workflow] * 6
)
added = []
original = prompt_inspector._PngRecords.add
def counting(records, kind, data):
added.append(len(data))
return original(records, kind, data)
with (
patch.object(
prompt_inspector, "DUPLICATE_WORKFLOW_BYTES", size * 2 + size // 2
),
patch.object(prompt_inspector._PngRecords, "add", counting),
):
read = read_inspection_file(path)
# Original plus two budgeted duplicates were read; the third is limited.
self.assertEqual(added.count(size), 3)
self.assertEqual(read.record_formats["workflow"], "limited")
self.assertNotIn("workflow", read.metadata)
self.assertEqual(read.record_formats["prompt"], "original-text")
self.assert_prompts_resolved(read)
def test_role_duplicates_past_their_budget_fail_closed(self):
from utils import prompt_inspector
chunk = self.prompt_chunk()
size = len(chunk) - 12
path = _png_with_chunks(self.directory / "dupes.png", [chunk] * 6)
with patch.object(
prompt_inspector, "DUPLICATE_ROLE_BYTES", size * 2 + size // 2
):
with self.assertRaisesRegex(
MetadataReadError, "Metadata exceeds inspection limits"
):
read_inspection_file(path)
def test_over_budget_role_record_never_falls_back_to_other_evidence(self):
from utils import prompt_inspector
chunk = self.prompt_chunk()
size = len(chunk) - 12
extras = [
_text_chunk(b"tEXt", "parameters", b"fallback parameters"),
_text_chunk(b"tEXt", "workflow", b"{}"),
]
path = _png_with_chunks(self.directory / "fallback.png", [chunk] * 6 + extras)
with patch.object(prompt_inspector, "DUPLICATE_ROLE_BYTES", size * 2):
with self.assertRaisesRegex(
MetadataReadError, "Metadata exceeds inspection limits"
):
read_inspection_file(path)
def test_role_key_can_never_end_up_limited(self):
from utils import prompt_inspector
path = _png_with_chunks(self.directory / "plain.png", [self.prompt_chunk()])
original = prompt_inspector._PngRecords.skip
def limit_role(records, key, length):
records.limited.add("prompt")
return original(records, key, length)
with patch.object(prompt_inspector._PngRecords, "skip", limit_role):
with self.assertRaisesRegex(
MetadataReadError, "Metadata exceeds inspection limits"
):
read_inspection_file(path)
def test_duplicates_within_the_total_budget_are_unchanged(self):
path = _png_with_chunks(self.directory / "few.png", [self.prompt_chunk()] * 3)
read = read_inspection_file(path)
self.assertEqual(read.record_formats["prompt"], "original-text")
self.assert_prompts_resolved(read)
def test_unretained_text_chunks_are_skipped_without_retention(self):
software = _text_chunk(b"tEXt", "Software", b"S" * (5 * MIB))
compressed = _text_chunk(b"zTXt", "Comment", b"\0" + b"not zlib at all")
path = _png_with_chunks(
self.directory / "software.png",
[software, compressed, self.prompt_chunk()],
)
reads = self.tracked_reads(path)
self.assertLess(sum(reads), 64 * 1024)
read = read_inspection_file(path)
self.assertEqual(set(read.metadata), {"prompt"})
def test_unretained_chunk_with_malformed_keyword_still_fails(self):
for body in (b"no-separator", b"k" * 100 + b"\0text", b"\0text"):
with self.subTest(body=body[:12]):
path = _png_with_chunks(
self.directory / "keyword.png",
[self.prompt_chunk(), parity._raw_chunk(b"tEXt", body)],
)
with self.assertRaisesRegex(MetadataReadError, "Malformed"):
read_inspection_file(path)
def test_chunk_count_limit_unchanged(self):
from utils import prompt_inspector
self.assertEqual(prompt_inspector.MAX_PNG_CHUNKS, 10_000)
filler = _text_chunk(b"tEXt", "Comment", b"x")
for total, readable in ((10_000, True), (10_001, False)):
with self.subTest(total=total):
path = _png_with_chunks(self.directory / "count.png", [])
structural = self.chunk_count(path)
path = _png_with_chunks(
self.directory / "count.png",
[self.prompt_chunk()] + [filler] * (total - structural - 1),
)
self.assertEqual(self.chunk_count(path), total)
if readable:
self.assert_prompts_resolved(read_inspection_file(path))
else:
with self.assertRaisesRegex(MetadataReadError, "chunk count"):
read_inspection_file(path)
@staticmethod
def chunk_count(path):
data, offset, count = path.read_bytes(), 8, 0
while offset < len(data):
(length,) = struct.unpack(">I", data[offset : offset + 4])
offset += 12 + length
count += 1
return count
def test_non_png_workflow_over_budget_is_limited_not_unreadable(self):
from utils.prompt_inspector import WORKFLOW_METADATA_BYTES
prompt = json.dumps(parity.build("f04")["prompt"])
metadata = {"prompt": prompt, "workflow": "w" * (WORKFLOW_METADATA_BYTES + 1)}
fake = MagicMock()
fake.__enter__.return_value = fake
fake.info, fake.format, fake.size = metadata, "WEBP", (4, 4)
path = self.directory / "image.webp"
path.write_bytes(b"RIFF0000WEBP")
with patch("utils.prompt_inspector.Image.open", return_value=fake):
read = read_inspection_file(path)
self.assertEqual(read.limited_records, frozenset({"workflow"}))
self.assertEqual(
read.record_formats, {"prompt": "original-text", "workflow": "limited"}
)
self.assert_prompts_resolved(read)
def test_limited_record_format_omits_raw_and_limits_graph(self):
read = InspectionRead(
metadata={"prompt": graph_text(), "workflow": "{}"},
record_formats={"prompt": "original-text", "workflow": "original-text"},
video_carrier_text=None,
format="PNG",
width=4,
height=4,
limited_records=frozenset({"workflow"}),
)
metadata = json.loads(encode(read))["metadata"]
self.assertEqual(metadata["record_formats"]["workflow"], "limited")
self.assertNotIn("workflow", metadata["raw"])
self.assertEqual(
metadata["workflow_graph"], {"status": "limited", "value": None}
)
self.assertEqual(metadata["prompt_graph"]["status"], "parsed")
only = InspectionRead({}, {"workflow": "limited"}, None, "PNG", 4, 4)
metadata = json.loads(encode(only))["metadata"]
self.assertEqual(metadata["status"], "available")
self.assertEqual(metadata["raw"], {})
self.assertEqual(metadata["record_formats"], {"workflow": "limited"})
self.assertEqual(metadata["workflow_graph"]["status"], "limited")
self.assertEqual(metadata["prompt_graph"]["status"], "absent")
+620
View File
@@ -1,9 +1,17 @@
"""Conservative prompt-pair resolution contracts, independent of ComfyUI."""
import random
import unittest
from dataclasses import replace
from utils import prompt_pair_resolver as resolver
from utils.prompt_pair_resolver import OutputAdapter, resolve_prompt_pair
try:
from tests import inspection_parity as parity
except ImportError: # discovered with ``-s tests``
import inspection_parity as parity
def graph(positive: str = "portrait", negative: str = "blurry") -> dict:
return {
@@ -423,5 +431,617 @@ class TestPromptPairResolver(unittest.TestCase):
self.assertEqual(result["state"], "unresolved")
def _f03_metadata() -> dict:
return {"prompt": parity.build("f03")["prompt"]}
def _vector(name: str) -> dict:
return next(v for v in parity.load_vectors() if v["name"] == name)["expect"]
def _diamond(prefix: str = "d", leaves: int = 575, middles: int = 24) -> dict:
"""A ~600-node string diamond: 24 fans over 575 leaves, one leaf shared."""
letters = "abcdefghijklmnopqrstuvwxyz"
nodes = {
f"{prefix}l{index}": {"class_type": "PrimitiveString", "inputs": {"value": "a"}}
for index in range(leaves)
}
for middle in range(middles):
nodes[f"{prefix}m{middle}"] = {
"class_type": "Text Concatenate",
"inputs": {
f"text_{letters[index]}": [
f"{prefix}l{(middle * middles + index) % leaves}",
0,
]
for index in range(middles)
},
}
nodes[f"{prefix}r"] = {
"class_type": "Text Concatenate",
"inputs": {
f"text_{letters[index]}": [f"{prefix}m{index}", 0]
for index in range(middles)
},
}
return nodes
def _diamond_samplers(count: int) -> dict:
"""``count`` samplers, each encoding the same diamond through its own encoder."""
source = {"neg": {"class_type": "CLIPTextEncode", "inputs": {"text": "bad"}}}
source.update(_diamond())
for index in range(count):
source[f"e{index}"] = {
"class_type": "CLIPTextEncode",
"inputs": {"text": ["dr", 0]},
}
source[f"s{index}"] = {
"class_type": "KSampler",
"inputs": {"positive": [f"e{index}", 0], "negative": ["neg", 0]},
}
return source
def _distinct_diamond_samplers(count: int, chain: int = 60) -> dict:
"""``count`` samplers, each with its own 590-node diamond, sharing a latent chain.
Real work exceeds the shared 20,000-visit total while the graph stays under
the node ceiling: every candidate re-walks the (unmemoized) latent chain.
"""
source = {
"neg": {"class_type": "CLIPTextEncode", "inputs": {"text": "bad"}},
"z0": {"class_type": "EmptyLatentImage", "inputs": {}},
}
for index in range(1, chain + 1):
source[f"z{index}"] = {
"class_type": "Reroute",
"inputs": {"value": [f"z{index - 1}", 0]},
}
for index in range(count):
prefix = f"d{index}_"
source.update(_diamond(prefix, leaves=565))
source[f"e{index}"] = {
"class_type": "CLIPTextEncode",
"inputs": {"text": [f"{prefix}r", 0]},
}
source[f"s{index}"] = {
"class_type": "KSampler",
"inputs": {
"positive": [f"e{index}", 0],
"negative": ["neg", 0],
"latent_image": [f"z{chain}", 0],
},
}
return source
def _depth_straddling_samplers() -> dict:
"""A at depth 9 below its sampler, B four Reroutes deeper on the same chain."""
source = {"p": {"class_type": "PrimitiveString", "inputs": {"value": "cat"}}}
previous = "p"
for index in range(6):
source[f"c{index}"] = {
"class_type": "Reroute",
"inputs": {"value": [previous, 0]},
}
previous = f"c{index}"
source["encA"] = {"class_type": "CLIPTextEncode", "inputs": {"text": [previous, 0]}}
source["A"] = {
"class_type": "KSampler",
"inputs": {"positive": ["encA", 0], "negative": ["encA", 0]},
}
for index in range(4):
source[f"b{index}"] = {
"class_type": "Reroute",
"inputs": {"value": [previous, 0]},
}
previous = f"b{index}"
source["encB"] = {"class_type": "CLIPTextEncode", "inputs": {"text": [previous, 0]}}
source["B"] = {
"class_type": "KSampler",
"inputs": {"positive": ["encB", 0], "negative": ["encB", 0]},
}
return source
def _nested_shared_chain_samplers() -> dict:
"""A, B and C share one chain; each later sampler hangs below a memo hit."""
source = {"p": {"class_type": "PrimitiveString", "inputs": {"value": "cat"}}}
def chain(start: str, prefix: str, count: int) -> str:
previous = start
for index in range(count):
source[f"{prefix}{index}"] = {
"class_type": "Reroute",
"inputs": {"value": [previous, 0]},
}
previous = f"{prefix}{index}"
return previous
def sampler(name: str, text_source: str) -> None:
source[f"enc{name}"] = {
"class_type": "CLIPTextEncode",
"inputs": {"text": [text_source, 0]},
}
source[name] = {
"class_type": "KSampler",
"inputs": {"positive": [f"enc{name}", 0], "negative": [f"enc{name}", 0]},
}
top = chain("p", "c", 6) # A memoizes c5 with height 6
sampler("A", top)
middle = chain(top, "r", 1) # r0 is built over a memo hit on c5
sampler("B", middle)
sampler("C", chain(middle, "d", 3)) # C reaches r0 three Reroutes deeper
return source
def _random_graph(rng: random.Random) -> dict:
ids = [str(index) for index in range(rng.randrange(5, 60))]
classes = (
"StringConcatenate",
"Text Concatenate",
"Reroute",
"PrimitiveString",
"CLIPTextEncode",
"KSampler",
"PromptManager",
"EmptyLatentImage",
)
def link():
return [rng.choice(ids), 0]
source = {}
for node_id in ids:
name = rng.choice(classes)
if name == "StringConcatenate":
inputs = {"string_a": link(), "string_b": rng.choice([link(), "x"])}
elif name == "Text Concatenate":
inputs = {"text_a": link(), "text_b": link(), "delimiter": ""}
elif name == "Reroute":
inputs = {"value": link()}
elif name == "PrimitiveString":
inputs = {"value": rng.choice(["a", " b "])}
elif name == "CLIPTextEncode":
inputs = {"text": rng.choice([link(), "p"])}
elif name == "KSampler":
inputs = {"positive": link(), "negative": link()}
if rng.random() < 0.3:
inputs["latent_image"] = link()
elif name == "PromptManager":
inputs = {"text": rng.choice(["q", link()]), "prepend_text": ""}
else:
inputs = {}
source[node_id] = {"class_type": name, "inputs": inputs}
return source
def _outcomes(result: dict) -> dict:
items = result["candidates"] or [result]
return {
item["sampler_id"]: (item["state"], item["reason_code"], item["reason"])
for item in items
}
class _CountingInputs(dict):
"""Node inputs that count how often a resolver iterates (scans) their keys."""
scans = 0
def __iter__(self):
type(self).scans += 1
return super().__iter__()
class _CountingText(str):
"""A literal that tallies the characters a resolver strips from it."""
stripped = 0
def strip(self, *args):
type(self).stripped += len(self)
return super().strip(*args)
class TestResolverProfiles(unittest.TestCase):
"""Inspection bounds work, not graph size; the default profile is pinned."""
def test_inspection_profile_resolves_reachable_branch_in_506_node_graph(self):
expected = _vector("f03-default")
self.assertGreater(len(_f03_metadata()["prompt"]), 500)
result = resolve_prompt_pair(
_f03_metadata(), profile=resolver.INSPECTION_PROFILE
)
self.assertEqual(result["state"], "resolved")
self.assertIsNone(result["reason_code"])
self.assertEqual(result["positive"]["text"], parity.POS)
self.assertEqual(result["negative"]["text"], parity.NEG)
self.assertEqual(expected["state"], "unresolved") # ingestion baseline
def test_default_profile_keeps_500_node_gate(self):
for result in (
resolve_prompt_pair(_f03_metadata()),
resolve_prompt_pair(_f03_metadata(), profile=resolver.DEFAULT_PROFILE),
):
self.assertEqual(result["state"], "unresolved")
self.assertEqual(result["reason"], "Resolution limit reached")
self.assertIsNone(result["reason_code"])
self.assertNotIn("node", result)
def test_total_visit_budget_is_shared_across_candidates(self):
source = _distinct_diamond_samplers(33)
self.assertEqual(resolver.INSPECTION_PROFILE.total_visits, 20_000)
self.assertLess(len(source), resolver.INSPECTION_PROFILE.node_ceiling)
result = resolve_prompt_pair(
{"prompt": source}, profile=resolver.INSPECTION_PROFILE
)
# While selection is off (until PR3) many samplers are always ambiguous;
# PR3 must add "work-limit never upgrades to resolved" once one is chosen.
self.assertNotEqual(result["state"], "resolved")
self.assertEqual(result["state"], "ambiguous")
candidates = result["candidates"]
self.assertEqual(len(candidates), 33)
self.assertEqual(candidates[0]["state"], "resolved")
codes = [candidate["reason_code"] for candidate in candidates]
first_limited = codes.index("work-limit")
self.assertGreater(first_limited, 0)
self.assertTrue(all(code == "work-limit" for code in codes[first_limited:]))
self.assertTrue(
all(c["state"] == "unresolved" for c in candidates[first_limited:])
)
# The default profile keeps its historical gate on the same graph.
self.assertEqual(
resolve_prompt_pair({"prompt": source})["reason"],
"Resolution limit reached",
)
def test_memo_hit_respects_the_depth_left_at_the_reusing_candidate(self):
source = _depth_straddling_samplers()
reverse = dict(reversed(list(source.items())))
profile = resolver.INSPECTION_PROFILE
forward = _outcomes(
resolve_prompt_pair({"prompt": source}, profile=profile, max_depth=10)
)
backward = _outcomes(
resolve_prompt_pair({"prompt": reverse}, profile=profile, max_depth=10)
)
self.assertEqual(forward, backward)
self.assertEqual(forward["A"][:2], ("resolved", None))
self.assertEqual(forward["B"][:2], ("unresolved", "depth-limit"))
anchored = resolve_prompt_pair(
{"prompt": source}, profile=profile, max_depth=10, output_node_id="B"
)
self.assertEqual(
(anchored["state"], anchored["reason_code"]), ("unresolved", "depth-limit")
)
def test_memo_height_propagates_through_a_nested_memo_hit(self):
source = _nested_shared_chain_samplers()
reverse = dict(reversed(list(source.items())))
profile = resolver.INSPECTION_PROFILE
for graph in (source, reverse):
outcomes = _outcomes(
resolve_prompt_pair({"prompt": graph}, profile=profile, max_depth=12)
)
self.assertEqual(outcomes["C"][:2], ("unresolved", "depth-limit"))
anchored = resolve_prompt_pair(
{"prompt": source}, profile=profile, max_depth=12, output_node_id="C"
)
self.assertEqual(
(anchored["state"], anchored["reason_code"]), ("unresolved", "depth-limit")
)
def test_candidate_outcomes_match_anchored_and_ignore_node_order(self):
profile = resolver.INSPECTION_PROFILE
limits = ({}, {"max_depth": 4}, {"max_depth": 8}, {"max_visits": 6})
compared = 0
for seed in range(240, 246): # 6,000 deterministic graphs, about 1.5 s
rng = random.Random(seed)
for _ in range(1_000):
source = _random_graph(rng)
kwargs = rng.choice(limits)
base = _outcomes(
resolve_prompt_pair({"prompt": source}, profile=profile, **kwargs)
)
items = list(source.items())
rng.shuffle(items)
shuffled = _outcomes(
resolve_prompt_pair(
{"prompt": dict(items)}, profile=profile, **kwargs
)
)
self.assertEqual(base, shuffled, (seed, source, kwargs))
if len(base) < 2:
continue
for sampler_id, outcome in base.items():
anchored = resolve_prompt_pair(
{"prompt": source},
profile=profile,
output_node_id=sampler_id,
**kwargs,
)
self.assertEqual(
_outcomes(anchored)[sampler_id],
outcome,
(seed, sampler_id, source, kwargs),
)
compared += 1
self.assertGreater(compared, 1_000)
def test_provenance_entry_budget_reports_the_path_limit(self):
letters = "abcdefghijklmnopqrstuvwxyz"
source = {
"p": {"class_type": "PrimitiveString", "inputs": {"value": "cat"}},
"neg": {"class_type": "CLIPTextEncode", "inputs": {"text": "bad"}},
}
previous = "p"
for index in range(60):
source[f"c{index}"] = {
"class_type": "Reroute",
"inputs": {"value": [previous, 0]},
}
previous = f"c{index}"
for index in range(500):
source[f"w{index}"] = {
"class_type": "Text Concatenate",
"inputs": {f"text_{c}": [previous, 0] for c in letters},
}
source[f"e{index}"] = {
"class_type": "CLIPTextEncode",
"inputs": {"text": [f"w{index}", 0]},
}
source[f"s{index}"] = {
"class_type": "KSampler",
"inputs": {"positive": [f"e{index}", 0], "negative": ["neg", 0]},
}
result = resolve_prompt_pair(
{"prompt": source}, profile=resolver.INSPECTION_PROFILE
)
reasons = [candidate["reason"] for candidate in result["candidates"]]
self.assertEqual(result["candidates"][0]["state"], "resolved")
first = reasons.index("Provenance path limit reached")
self.assertGreater(first, 0)
self.assertLessEqual(
set(reasons[first:]),
{"Provenance path limit reached", "Resolution visit limit reached"},
)
self.assertEqual(reasons[first], "Provenance path limit reached")
self.assertEqual(
{c["reason_code"] for c in result["candidates"][first:]}, {"work-limit"}
)
# A small entry budget pins the same reason on one candidate.
small = replace(resolver.INSPECTION_PROFILE, max_path_entries=10)
result = resolve_prompt_pair({"prompt": source}, profile=small)
self.assertEqual(
(result["candidates"][0]["reason"], result["candidates"][0]["reason_code"]),
("Provenance path limit reached", "work-limit"),
)
def test_shared_node_is_scanned_once_however_many_candidates_use_it(self):
_CountingInputs.scans = 0
fat = _CountingInputs({"text": "portrait"})
fat.update({f"extra_{index}": 0 for index in range(20_000)})
source = {
"fat": {"class_type": "CLIPTextEncode", "inputs": fat},
"neg": {"class_type": "CLIPTextEncode", "inputs": {"text": "bad"}},
}
for index in range(1_000):
source[f"s{index}"] = {
"class_type": "KSampler",
"inputs": {"positive": ["fat", 0], "negative": ["neg", 0]},
}
result = resolve_prompt_pair(
{"prompt": source}, profile=resolver.INSPECTION_PROFILE
)
states = {candidate["state"] for candidate in result["candidates"]}
self.assertEqual(states, {"resolved"})
# Work is O(nodes), not O(candidates x node size).
self.assertLessEqual(_CountingInputs.scans, 2)
def test_stripped_concatenation_text_is_charged_to_a_work_budget(self):
_CountingText.stripped = 0
letters = "abcdefghijklmnopqrstuvwxyz"
hub = _CountingText(" " * 100_000)
source = graph()
source["hub"] = {"class_type": "PrimitiveString", "inputs": {"value": hub}}
for leaf in range(300):
source[f"l{leaf}"] = {
"class_type": "Text Concatenate",
"inputs": {f"text_{c}": ["hub", 0] for c in letters},
}
for middle in range(12):
source[f"m{middle}"] = {
"class_type": "Text Concatenate",
"inputs": {
f"text_{letters[i]}": [f"l{middle * 25 + i}", 0] for i in range(25)
},
}
source["root"] = {
"class_type": "Text Concatenate",
"inputs": {f"text_{letters[i]}": [f"m{i}", 0] for i in range(12)},
}
source["1"]["inputs"]["text"] = ["root", 0]
result = resolve_prompt_pair(source, profile=resolver.INSPECTION_PROFILE)
self.assertEqual(
(result["state"], result["reason_code"]), ("unresolved", "work-limit")
)
self.assertLessEqual(
_CountingText.stripped, resolver.INSPECTION_PROFILE.max_scanned_units
)
def test_per_candidate_visit_budget_resets_for_each_sampler(self):
source = _diamond_samplers(3)
result = resolve_prompt_pair(
{"prompt": source}, profile=resolver.INSPECTION_PROFILE, max_visits=700
)
self.assertEqual(
[candidate["state"] for candidate in result["candidates"]],
["resolved"] * 3,
)
result = resolve_prompt_pair(
{"prompt": source}, profile=resolver.INSPECTION_PROFILE, max_visits=300
)
self.assertEqual(
[candidate["reason_code"] for candidate in result["candidates"]],
["work-limit"] * 3,
)
def test_node_ceiling_reports_graph_ceiling(self):
source = graph()
for index in range(20_001 - len(source)):
source[f"p{index}"] = {"class_type": "PrimitiveInt", "inputs": {"value": 1}}
self.assertEqual(len(source), 20_001)
result = resolve_prompt_pair(
{"prompt": source}, profile=resolver.INSPECTION_PROFILE
)
self.assertEqual(result["reason_code"], "graph-ceiling")
self.assertNotEqual(result["state"], "resolved")
del source["p0"]
result = resolve_prompt_pair(
{"prompt": source}, profile=resolver.INSPECTION_PROFILE
)
self.assertEqual(result["state"], "resolved")
def test_prompt_graph_without_supported_sampler_is_unresolved_not_absent(self):
metadata = {"prompt": parity.build("f17")["prompt"]}
result = resolve_prompt_pair(metadata, profile=resolver.INSPECTION_PROFILE)
self.assertEqual(
(result["state"], result["reason_code"]), ("unresolved", "no-sampler")
)
self.assertIn("SamplerCustomAdvanced", result["reason"])
self.assertIsNone(result["positive"])
default = resolve_prompt_pair(metadata)
self.assertEqual(default["state"], "absent")
self.assertEqual(default["reason"], "No supported sampler pair found")
self.assertIsNone(default["reason_code"])
def test_no_sampler_reason_names_at_most_three_sampler_like_classes(self):
source = {
str(index): {
"class_type": f"FutureSampler{index % 5}",
"inputs": {"latent_image": ["x", 0]},
}
for index in range(10)
}
result = resolve_prompt_pair(
{"prompt": source}, profile=resolver.INSPECTION_PROFILE
)
self.assertEqual(result["reason_code"], "no-sampler")
self.assertEqual(
result["reason"],
"No supported sampler pair found "
"(saw FutureSampler0, FutureSampler1, FutureSampler2)",
)
def test_absent_only_without_prompt_or_parameters(self):
for value in (None, [], "raw text", {}, {"parameters": None}):
with self.subTest(value=value):
result = resolve_prompt_pair(value, profile=resolver.INSPECTION_PROFILE)
self.assertEqual(result["state"], "absent")
self.assertIsNone(result["positive"])
note = {"1": {"class_type": "Note", "inputs": {"text": "hello"}}}
for value in (
{"prompt": note},
note,
{"prompt": {}},
{"prompt": []},
{"prompt": {"1": None}},
{"parameters": "unstructured arbitrary text"},
):
with self.subTest(value=value):
result = resolve_prompt_pair(value, profile=resolver.INSPECTION_PROFILE)
self.assertEqual(result["state"], "unresolved")
self.assertIsNotNone(result["reason_code"])
self.assertEqual(resolve_prompt_pair({"prompt": note})["state"], "absent")
# A pair-level "absent" names the missing negative; it always has a positive.
source = graph()
del source["3"]["inputs"]["negative"]
result = resolve_prompt_pair(source, profile=resolver.INSPECTION_PROFILE)
self.assertEqual(result["state"], "absent")
self.assertEqual(result["positive"]["text"], "portrait")
def test_existing_limits_map_to_reason_codes(self):
cases = (
({"max_visits": 1}, "work-limit", "Resolution visit limit reached"),
({"max_depth": 0}, "depth-limit", None),
({"max_depth": 1}, "depth-limit", "Resolution depth limit reached"),
({"max_text_length": 3}, "text-limit", None),
)
for limits, code, reason in cases:
with self.subTest(limits=limits):
default = resolve_prompt_pair(graph(), **limits)
self.assertEqual(default["state"], "unresolved")
self.assertIsNone(default["reason_code"])
result = resolve_prompt_pair(
graph(), profile=resolver.INSPECTION_PROFILE, **limits
)
self.assertEqual(result["state"], "unresolved")
self.assertEqual(result["reason_code"], code)
if reason:
self.assertEqual(result["reason"], reason)
result = resolve_prompt_pair(
{"parameters": "x" * 20},
profile=resolver.INSPECTION_PROFILE,
max_text_length=10,
)
self.assertEqual(result["reason_code"], "text-limit")
def test_unsupported_node_names_the_blocking_node(self):
source = graph()
source["1"]["class_type"] = "UnknownEncoder"
result = resolve_prompt_pair(source, profile=resolver.INSPECTION_PROFILE)
self.assertEqual(result["reason_code"], "unsupported-node")
self.assertEqual(result["node"], {"id": "1", "class_type": "UnknownEncoder"})
default = resolve_prompt_pair(source)
self.assertIsNone(default["reason_code"])
self.assertNotIn("node", default)
self.assertEqual(default["reason"], result["reason"])
def test_default_profile_emits_only_a_null_reason_code_beyond_todays_keys(self):
source = graph()
source["4"] = {"class_type": "KSampler", "inputs": source["3"]["inputs"]}
for value in (graph(), source, {"parameters": "a\nSteps: 1, Sampler: x"}):
result = resolve_prompt_pair(value)
self.assertEqual(
set(result),
{
"state",
"positive",
"negative",
"sampler_id",
"reason",
"candidates",
"reason_code",
},
)
self.assertIsNone(result["reason_code"])
for candidate in result["candidates"]:
self.assertNotIn("node", candidate)
self.assertIsNone(candidate["reason_code"])
def test_diamond_provenance_paths_are_bounded_in_inspection(self):
source = graph()
source["8"] = {"class_type": "PrimitiveString", "inputs": {"value": ""}}
for index in range(9, 49):
source[str(index)] = {
"class_type": "StringConcatenate",
"inputs": {
"string_a": [str(index - 1), 0],
"string_b": [str(index - 1), 0],
"delimiter": "",
},
}
source["1"]["inputs"]["text"] = ["48", 0]
result = resolve_prompt_pair(source, profile=resolver.INSPECTION_PROFILE)
self.assertEqual(result["state"], "unresolved")
self.assertEqual(result["reason_code"], "work-limit")
# A memo hit charges its unfolded cost: the 2^k unfolded visits exhaust
# the 2,000 allowance (per-entry paths never outgrow the allowance).
self.assertEqual(result["reason"], "Resolution visit limit reached")
if __name__ == "__main__":
unittest.main()
+672
View File
@@ -0,0 +1,672 @@
"""Read-only inspection corpus tool for the #240 invariants (H1 to H8).
Runs the workspace current-file inspection path (``inspect_current_file`` then
``encode_inspection_response``) of a code tree on every servable media file under
``--root``, and hands each exact response to the workspace's JS facts. Example::
python3 tools/inspection_corpus.py --root OUTPUT_DIR --out SCRATCH/after.jsonl \\
--workers 8 --baseline SCRATCH/before.jsonl --summary SCRATCH/after.txt
``--code`` selects the tree whose Python and ``web/js`` are imported (default: the
tree holding this tool), so a copy of the tool can measure an older commit. Run
it against a base checkout first and pass that output as ``--baseline``.
Read-only by construction: media is opened only for reading, symlinks are not
followed, no database or config is imported, bytecode is not written, and
``--out``/``--summary`` are refused inside the scanned root or a code tree. The
JSONL rows and the summary carry counts, statuses and SHA-256 digests of resolved
role texts, never file names or prompt text, unless ``--with-paths`` is given.
"""
import argparse
import hashlib
import importlib
import importlib.util
import json
import math
import multiprocessing
import os
import shutil
import stat
import struct
import subprocess # nosec B404
import sys
import time
from collections import Counter
from pathlib import Path
from types import SimpleNamespace
TOOL_TREE = Path(__file__).resolve().parents[1]
FACTS_HARNESS = Path(__file__).resolve().with_name("inspection_corpus_facts.mjs")
SERVICE = Path("py", "api", "inspection_service.py")
LIBRARY = {"status": "none"}
RECORD_KEYS = ("prompt", "parameters", "workflow", "comfyassets_prompt_pairs")
ROLE_RECORDS = frozenset({"prompt", "parameters"})
FACTS_KNOWN_TARGET = 4
TOP_CLASSES = 20
BASELINE_P99_MS = 520
# H8 compares p99 with twice the baseline p99; below this floor a tiny baseline
# (a few ms on fixtures) would only measure scheduler noise.
H8_FLOOR_MS = 50
P99_FACTOR = 2
MAX_MS = 5000
PNG_SIGNATURE = b"\x89PNG\r\n\x1a\n"
PNG_TEXT_CHUNKS = (b"tEXt", b"zTXt", b"iTXt")
MAX_PNG_CHUNKS = 10_000
HASH_BLOCK = 1 << 20
MAX_JSON_DEPTH = 64
EXIT_OK, EXIT_INVARIANT_FAILED, EXIT_REFUSED = 0, 1, 2
_STATE = SimpleNamespace(code=None, node=None)
class Refused(Exception):
"""An argument would make the run write somewhere it must not."""
def _inside(path: Path, area: Path) -> bool:
child = os.path.normcase(os.path.realpath(path))
parent = os.path.normcase(os.path.realpath(area))
try:
return os.path.commonpath([child, parent]) == parent
except ValueError: # different drives on Windows
return False
def _check_outputs(args) -> None:
areas = [("--root", args.root), ("the code tree", args.code)]
if (TOOL_TREE / SERVICE).is_file(): # a copied tool may live in scratch
areas.append(("this tool's code tree", TOOL_TREE))
outputs = [("--out", args.out)] + (
[("--summary", args.summary)] if args.summary else []
)
for label, path in outputs:
if Path(path).is_dir():
raise Refused(f"{label} is a directory")
for name, area in areas:
if _inside(Path(path), area):
raise Refused(f"refusing to write {label} inside {name}")
targets = [os.path.realpath(path) for _, path in outputs]
if args.baseline:
targets.append(os.path.realpath(args.baseline))
if len(set(map(os.path.normcase, targets))) != len(targets):
raise Refused("--out, --summary and --baseline must be different files")
def _load_code(code_dir: Path) -> SimpleNamespace:
"""Import only the inspection modules of ``code_dir``: no routes, no database."""
location = code_dir / SERVICE
if not location.is_file():
raise Refused("--code is not a PromptManager code tree")
loaded = sys.modules.get("utils")
if loaded is None:
sys.path.insert(0, str(code_dir))
elif not any(_inside(Path(p), code_dir) for p in loaded.__path__):
raise Refused("another code tree's utils package is already imported")
spec = importlib.util.spec_from_file_location("_corpus_inspection", location)
service = importlib.util.module_from_spec(spec)
spec.loader.exec_module(service)
response = importlib.import_module("utils.inspection_response")
inspector = importlib.import_module("utils.prompt_inspector")
return SimpleNamespace(
inspect=service.inspect_current_file,
extensions=frozenset(service.SERVABLE_EXTENSIONS),
encode=response.encode_inspection_response,
cap=response.MAX_RESPONSE_BYTES,
read=inspector.read_inspection_file,
read_error=inspector.MetadataReadError,
)
def _start_node(node: str | None, code_dir: Path):
if not node:
return None
# Fixed local Node harness, no shell.
return subprocess.Popen( # nosec B603
[node, str(FACTS_HARNESS), str(code_dir)],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.DEVNULL,
)
def _close_node(process) -> None:
for stream in (process.stdin, process.stdout):
try:
stream.close()
except OSError: # the pipe of an exited harness
pass
process.wait()
def _stop_node() -> None:
process, _STATE.node = _STATE.node, None
if process is not None:
_close_node(process)
def _init_worker(code_dir: str, node: str | None) -> None:
sys.dont_write_bytecode = True # never write into the measured code tree
_STATE.code = _load_code(Path(code_dir))
_STATE.node = _start_node(node, Path(code_dir))
def _facts(body: bytes) -> dict:
process = _STATE.node
if process is None:
return {"facts": None, "facts_error": "node-unavailable"}
try:
process.stdin.write(body + b"\n")
process.stdin.flush()
line = process.stdout.readline()
except OSError:
line = b""
if not line:
_STATE.node = None
_close_node(process)
return {"facts": None, "facts_error": "node-exited"}
facts = json.loads(line)
if "error" in facts:
return {"facts": None, "facts_error": facts["error"]}
known = sum(1 for fact in facts.values() if fact["status"] == "known")
return {"facts": facts, "facts_known": known}
def _digest(text: object) -> str | None:
if not isinstance(text, str):
return None
return hashlib.sha256(text.encode("utf-8", "surrogatepass")).hexdigest()
def _roles(prompts: dict) -> dict:
return {
role: {
"status": side.get("status"),
"source": side.get("source"),
"sha256": (
_digest(side.get("text")) if side.get("status") == "resolved" else None
),
}
for role, side in prompts.items()
}
def _texts(side_pair: dict) -> tuple:
return tuple(
(side_pair.get(role) or {}).get("text") for role in ("positive", "negative")
)
def _resolution(response: dict) -> dict:
# ``candidates`` (and so H2) only appears with the selection slice (PR3);
# PR1 responses carry none, so H2 is vacuous until then.
resolution = response["resolution"]
candidates = resolution.get("candidates") or []
node = resolution.get("node")
resolved = _texts(response["prompts"])
return {
"state": resolution.get("state"),
"reason": resolution.get("reason"),
"reason_code": resolution.get("reason_code"),
"selection": resolution.get("selection"),
"node_class": node.get("class_type") if isinstance(node, dict) else None,
"candidate_count": resolution.get("candidate_count", len(candidates)),
"h2": resolution.get("state") == "resolved"
and any(_texts(candidate) != resolved for candidate in candidates),
}
def _nonfinite(value: object) -> bool:
stack = [(value, 1)]
while stack:
item, depth = stack.pop()
if isinstance(item, float) and not math.isfinite(item):
return True
if depth < MAX_JSON_DEPTH and isinstance(item, (dict, list)):
children = item.values() if isinstance(item, dict) else item
stack.extend((child, depth + 1) for child in children)
return False
def _unique(pairs: list) -> dict:
keys = [key for key, _ in pairs]
if len(set(keys)) != len(keys):
raise ValueError("duplicate key")
return dict(pairs)
def malformed_cause(text: str) -> str:
"""Why a prompt record may be malformed: ``non-input`` means only quirks outside
node ``inputs`` (``_meta``, ``is_changed``, nonfinite numbers) are left."""
try:
tree = json.loads(text, object_pairs_hook=_unique)
except (ValueError, RecursionError):
return "unparseable"
nodes = tree.values() if isinstance(tree, dict) else None
if nodes is None or not all(
isinstance(node, dict)
and isinstance(node.get("class_type"), str)
and isinstance(node.get("inputs", {}), dict)
for node in nodes
):
return "shape"
if any(_nonfinite(node.get("inputs")) for node in nodes):
return "input-nonfinite"
return "non-input"
def _hash_chunk(handle, length: int) -> tuple:
digest, keyword, remaining = hashlib.sha256(), b"", length
while remaining:
block = handle.read(min(HASH_BLOCK, remaining))
if not block:
break
if remaining == length:
keyword = block.split(b"\0", 1)[0]
digest.update(block)
remaining -= len(block)
return keyword.decode("latin-1"), digest.digest()
def png_duplicates_identical(path: Path) -> bool | None:
"""True when every duplicated record key repeats byte-identical chunk data."""
seen: dict = {}
with open(path, "rb") as handle:
if handle.read(8) != PNG_SIGNATURE:
return None
for _ in range(MAX_PNG_CHUNKS):
head = handle.read(8)
if len(head) < 8:
break
length, kind = struct.unpack(">I4s", head)
if kind in PNG_TEXT_CHUNKS:
keyword, digest = _hash_chunk(handle, length)
if keyword in RECORD_KEYS:
seen.setdefault(keyword, []).append(digest)
handle.seek(4, os.SEEK_CUR)
else:
handle.seek(length + 4, os.SEEK_CUR)
if kind == b"IEND":
break
repeated = [digests for digests in seen.values() if len(digests) > 1]
if not repeated:
return None
return all(len(set(digests)) == 1 for digests in repeated)
def _read_evidence(path: Path, row: dict) -> None:
code = _STATE.code
try:
code.read(path)
row["read_error"] = None
except code.read_error as error:
row["read_error"] = str(error)
if row["read_error"] == "Duplicate prompt metadata" and row["ext"] == ".png":
row["duplicates_identical"] = png_duplicates_identical(path)
def _metadata(response: dict, inspection) -> dict:
metadata = response["metadata"]
records = set() if inspection is None else set(inspection.metadata)
records |= set(getattr(inspection, "limited_records", ()) or ())
row = {
"records": sorted(records & set(RECORD_KEYS)),
"metadata_status": metadata["status"],
"omitted_reason": metadata["omitted_reason"],
"graphs": {
"prompt": metadata["prompt_graph"]["status"],
"workflow": metadata["workflow_graph"]["status"],
},
"record_formats": metadata["record_formats"],
}
text = None if inspection is None else inspection.metadata.get("prompt")
if row["graphs"]["prompt"] == "malformed" and isinstance(text, str):
row["malformed_cause"] = malformed_cause(text)
return row
def _observe(path: Path, rel: str, row: dict) -> None:
code = _STATE.code
row["size"] = path.stat().st_size
_read_evidence(path, row)
started = time.perf_counter()
observed = code.inspect(path, source_label="corpus", saved=None, library=LIBRARY)
image = {"root_index": 0, "path": rel, "filename": path.name}
try:
body = code.encode(image=image, root_scope="0" * 64, **observed)
except ValueError:
row["ms"] = round((time.perf_counter() - started) * 1000, 1)
row["over_cap"] = True
return
row["ms"] = round((time.perf_counter() - started) * 1000, 1)
row["response_bytes"] = len(body)
row["over_cap"] = len(body) > code.cap
response = json.loads(body)
row["prompts"] = _roles(response["prompts"])
row.update(_resolution(response))
row.update(_metadata(response, observed["inspection"]))
row.update(_facts(body))
def file_id(rel: str) -> str:
"""Stable, name-free join key for ``--baseline`` comparisons."""
return hashlib.sha256(rel.encode("utf-8", "surrogatepass")).hexdigest()[:32]
def inspect_one(task: tuple) -> dict:
path, rel, with_paths = Path(task[0]), task[1], task[2]
row = {"id": file_id(rel), "ext": path.suffix.lower()}
if with_paths:
row["path"] = rel
try:
_observe(path, rel, row)
except Exception as error: # one file never stops the corpus; class only
row["error"] = type(error).__name__
return row
def media_files(root: Path, extensions: frozenset):
"""Regular servable files under ``root``; symlinks are neither followed nor read."""
for directory, subdirectories, names in os.walk(root, followlinks=False):
subdirectories.sort()
for name in sorted(names):
path = Path(directory, name)
if path.suffix.lower() not in extensions:
continue
try:
mode = path.lstat().st_mode
except OSError:
continue
if stat.S_ISREG(mode):
yield str(path), path.relative_to(root).as_posix()
def _rows(tasks: list, workers: int, code_dir: Path, node: str | None):
if workers <= 1:
_STATE.node = _start_node(node, code_dir)
try:
for task in tasks:
yield inspect_one(task)
finally:
_stop_node()
return
context = multiprocessing.get_context("spawn")
with context.Pool(workers, _init_worker, (str(code_dir), node)) as pool:
yield from pool.imap_unordered(inspect_one, tasks, chunksize=4)
def percentile(values: list, fraction: float) -> float | None:
"""Nearest-rank percentile; ``None`` for no values."""
if not values:
return None
ordered = sorted(values)
return ordered[max(0, math.ceil(fraction * len(ordered)) - 1)]
def _fact_reasons(row: dict) -> set:
return {fact.get("reason") for fact in (row.get("facts") or {}).values()}
def _h1(rows: list, baseline: list | None) -> dict:
if baseline is None:
return {"status": "N/A", "detail": "no --baseline"}
current = {row["id"]: row for row in rows}
before = [row for row in baseline if row.get("state") == "resolved"]
regressions = Counter()
for old in before:
new = current.get(old["id"])
if new is None:
regressions["missing"] += 1
elif new.get("state") != "resolved":
regressions[f"{new.get('state')}:{new.get('reason_code')}"] += 1
elif _role_digests(new) != _role_digests(old):
regressions["resolved:texts-differ"] += 1
resolved_now = sum(1 for row in rows if row.get("state") == "resolved")
return {
"status": "FAIL" if regressions else "PASS",
"count": sum(regressions.values()),
"detail": f"resolved at baseline {len(before)}, now {resolved_now}",
"breakdown": dict(sorted(regressions.items())),
}
def _role_digests(row: dict) -> tuple:
prompts = row.get("prompts") or {}
return tuple(
(
(prompts.get(role) or {}).get("status"),
(prompts.get(role) or {}).get("sha256"),
)
for role in ("positive", "negative")
)
def _false_absent(row: dict) -> bool:
if not ROLE_RECORDS & set(row.get("records") or ()):
return False
positive = ((row.get("prompts") or {}).get("positive") or {}).get("status")
# Negative-only absence beside a resolved positive is honest (ledger ruling).
return (
positive == "absent" or row.get("state") == "absent" and positive != "resolved"
)
def _counted(count: int, detail: str = "") -> dict:
return {"status": "FAIL" if count else "PASS", "count": count, "detail": detail}
def _h8(rows: list, baseline: list | None, cap: int) -> dict:
timings = [row["ms"] for row in rows if "ms" in row]
p99, peak = percentile(timings, 0.99), max(timings, default=None)
base = percentile([row["ms"] for row in baseline or () if "ms" in row], 0.99)
limit = max(
P99_FACTOR * (base if base is not None else BASELINE_P99_MS), H8_FLOOR_MS
)
over_cap = sum(1 for row in rows if row.get("over_cap"))
failed = (p99 or 0) > limit or (peak or 0) >= MAX_MS or over_cap
return {
"status": "FAIL" if failed else "PASS",
"count": over_cap,
"detail": f"p99 {p99} ms (limit {limit}), max {peak} ms (limit <{MAX_MS}), "
f"responses over the {cap}-byte cap {over_cap}",
}
def _h3(rows: list) -> dict:
limited = sum(
1
for row in rows
if row.get("read_error") is None and "projection-limited" in _fact_reasons(row)
)
# Facts that were never computed cannot pass: count them as failures.
missing = sum(1 for row in rows if row.get("facts_error") or row.get("error"))
detail = f"projection-limited {limited}, facts not computed {missing}"
return _counted(limited + missing, detail)
def _harness(rows: list) -> dict:
"""The tool's own failures, kept apart from inspector outcomes: an unreadable
or unsupported file is a row with ``read_error``/``state``, not an ``error``."""
crashed = sum(1 for row in rows if row.get("error"))
no_facts = sum(1 for row in rows if row.get("facts_error"))
detail = f"inspection exceptions {crashed}, facts not computed {no_facts}"
return _counted(crashed + no_facts, detail)
def invariants(rows: list, baseline: list | None, cap: int) -> dict:
"""H1 to H8 from the design; H7 is informational until the video slice."""
videos = [row for row in rows if row["ext"] == ".mp4"]
with_records = sum(1 for row in videos if row.get("records"))
return {
"H1": _h1(rows, baseline),
"H2": _counted(sum(1 for row in rows if row.get("h2"))),
"H3": _h3(rows),
"H4": _counted(
sum(1 for row in rows if row.get("malformed_cause") == "non-input")
),
"H5": _counted(sum(1 for row in rows if _false_absent(row))),
"H6": _counted(
sum(1 for row in rows if row.get("duplicates_identical") is True)
),
"H7": {
"status": "N/A",
"detail": "informational until PR4: mp4 with records "
f"{with_records} of {len(videos)}",
},
"H8": _h8(rows, baseline, cap),
"harness": _harness(rows),
}
def build_report(rows: list, baseline: list | None, cap: int) -> dict:
"""Aggregate counts only: no names, no text."""
timings = [row["ms"] for row in rows if "ms" in row]
unsupported = Counter(row["node_class"] for row in rows if row.get("node_class"))
return {
"files": len(rows),
"extensions": Counter(row["ext"] for row in rows),
"harness_errors": Counter(row["error"] for row in rows if row.get("error")),
"states": Counter(row.get("state") for row in rows if not row.get("error")),
"states_by_extension": Counter(
f"{row['ext']} {row.get('state')}" for row in rows if not row.get("error")
),
"reason_codes": Counter(
str(row.get("reason_code") or "null")
for row in rows
if not row.get("error")
),
"selection_kinds": Counter(
str(row.get("selection") or "null") for row in rows if not row.get("error")
),
"metadata_status": Counter(
row.get("metadata_status") for row in rows if "metadata_status" in row
),
"read_errors": Counter(
row["read_error"] for row in rows if row.get("read_error")
),
"malformed_causes": Counter(
row["malformed_cause"] for row in rows if "malformed_cause" in row
),
"facts_known_at_least_4": sum(
1 for row in rows if (row.get("facts_known") or 0) >= FACTS_KNOWN_TARGET
),
"facts_errors": Counter(
row["facts_error"] for row in rows if row.get("facts_error")
),
"top_unsupported_classes": unsupported.most_common(TOP_CLASSES),
"timing_ms": {
"p50": percentile(timings, 0.5),
"p99": percentile(timings, 0.99),
"max": max(timings, default=None),
},
"invariants": invariants(rows, baseline, cap),
}
def _counter_line(label: str, counts) -> str:
items = counts.most_common() if isinstance(counts, Counter) else counts
body = ", ".join(f"{key} {value}" for key, value in items) or "none"
return f"{label}: {body}"
def format_report(report: dict) -> str:
timing = report["timing_ms"]
lines = [
"Inspection corpus summary",
f"files: {report['files']}",
_counter_line("extensions", report["extensions"]),
_counter_line("harness errors", report["harness_errors"]),
_counter_line("states", report["states"]),
_counter_line("states by extension", report["states_by_extension"]),
_counter_line("reason codes", report["reason_codes"]),
_counter_line("selection kinds", report["selection_kinds"]),
_counter_line("metadata status", report["metadata_status"]),
_counter_line("read errors", report["read_errors"]),
_counter_line("malformed prompt graph causes", report["malformed_causes"]),
f"facts at least {FACTS_KNOWN_TARGET} known: "
f"{report['facts_known_at_least_4']}",
_counter_line("facts errors", report["facts_errors"]),
_counter_line(
f"top {TOP_CLASSES} unsupported classes", report["top_unsupported_classes"]
),
f"timing ms: p50 {timing['p50']}, p99 {timing['p99']}, max {timing['max']}",
"invariants:",
]
for name, result in report["invariants"].items():
count = f" {result['count']}" if "count" in result else ""
detail = f" ({result['detail']})" if result.get("detail") else ""
lines.append(f" {name} {result['status']}{count}{detail}")
for key, value in (result.get("breakdown") or {}).items():
lines.append(f" {key} {value}")
return "\n".join(lines) + "\n"
def _by_id(row: dict) -> str:
return row["id"]
def _load_baseline(path: str | None) -> list | None:
if path is None:
return None
with open(path, encoding="utf-8") as handle:
return [json.loads(line) for line in handle if line.strip()]
def _parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
parser.add_argument("--root", required=True, help="media folder, scanned read-only")
parser.add_argument("--out", required=True, help="per-file JSONL to write")
parser.add_argument("--workers", type=int, default=1)
parser.add_argument("--baseline", help="JSONL from an earlier run (H1, H8)")
parser.add_argument("--summary", help="also write the summary text here")
parser.add_argument("--code", default=str(TOOL_TREE), help="code tree to measure")
parser.add_argument("--node", default=shutil.which("node"), help="Node.js binary")
parser.add_argument(
"--with-paths", action="store_true", help="add relative paths to the JSONL"
)
return parser
def main(argv: list | None = None) -> int:
args = _parser().parse_args(argv)
previous, sys.dont_write_bytecode = sys.dont_write_bytecode, True
try: # never write bytecode into the measured code tree
return _run(args)
finally:
sys.dont_write_bytecode = previous
def _run(args) -> int:
args.root = Path(os.path.realpath(args.root))
args.code = Path(os.path.realpath(args.code))
try:
if not args.root.is_dir():
raise Refused("--root is not a directory")
if args.node and shutil.which(args.node) is None:
raise Refused("--node is not an executable")
_check_outputs(args)
_STATE.code = _load_code(args.code)
baseline = _load_baseline(args.baseline)
except (Refused, OSError, ValueError) as error:
print(f"inspection_corpus: {error}", file=sys.stderr)
return EXIT_REFUSED
tasks = [
(path, rel, args.with_paths)
for path, rel in media_files(args.root, _STATE.code.extensions)
]
rows = sorted(_rows(tasks, args.workers, args.code, args.node), key=_by_id)
with open(args.out, "w", encoding="utf-8", newline="\n") as out:
for row in rows:
out.write(json.dumps(row, ensure_ascii=False, sort_keys=True) + "\n")
report = build_report(rows, baseline, _STATE.code.cap)
text = format_report(report)
sys.stdout.write(text)
if args.summary:
with open(args.summary, "w", encoding="utf-8", newline="\n") as handle:
handle.write(text)
failed = any(result["status"] == "FAIL" for result in report["invariants"].values())
return EXIT_INVARIANT_FAILED if failed else EXIT_OK
if __name__ == "__main__":
sys.exit(main())
+28
View File
@@ -0,0 +1,28 @@
// Facts for tools/inspection_corpus.py, through the workspace's own path:
// projectInspectionPayload validates the wire, then extractGenerationFacts reads
// the bounded projection. argv[2] is the code tree whose web/js is loaded.
// stdin: one exact inspection response JSON text per line.
// stdout: one line per input, {field: {status, reason}} or {error}.
import { createRequire } from 'node:module';
import { pathToFileURL } from 'node:url';
import path from 'node:path';
import readline from 'node:readline';
const web = path.join(path.resolve(process.argv[2]), 'web', 'js');
const require = createRequire(import.meta.url);
const { extractGenerationFacts } = require(path.join(web, 'comfy-metadata.js'));
const { projectInspectionPayload } = await import(pathToFileURL(path.join(web, 'workspace', 'inspection-state.mjs')).href);
const identity = { kind: 'rooted-file', key: '0:corpus.png', root_index: 0, path: 'corpus.png', root_scope: null };
function facts(line) {
try {
const record = projectInspectionPayload(JSON.parse(line), identity, null, extractGenerationFacts);
return Object.fromEntries(Object.entries(record.generation).map(([field, fact]) => [field, { status: fact.status, reason: fact.reason ?? null }]));
} catch (error) {
return { error: error?.name || 'Error' };
}
}
for await (const line of readline.createInterface({ input: process.stdin, crlfDelay: Infinity })) {
process.stdout.write(`${JSON.stringify(facts(line))}\n`);
}
+1 -1
View File
@@ -241,7 +241,7 @@
},
"workspace/library.mjs": {
"lines": 95,
"branches": 84,
"branches": 85,
"functions": 96,
"statements": 95
},
+118 -40
View File
@@ -3,7 +3,7 @@
import json
import math
from decimal import Decimal, InvalidOperation
from typing import TYPE_CHECKING
from typing import TYPE_CHECKING, Literal
if TYPE_CHECKING:
from .prompt_inspector import InspectionRead
@@ -14,7 +14,13 @@ _MAX_SAFE_INTEGER = 2**53 - 1
_RECORD_KEYS = frozenset(
{"prompt", "workflow", "parameters", "comfyassets_prompt_pairs"}
)
_FORMATS = frozenset({"original-text", "reserialized-json", "invalid"})
_FORMATS = frozenset({"original-text", "reserialized-json", "invalid", "limited"})
_PROMPT_NODE_KEYS = frozenset({"class_type", "inputs"})
MAX_PROJECTION_DEPTH = 64
# Backstop, not an expected outcome: executable API nodes for a prompt; UI nodes
# including subgraph definitions for a workflow.
GRAPH_NODE_CEILING = 20_000
ProjectionKind = Literal["prompt", "workflow", "carrier"]
class _ProjectionLimit(ValueError):
@@ -30,23 +36,27 @@ def _pairs(items: list[tuple]) -> dict:
return result
def _nonfinite(value: str) -> None:
raise ValueError("Nonfinite JSON number")
class _Nonfinite:
"""A parsed ``NaN``/``Infinity`` literal: never a fact, projected as null."""
def _normalize(
value: object, depth: int = 1, *, retain: bool = True, carrier: bool = False
) -> object:
"""Walk bounded source once; unrelated carrier fields are only validated."""
_NONFINITE = _Nonfinite()
def _nonfinite(value: str) -> _Nonfinite:
return _NONFINITE
def _normalize(value: object, depth: int = 1, *, retain: bool = True) -> object:
"""Walk bounded source once; unretained values are only validated as JSON."""
if isinstance(value, (dict, list)):
if depth > 64:
if depth > MAX_PROJECTION_DEPTH:
raise _ProjectionLimit()
if isinstance(value, dict):
result = {} if retain else None
for key, child in value.items():
keep = retain and (not carrier or key in ("prompt", "workflow"))
normalized = _normalize(child, depth + 1, retain=keep)
if keep:
normalized = _normalize(child, depth + 1, retain=retain)
if retain:
result[key] = normalized
else:
result = [] if retain else None
@@ -55,17 +65,65 @@ def _normalize(
if retain:
result.append(normalized)
return result
if not retain or value is _NONFINITE:
return None
if isinstance(value, Decimal):
if not value.is_finite():
raise ValueError("Nonfinite JSON number")
# str keeps large exponents compact and never rounds through a float.
return str(value) if retain else None
return str(value)
if type(value) is int and abs(value) > _MAX_SAFE_INTEGER:
return str(value) if retain else None
return value if retain else None
return str(value)
return value
def _strict_projection(text: str, *, carrier: bool = False) -> object:
def _project_prompt(tree: object, depth: int = 1) -> dict:
"""Each API node keeps exactly ``class_type`` and ``inputs``.
Other node keys (``_meta``, ``is_changed``, future keys) are validated and
dropped, so a nonfinite value there never reaches the projection.
"""
if not isinstance(tree, dict):
raise ValueError("Prompt graph must be an object of nodes")
if depth + 1 > MAX_PROJECTION_DEPTH: # the root and its node objects
raise _ProjectionLimit()
result = {}
for node_id, node in tree.items():
if not isinstance(node, dict):
result[node_id] = _normalize(node, depth + 1)
continue
projected = {}
for key, child in node.items():
keep = key in _PROMPT_NODE_KEYS
normalized = _normalize(child, depth + 2, retain=keep)
if keep:
projected[key] = normalized
result[node_id] = projected
return result
def _project(tree: object, kind: str, depth: int = 1) -> object:
if kind == "prompt":
return _project_prompt(tree, depth)
if kind == "workflow":
return _normalize(tree, depth)
if not isinstance(tree, dict):
raise ValueError("Malformed carrier")
result = {}
for key, member in tree.items():
if key not in ("prompt", "workflow"):
_normalize(member, depth + 1, retain=False)
elif isinstance(member, str):
result[key] = member # projected by its own kind when read
elif key == "prompt" and not isinstance(member, dict):
result[key] = _normalize(member, depth + 1)
else:
result[key] = _project(member, key, depth + 1)
return result
def _strict_projection(text: str, *, kind: ProjectionKind) -> object:
"""Parse exactly (Decimal, no duplicate keys) and project by record kind."""
try:
tree = json.loads(
text,
@@ -73,34 +131,35 @@ def _strict_projection(text: str, *, carrier: bool = False) -> object:
object_pairs_hook=_pairs,
parse_constant=_nonfinite,
)
if carrier and not isinstance(tree, dict):
raise ValueError("Malformed carrier")
return _normalize(tree, carrier=carrier)
return _project(tree, kind)
except RecursionError:
raise _ProjectionLimit() from None
except InvalidOperation:
raise ValueError("JSON decimal exceeds supported range") from None
def _node_count(value: dict, *, workflow: bool) -> int:
if not workflow:
return len(value)
nodes = value.get("nodes")
count = len(nodes) if isinstance(nodes, list) else 0
definitions = value.get("definitions")
subgraphs = definitions.get("subgraphs") if isinstance(definitions, dict) else None
for subgraph in subgraphs if isinstance(subgraphs, list) else ():
inner = subgraph.get("nodes") if isinstance(subgraph, dict) else None
count += len(inner) if isinstance(inner, list) else 0
return count
def _graph(value: object, *, workflow: bool = False) -> dict:
if not isinstance(value, dict):
return {"status": "malformed", "value": None}
nodes = value.get("nodes") if workflow else value
if isinstance(nodes, (dict, list)) and len(nodes) > 500:
if _node_count(value, workflow=workflow) > GRAPH_NODE_CEILING:
return {"status": "limited", "value": None}
return {"status": "parsed", "value": value}
def _metadata(inspection: "InspectionRead | None") -> dict:
raw = (
{}
if inspection is None
else {
key: value
for key, value in inspection.metadata.items()
if key in _RECORD_KEYS and isinstance(value, str)
}
)
formats = (
{}
if inspection is None
@@ -110,6 +169,22 @@ def _metadata(inspection: "InspectionRead | None") -> dict:
if key in _RECORD_KEYS and value in _FORMATS
}
)
if inspection is not None:
# A record over its read budget has no text: like ``invalid``, raw omits
# it, and its graph is ``limited`` rather than parsed from nothing.
for key in inspection.limited_records & _RECORD_KEYS:
formats[key] = "limited"
raw = (
{}
if inspection is None
else {
key: value
for key, value in inspection.metadata.items()
if key in _RECORD_KEYS
and isinstance(value, str)
and formats.get(key) != "limited"
}
)
result = {
"status": (
"unreadable"
@@ -124,7 +199,7 @@ def _metadata(inspection: "InspectionRead | None") -> dict:
carrier, failed = None, None
if inspection is not None and inspection.video_carrier_text is not None:
try:
carrier = _strict_projection(inspection.video_carrier_text, carrier=True)
carrier = _strict_projection(inspection.video_carrier_text, kind="carrier")
except _ProjectionLimit:
failed = "limited"
except (ValueError, TypeError, OverflowError):
@@ -132,8 +207,8 @@ def _metadata(inspection: "InspectionRead | None") -> dict:
for key, field in (("prompt", "prompt_graph"), ("workflow", "workflow_graph")):
graph = {"status": "absent", "value": None}
if key in raw or key in formats:
if failed:
graph["status"] = failed
if failed or formats.get(key) == "limited":
graph["status"] = failed or "limited"
elif formats.get(key) == "invalid":
graph["status"] = "malformed"
else:
@@ -141,11 +216,11 @@ def _metadata(inspection: "InspectionRead | None") -> dict:
if carrier is not None:
value = carrier.get(key)
if isinstance(value, str):
value = _strict_projection(value)
value = _strict_projection(value, kind=key)
elif formats.get(key) == "reserialized-json":
raise _ProjectionLimit()
else:
value = _strict_projection(raw[key])
value = _strict_projection(raw[key], kind=key)
graph = _graph(value, workflow=key == "workflow")
except _ProjectionLimit:
graph["status"] = "limited"
@@ -259,11 +334,14 @@ def _encode_inspection(
"library": _library_projection(library),
}
body = _bytes(response)
if len(body) > MAX_RESPONSE_BYTES:
for field in ("prompt_graph", "workflow_graph"):
if metadata[field]["status"] != "absent":
metadata[field] = {"status": "limited", "value": None}
body = _bytes(response)
# Omission order keeps prompt evidence longest: the workflow graph first,
# then the prompt graph, then all raw records atomically.
for field in ("workflow_graph", "prompt_graph"):
if len(body) <= MAX_RESPONSE_BYTES:
break
if metadata[field]["status"] != "absent":
metadata[field] = {"status": "limited", "value": None}
body = _bytes(response)
if len(body) > MAX_RESPONSE_BYTES:
metadata.update(
raw={},
+186 -51
View File
@@ -1,26 +1,44 @@
"""Read prompt evidence without inference, indexing, or decoding image pixels."""
import hashlib
import io
import json
import struct
import subprocess # nosec B404
import threading
import zlib
from dataclasses import dataclass
from dataclasses import dataclass, field as dataclass_field
from pathlib import Path
from typing import Any
from PIL import Image
from .pair_execution import EXECUTION_KEY
from .prompt_pair_resolver import resolve_prompt_pair
from .prompt_pair_resolver import (
INSPECTION_PROFILE,
ResolverProfile,
resolve_prompt_pair,
)
from .video_metadata import ffprobe_path, is_video_path
MAX_METADATA_BYTES = 2 * 1024 * 1024
# Role records (prompt, parameters, the pair execution record) share one
# budget, encoded and decoded: roles cannot be trusted from a partial record,
# so exceeding it fails the whole read. A workflow has its own budget; past it
# only the workflow is ``limited``. Other text chunks are never retained.
ROLE_METADATA_BYTES = 2 * 1024 * 1024
WORKFLOW_METADATA_BYTES = 4 * 1024 * 1024
# Total bytes read for byte-identical duplicate chunks of one file, per class
# so workflow repeats can never starve role records: two budgets each, making
# repeated chunks cost bounded I/O instead of file size.
DUPLICATE_WORKFLOW_BYTES = 2 * WORKFLOW_METADATA_BYTES
DUPLICATE_ROLE_BYTES = 2 * ROLE_METADATA_BYTES
MAX_METADATA_BYTES = ROLE_METADATA_BYTES # video stdout bound, unchanged here
MAX_KEYWORD_BYTES = 80 # a 1-79 byte PNG keyword plus its NUL separator
MAX_HEADER_BYTES = 8 * 1024 * 1024
MAX_PNG_CHUNKS = 10_000
_ROLE_METADATA_KEYS = frozenset({"prompt", "parameters", EXECUTION_KEY})
_METADATA_KEYS = _ROLE_METADATA_KEYS | {"workflow"}
_TEXT_CHUNKS = (b"tEXt", b"zTXt", b"iTXt")
_SIGNATURE = b"\x89PNG\r\n\x1a\n"
@@ -38,6 +56,8 @@ class InspectionRead:
format: str | None
width: int | None
height: int | None
# Records present but over their budget: no text, format ``limited``.
limited_records: frozenset[str] = dataclass_field(default_factory=frozenset)
@dataclass(frozen=True)
@@ -47,24 +67,33 @@ class _VideoInspection:
carrier_text: str | None
def _inflate(data: bytes) -> bytes:
def _inflate(data: bytes, limit: int) -> bytes | None:
"""Inflate at most ``limit + 1`` bytes; ``None`` when the output is over limit."""
inflater = zlib.decompressobj()
result = inflater.decompress(data, MAX_METADATA_BYTES + 1)
if len(result) > MAX_METADATA_BYTES or not inflater.eof or inflater.unused_data:
result = inflater.decompress(data, limit + 1)
if len(result) > limit:
return None
if not inflater.eof or inflater.unused_data:
raise MetadataReadError("Compressed metadata is invalid or exceeds limits")
return result
def _png_text(kind: bytes, data: bytes) -> tuple[str, str]:
keyword, separator, body = data.partition(b"\0")
def _keyword(prefix: bytes) -> str:
keyword, separator, _ = prefix.partition(b"\0")
if not separator or not 1 <= len(keyword) <= 79:
raise MetadataReadError("Malformed PNG text metadata")
key = keyword.decode("latin-1")
return keyword.decode("latin-1")
def _png_text(kind: bytes, data: bytes, limit: int) -> tuple[str, str | None]:
"""Decode one retained text chunk; ``None`` text when it inflates past limit."""
_, _, body = data.partition(b"\0")
key = _keyword(data[:MAX_KEYWORD_BYTES])
encoding = "latin-1"
if kind == b"zTXt":
if not body or body[0] != 0:
raise MetadataReadError("Unsupported PNG text compression")
body = _inflate(body[1:])
body = _inflate(body[1:], limit)
elif kind == b"iTXt":
if len(body) < 2 or body[0] not in (0, 1):
raise MetadataReadError("Malformed international PNG metadata")
@@ -74,21 +103,91 @@ def _png_text(kind: bytes, data: bytes) -> tuple[str, str]:
raise MetadataReadError("Malformed international PNG metadata")
fields[0].decode("ascii")
fields[1].decode("utf-8")
body = _inflate(fields[2]) if flag else fields[2]
body = _inflate(fields[2], limit) if flag else fields[2]
encoding = "utf-8"
if body is None:
return key, None
if b"\0" in body:
raise MetadataReadError("Malformed PNG text metadata")
return key, body.decode(encoding)
def _png_metadata(stream: Any, size: int) -> tuple[dict, int, int]:
"""CRC-check text and structural chunks, seeking past compressed pixels.
class _PngRecords:
"""Retained records with per-class budgets and byte-identical duplicates."""
def __init__(self) -> None:
self.found: dict[str, str] = {}
self.limited: set[str] = set()
self.digests: dict[str, tuple[int, bytes]] = {}
self.role_encoded = self.role_decoded = 0
self.duplicate_workflow = self.duplicate_role = 0
def skip(self, key: str, length: int) -> bool:
"""Decide from the keyword and length alone whether to seek past a chunk."""
if key not in _METADATA_KEYS or key in self.limited:
return True # never retained, or a duplicate of a limited workflow
if key in self.digests:
if self.digests[key][0] != length:
raise MetadataReadError("Duplicate prompt metadata")
if key == "workflow":
if self.duplicate_workflow + length > DUPLICATE_WORKFLOW_BYTES:
self.limited.add(key)
self.found.pop(key, None)
return True
self.duplicate_workflow += length
else:
if self.duplicate_role + length > DUPLICATE_ROLE_BYTES:
raise MetadataReadError("Metadata exceeds inspection limits")
self.duplicate_role += length
return False # same length: read and compare bytes
if key == "workflow":
if length > WORKFLOW_METADATA_BYTES:
self.limited.add(key)
return True
return False
self.role_encoded += length
if self.role_encoded > ROLE_METADATA_BYTES:
raise MetadataReadError("Metadata exceeds inspection limits")
return False
def add(self, kind: bytes, data: bytes) -> None:
key = _keyword(data[:MAX_KEYWORD_BYTES])
digest = hashlib.sha256(kind + data).digest()
if key in self.digests:
if self.digests[key][1] != digest:
raise MetadataReadError("Duplicate prompt metadata")
return
self.digests[key] = (len(data), digest)
workflow = key == "workflow"
limit = (
WORKFLOW_METADATA_BYTES
if workflow
else ROLE_METADATA_BYTES - self.role_decoded
)
_, value = _png_text(kind, data, limit)
size = None if value is None else len(value.encode("utf-8"))
if workflow:
if size is None or size > WORKFLOW_METADATA_BYTES:
self.limited.add(key)
return
else:
self.role_decoded += limit + 1 if size is None else size
if self.role_decoded > ROLE_METADATA_BYTES:
raise MetadataReadError("Metadata exceeds inspection limits")
self.found[key] = value
def _png_metadata(stream: Any, size: int) -> tuple[dict, frozenset, int, int]:
"""CRC-check retained text and structural chunks, seeking past everything else.
PNG text chunks may follow IDAT. Pillow's ``.text`` loads pixels to find
them; this bounded reader instead follows the W3C PNG chunk layout.
IDAT checksums/pixel validity are deliberately outside metadata inspection.
them; this bounded reader instead follows the W3C PNG chunk layout. A text
chunk's keyword is read first (at most ``MAX_KEYWORD_BYTES``); chunks that
are not retained, or are over their budget, are skipped by length, so their
bytes are never held in memory or checksummed. IDAT checksums/pixel
validity are deliberately outside metadata inspection.
"""
found, compressed_size, expanded_size = {}, 0, 0
records = _PngRecords()
saw_header = saw_pixels = False
for index in range(MAX_PNG_CHUNKS):
header = stream.read(8)
@@ -99,34 +198,32 @@ def _png_metadata(stream: Any, size: int) -> tuple[dict, int, int]:
raise MetadataReadError("Incomplete PNG chunk")
if index == 0 and (kind != b"IHDR" or length != 13):
raise MetadataReadError("Missing PNG header")
if kind in (b"tEXt", b"zTXt", b"iTXt", b"IHDR", b"IEND"):
compressed_size += length
if compressed_size > MAX_METADATA_BYTES:
raise MetadataReadError("Metadata exceeds inspection limits")
data, checksum = stream.read(length), stream.read(4)
if zlib.crc32(kind + data) != int.from_bytes(checksum, "big"):
raise MetadataReadError("PNG metadata checksum mismatch")
if kind == b"IHDR":
if saw_header or not all(struct.unpack(">II", data[:8])):
raise MetadataReadError("Malformed PNG header")
saw_header = True
width, height = struct.unpack(">II", data[:8])
elif kind == b"IEND":
if length or not saw_pixels:
raise MetadataReadError("Incomplete PNG image")
return found, width, height
else:
key, value = _png_text(kind, data)
expanded_size += len(value.encode("utf-8"))
if expanded_size > MAX_METADATA_BYTES:
raise MetadataReadError("Metadata exceeds inspection limits")
if key in _METADATA_KEYS:
if key in found:
raise MetadataReadError("Duplicate prompt metadata")
found[key] = value
if kind in _TEXT_CHUNKS:
prefix = stream.read(min(length, MAX_KEYWORD_BYTES))
if records.skip(_keyword(prefix), length):
stream.seek(length - len(prefix) + 4, 1)
continue
data = prefix + stream.read(length - len(prefix))
elif kind in (b"IHDR", b"IEND"):
data = stream.read(length)
else:
saw_pixels |= kind == b"IDAT"
stream.seek(length + 4, 1)
continue
checksum = stream.read(4)
if zlib.crc32(kind + data) != int.from_bytes(checksum, "big"):
raise MetadataReadError("PNG metadata checksum mismatch")
if kind == b"IHDR":
if saw_header or not all(struct.unpack(">II", data[:8])):
raise MetadataReadError("Malformed PNG header")
saw_header = True
width, height = struct.unpack(">II", data[:8])
elif kind == b"IEND":
if length or not saw_pixels:
raise MetadataReadError("Incomplete PNG image")
return records.found, frozenset(records.limited), width, height
else:
records.add(kind, data)
raise MetadataReadError("PNG chunk count exceeds inspection limits")
@@ -222,7 +319,7 @@ def _video_metadata(path: Path) -> _VideoInspection:
def read_inspection_file(path: Path) -> InspectionRead:
"""Read bounded metadata and header facts without decoding media pixels."""
try:
formats, carrier = {}, None
formats, carrier, limited = {}, None, frozenset()
format_name = width = height = None
if is_video_path(path):
video = _video_metadata(path)
@@ -235,7 +332,9 @@ def read_inspection_file(path: Path) -> InspectionRead:
with path.open("rb") as stream:
signature = stream.read(8)
if signature == _SIGNATURE:
metadata, width, height = _png_metadata(stream, path.stat().st_size)
metadata, limited, width, height = _png_metadata(
stream, path.stat().st_size
)
format_name = "PNG"
else:
stream.seek(0)
@@ -243,7 +342,7 @@ def read_inspection_file(path: Path) -> InspectionRead:
with Image.open(io.BytesIO(header)) as image:
metadata = dict(image.info)
format_name, (width, height) = image.format, image.size
result, size = {}, 0
result, role_size, limited = {}, 0, set(limited)
for key in _METADATA_KEYS:
if key not in metadata:
continue
@@ -253,12 +352,31 @@ def read_inspection_file(path: Path) -> InspectionRead:
continue
if not isinstance(value, str):
raise MetadataReadError("Malformed prompt metadata")
size += len(value.encode("utf-8", "surrogatepass"))
if size > MAX_METADATA_BYTES:
raise MetadataReadError("Metadata exceeds inspection limits")
size = len(value.encode("utf-8", "surrogatepass"))
if key == "workflow":
if size > WORKFLOW_METADATA_BYTES:
limited.add(key)
continue
else:
role_size += size
if role_size > ROLE_METADATA_BYTES:
raise MetadataReadError("Metadata exceeds inspection limits")
result[key] = value
formats.setdefault(key, "original-text")
return InspectionRead(result, formats, carrier, format_name, width, height)
if _ROLE_METADATA_KEYS.intersection(limited):
# A limited role record would vanish and let fallback evidence win.
raise MetadataReadError("Metadata exceeds inspection limits")
for key in limited:
formats[key] = "limited"
return InspectionRead(
result,
formats,
carrier,
format_name,
width,
height,
frozenset(limited),
)
except MetadataReadError:
raise
except (
@@ -282,8 +400,19 @@ def _side(status: str, text: str | None = None, source: str | None = None) -> di
return {"status": status, "text": text, "source": source}
def inspect_prompt_evidence(metadata: dict, saved: dict | None = None) -> dict:
"""Keep uncertainty visible; a saved association never replaces file evidence."""
def inspect_prompt_evidence(
metadata: dict,
saved: dict | None = None,
*,
profile: ResolverProfile = INSPECTION_PROFILE,
) -> dict:
"""Keep uncertainty visible; a saved association never replaces file evidence.
Inspection routes resolve with ``INSPECTION_PROFILE``: bounded work, not
graph size. Ingestion (scan, live monitor, admin import) calls the resolver
directly with its default profile. ``resolution`` carries the closed
``reason_code`` and, for a blocking node, ``node: {id, class_type}``.
"""
metadata = {
key: value for key, value in metadata.items() if key in _ROLE_METADATA_KEYS
}
@@ -291,6 +420,8 @@ def inspect_prompt_evidence(metadata: dict, saved: dict | None = None) -> dict:
"state": "unresolved",
"sampler_id": None,
"reason": "Prompt evidence is malformed",
"reason_code": "malformed",
"node": None,
}
if not metadata:
result = {role: _side("absent") for role in ("positive", "negative")}
@@ -311,6 +442,8 @@ def inspect_prompt_evidence(metadata: dict, saved: dict | None = None) -> dict:
"state": "absent",
"sampler_id": None,
"reason": "No parsed prompt metadata",
"reason_code": None,
"node": None,
},
}
canonical = dict(metadata)
@@ -329,7 +462,7 @@ def inspect_prompt_evidence(metadata: dict, saved: dict | None = None) -> dict:
},
"resolution": resolution,
}
resolved = resolve_prompt_pair(canonical)
resolved = resolve_prompt_pair(canonical, profile=profile)
state = resolved["state"]
result = {}
for role in ("positive", "negative"):
@@ -350,6 +483,8 @@ def inspect_prompt_evidence(metadata: dict, saved: dict | None = None) -> dict:
"state": state,
"sampler_id": resolved.get("sampler_id"),
"reason": resolved.get("reason", ""),
"reason_code": resolved.get("reason_code"),
"node": resolved.get("node"),
},
}
+418 -62
View File
@@ -14,10 +14,19 @@ Effective text uses a unique bounded execution record for traced paired nodes
when available; cached graph-only results remain explicitly inferred.
The resolver has no
ComfyUI, database, configuration or filesystem dependencies.
Two ``ResolverProfile`` values exist. ``DEFAULT_PROFILE`` (ingestion: scan, live
monitor, admin import) keeps the historical behavior exactly, including the
500-node gate, and only adds ``reason_code: None``. ``INSPECTION_PROFILE`` bounds
work instead of graph size: a per-candidate visit allowance, a total visit budget
shared by every candidate and attribution walk, a node ceiling backstop, and
bounds on built provenance and text. It also names why a result is not resolved
(``reason_code``, and ``node`` for the first unsupported node), and a prompt graph
without a supported sampler is ``unresolved`` (``no-sampler``), never ``absent``.
"""
import re
from dataclasses import dataclass
from dataclasses import dataclass, replace
from typing import Any, Dict, Mapping, Optional, Tuple
from .pair_execution import ExecutionEvidenceError, read_pair_execution
@@ -62,8 +71,109 @@ for _name in ("Text Multiline", "String Literal", "ShowText|pysssss"):
_DEFAULT_ADAPTERS[(_name, 0)] = OutputAdapter("text", "string")
@dataclass(frozen=True)
class ResolverProfile:
"""Resolution bounds. ``legacy_node_gate`` set means the historical code path.
``max_path_entries`` and ``max_built_chars`` bound memory across the whole
resolution (provenance entries and newly built text). ``max_scanned_units``
bounds linear work across the whole resolution: input keys validated or
scanned, characters stripped or joined, and provenance entries copied, each
charged before the work is done. ``None`` disables each bound.
``semantics`` and ``selection`` are reserved for later slices and stay off.
"""
name: str
per_candidate_visits: int
total_visits: int
max_depth: int
max_text_length: int
node_ceiling: Optional[int]
legacy_node_gate: Optional[int]
semantics: bool = False
selection: bool = False
max_path_entries: Optional[int] = None
max_built_chars: Optional[int] = None
max_scanned_units: Optional[int] = None
@property
def legacy(self) -> bool:
return self.legacy_node_gate is not None
DEFAULT_PROFILE = ResolverProfile(
name="default",
per_candidate_visits=500,
total_visits=500,
max_depth=64,
max_text_length=100_000,
node_ceiling=None,
legacy_node_gate=500,
)
INSPECTION_PROFILE = ResolverProfile(
name="inspection",
per_candidate_visits=2_000,
total_visits=20_000,
max_depth=64,
max_text_length=100_000,
node_ceiling=20_000,
legacy_node_gate=None,
max_path_entries=64 * 20_000,
max_built_chars=16 * 100_000,
# About four passes over the 2 MiB role-record budget.
max_scanned_units=8_000_000,
)
_SAMPLER_LIKE_INPUTS = ("latent_image", "guider", "positive", "negative")
_NO_SAMPLER_NAMES = 3
_CLASS_NAME_LENGTH = 64
class _Unresolved(Exception):
pass
"""A non-resolved outcome; ``code`` is the closed machine vocabulary."""
def __init__(
self, message: str, code: Optional[str] = None, node: Optional[dict] = None
) -> None:
super().__init__(message)
self.code = code
self.node = node
def _node_ref(node_id: str, class_type: Any) -> dict:
name = class_type[:_CLASS_NAME_LENGTH] if isinstance(class_type, str) else None
return {"id": node_id[:_CLASS_NAME_LENGTH], "class_type": name}
class _Budget:
"""Visits shared by every candidate, plus a per-candidate allowance.
With ``total`` equal to the allowance (the default profile) this is exactly
the historical single counter, so the reset per sampler is a no-op there.
"""
def __init__(self, total: int) -> None:
self.total = total
self.allowance = total
def reset(self, allowance: int) -> None:
self.allowance = allowance
@property
def remaining(self) -> int:
return min(self.total, self.allowance)
def spend(self) -> None:
# Checked before decrementing, so an exhausted allowance costs no total.
if self.total <= 0 or self.allowance <= 0:
raise _Unresolved("Resolution visit limit reached", "work-limit")
self.total -= 1
self.allowance -= 1
def charge(self, cost: int) -> None:
"""Charge a memoized subtree's visit cost to this candidate only."""
if cost > self.allowance:
raise _Unresolved("Resolution visit limit reached", "work-limit")
self.allowance -= cost
def _link(value: Any) -> bool:
@@ -85,6 +195,8 @@ def _result(
sampler_id: Optional[str] = None,
reason: str = "",
candidates: Optional[list] = None,
reason_code: Optional[str] = None,
node: Optional[dict] = None,
) -> Dict[str, Any]:
return {
"state": state,
@@ -93,9 +205,19 @@ def _result(
"sampler_id": sampler_id,
"reason": reason,
"candidates": candidates or [],
"reason_code": reason_code,
"node": node,
}
def _legacy_result(result: Dict[str, Any]) -> Dict[str, Any]:
"""The default profile's wire: today's keys plus ``reason_code: None``."""
legacy = {key: value for key, value in result.items() if key != "node"}
legacy["reason_code"] = None
legacy["candidates"] = [_legacy_result(item) for item in result["candidates"]]
return legacy
class _GraphResolver:
def __init__(
self,
@@ -105,10 +227,32 @@ class _GraphResolver:
max_depth: int,
max_text_length: int,
metadata: Optional[dict] = None,
*,
budget: Optional[_Budget] = None,
profile: ResolverProfile = DEFAULT_PROFILE,
) -> None:
# ``max_visits`` is the per-candidate allowance; ``profile`` supplies only
# the path selection and whole-resolution bounds. ``resolve_text_input``
# relies on the legacy default profile here.
self.graph = graph
self.adapters = {**_DEFAULT_ADAPTERS, **adapters}
self.budget = max_visits
self.per_candidate_visits = max_visits
self.budget = budget if budget is not None else _Budget(max_visits)
self.budget.reset(max_visits)
self.legacy = profile.legacy
self.path_entries = profile.max_path_entries
self.built_chars = profile.max_built_chars
self.scanned = profile.max_scanned_units
self.checked: dict = {}
# Unfolded visit cost of each memo entry (inspection): a memo hit charges
# it to the current candidate, so each candidate's outcome is independent
# of tracing order while the shared work is done only once.
self.costs: dict = {}
# Height of each memo entry (deepest depth reached below it, relative to
# its own depth): a hit raises ``depth-limit`` exactly where a fresh walk
# from the reusing depth would, so depth too is independent of order.
self.heights: dict = {}
self.deepest = 0
self.max_depth = max_depth
self.max_text_length = max_text_length
self.memo: dict = {}
@@ -117,46 +261,99 @@ class _GraphResolver:
self.text_sources: dict = {}
def node(self, node_id: str) -> Tuple[str, dict]:
if self.legacy:
return self.checked_node(node_id)
if node_id not in self.checked:
try:
self.checked[node_id] = self.checked_node(node_id)
except _Unresolved as error:
if error.code == "work-limit":
raise
self.checked[node_id] = error
checked = self.checked[node_id]
if isinstance(checked, _Unresolved):
raise _Unresolved(str(checked), checked.code, checked.node)
return checked
def checked_node(self, node_id: str) -> Tuple[str, dict]:
node = self.graph.get(node_id)
if not isinstance(node, dict) or not isinstance(node.get("inputs"), dict):
raise _Unresolved("Missing or malformed graph node")
raise _Unresolved("Missing or malformed graph node", "malformed")
self.scan(len(node["inputs"]))
if any(not isinstance(key, str) for key in node["inputs"]):
raise _Unresolved("Malformed node input names")
raise _Unresolved("Malformed node input names", "malformed")
return str(node.get("class_type", "")), node["inputs"]
def scan(self, units: int) -> None:
"""Charge linear work before doing it (inspection; no-op when legacy)."""
if self.scanned is not None:
if units > self.scanned:
self.scanned = -1
raise _Unresolved("Resolution work limit reached", "work-limit")
self.scanned -= units
def unsupported(self, message: str, node_id: str) -> _Unresolved:
node = self.graph.get(node_id)
class_type = node.get("class_type") if isinstance(node, dict) else None
if isinstance(class_type, str):
class_type = class_type[:_CLASS_NAME_LENGTH]
return _Unresolved(message, "unsupported-node", _node_ref(node_id, class_type))
def bounded(self, text: str) -> str:
if len(text) > self.max_text_length:
raise _Unresolved("Resolved text exceeds the length limit")
raise _Unresolved("Resolved text exceeds the length limit", "text-limit")
return text
def built(self, text: str) -> str:
"""Charge newly built text against the profile's whole-resolution bound."""
if self.built_chars is not None:
self.built_chars -= len(text)
if self.built_chars < 0:
raise _Unresolved("Resolution work limit reached", "work-limit")
return text
def text(self, value: Any, kind: str, depth: int = 0) -> Tuple[str, list, str]:
if not self.legacy:
self.deepest = max(self.deepest, depth)
if depth >= self.max_depth:
raise _Unresolved("Resolution depth limit reached")
raise _Unresolved("Resolution depth limit reached", "depth-limit")
if isinstance(value, str) and kind == "string":
return self.bounded(value), [], value
if not _link(value):
raise _Unresolved("Expected a typed graph link or text input")
raise _Unresolved("Expected a typed graph link or text input", "malformed")
node_id, slot = str(value[0]), value[1]
key = (node_id, slot, kind)
if key in self.active:
raise _Unresolved("Cyclic prompt graph")
raise _Unresolved("Cyclic prompt graph", "malformed")
if key in self.memo:
if not self.legacy:
height = self.heights[key]
if depth + height >= self.max_depth:
raise _Unresolved("Resolution depth limit reached", "depth-limit")
self.budget.charge(self.costs[key])
self.deepest = max(self.deepest, depth + height)
return self.memo[key]
self.budget -= 1
if self.budget < 0:
raise _Unresolved("Resolution visit limit reached")
name, inputs = self.node(node_id)
self.active.add(key)
allowance = self.budget.allowance
outer_deepest = self.deepest
self.deepest = depth
try:
self.budget.spend()
name, inputs = self.node(node_id)
self.scan(len(inputs))
self.active.add(key)
if name == "Reroute" and slot == 0:
keys = [key for key in ("value", "input", "in") if key in inputs]
if len(keys) != 1:
raise _Unresolved("Reroute input is not uniquely identifiable")
raise _Unresolved(
"Reroute input is not uniquely identifiable", "malformed"
)
text, path, storage_text = self.text(inputs[keys[0]], kind, depth + 1)
elif (name, slot) in self.adapters:
adapter = self.adapters[(name, slot)]
if not isinstance(adapter, OutputAdapter) or adapter.kind != kind:
raise _Unresolved("Output slot has an incompatible type")
raise _Unresolved(
"Output slot has an incompatible type", "malformed"
)
evidence = (
self.execution_record(node_id, inputs)
if name == "PromptManagerPair"
@@ -186,11 +383,20 @@ class _GraphResolver:
after, after_path, _ = self.text(
inputs.get("append_text", ""), "string", depth + 1
)
text = self.bounded(
" ".join(
part
for part in (before.strip(), text, after.strip())
if part
self.scan(
len(before)
+ len(after)
+ len(before_path)
+ len(path)
+ len(after_path)
)
text = self.built(
self.bounded(
" ".join(
part
for part in (before.strip(), text, after.strip())
if part
)
)
)
path = before_path + path + after_path
@@ -212,38 +418,56 @@ class _GraphResolver:
str(inputs.get("clean_whitespace", "true")).lower() == "true"
)
if not keys or not isinstance(delimiter, str):
raise _Unresolved("Malformed concatenation")
raise _Unresolved("Malformed concatenation", "malformed")
parts, path, size = [], [], 0
for input_name in keys:
part, followed, _ = self.text(
inputs.get(input_name, ""), "string", depth + 1
)
# Charged before stripping: stripped-away text is work too.
self.scan(len(part) + len(followed))
if clean:
part = part.strip()
path.extend(followed)
if part or name == "StringConcatenate":
size += len(part) + (len(delimiter) if parts else 0)
if size > self.max_text_length:
raise _Unresolved("Resolved text exceeds the length limit")
raise _Unresolved(
"Resolved text exceeds the length limit", "text-limit"
)
parts.append(part)
text = delimiter.join(parts)
text = self.built(delimiter.join(parts))
storage_text = text
else:
raise _Unresolved("Unsupported node semantics or output slot")
raise self.unsupported(
"Unsupported node semantics or output slot", node_id
)
# Memoized paths stay bounded even for diamond-shaped string graphs.
self.scan(len(path) + 1)
path = [{"node_id": node_id, "output_slot": slot}] + path
if len(path) > self.max_depth * max(1, len(self.graph)):
raise _Unresolved("Provenance path limit reached")
if self.legacy:
path_limit = self.max_depth * max(1, len(self.graph))
else:
path_limit = self.max_depth * max(1, self.per_candidate_visits)
if self.path_entries is not None:
self.path_entries -= len(path)
if len(path) > path_limit or (
self.path_entries is not None and self.path_entries < 0
):
raise _Unresolved("Provenance path limit reached", "work-limit")
self.memo[key] = (text, path, storage_text)
self.costs[key] = allowance - self.budget.allowance
self.heights[key] = self.deepest - depth
return text, path, storage_text
finally:
self.deepest = max(outer_deepest, self.deepest)
self.active.discard(key)
def execution_record(self, node_id: str, inputs: dict) -> Optional[dict]:
try:
record = read_pair_execution(self.metadata, node_id)
except ExecutionEvidenceError as error:
raise _Unresolved(str(error)) from error
raise _Unresolved(str(error), "malformed") from error
if record is None:
return None
for name in ("text", "negative_text", "prepend_text", "append_text"):
@@ -273,7 +497,8 @@ class _GraphResolver:
expected = expected.strip()
if expected != record[name]:
raise _Unresolved(
"Execution evidence does not match recorded node inputs"
"Execution evidence does not match recorded node inputs",
"malformed",
)
return record
@@ -296,6 +521,14 @@ class _GraphResolver:
def pair(self, sampler_id: str) -> dict:
positive = negative = None
self.budget.reset(self.per_candidate_visits)
if not self.legacy and self.budget.total <= 0:
return _result(
"unresolved",
sampler_id=sampler_id,
reason="Resolution visit limit reached",
reason_code="work-limit",
)
try:
_, inputs = self.node(sampler_id)
# A second sampler in the latent ancestry denotes a multistage run;
@@ -315,31 +548,50 @@ class _GraphResolver:
negative = self.side(inputs["negative"], sampler_id)
return _result("resolved", positive, negative, sampler_id)
except _Unresolved as error:
return _result("unresolved", positive, negative, sampler_id, str(error))
return _result(
"unresolved",
positive,
negative,
sampler_id,
str(error),
reason_code=error.code,
node=error.node,
)
def latent_source(self, value: Any, seen: set, depth: int) -> None:
if not _link(value) or depth >= self.max_depth:
raise _Unresolved("Unknown latent ancestry")
code = "malformed" if not _link(value) else "depth-limit"
raise _Unresolved("Unknown latent ancestry", code)
node_id, slot = str(value[0]), value[1]
if node_id in seen or len(seen) >= max(0, self.budget):
raise _Unresolved("Cyclic or excessive latent ancestry")
if self.legacy:
if node_id in seen or len(seen) >= max(0, self.budget.remaining):
raise _Unresolved("Cyclic or excessive latent ancestry")
elif node_id in seen:
raise _Unresolved("Cyclic or excessive latent ancestry", "malformed")
else:
self.budget.spend()
seen.add(node_id)
name, inputs = self.node(node_id)
if name in _SAMPLERS:
# ``multistage-differs`` is reserved for when stage texts are compared.
raise _Unresolved("Multistage sampler ancestry is ambiguous")
if name in ("EmptyLatentImage", "EmptySD3LatentImage") and slot == 0:
return
if name == "Reroute" and slot == 0 and "value" in inputs:
self.latent_source(inputs["value"], seen, depth + 1)
return
raise _Unresolved("Unsupported latent ancestry")
raise self.unsupported("Unsupported latent ancestry", node_id)
def anchor(self, node_id: str) -> str:
seen = set()
slot = 0
while len(seen) < self.max_depth and len(seen) < self.budget:
while len(seen) < self.max_depth and (
not self.legacy or len(seen) < self.budget.remaining
):
if node_id in seen:
raise _Unresolved("Cyclic output ancestry")
raise _Unresolved("Cyclic output ancestry", "malformed")
if not self.legacy:
self.budget.spend()
seen.add(node_id)
name, inputs = self.node(node_id)
if name in _SAMPLERS and slot == 0:
@@ -353,19 +605,28 @@ class _GraphResolver:
}.get(name)
value = inputs.get(key) if key and slot == 0 else None
if not _link(value):
raise _Unresolved("Unsupported output ancestry")
raise self.unsupported("Unsupported output ancestry", node_id)
node_id, slot = str(value[0]), value[1]
raise _Unresolved("Output ancestry limit reached")
code = "depth-limit" if len(seen) >= self.max_depth else "work-limit"
raise _Unresolved("Output ancestry limit reached", code)
def _parameters(value: Any, max_text_length: int) -> dict:
if not isinstance(value, str) or not value:
return _result("absent", reason="No A1111 parameters")
if len(value) > max_text_length:
return _result("unresolved", reason="Parameters exceed the length limit")
return _result(
"unresolved",
reason="Parameters exceed the length limit",
reason_code="text-limit",
)
footers = list(re.finditer(r"(?m)^Steps:\s*\d+\s*,[^\n]*$", value))
if len(footers) != 1 or value[footers[0].end() :].strip():
return _result("unresolved", reason="A1111 settings footer is not unique")
return _result(
"unresolved",
reason="A1111 settings footer is not unique",
reason_code="malformed",
)
body = value[: footers[0].start()].rstrip("\r\n")
delimiter = "\nNegative prompt:"
if body.count(delimiter) > 1:
@@ -396,49 +657,136 @@ def _parameters(value: Any, max_text_length: int) -> dict:
return _result("resolved", positive_side, negative_side)
def _profile_with(
profile: ResolverProfile,
max_visits: Optional[int],
max_depth: Optional[int],
max_text_length: Optional[int],
) -> ResolverProfile:
"""Explicit ``max_*`` keywords override the profile they are given with."""
changes: Dict[str, Any] = {}
if max_visits is not None:
changes["per_candidate_visits"] = max_visits
if profile.legacy: # one historical number: gate, total and allowance
changes.update(total_visits=max_visits, legacy_node_gate=max_visits)
if max_depth is not None:
changes["max_depth"] = max_depth
if max_text_length is not None:
changes["max_text_length"] = max_text_length
return replace(profile, **changes) if changes else profile
def _sampler_like_classes(graph: dict) -> list:
"""Distinct classes, in graph order, of nodes shaped like samplers/guiders."""
names: list = []
for node in graph.values():
if not isinstance(node, dict) or not isinstance(node.get("inputs"), dict):
continue
name = node.get("class_type")
if not isinstance(name, str) or not name or name in names:
continue
if any(_link(node["inputs"].get(key)) for key in _SAMPLER_LIKE_INPUTS):
names.append(name[:_CLASS_NAME_LENGTH])
if len(names) == _NO_SAMPLER_NAMES:
break
return names
def resolve_prompt_pair(
metadata: Any,
*,
output_node_id: Optional[str] = None,
adapters: Optional[Mapping[Tuple[str, int], OutputAdapter]] = None,
max_visits: int = 500,
max_depth: int = 64,
max_text_length: int = 100_000,
profile: ResolverProfile = DEFAULT_PROFILE,
max_visits: Optional[int] = None,
max_depth: Optional[int] = None,
max_text_length: Optional[int] = None,
) -> Dict[str, Any]:
"""Resolve parsed metadata; a present API graph is authoritative over text.
Accepts ``{"prompt": api_graph, "parameters": optional_a1111_text}`` or a
direct API graph. An optional saver/sampler node ID anchors branch selection.
Unanchored multiple samplers are ambiguous, even when their texts are equal.
``profile`` selects the bounds; explicit ``max_*`` keywords override it.
"""
profile = _profile_with(profile, max_visits, max_depth, max_text_length)
result = _resolve(metadata, output_node_id, adapters, profile)
return _legacy_result(result) if profile.legacy else result
def _limit_result(profile: ResolverProfile, graph: dict) -> Optional[Dict[str, Any]]:
if profile.legacy:
if (
len(graph) > profile.legacy_node_gate
or profile.max_depth <= 0
or profile.max_text_length <= 0
):
return _result("unresolved", reason="Resolution limit reached")
return None
if profile.node_ceiling is not None and len(graph) > profile.node_ceiling:
return _result(
"unresolved",
reason="Prompt graph exceeds the inspection node ceiling",
reason_code="graph-ceiling",
)
if profile.max_depth <= 0:
return _result(
"unresolved", reason="Resolution limit reached", reason_code="depth-limit"
)
if profile.max_text_length <= 0:
return _result(
"unresolved", reason="Resolution limit reached", reason_code="text-limit"
)
return None
def _resolve(
metadata: Any,
output_node_id: Optional[str],
adapters: Optional[Mapping[Tuple[str, int], OutputAdapter]],
profile: ResolverProfile,
) -> Dict[str, Any]:
if not isinstance(metadata, dict) or not metadata:
return _result("absent", reason="No parsed prompt metadata")
if "prompt" in metadata:
graph = metadata["prompt"]
elif "parameters" in metadata:
return _parameters(metadata["parameters"], max_text_length)
return _parameters(metadata["parameters"], profile.max_text_length)
else:
graph = metadata
if not isinstance(graph, dict) or not graph:
return _result("unresolved", reason="API graph is missing or malformed")
if len(graph) > max_visits or max_depth <= 0 or max_text_length <= 0:
return _result("unresolved", reason="Resolution limit reached")
return _result(
"unresolved",
reason="API graph is missing or malformed",
reason_code="malformed",
)
limited = _limit_result(profile, graph)
if limited is not None:
return limited
normalized = {str(node_id): node for node_id, node in graph.items()}
if len(normalized) != len(graph):
return _result("unresolved", reason="Graph node IDs are not unique")
return _result(
"unresolved",
reason="Graph node IDs are not unique",
reason_code="malformed",
)
resolver = _GraphResolver(
normalized,
adapters or {},
max_visits,
max_depth,
max_text_length,
profile.per_candidate_visits,
profile.max_depth,
profile.max_text_length,
metadata if "prompt" in metadata else None,
budget=_Budget(profile.total_visits),
profile=profile,
)
if output_node_id is not None:
try:
sampler_ids = [resolver.anchor(str(output_node_id))]
except _Unresolved as error:
return _result("unresolved", reason=str(error))
return _result(
"unresolved", reason=str(error), reason_code=error.code, node=error.node
)
else:
sampler_ids = [
node_id
@@ -448,16 +796,7 @@ def resolve_prompt_pair(
and node["class_type"] in _SAMPLERS
]
if not sampler_ids:
has_pair_inputs = any(
isinstance(node, dict)
and isinstance(node.get("inputs"), dict)
and ("positive" in node["inputs"] or "negative" in node["inputs"])
for node in normalized.values()
)
return _result(
"unresolved" if has_pair_inputs else "absent",
reason="No supported sampler pair found",
)
return _no_sampler(normalized, profile)
if len(sampler_ids) > 1:
return _result(
"ambiguous",
@@ -467,6 +806,23 @@ def resolve_prompt_pair(
return resolver.pair(sampler_ids[0])
def _no_sampler(graph: dict, profile: ResolverProfile) -> Dict[str, Any]:
reason = "No supported sampler pair found"
if profile.legacy:
has_pair_inputs = any(
isinstance(node, dict)
and isinstance(node.get("inputs"), dict)
and ("positive" in node["inputs"] or "negative" in node["inputs"])
for node in graph.values()
)
return _result("unresolved" if has_pair_inputs else "absent", reason=reason)
# A prompt graph is a record: its prompt is unresolved, never "not recorded".
seen = _sampler_like_classes(graph)
if seen:
reason = f"{reason} (saw {', '.join(seen)})"
return _result("unresolved", reason=reason, reason_code="no-sampler")
def resolve_text_input(
graph: Any,
node_id: str,
+2 -2
View File
@@ -469,8 +469,9 @@
function graphFacts(graph, source, resolution) {
if (!object(graph)) return factSet("unresolved", source);
// Graph size is not a bound: each lookup is bounded by its MAX_VISITS seen-set,
// and the server's node ceiling arrives as a `limited` projection.
const entries = Object.entries(graph);
if (entries.length > MAX_VISITS) return factSet("unresolved", source, "projection-limited");
if (entries.some(([, node]) => !object(node) || !object(node.inputs) || typeof node.class_type !== "string")) return factSet("unresolved", source);
const candidates = entries.filter(([, node]) => GENERATION_SAMPLERS.has(node.class_type));
const selected = candidates.find(([id]) => typeof resolution?.sampler_id === "string" && id === resolution.sampler_id);
@@ -560,7 +561,6 @@
if (record.status === "limited") return factSet("unresolved", source, "projection-limited");
if (record.status !== "parsed") return factSet("unresolved", source);
if (source === "workflow") {
if (record.value?.nodes?.length > MAX_VISITS) return factSet("unresolved", source, "projection-limited");
if (!validFactWorkflow(record.value)) return factSet("unresolved", source);
return graphFacts(workflowToGraph(record.value, true), source, null);
}
+5 -1
View File
@@ -7,11 +7,14 @@ import { createWorkspaceIcon } from './icons.mjs';
const RAW_KEYS = ['prompt', 'workflow', 'parameters', 'comfyassets_prompt_pairs'];
const RAW_LIMIT = 'Raw metadata exceeds the response limit. Download the original file to inspect it externally.';
const CONTAINER_WORKFLOW = 'Original workflow JSON text is unavailable for this container record. Download metadata or the original file.';
const LIMITED_WORKFLOW = 'Workflow exceeds inspection limits. Download the original file to inspect it externally.';
const REPRESENTATIONS = {
'original-text': 'Original embedded text',
'reserialized-json': 'Reserialized container JSON; numeric spelling/precision may differ',
invalid: 'Invalid container record; text is unavailable',
limited: 'Record exceeds inspection limits; text is unavailable',
};
const TEXTLESS = new Set(['invalid', 'limited']);
const FACTS = [['seed', 'Seed'], ['steps', 'Steps'], ['cfg', 'CFG'], ['sampler', 'Sampler'], ['scheduler', 'Scheduler'], ['model', 'Model']];
const FACT_SOURCES = {'api-graph': 'Embedded API configuration', workflow: 'Saved workflow configuration', 'embedded-parameters': 'Embedded generation parameters'};
@@ -33,7 +36,7 @@ export function inspectionDownloads(record) {
if (Object.hasOwn(REPRESENTATIONS, metadata.record_formats?.[key])) {
record_formats[key] = metadata.record_formats[key];
representations[key] = REPRESENTATIONS[record_formats[key]];
if (record_formats[key] !== 'invalid' && typeof metadata.raw?.[key] === 'string') raw[key] = metadata.raw[key];
if (!TEXTLESS.has(record_formats[key]) && typeof metadata.raw?.[key] === 'string') raw[key] = metadata.raw[key];
}
}
const envelope = {inspection_version:1, identity:record.identity, file:record.file, prompts:record.prompts,
@@ -42,6 +45,7 @@ export function inspectionDownloads(record) {
scope:'Supported embedded records only; one selected container carrier for video. Other metadata and private carrier text are excluded.'}};
result.metadata = {...result.metadata, available:true, reason:null, bodyText:JSON.stringify(envelope, null, 2)};
if (record_formats.workflow === 'reserialized-json') result.workflow.reason = CONTAINER_WORKFLOW;
else if (record_formats.workflow === 'limited') result.workflow.reason = LIMITED_WORKFLOW;
else if (record_formats.workflow === 'original-text' && metadata.workflow_graph?.status === 'parsed' && typeof raw.workflow === 'string') {
result.workflow = {...result.workflow, available:true, reason:null, bodyText:raw.workflow};
} else result.workflow.reason = 'A valid original workflow JSON record is not available in this file.';
+24 -8
View File
@@ -4,6 +4,11 @@ import { promptRoles } from './prompt-blocks.mjs';
const SOURCE_CHANGED = 'The source changed. Refresh the image collection before inspecting this file.';
const RECORD_KEYS = ['prompt', 'workflow', 'parameters', 'comfyassets_prompt_pairs'];
const FACT_FIELDS = ['seed', 'steps', 'cfg', 'sampler', 'scheduler', 'model'];
// `invalid` and `limited` records carry no text. Only a workflow can be `limited`:
// role records over their budget fail the whole read on the server.
const RECORD_FORMATS = ['original-text', 'reserialized-json', 'invalid', 'limited'];
const TEXTLESS_FORMATS = ['invalid', 'limited'];
const REASON_CODES = ['work-limit', 'depth-limit', 'text-limit', 'graph-ceiling', 'unsupported-node', 'runtime-value', 'composite', 'no-sampler', 'malformed', 'multistage-differs'];
const object = value => value !== null && typeof value === 'object' && !Array.isArray(value);
const validScope = value => typeof value === 'string' && /^[a-f0-9]{64}$/.test(value) && value.length === 64;
@@ -103,30 +108,43 @@ function metadataFor(value) {
(value.parameters !== null && typeof value.parameters !== 'string')) throw new TypeError('Invalid inspection metadata');
const raw = {}, formats = {};
for (const [key, format] of Object.entries(value.record_formats)) {
if (!RECORD_KEYS.includes(key) || !['original-text', 'reserialized-json', 'invalid'].includes(format)) throw new TypeError('Invalid inspection record format');
if (!RECORD_KEYS.includes(key) || !RECORD_FORMATS.includes(format) || (format === 'limited' && key !== 'workflow')) throw new TypeError('Invalid inspection record format');
formats[key] = format;
}
for (const [key, text] of Object.entries(value.raw)) {
if (!RECORD_KEYS.includes(key) || typeof text !== 'string' || !Object.hasOwn(formats, key) || formats[key] === 'invalid') throw new TypeError('Invalid inspection raw record');
if (!RECORD_KEYS.includes(key) || typeof text !== 'string' || !Object.hasOwn(formats, key) || TEXTLESS_FORMATS.includes(formats[key])) throw new TypeError('Invalid inspection raw record');
raw[key] = text;
}
if (value.parameters !== (raw.parameters ?? null)) throw new TypeError('Invalid inspection parameters projection');
const prompt = graphFor(value.prompt_graph), workflow = graphFor(value.workflow_graph);
const wholeRawOmitted = value.status === 'limited';
if (wholeRawOmitted && (Object.keys(raw).length || value.parameters !== null)) throw new TypeError('Invalid atomic raw omission');
if (!wholeRawOmitted && Object.entries(formats).some(([key, format]) => format !== 'invalid' && !Object.hasOwn(raw, key))) throw new TypeError('Missing represented raw record');
const invalidGraphOmitted = [['prompt', prompt], ['workflow', workflow]].some(([key, graph]) => formats[key] === 'invalid' && graph.status === 'limited');
if (!wholeRawOmitted && Object.entries(formats).some(([key, format]) => !TEXTLESS_FORMATS.includes(format) && !Object.hasOwn(raw, key))) throw new TypeError('Missing represented raw record');
// The response budget omits the workflow graph, then the prompt graph, then all raw.
// An invalid record's graph is otherwise malformed, so a limited one marks omission.
const promptGraphOmitted = formats.prompt === 'invalid' && prompt.status === 'limited';
for (const [key, graph] of [['prompt', prompt], ['workflow', workflow]]) {
const hasRecord = Object.hasOwn(formats, key);
if ((formats[key] === 'invalid' && !['malformed', 'limited'].includes(graph.status)) ||
(formats[key] === 'limited' && graph.status !== 'limited') ||
hasRecord !== (graph.status !== 'absent')) throw new TypeError('Invalid graph representation provenance');
if (hasRecord && (wholeRawOmitted || invalidGraphOmitted) && graph.status !== 'limited') throw new TypeError('Invalid atomic graph omission');
if (hasRecord && (wholeRawOmitted || (promptGraphOmitted && key === 'workflow')) && graph.status !== 'limited') throw new TypeError('Invalid atomic graph omission');
}
if ((value.status === 'limited') !== (value.omitted_reason === 'response-size-limit')) throw new TypeError('Invalid metadata omission status');
if (['absent', 'unreadable'].includes(value.status) && (Object.keys(raw).length || Object.keys(formats).length || prompt.status !== 'absent' || workflow.status !== 'absent')) throw new TypeError('Invalid unavailable metadata');
return { status: value.status, omitted_reason: value.omitted_reason, raw, record_formats: formats, prompt_graph: prompt, workflow_graph: workflow, parameters: value.parameters };
}
function resolutionFor(value) {
if (!object(value) || !['resolved', 'absent', 'ambiguous', 'unresolved'].includes(value.state) ||
(value.sampler_id !== null && (typeof value.sampler_id !== 'string' || !value.sampler_id)) || typeof value.reason !== 'string') throw new TypeError('Invalid inspection resolution');
const code = value.reason_code ?? null, node = value.node ?? null;
// Codes explain a non-resolved outcome; a blocking node is named only with its code.
if ((code !== null && (!REASON_CODES.includes(code) || ['resolved', 'absent'].includes(value.state))) ||
(node !== null && (code === null || !object(node) || typeof node.id !== 'string' || (node.class_type !== null && typeof node.class_type !== 'string')))) throw new TypeError('Invalid inspection resolution reason');
return value;
}
function libraryFor(value) {
if (!object(value) || !['linked', 'none', 'ambiguous', 'unavailable', 'limited'].includes(value.status) || value.scope !== 'saved-library' ||
!Array.isArray(value.tags) || value.tags.some(tag => typeof tag !== 'string') || ![null, 'pair', 'positive'].includes(value.metadata_scope)) throw new TypeError('Invalid inspection library');
@@ -163,9 +181,7 @@ export function projectInspectionPayload(result, identity, download, extractFact
generation: unavailableFacts(), historyTarget: null, download,
};
if (legacy) return record;
const resolution = result.resolution;
if (!object(resolution) || !['resolved', 'absent', 'ambiguous', 'unresolved'].includes(resolution.state) ||
(resolution.sampler_id !== null && (typeof resolution.sampler_id !== 'string' || !resolution.sampler_id)) || typeof resolution.reason !== 'string') throw new TypeError('Invalid inspection resolution');
const resolution = resolutionFor(result.resolution);
record.file = fileFor(result.file);
record.metadata = metadataFor(result.metadata);
record.library = libraryFor(result.library);