Compare commits
16
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
03a8ccc10e | ||
|
|
70a418ecac | ||
|
|
8890c56c30 | ||
|
|
2fcbf5389d | ||
|
|
20c8e9eef9 | ||
|
|
0bdb33230f | ||
|
|
436ae95cda | ||
|
|
da4d6cdc84 | ||
|
|
8e014c0ebe | ||
|
|
8b922c6b64 | ||
|
|
98a8475d10 | ||
|
|
99e4d09419 | ||
|
|
600a479f0b | ||
|
|
cc6e7f9ec0 | ||
|
|
523c136455 | ||
|
|
08e77de05f |
+44
-9
@@ -200,10 +200,10 @@ loading a workflow or queueing generation.
|
||||
|
||||
## Durable execution ledger
|
||||
|
||||
`npm run --prefix tools/vendor test:executions` adds ten real execution cases
|
||||
(five each in Classic and Nodes 2.0) to the existing 78 stock host cases and six
|
||||
pack compatibility cases. Run these suites serially because they share report
|
||||
cleanup and owned host resources. Use the same pinned disposable `PM_COMFY_ROOT`,
|
||||
`npm run --prefix tools/vendor test:executions` runs 26 real execution cases
|
||||
(13 each in Classic and Nodes 2.0, retaining the original ten ledger cases),
|
||||
alongside the existing 78 stock host cases and six pack compatibility cases. Run
|
||||
these suites serially because they share report cleanup and owned host resources. Use the same pinned disposable `PM_COMFY_ROOT`,
|
||||
`PM_TEST_PYTHON` and Playwright browser installation as `test:comfyui`:
|
||||
|
||||
```sh
|
||||
@@ -218,7 +218,11 @@ server. Each renderer worker launches a CPU/offline host with independent
|
||||
configuration, database, user and output directories. The shared launcher installs
|
||||
tracked snapshots of PromptManager and a tiny temporary Git-tracked fixture pack.
|
||||
Only that owned test pack provides cached output, deliberate failure, an
|
||||
interruptible queue gate and gate-control routes; production has no test nodes or
|
||||
interruptible queue gate, deterministic CLIP/model/latent inputs and control routes.
|
||||
It replaces external sampler inference and trigger injection only inside the
|
||||
disposable host; real registered PromptManager classes, queue, executor and cache
|
||||
remain active. Its cache control resets the real executor on its worker to exercise
|
||||
Classic, LRU, RAM-pressure and disabled caching; production has no test nodes or
|
||||
endpoints. Cleanup always releases the gate, stops the host and removes both
|
||||
owned directories. Logs and Playwright failures appear under
|
||||
`tools/vendor/test-results/executions/` and the existing CI artifact upload.
|
||||
@@ -230,8 +234,10 @@ removal before dequeue. Actual host history is checked alongside finite polling
|
||||
of SQLite after the background writer. The read helper uses SQLite read-only
|
||||
mode, exact UUID job/session filters and a 256-row limit; absent ledger schema is
|
||||
explicit, while missing/corrupt databases and other read errors fail the suite.
|
||||
Sensitive sentinels in graph inputs, extra data and error messages must never
|
||||
appear in any stored row.
|
||||
Sensitive diagnostic and extra-data sentinels must never appear in lifecycle
|
||||
rows. Exact prompt fields are deliberately persisted only in context snapshots.
|
||||
The same read-only helper calls the internal `get_execution_contexts` reader for
|
||||
requested run IDs and inspects their snapshot identities and current usage counts.
|
||||
|
||||
For a meaningful historical failure, create a separate disposable detached
|
||||
checkout at `fdba6fd0126a8a40cbd6f3e50ed4c5de8991efd4` and pass its path to
|
||||
@@ -253,5 +259,34 @@ before restarting an existing deployment.
|
||||
|
||||
The ledger records only new dequeued jobs and observed outcomes. Existing
|
||||
counters, prompts, pairs and image links are preserved, with no reconstructed old
|
||||
runs. Prompt contexts, output associations and the chronological timeline remain
|
||||
future work under #209. Deploying the Python changes requires a ComfyUI restart.
|
||||
runs. Schema v2 migrates automatically when the configured database opens, adding
|
||||
immutable prompt-context tables without backfilling old runs. The internal reader
|
||||
returns `not_captured` for preexisting runs; a new capture can instead be complete,
|
||||
partial, ambiguous or unavailable. Keep the existing database backup/restore
|
||||
procedure: restored history changes storage epoch, and old library navigation
|
||||
hints must never be rebound to unrelated current rows.
|
||||
|
||||
Context cases verify A,A,B as three jobs/two combinations/one positive snapshot
|
||||
when every positive field matches; all five registered PM variants; exact raw
|
||||
whitespace and Unicode; known empty negatives; Text negative roles and crossed
|
||||
pairs; fully cached repeats after trigger configuration changes; successful source
|
||||
text before downstream failure/interruption; unknown cached dynamic raw; unsupported
|
||||
transforms and duplicate list invocation ambiguity. Fresh values use
|
||||
`runtime_return`, cached effective STRING outputs use `host_cache_output`, and
|
||||
submitted literals use graph inference. SQL NULL means unknown; empty text is
|
||||
known. Text is never trimmed or normalized by capture. Cache availability and
|
||||
inferred sampler roles prove neither sampler execution nor consumption, and list
|
||||
indices do not identify generated images. External trigger configuration changes
|
||||
must not rewrite the effective text of an existing cached result. The cache matrix
|
||||
also checks that positive/pair usage increments once per dequeued job while a
|
||||
negative-only component retains its existing zero standalone-use count.
|
||||
|
||||
Fault injection for restore fencing, inherited async capability revocation, queued
|
||||
finalization, epoch and generation revocation, provider-local cache entries, eviction/reset, malformed or
|
||||
oversized payloads and missing host interfaces lives in the focused Python
|
||||
`test_execution*.py` suites. These use disposable storage and mocked host interfaces;
|
||||
they are distinct from the pinned real executor/cache checks above. No output-file
|
||||
associations or chronological timeline are introduced. Deploying the Python changes
|
||||
requires a ComfyUI restart. After restart, queue three new jobs A,A,B with identical
|
||||
positive fields and two negative recipes; existing counters should advance as usual,
|
||||
and internal context inspection should find three attempts and two combinations.
|
||||
|
||||
@@ -0,0 +1,406 @@
|
||||
"""Owned execution-context v1 tables within execution-history schema v2.
|
||||
|
||||
Content IDs hash exact scalar values; evidence JSON contains metadata only.
|
||||
All validation scans use cursors and bounded per-row payloads.
|
||||
"""
|
||||
|
||||
import json
|
||||
import sqlite3
|
||||
|
||||
from .execution_history_schema import _compact, _uuid_check
|
||||
|
||||
try:
|
||||
from ..utils.execution_contexts import (
|
||||
BUNDLE_STATES,
|
||||
CAPTURE_REASONS,
|
||||
EVIDENCE_SOURCES,
|
||||
UNKNOWN_REASONS,
|
||||
MAX_CONTEXT_BYTES,
|
||||
ContextBundle,
|
||||
PromptSnapshot,
|
||||
RecipeContext,
|
||||
TextEvidence,
|
||||
validate_bundle,
|
||||
content_digest,
|
||||
text_size,
|
||||
)
|
||||
except ImportError:
|
||||
from utils.execution_contexts import (
|
||||
BUNDLE_STATES,
|
||||
CAPTURE_REASONS,
|
||||
EVIDENCE_SOURCES,
|
||||
UNKNOWN_REASONS,
|
||||
MAX_CONTEXT_BYTES,
|
||||
ContextBundle,
|
||||
PromptSnapshot,
|
||||
RecipeContext,
|
||||
TextEvidence,
|
||||
validate_bundle,
|
||||
content_digest,
|
||||
text_size,
|
||||
)
|
||||
|
||||
|
||||
FIELDS = ("raw", "effective", "prepend", "append")
|
||||
TABLE_COLUMNS = {
|
||||
"execution_context_texts": (
|
||||
("text_id", "TEXT", 1, 1),
|
||||
("text_value", "TEXT", 1, 0),
|
||||
),
|
||||
"execution_prompt_snapshots": (
|
||||
("snapshot_id", "TEXT", 1, 1),
|
||||
*((f"{field}_text_id", "TEXT", 0, 0) for field in FIELDS),
|
||||
),
|
||||
"execution_contexts": (
|
||||
("context_id", "TEXT", 1, 1),
|
||||
("positive_snapshot_id", "TEXT", 0, 0),
|
||||
("negative_snapshot_id", "TEXT", 0, 0),
|
||||
),
|
||||
"execution_run_contexts": (
|
||||
("run_id", "TEXT", 1, 1),
|
||||
("anchor_id", "TEXT", 1, 2),
|
||||
("context_id", "TEXT", 1, 0),
|
||||
("evidence_json", "TEXT", 1, 0),
|
||||
("positive_prompt_id", "INTEGER", 0, 0),
|
||||
("negative_prompt_id", "INTEGER", 0, 0),
|
||||
("pair_id", "INTEGER", 0, 0),
|
||||
("reference_epoch", "TEXT", 0, 0),
|
||||
("capture_version", "INTEGER", 1, 0),
|
||||
),
|
||||
"execution_context_capture": (
|
||||
("run_id", "TEXT", 1, 1),
|
||||
("state", "TEXT", 1, 0),
|
||||
("reason", "TEXT", 1, 0),
|
||||
("format_version", "INTEGER", 1, 0),
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def _digest_check(column: str, nullable: bool = False) -> str:
|
||||
check = (
|
||||
f"typeof({column}) = 'text' AND length({column}) = 64 "
|
||||
f"AND length(CAST({column} AS BLOB)) = 64 "
|
||||
f"AND {column} NOT GLOB '*[^0-9a-f]*'"
|
||||
)
|
||||
return f"{column} IS NULL OR ({check})" if nullable else check
|
||||
|
||||
|
||||
def _bounded_uuid_check(column: str) -> str:
|
||||
return f"{_uuid_check(column)} AND length(CAST({column} AS BLOB)) = 36"
|
||||
|
||||
|
||||
def _enum_check(column: str, values: frozenset[str]) -> str:
|
||||
return (
|
||||
f"typeof({column}) = 'text' AND {column} IN ("
|
||||
+ ",".join("'" + value + "'" for value in sorted(values))
|
||||
+ ")"
|
||||
)
|
||||
|
||||
|
||||
TABLE_CHECKS = {
|
||||
"execution_context_texts": (
|
||||
_digest_check("text_id"),
|
||||
"typeof(text_value) = 'text' AND length(CAST(text_value AS BLOB)) <= 65536",
|
||||
),
|
||||
"execution_prompt_snapshots": (
|
||||
_digest_check("snapshot_id"),
|
||||
*(_digest_check(f"{field}_text_id", True) for field in FIELDS),
|
||||
),
|
||||
"execution_contexts": (
|
||||
_digest_check("context_id"),
|
||||
_digest_check("positive_snapshot_id", True),
|
||||
_digest_check("negative_snapshot_id", True),
|
||||
"positive_snapshot_id IS NOT NULL OR negative_snapshot_id IS NOT NULL",
|
||||
),
|
||||
"execution_run_contexts": (
|
||||
_bounded_uuid_check("run_id"),
|
||||
"typeof(anchor_id) = 'text' AND length(anchor_id) BETWEEN 1 AND 128 "
|
||||
"AND length(CAST(anchor_id AS BLOB)) <= 512",
|
||||
_digest_check("context_id"),
|
||||
"typeof(evidence_json) = 'text' AND length(CAST(evidence_json AS "
|
||||
"BLOB)) <= 8192",
|
||||
*(
|
||||
f"{column} IS NULL OR (typeof({column}) = 'integer' AND {column} > 0)"
|
||||
for column in ("positive_prompt_id", "negative_prompt_id", "pair_id")
|
||||
),
|
||||
f"reference_epoch IS NULL OR ({_bounded_uuid_check('reference_epoch')})",
|
||||
"typeof(capture_version) = 'integer' AND capture_version = 1",
|
||||
),
|
||||
"execution_context_capture": (
|
||||
_bounded_uuid_check("run_id"),
|
||||
_enum_check("state", BUNDLE_STATES),
|
||||
_enum_check("reason", CAPTURE_REASONS),
|
||||
"length(CAST(reason AS BLOB)) <= 64",
|
||||
"typeof(format_version) = 'integer' AND format_version = 1",
|
||||
),
|
||||
}
|
||||
# (source column, destination table, destination column); actions always NO ACTION.
|
||||
TABLE_FOREIGN_KEYS = {
|
||||
"execution_context_texts": (),
|
||||
"execution_prompt_snapshots": tuple(
|
||||
(f"{field}_text_id", "execution_context_texts", "text_id") for field in FIELDS
|
||||
),
|
||||
"execution_contexts": tuple(
|
||||
(f"{role}_snapshot_id", "execution_prompt_snapshots", "snapshot_id")
|
||||
for role in ("positive", "negative")
|
||||
),
|
||||
"execution_run_contexts": (
|
||||
("run_id", "execution_runs", "run_id"),
|
||||
("context_id", "execution_contexts", "context_id"),
|
||||
),
|
||||
"execution_context_capture": (("run_id", "execution_runs", "run_id"),),
|
||||
}
|
||||
# Only SQLite-generated primary-key indexes are unique. No secondary indexes required.
|
||||
|
||||
|
||||
def install_context_schema(conn: sqlite3.Connection) -> None:
|
||||
"""Install tables inside the caller's transaction, without committing."""
|
||||
for table, columns in TABLE_COLUMNS.items():
|
||||
definitions = [
|
||||
f"{name} {kind}{' NOT NULL' if required else ''}"
|
||||
for name, kind, required, _ in columns
|
||||
]
|
||||
primary = [name for name, _, _, order in columns if order]
|
||||
definitions.append(f"PRIMARY KEY ({', '.join(primary)})")
|
||||
definitions.extend(f"CHECK({check})" for check in TABLE_CHECKS[table])
|
||||
definitions.extend(
|
||||
f"FOREIGN KEY ({column}) REFERENCES {target}({key}) "
|
||||
"ON UPDATE NO ACTION ON DELETE NO ACTION"
|
||||
for column, target, key in TABLE_FOREIGN_KEYS[table]
|
||||
)
|
||||
conn.execute(f"CREATE TABLE {table} ({', '.join(definitions)})")
|
||||
|
||||
|
||||
def _unique_object(pairs: list[tuple[str, object]]) -> dict:
|
||||
result = {}
|
||||
for key, value in pairs:
|
||||
if key in result:
|
||||
raise ValueError("Duplicate evidence key")
|
||||
result[key] = value
|
||||
return result
|
||||
|
||||
|
||||
def _snapshot_values(conn: sqlite3.Connection, snapshot_id: str) -> tuple:
|
||||
row = conn.execute(
|
||||
"SELECT raw_text_id,effective_text_id,prepend_text_id,append_text_id "
|
||||
"FROM execution_prompt_snapshots WHERE snapshot_id=?",
|
||||
(snapshot_id,),
|
||||
).fetchone()
|
||||
if row is None:
|
||||
raise ValueError("Missing snapshot")
|
||||
values = []
|
||||
for text_id in row:
|
||||
if text_id is None:
|
||||
values.append(None)
|
||||
else:
|
||||
text = conn.execute(
|
||||
"SELECT text_value FROM execution_context_texts WHERE text_id=?",
|
||||
(text_id,),
|
||||
).fetchone()
|
||||
if text is None:
|
||||
raise ValueError("Missing context text")
|
||||
text_size(text[0])
|
||||
values.append(text[0])
|
||||
return tuple(values)
|
||||
|
||||
|
||||
def _validate_evidence(
|
||||
conn: sqlite3.Connection, anchor_id: str, context_id: str, encoded: str
|
||||
) -> tuple[RecipeContext, bool]:
|
||||
text_size(encoded, 8192)
|
||||
evidence = json.loads(encoded, object_pairs_hook=_unique_object)
|
||||
if (
|
||||
type(evidence) is not dict
|
||||
or set(evidence)
|
||||
!= {"format_version", "role_provenance", "positive", "negative"}
|
||||
or type(evidence["format_version"]) is not int
|
||||
or evidence["format_version"] != 1
|
||||
or evidence["role_provenance"] != "inferred"
|
||||
):
|
||||
raise ValueError("Invalid context evidence")
|
||||
row = conn.execute(
|
||||
"SELECT positive_snapshot_id,negative_snapshot_id FROM "
|
||||
"execution_contexts WHERE context_id=?",
|
||||
(context_id,),
|
||||
).fetchone()
|
||||
if row is None:
|
||||
raise ValueError("Missing context")
|
||||
complete = True
|
||||
snapshots = []
|
||||
for role, snapshot_id in zip(("positive", "negative"), row):
|
||||
metadata = evidence[role]
|
||||
if snapshot_id is None:
|
||||
if metadata is not None:
|
||||
raise ValueError("Unexpected role evidence")
|
||||
complete = False
|
||||
snapshots.append(None)
|
||||
continue
|
||||
values = _snapshot_values(conn, snapshot_id)
|
||||
if type(metadata) is not dict or set(metadata) != set(FIELDS):
|
||||
raise ValueError("Invalid snapshot evidence")
|
||||
fields = []
|
||||
for field, value in zip(FIELDS, values):
|
||||
item = metadata[field]
|
||||
if type(item) is not dict or set(item) != {
|
||||
"provenance",
|
||||
"source",
|
||||
"reason",
|
||||
}:
|
||||
raise ValueError("Invalid field evidence")
|
||||
provenance, source, reason = (
|
||||
item["provenance"],
|
||||
item["source"],
|
||||
item["reason"],
|
||||
)
|
||||
if (
|
||||
type(provenance) is not str
|
||||
or type(source) is not str
|
||||
or source not in EVIDENCE_SOURCES.get(provenance, ())
|
||||
):
|
||||
raise ValueError("Invalid evidence source")
|
||||
if value is None:
|
||||
complete = False
|
||||
if (
|
||||
provenance != "unknown"
|
||||
or type(reason) is not str
|
||||
or reason not in UNKNOWN_REASONS
|
||||
):
|
||||
raise ValueError("Unknown evidence requires reason")
|
||||
elif provenance == "unknown" or reason is not None:
|
||||
raise ValueError("Known evidence metadata mismatch")
|
||||
fields.append(TextEvidence(value, provenance, source, reason))
|
||||
snapshots.append(PromptSnapshot(*fields))
|
||||
return RecipeContext(anchor_id, *snapshots, evidence["role_provenance"]), complete
|
||||
|
||||
|
||||
def context_schema_error(conn: sqlite3.Connection) -> str | None:
|
||||
"""Validate exact owned schema, bounded evidence, fingerprints and links."""
|
||||
if conn.execute(
|
||||
"SELECT 1 FROM sqlite_master WHERE type IN ('trigger','view') LIMIT 1"
|
||||
).fetchone():
|
||||
return "Unsupported execution context trigger/view"
|
||||
tables = {
|
||||
row[0]: row[1]
|
||||
for row in conn.execute("SELECT name,sql FROM sqlite_master WHERE type='table'")
|
||||
if row[0] in TABLE_COLUMNS
|
||||
}
|
||||
if tables.keys() != TABLE_COLUMNS.keys():
|
||||
return "Incomplete execution context schema"
|
||||
for table, expected in TABLE_COLUMNS.items():
|
||||
columns = tuple(
|
||||
(row[1], row[2].upper(), row[3], row[5])
|
||||
for row in conn.execute(f"PRAGMA table_xinfo({table})")
|
||||
)
|
||||
if columns != expected:
|
||||
return f"{table} has invalid execution context columns"
|
||||
sql = _compact(tables[table])
|
||||
checks = TABLE_CHECKS[table]
|
||||
if sql.count("check(") != len(checks) or any(
|
||||
f"check({_compact(check)})" not in sql for check in checks
|
||||
):
|
||||
return f"{table} has invalid execution context checks"
|
||||
foreign = tuple(
|
||||
sorted(
|
||||
(row[3], row[2], row[4], row[5], row[6], row[7])
|
||||
for row in conn.execute(f"PRAGMA foreign_key_list({table})")
|
||||
)
|
||||
)
|
||||
expected_foreign = tuple(
|
||||
sorted(
|
||||
(*key, "NO ACTION", "NO ACTION", "NONE")
|
||||
for key in TABLE_FOREIGN_KEYS[table]
|
||||
)
|
||||
)
|
||||
if foreign != expected_foreign:
|
||||
return f"{table} has invalid execution context foreign keys"
|
||||
for index in conn.execute(f"PRAGMA index_list({table})"):
|
||||
if not index[2]:
|
||||
continue
|
||||
name = index[1].replace('"', '""')
|
||||
keys = tuple(
|
||||
row for row in conn.execute(f'PRAGMA index_xinfo("{name}")') if row[5]
|
||||
)
|
||||
primary = tuple(column[0] for column in expected if column[3])
|
||||
if (
|
||||
index[3] != "pk"
|
||||
or index[4]
|
||||
or tuple(key[2] for key in keys) != primary
|
||||
or any(key[3] or key[4] != "BINARY" for key in keys)
|
||||
):
|
||||
return f"{table} has unexpected execution context uniqueness"
|
||||
predicate = " AND ".join(f"({check})" for check in checks)
|
||||
if conn.execute(
|
||||
f"SELECT 1 FROM {table} WHERE NOT ({predicate}) LIMIT 1"
|
||||
).fetchone():
|
||||
return f"{table} contains invalid execution context records"
|
||||
if conn.execute(f"PRAGMA foreign_key_check({table})").fetchone():
|
||||
return f"{table} contains orphan execution context records"
|
||||
if conn.execute(
|
||||
"SELECT 1 FROM execution_run_contexts GROUP BY run_id HAVING "
|
||||
"COUNT(*) > 16 LIMIT 1"
|
||||
).fetchone():
|
||||
return "Too many execution contexts per run"
|
||||
try:
|
||||
for text_id, text in conn.execute(
|
||||
"SELECT text_id,text_value FROM execution_context_texts"
|
||||
):
|
||||
text_size(text)
|
||||
if content_digest("text", (text,)) != text_id:
|
||||
raise ValueError("Invalid text fingerprint")
|
||||
for (snapshot_id,) in conn.execute(
|
||||
"SELECT snapshot_id FROM execution_prompt_snapshots"
|
||||
):
|
||||
if (
|
||||
content_digest("snapshot", _snapshot_values(conn, snapshot_id))
|
||||
!= snapshot_id
|
||||
):
|
||||
raise ValueError("Invalid snapshot fingerprint")
|
||||
for context_id, positive, negative in conn.execute(
|
||||
"SELECT context_id,positive_snapshot_id,negative_snapshot_id FROM "
|
||||
"execution_contexts"
|
||||
):
|
||||
if content_digest("context", (positive, negative)) != context_id:
|
||||
raise ValueError("Invalid context fingerprint")
|
||||
payload_size = 0
|
||||
for snapshot_id in (positive, negative):
|
||||
if snapshot_id is not None:
|
||||
payload_size += sum(
|
||||
text_size(value)
|
||||
for value in _snapshot_values(conn, snapshot_id)
|
||||
if value is not None
|
||||
)
|
||||
if payload_size > MAX_CONTEXT_BYTES:
|
||||
raise ValueError("Oversized detached context")
|
||||
previous_run = None
|
||||
contexts = ()
|
||||
for run_id, anchor_id, context_id, evidence in conn.execute(
|
||||
"SELECT run_id,anchor_id,context_id,evidence_json FROM "
|
||||
"execution_run_contexts ORDER BY run_id,anchor_id"
|
||||
):
|
||||
if run_id != previous_run:
|
||||
contexts = ()
|
||||
previous_run = run_id
|
||||
context, complete = _validate_evidence(
|
||||
conn, anchor_id, context_id, evidence
|
||||
)
|
||||
status = conn.execute(
|
||||
"SELECT state,reason FROM execution_context_capture WHERE run_id=?",
|
||||
(run_id,),
|
||||
).fetchone()
|
||||
if status is None:
|
||||
raise ValueError("Context link has no capture header")
|
||||
state, reason = status
|
||||
if state == "complete" and not complete:
|
||||
raise ValueError("Incomplete evidence marked complete")
|
||||
candidate = (*contexts, context)
|
||||
validate_bundle(ContextBundle(candidate, state, reason))
|
||||
contexts = candidate
|
||||
if conn.execute(
|
||||
"SELECT 1 FROM execution_context_capture c WHERE state='complete' "
|
||||
"AND NOT EXISTS (SELECT 1 FROM execution_run_contexts r WHERE "
|
||||
"r.run_id=c.run_id) LIMIT 1"
|
||||
).fetchone():
|
||||
raise ValueError("Complete capture has no contexts")
|
||||
except (ValueError, TypeError, RecursionError, OverflowError) as exc:
|
||||
return f"Invalid execution context content: {type(exc).__name__}"
|
||||
return None
|
||||
@@ -0,0 +1,376 @@
|
||||
"""Immutable execution capture and exact, epoch-fenced library navigation hints."""
|
||||
|
||||
from contextlib import ExitStack
|
||||
import json
|
||||
import sqlite3
|
||||
from typing import Callable, ContextManager, Iterator
|
||||
|
||||
try:
|
||||
from ..utils.execution_history_events import RunFinish, encode_command
|
||||
from ..utils.execution_contexts import (
|
||||
CAPTURE_REASONS,
|
||||
ContextBundle,
|
||||
content_digest,
|
||||
validate_bundle,
|
||||
)
|
||||
except ImportError:
|
||||
from utils.execution_history_events import RunFinish, encode_command
|
||||
from utils.execution_contexts import (
|
||||
CAPTURE_REASONS,
|
||||
ContextBundle,
|
||||
content_digest,
|
||||
validate_bundle,
|
||||
)
|
||||
|
||||
|
||||
FIELDS = ("raw", "effective", "prepend", "append")
|
||||
ROLES = ("positive", "negative")
|
||||
|
||||
|
||||
def _prepare(bundle: ContextBundle) -> list[dict]:
|
||||
validate_bundle(bundle)
|
||||
if bundle.reason not in CAPTURE_REASONS:
|
||||
raise ValueError("Invalid capture reason")
|
||||
prepared = []
|
||||
complete = bool(bundle.contexts)
|
||||
for context in bundle.contexts:
|
||||
evidence = {"format_version": 1, "role_provenance": context.role_provenance}
|
||||
snapshots = []
|
||||
for role in ROLES:
|
||||
snapshot = getattr(context, role)
|
||||
if snapshot is None:
|
||||
snapshots.append(None)
|
||||
evidence[role] = None
|
||||
complete = False
|
||||
continue
|
||||
values = tuple(getattr(snapshot, field).value for field in FIELDS)
|
||||
complete = complete and all(value is not None for value in values)
|
||||
snapshots.append((content_digest("snapshot", values), values))
|
||||
evidence[role] = {
|
||||
field: {
|
||||
"provenance": getattr(snapshot, field).provenance,
|
||||
"source": getattr(snapshot, field).source,
|
||||
"reason": getattr(snapshot, field).reason,
|
||||
}
|
||||
for field in FIELDS
|
||||
}
|
||||
if all(snapshot is None for snapshot in snapshots):
|
||||
raise ValueError("Context must contain a snapshot")
|
||||
context_id = content_digest(
|
||||
"context", tuple(s[0] if s else None for s in snapshots)
|
||||
)
|
||||
encoded = json.dumps(
|
||||
evidence,
|
||||
ensure_ascii=False,
|
||||
allow_nan=False,
|
||||
separators=(",", ":"),
|
||||
sort_keys=True,
|
||||
)
|
||||
if len(encoded.encode("utf-8")) > 8192:
|
||||
raise ValueError("Oversized context evidence")
|
||||
prepared.append(
|
||||
{
|
||||
"anchor_id": context.anchor_id,
|
||||
"context_id": context_id,
|
||||
"snapshots": snapshots,
|
||||
"evidence_json": encoded,
|
||||
}
|
||||
)
|
||||
if bundle.state == "complete" and not complete:
|
||||
raise ValueError("Incomplete scalar capture")
|
||||
return sorted(prepared, key=lambda item: item["anchor_id"])
|
||||
|
||||
|
||||
def _existing_row(
|
||||
conn: sqlite3.Connection, table: str, columns: tuple[str, ...], values: tuple
|
||||
) -> bool:
|
||||
row = conn.execute(
|
||||
f"SELECT {','.join(columns[1:])} FROM {table} WHERE {columns[0]}=?",
|
||||
(values[0],),
|
||||
).fetchone()
|
||||
if row is None:
|
||||
return False
|
||||
if tuple(row) != values[1:]:
|
||||
raise ValueError("Execution context digest collision")
|
||||
return True
|
||||
|
||||
|
||||
def _content_rows(item: dict) -> Iterator[tuple[str, tuple[str, ...], tuple]]:
|
||||
snapshot_ids = []
|
||||
for snapshot in item["snapshots"]:
|
||||
if snapshot is None:
|
||||
snapshot_ids.append(None)
|
||||
continue
|
||||
snapshot_id, values = snapshot
|
||||
text_ids = []
|
||||
for value in values:
|
||||
text_id = None if value is None else content_digest("text", (value,))
|
||||
if value is not None:
|
||||
yield "execution_context_texts", ("text_id", "text_value"), (
|
||||
text_id,
|
||||
value,
|
||||
)
|
||||
text_ids.append(text_id)
|
||||
yield "execution_prompt_snapshots", (
|
||||
"snapshot_id",
|
||||
*(f"{field}_text_id" for field in FIELDS),
|
||||
), (snapshot_id, *text_ids)
|
||||
snapshot_ids.append(snapshot_id)
|
||||
yield "execution_contexts", (
|
||||
"context_id",
|
||||
"positive_snapshot_id",
|
||||
"negative_snapshot_id",
|
||||
), (item["context_id"], *snapshot_ids)
|
||||
|
||||
|
||||
def _insert_exact(
|
||||
conn: sqlite3.Connection, table: str, columns: tuple[str, ...], values: tuple
|
||||
) -> None:
|
||||
if not _existing_row(conn, table, columns, values):
|
||||
names = ",".join(columns)
|
||||
placeholders = ",".join("?" for _ in columns)
|
||||
conn.execute(f"INSERT INTO {table} ({names}) VALUES ({placeholders})", values)
|
||||
|
||||
|
||||
def _prompt_hint(conn: sqlite3.Connection, raw: str | None) -> int | None:
|
||||
if raw is None:
|
||||
return None
|
||||
rows = conn.execute(
|
||||
"SELECT id FROM prompts WHERE text=? COLLATE BINARY LIMIT 2", (raw,)
|
||||
).fetchall()
|
||||
return rows[0][0] if len(rows) == 1 else None
|
||||
|
||||
|
||||
def _hints(
|
||||
conn: sqlite3.Connection, positive: str | None, negative: str | None
|
||||
) -> tuple:
|
||||
pos_id, neg_id = _prompt_hint(conn, positive), _prompt_hint(conn, negative)
|
||||
pair_id = None
|
||||
negative_ambiguous = (
|
||||
negative is not None
|
||||
and neg_id is None
|
||||
and conn.execute(
|
||||
"SELECT 1 FROM prompts WHERE text=? COLLATE BINARY LIMIT 1", (negative,)
|
||||
).fetchone()
|
||||
is not None
|
||||
)
|
||||
if pos_id is not None and negative is not None and not negative_ambiguous:
|
||||
rows = conn.execute(
|
||||
"SELECT pp.id FROM prompt_pairs pp LEFT JOIN prompts n ON "
|
||||
"n.id=pp.negative_prompt_id "
|
||||
"WHERE pp.positive_prompt_id=? AND ((pp.negative_prompt_id IS NULL AND "
|
||||
"?='') "
|
||||
"OR (pp.negative_prompt_id=? AND n.text=? COLLATE BINARY)) LIMIT 2",
|
||||
(pos_id, negative, neg_id, negative),
|
||||
).fetchall()
|
||||
if len(rows) == 1:
|
||||
pair_id = rows[0][0]
|
||||
return pos_id, neg_id, pair_id
|
||||
|
||||
|
||||
def _values(conn: sqlite3.Connection, snapshot_id: str | None) -> dict | None:
|
||||
if snapshot_id is None:
|
||||
return None
|
||||
row = conn.execute(
|
||||
"SELECT r.text_value,e.text_value,p.text_value,a.text_value "
|
||||
"FROM execution_prompt_snapshots s "
|
||||
"LEFT JOIN execution_context_texts r ON r.text_id=s.raw_text_id "
|
||||
"LEFT JOIN execution_context_texts e ON e.text_id=s.effective_text_id "
|
||||
"LEFT JOIN execution_context_texts p ON p.text_id=s.prepend_text_id "
|
||||
"LEFT JOIN execution_context_texts a ON a.text_id=s.append_text_id "
|
||||
"WHERE s.snapshot_id=?",
|
||||
(snapshot_id,),
|
||||
).fetchone()
|
||||
if row is None:
|
||||
raise sqlite3.DatabaseError("Missing execution context snapshot")
|
||||
return dict(zip(FIELDS, row))
|
||||
|
||||
|
||||
def _write_capture(
|
||||
conn: sqlite3.Connection,
|
||||
run_id: str,
|
||||
expected_epoch: str,
|
||||
bundle: ContextBundle,
|
||||
prepared: list[dict],
|
||||
) -> bool:
|
||||
capture = conn.execute(
|
||||
"SELECT state,reason,format_version FROM execution_context_capture "
|
||||
"WHERE run_id=?",
|
||||
(run_id,),
|
||||
).fetchone()
|
||||
if capture is not None:
|
||||
for item in prepared:
|
||||
for table, columns, values in _content_rows(item):
|
||||
_existing_row(conn, table, columns, values)
|
||||
links = conn.execute(
|
||||
"SELECT anchor_id,context_id,evidence_json FROM "
|
||||
"execution_run_contexts WHERE run_id=? ORDER BY anchor_id",
|
||||
(run_id,),
|
||||
).fetchall()
|
||||
expected = [
|
||||
(item["anchor_id"], item["context_id"], item["evidence_json"])
|
||||
for item in prepared
|
||||
]
|
||||
# Compare JSON objects so serialization order never creates a conflict.
|
||||
original = [(row[0], row[1], json.loads(row[2])) for row in links]
|
||||
proposed = [
|
||||
(anchor, context, json.loads(evidence))
|
||||
for anchor, context, evidence in expected
|
||||
]
|
||||
if capture[0] != "conflicting" and (
|
||||
tuple(capture) != (bundle.state, bundle.reason, bundle.format_version)
|
||||
or original != proposed
|
||||
):
|
||||
conn.execute(
|
||||
"UPDATE execution_context_capture SET "
|
||||
"state='conflicting',reason='conflicting' WHERE run_id=?",
|
||||
(run_id,),
|
||||
)
|
||||
return True
|
||||
for item in prepared:
|
||||
for table, columns, values in _content_rows(item):
|
||||
_insert_exact(conn, table, columns, values)
|
||||
raw = [snapshot[1][0] if snapshot else None for snapshot in item["snapshots"]]
|
||||
positive_id, negative_id, pair_id = _hints(conn, *raw)
|
||||
conn.execute(
|
||||
"INSERT INTO "
|
||||
"execution_run_contexts(run_id,anchor_id,context_id,evidence_json,"
|
||||
"positive_prompt_id,negative_prompt_id,pair_id,reference_epoch,"
|
||||
"capture_version) VALUES(?,?,?,?,?,?,?,?,1)",
|
||||
(
|
||||
run_id,
|
||||
item["anchor_id"],
|
||||
item["context_id"],
|
||||
item["evidence_json"],
|
||||
positive_id,
|
||||
negative_id,
|
||||
pair_id,
|
||||
expected_epoch,
|
||||
),
|
||||
)
|
||||
conn.execute(
|
||||
"INSERT INTO "
|
||||
"execution_context_capture(run_id,state,reason,format_version) "
|
||||
"VALUES(?,?,?,?)",
|
||||
(run_id, bundle.state, bundle.reason, bundle.format_version),
|
||||
)
|
||||
return True
|
||||
|
||||
|
||||
class ExecutionContextOperations:
|
||||
"""Compose with ExecutionHistoryOperations on PromptDatabase."""
|
||||
|
||||
def write_execution_contexts(
|
||||
self, run_id: str, expected_epoch: str, bundle: ContextBundle
|
||||
) -> bool:
|
||||
prepared = _prepare(bundle)
|
||||
with self._execution_transaction(expected_epoch) as conn:
|
||||
if (
|
||||
conn.execute(
|
||||
"SELECT 1 FROM execution_runs WHERE run_id=? AND storage_epoch=?",
|
||||
(run_id, expected_epoch),
|
||||
).fetchone()
|
||||
is None
|
||||
):
|
||||
return False
|
||||
return _write_capture(conn, run_id, expected_epoch, bundle, prepared)
|
||||
|
||||
def write_execution_contexts_for_finish(
|
||||
self,
|
||||
command: RunFinish,
|
||||
bundle: ContextBundle,
|
||||
*,
|
||||
generation_guard: Callable[[], ContextManager[bool]] | None = None,
|
||||
) -> bool:
|
||||
encode_command(command)
|
||||
if type(command) is not RunFinish:
|
||||
raise ValueError("Expected execution finish command")
|
||||
prepared = _prepare(bundle)
|
||||
# Enter the guard after BEGIN, and keep it until COMMIT/ROLLBACK exits.
|
||||
with ExitStack() as guards:
|
||||
with self._execution_transaction(command.storage_epoch) as conn:
|
||||
if generation_guard is not None and not guards.enter_context(
|
||||
generation_guard()
|
||||
):
|
||||
return False
|
||||
row = conn.execute(
|
||||
"SELECT run_id FROM execution_runs WHERE session_id=? "
|
||||
"AND queue_item_id=? AND storage_epoch=?",
|
||||
(command.session_id, command.queue_item_id, command.storage_epoch),
|
||||
).fetchone()
|
||||
if row is None:
|
||||
return False
|
||||
return _write_capture(
|
||||
conn, row[0], command.storage_epoch, bundle, prepared
|
||||
)
|
||||
|
||||
def get_execution_contexts(self, run_id: str) -> dict:
|
||||
"""Return first-capture scalars; hints are None unless currently valid.
|
||||
|
||||
reference_epoch remains historical evidence even when hints are invalidated.
|
||||
Reads never rebind a first-capture reference or alter stored capture data.
|
||||
"""
|
||||
conn = self.model.get_connection()
|
||||
own_transaction = not conn.in_transaction
|
||||
if own_transaction:
|
||||
conn.execute("BEGIN")
|
||||
try:
|
||||
capture = conn.execute(
|
||||
"SELECT state,reason,format_version FROM execution_context_capture "
|
||||
"WHERE run_id=?",
|
||||
(run_id,),
|
||||
).fetchone()
|
||||
if capture is None:
|
||||
return {
|
||||
"state": "not_captured",
|
||||
"reason": "missing",
|
||||
"format_version": None,
|
||||
"contexts": [],
|
||||
}
|
||||
epoch = self.get_execution_epoch()
|
||||
contexts = []
|
||||
rows = conn.execute(
|
||||
"SELECT r.*,c.positive_snapshot_id,c.negative_snapshot_id FROM "
|
||||
"execution_run_contexts r JOIN execution_contexts c ON "
|
||||
"c.context_id=r.context_id WHERE r.run_id=? ORDER BY r.anchor_id",
|
||||
(run_id,),
|
||||
)
|
||||
for row in rows:
|
||||
positive = _values(conn, row["positive_snapshot_id"])
|
||||
negative = _values(conn, row["negative_snapshot_id"])
|
||||
hints = (None, None, None)
|
||||
if row["reference_epoch"] == epoch:
|
||||
current = _hints(
|
||||
conn,
|
||||
positive["raw"] if positive else None,
|
||||
negative["raw"] if negative else None,
|
||||
)
|
||||
hints = tuple(
|
||||
row[key] if row[key] == value else None
|
||||
for key, value in zip(
|
||||
("positive_prompt_id", "negative_prompt_id", "pair_id"),
|
||||
current,
|
||||
)
|
||||
)
|
||||
contexts.append(
|
||||
{
|
||||
"anchor_id": row["anchor_id"],
|
||||
"context_id": row["context_id"],
|
||||
"positive": positive,
|
||||
"negative": negative,
|
||||
"evidence": json.loads(row["evidence_json"]),
|
||||
"positive_prompt_id": hints[0],
|
||||
"negative_prompt_id": hints[1],
|
||||
"pair_id": hints[2],
|
||||
"reference_epoch": row["reference_epoch"],
|
||||
}
|
||||
)
|
||||
return {
|
||||
"state": capture[0],
|
||||
"reason": capture[1],
|
||||
"format_version": capture[2],
|
||||
"contexts": contexts,
|
||||
}
|
||||
finally:
|
||||
if own_transaction:
|
||||
conn.rollback()
|
||||
@@ -4,7 +4,7 @@ import re
|
||||
import sqlite3
|
||||
from uuid import uuid4
|
||||
|
||||
SCHEMA_VERSION = 1
|
||||
SCHEMA_VERSION = 2
|
||||
|
||||
|
||||
def _uuid_check(column: str) -> str:
|
||||
@@ -24,7 +24,7 @@ def _integer_check(column: str, *, nullable: bool = False) -> str:
|
||||
|
||||
_METADATA_CHECKS = (
|
||||
"singleton_id = 1",
|
||||
"schema_version = 1",
|
||||
"schema_version = 2",
|
||||
_uuid_check("storage_epoch"),
|
||||
)
|
||||
_RUN_CHECKS = (
|
||||
@@ -44,13 +44,13 @@ _RUN_CHECKS = (
|
||||
)
|
||||
),
|
||||
"outcome IN ('observed', 'succeeded', 'failed', 'interrupted', 'unknown')",
|
||||
"(outcome = 'observed' AND reason IS NULL AND observed_callback_at IS NULL) OR "
|
||||
"(outcome != 'observed' AND reason IS NOT NULL "
|
||||
"AND observed_callback_at IS NOT NULL AND "
|
||||
"(outcome = 'observed' AND reason IS NULL AND "
|
||||
"observed_callback_at IS NULL) OR (outcome != 'observed' AND "
|
||||
"reason IS NOT NULL AND observed_callback_at IS NOT NULL AND "
|
||||
"((outcome = 'succeeded' AND reason = 'terminal_success') OR "
|
||||
"(outcome = 'failed' AND reason = 'terminal_error') OR "
|
||||
"(outcome = 'interrupted' AND reason = 'terminal_interrupted') OR "
|
||||
"(outcome = 'unknown' AND reason IN ('missing_terminal', "
|
||||
"(outcome = 'failed' AND reason = 'terminal_error') OR (outcome = "
|
||||
"'interrupted' AND reason = 'terminal_interrupted') OR (outcome = "
|
||||
"'unknown' AND reason IN ('missing_terminal', "
|
||||
"'conflicting_terminal', 'invalid_evidence', 'evidence_limit'))))",
|
||||
"diagnostics_omitted IN (0, 1)",
|
||||
*(
|
||||
@@ -92,6 +92,30 @@ _TABLE_CHECKS = {
|
||||
"execution_history_metadata": _METADATA_CHECKS,
|
||||
"execution_runs": _RUN_CHECKS,
|
||||
}
|
||||
# Row-only guards preserve the supported v1 DDL fingerprint while bounding reads.
|
||||
_ROW_BYTE_CHECKS = {
|
||||
"execution_history_metadata": ("length(CAST(storage_epoch AS BLOB)) = 36",),
|
||||
"execution_runs": (
|
||||
*(
|
||||
f"length(CAST({column} AS BLOB)) = 36"
|
||||
for column in (
|
||||
"run_id",
|
||||
"session_id",
|
||||
"job_id",
|
||||
"storage_epoch",
|
||||
)
|
||||
),
|
||||
*(
|
||||
f"{column} IS NULL OR length(CAST({column} AS BLOB)) <= {limit}"
|
||||
for column, limit in (
|
||||
("node_id", 512),
|
||||
("node_type", 512),
|
||||
("exception_type", 1024),
|
||||
)
|
||||
),
|
||||
),
|
||||
}
|
||||
|
||||
_INDEXES = {
|
||||
"idx_execution_runs_identity": (True, ("session_id", "queue_item_id")),
|
||||
"idx_execution_runs_observed": (False, ("observed_dequeue_at", "run_id")),
|
||||
@@ -114,10 +138,29 @@ def history_schema_error(
|
||||
for row in conn.execute("SELECT name,sql FROM sqlite_master WHERE type='table'")
|
||||
if row[0] in _TABLE_COLUMNS
|
||||
}
|
||||
from .execution_context_schema import TABLE_COLUMNS, context_schema_error
|
||||
|
||||
context_present = any(
|
||||
row[0] in TABLE_COLUMNS
|
||||
for row in conn.execute("SELECT name FROM sqlite_master WHERE type='table'")
|
||||
)
|
||||
if not tables and context_present:
|
||||
return "Execution contexts require execution history"
|
||||
if not tables:
|
||||
return None if allow_missing else "Missing execution history schema"
|
||||
if tables.keys() != _TABLE_COLUMNS.keys():
|
||||
return "Incomplete execution history schema"
|
||||
metadata = conn.execute(
|
||||
"SELECT CASE WHEN typeof(schema_version) = 'integer' "
|
||||
"THEN schema_version ELSE NULL END FROM execution_history_metadata LIMIT 2"
|
||||
).fetchall()
|
||||
if len(metadata) != 1 or len(metadata[0]) != 1 or metadata[0][0] not in (1, 2):
|
||||
return "Unsupported execution history metadata/version"
|
||||
version = metadata[0][0]
|
||||
if version == 1 and context_present:
|
||||
return "Unexpected execution contexts for history v1"
|
||||
if not allow_missing and version != SCHEMA_VERSION:
|
||||
return "Execution history migration required"
|
||||
for table, expected in _TABLE_COLUMNS.items():
|
||||
columns = tuple(
|
||||
(row[1], row[2].upper(), row[3], row[5])
|
||||
@@ -127,6 +170,11 @@ def history_schema_error(
|
||||
return f"{table} has invalid execution history columns"
|
||||
sql = _compact(tables[table])
|
||||
checks = _TABLE_CHECKS[table]
|
||||
if table == "execution_history_metadata" and version == 1:
|
||||
checks = tuple(
|
||||
check.replace("schema_version = 2", "schema_version = 1")
|
||||
for check in checks
|
||||
)
|
||||
if sql.count("check(") != len(checks) or any(
|
||||
f"check({_compact(check)})" not in sql for check in checks
|
||||
):
|
||||
@@ -150,16 +198,18 @@ def history_schema_error(
|
||||
or any(key[3] or key[4] != "BINARY" for key in keys)
|
||||
):
|
||||
return f"{table} has unexpected execution history uniqueness"
|
||||
predicate = " AND ".join(f"({check})" for check in checks)
|
||||
predicate = " AND ".join(
|
||||
f"({check})" for check in (*checks, *_ROW_BYTE_CHECKS[table])
|
||||
)
|
||||
if conn.execute(
|
||||
f"SELECT 1 FROM {table} WHERE NOT ({predicate}) LIMIT 1"
|
||||
).fetchone():
|
||||
return f"{table} contains invalid execution history records"
|
||||
metadata = conn.execute(
|
||||
"SELECT singleton_id,schema_version,storage_epoch "
|
||||
"FROM execution_history_metadata"
|
||||
"SELECT singleton_id,schema_version,storage_epoch FROM "
|
||||
"execution_history_metadata"
|
||||
).fetchall()
|
||||
if len(metadata) != 1 or metadata[0][0] != 1 or metadata[0][1] != SCHEMA_VERSION:
|
||||
if len(metadata) != 1 or metadata[0][0] != 1 or metadata[0][1] != version:
|
||||
return "Unsupported execution history metadata/version"
|
||||
indexes = {row[1]: row for row in conn.execute("PRAGMA index_list(execution_runs)")}
|
||||
for name, (unique, columns) in _INDEXES.items():
|
||||
@@ -172,13 +222,24 @@ def history_schema_error(
|
||||
item[3] or item[4] != "BINARY" for item in keys
|
||||
):
|
||||
return f"Invalid execution history index {name}"
|
||||
if version == 2:
|
||||
return context_schema_error(conn)
|
||||
return None
|
||||
|
||||
|
||||
def install_context_schema(conn: sqlite3.Connection) -> None:
|
||||
from .execution_context_schema import install_context_schema as install
|
||||
|
||||
install(conn)
|
||||
|
||||
|
||||
def migrate_execution_history(conn: sqlite3.Connection) -> None:
|
||||
"""Install both tables and indexes atomically, without backfilling runs."""
|
||||
"""Install versioned history and context tables atomically, without backfill."""
|
||||
conn.execute("SAVEPOINT execution_history_schema")
|
||||
try:
|
||||
error = history_schema_error(conn)
|
||||
if error:
|
||||
raise sqlite3.DatabaseError(error)
|
||||
present = conn.execute(
|
||||
"SELECT 1 FROM sqlite_master WHERE name IN "
|
||||
"('execution_runs','execution_history_metadata') LIMIT 1"
|
||||
@@ -201,6 +262,33 @@ def migrate_execution_history(conn: sqlite3.Connection) -> None:
|
||||
"INSERT INTO execution_history_metadata VALUES(1,?,?)",
|
||||
(SCHEMA_VERSION, str(uuid4())),
|
||||
)
|
||||
install_context_schema(conn)
|
||||
else:
|
||||
version, epoch = conn.execute(
|
||||
"SELECT schema_version,storage_epoch FROM execution_history_metadata"
|
||||
).fetchone()
|
||||
if version == 1:
|
||||
conn.execute(
|
||||
"ALTER TABLE execution_history_metadata RENAME TO "
|
||||
"execution_history_metadata_v1"
|
||||
)
|
||||
definitions = [
|
||||
f"{name} {kind}{' NOT NULL' if required else ''}"
|
||||
f"{' PRIMARY KEY' if primary else ''}"
|
||||
for name, kind, required, primary in _TABLE_COLUMNS[
|
||||
"execution_history_metadata"
|
||||
]
|
||||
]
|
||||
definitions.extend(f"CHECK({check})" for check in _METADATA_CHECKS)
|
||||
conn.execute(
|
||||
"CREATE TABLE execution_history_metadata "
|
||||
f"({', '.join(definitions)})"
|
||||
)
|
||||
conn.execute(
|
||||
"INSERT INTO execution_history_metadata VALUES(1,2,?)", (epoch,)
|
||||
)
|
||||
conn.execute("DROP TABLE execution_history_metadata_v1")
|
||||
install_context_schema(conn)
|
||||
error = history_schema_error(conn, allow_missing=False)
|
||||
if error:
|
||||
raise sqlite3.DatabaseError(error)
|
||||
@@ -216,8 +304,8 @@ def rotate_history_epoch(conn: sqlite3.Connection) -> str:
|
||||
epoch = str(uuid4())
|
||||
with conn:
|
||||
cursor = conn.execute(
|
||||
"UPDATE execution_history_metadata SET storage_epoch=? "
|
||||
"WHERE singleton_id=1",
|
||||
"UPDATE execution_history_metadata SET storage_epoch=? WHERE "
|
||||
"singleton_id=1",
|
||||
(epoch,),
|
||||
)
|
||||
if cursor.rowcount != 1:
|
||||
|
||||
@@ -12,6 +12,7 @@ from .models import PromptModel, normalize_image_path, utc_now_iso
|
||||
from .pairs import PairDependencyError, PairOperations
|
||||
from .library import LibraryOperations
|
||||
from .execution_history import ExecutionHistoryOperations
|
||||
from .execution_contexts import ExecutionContextOperations
|
||||
|
||||
# Import logging system
|
||||
try:
|
||||
@@ -152,7 +153,12 @@ def _coerce_file_info(metadata: Any) -> tuple:
|
||||
)
|
||||
|
||||
|
||||
class PromptDatabase(PairOperations, LibraryOperations, ExecutionHistoryOperations):
|
||||
class PromptDatabase(
|
||||
PairOperations,
|
||||
LibraryOperations,
|
||||
ExecutionHistoryOperations,
|
||||
ExecutionContextOperations,
|
||||
):
|
||||
"""Database operations class for managing prompts."""
|
||||
|
||||
def __init__(self, db_path: Optional[str] = None):
|
||||
|
||||
+10
-1
@@ -6,6 +6,11 @@ with persistent prompt storage and search capabilities.
|
||||
import time
|
||||
from typing import Any, Dict, Optional, Tuple
|
||||
|
||||
if __package__:
|
||||
from .utils.execution_context_capture import record_runtime_context
|
||||
else:
|
||||
from utils.execution_context_capture import record_runtime_context
|
||||
|
||||
try:
|
||||
from comfy.comfy_types import IO, ComfyNodeABC, InputTypeDict
|
||||
except ImportError:
|
||||
@@ -256,7 +261,11 @@ class PromptManager(PromptManagerBase, ComfyNodeABC):
|
||||
)
|
||||
|
||||
self.logger.info(f"CLIP encoding completed, text: {repr(encoding_text)[:80]}")
|
||||
return (conditioning, encoding_text)
|
||||
result = (conditioning, encoding_text)
|
||||
record_runtime_context(
|
||||
type(self).__name__, text, prepend_text, append_text, encoding_text
|
||||
)
|
||||
return result
|
||||
|
||||
@classmethod
|
||||
def IS_CHANGED(
|
||||
|
||||
+18
-1
@@ -5,6 +5,11 @@ import json
|
||||
import time
|
||||
from typing import Any, Dict, Optional, Tuple
|
||||
|
||||
if __package__:
|
||||
from .utils.execution_context_capture import record_runtime_context
|
||||
else:
|
||||
from utils.execution_context_capture import record_runtime_context
|
||||
|
||||
try:
|
||||
from .prompt_manager import IO, InputTypeDict, PromptManagerFocused
|
||||
from .utils.hashing import generate_prompt_hash
|
||||
@@ -204,7 +209,19 @@ class PromptManagerPair(PromptManagerFocused):
|
||||
)
|
||||
except Exception as error:
|
||||
self.logger.warning(f"Could not register paired metadata: {error}")
|
||||
return positive_conditioning, negative_conditioning, positive, negative_text
|
||||
result = positive_conditioning, negative_conditioning, positive, negative_text
|
||||
record_runtime_context(
|
||||
type(self).__name__,
|
||||
text,
|
||||
prepend_text,
|
||||
append_text,
|
||||
positive,
|
||||
role="positive",
|
||||
)
|
||||
record_runtime_context(
|
||||
type(self).__name__, negative_text, "", "", negative_text, role="negative"
|
||||
)
|
||||
return result
|
||||
|
||||
@classmethod
|
||||
def IS_CHANGED(
|
||||
|
||||
+10
-1
@@ -6,6 +6,11 @@ without CLIP encoding, while maintaining all database and search features.
|
||||
import time
|
||||
from typing import Any, Dict, Optional, Tuple
|
||||
|
||||
if __package__:
|
||||
from .utils.execution_context_capture import record_runtime_context
|
||||
else:
|
||||
from utils.execution_context_capture import record_runtime_context
|
||||
|
||||
try:
|
||||
from comfy.comfy_types import IO, ComfyNodeABC, InputTypeDict
|
||||
except ImportError:
|
||||
@@ -227,7 +232,11 @@ class PromptManagerText(PromptManagerBase, ComfyNodeABC):
|
||||
)
|
||||
|
||||
self.logger.debug(f"Text processing completed: {final_text[:100]}...")
|
||||
return (final_text,)
|
||||
result = (final_text,)
|
||||
record_runtime_context(
|
||||
type(self).__name__, text, prepend_text, append_text, final_text
|
||||
)
|
||||
return result
|
||||
|
||||
@classmethod
|
||||
def IS_CHANGED(
|
||||
|
||||
@@ -166,3 +166,185 @@ test('rejected and removed-before-dequeue jobs create no persisted attempts', as
|
||||
}
|
||||
} finally { await control(request, 'release'); }
|
||||
});
|
||||
|
||||
function recipeGraph({ text = 'context positive', negative = '', kind = 'PromptManagerPair', prepend = '', append = '', dynamic = false } = {}) {
|
||||
const graph = {
|
||||
'10': { class_type: 'PMHistoryInputs', inputs: {} },
|
||||
'20': { class_type: kind, inputs: { text: dynamic ? ['11', 0] : text,
|
||||
prepend_text: prepend, append_text: append, ...(kind === 'PromptManagerText' ? {} : { clip: ['10', 0] }),
|
||||
...(kind === 'PromptManagerPair' ? { negative_text: negative } : {}) } },
|
||||
'30': { class_type: 'CLIPTextEncode', inputs: { text: negative, clip: ['10', 0] } },
|
||||
'40': { class_type: 'KSampler', inputs: { model: ['10', 1], latent_image: ['10', 2],
|
||||
positive: ['20', 0], negative: kind === 'PromptManagerPair' ? ['20', 1] : ['30', 0],
|
||||
seed: 1, steps: 1, cfg: 1, sampler_name: 'euler', scheduler: 'normal', denoise: 1 } },
|
||||
'50': { class_type: 'PMHistoryOutput', inputs: { value: ['40', 0] } },
|
||||
};
|
||||
if (kind === 'PromptManagerText') {
|
||||
graph['31'] = { class_type: 'CLIPTextEncode', inputs: { text: ['20', 0], clip: ['10', 0] } };
|
||||
graph['40'].inputs.positive = ['31', 0];
|
||||
}
|
||||
if (dynamic) graph['11'] = { class_type: 'PMHistoryCached', inputs: { value: text } };
|
||||
return graph;
|
||||
}
|
||||
|
||||
async function capture(execution, id, outcome = 'succeeded') {
|
||||
const [run] = await rows(execution, id, [outcome]);
|
||||
let result;
|
||||
await expect.poll(async () => {
|
||||
result = await readExecutionRuns(execution.databasePath, id);
|
||||
return result.captures[run.run_id].state;
|
||||
}, { timeout: 10000, message: 'Background writer persisted context finalization' }).not.toBe('not_captured');
|
||||
return result.captures[run.run_id];
|
||||
}
|
||||
|
||||
test('actual PM A,A,B jobs persist two combinations and one identical positive snapshot', async ({ execution }) => {
|
||||
const captures = [];
|
||||
const positiveSnapshotIds = [];
|
||||
for (const negative of ['negative A', 'negative A', 'negative B']) {
|
||||
const id = await submit(execution.request, recipeGraph({ negative }));
|
||||
await history(execution.request, id);
|
||||
captures.push(await capture(execution, id));
|
||||
const persisted = await readExecutionRuns(execution.databasePath, id);
|
||||
positiveSnapshotIds.push(persisted.snapshotIds[persisted.rows[0].run_id][0].positive_snapshot_id);
|
||||
}
|
||||
expect(new Set(positiveSnapshotIds).size).toBe(1);
|
||||
expect(captures.map(value => value.state)).toEqual(['complete', 'complete', 'complete']);
|
||||
const contexts = captures.map(value => value.contexts[0]);
|
||||
expect(new Set(contexts.map(value => value.context_id)).size).toBe(2);
|
||||
expect(new Set(contexts.map(value => JSON.stringify(value.positive))).size).toBe(1);
|
||||
expect(contexts[0].evidence.positive.effective.source).toBe('runtime_return');
|
||||
expect(contexts[1].evidence.positive.effective.source).toBe('host_cache_output');
|
||||
expect(contexts[1].evidence.positive.raw.source).toBe('graph_literal');
|
||||
expect(contexts[1].evidence.role_provenance).toBe('inferred');
|
||||
expect(contexts[2].negative.raw).toBe('negative B');
|
||||
});
|
||||
|
||||
test('fresh registered PM variants preserve exact raw whitespace and known empty negative', async ({ execution }) => {
|
||||
for (const kind of ['PromptManager', 'PromptManagerFocused', 'PromptManagerSplit', 'PromptManagerPair', 'PromptManagerText']) {
|
||||
const graph = recipeGraph({ kind, text: ` ${kind} e\u0301\n`, prepend: ' pre \t', append: '\n post ' });
|
||||
const id = await submit(execution.request, graph);
|
||||
await history(execution.request, id);
|
||||
const value = (await capture(execution, id)).contexts[0];
|
||||
expect(value.positive).toEqual({ raw: ` ${kind} e\u0301\n`, prepend: ' pre \t', append: '\n post ', effective: `pre ${kind} e\u0301\n post` });
|
||||
expect(value.negative.raw).toBe('');
|
||||
expect(value.negative.effective).toBe('');
|
||||
expect(value.evidence.positive.raw).toMatchObject({ provenance: 'recorded', source: 'runtime_return' });
|
||||
}
|
||||
});
|
||||
|
||||
test('fully cached text uses old effective output after external trigger configuration changes', async ({ execution }) => {
|
||||
const request = execution.request;
|
||||
const configure = suffix => request.post('/pm_execution_fixture/injection', { data: { suffix } });
|
||||
try {
|
||||
expect((await configure(' trigger-A')).ok()).toBe(true);
|
||||
const graph = recipeGraph({ text: 'cached trigger recipe', negative: 'trigger negative' });
|
||||
const first = await submit(request, graph);
|
||||
await history(request, first);
|
||||
const original = (await capture(execution, first)).contexts[0];
|
||||
expect(original.positive.effective).toBe('cached trigger recipe trigger-A');
|
||||
expect((await configure(' trigger-B')).ok()).toBe(true);
|
||||
const second = await submit(request, graph);
|
||||
const item = await history(request, second);
|
||||
expect(item.status.messages.find(([kind]) => kind === 'execution_cached')[1].nodes).toEqual(expect.arrayContaining(['20', '40', '50']));
|
||||
const repeated = (await capture(execution, second)).contexts[0];
|
||||
expect(repeated.context_id).toBe(original.context_id);
|
||||
expect(repeated.positive.effective).toBe(original.positive.effective);
|
||||
expect(repeated.evidence.positive.effective.source).toBe('host_cache_output');
|
||||
} finally { expect((await configure('')).ok()).toBe(true); }
|
||||
});
|
||||
|
||||
test('crossed Pair and Text negative roles remain independent declared contexts', async ({ execution }) => {
|
||||
const graph = recipeGraph({ text: 'pair one positive', negative: 'pair one negative' });
|
||||
graph['21'] = { class_type: 'PromptManagerPair', inputs: { text: 'pair two positive', negative_text: 'pair two negative', clip: ['10', 0] } };
|
||||
graph['22'] = { class_type: 'PromptManagerText', inputs: { text: 'text negative role' } };
|
||||
graph['32'] = { class_type: 'CLIPTextEncode', inputs: { text: ['22', 0], clip: ['10', 0] } };
|
||||
graph['40'].inputs.negative = ['21', 1];
|
||||
graph['41'] = structuredClone(graph['40']);
|
||||
graph['41'].inputs.positive = ['21', 0];
|
||||
graph['41'].inputs.negative = ['32', 0];
|
||||
graph['51'] = { class_type: 'PMHistoryOutput', inputs: { value: ['41', 0] } };
|
||||
const id = await submit(execution.request, graph);
|
||||
await history(execution.request, id);
|
||||
const result = await capture(execution, id);
|
||||
expect(result.contexts.map(value => [value.anchor_id, value.positive.raw, value.negative.raw])).toEqual([
|
||||
['40', 'pair one positive', 'pair two negative'], ['41', 'pair two positive', 'text negative role'],
|
||||
]);
|
||||
for (const value of result.contexts) expect(value.evidence.negative.effective.source).toBe('runtime_return');
|
||||
});
|
||||
|
||||
test('successful source capture survives downstream failure and real interruption', async ({ execution }) => {
|
||||
const request = execution.request;
|
||||
for (const mode of ['failure', 'gate']) {
|
||||
const graph = recipeGraph({ text: `source before ${mode}` });
|
||||
graph['50'].inputs.mode = mode;
|
||||
if (mode === 'gate') await control(request, 'hold');
|
||||
try {
|
||||
const id = await submit(request, graph);
|
||||
if (mode === 'gate') {
|
||||
await entered(request);
|
||||
expect((await request.post('/interrupt', { data: {} })).ok()).toBe(true);
|
||||
}
|
||||
await history(request, id, mode === 'gate' ? 'execution_interrupted' : 'execution_error');
|
||||
const result = await capture(execution, id, mode === 'gate' ? 'interrupted' : 'failed');
|
||||
expect(result.contexts[0].positive.effective).toBe(`source before ${mode}`);
|
||||
expect(result.contexts[0].evidence.positive.effective.source).toBe('runtime_return');
|
||||
} finally { await control(request, 'release'); }
|
||||
}
|
||||
});
|
||||
|
||||
test('cached dynamic raw stays unknown while literal affixes and effective cache text survive', async ({ execution }) => {
|
||||
const graph = recipeGraph({ dynamic: true, text: 'runtime dynamic recipe', prepend: ' pre ', append: ' post ' });
|
||||
const first = await submit(execution.request, graph);
|
||||
await history(execution.request, first);
|
||||
expect((await capture(execution, first)).contexts[0].positive.raw).toBe('runtime dynamic recipe');
|
||||
const second = await submit(execution.request, graph);
|
||||
await history(execution.request, second);
|
||||
const value = (await capture(execution, second)).contexts[0];
|
||||
expect(value.positive).toEqual({ raw: null, prepend: ' pre ', append: ' post ', effective: 'pre runtime dynamic recipe post' });
|
||||
expect(value.evidence.positive.raw.provenance).toBe('unknown');
|
||||
expect(value.evidence.positive.effective.source).toBe('host_cache_output');
|
||||
});
|
||||
|
||||
test('unsupported conditioning transforms and duplicate list invocations preserve uncertainty', async ({ execution }) => {
|
||||
const transformed = recipeGraph({ text: 'unsupported transformed recipe' });
|
||||
transformed['35'] = { class_type: 'PMHistoryTransform', inputs: { conditioning: ['20', 0] } };
|
||||
transformed['40'].inputs.positive = ['35', 0];
|
||||
const unsupported = await submit(execution.request, transformed);
|
||||
await history(execution.request, unsupported);
|
||||
const unknown = await capture(execution, unsupported);
|
||||
expect(unknown.state).toBe('partial');
|
||||
expect(unknown.contexts[0].positive.effective).toBeNull();
|
||||
const graph = recipeGraph({ text: 'duplicate runtime list' });
|
||||
graph['11'] = { class_type: 'PMHistoryList', inputs: { value: 'duplicate runtime list' } };
|
||||
graph['20'].inputs.text = ['11', 0];
|
||||
const id = await submit(execution.request, graph);
|
||||
await history(execution.request, id);
|
||||
const result = await capture(execution, id);
|
||||
expect(result.state).toBe('ambiguous');
|
||||
expect(result.reason).toBe('multiple_samples');
|
||||
expect(result.contexts[0].positive.effective).toBeNull();
|
||||
});
|
||||
|
||||
test('real executor Classic LRU RAM and disabled caches preserve honest evidence and usage', async ({ execution }) => {
|
||||
const request = execution.request;
|
||||
for (const mode of ['CLASSIC', 'LRU', 'RAM_PRESSURE', 'NONE']) {
|
||||
expect((await request.post('/pm_execution_fixture/cache', { data: { mode } })).ok()).toBe(true);
|
||||
const graph = recipeGraph({ text: `cache mode ${mode}`, negative: `cache mode negative ${mode}` });
|
||||
const results = [];
|
||||
for (let index = 0; index < 2; index++) {
|
||||
const id = await submit(request, graph);
|
||||
await history(request, id);
|
||||
results.push(await capture(execution, id));
|
||||
const persisted = await readExecutionRuns(execution.databasePath, id);
|
||||
const [run] = persisted.rows;
|
||||
expect(persisted.usage[run.run_id]).toEqual([
|
||||
{ role: 'positive', run_count: index + 1 },
|
||||
{ role: 'negative', run_count: 0 },
|
||||
{ role: 'pair', run_count: index + 1 },
|
||||
]);
|
||||
}
|
||||
expect(results[0].state).toBe('complete');
|
||||
expect(results[1].contexts[0].context_id).toBe(results[0].contexts[0].context_id);
|
||||
expect(results[1].contexts[0].evidence.positive.effective.source).toBe(mode === 'NONE' ? 'runtime_return' : 'host_cache_output');
|
||||
}
|
||||
});
|
||||
|
||||
@@ -3,6 +3,9 @@
|
||||
import threading
|
||||
import time
|
||||
|
||||
import nodes
|
||||
import execution
|
||||
|
||||
from aiohttp import web
|
||||
import comfy.model_management
|
||||
from server import PromptServer
|
||||
@@ -69,3 +72,129 @@ NODE_CLASS_MAPPINGS = {
|
||||
"PMHistoryFailure": PMHistoryFailure,
|
||||
"PMHistoryGate": PMHistoryGate,
|
||||
}
|
||||
|
||||
# Only external inference is replaced; PM classes and host execution remain real.
|
||||
|
||||
|
||||
class _Clip:
|
||||
def tokenize(self, text: str) -> str:
|
||||
return text
|
||||
|
||||
def encode_from_tokens_scheduled(self, tokens: str) -> list:
|
||||
return [[tokens, {}]]
|
||||
|
||||
|
||||
class PMHistoryInputs:
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls) -> dict:
|
||||
return {"required": {}}
|
||||
|
||||
RETURN_TYPES = ("CLIP", "MODEL", "LATENT")
|
||||
FUNCTION = "execute"
|
||||
CATEGORY = "testing/execution-history"
|
||||
|
||||
def execute(self) -> tuple:
|
||||
return (_Clip(), object(), {"samples": "owned deterministic latent"})
|
||||
|
||||
|
||||
class PMHistoryOutput:
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls) -> dict:
|
||||
return {
|
||||
"required": {"value": ("LATENT",)},
|
||||
"optional": {"mode": (["normal", "failure", "gate"],)},
|
||||
}
|
||||
|
||||
RETURN_TYPES = ()
|
||||
FUNCTION = "execute"
|
||||
CATEGORY = "testing/execution-history"
|
||||
OUTPUT_NODE = True
|
||||
|
||||
def execute(self, value: dict, mode: str = "normal") -> tuple:
|
||||
if mode == "failure":
|
||||
raise ValueError("owned downstream failure")
|
||||
if mode == "gate":
|
||||
PMHistoryGate().execute("owned downstream gate")
|
||||
return ()
|
||||
|
||||
|
||||
def _sample(self: object, **kwargs: object) -> tuple:
|
||||
return (kwargs["latent_image"],)
|
||||
|
||||
|
||||
nodes.KSampler.sample = _sample
|
||||
NODE_CLASS_MAPPINGS.update(
|
||||
{"PMHistoryInputs": PMHistoryInputs, "PMHistoryOutput": PMHistoryOutput}
|
||||
)
|
||||
|
||||
|
||||
class PMHistoryList(PMHistoryCached):
|
||||
OUTPUT_IS_LIST = (True,)
|
||||
|
||||
def execute(self, value: str) -> tuple:
|
||||
return ([value, value],)
|
||||
|
||||
|
||||
class PMHistoryTransform:
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls) -> dict:
|
||||
return {"required": {"conditioning": ("CONDITIONING",)}}
|
||||
|
||||
RETURN_TYPES = ("CONDITIONING",)
|
||||
FUNCTION = "execute"
|
||||
CATEGORY = "testing/execution-history"
|
||||
|
||||
def execute(self, conditioning: list) -> tuple:
|
||||
return (conditioning,)
|
||||
|
||||
|
||||
NODE_CLASS_MAPPINGS.update(
|
||||
{"PMHistoryList": PMHistoryList, "PMHistoryTransform": PMHistoryTransform}
|
||||
)
|
||||
|
||||
_INJECTION_ORIGINALS = {}
|
||||
|
||||
|
||||
@PromptServer.instance.routes.post("/pm_execution_fixture/injection")
|
||||
async def injection(request: web.Request) -> web.Response:
|
||||
suffix = (await request.json())["suffix"]
|
||||
if not isinstance(suffix, str) or len(suffix) > 128:
|
||||
raise web.HTTPBadRequest()
|
||||
for name in ("PromptManager", "PromptManagerText"):
|
||||
cls = nodes.NODE_CLASS_MAPPINGS[name]
|
||||
if name not in _INJECTION_ORIGINALS:
|
||||
_INJECTION_ORIGINALS[name] = cls._inject_lora_trigger_words
|
||||
cls._inject_lora_trigger_words = (
|
||||
(lambda self, text: text + suffix) if suffix else _INJECTION_ORIGINALS[name]
|
||||
)
|
||||
return web.json_response({"configured": True})
|
||||
|
||||
|
||||
# Apply a requested host cache configuration on its own worker before execution.
|
||||
# Retain only the mode scalar; never retain the worker/executor/cache objects.
|
||||
|
||||
_CACHE_MODE = None
|
||||
_EXECUTE = execution.PromptExecutor.execute
|
||||
|
||||
|
||||
def _execute_with_cache_mode(self: object, *args: object, **kwargs: object) -> object:
|
||||
global _CACHE_MODE
|
||||
if _CACHE_MODE is not None:
|
||||
self.cache_type = execution.CacheType[_CACHE_MODE]
|
||||
self.cache_args = {**self.cache_args, "lru": 100, "ram": 0}
|
||||
self.reset()
|
||||
_CACHE_MODE = None
|
||||
return _EXECUTE(self, *args, **kwargs)
|
||||
|
||||
|
||||
execution.PromptExecutor.execute = _execute_with_cache_mode
|
||||
|
||||
|
||||
@PromptServer.instance.routes.post("/pm_execution_fixture/cache")
|
||||
async def cache_mode(request: web.Request) -> web.Response:
|
||||
global _CACHE_MODE
|
||||
mode = (await request.json())["mode"]
|
||||
if mode not in ("CLASSIC", "LRU", "RAM_PRESSURE", "NONE"):
|
||||
raise web.HTTPBadRequest()
|
||||
_CACHE_MODE = mode
|
||||
return web.json_response({"configured": True})
|
||||
|
||||
@@ -4,8 +4,19 @@ import argparse
|
||||
import json
|
||||
from pathlib import Path
|
||||
import sqlite3
|
||||
import sys
|
||||
from types import SimpleNamespace
|
||||
from uuid import UUID
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
|
||||
from database.execution_contexts import ExecutionContextOperations # noqa: E402
|
||||
from database.execution_history import ExecutionHistoryOperations # noqa: E402
|
||||
|
||||
|
||||
class _ReadOnlyHistory(ExecutionContextOperations, ExecutionHistoryOperations):
|
||||
def __init__(self, connection: sqlite3.Connection) -> None:
|
||||
self.model = SimpleNamespace(get_connection=lambda: connection)
|
||||
|
||||
|
||||
def read_runs(database: str, job_id: str, session_id: str | None = None) -> dict:
|
||||
job_id = str(UUID(job_id))
|
||||
@@ -30,7 +41,45 @@ def read_runs(database: str, job_id: str, session_id: str | None = None) -> dict
|
||||
).fetchall()
|
||||
if len(rows) > 256:
|
||||
raise ValueError("Execution query exceeds the 256-row bound")
|
||||
return {"schemaAvailable": True, "rows": [dict(row) for row in rows]}
|
||||
reader = _ReadOnlyHistory(connection)
|
||||
captures = {
|
||||
row["run_id"]: reader.get_execution_contexts(row["run_id"]) for row in rows
|
||||
}
|
||||
usage = {}
|
||||
snapshot_ids = {}
|
||||
for run_id, capture in captures.items():
|
||||
usage[run_id] = []
|
||||
snapshot_ids[run_id] = []
|
||||
for context in capture["contexts"]:
|
||||
for role in ("positive", "negative"):
|
||||
prompt_id = context[f"{role}_prompt_id"]
|
||||
if prompt_id is not None:
|
||||
count = connection.execute(
|
||||
"SELECT run_count FROM prompts WHERE id=?", (prompt_id,)
|
||||
).fetchone()[0]
|
||||
usage[run_id].append({"role": role, "run_count": count})
|
||||
pair_id = context["pair_id"]
|
||||
if pair_id is not None:
|
||||
count = connection.execute(
|
||||
"SELECT run_count FROM prompt_pairs WHERE id=?", (pair_id,)
|
||||
).fetchone()[0]
|
||||
usage[run_id].append({"role": "pair", "run_count": count})
|
||||
snapshot_ids[run_id].append(
|
||||
dict(
|
||||
connection.execute(
|
||||
"SELECT positive_snapshot_id,negative_snapshot_id "
|
||||
"FROM execution_contexts WHERE context_id=?",
|
||||
(context["context_id"],),
|
||||
).fetchone()
|
||||
)
|
||||
)
|
||||
return {
|
||||
"schemaAvailable": True,
|
||||
"rows": [dict(row) for row in rows],
|
||||
"captures": captures,
|
||||
"usage": usage,
|
||||
"snapshotIds": snapshot_ids,
|
||||
}
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
@@ -0,0 +1,115 @@
|
||||
"""Ownership and accounting behavior for shared context payloads."""
|
||||
|
||||
import unittest
|
||||
|
||||
|
||||
class ContextBudgetTests(unittest.TestCase):
|
||||
def test_reservation_transfer_shrink_and_idempotent_release(self):
|
||||
from utils.execution_context_budget import ContextBudget
|
||||
|
||||
pool = ContextBudget(100)
|
||||
lease = pool.reserve(100)
|
||||
self.assertIsNotNone(lease)
|
||||
self.assertIsNone(pool.reserve(1))
|
||||
owner = lease.transfer()
|
||||
lease.release()
|
||||
self.assertEqual(pool.charged_bytes, 100)
|
||||
with self.assertRaises(ValueError):
|
||||
lease.transfer()
|
||||
with self.assertRaises(ValueError):
|
||||
lease.shrink_to(1)
|
||||
owner.shrink_to(60)
|
||||
self.assertEqual(owner.byte_count, 60)
|
||||
self.assertEqual(pool.charged_bytes, 60)
|
||||
for invalid in (-1, 61, True, 1.0):
|
||||
with self.assertRaises(ValueError):
|
||||
owner.shrink_to(invalid)
|
||||
owner.release()
|
||||
owner.release()
|
||||
self.assertEqual(pool.charged_bytes, 0)
|
||||
|
||||
def test_default_pool_and_invalid_reservations(self):
|
||||
from utils.execution_context_budget import ContextBudget
|
||||
|
||||
pool = ContextBudget()
|
||||
lease = pool.reserve(8388608)
|
||||
self.assertIsNotNone(lease)
|
||||
self.assertIsNone(pool.reserve(1))
|
||||
lease.release()
|
||||
for value in (-1, True, 1.0):
|
||||
with self.assertRaises(ValueError):
|
||||
pool.reserve(value)
|
||||
with self.assertRaises(ValueError):
|
||||
ContextBudget(value)
|
||||
|
||||
def test_only_pool_issued_leases_can_charge_or_release(self):
|
||||
from utils.execution_context_budget import ContextBudget, PayloadLease
|
||||
|
||||
pool = ContextBudget(100)
|
||||
with self.assertRaises(ValueError):
|
||||
PayloadLease(pool, 100)
|
||||
self.assertEqual(pool.charged_bytes, 0)
|
||||
|
||||
def test_measured_payload_field_context_and_attempt_boundaries(self):
|
||||
from dataclasses import replace
|
||||
from utils.execution_contexts import (
|
||||
ContextBundle,
|
||||
RecipeContext,
|
||||
PromptSnapshot,
|
||||
TextEvidence,
|
||||
bundle_payload_size,
|
||||
validate_bundle,
|
||||
)
|
||||
|
||||
def recipe(anchor, fourth):
|
||||
values = (
|
||||
"x" * 65536,
|
||||
"x" * 65536,
|
||||
"x" * 65536,
|
||||
"x" * fourth,
|
||||
"",
|
||||
"",
|
||||
"",
|
||||
"",
|
||||
)
|
||||
fields = tuple(
|
||||
TextEvidence(v, "recorded", "runtime_return", None) for v in values
|
||||
)
|
||||
return RecipeContext(
|
||||
anchor,
|
||||
PromptSnapshot(*fields[:4]),
|
||||
PromptSnapshot(*fields[4:]),
|
||||
"inferred",
|
||||
)
|
||||
|
||||
# 8 * (8 provenance + 14 source) + 8 role + 1 anchor = 185.
|
||||
context = recipe("1", 65351)
|
||||
bundle = ContextBundle((context,), "complete", "captured")
|
||||
self.assertEqual(bundle_payload_size(bundle), 262160)
|
||||
self.assertIsNone(validate_bundle(bundle))
|
||||
with self.assertRaises(ValueError):
|
||||
bundle_payload_size(replace(bundle, contexts=(recipe("1", 65352),)))
|
||||
field = replace(context.positive.raw, value="é" * 32769)
|
||||
bad = replace(context, positive=replace(context.positive, raw=field))
|
||||
with self.assertRaises(ValueError):
|
||||
bundle_payload_size(replace(bundle, contexts=(bad,)))
|
||||
attempt = replace(
|
||||
bundle, contexts=tuple(recipe(str(i), 65347) for i in range(4))
|
||||
)
|
||||
self.assertEqual(bundle_payload_size(attempt), 1048576)
|
||||
with self.assertRaises(ValueError):
|
||||
bundle_payload_size(
|
||||
replace(attempt, contexts=(recipe("0", 65348), *attempt.contexts[1:]))
|
||||
)
|
||||
small = replace(context, positive=None, negative=None)
|
||||
sixteen = replace(
|
||||
bundle, contexts=tuple(replace(small, anchor_id=str(i)) for i in range(16))
|
||||
)
|
||||
validate_bundle(sixteen)
|
||||
with self.assertRaises(ValueError):
|
||||
bundle_payload_size(
|
||||
replace(
|
||||
sixteen,
|
||||
contexts=(*sixteen.contexts, replace(small, anchor_id="17")),
|
||||
)
|
||||
)
|
||||
@@ -0,0 +1,573 @@
|
||||
"""Attempt fencing and bounded runtime evidence, without a running host."""
|
||||
|
||||
from contextvars import copy_context
|
||||
from dataclasses import replace
|
||||
import gc
|
||||
import sys
|
||||
import threading
|
||||
import types
|
||||
import unittest
|
||||
import weakref
|
||||
from unittest.mock import patch
|
||||
|
||||
from test_execution_context_graph import transformed_input_graph
|
||||
|
||||
from utils.execution_context_budget import ContextBudget
|
||||
from utils.execution_context_graph import SourceRef
|
||||
from utils.execution_contexts import PromptSnapshot, TextEvidence
|
||||
from utils.execution_history_events import RunStart
|
||||
from utils.execution_history_writer import HistoryWriterStatus
|
||||
|
||||
try:
|
||||
from utils import execution_context_capture as capture
|
||||
except ImportError:
|
||||
capture = None
|
||||
|
||||
SESSION = "11111111-1111-4111-8111-111111111111"
|
||||
EPOCH = "22222222-2222-4222-8222-222222222222"
|
||||
JOB = "33333333-3333-4333-8333-333333333333"
|
||||
|
||||
|
||||
def graph():
|
||||
return {
|
||||
"p": {
|
||||
"class_type": "PromptManagerPair",
|
||||
"inputs": {"text": "raw", "negative_text": "negative"},
|
||||
},
|
||||
"s": {
|
||||
"class_type": "KSampler",
|
||||
"inputs": {"positive": ["p", 0], "negative": ["p", 1]},
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def snapshot(raw=" raw ", effective="injected", prepend=" pre ", append=" post "):
|
||||
return PromptSnapshot(
|
||||
*(
|
||||
TextEvidence(v, "recorded", "runtime_return", None)
|
||||
for v in (raw, effective, prepend, append)
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
class CaptureTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.assertIsNotNone(capture, "attempt capture API is missing")
|
||||
self.pool = ContextBudget()
|
||||
self.status = HistoryWriterStatus(EPOCH, True, 0, 0, True, 3)
|
||||
self.manager = capture.ContextCapture(10, lambda: self.status, self.pool)
|
||||
self.addCleanup(self.manager.revoke_all)
|
||||
self.start = RunStart(SESSION, SESSION, 1, JOB, EPOCH, 0)
|
||||
|
||||
def begin(self, item=1, g=None):
|
||||
return self.manager.begin(
|
||||
replace(self.start, queue_item_id=item),
|
||||
3,
|
||||
graph() if g is None else g,
|
||||
None,
|
||||
)
|
||||
|
||||
def seal(self, cap):
|
||||
result = self.manager.seal(cap)
|
||||
if result.lease:
|
||||
self.addCleanup(result.lease.release)
|
||||
return result
|
||||
|
||||
def test_runtime_keeps_raw_affix_whitespace_and_injected_effective_text(self):
|
||||
cap = self.begin()
|
||||
self.manager.observe(cap, SourceRef("p", "PromptManagerPair", 0), 0, snapshot())
|
||||
result = self.seal(cap)
|
||||
side = result.bundle.contexts[0].positive
|
||||
self.assertEqual(
|
||||
[
|
||||
side.raw.value,
|
||||
side.effective.value,
|
||||
side.prepend.value,
|
||||
side.append.value,
|
||||
],
|
||||
[" raw ", "injected", " pre ", " post "],
|
||||
)
|
||||
self.assertIsNone(result.bundle.contexts[0].negative.effective.value)
|
||||
self.assertEqual(result.state, "partial")
|
||||
self.assertLess(self.pool.charged_bytes, 1048576)
|
||||
|
||||
def test_reused_uuid_late_async_and_same_epoch_generation_are_fenced(self):
|
||||
a = self.begin()
|
||||
inherited = copy_context()
|
||||
b = self.begin(2)
|
||||
inherited.run(
|
||||
self.manager.observe,
|
||||
a,
|
||||
SourceRef("p", "PromptManagerPair", 0),
|
||||
0,
|
||||
snapshot(),
|
||||
)
|
||||
self.assertIsNone(self.seal(a).bundle)
|
||||
self.status = replace(self.status, capture_generation=4)
|
||||
self.manager.observe(b, SourceRef("p", "PromptManagerPair", 0), 0, snapshot())
|
||||
self.assertIsNone(self.seal(b).bundle)
|
||||
self.assertEqual(self.pool.charged_bytes, 0)
|
||||
|
||||
def test_unbound_thread_and_forged_metadata_cannot_record(self):
|
||||
cap = self.begin()
|
||||
host = types.ModuleType("comfy_execution.utils")
|
||||
host.get_executing_context = lambda: types.SimpleNamespace(
|
||||
prompt_id=JOB, node_id="p", list_index=0
|
||||
)
|
||||
with patch.dict(sys.modules, {"comfy_execution.utils": host}):
|
||||
thread = threading.Thread(
|
||||
target=capture.record_runtime_context,
|
||||
args=("PromptManagerPair", "bad", "", "", "bad"),
|
||||
kwargs={"role": "positive"},
|
||||
)
|
||||
thread.start()
|
||||
thread.join(2)
|
||||
self.manager.observe(
|
||||
replace(cap, nonce="forged"),
|
||||
SourceRef("p", "PromptManagerPair", 0),
|
||||
0,
|
||||
snapshot(),
|
||||
)
|
||||
host.get_executing_context = lambda: types.SimpleNamespace(
|
||||
prompt_id="forged", node_id="p", list_index=0
|
||||
)
|
||||
capture.record_runtime_context(
|
||||
"PromptManagerPair", "bad", "", "", "bad", role="positive"
|
||||
)
|
||||
self.assertIsNone(self.seal(cap).bundle.contexts[0].positive.effective.value)
|
||||
|
||||
def test_new_dequeue_revokes_unfinished_attempt(self):
|
||||
a = self.begin()
|
||||
b = self.begin(2)
|
||||
self.assertEqual(capture.active_capture(), (self.manager, b))
|
||||
self.manager.revoke(a)
|
||||
self.assertEqual(capture.active_capture(), (self.manager, b))
|
||||
self.manager.revoke_all()
|
||||
self.assertIsNone(capture.active_capture())
|
||||
self.assertEqual(self.pool.charged_bytes, 0)
|
||||
|
||||
def test_equal_distinct_invocations_are_ambiguous_and_replay_is_idempotent(self):
|
||||
for count, state in ((1, "partial"), (64, "ambiguous"), (65, "ambiguous")):
|
||||
cap = self.begin(count)
|
||||
for i in range(count):
|
||||
self.manager.observe(
|
||||
cap, SourceRef("p", "PromptManagerPair", 0), i, snapshot()
|
||||
)
|
||||
self.manager.observe(
|
||||
cap, SourceRef("p", "PromptManagerPair", 0), i, snapshot()
|
||||
)
|
||||
result = self.seal(cap)
|
||||
self.assertEqual(result.state, state)
|
||||
if count > 1:
|
||||
self.assertIsNone(result.bundle.contexts[0].positive.effective.value)
|
||||
|
||||
def test_projection_never_retains_original_containers_and_budget_denial(self):
|
||||
class Envelope(dict):
|
||||
pass
|
||||
|
||||
g = Envelope(graph())
|
||||
ref = weakref.ref(g)
|
||||
cap = self.begin(g=g)
|
||||
del g
|
||||
gc.collect()
|
||||
self.assertIsNone(ref())
|
||||
self.manager.revoke(cap)
|
||||
small = capture.ContextCapture(10, lambda: self.status, ContextBudget(1))
|
||||
cap = small.begin(self.start, 3, graph(), None)
|
||||
self.assertEqual(small.seal(cap).reason, "budget_exhausted")
|
||||
|
||||
def test_crossed_roles_and_multiple_anchors_use_exact_slots(self):
|
||||
g = graph()
|
||||
g["t"] = {
|
||||
"class_type": "KSampler",
|
||||
"inputs": {"positive": ["p", 1], "negative": ["p", 0]},
|
||||
}
|
||||
cap = self.begin(g=g)
|
||||
self.manager.observe(
|
||||
cap, SourceRef("p", "PromptManagerPair", 0), 0, snapshot(effective="POS")
|
||||
)
|
||||
self.manager.observe(
|
||||
cap, SourceRef("p", "PromptManagerPair", 1), 0, snapshot(effective="NEG")
|
||||
)
|
||||
result = self.seal(cap)
|
||||
self.assertEqual(result.state, "complete")
|
||||
self.assertEqual(
|
||||
[
|
||||
(r.positive.effective.value, r.negative.effective.value)
|
||||
for r in result.bundle.contexts
|
||||
],
|
||||
[("POS", "NEG"), ("NEG", "POS")],
|
||||
)
|
||||
|
||||
def test_transformed_sources_cannot_replace_composed_text(self):
|
||||
g = graph()
|
||||
g["text"] = {
|
||||
"class_type": "StringConcatenate",
|
||||
"inputs": {"string_a": ["p", 2], "string_b": " suffix"},
|
||||
}
|
||||
g["clip"] = {"class_type": "CLIPTextEncode", "inputs": {"text": ["text", 0]}}
|
||||
g["s"]["inputs"]["positive"] = ["clip", 0]
|
||||
cap = self.begin(g=g)
|
||||
self.manager.observe(
|
||||
cap,
|
||||
SourceRef("p", "PromptManagerPair", 2),
|
||||
0,
|
||||
snapshot(effective="TRIGGER"),
|
||||
)
|
||||
result = self.seal(cap)
|
||||
self.assertIsNone(result.bundle.contexts[0].positive.effective.value)
|
||||
self.assertNotEqual(result.state, "complete")
|
||||
|
||||
def test_upstream_transform_preserves_final_pm_runtime_snapshot(self):
|
||||
for field in ("text", "prepend_text", "append_text"):
|
||||
for unresolved in (False, True):
|
||||
with self.subTest(field=field, unresolved=unresolved):
|
||||
cap = self.begin(g=transformed_input_graph(field, unresolved))
|
||||
self.host_context()
|
||||
capture.record_runtime_context(
|
||||
"PromptManagerPair",
|
||||
" actual raw ",
|
||||
" actual pre ",
|
||||
" actual post ",
|
||||
" actual injected effective ",
|
||||
role="positive",
|
||||
)
|
||||
side = self.seal(cap).bundle.contexts[0].positive
|
||||
for name, expected in (
|
||||
("raw", " actual raw "),
|
||||
("prepend", " actual pre "),
|
||||
("append", " actual post "),
|
||||
("effective", " actual injected effective "),
|
||||
):
|
||||
evidence = getattr(side, name)
|
||||
self.assertEqual(evidence.value, expected)
|
||||
self.assertEqual(evidence.provenance, "recorded")
|
||||
self.assertEqual(evidence.source, "runtime_return")
|
||||
|
||||
def test_unsupported_graph_has_no_supported_context(self):
|
||||
result = self.seal(self.begin(g={}))
|
||||
self.assertEqual(result.reason, "no_supported_context")
|
||||
|
||||
def host_context(self, kind="PromptManagerPair", index=0, job=JOB, node_id="p"):
|
||||
host = types.ModuleType("comfy_execution.utils")
|
||||
host.get_executing_context = lambda: types.SimpleNamespace(
|
||||
prompt_id=job, node_id=node_id, list_index=index
|
||||
)
|
||||
stack = self.enter_stack()
|
||||
stack.enter_context(patch.dict(sys.modules, {"comfy_execution.utils": host}))
|
||||
return host
|
||||
|
||||
def enter_stack(self):
|
||||
from contextlib import ExitStack
|
||||
|
||||
stack = ExitStack()
|
||||
self.addCleanup(stack.close)
|
||||
return stack
|
||||
|
||||
def test_nodes_record_only_successful_returns_and_save_failure_preserves_text(self):
|
||||
import logging
|
||||
from unittest.mock import Mock
|
||||
from prompt_manager import (
|
||||
PromptManager,
|
||||
PromptManagerFocused,
|
||||
PromptManagerSplit,
|
||||
)
|
||||
from prompt_manager_text import PromptManagerText
|
||||
from prompt_manager_pair import PromptManagerPair
|
||||
|
||||
for cls in (
|
||||
PromptManager,
|
||||
PromptManagerFocused,
|
||||
PromptManagerSplit,
|
||||
PromptManagerText,
|
||||
PromptManagerPair,
|
||||
):
|
||||
kind = cls.__name__
|
||||
g = graph()
|
||||
g["p"]["class_type"] = kind
|
||||
if kind != "PromptManagerPair":
|
||||
g["s"]["inputs"]["negative"] = ["p", 0]
|
||||
if kind == "PromptManagerText":
|
||||
g["c"] = {"class_type": "CLIPTextEncode", "inputs": {"text": ["p", 0]}}
|
||||
g["s"]["inputs"] = {"positive": ["c", 0], "negative": ["c", 0]}
|
||||
cap = self.begin(g=g)
|
||||
self.host_context(kind)
|
||||
node = cls.__new__(cls)
|
||||
node.db = Mock()
|
||||
node.db.get_prompt_by_hash.return_value = None
|
||||
node.db.save_prompt.side_effect = RuntimeError("offline")
|
||||
node.db.save_prompt_pair.side_effect = RuntimeError("offline")
|
||||
node.logger = logging.getLogger("test.capture.nodes")
|
||||
node.prompt_tracker = Mock()
|
||||
node.comfyui_integration = Mock()
|
||||
node._inject_lora_trigger_words = lambda text: text + " TRIGGER"
|
||||
clip = Mock()
|
||||
clip.tokenize.side_effect = lambda text: text
|
||||
clip.encode_from_tokens_scheduled.side_effect = lambda text: (
|
||||
"encoded",
|
||||
text,
|
||||
)
|
||||
kwargs = dict(text=" raw ", prepend_text=" pre ", append_text=" post ")
|
||||
if kind != "PromptManagerText":
|
||||
kwargs["clip"] = clip
|
||||
if kind == "PromptManagerPair":
|
||||
kwargs["negative_text"] = " neg "
|
||||
output = (
|
||||
node.process_text(**kwargs)
|
||||
if kind == "PromptManagerText"
|
||||
else node.encode_prompt(**kwargs)
|
||||
)
|
||||
result = self.seal(cap)
|
||||
side = result.bundle.contexts[0].positive
|
||||
self.assertEqual(side.effective.provenance, "recorded", kind)
|
||||
self.assertEqual(side.raw.value, " raw ")
|
||||
self.assertEqual(side.prepend.value, " pre ")
|
||||
self.assertEqual(side.append.value, " post ")
|
||||
self.assertEqual(
|
||||
side.effective.value,
|
||||
output[-2] if kind == "PromptManagerPair" else output[-1],
|
||||
)
|
||||
if kind == "PromptManagerPair":
|
||||
self.assertEqual(
|
||||
result.bundle.contexts[0].negative.effective.value, " neg "
|
||||
)
|
||||
if kind != "PromptManagerText":
|
||||
cap = self.begin(g=g)
|
||||
clip.encode_from_tokens_scheduled.side_effect = RuntimeError(
|
||||
"encode failed"
|
||||
)
|
||||
with self.assertRaises(RuntimeError):
|
||||
node.encode_prompt(**kwargs)
|
||||
self.assertIsNone(
|
||||
self.seal(cap).bundle.contexts[0].positive.effective.value
|
||||
)
|
||||
|
||||
def test_oversized_projection_and_observation_release_all_reservations(self):
|
||||
g = graph()
|
||||
g["p"]["inputs"].update(
|
||||
text="x" * 65536, prepend_text=" " * 65536, append_text=" " * 65536
|
||||
)
|
||||
cap = self.begin(g=g)
|
||||
self.manager.revoke(cap)
|
||||
self.assertEqual(self.pool.charged_bytes, 0)
|
||||
cap = self.begin()
|
||||
self.manager.observe(
|
||||
cap,
|
||||
SourceRef("p", "PromptManagerPair", 0),
|
||||
0,
|
||||
snapshot(effective="x" * 65537),
|
||||
)
|
||||
self.assertIsNone(self.seal(cap).bundle.contexts[0].positive.effective.value)
|
||||
|
||||
def test_recorder_rejects_wrong_class_slot_and_noninteger_invocation(self):
|
||||
for node_type, role, index, node_id in (
|
||||
("PromptManager", None, 0, "p"),
|
||||
("PromptManagerPair", "invalid", 0, "p"),
|
||||
("PromptManagerPair", "positive", True, "p"),
|
||||
("PromptManagerPair", "positive", -1, "p"),
|
||||
("PromptManagerPair", "positive", 0, "other"),
|
||||
):
|
||||
cap = self.begin()
|
||||
self.host_context(index=index, node_id=node_id)
|
||||
capture.record_runtime_context(node_type, "bad", "", "", "bad", role=role)
|
||||
self.assertIsNone(
|
||||
self.seal(cap).bundle.contexts[0].positive.effective.value
|
||||
)
|
||||
|
||||
def test_async_inherited_capability_after_await_cannot_record_new_attempt(self):
|
||||
import asyncio
|
||||
|
||||
async def scenario():
|
||||
old = self.begin()
|
||||
entered, release = asyncio.Event(), asyncio.Event()
|
||||
|
||||
async def late():
|
||||
entered.set()
|
||||
await release.wait()
|
||||
capture.record_runtime_context(
|
||||
"PromptManagerPair", "late", "", "", "late", role="positive"
|
||||
)
|
||||
|
||||
self.host_context()
|
||||
task = asyncio.create_task(late())
|
||||
await entered.wait()
|
||||
new = self.begin(2)
|
||||
release.set()
|
||||
await task
|
||||
self.assertIsNone(self.seal(old).bundle)
|
||||
self.assertIsNone(
|
||||
self.seal(new).bundle.contexts[0].positive.effective.value
|
||||
)
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
def test_writer_restore_or_close_revokes_callback_paused_during_encoding(self):
|
||||
import asyncio
|
||||
|
||||
from utils.execution_history_writer import HistoryWriter
|
||||
|
||||
for action in ("invalidate", "close"):
|
||||
with self.subTest(action=action):
|
||||
pool = ContextBudget()
|
||||
current = [HistoryWriterStatus(EPOCH, True, 0, 0, True, 3)]
|
||||
manager = capture.ContextCapture(10, lambda: current[0], pool)
|
||||
self.addCleanup(manager.revoke_all)
|
||||
writer = HistoryWriter(
|
||||
lambda: self.fail("This callback test must not open storage"),
|
||||
context_budget=pool,
|
||||
on_invalidation=manager.revoke_all,
|
||||
)
|
||||
self.addCleanup(writer.close)
|
||||
self.host_context()
|
||||
|
||||
async def scenario():
|
||||
old = manager.begin(self.start, 3, graph(), None)
|
||||
manager.observe(
|
||||
old, SourceRef("p", "PromptManagerPair", 0), 0, snapshot()
|
||||
)
|
||||
entered, release = asyncio.Event(), asyncio.Event()
|
||||
|
||||
async def encoding_return():
|
||||
entered.set()
|
||||
await release.wait()
|
||||
capture.record_runtime_context(
|
||||
"PromptManagerPair", "late", "", "", "late", role="positive"
|
||||
)
|
||||
|
||||
task = asyncio.create_task(encoding_return())
|
||||
await entered.wait()
|
||||
self.assertGreater(pool.charged_bytes, 0)
|
||||
if action == "invalidate":
|
||||
writer.invalidate()
|
||||
else:
|
||||
self.assertTrue(writer.close())
|
||||
self.assertEqual(pool.charged_bytes, 0)
|
||||
self.assertIsNone(capture.active_capture())
|
||||
new_epoch = "44444444-4444-4444-8444-444444444444"
|
||||
current[0] = HistoryWriterStatus(new_epoch, True, 0, 0, True, 4)
|
||||
new = manager.begin(
|
||||
replace(
|
||||
self.start,
|
||||
run_id=JOB,
|
||||
queue_item_id=2,
|
||||
storage_epoch=new_epoch,
|
||||
),
|
||||
4,
|
||||
graph(),
|
||||
None,
|
||||
)
|
||||
release.set()
|
||||
await task
|
||||
self.assertIsNone(manager.seal(old).bundle)
|
||||
sealed = manager.seal(new)
|
||||
try:
|
||||
self.assertIsNone(
|
||||
sealed.bundle.contexts[0].positive.effective.value
|
||||
)
|
||||
finally:
|
||||
if sealed.lease:
|
||||
sealed.lease.release()
|
||||
self.assertEqual(pool.charged_bytes, 0)
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
def test_oversized_runtime_field_is_unknown_without_erasing_other_fields(self):
|
||||
cap = self.begin()
|
||||
self.host_context()
|
||||
capture.record_runtime_context(
|
||||
"PromptManagerPair",
|
||||
" actual ",
|
||||
" pre ",
|
||||
" post ",
|
||||
"x" * 65537,
|
||||
role="positive",
|
||||
)
|
||||
result = self.seal(cap)
|
||||
side = result.bundle.contexts[0].positive
|
||||
self.assertEqual(side.raw.value, " actual ")
|
||||
self.assertEqual(side.prepend.value, " pre ")
|
||||
self.assertIsNone(side.effective.value)
|
||||
self.assertEqual(side.effective.reason, "limit_reached")
|
||||
self.assertEqual(result.reason, "field_limit")
|
||||
|
||||
def test_mixed_cache_provenance_cannot_promote_inferred_pm_effective(self):
|
||||
cap = self.begin()
|
||||
field = TextEvidence("literal", "inferred", "graph_literal", None)
|
||||
cached = TextEvidence("cached TRIGGER", "recorded", "host_cache_output", None)
|
||||
self.manager.observe(
|
||||
cap,
|
||||
SourceRef("p", "PromptManagerPair", 0),
|
||||
0,
|
||||
PromptSnapshot(field, cached, field, field),
|
||||
)
|
||||
side = self.seal(cap).bundle.contexts[0].positive
|
||||
self.assertEqual(side.raw.provenance, "inferred")
|
||||
self.assertEqual(side.effective.value, "cached TRIGGER")
|
||||
cap = self.begin(2)
|
||||
self.manager.observe(
|
||||
cap,
|
||||
SourceRef("p", "PromptManagerPair", 0),
|
||||
0,
|
||||
PromptSnapshot(field, field, field, field),
|
||||
)
|
||||
self.assertIsNone(self.seal(cap).bundle.contexts[0].positive.effective.value)
|
||||
|
||||
def test_restore_and_close_release_weakly_bound_payload_immediately(self):
|
||||
cap = self.begin()
|
||||
binding = copy_context()
|
||||
self.manager.revoke_all()
|
||||
self.assertEqual(self.pool.charged_bytes, 0)
|
||||
self.assertIsNone(binding.run(capture.active_capture))
|
||||
self.manager.observe(cap, SourceRef("p", "PromptManagerPair", 0), 0, snapshot())
|
||||
self.assertIsNone(self.seal(cap).bundle)
|
||||
owner = capture.ContextCapture(11, lambda: self.status, self.pool)
|
||||
owner.begin(self.start, 3, graph(), None)
|
||||
binding = copy_context()
|
||||
owner.revoke_all()
|
||||
ref = weakref.ref(owner)
|
||||
del owner
|
||||
gc.collect()
|
||||
self.assertIsNone(ref())
|
||||
self.assertIsNone(binding.run(capture.active_capture))
|
||||
|
||||
def test_conflicting_replay_stays_ambiguous_after_limited_other_source(self):
|
||||
cap = self.begin()
|
||||
source = SourceRef("p", "PromptManagerPair", 0)
|
||||
self.manager.observe(cap, source, 0, snapshot(effective="first"))
|
||||
self.manager.observe(cap, source, 0, snapshot(effective="different"))
|
||||
self.manager.observe(
|
||||
cap,
|
||||
SourceRef("p", "PromptManagerPair", 1),
|
||||
0,
|
||||
snapshot(effective="x" * 65537),
|
||||
)
|
||||
sealed = self.seal(cap)
|
||||
self.assertEqual(sealed.state, "ambiguous")
|
||||
self.assertIsNone(sealed.bundle.contexts[0].positive.effective.value)
|
||||
|
||||
def test_capture_owner_bounds_unavailable_attempts_to_128(self):
|
||||
owner = capture.ContextCapture(12, lambda: self.status, ContextBudget(0))
|
||||
self.addCleanup(owner.revoke_all)
|
||||
capabilities = [
|
||||
copy_context().run(
|
||||
owner.begin, replace(self.start, queue_item_id=i), 3, {}, None
|
||||
)
|
||||
for i in range(129)
|
||||
]
|
||||
self.assertEqual(sum(owner.valid(cap) for cap in capabilities), 128)
|
||||
|
||||
def test_revocation_discards_owned_samples_before_releasing_budget(self):
|
||||
cap = self.begin()
|
||||
value = snapshot()
|
||||
ref = weakref.ref(value.raw)
|
||||
self.manager.observe(cap, SourceRef("p", "PromptManagerPair", 0), 0, value)
|
||||
del value
|
||||
lease = self.manager._attempts[cap.nonce].lease
|
||||
original = lease.release
|
||||
|
||||
def release():
|
||||
self.assertIsNone(ref(), "sample was uncharged while still owned")
|
||||
original()
|
||||
|
||||
with patch.object(lease, "release", release):
|
||||
self.manager.revoke(cap)
|
||||
@@ -0,0 +1,613 @@
|
||||
"""Immutable scalar capture, deduplication, and conservative navigation hints."""
|
||||
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from contextlib import contextmanager
|
||||
from uuid import uuid4
|
||||
from dataclasses import replace
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
import textwrap
|
||||
from unittest.mock import patch
|
||||
import unittest
|
||||
|
||||
from database.execution_history import HistoryEpochChanged
|
||||
from database.execution_context_schema import context_schema_error
|
||||
from utils.execution_contexts import (
|
||||
ContextBundle,
|
||||
PromptSnapshot,
|
||||
RecipeContext,
|
||||
TextEvidence,
|
||||
)
|
||||
|
||||
try:
|
||||
from tests import test_execution_history_database as history_tests
|
||||
except ImportError:
|
||||
import test_execution_history_database as history_tests
|
||||
|
||||
|
||||
def known(value):
|
||||
return TextEvidence(value, "recorded", "runtime_return", None)
|
||||
|
||||
|
||||
def snapshot(raw):
|
||||
return PromptSnapshot(known(raw), known(raw), known(""), known(""))
|
||||
|
||||
|
||||
def bundle(negative="negative", positive="positive", anchor="4"):
|
||||
return ContextBundle(
|
||||
(RecipeContext(anchor, snapshot(positive), snapshot(negative), "inferred"),),
|
||||
"complete",
|
||||
"captured",
|
||||
)
|
||||
|
||||
|
||||
class ContextDatabaseTests(unittest.TestCase):
|
||||
setUp = history_tests.HistoryDatabaseTests.setUp
|
||||
close_databases = history_tests.HistoryDatabaseTests.close_databases
|
||||
database = history_tests.HistoryDatabaseTests.database
|
||||
start = history_tests.HistoryDatabaseTests.start
|
||||
finish = history_tests.HistoryDatabaseTests.finish
|
||||
|
||||
def capture(self, db, item=0, capture=None):
|
||||
start = self.start(db, item)
|
||||
db.write_execution_start(start)
|
||||
db.write_execution_finish(self.finish(start))
|
||||
self.assertTrue(
|
||||
db.write_execution_contexts(
|
||||
start.run_id, start.storage_epoch, capture or bundle()
|
||||
)
|
||||
)
|
||||
return start
|
||||
|
||||
def counts(self, db):
|
||||
conn = db.model.get_connection()
|
||||
return tuple(
|
||||
conn.execute(f"SELECT COUNT(*) FROM {table}").fetchone()[0]
|
||||
for table in (
|
||||
"execution_runs",
|
||||
"execution_contexts",
|
||||
"execution_prompt_snapshots",
|
||||
"execution_context_capture",
|
||||
)
|
||||
)
|
||||
|
||||
def test_finish_capture_uses_stored_parent_identity(self):
|
||||
db = self.database()
|
||||
start = replace(self.start(db), run_id=str(uuid4()))
|
||||
db.write_execution_start(start)
|
||||
finish = self.finish(start)
|
||||
self.assertTrue(db.write_execution_contexts_for_finish(finish, bundle()))
|
||||
self.assertEqual(db.get_execution_contexts(start.run_id)["state"], "complete")
|
||||
missing = replace(finish, queue_item_id=999)
|
||||
self.assertFalse(db.write_execution_contexts_for_finish(missing, bundle()))
|
||||
self.assertEqual(self.counts(db), (1, 1, 2, 1))
|
||||
|
||||
def test_finish_generation_guard_encloses_commit_and_rollback(self):
|
||||
db = self.database()
|
||||
start = self.start(db)
|
||||
db.write_execution_start(start)
|
||||
finish = self.finish(start)
|
||||
observed = []
|
||||
|
||||
@contextmanager
|
||||
def guard():
|
||||
observed.append(db.model.get_connection().in_transaction)
|
||||
try:
|
||||
yield True
|
||||
finally:
|
||||
observed.append(db.model.get_connection().in_transaction)
|
||||
|
||||
self.assertTrue(
|
||||
db.write_execution_contexts_for_finish(
|
||||
finish, bundle(), generation_guard=guard
|
||||
)
|
||||
)
|
||||
self.assertEqual(observed, [True, False])
|
||||
observed.clear()
|
||||
other = replace(self.start(db, 1), run_id=str(uuid4()))
|
||||
db.write_execution_start(other)
|
||||
with patch(
|
||||
"database.execution_contexts._insert_exact",
|
||||
side_effect=ValueError("collision"),
|
||||
):
|
||||
with self.assertRaises(ValueError):
|
||||
db.write_execution_contexts_for_finish(
|
||||
self.finish(other), bundle(), generation_guard=guard
|
||||
)
|
||||
self.assertEqual(observed, [True, False])
|
||||
self.assertEqual(
|
||||
db.get_execution_contexts(other.run_id)["state"], "not_captured"
|
||||
)
|
||||
|
||||
def test_finish_generation_guard_rejects_without_capture(self):
|
||||
db = self.database()
|
||||
start = self.start(db)
|
||||
db.write_execution_start(start)
|
||||
|
||||
@contextmanager
|
||||
def stale():
|
||||
yield False
|
||||
|
||||
self.assertFalse(
|
||||
db.write_execution_contexts_for_finish(
|
||||
self.finish(start), bundle(), generation_guard=stale
|
||||
)
|
||||
)
|
||||
self.assertEqual(self.counts(db), (1, 0, 0, 0))
|
||||
|
||||
def test_a_a_b_cardinality(self):
|
||||
db = self.database()
|
||||
self.assertTrue(
|
||||
callable(getattr(db, "write_execution_contexts", None)),
|
||||
"PromptDatabase must expose immutable context persistence",
|
||||
)
|
||||
starts = [self.capture(db, i, bundle(n)) for i, n in enumerate(("A", "A", "B"))]
|
||||
self.assertEqual(self.counts(db), (3, 2, 3, 3))
|
||||
reads = [db.get_execution_contexts(s.run_id) for s in starts]
|
||||
self.assertEqual(
|
||||
reads[0]["contexts"][0]["context_id"], reads[1]["contexts"][0]["context_id"]
|
||||
)
|
||||
self.assertNotEqual(
|
||||
reads[0]["contexts"][0]["context_id"], reads[2]["contexts"][0]["context_id"]
|
||||
)
|
||||
self.assertIsNone(context_schema_error(db.model.get_connection()))
|
||||
|
||||
def test_parallel_identical_inserts_deduplicate(self):
|
||||
db = self.database()
|
||||
starts = [self.start(db, i) for i in range(8)]
|
||||
for start in starts:
|
||||
db.write_execution_start(start)
|
||||
|
||||
def write(start):
|
||||
return db.write_execution_contexts(
|
||||
start.run_id, start.storage_epoch, bundle()
|
||||
)
|
||||
|
||||
with ThreadPoolExecutor(max_workers=4) as pool:
|
||||
self.assertEqual(list(pool.map(write, starts)), [True] * 8)
|
||||
self.assertEqual(self.counts(db), (8, 1, 2, 8))
|
||||
|
||||
def test_replay_freezes_first_capture(self):
|
||||
db = self.database()
|
||||
start = self.capture(db)
|
||||
original = db.get_execution_contexts(start.run_id)
|
||||
counts = self.counts(db)
|
||||
self.assertTrue(
|
||||
db.write_execution_contexts(start.run_id, start.storage_epoch, bundle())
|
||||
)
|
||||
db.write_execution_finish(self.finish(start))
|
||||
self.assertEqual(db.get_execution_contexts(start.run_id), original)
|
||||
self.assertTrue(
|
||||
db.write_execution_contexts(
|
||||
start.run_id, start.storage_epoch, bundle("changed")
|
||||
)
|
||||
)
|
||||
read = db.get_execution_contexts(start.run_id)
|
||||
self.assertEqual(
|
||||
(read["state"], read["reason"]), ("conflicting", "conflicting")
|
||||
)
|
||||
self.assertEqual(read["contexts"], original["contexts"])
|
||||
self.assertEqual(self.counts(db), counts)
|
||||
db.write_execution_contexts(start.run_id, start.storage_epoch, bundle())
|
||||
self.assertEqual(db.get_execution_contexts(start.run_id), read)
|
||||
|
||||
def test_evidence_replay_and_anchor_changes_conflict_without_inserting(self):
|
||||
for change in ("evidence", "anchor", "status"):
|
||||
with self.subTest(change=change):
|
||||
db = self.database(str(Path(self.temp.name) / f"{change}.db"))
|
||||
start = self.capture(db)
|
||||
original = db.get_execution_contexts(start.run_id)["contexts"]
|
||||
candidate = bundle()
|
||||
if change == "evidence":
|
||||
ctx = candidate.contexts[0]
|
||||
candidate = replace(
|
||||
candidate,
|
||||
contexts=(
|
||||
replace(
|
||||
ctx,
|
||||
positive=replace(
|
||||
ctx.positive,
|
||||
raw=TextEvidence(
|
||||
"positive", "inferred", "graph_literal", None
|
||||
),
|
||||
),
|
||||
),
|
||||
),
|
||||
)
|
||||
elif change == "anchor":
|
||||
candidate = bundle(anchor="5")
|
||||
else:
|
||||
candidate = replace(candidate, state="partial", reason="partial")
|
||||
db.write_execution_contexts(
|
||||
start.run_id, start.storage_epoch, candidate
|
||||
)
|
||||
read = db.get_execution_contexts(start.run_id)
|
||||
self.assertEqual(read["state"], "conflicting")
|
||||
self.assertEqual(read["contexts"], original)
|
||||
self.assertEqual(self.counts(db), (1, 1, 2, 1))
|
||||
|
||||
def test_hash_collision_rejected_without_partial_insert(self):
|
||||
db = self.database()
|
||||
start = self.capture(db)
|
||||
original = db.get_execution_contexts(start.run_id)
|
||||
second = self.start(db, 1)
|
||||
db.write_execution_start(second)
|
||||
# A collision must compare stored scalar content, then roll back all rows.
|
||||
from utils.execution_contexts import content_digest
|
||||
|
||||
collided = content_digest("text", ("negative",))
|
||||
|
||||
def digest(kind, fields):
|
||||
return (
|
||||
collided
|
||||
if kind == "text" and fields == ("changed",)
|
||||
else content_digest(kind, fields)
|
||||
)
|
||||
|
||||
before = tuple(db.model.get_connection().iterdump())
|
||||
with patch("database.execution_contexts.content_digest", side_effect=digest):
|
||||
with self.assertRaises(ValueError):
|
||||
db.write_execution_contexts(
|
||||
second.run_id,
|
||||
second.storage_epoch,
|
||||
bundle("changed", "new-positive"),
|
||||
)
|
||||
self.assertEqual(tuple(db.model.get_connection().iterdump()), before)
|
||||
self.assertEqual(db.get_execution_contexts(start.run_id), original)
|
||||
|
||||
def test_missing_parent_and_epoch_mismatch_insert_nothing(self):
|
||||
db = self.database()
|
||||
start = self.start(db)
|
||||
self.assertEqual(
|
||||
db.get_execution_contexts(start.run_id)["state"], "not_captured"
|
||||
)
|
||||
self.assertFalse(
|
||||
db.write_execution_contexts(start.run_id, start.storage_epoch, bundle())
|
||||
)
|
||||
self.assertEqual(self.counts(db), (0, 0, 0, 0))
|
||||
db.write_execution_start(start)
|
||||
with self.assertRaises(HistoryEpochChanged):
|
||||
db.write_execution_contexts(
|
||||
start.run_id, "11111111-1111-4111-8111-111111111111", bundle()
|
||||
)
|
||||
with db.model.get_connection() as conn:
|
||||
conn.execute(
|
||||
"UPDATE execution_runs SET storage_epoch=? WHERE run_id=?",
|
||||
("11111111-1111-4111-8111-111111111111", start.run_id),
|
||||
)
|
||||
self.assertFalse(
|
||||
db.write_execution_contexts(start.run_id, start.storage_epoch, bundle())
|
||||
)
|
||||
self.assertEqual(self.counts(db), (1, 0, 0, 0))
|
||||
|
||||
def test_invalid_bundle_has_no_partial_rows(self):
|
||||
db = self.database()
|
||||
start = self.start(db)
|
||||
db.write_execution_start(start)
|
||||
for candidate in (
|
||||
replace(bundle(), reason="arbitrary"),
|
||||
replace(bundle(), contexts=()),
|
||||
replace(bundle(), contexts=(RecipeContext("4", None, None, "inferred"),)),
|
||||
):
|
||||
with self.subTest(candidate=candidate), self.assertRaises(ValueError):
|
||||
db.write_execution_contexts(
|
||||
start.run_id, start.storage_epoch, candidate
|
||||
)
|
||||
self.assertEqual(self.counts(db), (1, 0, 0, 0))
|
||||
|
||||
def test_unknown_empty_unicode_and_evidence_round_trip(self):
|
||||
db = self.database()
|
||||
unknown = TextEvidence(None, "unknown", "unknown", "missing")
|
||||
ctx = RecipeContext(
|
||||
"3",
|
||||
PromptSnapshot(known(" e\u0301 \n"), unknown, known(""), known("é")),
|
||||
None,
|
||||
"inferred",
|
||||
)
|
||||
start = self.capture(db, capture=ContextBundle((ctx,), "partial", "missing"))
|
||||
read = db.get_execution_contexts(start.run_id)["contexts"][0]
|
||||
self.assertEqual(
|
||||
read["positive"],
|
||||
{"raw": " e\u0301 \n", "effective": None, "prepend": "", "append": "é"},
|
||||
)
|
||||
self.assertIsNone(read["negative"])
|
||||
self.assertEqual(
|
||||
read["evidence"]["positive"]["effective"],
|
||||
{"provenance": "unknown", "source": "unknown", "reason": "missing"},
|
||||
)
|
||||
self.assertEqual(
|
||||
set(read["evidence"]),
|
||||
{"format_version", "role_provenance", "positive", "negative"},
|
||||
)
|
||||
self.assertIsNone(context_schema_error(db.model.get_connection()))
|
||||
|
||||
def test_library_mutations_and_id_reuse_do_not_rewrite_history(self):
|
||||
db = self.database()
|
||||
conn = db.model.get_connection()
|
||||
with conn:
|
||||
conn.execute(
|
||||
"INSERT INTO prompts(id,text) VALUES(1,'positive'),(2,'negative')"
|
||||
)
|
||||
conn.execute(
|
||||
"INSERT INTO prompt_pairs(id,positive_prompt_id,negative_prompt_id) "
|
||||
"VALUES(7,1,2)"
|
||||
)
|
||||
start = self.capture(db)
|
||||
original = db.get_execution_contexts(start.run_id)["contexts"][0]
|
||||
self.assertEqual(
|
||||
(
|
||||
original["positive_prompt_id"],
|
||||
original["negative_prompt_id"],
|
||||
original["pair_id"],
|
||||
),
|
||||
(1, 2, 7),
|
||||
)
|
||||
db.archive_prompt_pair(7)
|
||||
self.assertEqual(
|
||||
db.get_execution_contexts(start.run_id)["contexts"][0], original
|
||||
)
|
||||
db.delete_prompt_pair(7)
|
||||
db.update_prompt_text(1, "edited")
|
||||
read = db.get_execution_contexts(start.run_id)["contexts"][0]
|
||||
self.assertIsNone(read["positive_prompt_id"])
|
||||
self.assertIsNone(read["pair_id"])
|
||||
db.delete_prompt(2)
|
||||
with conn:
|
||||
conn.execute("INSERT INTO prompts(id,text) VALUES(2,'reused')")
|
||||
self.assertIsNone(
|
||||
db.get_execution_contexts(start.run_id)["contexts"][0]["negative_prompt_id"]
|
||||
)
|
||||
self.assertEqual(db.prune_orphaned_prompts(), 2)
|
||||
after = db.get_execution_contexts(start.run_id)["contexts"][0]
|
||||
for key in ("positive", "negative", "evidence", "context_id"):
|
||||
self.assertEqual(after[key], original[key])
|
||||
self.assertEqual(conn.execute("SELECT COUNT(*) FROM prompts").fetchone()[0], 0)
|
||||
self.assertIsNone(context_schema_error(conn))
|
||||
|
||||
def test_hints_require_exact_unique_raw_text_and_never_refresh(self):
|
||||
db = self.database()
|
||||
conn = db.model.get_connection()
|
||||
with conn:
|
||||
conn.execute(
|
||||
"INSERT INTO prompts(id,text) VALUES(1,' positive "
|
||||
"'),(2,'negative'),(3,'negative')"
|
||||
)
|
||||
start = self.capture(db)
|
||||
read = db.get_execution_contexts(start.run_id)["contexts"][0]
|
||||
self.assertIsNone(read["positive_prompt_id"])
|
||||
self.assertIsNone(read["negative_prompt_id"])
|
||||
with conn:
|
||||
conn.execute("DELETE FROM prompts WHERE id=3")
|
||||
conn.execute("UPDATE prompts SET text='positive' WHERE id=1")
|
||||
db.write_execution_contexts(start.run_id, start.storage_epoch, bundle())
|
||||
self.assertEqual(db.get_execution_contexts(start.run_id)["contexts"], [read])
|
||||
fresh = self.capture(db, 1)
|
||||
self.assertEqual(
|
||||
db.get_execution_contexts(fresh.run_id)["contexts"][0][
|
||||
"positive_prompt_id"
|
||||
],
|
||||
1,
|
||||
)
|
||||
with conn:
|
||||
conn.execute("INSERT INTO prompts(id,text) VALUES(3,'positive')")
|
||||
self.assertIsNone(
|
||||
db.get_execution_contexts(fresh.run_id)["contexts"][0]["positive_prompt_id"]
|
||||
)
|
||||
|
||||
def test_null_negative_pair_only_matches_known_exact_empty(self):
|
||||
db = self.database()
|
||||
with db.model.get_connection() as conn:
|
||||
conn.execute("INSERT INTO prompts(id,text) VALUES(1,'positive')")
|
||||
conn.execute("INSERT INTO prompt_pairs(id,positive_prompt_id) VALUES(7,1)")
|
||||
for item, raw, expected in ((0, "", 7), (1, " ", None), (2, None, None)):
|
||||
candidate = bundle(negative=raw or "")
|
||||
if raw is None:
|
||||
ctx = candidate.contexts[0]
|
||||
candidate = replace(
|
||||
candidate,
|
||||
contexts=(
|
||||
replace(
|
||||
ctx,
|
||||
negative=replace(
|
||||
ctx.negative,
|
||||
raw=TextEvidence(None, "unknown", "unknown", "missing"),
|
||||
),
|
||||
),
|
||||
),
|
||||
state="partial",
|
||||
reason="partial",
|
||||
)
|
||||
start = self.capture(db, item, candidate)
|
||||
self.assertEqual(
|
||||
db.get_execution_contexts(start.run_id)["contexts"][0]["pair_id"],
|
||||
expected,
|
||||
)
|
||||
|
||||
def test_snapshot_and_context_collisions_roll_back_all_new_rows(self):
|
||||
from utils.execution_contexts import content_digest
|
||||
|
||||
for kind in ("snapshot", "context"):
|
||||
with self.subTest(kind=kind):
|
||||
db = self.database(str(Path(self.temp.name) / f"{kind}.db"))
|
||||
first = self.capture(db)
|
||||
old_context = db.get_execution_contexts(first.run_id)["contexts"][0][
|
||||
"context_id"
|
||||
]
|
||||
old_snapshot = (
|
||||
db.model.get_connection()
|
||||
.execute(
|
||||
"SELECT negative_snapshot_id FROM execution_contexts WHERE "
|
||||
"context_id=?",
|
||||
(old_context,),
|
||||
)
|
||||
.fetchone()[0]
|
||||
)
|
||||
second = self.start(db, 1)
|
||||
db.write_execution_start(second)
|
||||
before = tuple(db.model.get_connection().iterdump())
|
||||
|
||||
def digest(current_kind, fields):
|
||||
if current_kind == kind:
|
||||
return old_snapshot if kind == "snapshot" else old_context
|
||||
return content_digest(current_kind, fields)
|
||||
|
||||
with (
|
||||
patch(
|
||||
"database.execution_contexts.content_digest", side_effect=digest
|
||||
),
|
||||
self.assertRaises(ValueError),
|
||||
):
|
||||
db.write_execution_contexts(
|
||||
second.run_id,
|
||||
second.storage_epoch,
|
||||
bundle("changed", "new-positive"),
|
||||
)
|
||||
self.assertEqual(tuple(db.model.get_connection().iterdump()), before)
|
||||
|
||||
def test_parallel_contradictory_replay_keeps_one_first_capture(self):
|
||||
db = self.database()
|
||||
start = self.start(db)
|
||||
db.write_execution_start(start)
|
||||
|
||||
def write(negative):
|
||||
return db.write_execution_contexts(
|
||||
start.run_id, start.storage_epoch, bundle(negative)
|
||||
)
|
||||
|
||||
with ThreadPoolExecutor(max_workers=2) as pool:
|
||||
self.assertEqual(list(pool.map(write, ("A", "B"))), [True, True])
|
||||
read = db.get_execution_contexts(start.run_id)
|
||||
self.assertEqual(read["state"], "conflicting")
|
||||
self.assertIn(read["contexts"][0]["negative"]["raw"], ("A", "B"))
|
||||
self.assertEqual(self.counts(db), (1, 1, 2, 1))
|
||||
|
||||
def test_merge_never_rebinds_a_hint_to_normalized_equal_primary(self):
|
||||
db = self.database()
|
||||
with db.model.get_connection() as conn:
|
||||
conn.execute(
|
||||
"INSERT INTO prompts(id,text,created_at) VALUES(1,' POSITIVE "
|
||||
"','2020-01-01'),(2,'positive','2021-01-01')"
|
||||
)
|
||||
start = self.capture(db)
|
||||
before = db.get_execution_contexts(start.run_id)["contexts"][0]
|
||||
self.assertEqual(before["positive_prompt_id"], 2)
|
||||
self.assertEqual(db.cleanup_duplicates(), 1)
|
||||
after = db.get_execution_contexts(start.run_id)["contexts"][0]
|
||||
self.assertIsNone(after["positive_prompt_id"])
|
||||
self.assertEqual(after["positive"], before["positive"])
|
||||
self.assertEqual(after["evidence"], before["evidence"])
|
||||
self.assertEqual(
|
||||
db.model.get_connection().execute("SELECT text FROM prompts").fetchone()[0],
|
||||
" POSITIVE ",
|
||||
)
|
||||
|
||||
def test_ambiguous_empty_negative_omits_pair_hint(self):
|
||||
db = self.database()
|
||||
with db.model.get_connection() as conn:
|
||||
conn.execute(
|
||||
"INSERT INTO prompts(id,text) VALUES(1,'positive'),(2,''),(3,'')"
|
||||
)
|
||||
conn.execute("INSERT INTO prompt_pairs(id,positive_prompt_id) VALUES(7,1)")
|
||||
start = self.capture(db, capture=bundle(negative=""))
|
||||
read = db.get_execution_contexts(start.run_id)["contexts"][0]
|
||||
self.assertIsNone(read["negative_prompt_id"])
|
||||
self.assertIsNone(read["pair_id"])
|
||||
|
||||
def test_replay_checks_scalar_content_even_if_digests_collide(self):
|
||||
db = self.database()
|
||||
start = self.capture(db)
|
||||
before = tuple(db.model.get_connection().iterdump())
|
||||
from utils.execution_contexts import content_digest
|
||||
|
||||
original_snapshot = content_digest("snapshot", ("negative", "negative", "", ""))
|
||||
|
||||
def digest(kind, fields):
|
||||
if kind == "snapshot" and fields == ("changed", "changed", "", ""):
|
||||
return original_snapshot
|
||||
return content_digest(kind, fields)
|
||||
|
||||
with (
|
||||
patch("database.execution_contexts.content_digest", side_effect=digest),
|
||||
self.assertRaises(ValueError),
|
||||
):
|
||||
db.write_execution_contexts(
|
||||
start.run_id, start.storage_epoch, bundle("changed")
|
||||
)
|
||||
self.assertEqual(tuple(db.model.get_connection().iterdump()), before)
|
||||
|
||||
def test_package_qualified_capture_with_and_without_repository_on_sys_path(self):
|
||||
# A host imports node packs under a package name, not as top-level utils.
|
||||
script = textwrap.dedent("""\
|
||||
import importlib
|
||||
import sys
|
||||
import tempfile
|
||||
import types
|
||||
from pathlib import Path
|
||||
root, expose_root = sys.argv[1:]
|
||||
if expose_root == "yes":
|
||||
sys.path.insert(0, root)
|
||||
else:
|
||||
assert all(Path(p).resolve() != Path(root) for p in sys.path)
|
||||
package = types.ModuleType("context_pack")
|
||||
package.__path__ = [root]
|
||||
sys.modules[package.__name__] = package
|
||||
records = importlib.import_module("context_pack.utils.execution_contexts")
|
||||
operations = importlib.import_module("context_pack.database.operations")
|
||||
events = importlib.import_module(
|
||||
"context_pack.utils.execution_history_events"
|
||||
)
|
||||
schema = importlib.import_module(
|
||||
"context_pack.database.execution_context_schema"
|
||||
)
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
db = operations.PromptDatabase(str(Path(directory) / "capture.db"))
|
||||
try:
|
||||
start = events.make_start(
|
||||
"11111111-1111-4111-8111-111111111111",
|
||||
((0, "22222222-2222-4222-8222-222222222222", {}, {}), 0),
|
||||
db.get_execution_epoch(), 100,
|
||||
)
|
||||
db.write_execution_start(start)
|
||||
def known(value):
|
||||
return records.TextEvidence(
|
||||
value, "recorded", "runtime_return", None
|
||||
)
|
||||
snapshot = records.PromptSnapshot(
|
||||
known("raw"), known("effective"), known(""), known("")
|
||||
)
|
||||
context = records.RecipeContext(
|
||||
"4", snapshot, snapshot, "inferred"
|
||||
)
|
||||
bundle = records.ContextBundle(
|
||||
(context,), "complete", "captured"
|
||||
)
|
||||
assert db.write_execution_contexts(
|
||||
start.run_id, start.storage_epoch, bundle
|
||||
)
|
||||
read = db.get_execution_contexts(start.run_id)
|
||||
assert read["state"] == "complete", read
|
||||
assert read["contexts"][0]["positive"]["raw"] == "raw", read
|
||||
role = read["contexts"][0]["positive"]
|
||||
assert role["effective"] == "effective", read
|
||||
error = schema.context_schema_error(db.model.get_connection())
|
||||
assert error is None, error
|
||||
verified = db.model.verify_database_file(db.model.db_path)
|
||||
assert verified == (True, "ok"), verified
|
||||
if expose_root == "no":
|
||||
assert "utils.execution_contexts" not in sys.modules
|
||||
assert "database.execution_context_schema" not in sys.modules
|
||||
finally:
|
||||
db.close_all()
|
||||
""")
|
||||
root = str(Path(__file__).resolve().parents[1])
|
||||
for expose_root in ("yes", "no"):
|
||||
with self.subTest(repository_on_sys_path=expose_root):
|
||||
result = subprocess.run(
|
||||
[sys.executable, "-I", "-c", script, root, expose_root],
|
||||
cwd=self.temp.name,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=30,
|
||||
)
|
||||
self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
|
||||
@@ -0,0 +1,340 @@
|
||||
"""Projection must retain declared roles without claiming sampler consumption."""
|
||||
|
||||
import unittest
|
||||
from test_execution_contexts import load_module
|
||||
|
||||
|
||||
def node(kind, **inputs):
|
||||
return {"class_type": kind, "inputs": inputs}
|
||||
|
||||
|
||||
def graph(kind="KSampler", source="PromptManagerPair"):
|
||||
return {
|
||||
"p": node(
|
||||
source,
|
||||
text=" cat ",
|
||||
negative_text="",
|
||||
prepend_text="pre",
|
||||
append_text="post",
|
||||
),
|
||||
"s": node(kind, positive=["p", 0], negative=["p", 1]),
|
||||
}
|
||||
|
||||
|
||||
def transformed_input_graph(field: str, unresolved: bool = False) -> dict:
|
||||
g = graph()
|
||||
g["upstream"] = node("PromptManagerText", text="looks literal")
|
||||
g["dynamic"] = node("RuntimeGenerator")
|
||||
g["join"] = node(
|
||||
"StringConcatenate",
|
||||
string_a=["upstream", 0],
|
||||
string_b=["dynamic", 0] if unresolved else " suffix",
|
||||
)
|
||||
if unresolved:
|
||||
g["inner"] = node(
|
||||
"StringConcatenate", string_a=["upstream", 0], string_b=" inner"
|
||||
)
|
||||
g["join"]["inputs"]["string_a"] = ["inner", 0]
|
||||
g["p"]["inputs"][field] = ["join", 0]
|
||||
return g
|
||||
|
||||
|
||||
class ProjectionTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.m = load_module("execution_context_graph")
|
||||
|
||||
def test_projection_preserves_roles(self):
|
||||
for kind in ("KSampler", "KSamplerAdvanced", "SamplerCustom"):
|
||||
g = graph(kind)
|
||||
result = self.m.project_contexts(g, None)
|
||||
self.assertEqual(len(result.recipes), 1)
|
||||
recipe = result.recipes[0]
|
||||
self.assertEqual(recipe.anchor_id, "s")
|
||||
self.assertEqual(recipe.positive.inferred.raw.value, " cat ")
|
||||
self.assertEqual(recipe.positive.inferred.effective.value, "pre cat post")
|
||||
self.assertEqual(recipe.negative.inferred.effective.value, "")
|
||||
self.assertEqual(recipe.positive.sources[0].output_slot, 0)
|
||||
g["s"]["inputs"]["positive"], g["s"]["inputs"]["negative"] = ["p", 1], [
|
||||
"p",
|
||||
0,
|
||||
]
|
||||
crossed = self.m.project_contexts(g, None).recipes[0]
|
||||
self.assertEqual(crossed.positive.inferred.raw.value, "")
|
||||
self.assertEqual(crossed.negative.sources[0].output_slot, 0)
|
||||
for source in ("PromptManager", "PromptManagerFocused", "PromptManagerSplit"):
|
||||
g = graph(source=source)
|
||||
g["clip"] = node("CLIPTextEncode", text=["p", 1])
|
||||
g["s"]["inputs"]["negative"] = ["clip", 0]
|
||||
recipe = self.m.project_contexts(g, None).recipes[0]
|
||||
self.assertEqual(recipe.negative.inferred.raw.value, " cat ")
|
||||
self.assertEqual(recipe.negative.sources[0].output_slot, 1)
|
||||
for source, slot in (
|
||||
("PromptManagerText", 0),
|
||||
("PromptManagerPair", 2),
|
||||
("PromptManagerPair", 3),
|
||||
):
|
||||
g = graph(source=source)
|
||||
g["clip"] = node("CLIPTextEncode", text=["p", slot])
|
||||
g["s"]["inputs"] = {"positive": ["clip", 0], "negative": ["clip", 0]}
|
||||
self.assertEqual(
|
||||
self.m.project_contexts(g, None)
|
||||
.recipes[0]
|
||||
.positive.sources[0]
|
||||
.output_slot,
|
||||
slot,
|
||||
)
|
||||
g = graph()
|
||||
g["p"] = node("CLIPTextEncode", text=["literal", 0])
|
||||
g["literal"] = node("PrimitiveString", value=" exact ")
|
||||
g["s"]["inputs"]["negative"] = ["p", 0]
|
||||
self.assertEqual(
|
||||
self.m.project_contexts(g, None)
|
||||
.recipes[0]
|
||||
.positive.inferred.effective.value,
|
||||
" exact ",
|
||||
)
|
||||
g["p"] = node("UnknownTransform", text="secret")
|
||||
role = self.m.project_contexts(g, None).recipes[0].positive
|
||||
self.assertIsNone(role.inferred)
|
||||
self.assertTrue(role.reason)
|
||||
|
||||
def test_output_selection_and_scalar_isolation(self):
|
||||
g = graph()
|
||||
g["other"] = node("KSampler", positive=["p", 0], negative=["p", 1])
|
||||
g["decode"] = node("VAEDecode", samples=["s", 0])
|
||||
g["save"] = node("SaveImage", images=["decode", 0])
|
||||
result = self.m.project_contexts(g, ["save"])
|
||||
self.assertEqual(tuple(r.anchor_id for r in result.recipes), ("s",))
|
||||
g["p"]["inputs"]["text"] = "changed"
|
||||
self.assertEqual(result.recipes[0].positive.inferred.raw.value, " cat ")
|
||||
self.assertEqual(self.m.project_contexts(g, []).recipes, ())
|
||||
for outputs in (["missing"], [object()], "save"):
|
||||
result = self.m.project_contexts(g, outputs)
|
||||
self.assertEqual(result.recipes, ())
|
||||
self.assertTrue(result.reason)
|
||||
|
||||
def test_selected_ancestry_ignores_unrelated_widget_arrays(self):
|
||||
for widget in (
|
||||
[1, 2, 3],
|
||||
("s", 0, "widget"),
|
||||
[],
|
||||
["s"],
|
||||
["s", "0"],
|
||||
["s", True],
|
||||
["s", -1],
|
||||
[["s", 0], 0],
|
||||
):
|
||||
with self.subTest(widget=widget):
|
||||
g = graph()
|
||||
g["other"] = node("KSampler", positive=["p", 0], negative=["p", 1])
|
||||
g["adapter"] = node("CustomAdapter", samples=("s", 0), widget=widget)
|
||||
g["save"] = node("CustomOutput", images=["adapter", 0])
|
||||
result = self.m.project_contexts(g, ["save"])
|
||||
self.assertEqual(tuple(r.anchor_id for r in result.recipes), ("s",))
|
||||
self.assertEqual(result.recipes[0].positive.inferred.raw.value, " cat ")
|
||||
|
||||
def test_two_element_link_shapes_are_followed_conservatively(self):
|
||||
g = graph()
|
||||
# A typed two-element value cannot be distinguished from a real link.
|
||||
g["save"] = node("CustomOutput", widget=["s", 0])
|
||||
self.assertEqual(
|
||||
tuple(r.anchor_id for r in self.m.project_contexts(g, ["save"]).recipes),
|
||||
("s",),
|
||||
)
|
||||
for value in (["missing", 0], ["x" * 129, 0]):
|
||||
g["save"]["inputs"]["widget"] = value
|
||||
result = self.m.project_contexts(g, ["save"])
|
||||
self.assertEqual(result.recipes, ())
|
||||
self.assertTrue(result.reason)
|
||||
|
||||
def test_projection_limits(self):
|
||||
for count, succeeds in ((500, True), (501, False)):
|
||||
g = graph()
|
||||
g.update({str(i): node("Unused") for i in range(count - 2)})
|
||||
self.assertEqual(bool(self.m.project_contexts(g, None).recipes), succeeds)
|
||||
for count, succeeds in ((16, True), (17, False)):
|
||||
g = {"p": graph()["p"]}
|
||||
g.update({str(i): graph()["s"] for i in range(count)})
|
||||
self.assertEqual(bool(self.m.project_contexts(g, None).recipes), succeeds)
|
||||
for depth, succeeds in ((64, True), (65, False)):
|
||||
g = graph()
|
||||
# The CLIP plus reroutes plus literal give exactly depth visited links.
|
||||
g["p"] = node("CLIPTextEncode", text=["0", 0])
|
||||
for i in range(depth - 2):
|
||||
g[str(i)] = node("Reroute", value=[str(i + 1), 0])
|
||||
g[str(depth - 2)] = node("PrimitiveString", value="known")
|
||||
role = self.m.project_contexts(g, None).recipes[0].positive
|
||||
self.assertEqual(role.inferred is not None, succeeds)
|
||||
g = graph()
|
||||
g["s" * 129] = g.pop("s")
|
||||
self.assertFalse(self.m.project_contexts(g, None).recipes)
|
||||
|
||||
def test_dynamic_pm_inputs_preserve_observable_sources(self):
|
||||
for kind, positive_slot, negative_slot in (
|
||||
("PromptManagerText", 0, 0),
|
||||
("PromptManagerPair", 2, 3),
|
||||
("PromptManager", 1, 1),
|
||||
("PromptManagerFocused", 1, 1),
|
||||
("PromptManagerSplit", 1, 1),
|
||||
("PromptManagerPair", 0, 1),
|
||||
):
|
||||
with self.subTest(kind=kind, slots=(positive_slot, negative_slot)):
|
||||
g = graph(source=kind)
|
||||
g["generator"] = node("UnknownGenerator")
|
||||
g["p"]["inputs"]["text"] = ["generator", 0]
|
||||
g["p"]["inputs"]["negative_text"] = ["generator", 0]
|
||||
g["pos"] = node("CLIPTextEncode", text=["p", positive_slot])
|
||||
g["neg"] = node("CLIPTextEncode", text=["p", negative_slot])
|
||||
g["s"]["inputs"] = {"positive": ["pos", 0], "negative": ["neg", 0]}
|
||||
if kind == "PromptManagerPair" and positive_slot == 0:
|
||||
g["s"]["inputs"] = {"positive": ["p", 1], "negative": ["p", 0]}
|
||||
positive_slot, negative_slot = 1, 0
|
||||
result = self.m.project_contexts(g, None)
|
||||
recipe = result.recipes[0]
|
||||
for role, slot in (
|
||||
(recipe.positive, positive_slot),
|
||||
(recipe.negative, negative_slot),
|
||||
):
|
||||
self.assertEqual(
|
||||
tuple(ref.output_slot for ref in role.sources), (slot,)
|
||||
)
|
||||
self.assertIsNone(role.inferred)
|
||||
self.assertTrue(role.reason)
|
||||
self.assertTrue(result.sources)
|
||||
|
||||
def test_graph_evidence_distinguishes_literals_from_semantics(self):
|
||||
result = self.m.project_contexts(graph(), None)
|
||||
evidence = result.recipes[0].positive.inferred
|
||||
self.assertEqual(evidence.raw.provenance, "inferred")
|
||||
self.assertEqual(evidence.raw.source, "graph_literal")
|
||||
self.assertEqual(evidence.effective.source, "graph_semantics")
|
||||
|
||||
def test_derived_raw_and_composition_preserve_field_sources(self):
|
||||
g = graph(source="PromptManagerText")
|
||||
g["join"] = node(
|
||||
"Text Concatenate",
|
||||
text_a=" a ",
|
||||
text_b=" b ",
|
||||
delimiter="-",
|
||||
clean_whitespace="true",
|
||||
)
|
||||
g["p"]["inputs"]["text"] = ["join", 0]
|
||||
g["clip"] = node("CLIPTextEncode", text=["p", 0])
|
||||
g["s"]["inputs"] = {"positive": ["clip", 0], "negative": ["clip", 0]}
|
||||
role = self.m.project_contexts(g, None).recipes[0].positive
|
||||
self.assertIsNotNone(role.inferred)
|
||||
self.assertEqual(role.inferred.raw.value, "a-b")
|
||||
self.assertEqual(role.inferred.raw.source, "graph_semantics")
|
||||
self.assertEqual(role.inferred.prepend.source, "graph_literal")
|
||||
|
||||
def test_selected_custom_outputs_and_multistage_ancestry(self):
|
||||
g = graph()
|
||||
g["earlier"] = node("SamplerCustom", positive=["p", 0], negative=["p", 1])
|
||||
g["s"]["inputs"]["latent_image"] = ["earlier", 0]
|
||||
g["adapter"] = node("UnknownImageAdapter", source=["s", 0])
|
||||
g["custom"] = node("CustomOutput", images=["adapter", 0])
|
||||
result = self.m.project_contexts(g, ["custom"])
|
||||
self.assertEqual(tuple(r.anchor_id for r in result.recipes), ("s", "earlier"))
|
||||
|
||||
def test_total_resolution_visits_are_bounded(self):
|
||||
# Sixteen recipes each resolving two independent chains of length 16
|
||||
# exceed 500 total visits even though the graph has fewer than 500 nodes.
|
||||
g = {"v": node("PrimitiveString", value="known")}
|
||||
for i in range(15):
|
||||
g[str(i)] = node("Reroute", value=[str(i + 1), 0] if i < 14 else ["v", 0])
|
||||
g["clip"] = node("CLIPTextEncode", text=["0", 0])
|
||||
for i in range(16):
|
||||
g["s" + str(i)] = node(
|
||||
"KSampler", positive=["clip", 0], negative=["clip", 0]
|
||||
)
|
||||
result = self.m.project_contexts(g, None)
|
||||
self.assertTrue(result.reason)
|
||||
self.assertFalse(
|
||||
result.recipes
|
||||
and all(
|
||||
r.positive.inferred is not None and r.negative.inferred is not None
|
||||
for r in result.recipes
|
||||
)
|
||||
)
|
||||
|
||||
def test_known_concatenation_remains_inferred(self):
|
||||
g = graph(source="PromptManagerText")
|
||||
g["join"] = node(
|
||||
"StringConcatenate", string_a=["p", 0], string_b="!", delimiter=""
|
||||
)
|
||||
g["clip"] = node("CLIPTextEncode", text=["join", 0])
|
||||
g["s"]["inputs"] = {"positive": ["clip", 0], "negative": ["clip", 0]}
|
||||
role = self.m.project_contexts(g, None).recipes[0].positive
|
||||
self.assertIsNotNone(role.inferred)
|
||||
self.assertEqual(role.inferred.effective.value, "pre cat post!")
|
||||
self.assertEqual(role.inferred.effective.provenance, "inferred")
|
||||
self.assertEqual(role.sources[0].node_id, "p")
|
||||
self.assertEqual(role.reason, "source-transformed")
|
||||
|
||||
def test_transform_provenance_stops_at_selected_pm_capture_boundary(self):
|
||||
for field in ("text", "prepend_text", "append_text"):
|
||||
for unresolved in (False, True):
|
||||
with self.subTest(field=field, unresolved=unresolved):
|
||||
g = transformed_input_graph(field, unresolved)
|
||||
g["outer"] = node(
|
||||
"StringConcatenate", string_a=["p", 2], string_b=" outer"
|
||||
)
|
||||
g["clip"] = node("CLIPTextEncode", text=["outer", 0])
|
||||
g["s"]["inputs"]["negative"] = ["clip", 0]
|
||||
result = self.m.project_contexts(g, None)
|
||||
positive, negative = (
|
||||
result.recipes[0].positive,
|
||||
result.recipes[0].negative,
|
||||
)
|
||||
self.assertEqual(
|
||||
[(ref.node_id, ref.output_slot) for ref in positive.sources],
|
||||
[("p", 0)],
|
||||
)
|
||||
self.assertNotEqual(positive.reason, "source-transformed")
|
||||
self.assertEqual(negative.reason, "source-transformed")
|
||||
self.assertEqual(negative.sources[0].node_id, "p")
|
||||
|
||||
def test_malformed_inputs_do_not_stringify(self):
|
||||
class HostValue:
|
||||
def __str__(self):
|
||||
raise AssertionError("host object stringified")
|
||||
|
||||
for g in (
|
||||
{HostValue(): node("KSampler")},
|
||||
{"s": node(HostValue())},
|
||||
{"s": node("KSampler", positive=HostValue())},
|
||||
):
|
||||
self.m.project_contexts(g, None)
|
||||
|
||||
def test_composed_affixes_and_retained_projection_are_bounded(self):
|
||||
g = graph()
|
||||
g["p"]["inputs"].update(
|
||||
text="x" * 65536, prepend_text="y" * 65536, append_text="z" * 65536
|
||||
)
|
||||
result = self.m.project_contexts(g, None)
|
||||
self.assertIsNone(result.recipes[0].positive.inferred)
|
||||
g = graph()
|
||||
g["p"]["inputs"]["text"] = "x" * 60000
|
||||
for i in range(15):
|
||||
g[str(i)] = node("KSampler", positive=["p", 0], negative=["p", 0])
|
||||
result = self.m.project_contexts(g, None)
|
||||
self.assertEqual(result.recipes, ())
|
||||
|
||||
def test_partial_source_snapshots_keep_literal_fields_without_stateful_substitution(
|
||||
self,
|
||||
):
|
||||
for upstream in ("RuntimeGenerator", "PromptManagerText"):
|
||||
g = graph()
|
||||
g["dynamic"] = node(upstream, text="looks literal")
|
||||
g["p"]["inputs"]["text"] = ["dynamic", 0]
|
||||
projection = self.m.project_contexts(g, None)
|
||||
positive = next(
|
||||
snapshot
|
||||
for ref, snapshot in projection.source_snapshots
|
||||
if ref.output_slot == 0
|
||||
)
|
||||
self.assertIsNone(positive.raw.value)
|
||||
self.assertIsNone(positive.effective.value)
|
||||
self.assertEqual(positive.prepend.value, "pre")
|
||||
self.assertEqual(positive.append.value, "post")
|
||||
self.assertEqual(positive.append.provenance, "inferred")
|
||||
@@ -0,0 +1,447 @@
|
||||
"""Read-only cached STRING evidence and cooperative executor notifications."""
|
||||
|
||||
from collections import namedtuple
|
||||
from contextvars import Context
|
||||
from dataclasses import replace
|
||||
import gc
|
||||
import sys
|
||||
import weakref
|
||||
import threading
|
||||
import types
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
from test_execution_context_graph import transformed_input_graph
|
||||
|
||||
from utils.execution_context_budget import ContextBudget
|
||||
from utils.execution_context_capture import ContextCapture
|
||||
from utils.execution_history_events import RunStart
|
||||
from utils.execution_history_writer import HistoryWriterStatus
|
||||
|
||||
try:
|
||||
from utils import execution_context_host as host
|
||||
except ImportError:
|
||||
host = None
|
||||
|
||||
SESSION = "11111111-1111-4111-8111-111111111111"
|
||||
EPOCH = "22222222-2222-4222-8222-222222222222"
|
||||
JOB = "33333333-3333-4333-8333-333333333333"
|
||||
Entry = namedtuple("CacheEntry", "ui outputs")
|
||||
|
||||
|
||||
class HostTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.assertIsNotNone(host, "read-only host adapter is missing")
|
||||
self.queue = object()
|
||||
self.pool = ContextBudget()
|
||||
self.status = HistoryWriterStatus(EPOCH, True, 0, 0, True, 3)
|
||||
self.capture = ContextCapture(id(self.queue), lambda: self.status, self.pool)
|
||||
self.addCleanup(self.capture.revoke_all)
|
||||
self.calls = []
|
||||
calls = self.calls
|
||||
|
||||
class BasicCache:
|
||||
def __init__(self):
|
||||
self.entries = {}
|
||||
|
||||
def get_local(self, node_id):
|
||||
calls.append(("local", node_id))
|
||||
return self.entries.get(node_id)
|
||||
|
||||
async def get(self, node_id):
|
||||
raise AssertionError("async get must never be called")
|
||||
|
||||
class HierarchicalCache(BasicCache):
|
||||
pass
|
||||
|
||||
class LRUCache(BasicCache):
|
||||
pass
|
||||
|
||||
class RAMPressureCache(LRUCache):
|
||||
pass
|
||||
|
||||
class NullCache:
|
||||
def get_local(self, node_id):
|
||||
raise AssertionError("null cache is not usable")
|
||||
|
||||
class PromptExecutor:
|
||||
def add_message(self, event, data, broadcast):
|
||||
calls.append(("original", event, data, broadcast))
|
||||
if event == "raise":
|
||||
raise RuntimeError("original")
|
||||
return "original-return"
|
||||
|
||||
self.execution = types.ModuleType("execution")
|
||||
self.execution.PromptExecutor = PromptExecutor
|
||||
self.execution.CacheEntry = Entry
|
||||
self.caching = types.ModuleType("comfy_execution.caching")
|
||||
for cls in (
|
||||
BasicCache,
|
||||
HierarchicalCache,
|
||||
LRUCache,
|
||||
RAMPressureCache,
|
||||
NullCache,
|
||||
):
|
||||
setattr(self.caching, cls.__name__, cls)
|
||||
package = types.ModuleType("comfy_execution")
|
||||
package.caching = self.caching
|
||||
self.patch = patch.dict(
|
||||
sys.modules,
|
||||
{
|
||||
"comfy_execution": package,
|
||||
"execution": self.execution,
|
||||
"comfy_execution.caching": self.caching,
|
||||
},
|
||||
)
|
||||
self.patch.start()
|
||||
self.addCleanup(self.patch.stop)
|
||||
self.executor = PromptExecutor()
|
||||
self.executor.server = types.SimpleNamespace(prompt_queue=self.queue)
|
||||
self.executor.caches = types.SimpleNamespace(outputs=HierarchicalCache())
|
||||
|
||||
def begin(
|
||||
self, kind="PromptManager", raw=" raw ", prepend=" pre ", append=" post "
|
||||
):
|
||||
g = {
|
||||
"p": {
|
||||
"class_type": kind,
|
||||
"inputs": {
|
||||
"text": raw,
|
||||
"negative_text": "negative",
|
||||
"prepend_text": prepend,
|
||||
"append_text": append,
|
||||
},
|
||||
},
|
||||
"clip": {"class_type": "CLIPTextEncode", "inputs": {"text": ["p", 0]}},
|
||||
"s": {
|
||||
"class_type": "KSampler",
|
||||
"inputs": {
|
||||
"positive": (
|
||||
["clip", 0] if kind == "PromptManagerText" else ["p", 0]
|
||||
),
|
||||
"negative": (
|
||||
["p", 1] if kind == "PromptManagerPair" else ["clip", 0]
|
||||
),
|
||||
},
|
||||
},
|
||||
}
|
||||
if type(raw) is list or type(prepend) is list:
|
||||
g["dynamic"] = {"class_type": "RuntimeGenerator", "inputs": {}}
|
||||
g["stateful"] = {
|
||||
"class_type": "PromptManagerText",
|
||||
"inputs": {"text": "looks literal"},
|
||||
}
|
||||
self.cap = self.capture.begin(
|
||||
RunStart(SESSION, SESSION, 1, JOB, EPOCH, 0), 3, g, None
|
||||
)
|
||||
return self.cap
|
||||
|
||||
def observe(self, event, executor=None, **data):
|
||||
host.observe_cached_context(
|
||||
executor or self.executor, event, {"prompt_id": JOB, **data}
|
||||
)
|
||||
|
||||
def seal(self):
|
||||
sealed = self.capture.seal(self.cap)
|
||||
if sealed.lease:
|
||||
self.addCleanup(sealed.lease.release)
|
||||
return sealed.bundle.contexts[0]
|
||||
|
||||
def test_original_return_exception_and_wrapper_cleanup_preserved(self):
|
||||
cls = self.execution.PromptExecutor
|
||||
original = cls.add_message
|
||||
|
||||
def earlier(executor, *args, **kwargs):
|
||||
self.calls.append("earlier")
|
||||
return original(executor, *args, **kwargs)
|
||||
|
||||
cls.add_message = earlier
|
||||
seen = []
|
||||
stop_a = host.subscribe_executor_messages("a", lambda *args: seen.append(args))
|
||||
stop_b = host.subscribe_executor_messages("b", lambda *args: seen.append(args))
|
||||
wrapped = cls.add_message
|
||||
|
||||
def later(executor, *args, **kwargs):
|
||||
self.calls.append("later")
|
||||
return wrapped(executor, *args, **kwargs)
|
||||
|
||||
cls.add_message = later
|
||||
data = {"prompt_id": JOB}
|
||||
self.assertEqual(
|
||||
self.executor.add_message("ok", data, broadcast=False), "original-return"
|
||||
)
|
||||
self.assertEqual(len(seen), 2)
|
||||
self.assertIs(seen[0][0], self.executor)
|
||||
self.assertIs(seen[0][1][1], data)
|
||||
with self.assertRaisesRegex(RuntimeError, "original"):
|
||||
self.executor.add_message("raise", data, False)
|
||||
self.assertEqual(len(seen), 2)
|
||||
stop_a()
|
||||
self.executor.add_message("ok", data, False)
|
||||
self.assertEqual(len(seen), 3)
|
||||
stop_b()
|
||||
self.assertIs(cls.add_message, later)
|
||||
self.executor.add_message("ok", data, False)
|
||||
self.assertEqual(len(seen), 3)
|
||||
stop_c = host.subscribe_executor_messages("c", lambda *args: seen.append(args))
|
||||
self.executor.add_message("ok", data, False)
|
||||
self.assertEqual(len(seen), 4)
|
||||
stop_c()
|
||||
|
||||
def test_cache_reads_only_local_string_slots(self):
|
||||
for mode in ("HierarchicalCache", "LRUCache", "RAMPressureCache"):
|
||||
for kind, values in (
|
||||
("PromptManager", [object(), ["cached"]]),
|
||||
("PromptManagerFocused", [object(), ["cached"]]),
|
||||
("PromptManagerSplit", [object(), ["cached"]]),
|
||||
("PromptManagerText", [["cached"]]),
|
||||
("PromptManagerPair", [object(), object(), ["cached"], [""]]),
|
||||
):
|
||||
with self.subTest(mode=mode, kind=kind):
|
||||
self.begin(kind)
|
||||
cache = getattr(self.caching, mode)()
|
||||
entry = Entry({"secret": object()}, values)
|
||||
cache.entries["p"] = entry
|
||||
self.executor.caches.outputs = cache
|
||||
self.observe("execution_start")
|
||||
self.observe("execution_cached", nodes=["p"])
|
||||
result = self.seal()
|
||||
self.assertEqual(result.positive.effective.value, "cached")
|
||||
self.assertEqual(
|
||||
result.positive.effective.source, "host_cache_output"
|
||||
)
|
||||
if kind == "PromptManagerPair":
|
||||
self.assertEqual(result.negative.effective.value, "")
|
||||
self.assertIs(cache.entries["p"], entry)
|
||||
self.assertEqual(set(cache.__dict__), {"entries"})
|
||||
self.assertTrue(all(call[0] == "local" for call in self.calls))
|
||||
|
||||
def test_changed_trigger_configuration_keeps_cached_effective_text(self):
|
||||
self.begin()
|
||||
self.executor.caches.outputs.entries["p"] = Entry(
|
||||
None, [object(), ["old injected text"]]
|
||||
)
|
||||
self.observe("execution_start")
|
||||
self.observe("execution_cached", nodes=["p"])
|
||||
result = self.seal().positive
|
||||
self.assertEqual(result.effective.value, "old injected text")
|
||||
self.assertEqual(result.raw.value, " raw ")
|
||||
self.assertEqual(result.prepend.value, " pre ")
|
||||
self.assertEqual(result.append.value, " post ")
|
||||
self.assertEqual(result.raw.provenance, "inferred")
|
||||
|
||||
def test_dynamic_cached_raw_is_unknown(self):
|
||||
for dynamic in (["dynamic", 0], ["stateful", 0]):
|
||||
for field in ("raw", "prepend"):
|
||||
with self.subTest(dynamic=dynamic, field=field):
|
||||
self.begin(**{field: dynamic})
|
||||
self.executor.caches.outputs.entries["p"] = Entry(
|
||||
None, [object(), ["cached"]]
|
||||
)
|
||||
self.observe("execution_start")
|
||||
self.observe("execution_cached", nodes=["p"])
|
||||
result = self.seal().positive
|
||||
self.assertEqual(result.effective.value, "cached")
|
||||
self.assertIsNone(getattr(result, field).value)
|
||||
self.assertEqual(result.append.value, " post ")
|
||||
if field == "raw":
|
||||
self.assertEqual(result.prepend.value, " pre ")
|
||||
else:
|
||||
self.assertEqual(result.raw.value, " raw ")
|
||||
|
||||
def test_upstream_transform_preserves_final_pm_cached_effective_only(self):
|
||||
for field, snapshot_field in (
|
||||
("text", "raw"),
|
||||
("prepend_text", "prepend"),
|
||||
("append_text", "append"),
|
||||
):
|
||||
for unresolved in (False, True):
|
||||
with self.subTest(field=field, unresolved=unresolved):
|
||||
g = transformed_input_graph(field, unresolved)
|
||||
g["outer"] = {
|
||||
"class_type": "StringConcatenate",
|
||||
"inputs": {"string_a": ["p", 2], "string_b": " outer"},
|
||||
}
|
||||
g["clip"] = {
|
||||
"class_type": "CLIPTextEncode",
|
||||
"inputs": {"text": ["outer", 0]},
|
||||
}
|
||||
g["s"]["inputs"]["negative"] = ["clip", 0]
|
||||
self.cap = self.capture.begin(
|
||||
RunStart(SESSION, SESSION, 1, JOB, EPOCH, 0),
|
||||
3,
|
||||
g,
|
||||
None,
|
||||
)
|
||||
self.executor.caches.outputs.entries["p"] = Entry(
|
||||
None, [object(), object(), [" cached injected "], [""]]
|
||||
)
|
||||
self.observe("execution_start")
|
||||
self.observe("execution_cached", nodes=["upstream", "p"])
|
||||
result = self.seal()
|
||||
self.assertIsNone(result.negative.effective.value)
|
||||
side = result.positive
|
||||
self.assertEqual(side.effective.value, " cached injected ")
|
||||
self.assertEqual(side.effective.provenance, "recorded")
|
||||
self.assertEqual(side.effective.source, "host_cache_output")
|
||||
for name, expected in (
|
||||
("raw", " cat "),
|
||||
("prepend", "pre"),
|
||||
("append", "post"),
|
||||
):
|
||||
evidence = getattr(side, name)
|
||||
if name == snapshot_field:
|
||||
self.assertIsNone(evidence.value)
|
||||
self.assertEqual(evidence.provenance, "unknown")
|
||||
else:
|
||||
self.assertEqual(evidence.value, expected)
|
||||
self.assertEqual(evidence.provenance, "inferred")
|
||||
|
||||
def test_provider_local_reset_eviction_and_unknown_modes_fail_closed(self):
|
||||
for mode in (
|
||||
"NullCache",
|
||||
"unknown",
|
||||
"reset",
|
||||
"evicted",
|
||||
"malformed",
|
||||
"provider-local",
|
||||
):
|
||||
with self.subTest(mode=mode):
|
||||
self.begin()
|
||||
cache = self.caching.HierarchicalCache()
|
||||
if mode == "NullCache":
|
||||
cache = self.caching.NullCache()
|
||||
elif mode == "unknown":
|
||||
cache = types.SimpleNamespace(
|
||||
get_local=lambda _: self.fail("unknown cache read")
|
||||
)
|
||||
elif mode == "malformed":
|
||||
cache.entries["p"] = Entry(None, [object(), [object()]])
|
||||
elif mode in ("reset", "provider-local"):
|
||||
cache.entries["p"] = Entry(
|
||||
None, [object(), ["resident provider output"]]
|
||||
)
|
||||
self.executor.caches.outputs = cache
|
||||
self.observe("execution_start")
|
||||
if mode == "reset":
|
||||
self.executor.caches.outputs = self.caching.HierarchicalCache()
|
||||
self.observe("execution_cached", nodes=["p"])
|
||||
result = self.seal().positive
|
||||
self.assertEqual(
|
||||
result.effective.value,
|
||||
"resident provider output" if mode == "provider-local" else None,
|
||||
)
|
||||
|
||||
def test_worker_queue_job_executor_and_generation_fences(self):
|
||||
for failure in (
|
||||
"unbound",
|
||||
"thread",
|
||||
"queue",
|
||||
"job",
|
||||
"executor",
|
||||
"recursive",
|
||||
"generation",
|
||||
"no-start",
|
||||
):
|
||||
with self.subTest(failure=failure):
|
||||
self.begin()
|
||||
self.executor.server.prompt_queue = self.queue
|
||||
self.executor.caches.outputs.entries["p"] = Entry(
|
||||
None, [object(), ["cached"]]
|
||||
)
|
||||
if failure != "no-start":
|
||||
self.observe("execution_start")
|
||||
target = self.executor
|
||||
if failure == "queue":
|
||||
self.executor.server.prompt_queue = object()
|
||||
if failure in ("executor", "recursive"):
|
||||
target = self.execution.PromptExecutor()
|
||||
target.server = self.executor.server
|
||||
target.caches = self.executor.caches
|
||||
if failure == "recursive":
|
||||
self.observe("execution_start", target)
|
||||
target = self.executor
|
||||
data = {
|
||||
"prompt_id": "wrong" if failure == "job" else JOB,
|
||||
"nodes": ["p"],
|
||||
}
|
||||
if failure == "generation":
|
||||
self.status = replace(self.status, capture_generation=4)
|
||||
|
||||
def action():
|
||||
host.observe_cached_context(target, "execution_cached", data)
|
||||
|
||||
if failure == "unbound":
|
||||
Context().run(action)
|
||||
elif failure == "thread":
|
||||
thread = threading.Thread(target=action)
|
||||
thread.start()
|
||||
thread.join(2)
|
||||
else:
|
||||
action()
|
||||
if failure == "generation":
|
||||
self.assertIsNone(self.capture.seal(self.cap).bundle)
|
||||
self.status = replace(self.status, capture_generation=3)
|
||||
else:
|
||||
self.assertIsNone(self.seal().positive.effective.value)
|
||||
|
||||
def test_missing_cache_preserves_known_affixes_when_raw_is_dynamic(self):
|
||||
self.begin(raw=["dynamic", 0])
|
||||
self.observe("execution_start")
|
||||
self.observe("execution_cached", nodes=["p"])
|
||||
result = self.seal().positive
|
||||
self.assertIsNone(result.raw.value)
|
||||
self.assertIsNone(result.effective.value)
|
||||
self.assertEqual(result.prepend.value, " pre ")
|
||||
self.assertEqual(result.append.value, " post ")
|
||||
|
||||
def test_cache_entries_tensors_and_executor_are_not_retained(self):
|
||||
class Tensor:
|
||||
pass
|
||||
|
||||
tensor = Tensor()
|
||||
tensor_ref = weakref.ref(tensor)
|
||||
self.begin()
|
||||
self.executor.caches.outputs.entries["p"] = Entry(tensor, [tensor, ["cached"]])
|
||||
self.observe("execution_start")
|
||||
self.observe("execution_cached", nodes=["p"])
|
||||
self.executor.caches.outputs.entries.clear()
|
||||
executor_ref = weakref.ref(self.executor)
|
||||
del tensor
|
||||
del self.executor
|
||||
gc.collect()
|
||||
self.assertIsNone(tensor_ref())
|
||||
self.assertIsNone(executor_ref())
|
||||
self.assertEqual(self.seal().positive.effective.value, "cached")
|
||||
|
||||
def test_unknown_and_malformed_cache_payloads_are_bounded(self):
|
||||
for values in (["x"] * 65, ["x" * 65537], [object()], ["x", "x"]):
|
||||
with self.subTest(count=len(values)):
|
||||
self.begin()
|
||||
self.executor.caches.outputs.entries["p"] = Entry(
|
||||
None, [object(), values]
|
||||
)
|
||||
self.observe("execution_start")
|
||||
self.observe("execution_cached", nodes=["p"])
|
||||
result = self.seal().positive
|
||||
self.assertIsNone(result.effective.value)
|
||||
self.assertEqual(
|
||||
result.append.value, " post " if len(values) != 2 else None
|
||||
)
|
||||
|
||||
def test_disposed_subscriptions_do_not_retain_dynamic_executor_classes(self):
|
||||
def install_and_remove():
|
||||
class TemporaryExecutor:
|
||||
def add_message(self, *args, **kwargs):
|
||||
return None
|
||||
|
||||
self.execution.PromptExecutor = TemporaryExecutor
|
||||
ref = weakref.ref(TemporaryExecutor)
|
||||
stop = host.subscribe_executor_messages("temporary", lambda *args: None)
|
||||
stop()
|
||||
del self.execution.PromptExecutor
|
||||
return ref
|
||||
|
||||
ref = install_and_remove()
|
||||
gc.collect()
|
||||
self.assertIsNone(ref())
|
||||
@@ -0,0 +1,433 @@
|
||||
"""Versioned migration and hostile context restore boundaries."""
|
||||
|
||||
import json
|
||||
import sqlite3
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
from database import execution_history_schema as history
|
||||
from utils.execution_contexts import content_digest
|
||||
|
||||
EPOCH = "11111111-1111-4111-8111-111111111111"
|
||||
RUN = "22222222-2222-4222-8222-222222222222"
|
||||
|
||||
|
||||
def legacy(conn):
|
||||
for table, columns in history._TABLE_COLUMNS.items():
|
||||
definitions = [
|
||||
f"{name} {kind}{' NOT NULL' if required else ''}"
|
||||
f"{' PRIMARY KEY' if primary else ''}"
|
||||
for name, kind, required, primary in columns
|
||||
]
|
||||
checks = (
|
||||
(
|
||||
"singleton_id = 1",
|
||||
"schema_version = 1",
|
||||
history._uuid_check("storage_epoch"),
|
||||
)
|
||||
if table == "execution_history_metadata"
|
||||
else history._RUN_CHECKS
|
||||
)
|
||||
definitions.extend(f"CHECK({check})" for check in checks)
|
||||
conn.execute(f"CREATE TABLE {table} ({', '.join(definitions)})")
|
||||
for name, (unique, columns) in history._INDEXES.items():
|
||||
conn.execute(
|
||||
f"CREATE {'UNIQUE ' if unique else ''}INDEX {name} "
|
||||
f"ON execution_runs({', '.join(columns)})"
|
||||
)
|
||||
conn.execute("INSERT INTO execution_history_metadata VALUES(1,1,?)", (EPOCH,))
|
||||
conn.execute(
|
||||
"INSERT INTO execution_runs "
|
||||
"VALUES(?,?,?,?,?,1,100,NULL,NULL,NULL,'observed',NULL,0,NULL,NULL"
|
||||
",NULL)",
|
||||
(RUN, EPOCH, 0, RUN, EPOCH),
|
||||
)
|
||||
conn.execute(
|
||||
"CREATE TABLE prompts(id INTEGER PRIMARY KEY, text TEXT, run_count INTEGER)"
|
||||
)
|
||||
conn.execute("INSERT INTO prompts VALUES(7,'kept',19)")
|
||||
conn.commit()
|
||||
|
||||
|
||||
class ContextSchemaTests(unittest.TestCase):
|
||||
def connection(self):
|
||||
conn = sqlite3.connect(":memory:")
|
||||
self.addCleanup(conn.close)
|
||||
return conn
|
||||
|
||||
def test_v1_upgrade_preserves_epoch_and_records(self):
|
||||
conn = self.connection()
|
||||
legacy(conn)
|
||||
before = conn.execute("SELECT * FROM execution_runs").fetchall()
|
||||
self.assertIsNone(history.history_schema_error(conn))
|
||||
history.migrate_execution_history(conn)
|
||||
self.assertEqual(
|
||||
conn.execute(
|
||||
"SELECT schema_version,storage_epoch FROM execution_history_metadata"
|
||||
).fetchone(),
|
||||
(2, EPOCH),
|
||||
)
|
||||
self.assertEqual(
|
||||
conn.execute("SELECT * FROM execution_runs").fetchall(), before
|
||||
)
|
||||
self.assertEqual(
|
||||
conn.execute("SELECT * FROM prompts").fetchall(), [(7, "kept", 19)]
|
||||
)
|
||||
for table in (
|
||||
"execution_context_texts",
|
||||
"execution_prompt_snapshots",
|
||||
"execution_contexts",
|
||||
"execution_run_contexts",
|
||||
"execution_context_capture",
|
||||
):
|
||||
self.assertEqual(
|
||||
conn.execute(f"SELECT COUNT(*) FROM {table}").fetchone()[0], 0
|
||||
)
|
||||
|
||||
def test_migration_idempotent(self):
|
||||
conn = self.connection()
|
||||
history.migrate_execution_history(conn)
|
||||
first = "\n".join(conn.iterdump())
|
||||
history.migrate_execution_history(conn)
|
||||
self.assertEqual("\n".join(conn.iterdump()), first)
|
||||
self.assertEqual(
|
||||
conn.execute(
|
||||
"SELECT schema_version FROM execution_history_metadata"
|
||||
).fetchone()[0],
|
||||
2,
|
||||
)
|
||||
|
||||
def test_migration_failure_rolls_back_metadata_and_all_tables(self):
|
||||
conn = self.connection()
|
||||
legacy(conn)
|
||||
before = "\n".join(conn.iterdump())
|
||||
|
||||
def fail(connection):
|
||||
connection.execute("CREATE TABLE execution_context_texts(x TEXT)")
|
||||
raise sqlite3.OperationalError("injected context DDL failure")
|
||||
|
||||
with patch(
|
||||
"database.execution_history_schema.install_context_schema",
|
||||
fail,
|
||||
create=True,
|
||||
):
|
||||
with self.assertRaisesRegex(sqlite3.OperationalError, "injected"):
|
||||
history.migrate_execution_history(conn)
|
||||
self.assertEqual("\n".join(conn.iterdump()), before)
|
||||
|
||||
def test_partial_context_schema_rejected_even_for_v1_or_absent(self):
|
||||
for old in (False, True):
|
||||
conn = self.connection()
|
||||
if old:
|
||||
legacy(conn)
|
||||
conn.execute("CREATE TABLE execution_context_texts(text_id TEXT)")
|
||||
self.assertIsNotNone(history.history_schema_error(conn))
|
||||
with self.assertRaises(sqlite3.DatabaseError):
|
||||
history.migrate_execution_history(conn)
|
||||
|
||||
def populate(self, conn):
|
||||
legacy(conn)
|
||||
history.migrate_execution_history(conn)
|
||||
text = content_digest("text", ("hello",))
|
||||
snapshot = content_digest("snapshot", ("hello", None, None, None))
|
||||
context = content_digest("context", (snapshot, None))
|
||||
conn.execute("INSERT INTO execution_context_texts VALUES(?,?)", (text, "hello"))
|
||||
conn.execute(
|
||||
"INSERT INTO execution_prompt_snapshots VALUES(?,?,NULL,NULL,NULL)",
|
||||
(snapshot, text),
|
||||
)
|
||||
conn.execute(
|
||||
"INSERT INTO execution_contexts VALUES(?,?,NULL)", (context, snapshot)
|
||||
)
|
||||
unknown = dict(provenance="unknown", source="unknown", reason="missing")
|
||||
evidence = dict(
|
||||
format_version=1,
|
||||
role_provenance="inferred",
|
||||
positive=dict(
|
||||
raw=dict(provenance="recorded", source="runtime_return", reason=None),
|
||||
effective=unknown,
|
||||
prepend=unknown,
|
||||
append=unknown,
|
||||
),
|
||||
negative=None,
|
||||
)
|
||||
conn.execute(
|
||||
"INSERT INTO execution_run_contexts VALUES(?,?,?,?,NULL,NULL,NULL,NULL,1)",
|
||||
(RUN, "node", context, json.dumps(evidence)),
|
||||
)
|
||||
conn.execute(
|
||||
"INSERT INTO execution_context_capture VALUES(?,'partial','missing',1)",
|
||||
(RUN,),
|
||||
)
|
||||
return evidence
|
||||
|
||||
def test_owned_identifier_constraints_reject_nul_suffixed_bytes(self):
|
||||
columns = {
|
||||
"execution_context_texts": ("text_id",),
|
||||
"execution_prompt_snapshots": (
|
||||
"snapshot_id",
|
||||
"raw_text_id",
|
||||
"effective_text_id",
|
||||
"prepend_text_id",
|
||||
"append_text_id",
|
||||
),
|
||||
"execution_contexts": (
|
||||
"context_id",
|
||||
"positive_snapshot_id",
|
||||
"negative_snapshot_id",
|
||||
),
|
||||
"execution_run_contexts": (
|
||||
"run_id",
|
||||
"anchor_id",
|
||||
"context_id",
|
||||
"reference_epoch",
|
||||
),
|
||||
"execution_context_capture": ("run_id",),
|
||||
}
|
||||
for table, names in columns.items():
|
||||
for column in names:
|
||||
with self.subTest(table=table, column=column):
|
||||
conn = self.connection()
|
||||
self.populate(conn)
|
||||
prefix = (
|
||||
RUN
|
||||
if column in ("run_id", "reference_epoch")
|
||||
else "node" if column == "anchor_id" else "a" * 64
|
||||
)
|
||||
with self.assertRaises(sqlite3.IntegrityError):
|
||||
conn.execute(
|
||||
f"UPDATE {table} SET {column}=?",
|
||||
(prefix + "\0" + "x" * 1024,),
|
||||
)
|
||||
|
||||
def test_context_validation_rejects_oversized_keys_before_fetch(self):
|
||||
for table, column, prefix in (
|
||||
("execution_context_texts", "text_id", "a" * 64),
|
||||
("execution_run_contexts", "anchor_id", "node"),
|
||||
):
|
||||
with self.subTest(table=table, column=column):
|
||||
conn = self.connection()
|
||||
self.populate(conn)
|
||||
malicious = prefix + "\0" + "x" * 1024
|
||||
if table == "execution_context_texts":
|
||||
for dependent in (
|
||||
"execution_context_capture",
|
||||
"execution_run_contexts",
|
||||
"execution_contexts",
|
||||
"execution_prompt_snapshots",
|
||||
):
|
||||
conn.execute(f"DELETE FROM {dependent}")
|
||||
conn.execute("PRAGMA ignore_check_constraints=ON")
|
||||
conn.execute(f"UPDATE {table} SET {column}=?", (malicious,))
|
||||
|
||||
def bounded_text(value):
|
||||
self.assertNotEqual(value, malicious.encode())
|
||||
return value.decode()
|
||||
|
||||
conn.text_factory = bounded_text
|
||||
self.assertIsNotNone(history.history_schema_error(conn))
|
||||
|
||||
def test_unicode_anchor_boundary_and_prompt_nuls_preserve_exact_text(self):
|
||||
conn = self.connection()
|
||||
self.populate(conn)
|
||||
text = "hello\0world"
|
||||
text_id = content_digest("text", (text,))
|
||||
snapshot_id = content_digest("snapshot", (text, None, None, None))
|
||||
context_id = content_digest("context", (snapshot_id, None))
|
||||
conn.execute(
|
||||
"UPDATE execution_context_texts SET text_id=?,text_value=?",
|
||||
(text_id, text),
|
||||
)
|
||||
conn.execute(
|
||||
"UPDATE execution_prompt_snapshots SET snapshot_id=?,raw_text_id=?",
|
||||
(snapshot_id, text_id),
|
||||
)
|
||||
conn.execute(
|
||||
"UPDATE execution_contexts SET context_id=?,positive_snapshot_id=?",
|
||||
(context_id, snapshot_id),
|
||||
)
|
||||
conn.execute(
|
||||
"UPDATE execution_run_contexts SET context_id=?,anchor_id=?",
|
||||
(context_id, "😀" * 128),
|
||||
)
|
||||
self.assertIsNone(history.history_schema_error(conn))
|
||||
self.assertEqual(
|
||||
conn.execute("SELECT text_value FROM execution_context_texts").fetchone(),
|
||||
(text,),
|
||||
)
|
||||
|
||||
def test_link_without_capture_header_is_malformed_v2(self):
|
||||
from database.execution_context_schema import context_schema_error
|
||||
|
||||
conn = self.connection()
|
||||
self.populate(conn)
|
||||
conn.execute("DELETE FROM execution_context_capture")
|
||||
before = "\n".join(conn.iterdump())
|
||||
self.assertIsNotNone(context_schema_error(conn))
|
||||
self.assertIsNotNone(history.history_schema_error(conn, allow_missing=False))
|
||||
self.assertEqual("\n".join(conn.iterdump()), before)
|
||||
|
||||
def test_valid_context_and_fingerprints(self):
|
||||
conn = self.connection()
|
||||
self.populate(conn)
|
||||
self.assertIsNone(history.history_schema_error(conn, allow_missing=False))
|
||||
|
||||
def test_v2_rejects_partial_schema_extra_uniqueness_bad_fingerprint_or_orphan(self):
|
||||
mutations = (
|
||||
"DROP TABLE execution_context_capture",
|
||||
"CREATE UNIQUE INDEX extra_text ON execution_context_texts(text_value)",
|
||||
"UPDATE execution_context_texts SET text_value='changed'",
|
||||
"UPDATE execution_prompt_snapshots SET raw_text_id='" + "0" * 64 + "'",
|
||||
"UPDATE execution_contexts SET context_id='" + "0" * 64 + "'",
|
||||
"UPDATE execution_run_contexts SET "
|
||||
"run_id='33333333-3333-4333-8333-333333333333'",
|
||||
"UPDATE execution_run_contexts SET evidence_json='{}'",
|
||||
"UPDATE execution_context_capture SET "
|
||||
"run_id='33333333-3333-4333-8333-333333333333'",
|
||||
)
|
||||
for mutation in mutations:
|
||||
with self.subTest(mutation=mutation):
|
||||
conn = self.connection()
|
||||
self.populate(conn)
|
||||
conn.execute(mutation)
|
||||
self.assertIsNotNone(history.history_schema_error(conn))
|
||||
|
||||
def test_evidence_rejects_extra_metadata_and_false_unknown(self):
|
||||
for change in ("extra", "known", "role", "version"):
|
||||
conn = self.connection()
|
||||
evidence = self.populate(conn)
|
||||
if change == "extra":
|
||||
evidence["graph"] = {}
|
||||
elif change == "known":
|
||||
evidence["positive"]["effective"] = dict(
|
||||
provenance="inferred", source="graph_literal", reason=None
|
||||
)
|
||||
elif change == "role":
|
||||
evidence["negative"] = evidence["positive"]
|
||||
else:
|
||||
evidence["format_version"] = True
|
||||
conn.execute(
|
||||
"UPDATE execution_run_contexts SET evidence_json=?",
|
||||
(json.dumps(evidence),),
|
||||
)
|
||||
self.assertIsNotNone(history.history_schema_error(conn))
|
||||
|
||||
def test_more_than_sixteen_context_links_rejected(self):
|
||||
conn = self.connection()
|
||||
self.populate(conn)
|
||||
for i in range(16):
|
||||
conn.execute(
|
||||
"INSERT INTO execution_run_contexts SELECT "
|
||||
"run_id,?,context_id,evidence_json,positive_prompt_id,negative_pro"
|
||||
"mpt_id,pair_id,reference_epoch,capture_version FROM "
|
||||
"execution_run_contexts LIMIT 1",
|
||||
(str(i),),
|
||||
)
|
||||
self.assertIsNotNone(history.history_schema_error(conn))
|
||||
|
||||
def test_owned_schema_rejects_triggers_views_and_generated_columns(self):
|
||||
for mutation in (
|
||||
"CREATE TRIGGER hidden_write AFTER INSERT ON "
|
||||
"execution_context_texts BEGIN DELETE FROM "
|
||||
"execution_context_texts; END",
|
||||
"CREATE VIEW context_view AS SELECT * FROM execution_context_texts",
|
||||
"ALTER TABLE execution_context_texts ADD COLUMN hidden TEXT "
|
||||
"GENERATED ALWAYS AS (text_value) VIRTUAL",
|
||||
):
|
||||
with self.subTest(mutation=mutation):
|
||||
conn = self.connection()
|
||||
self.populate(conn)
|
||||
conn.execute(mutation)
|
||||
self.assertIsNotNone(history.history_schema_error(conn))
|
||||
|
||||
def test_duplicate_json_keys_and_unbounded_or_false_complete_rows_rejected(self):
|
||||
for change in (
|
||||
"duplicate",
|
||||
"oversized",
|
||||
"reason",
|
||||
"complete",
|
||||
"empty_complete",
|
||||
):
|
||||
with self.subTest(change=change):
|
||||
conn = self.connection()
|
||||
self.populate(conn)
|
||||
conn.execute("PRAGMA ignore_check_constraints=ON")
|
||||
if change == "duplicate":
|
||||
conn.execute(
|
||||
"UPDATE execution_run_contexts SET "
|
||||
"evidence_json=replace(evidence_json, '{', "
|
||||
"'{\"format_version\":1,')"
|
||||
)
|
||||
elif change == "oversized":
|
||||
conn.execute(
|
||||
"UPDATE execution_context_texts SET text_value=?",
|
||||
("é" * 32769,),
|
||||
)
|
||||
elif change == "reason":
|
||||
conn.execute(
|
||||
"UPDATE execution_context_capture SET reason='arbitrary' "
|
||||
)
|
||||
else:
|
||||
conn.execute(
|
||||
"UPDATE execution_context_capture SET state='complete'"
|
||||
)
|
||||
if change == "empty_complete":
|
||||
conn.execute("DELETE FROM execution_run_contexts")
|
||||
self.assertIsNotNone(history.history_schema_error(conn))
|
||||
|
||||
def test_restore_enforces_context_and_run_aggregate_payload_bounds(self):
|
||||
for size, links, valid in (
|
||||
(40000, 0, False),
|
||||
(40000, 1, False),
|
||||
(25000, 5, True),
|
||||
(25000, 6, False),
|
||||
):
|
||||
with self.subTest(size=size, links=links):
|
||||
conn = self.connection()
|
||||
evidence = self.populate(conn)
|
||||
text = "x" * size
|
||||
text_id = content_digest("text", (text,))
|
||||
snapshot_id = content_digest("snapshot", (text,) * 4)
|
||||
context_id = content_digest("context", (snapshot_id, snapshot_id))
|
||||
conn.execute(
|
||||
"UPDATE execution_context_texts SET text_id=?,text_value=?",
|
||||
(text_id, text),
|
||||
)
|
||||
conn.execute(
|
||||
"UPDATE execution_prompt_snapshots SET "
|
||||
"snapshot_id=?,raw_text_id=?,effective_text_id=?,prepend_text_id=?"
|
||||
",append_text_id=?",
|
||||
(snapshot_id,) + (text_id,) * 4,
|
||||
)
|
||||
conn.execute(
|
||||
"UPDATE execution_contexts SET "
|
||||
"context_id=?,positive_snapshot_id=?,negative_snapshot_id=?",
|
||||
(context_id, snapshot_id, snapshot_id),
|
||||
)
|
||||
known = dict(
|
||||
provenance="recorded", source="runtime_return", reason=None
|
||||
)
|
||||
evidence["positive"] = dict.fromkeys(
|
||||
("raw", "effective", "prepend", "append"), known
|
||||
)
|
||||
evidence["negative"] = evidence["positive"]
|
||||
conn.execute(
|
||||
"UPDATE execution_run_contexts SET context_id=?,evidence_json=?",
|
||||
(context_id, json.dumps(evidence)),
|
||||
)
|
||||
for i in range(links - 1):
|
||||
conn.execute(
|
||||
"INSERT INTO execution_run_contexts SELECT "
|
||||
"run_id,?,context_id,evidence_json,positive_prompt_id,"
|
||||
"negative_prompt_id,pair_id,reference_epoch,"
|
||||
"capture_version FROM "
|
||||
"execution_run_contexts LIMIT 1",
|
||||
(str(i),),
|
||||
)
|
||||
if not links:
|
||||
conn.execute("DELETE FROM execution_run_contexts")
|
||||
error = history.history_schema_error(conn)
|
||||
if valid:
|
||||
self.assertIsNone(error)
|
||||
else:
|
||||
self.assertIsNotNone(error)
|
||||
@@ -0,0 +1,218 @@
|
||||
"""Exact identities and bounded scalar records; normalization would lose history."""
|
||||
|
||||
import hashlib
|
||||
import importlib.util
|
||||
import sys
|
||||
import types
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def load_module(name):
|
||||
package = "execution_context_test_utils"
|
||||
if package not in sys.modules:
|
||||
module = types.ModuleType(package)
|
||||
module.__path__ = [str(Path(__file__).parents[1] / "utils")]
|
||||
sys.modules[package] = module
|
||||
qualified = package + "." + name
|
||||
if qualified not in sys.modules:
|
||||
spec = importlib.util.spec_from_file_location(
|
||||
qualified, Path(__file__).parents[1] / "utils" / (name + ".py")
|
||||
)
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
sys.modules[qualified] = module
|
||||
spec.loader.exec_module(module)
|
||||
return sys.modules[qualified]
|
||||
|
||||
|
||||
class ExactContextsTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.m = load_module("execution_contexts")
|
||||
|
||||
def evidence(self, value, provenance="inferred"):
|
||||
return self.m.TextEvidence(
|
||||
value,
|
||||
"unknown" if value is None else provenance,
|
||||
(
|
||||
"unknown"
|
||||
if value is None
|
||||
else "runtime_return" if provenance == "recorded" else "graph_literal"
|
||||
),
|
||||
"unavailable" if value is None else None,
|
||||
)
|
||||
|
||||
def snapshot(self, value):
|
||||
return self.m.PromptSnapshot(
|
||||
*(self.evidence(v) for v in (value, value, "", ""))
|
||||
)
|
||||
|
||||
def test_exact_identity(self):
|
||||
m = self.m
|
||||
self.assertEqual(
|
||||
m.canonical_content("snapshot", ("é", None, "", "")),
|
||||
'[1,"snapshot","é",null,"",""]'.encode(),
|
||||
)
|
||||
values = ("cat", "Cat", " cat", "cat ", "é", "e\u0301", None, "")
|
||||
digests = {m.content_digest("snapshot", (v, v, "", "")) for v in values}
|
||||
self.assertEqual(len(digests), len(values))
|
||||
self.assertEqual(
|
||||
m.content_digest("snapshot", ("cat", "cat", "", "")),
|
||||
hashlib.sha256(b'[1,"snapshot","cat","cat","",""]').hexdigest(),
|
||||
)
|
||||
a = self.snapshot("cat")
|
||||
b = m.PromptSnapshot(
|
||||
*(self.evidence(v, "recorded") for v in ("cat", "cat", "", ""))
|
||||
)
|
||||
self.assertEqual(m.snapshot_digest(a), m.snapshot_digest(b))
|
||||
self.assertNotEqual(
|
||||
m.context_digest(m.RecipeContext("1", a, None, "inferred")),
|
||||
m.context_digest(m.RecipeContext("1", None, a, "inferred")),
|
||||
)
|
||||
|
||||
def test_invalid_text_and_bounds(self):
|
||||
m = self.m
|
||||
for value in ("\ud800", "x" * 65537, "é" * 32769, object(), True):
|
||||
with self.subTest(value_type=type(value).__name__):
|
||||
with self.assertRaises(ValueError):
|
||||
m.canonical_content("snapshot", (value,))
|
||||
m.canonical_content("snapshot", ("é" * 32768,))
|
||||
for identifier in ("x" * 129, object()):
|
||||
with self.assertRaises(ValueError):
|
||||
m.validate_bundle(
|
||||
m.ContextBundle(
|
||||
(m.RecipeContext(identifier, None, None, "inferred"),),
|
||||
"unavailable",
|
||||
"missing",
|
||||
)
|
||||
)
|
||||
with self.assertRaises(ValueError):
|
||||
m.validate_bundle(
|
||||
m.ContextBundle(
|
||||
tuple(
|
||||
m.RecipeContext(str(i), None, None, "inferred")
|
||||
for i in range(17)
|
||||
),
|
||||
"partial",
|
||||
"limit",
|
||||
)
|
||||
)
|
||||
with self.assertRaises(ValueError):
|
||||
m.validate_bundle(
|
||||
m.ContextBundle(
|
||||
(m.RecipeContext("1", self.snapshot("\ud800"), None, "inferred"),),
|
||||
"complete",
|
||||
"",
|
||||
)
|
||||
)
|
||||
|
||||
def test_bundle_preserves_unknown_and_empty(self):
|
||||
m = self.m
|
||||
bundle = m.ContextBundle(
|
||||
(m.RecipeContext("1", self.snapshot(""), self.snapshot(None), "inferred"),),
|
||||
"partial",
|
||||
"unknown",
|
||||
)
|
||||
self.assertIsNone(m.validate_bundle(bundle))
|
||||
with self.assertRaises(ValueError):
|
||||
m.validate_bundle(
|
||||
m.ContextBundle(bundle.contexts, "partial", "", format_version=2)
|
||||
)
|
||||
|
||||
def test_evidence_contract_rejects_incompatible_metadata(self):
|
||||
m = self.m
|
||||
invalid = (
|
||||
("", "graph-inferred", "graph_literal", None),
|
||||
("", "recorded", "graph_literal", None),
|
||||
("", "inferred", "runtime_return", None),
|
||||
(None, "inferred", "graph_literal", None),
|
||||
("", "unknown", "unknown", "unavailable"),
|
||||
(None, "unknown", "unknown", None),
|
||||
(None, "unknown", "unknown", "an arbitrary exception message"),
|
||||
)
|
||||
for fields in invalid:
|
||||
with self.subTest(fields=fields):
|
||||
evidence = m.TextEvidence(*fields)
|
||||
snapshot = m.PromptSnapshot(evidence, evidence, evidence, evidence)
|
||||
with self.assertRaises(ValueError):
|
||||
m.validate_bundle(
|
||||
m.ContextBundle(
|
||||
(m.RecipeContext("1", snapshot, None, "inferred"),),
|
||||
"partial",
|
||||
"",
|
||||
)
|
||||
)
|
||||
for provenance, source in (
|
||||
("recorded", "runtime_return"),
|
||||
("recorded", "host_cache_output"),
|
||||
("inferred", "graph_literal"),
|
||||
("inferred", "graph_semantics"),
|
||||
):
|
||||
evidence = m.TextEvidence("", provenance, source, None)
|
||||
snapshot = m.PromptSnapshot(evidence, evidence, evidence, evidence)
|
||||
m.validate_bundle(
|
||||
m.ContextBundle(
|
||||
(m.RecipeContext("1", snapshot, None, "inferred"),), "complete", ""
|
||||
)
|
||||
)
|
||||
|
||||
def test_bundle_state_and_declared_role_contract(self):
|
||||
m = self.m
|
||||
for state, role in (
|
||||
("unknown", "inferred"),
|
||||
("complete", "recorded"),
|
||||
("complete", "declared"),
|
||||
):
|
||||
with self.subTest(state=state, role=role):
|
||||
with self.assertRaises(ValueError):
|
||||
m.validate_bundle(
|
||||
m.ContextBundle(
|
||||
(m.RecipeContext("1", None, None, role),), state, ""
|
||||
)
|
||||
)
|
||||
for state in ("complete", "partial", "ambiguous", "unavailable", "conflicting"):
|
||||
m.validate_bundle(
|
||||
m.ContextBundle(
|
||||
(m.RecipeContext("1", None, None, "inferred"),), state, ""
|
||||
)
|
||||
)
|
||||
|
||||
def test_identifier_limit_counts_characters(self):
|
||||
m = self.m
|
||||
m.validate_bundle(
|
||||
m.ContextBundle(
|
||||
(m.RecipeContext("é" * 128, None, None, "inferred"),), "unavailable", ""
|
||||
)
|
||||
)
|
||||
with self.assertRaises(ValueError):
|
||||
m.validate_bundle(
|
||||
m.ContextBundle(
|
||||
(m.RecipeContext("é" * 129, None, None, "inferred"),),
|
||||
"unavailable",
|
||||
"",
|
||||
)
|
||||
)
|
||||
|
||||
def test_aggregate_payload_limits(self):
|
||||
m = self.m
|
||||
large = self.snapshot("x" * 65536)
|
||||
with self.assertRaises(ValueError):
|
||||
m.validate_bundle(
|
||||
m.ContextBundle(
|
||||
(m.RecipeContext("1", large, large, "inferred"),), "complete", ""
|
||||
)
|
||||
)
|
||||
contexts = tuple(
|
||||
m.RecipeContext(str(i), large, None, "inferred") for i in range(8)
|
||||
)
|
||||
with self.assertRaises(ValueError):
|
||||
m.validate_bundle(m.ContextBundle(contexts, "complete", ""))
|
||||
|
||||
def test_equal_repeated_samples_are_ambiguous(self):
|
||||
# Projection contains references, never runtime samples. Duplicate anchors
|
||||
# remain separate invalid records even if their prompt values are equal.
|
||||
m = self.m
|
||||
context = m.RecipeContext(
|
||||
"1", self.snapshot("same"), self.snapshot("same"), "inferred"
|
||||
)
|
||||
with self.assertRaises(ValueError):
|
||||
m.validate_bundle(m.ContextBundle((context, context), "complete", ""))
|
||||
@@ -95,6 +95,10 @@ class HistoryDatabaseTests(unittest.TestCase):
|
||||
for pre_pair in (False, True):
|
||||
with self.subTest(pre_pair=pre_pair):
|
||||
with conn:
|
||||
from database.execution_context_schema import TABLE_COLUMNS
|
||||
|
||||
for table in reversed(TABLE_COLUMNS):
|
||||
conn.execute(f"DROP TABLE {table}")
|
||||
conn.execute("DROP TABLE execution_runs")
|
||||
conn.execute("DROP TABLE execution_history_metadata")
|
||||
if pre_pair:
|
||||
@@ -232,7 +236,7 @@ class HistoryDatabaseTests(unittest.TestCase):
|
||||
else:
|
||||
conn.execute(
|
||||
"INSERT INTO execution_history_metadata "
|
||||
"VALUES(1,1,?)",
|
||||
"VALUES(1,2,?)",
|
||||
(epoch,),
|
||||
)
|
||||
else:
|
||||
@@ -432,7 +436,7 @@ class HistoryDatabaseTests(unittest.TestCase):
|
||||
"sql=replace(sql, 'queue_item_id >= 0', 'queue_item_id >= -1') "
|
||||
"WHERE name='execution_runs'; PRAGMA writable_schema=OFF",
|
||||
"PRAGMA ignore_check_constraints=ON; "
|
||||
"UPDATE execution_history_metadata SET schema_version=2",
|
||||
"UPDATE execution_history_metadata SET schema_version=3",
|
||||
"PRAGMA writable_schema=ON; UPDATE sqlite_master SET "
|
||||
"sql=replace(sql, \"'observed'\", \"'OBSERVED'\") "
|
||||
"WHERE name='execution_runs'; PRAGMA writable_schema=OFF",
|
||||
|
||||
@@ -76,6 +76,8 @@ class HistoryHooksTests(unittest.TestCase):
|
||||
logger.handlers, logger.propagate, logger.level = previous
|
||||
|
||||
def cleanup_registrations(self):
|
||||
for registration in self.registrations:
|
||||
registration.writer.invalidate()
|
||||
for release in self.releases:
|
||||
release.set()
|
||||
for registration in self.registrations:
|
||||
@@ -356,7 +358,7 @@ class HistoryHooksTests(unittest.TestCase):
|
||||
clock = ManualClock()
|
||||
with patch(
|
||||
"utils.execution_history.HistoryWriter",
|
||||
lambda factory: HistoryWriter(factory, clock=clock),
|
||||
lambda factory, **kwargs: HistoryWriter(factory, clock=clock, **kwargs),
|
||||
):
|
||||
registration = self.register()
|
||||
for index in range(128):
|
||||
@@ -448,3 +450,310 @@ class HistoryHooksTests(unittest.TestCase):
|
||||
self.drain(registration)
|
||||
self.assertEqual(seen, [JOB])
|
||||
self.assertEqual(len(self.rows()), 1)
|
||||
|
||||
def test_fixture_cleanup_discards_saturation_before_release_and_database_close(
|
||||
self,
|
||||
):
|
||||
from utils.execution_history_events import make_start
|
||||
|
||||
entered, release, rescue = (threading.Event() for _ in range(3))
|
||||
self.releases.append(release)
|
||||
pending_at_release, writes = [], []
|
||||
original = self.db.write_execution_start
|
||||
|
||||
def blocked(command):
|
||||
entered.set()
|
||||
release.wait(5)
|
||||
pending = registration.writer.status().pending
|
||||
pending_at_release.append(pending)
|
||||
# Model a slow drain without executing 1,024 SQLite transactions.
|
||||
if pending > 1:
|
||||
rescue.wait(5)
|
||||
writes.append(command.run_id)
|
||||
return original(command)
|
||||
|
||||
with patch.object(self.db, "write_execution_start", blocked):
|
||||
registration = self.register()
|
||||
try:
|
||||
self.dequeue(1)
|
||||
self.assertTrue(entered.wait(1))
|
||||
epoch = registration.writer.status().epoch
|
||||
for i in range(1023):
|
||||
command = make_start(
|
||||
registration._session_id, item(1000 + i), epoch, 0
|
||||
)
|
||||
self.assertTrue(registration.writer.submit(command))
|
||||
self.assertEqual(registration.writer.status().pending, 1024)
|
||||
before = time.monotonic()
|
||||
self.cleanup_registrations()
|
||||
self.assertLess(time.monotonic() - before, 2)
|
||||
self.assertEqual(pending_at_release, [1])
|
||||
self.assertEqual(len(writes), 1)
|
||||
self.assertFalse(registration.writer.status().alive)
|
||||
self.assertEqual(registration.writer.status().pending, 0)
|
||||
self.db.model.close_all()
|
||||
finally:
|
||||
registration.writer.invalidate()
|
||||
rescue.set()
|
||||
release.set()
|
||||
self.assertTrue(
|
||||
registration.dispose(3), "regression rescue leaked worker"
|
||||
)
|
||||
|
||||
def test_rejected_new_start_revokes_previous_worker_capability(self):
|
||||
from utils.execution_context_capture import active_capture
|
||||
from utils.execution_history_events import make_start
|
||||
|
||||
entered, release = threading.Event(), threading.Event()
|
||||
self.releases.append(release)
|
||||
original = self.db.write_execution_start
|
||||
|
||||
def blocked(command):
|
||||
entered.set()
|
||||
release.wait(5)
|
||||
return original(command)
|
||||
|
||||
with patch.object(self.db, "write_execution_start", blocked):
|
||||
registration = self.register()
|
||||
self.dequeue(1)
|
||||
self.assertTrue(entered.wait(1))
|
||||
previous = active_capture()
|
||||
self.assertIsNotNone(previous)
|
||||
epoch = registration.writer.status().epoch
|
||||
for i in range(1023):
|
||||
command = make_start(registration._session_id, item(1000 + i), epoch, 0)
|
||||
self.assertTrue(registration.writer.submit(command))
|
||||
self.assertEqual(registration.writer.status().pending, 1024)
|
||||
self.dequeue(2)
|
||||
self.assertIsNone(active_capture())
|
||||
self.assertFalse(previous[0].valid(previous[1]))
|
||||
registration.writer.invalidate()
|
||||
release.set()
|
||||
self.assertTrue(registration.dispose(3), "owned worker leaked")
|
||||
|
||||
def test_late_finish_does_not_clear_new_attempt_binding(self):
|
||||
from utils.execution_context_capture import active_capture
|
||||
|
||||
registration = self.register()
|
||||
self.dequeue(1)
|
||||
self.dequeue(2)
|
||||
current = active_capture()
|
||||
self.assertIsNotNone(current)
|
||||
self.queue.task_done(1, {}, status())
|
||||
self.assertEqual(active_capture(), current)
|
||||
# An identical callback replay keeps the already accepted binding.
|
||||
registration._dequeue(item(2))
|
||||
self.assertEqual(active_capture(), current)
|
||||
self.queue.task_done(2, {}, status())
|
||||
self.assertIsNone(active_capture())
|
||||
|
||||
def test_invalidation_releases_capture_immediately_outside_writer_locks(self):
|
||||
from utils.execution_context_capture import active_capture
|
||||
|
||||
registration = self.register()
|
||||
self.dequeue(1)
|
||||
current = active_capture()
|
||||
pool = registration.writer.context_budget
|
||||
self.assertGreater(pool.charged_bytes, 0)
|
||||
registration.writer.invalidate()
|
||||
self.assertEqual(pool.charged_bytes, 0)
|
||||
self.assertFalse(current[0].valid(current[1]))
|
||||
|
||||
def test_replayed_older_accepted_dequeue_cannot_revoke_current_binding(self):
|
||||
from utils.execution_context_capture import active_capture
|
||||
|
||||
registration = self.register()
|
||||
old = self.dequeue(1)
|
||||
self.dequeue(2)
|
||||
current = active_capture()
|
||||
registration._dequeue(old)
|
||||
self.assertEqual(active_capture(), current)
|
||||
|
||||
def test_broken_finish_releases_active_capture(self):
|
||||
from utils.execution_context_capture import active_capture
|
||||
|
||||
registration = self.register()
|
||||
self.dequeue(1)
|
||||
|
||||
class Broken:
|
||||
@property
|
||||
def messages(self):
|
||||
raise RuntimeError("broken status")
|
||||
|
||||
self.queue.task_done(1, {}, Broken())
|
||||
self.assertEqual(registration.writer.context_budget.charged_bytes, 0)
|
||||
self.assertIsNone(active_capture())
|
||||
|
||||
def test_finished_accepted_replay_preserves_new_same_uuid_attempt(self):
|
||||
from utils.execution_context_capture import active_capture
|
||||
|
||||
registration = self.register()
|
||||
old = self.dequeue(1)
|
||||
self.queue.task_done(1, {}, status())
|
||||
self.dequeue(2)
|
||||
current = active_capture()
|
||||
registration._dequeue(old)
|
||||
self.assertEqual(active_capture(), current)
|
||||
self.queue.task_done(2, {}, status())
|
||||
self.drain(registration)
|
||||
self.assertEqual(len(self.rows()), 2)
|
||||
|
||||
def test_observed_upstream_survives_downstream_failure_in_database(self):
|
||||
from utils.execution_context_capture import record_runtime_context
|
||||
|
||||
g = {
|
||||
"p": {
|
||||
"class_type": "PromptManagerPair",
|
||||
"inputs": {"text": "raw", "negative_text": ""},
|
||||
},
|
||||
"s": {
|
||||
"class_type": "KSampler",
|
||||
"inputs": {"positive": ["p", 0], "negative": ["p", 1]},
|
||||
},
|
||||
}
|
||||
registration = self.register()
|
||||
self.queue.results.append(((0, JOB, g, {}, ["s"]), 1))
|
||||
self.queue.get()
|
||||
host = types.ModuleType("comfy_execution.utils")
|
||||
host.get_executing_context = lambda: SimpleNamespace(
|
||||
prompt_id=JOB, node_id="p", list_index=0
|
||||
)
|
||||
with patch.dict(sys.modules, {"comfy_execution.utils": host}):
|
||||
record_runtime_context(
|
||||
"PromptManagerPair",
|
||||
" raw ",
|
||||
" pre ",
|
||||
"",
|
||||
"pre raw TRIGGER",
|
||||
role="positive",
|
||||
)
|
||||
record_runtime_context("PromptManagerPair", "", "", "", "", role="negative")
|
||||
self.queue.task_done(1, {}, status("execution_error"))
|
||||
self.drain(registration)
|
||||
row = self.rows()[0]
|
||||
self.assertEqual(row["outcome"], "failed")
|
||||
result = self.db.get_execution_contexts(row["run_id"])
|
||||
self.assertEqual(result["state"], "complete")
|
||||
self.assertEqual(
|
||||
result["contexts"][0]["positive"],
|
||||
{
|
||||
"raw": " raw ",
|
||||
"effective": "pre raw TRIGGER",
|
||||
"prepend": " pre ",
|
||||
"append": "",
|
||||
},
|
||||
)
|
||||
self.assertEqual(result["contexts"][0]["negative"]["effective"], "")
|
||||
|
||||
def test_close_releases_capture_while_writer_is_blocked(self):
|
||||
entered, release = threading.Event(), threading.Event()
|
||||
self.releases.append(release)
|
||||
original = self.db.write_execution_start
|
||||
|
||||
def blocked(command):
|
||||
entered.set()
|
||||
release.wait(3)
|
||||
return original(command)
|
||||
|
||||
with patch.object(self.db, "write_execution_start", blocked):
|
||||
registration = self.register()
|
||||
self.dequeue(1)
|
||||
self.assertTrue(entered.wait(1))
|
||||
self.assertGreater(registration.writer.context_budget.charged_bytes, 0)
|
||||
self.assertFalse(registration.writer.close(0))
|
||||
self.assertEqual(registration.writer.context_budget.charged_bytes, 0)
|
||||
release.set()
|
||||
|
||||
def test_reregistration_cannot_rebind_already_observed_queue_item(self):
|
||||
from utils.execution_context_capture import active_capture
|
||||
|
||||
first = self.register()
|
||||
self.dequeue(10)
|
||||
self.queue.task_done(10, {}, status())
|
||||
self.drain(first)
|
||||
self.assertTrue(first.dispose())
|
||||
second = self.register()
|
||||
self.dequeue(10)
|
||||
self.assertIsNone(active_capture())
|
||||
self.queue.task_done(10, {}, status())
|
||||
self.drain(second)
|
||||
self.assertEqual(len(self.rows()), 1)
|
||||
|
||||
def test_cached_executor_subscription_persists_strings_and_disposes_cooperatively(
|
||||
self,
|
||||
):
|
||||
from utils.execution_context_host import subscribe_executor_messages
|
||||
|
||||
calls = []
|
||||
execution = types.ModuleType("execution")
|
||||
caching = types.ModuleType("comfy_execution.caching")
|
||||
package = types.ModuleType("comfy_execution")
|
||||
entry_type = namedtuple("CacheEntry", "ui outputs")
|
||||
|
||||
class PromptExecutor:
|
||||
def add_message(self, event, data, broadcast):
|
||||
calls.append(event)
|
||||
return "host-result"
|
||||
|
||||
class HierarchicalCache:
|
||||
def get_local(self, node_id):
|
||||
return entry_type(None, [object(), object(), ["old trigger"], [""]])
|
||||
|
||||
execution.PromptExecutor, execution.CacheEntry = PromptExecutor, entry_type
|
||||
caching.HierarchicalCache = HierarchicalCache
|
||||
package.caching = caching
|
||||
with patch.dict(
|
||||
sys.modules,
|
||||
{
|
||||
"execution": execution,
|
||||
"comfy_execution": package,
|
||||
"comfy_execution.caching": caching,
|
||||
},
|
||||
):
|
||||
observer_calls = []
|
||||
independent = subscribe_executor_messages(
|
||||
"independent", lambda *args: observer_calls.append(args)
|
||||
)
|
||||
try:
|
||||
registration = self.register()
|
||||
g = {
|
||||
"p": {
|
||||
"class_type": "PromptManagerPair",
|
||||
"inputs": {"text": "raw", "negative_text": ""},
|
||||
},
|
||||
"s": {
|
||||
"class_type": "KSampler",
|
||||
"inputs": {"positive": ["p", 0], "negative": ["p", 1]},
|
||||
},
|
||||
}
|
||||
self.queue.results.append(((0, JOB, g, {}, ["s"]), 1))
|
||||
self.queue.get()
|
||||
executor = PromptExecutor()
|
||||
executor.server = self.server
|
||||
executor.caches = SimpleNamespace(outputs=HierarchicalCache())
|
||||
self.assertEqual(
|
||||
executor.add_message("execution_start", {"prompt_id": JOB}, False),
|
||||
"host-result",
|
||||
)
|
||||
executor.add_message(
|
||||
event="execution_cached",
|
||||
data={"prompt_id": JOB, "nodes": ["p"]},
|
||||
broadcast=False,
|
||||
)
|
||||
self.queue.task_done(1, {}, status())
|
||||
self.drain(registration)
|
||||
result = self.db.get_execution_contexts(self.rows()[0]["run_id"])
|
||||
self.assertEqual(
|
||||
result["contexts"][0]["positive"]["effective"], "old trigger"
|
||||
)
|
||||
self.assertEqual(result["contexts"][0]["negative"]["effective"], "")
|
||||
self.assertTrue(registration.dispose())
|
||||
executor.add_message(
|
||||
"execution_cached", {"prompt_id": JOB, "nodes": ["p"]}, False
|
||||
)
|
||||
self.assertEqual(len(observer_calls), 3)
|
||||
finally:
|
||||
independent()
|
||||
self.assertEqual(
|
||||
calls, ["execution_start", "execution_cached", "execution_cached"]
|
||||
)
|
||||
|
||||
@@ -22,6 +22,91 @@ class HistoryRestoreTests(unittest.TestCase):
|
||||
finish = history_tests.HistoryDatabaseTests.finish
|
||||
count = history_tests.HistoryDatabaseTests.count
|
||||
|
||||
def test_legacy_identifier_and_diagnostic_bytes_rejected_before_fetch(self):
|
||||
try:
|
||||
from tests.test_execution_context_schema import legacy, RUN
|
||||
except ImportError:
|
||||
from test_execution_context_schema import legacy, RUN
|
||||
from database.execution_history_schema import history_schema_error
|
||||
|
||||
for table, column, prefix in (
|
||||
("execution_history_metadata", "storage_epoch", RUN),
|
||||
*(
|
||||
("execution_runs", column, RUN)
|
||||
for column in (
|
||||
"run_id",
|
||||
"session_id",
|
||||
"job_id",
|
||||
"storage_epoch",
|
||||
)
|
||||
),
|
||||
("execution_runs", "node_id", "x" * 128),
|
||||
("execution_runs", "node_type", "x" * 128),
|
||||
("execution_runs", "exception_type", "x" * 256),
|
||||
):
|
||||
with self.subTest(table=table, column=column):
|
||||
with closing(sqlite3.connect(":memory:")) as conn:
|
||||
legacy(conn)
|
||||
malicious = prefix + "\0" + "x" * 1024
|
||||
conn.execute(f"UPDATE {table} SET {column}=?", (malicious,))
|
||||
|
||||
def bounded_text(value):
|
||||
self.assertNotEqual(value, malicious.encode())
|
||||
return value.decode()
|
||||
|
||||
conn.text_factory = bounded_text
|
||||
self.assertIsNotNone(history_schema_error(conn))
|
||||
|
||||
def test_legacy_metadata_version_payload_is_rejected_before_fetch(self):
|
||||
try:
|
||||
from tests.test_execution_context_schema import legacy
|
||||
except ImportError:
|
||||
from test_execution_context_schema import legacy
|
||||
from database.execution_history_schema import history_schema_error
|
||||
|
||||
for malicious in ("x" * 1024, b"x" * 1024):
|
||||
with self.subTest(kind=type(malicious).__name__):
|
||||
with closing(sqlite3.connect(":memory:")) as conn:
|
||||
legacy(conn)
|
||||
conn.execute("PRAGMA ignore_check_constraints=ON")
|
||||
conn.execute(
|
||||
"UPDATE execution_history_metadata SET schema_version=?",
|
||||
(malicious,),
|
||||
)
|
||||
|
||||
def bounded_row(cursor, row):
|
||||
self.assertFalse(any(value == malicious for value in row))
|
||||
return row
|
||||
|
||||
conn.row_factory = bounded_row
|
||||
self.assertIsNotNone(history_schema_error(conn))
|
||||
|
||||
def test_legacy_four_byte_diagnostics_validate_and_migrate(self):
|
||||
try:
|
||||
from tests.test_execution_context_schema import legacy
|
||||
except ImportError:
|
||||
from test_execution_context_schema import legacy
|
||||
from database.execution_history_schema import (
|
||||
history_schema_error,
|
||||
migrate_execution_history,
|
||||
)
|
||||
|
||||
with closing(sqlite3.connect(":memory:")) as conn:
|
||||
legacy(conn)
|
||||
conn.execute(
|
||||
"UPDATE execution_runs SET node_id=?,node_type=?,exception_type=?",
|
||||
("😀" * 128, "😀" * 128, "😀" * 256),
|
||||
)
|
||||
self.assertIsNone(history_schema_error(conn))
|
||||
migrate_execution_history(conn)
|
||||
self.assertIsNone(history_schema_error(conn, allow_missing=False))
|
||||
self.assertEqual(
|
||||
conn.execute(
|
||||
"SELECT node_id,node_type,exception_type FROM " "execution_runs"
|
||||
).fetchone(),
|
||||
("😀" * 128, "😀" * 128, "😀" * 256),
|
||||
)
|
||||
|
||||
def test_committed_wal_backup_includes_history_and_epoch(self):
|
||||
db = self.database()
|
||||
conn = db.model.get_connection()
|
||||
@@ -87,14 +172,14 @@ class HistoryRestoreTests(unittest.TestCase):
|
||||
source = str(Path(self.temp.name) / "old.db")
|
||||
with closing(sqlite3.connect(source)) as conn, conn:
|
||||
conn.execute(
|
||||
"CREATE TABLE prompts (id INTEGER PRIMARY KEY AUTOINCREMENT,"
|
||||
" text TEXT NOT NULL, created_at TIMESTAMP, updated_at TIMESTAMP,"
|
||||
" category TEXT, tags TEXT, rating INTEGER, notes TEXT,"
|
||||
" hash TEXT UNIQUE, workflow_name TEXT)"
|
||||
"CREATE TABLE prompts (id INTEGER PRIMARY KEY AUTOINCREMENT, text "
|
||||
"TEXT NOT NULL, created_at TIMESTAMP, updated_at TIMESTAMP, "
|
||||
"category TEXT, tags TEXT, rating INTEGER, notes TEXT, hash TEXT "
|
||||
"UNIQUE, workflow_name TEXT)"
|
||||
)
|
||||
conn.execute(
|
||||
"INSERT INTO prompts(id,text,created_at) "
|
||||
"VALUES(7,'old prompt','2020-01-01')"
|
||||
"INSERT INTO prompts(id,text,created_at) VALUES(7,'old "
|
||||
"prompt','2020-01-01')"
|
||||
)
|
||||
original = Path(source).read_bytes()
|
||||
self.assertEqual(PromptModel.verify_database_file(source), (True, "ok"))
|
||||
@@ -113,6 +198,113 @@ class HistoryRestoreTests(unittest.TestCase):
|
||||
(7, "old prompt"),
|
||||
)
|
||||
|
||||
def test_wal_and_old_backup_staging_leave_source_unchanged(self):
|
||||
from database.execution_context_schema import TABLE_COLUMNS
|
||||
|
||||
source = str(Path(self.temp.name) / "v1.db")
|
||||
old = self.database(source)
|
||||
start = self.start(old)
|
||||
old.write_execution_start(start)
|
||||
with old.model.get_connection() as conn:
|
||||
conn.execute("INSERT INTO prompts(id,text,run_count) VALUES(7,'old',19)")
|
||||
for table in reversed(TABLE_COLUMNS):
|
||||
conn.execute(f"DROP TABLE {table}")
|
||||
sql = (
|
||||
conn.execute(
|
||||
"SELECT sql FROM sqlite_master WHERE "
|
||||
"name='execution_history_metadata'"
|
||||
)
|
||||
.fetchone()[0]
|
||||
.replace("schema_version = 2", "schema_version = 1")
|
||||
)
|
||||
conn.execute("DROP TABLE execution_history_metadata")
|
||||
conn.execute(sql)
|
||||
conn.execute(
|
||||
"INSERT INTO execution_history_metadata VALUES(1,1,?)",
|
||||
(start.storage_epoch,),
|
||||
)
|
||||
conn.execute("PRAGMA wal_autocheckpoint=0")
|
||||
with conn:
|
||||
conn.execute("UPDATE prompts SET run_count=31 WHERE id=7")
|
||||
source_bytes = Path(source).read_bytes()
|
||||
wal_bytes = Path(source + "-wal").read_bytes()
|
||||
self.assertEqual(PromptModel.verify_database_file(source), (True, "ok"))
|
||||
live = self.database()
|
||||
live.model.restore_from_file(source)
|
||||
self.assertEqual(Path(source).read_bytes(), source_bytes)
|
||||
self.assertEqual(Path(source + "-wal").read_bytes(), wal_bytes)
|
||||
self.assertEqual(live.get_execution_run(start.run_id)["record_version"], 1)
|
||||
self.assertEqual(
|
||||
live.model.get_connection()
|
||||
.execute("SELECT run_count FROM prompts WHERE id=7")
|
||||
.fetchone()[0],
|
||||
31,
|
||||
)
|
||||
self.assertEqual(
|
||||
live.model.get_connection()
|
||||
.execute("SELECT COUNT(*) FROM execution_run_contexts")
|
||||
.fetchone()[0],
|
||||
0,
|
||||
)
|
||||
self.assertEqual(
|
||||
conn.execute(
|
||||
"SELECT schema_version FROM execution_history_metadata"
|
||||
).fetchone()[0],
|
||||
1,
|
||||
)
|
||||
|
||||
def test_malformed_v2_context_restore_keeps_live_database_unchanged(self):
|
||||
live = self.database()
|
||||
start = self.start(live)
|
||||
live.write_execution_start(start)
|
||||
with live.model.get_connection() as conn:
|
||||
conn.execute("INSERT INTO prompts(text,run_count) VALUES('kept',19)")
|
||||
before = "\n".join(live.model.get_connection().iterdump())
|
||||
for mutation in (
|
||||
"DROP TABLE execution_context_capture",
|
||||
"CREATE UNIQUE INDEX bad_context_unique ON "
|
||||
"execution_context_texts(text_value)",
|
||||
"INSERT INTO execution_context_texts VALUES('"
|
||||
+ "0" * 64
|
||||
+ "','wrong fingerprint')",
|
||||
):
|
||||
with self.subTest(mutation=mutation):
|
||||
source = str(Path(self.temp.name) / "bad-context.db")
|
||||
self.assertTrue(live.model.backup_database(source))
|
||||
with closing(sqlite3.connect(source)) as conn, conn:
|
||||
conn.execute(mutation)
|
||||
original = Path(source).read_bytes()
|
||||
with self.assertRaises(ValueError):
|
||||
live.model.restore_from_file(source)
|
||||
self.assertEqual(
|
||||
"\n".join(live.model.get_connection().iterdump()), before
|
||||
)
|
||||
self.assertEqual(Path(source).read_bytes(), original)
|
||||
|
||||
def test_missing_capture_header_rejects_backup_verification_and_restore(self):
|
||||
try:
|
||||
from tests.test_execution_context_database import bundle
|
||||
except ImportError:
|
||||
from test_execution_context_database import bundle
|
||||
|
||||
live = self.database()
|
||||
start = self.start(live)
|
||||
live.write_execution_start(start)
|
||||
live.write_execution_contexts(start.run_id, start.storage_epoch, bundle())
|
||||
self.assertEqual(live.get_execution_contexts(start.run_id)["state"], "complete")
|
||||
before = "\n".join(live.model.get_connection().iterdump())
|
||||
source = str(Path(self.temp.name) / "missing-capture.db")
|
||||
self.assertTrue(live.model.backup_database(source))
|
||||
with closing(sqlite3.connect(source)) as conn, conn:
|
||||
conn.execute("DELETE FROM execution_context_capture")
|
||||
original = Path(source).read_bytes()
|
||||
valid, _ = PromptModel.verify_database_file(source)
|
||||
self.assertFalse(valid)
|
||||
with self.assertRaises(ValueError):
|
||||
live.model.restore_from_file(source)
|
||||
self.assertEqual("\n".join(live.model.get_connection().iterdump()), before)
|
||||
self.assertEqual(Path(source).read_bytes(), original)
|
||||
|
||||
def test_failed_staging_rotation_keeps_live_epoch_and_never_notifies(self):
|
||||
from database.execution_history import subscribe_history_restore
|
||||
|
||||
@@ -189,17 +381,17 @@ class HistoryRestoreTests(unittest.TestCase):
|
||||
)
|
||||
conn.execute("DROP TABLE old_runs")
|
||||
conn.execute(
|
||||
"CREATE UNIQUE INDEX idx_execution_runs_identity "
|
||||
"ON execution_runs(session_id, queue_item_id)"
|
||||
"CREATE UNIQUE INDEX idx_execution_runs_identity ON "
|
||||
"execution_runs(session_id, queue_item_id)"
|
||||
)
|
||||
conn.execute(
|
||||
"CREATE INDEX idx_execution_runs_observed "
|
||||
"ON execution_runs(observed_dequeue_at, run_id)"
|
||||
"CREATE INDEX idx_execution_runs_observed ON "
|
||||
"execution_runs(observed_dequeue_at, run_id)"
|
||||
)
|
||||
else:
|
||||
conn.execute(
|
||||
"CREATE UNIQUE INDEX unexpected_job_uniqueness "
|
||||
"ON execution_runs(job_id)"
|
||||
"CREATE UNIQUE INDEX unexpected_job_uniqueness ON "
|
||||
"execution_runs(job_id)"
|
||||
)
|
||||
original = Path(source).read_bytes()
|
||||
with self.assertRaisesRegex(ValueError, "uniqu"):
|
||||
@@ -241,6 +433,51 @@ class HistoryRestoreTests(unittest.TestCase):
|
||||
self.assertNotEqual(db.get_execution_epoch(), original_epoch)
|
||||
self.assertEqual(notices, [True])
|
||||
|
||||
def test_restore_preserves_context_scalars_but_never_reepochs_hints(self):
|
||||
try:
|
||||
from tests.test_execution_context_database import bundle
|
||||
except ImportError:
|
||||
from test_execution_context_database import bundle
|
||||
from database.execution_history import HistoryEpochChanged
|
||||
|
||||
db = self.database()
|
||||
with db.model.get_connection() as conn:
|
||||
conn.execute(
|
||||
"INSERT INTO prompts(id,text) VALUES(1,'positive'),(2,'negative')"
|
||||
)
|
||||
conn.execute(
|
||||
"INSERT INTO prompt_pairs(id,positive_prompt_id,negative_prompt_id) "
|
||||
"VALUES(7,1,2)"
|
||||
)
|
||||
start = self.start(db)
|
||||
db.write_execution_start(start)
|
||||
db.write_execution_contexts(start.run_id, start.storage_epoch, bundle())
|
||||
original = db.get_execution_contexts(start.run_id)
|
||||
backup = str(Path(self.temp.name) / "contexts.db")
|
||||
self.assertTrue(db.model.backup_database(backup))
|
||||
source_bytes = Path(backup).read_bytes()
|
||||
for _ in range(2):
|
||||
db.model.restore_from_file(backup)
|
||||
read = db.get_execution_contexts(start.run_id)
|
||||
expected = original["contexts"][0].copy()
|
||||
expected.update(
|
||||
positive_prompt_id=None, negative_prompt_id=None, pair_id=None
|
||||
)
|
||||
self.assertEqual(read["contexts"], [expected])
|
||||
self.assertEqual(read["state"], "complete")
|
||||
self.assertEqual(
|
||||
read["contexts"][0]["reference_epoch"], start.storage_epoch
|
||||
)
|
||||
with self.assertRaises(HistoryEpochChanged):
|
||||
db.write_execution_contexts(start.run_id, start.storage_epoch, bundle())
|
||||
self.assertFalse(
|
||||
db.write_execution_contexts(
|
||||
start.run_id, db.get_execution_epoch(), bundle()
|
||||
)
|
||||
)
|
||||
self.assertEqual(db.get_execution_contexts(start.run_id), read)
|
||||
self.assertEqual(Path(backup).read_bytes(), source_bytes)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
"""Real-thread, disposable-database coverage for the bounded history writer."""
|
||||
|
||||
from dataclasses import FrozenInstanceError, replace
|
||||
from contextlib import nullcontext
|
||||
import logging
|
||||
import os
|
||||
import sqlite3
|
||||
@@ -105,6 +106,491 @@ class HistoryWriterTests(unittest.TestCase):
|
||||
self.releases.append(release)
|
||||
return entered, release
|
||||
|
||||
def sealed(self, writer, value="exact text", generation=None):
|
||||
from utils.execution_context_budget import SealedContext
|
||||
from utils.execution_contexts import (
|
||||
ContextBundle,
|
||||
RecipeContext,
|
||||
PromptSnapshot,
|
||||
TextEvidence,
|
||||
bundle_payload_size,
|
||||
)
|
||||
|
||||
field = TextEvidence(value, "recorded", "runtime_return", None)
|
||||
snapshot = PromptSnapshot(field, field, field, field)
|
||||
bundle = ContextBundle(
|
||||
(RecipeContext("1", snapshot, snapshot, "inferred"),),
|
||||
"complete",
|
||||
"captured",
|
||||
)
|
||||
lease = writer.context_budget.reserve(bundle_payload_size(bundle))
|
||||
return SealedContext(
|
||||
bundle,
|
||||
lease,
|
||||
"complete",
|
||||
"captured",
|
||||
writer.status().capture_generation if generation is None else generation,
|
||||
)
|
||||
|
||||
def test_context_success_releases_and_legacy_finish_preserves_capture(self):
|
||||
writer = self.writer()
|
||||
start = self.start_command()
|
||||
sealed = self.sealed(writer)
|
||||
self.assertTrue(writer.submit(start))
|
||||
self.assertTrue(writer.submit_finalization(self.finish_command(start), sealed))
|
||||
self.wait_for(lambda: writer.status().pending == 0)
|
||||
self.assertEqual(writer.context_budget.charged_bytes, 0)
|
||||
self.assertEqual(
|
||||
self.db.get_execution_contexts(start.run_id)["state"], "complete"
|
||||
)
|
||||
self.assertTrue(writer.submit(self.finish_command(start)))
|
||||
self.wait_for(lambda: writer.status().pending == 0)
|
||||
self.assertEqual(
|
||||
self.db.get_execution_contexts(start.run_id)["state"], "complete"
|
||||
)
|
||||
|
||||
def test_context_saturation_keeps_lifecycle_and_status(self):
|
||||
from utils.execution_context_budget import SealedContext
|
||||
from utils.execution_history_events import encode_command
|
||||
|
||||
writer = self.writer()
|
||||
lease = writer.context_budget.reserve(8388608)
|
||||
self.assertIsNone(writer.context_budget.reserve(1))
|
||||
start = self.start_command()
|
||||
self.assertTrue(writer.submit(start))
|
||||
finish = self.finish_command(start)
|
||||
self.assertLessEqual(len(encode_command(finish)), 2048)
|
||||
self.assertTrue(
|
||||
writer.submit_finalization(
|
||||
finish, SealedContext(None, None, "unavailable", "budget_exhausted")
|
||||
)
|
||||
)
|
||||
self.wait_for(lambda: writer.status().pending == 0)
|
||||
self.assertEqual(
|
||||
self.db.get_execution_run(start.run_id)["outcome"], "succeeded"
|
||||
)
|
||||
self.assertEqual(
|
||||
self.db.get_execution_contexts(start.run_id)["reason"], "budget_exhausted"
|
||||
)
|
||||
lease.release()
|
||||
self.assertEqual(writer.context_budget.charged_bytes, 0)
|
||||
|
||||
def test_outcome_survives_context_transaction_failure(self):
|
||||
for error, ready in (
|
||||
(ValueError("invalid evidence"), True),
|
||||
(sqlite3.OperationalError("unavailable"), False),
|
||||
):
|
||||
with self.subTest(error=type(error)):
|
||||
writer = self.writer()
|
||||
start = self.start_command(10 if ready else 11)
|
||||
self.assertTrue(writer.submit(start))
|
||||
sealed = self.sealed(writer)
|
||||
entered, release = self.block()
|
||||
|
||||
def fail_context(*args, **kwargs):
|
||||
entered.set()
|
||||
release.wait(3)
|
||||
raise error
|
||||
|
||||
with patch.object(
|
||||
self.db, "write_execution_contexts_for_finish", fail_context
|
||||
):
|
||||
self.assertTrue(
|
||||
writer.submit_finalization(self.finish_command(start), sealed)
|
||||
)
|
||||
self.assertTrue(entered.wait(1))
|
||||
self.assertEqual(
|
||||
self.db.get_execution_run(start.run_id)["outcome"], "succeeded"
|
||||
)
|
||||
self.assertGreater(writer.context_budget.charged_bytes, 0)
|
||||
release.set()
|
||||
self.wait_for(lambda: writer.status().pending == 0)
|
||||
self.assertEqual(
|
||||
self.db.get_execution_run(start.run_id)["outcome"], "succeeded"
|
||||
)
|
||||
self.assertEqual(writer.status().accepting, ready)
|
||||
self.assertEqual(writer.context_budget.charged_bytes, 0)
|
||||
if not ready:
|
||||
self.clock.advance(5)
|
||||
self.wait_for(lambda: writer.status().accepting)
|
||||
self.assertEqual(
|
||||
self.db.get_execution_contexts(start.run_id)["state"],
|
||||
"not_captured",
|
||||
)
|
||||
self.assertTrue(writer.close())
|
||||
|
||||
def test_same_epoch_recovery_changes_capture_generation(self):
|
||||
writer = self.writer()
|
||||
start = self.start_command()
|
||||
old_generation = writer.status().capture_generation
|
||||
sealed = self.sealed(writer)
|
||||
writer.invalidate()
|
||||
self.clock.advance(5)
|
||||
self.wait_for(lambda: writer.status().accepting)
|
||||
self.assertEqual(writer.status().epoch, start.storage_epoch)
|
||||
self.assertGreater(writer.status().capture_generation, old_generation)
|
||||
self.assertTrue(writer.submit(start))
|
||||
self.assertTrue(writer.submit_finalization(self.finish_command(start), sealed))
|
||||
self.wait_for(lambda: writer.status().pending == 0)
|
||||
self.assertEqual(
|
||||
self.db.get_execution_run(start.run_id)["outcome"], "succeeded"
|
||||
)
|
||||
self.assertEqual(writer.context_budget.charged_bytes, 0)
|
||||
self.assertEqual(self.db.get_execution_contexts(start.run_id)["contexts"], [])
|
||||
|
||||
def test_restore_between_transactions_rejects_context(self):
|
||||
writer = self.writer()
|
||||
start = self.start_command()
|
||||
self.db.write_execution_start(start)
|
||||
backup = os.path.join(self.directory.name, "between.db")
|
||||
self.assertTrue(self.db.model.backup_database(backup))
|
||||
original = self.db.write_execution_finish
|
||||
|
||||
def finish(command):
|
||||
result = original(command)
|
||||
self.db.model.restore_from_file(backup)
|
||||
return result
|
||||
|
||||
with patch.object(self.db, "write_execution_finish", finish):
|
||||
self.assertTrue(
|
||||
writer.submit_finalization(
|
||||
self.finish_command(start), self.sealed(writer)
|
||||
)
|
||||
)
|
||||
self.wait_for(lambda: writer.status().pending == 0)
|
||||
self.assertEqual(
|
||||
self.db.get_execution_contexts(start.run_id)["state"], "not_captured"
|
||||
)
|
||||
self.assertEqual(writer.context_budget.charged_bytes, 0)
|
||||
self.assertFalse(writer.status().accepting)
|
||||
|
||||
def test_reject_discard_and_bounded_close_release_leases(self):
|
||||
writer = self.writer()
|
||||
start = self.start_command()
|
||||
sealed = self.sealed(writer)
|
||||
self.assertFalse(writer.submit_finalization(object(), sealed))
|
||||
self.assertEqual(writer.context_budget.charged_bytes, 0)
|
||||
entered, release = self.block()
|
||||
original = self.db.write_execution_start
|
||||
|
||||
def write(command):
|
||||
entered.set()
|
||||
release.wait(3)
|
||||
return original(command)
|
||||
|
||||
with patch.object(self.db, "write_execution_start", write):
|
||||
self.assertTrue(writer.submit(start))
|
||||
self.assertTrue(entered.wait(1))
|
||||
for _ in range(1023):
|
||||
self.assertTrue(
|
||||
writer.submit_finalization(
|
||||
self.finish_command(start), self.sealed(writer)
|
||||
)
|
||||
)
|
||||
self.assertEqual(writer.status().pending, 1024)
|
||||
self.assertFalse(
|
||||
writer.submit_finalization(
|
||||
self.finish_command(start), self.sealed(writer)
|
||||
)
|
||||
)
|
||||
self.assertFalse(writer.close(0.01))
|
||||
release.set()
|
||||
self.wait_for(lambda: not writer.status().alive)
|
||||
self.assertEqual(writer.context_budget.charged_bytes, 0)
|
||||
self.assertEqual(
|
||||
self.db.get_execution_contexts(start.run_id)["state"], "not_captured"
|
||||
)
|
||||
|
||||
def test_bounded_close_keeps_inflight_charge_and_releases_queued(self):
|
||||
self.check_bounded_close_payloads()
|
||||
|
||||
def test_early_join_return_still_waits_for_deadline_and_releases_queued(self):
|
||||
self.check_bounded_close_payloads(early_join=True)
|
||||
|
||||
def check_bounded_close_payloads(self, early_join=False):
|
||||
from utils.execution_context_budget import ContextBudget
|
||||
|
||||
pool = ContextBudget(1000)
|
||||
writer = HistoryWriter(lambda: self.db, clock=self.clock, context_budget=pool)
|
||||
self.writers.append(writer)
|
||||
writer.start()
|
||||
self.wait_for(lambda: writer.status().accepting)
|
||||
start = self.start_command()
|
||||
self.db.write_execution_start(start)
|
||||
entered, release = self.block()
|
||||
original = self.db.write_execution_contexts_for_finish
|
||||
first = self.sealed(writer)
|
||||
charge = first.lease.byte_count
|
||||
|
||||
def write(*args, **kwargs):
|
||||
entered.set()
|
||||
release.wait(3)
|
||||
return original(*args, **kwargs)
|
||||
|
||||
with patch.object(self.db, "write_execution_contexts_for_finish", write):
|
||||
self.assertTrue(
|
||||
writer.submit_finalization(self.finish_command(start), first)
|
||||
)
|
||||
self.assertTrue(entered.wait(1))
|
||||
self.assertTrue(
|
||||
writer.submit_finalization(
|
||||
self.finish_command(start), self.sealed(writer)
|
||||
)
|
||||
)
|
||||
native_join = writer._thread.join
|
||||
close_clock = ManualClock()
|
||||
|
||||
def return_early(timeout):
|
||||
# Keep the real worker blocked while simulating an early native join.
|
||||
self.assertTrue(writer.status().alive)
|
||||
self.assertFalse(writer.submit(self.finish_command(start)))
|
||||
native_join(0)
|
||||
close_clock.advance(min(timeout, 0.004))
|
||||
|
||||
def yield_wait(delay):
|
||||
self.assertGreater(delay, 0)
|
||||
self.assertLessEqual(delay, 0.001)
|
||||
close_clock.advance(delay)
|
||||
|
||||
clock = (
|
||||
patch(
|
||||
"utils.execution_history_writer.time",
|
||||
SimpleNamespace(monotonic=close_clock, sleep=yield_wait),
|
||||
)
|
||||
if early_join
|
||||
else nullcontext()
|
||||
)
|
||||
join = (
|
||||
patch.object(writer._thread, "join", return_early)
|
||||
if early_join
|
||||
else nullcontext()
|
||||
)
|
||||
with clock, join:
|
||||
monotonic = close_clock if early_join else time.monotonic
|
||||
before = monotonic()
|
||||
self.assertFalse(writer.close(0.02))
|
||||
elapsed = monotonic() - before
|
||||
self.assertGreaterEqual(elapsed, 0.02)
|
||||
self.assertLess(elapsed, 0.2)
|
||||
self.assertEqual(pool.charged_bytes, charge)
|
||||
replacement = HistoryWriter(lambda: self.db, context_budget=pool)
|
||||
self.writers.append(replacement)
|
||||
self.assertIsNone(replacement.context_budget.reserve(1000))
|
||||
release.set()
|
||||
self.wait_for(lambda: not writer.status().alive)
|
||||
self.assertEqual(pool.charged_bytes, 0)
|
||||
|
||||
def test_close_drain_exits_without_sleeping_after_worker_join(self):
|
||||
writer = self.writer()
|
||||
clock = ManualClock()
|
||||
native_join = writer._thread.join
|
||||
|
||||
def join(timeout):
|
||||
native_join(3)
|
||||
self.assertFalse(writer._thread.is_alive())
|
||||
clock.advance(0.001)
|
||||
|
||||
def sleep(delay):
|
||||
self.fail("Close slept after the worker exited")
|
||||
|
||||
with (
|
||||
patch(
|
||||
"utils.execution_history_writer.time",
|
||||
SimpleNamespace(monotonic=clock, sleep=sleep),
|
||||
),
|
||||
patch.object(writer._thread, "join", join),
|
||||
):
|
||||
self.assertTrue(writer.close(0.02))
|
||||
self.assertLess(clock(), 0.02)
|
||||
self.assertFalse(writer.status().alive)
|
||||
|
||||
def test_same_epoch_invalidation_between_transactions_drops_payload(self):
|
||||
writer = self.writer()
|
||||
start = self.start_command()
|
||||
self.db.write_execution_start(start)
|
||||
original = self.db.write_execution_finish
|
||||
|
||||
def finish(command):
|
||||
result = original(command)
|
||||
writer.invalidate()
|
||||
return result
|
||||
|
||||
with patch.object(self.db, "write_execution_finish", finish):
|
||||
self.assertTrue(
|
||||
writer.submit_finalization(
|
||||
self.finish_command(start), self.sealed(writer)
|
||||
)
|
||||
)
|
||||
self.wait_for(lambda: writer.status().pending == 0)
|
||||
self.assertEqual(
|
||||
self.db.get_execution_run(start.run_id)["outcome"], "succeeded"
|
||||
)
|
||||
self.assertEqual(
|
||||
self.db.get_execution_contexts(start.run_id)["state"], "not_captured"
|
||||
)
|
||||
self.assertEqual(writer.context_budget.charged_bytes, 0)
|
||||
self.clock.advance(5)
|
||||
self.wait_for(lambda: writer.status().accepting)
|
||||
self.assertEqual(
|
||||
self.db.get_execution_contexts(start.run_id)["state"], "not_captured"
|
||||
)
|
||||
|
||||
def test_invalid_seals_release_and_keep_valid_outcome(self):
|
||||
from utils.execution_context_budget import ContextBudget
|
||||
|
||||
writer = self.writer()
|
||||
cases = [
|
||||
dict(state="private text" * 100),
|
||||
dict(reason="arbitrary_exception"),
|
||||
dict(capture_generation=True),
|
||||
dict(capture_generation=-1),
|
||||
dict(capture_generation=None),
|
||||
dict(lease=None),
|
||||
]
|
||||
for item, changes in enumerate(cases, 20):
|
||||
start = self.start_command(item)
|
||||
self.db.write_execution_start(start)
|
||||
sealed = self.sealed(writer)
|
||||
malformed = replace(sealed, **changes)
|
||||
if changes == {"lease": None}:
|
||||
sealed.lease.release()
|
||||
self.assertTrue(
|
||||
writer.submit_finalization(self.finish_command(start), malformed)
|
||||
)
|
||||
self.wait_for(lambda: writer.status().pending == 0)
|
||||
self.assertEqual(
|
||||
self.db.get_execution_run(start.run_id)["outcome"], "succeeded"
|
||||
)
|
||||
self.assertEqual(
|
||||
self.db.get_execution_contexts(start.run_id)["reason"],
|
||||
"invalid_evidence",
|
||||
)
|
||||
self.assertEqual(writer.context_budget.charged_bytes, 0)
|
||||
foreign = ContextBudget(1000)
|
||||
sealed = self.sealed(writer)
|
||||
sealed.lease.release()
|
||||
sealed = replace(sealed, lease=foreign.reserve(1000))
|
||||
start = self.start_command(30)
|
||||
self.db.write_execution_start(start)
|
||||
self.assertTrue(writer.submit_finalization(self.finish_command(start), sealed))
|
||||
self.wait_for(lambda: writer.status().pending == 0)
|
||||
self.assertEqual(foreign.charged_bytes, 0)
|
||||
self.assertEqual(self.db.get_execution_contexts(start.run_id)["contexts"], [])
|
||||
|
||||
def test_custom_run_identity_receives_finalized_context(self):
|
||||
writer = self.writer()
|
||||
start = replace(self.start_command(), run_id=str(uuid4()))
|
||||
self.assertTrue(writer.submit(start))
|
||||
self.assertTrue(
|
||||
writer.submit_finalization(self.finish_command(start), self.sealed(writer))
|
||||
)
|
||||
self.wait_for(lambda: writer.status().pending == 0)
|
||||
self.assertEqual(
|
||||
self.db.get_execution_run(start.run_id)["outcome"], "succeeded"
|
||||
)
|
||||
self.assertEqual(
|
||||
self.db.get_execution_contexts(start.run_id)["state"], "complete"
|
||||
)
|
||||
self.assertEqual(writer.context_budget.charged_bytes, 0)
|
||||
|
||||
def test_invalidation_during_context_preparation_fences_actual_transaction(self):
|
||||
import database.execution_contexts as contexts_db
|
||||
|
||||
writer = self.writer()
|
||||
start = self.start_command()
|
||||
self.db.write_execution_start(start)
|
||||
entered, release = self.block()
|
||||
original = contexts_db._prepare
|
||||
|
||||
def prepare(bundle):
|
||||
prepared = original(bundle)
|
||||
entered.set()
|
||||
release.wait(3)
|
||||
return prepared
|
||||
|
||||
with patch.object(contexts_db, "_prepare", prepare):
|
||||
self.assertTrue(
|
||||
writer.submit_finalization(
|
||||
self.finish_command(start), self.sealed(writer)
|
||||
)
|
||||
)
|
||||
self.assertTrue(entered.wait(1))
|
||||
self.assertEqual(
|
||||
self.db.get_execution_run(start.run_id)["outcome"], "succeeded"
|
||||
)
|
||||
generation = writer.status().capture_generation
|
||||
writer.invalidate()
|
||||
self.assertGreater(writer.status().capture_generation, generation)
|
||||
release.set()
|
||||
self.wait_for(lambda: writer.status().pending == 0)
|
||||
self.assertEqual(
|
||||
self.db.get_execution_contexts(start.run_id)["state"], "not_captured"
|
||||
)
|
||||
self.assertEqual(writer.context_budget.charged_bytes, 0)
|
||||
self.clock.advance(5)
|
||||
self.wait_for(lambda: writer.status().accepting)
|
||||
self.assertEqual(writer.status().epoch, start.storage_epoch)
|
||||
self.assertEqual(
|
||||
self.db.get_execution_contexts(start.run_id)["state"], "not_captured"
|
||||
)
|
||||
|
||||
def test_context_commit_serializes_invalidation_without_blocking_queue_or_close(
|
||||
self,
|
||||
):
|
||||
import database.execution_contexts as contexts_db
|
||||
|
||||
writer = self.writer()
|
||||
start = self.start_command()
|
||||
self.db.write_execution_start(start)
|
||||
entered, release = self.block()
|
||||
invalidating, invalidated = threading.Event(), threading.Event()
|
||||
original = contexts_db._write_capture
|
||||
|
||||
def capture(*args, **kwargs):
|
||||
entered.set()
|
||||
release.wait(3)
|
||||
return original(*args, **kwargs)
|
||||
|
||||
def invalidate():
|
||||
invalidating.set()
|
||||
writer.invalidate()
|
||||
invalidated.set()
|
||||
|
||||
with patch.object(contexts_db, "_write_capture", capture):
|
||||
self.assertTrue(
|
||||
writer.submit_finalization(
|
||||
self.finish_command(start), self.sealed(writer)
|
||||
)
|
||||
)
|
||||
self.assertTrue(entered.wait(1))
|
||||
invalidator = threading.Thread(target=invalidate)
|
||||
invalidator.start()
|
||||
try:
|
||||
self.assertTrue(invalidating.wait(1))
|
||||
self.assertFalse(invalidated.wait(0.03))
|
||||
before = time.monotonic()
|
||||
self.assertTrue(writer.status().accepting)
|
||||
self.assertTrue(writer.submit(self.finish_command(start)))
|
||||
self.assertLess(time.monotonic() - before, 0.1)
|
||||
before = time.monotonic()
|
||||
self.assertFalse(writer.close(0))
|
||||
self.assertLess(time.monotonic() - before, 0.1)
|
||||
self.assertGreater(writer.context_budget.charged_bytes, 0)
|
||||
self.assertEqual(writer.status().pending, 1)
|
||||
finally:
|
||||
release.set()
|
||||
invalidator.join(3)
|
||||
self.assertFalse(invalidator.is_alive())
|
||||
self.assertTrue(invalidated.is_set())
|
||||
self.wait_for(lambda: not writer.status().alive)
|
||||
self.assertEqual(
|
||||
self.db.get_execution_run(start.run_id)["outcome"], "succeeded"
|
||||
)
|
||||
self.assertEqual(
|
||||
self.db.get_execution_contexts(start.run_id)["state"], "complete"
|
||||
)
|
||||
self.assertEqual(writer.context_budget.charged_bytes, 0)
|
||||
|
||||
def test_fifo_single_worker_and_thread_local_connection_cleanup(self):
|
||||
caller_conn = self.db.model.get_connection()
|
||||
threads, timeouts = [], []
|
||||
@@ -481,6 +967,29 @@ class HistoryWriterTests(unittest.TestCase):
|
||||
self.assertEqual(writer.status().recording_gaps, 2**63 - 1)
|
||||
self.assertTrue(writer.close())
|
||||
|
||||
def test_owner_invalidation_callback_runs_outside_both_locks_and_is_contained(self):
|
||||
observations = []
|
||||
writer = None
|
||||
|
||||
def callback():
|
||||
acquired = [
|
||||
lock.acquire(blocking=False)
|
||||
for lock in (writer._lock, writer._context_commit_gate)
|
||||
]
|
||||
observations.append(tuple(acquired))
|
||||
for lock, owned in zip(
|
||||
(writer._lock, writer._context_commit_gate), acquired
|
||||
):
|
||||
if owned:
|
||||
lock.release()
|
||||
raise RuntimeError("callback failure cannot escape")
|
||||
|
||||
writer = HistoryWriter(lambda: self.db, on_invalidation=callback)
|
||||
self.writers.append(writer)
|
||||
writer.invalidate()
|
||||
writer.close(0)
|
||||
self.assertEqual(observations, [(True, True), (True, True)])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
@@ -326,3 +326,23 @@ class TestPairExecution(unittest.TestCase):
|
||||
record_pair_execution(extra, "1", **kwargs)
|
||||
with self.assertRaises(ExecutionEvidenceError):
|
||||
read_pair_execution(extra, "1")
|
||||
|
||||
def test_png_evidence_cannot_populate_unobserved_attempt(self):
|
||||
from utils.execution_context_capture import ContextCapture
|
||||
from utils.execution_context_budget import ContextBudget
|
||||
from utils.execution_history_events import RunStart
|
||||
from utils.execution_history_writer import HistoryWriterStatus
|
||||
|
||||
identity = "11111111-1111-4111-8111-111111111111"
|
||||
self.execute()
|
||||
state = HistoryWriterStatus(identity, True, 0, 0, True, 1)
|
||||
manager = ContextCapture(1, lambda: state, ContextBudget())
|
||||
self.addCleanup(manager.revoke_all)
|
||||
cap = manager.begin(
|
||||
RunStart(identity, identity, 1, identity, identity, 0), 1, graph(), None
|
||||
)
|
||||
sealed = manager.seal(cap)
|
||||
if sealed.lease:
|
||||
self.addCleanup(sealed.lease.release)
|
||||
self.assertIsNone(sealed.bundle.contexts[0].positive.effective.value)
|
||||
self.assertIsNone(sealed.bundle.contexts[0].negative.effective.value)
|
||||
|
||||
@@ -158,3 +158,45 @@ class TestPromptPairNode(unittest.TestCase):
|
||||
)
|
||||
result = self.node.encode_prompt(self.clip, "cat", negative_text="blur")
|
||||
self.assertEqual(result[2:], ("cat LORA", "blur"))
|
||||
|
||||
def test_failed_negative_encoding_cannot_publish_positive_runtime_proof(self):
|
||||
from contextlib import ExitStack
|
||||
import sys
|
||||
import types
|
||||
from utils.execution_context_capture import ContextCapture
|
||||
from utils.execution_context_budget import ContextBudget
|
||||
from utils.execution_history_events import RunStart
|
||||
from utils.execution_history_writer import HistoryWriterStatus
|
||||
from unittest.mock import patch
|
||||
|
||||
identity = "11111111-1111-4111-8111-111111111111"
|
||||
state = HistoryWriterStatus(identity, True, 0, 0, True, 1)
|
||||
budget = ContextBudget()
|
||||
manager = ContextCapture(1, lambda: state, budget)
|
||||
self.addCleanup(manager.revoke_all)
|
||||
cap = manager.begin(
|
||||
RunStart(identity, identity, 1, identity, identity, 0),
|
||||
1,
|
||||
pair_graph(),
|
||||
None,
|
||||
)
|
||||
host = types.ModuleType("comfy_execution.utils")
|
||||
host.get_executing_context = lambda: types.SimpleNamespace(
|
||||
prompt_id=identity, node_id="1", list_index=0
|
||||
)
|
||||
self.clip.encode_from_tokens_scheduled.side_effect = [
|
||||
("conditioning", "cat LORA"),
|
||||
RuntimeError("negative failed"),
|
||||
]
|
||||
with ExitStack() as stack:
|
||||
stack.enter_context(
|
||||
patch.dict(sys.modules, {"comfy_execution.utils": host})
|
||||
)
|
||||
with self.assertRaisesRegex(RuntimeError, "negative failed"):
|
||||
self.node.encode_prompt(self.clip, "cat", negative_text="blur")
|
||||
sealed = manager.seal(cap)
|
||||
if sealed.lease:
|
||||
self.addCleanup(sealed.lease.release)
|
||||
self.assertIsNone(sealed.bundle.contexts[0].positive.effective.value)
|
||||
self.assertIsNone(sealed.bundle.contexts[0].negative.effective.value)
|
||||
self.assertEqual(self.db.list_prompt_pairs()["total"], 0)
|
||||
|
||||
Vendored
+4
-1
@@ -27,7 +27,10 @@ test('execution script discovers only the new matrix and preserves all 78 stock
|
||||
const executions = list('tools/vendor/playwright.execution.config.mjs');
|
||||
assert.match(executions, /\[classic-start\]/);
|
||||
assert.match(executions, /\[nodes2-start\]/);
|
||||
assert.match(executions, /Total: 10 tests in 1 file/);
|
||||
assert.match(executions, /Total: 26 tests in 1 file/);
|
||||
for (const project of ['classic-start', 'nodes2-start']) {
|
||||
assert.equal(executions.split('\n').filter(line => line.includes(`[${project}]`)).length, 13);
|
||||
}
|
||||
assert.doesNotMatch(executions, /prompt-manager.spec/);
|
||||
});
|
||||
|
||||
|
||||
Vendored
+102
-12
@@ -91,23 +91,113 @@ test('read-only ledger helper distinguishes absent schema from missing, corrupt
|
||||
const session = '20000000-0000-4000-8000-000000000001';
|
||||
execFileSync(python, ['-c', 'import sqlite3,sys; sqlite3.connect(sys.argv[1]).close()', database]);
|
||||
assert.deepEqual(await readExecutionRuns(database, job), { schemaAvailable: false, rows: [] });
|
||||
execFileSync(python, ['-c', `import sqlite3,sys
|
||||
c=sqlite3.connect(sys.argv[1])
|
||||
c.execute('CREATE TABLE execution_runs (run_id TEXT, job_id TEXT, session_id TEXT, outcome TEXT)')
|
||||
c.executemany('INSERT INTO execution_runs VALUES (?,?,?,?)', [('first',sys.argv[2],sys.argv[3],'succeeded'),('other','30000000-0000-4000-8000-000000000001',sys.argv[3],'failed')])
|
||||
c.commit()`, database, job, session]);
|
||||
const runId = '50000000-0000-4000-8000-000000000001';
|
||||
const otherSession = '40000000-0000-4000-8000-000000000001';
|
||||
const epoch = execFileSync(python, ['-c', `import sys
|
||||
from database.operations import PromptDatabase
|
||||
from utils.execution_history_events import RunStart, make_finish
|
||||
from utils.execution_contexts import ContextBundle, PromptSnapshot, RecipeContext, TextEvidence
|
||||
from types import SimpleNamespace
|
||||
|
||||
db = PromptDatabase(sys.argv[1])
|
||||
try:
|
||||
epoch = db.get_execution_epoch()
|
||||
c = db.model.get_connection()
|
||||
with c:
|
||||
c.executemany("INSERT INTO prompts(id,text,run_count) VALUES(?,?,?)",
|
||||
[(7, ' positive ', 19), (8, 'negative', 23)])
|
||||
c.execute("INSERT INTO prompt_pairs(id,positive_prompt_id,negative_prompt_id,run_count) VALUES(2,7,8,31)")
|
||||
for index, (job, session) in enumerate([
|
||||
(sys.argv[2], sys.argv[3]),
|
||||
('30000000-0000-4000-8000-000000000001', sys.argv[3]),
|
||||
(sys.argv[2], sys.argv[4]),
|
||||
], 1):
|
||||
run_id = '50000000-0000-4000-8000-%012d' % index
|
||||
db.write_execution_start(RunStart(run_id, session, index, job, epoch, 100))
|
||||
db.write_execution_finish(make_finish(session, index, epoch,
|
||||
SimpleNamespace(messages=[('execution_success', {'prompt_id': job})]), 200))
|
||||
def snapshot(raw):
|
||||
def known(value):
|
||||
return TextEvidence(value, 'recorded', 'runtime_return', None)
|
||||
return PromptSnapshot(known(raw), known(raw), known(''), known(''))
|
||||
db.write_execution_contexts(sys.argv[5], epoch, ContextBundle(
|
||||
(RecipeContext('4', snapshot(' positive '), snapshot('negative'), 'inferred'),),
|
||||
'complete', 'captured'))
|
||||
print(epoch)
|
||||
finally:
|
||||
db.close_all()
|
||||
`, database, job, session, otherSession, runId], { cwd: new URL('../../', import.meta.url), encoding: 'utf8' }).trim();
|
||||
const before = await readFile(database);
|
||||
assert.deepEqual(await readExecutionRuns(database, job, session), { schemaAvailable: true, rows: [
|
||||
{ run_id: 'first', job_id: job, session_id: session, outcome: 'succeeded' },
|
||||
const result = await readExecutionRuns(database, job, session);
|
||||
assert.equal(result.schemaAvailable, true);
|
||||
assert.deepEqual(result.rows, [{
|
||||
run_id: runId, job_id: job, session_id: session, queue_item_id: 1,
|
||||
storage_epoch: epoch, record_version: 1, observed_dequeue_at: 100,
|
||||
observed_callback_at: 200, executor_started_at: null, terminal_event_at: null,
|
||||
outcome: 'succeeded', reason: 'terminal_success', diagnostics_omitted: 0,
|
||||
node_id: null, node_type: null, exception_type: null,
|
||||
}]);
|
||||
assert.deepEqual(Object.keys(result.captures), [runId]);
|
||||
const capture = result.captures[runId];
|
||||
assert.equal(capture.contexts.length, 1);
|
||||
const contextId = capture.contexts[0].context_id;
|
||||
assert.match(contextId, /^[a-f0-9]{64}$/);
|
||||
const recorded = { provenance: 'recorded', source: 'runtime_return', reason: null };
|
||||
const evidence = { raw: recorded, effective: recorded, prepend: recorded, append: recorded };
|
||||
assert.deepEqual(capture, {
|
||||
state: 'complete', reason: 'captured', format_version: 1, contexts: [{
|
||||
anchor_id: '4', context_id: contextId,
|
||||
positive: { raw: ' positive ', effective: ' positive ', prepend: '', append: '' },
|
||||
negative: { raw: 'negative', effective: 'negative', prepend: '', append: '' },
|
||||
evidence: { format_version: 1, role_provenance: 'inferred', positive: evidence, negative: evidence },
|
||||
positive_prompt_id: 7, negative_prompt_id: 8, pair_id: 2, reference_epoch: epoch,
|
||||
}],
|
||||
});
|
||||
assert.deepEqual(result.usage, { [runId]: [
|
||||
{ role: 'positive', run_count: 19 }, { role: 'negative', run_count: 23 }, { role: 'pair', run_count: 31 },
|
||||
] });
|
||||
assert.deepEqual((await readExecutionRuns(database, job, '40000000-0000-4000-8000-000000000001')).rows, []);
|
||||
assert.deepEqual(Object.keys(result.snapshotIds), [runId]);
|
||||
assert.equal(result.snapshotIds[runId].length, 1);
|
||||
assert.deepEqual(Object.keys(result.snapshotIds[runId][0]).sort(), ['negative_snapshot_id', 'positive_snapshot_id']);
|
||||
const { positive_snapshot_id: positiveId, negative_snapshot_id: negativeId } = result.snapshotIds[runId][0];
|
||||
assert.equal(positiveId, 'd2d716108f66314e127fe87bb81327426d2a148e341a8ca1f7cf6a0b0df2ce8c');
|
||||
assert.equal(negativeId, '07db4923fea7f8d00d1a94a902e0b85137471178b5e7bb67ad664ef3370770ed');
|
||||
const unfiltered = await readExecutionRuns(database, job);
|
||||
assert.deepEqual(unfiltered.rows.map(row => row.session_id), [session, otherSession]);
|
||||
const uncapturedId = '50000000-0000-4000-8000-000000000003';
|
||||
assert.deepEqual(unfiltered.captures[uncapturedId], {
|
||||
state: 'not_captured', reason: 'missing', format_version: null, contexts: [],
|
||||
});
|
||||
assert.deepEqual(unfiltered.usage[uncapturedId], []);
|
||||
assert.deepEqual(unfiltered.snapshotIds[uncapturedId], []);
|
||||
assert.deepEqual(await readExecutionRuns(database, job, '60000000-0000-4000-8000-000000000001'), {
|
||||
schemaAvailable: true, rows: [], captures: {}, usage: {}, snapshotIds: {},
|
||||
});
|
||||
assert.deepEqual(await readFile(database), before);
|
||||
await assert.rejects(readExecutionRuns(database, "' OR 1=1 --"));
|
||||
execFileSync(python, ['-c', `import sqlite3,sys
|
||||
c=sqlite3.connect(sys.argv[1])
|
||||
c.executemany('INSERT INTO execution_runs VALUES (?,?,?,?)', [(str(i),sys.argv[2],sys.argv[3],'succeeded') for i in range(256)])
|
||||
c.commit()`, database, job, session]);
|
||||
await assert.rejects(readExecutionRuns(database, job, "' OR 1=1 --"));
|
||||
const addRuns = (start, stop) => execFileSync(python, ['-c', `import sys
|
||||
from database.operations import PromptDatabase
|
||||
from utils.execution_history_events import RunStart
|
||||
|
||||
db = PromptDatabase(sys.argv[1])
|
||||
try:
|
||||
epoch = db.get_execution_epoch()
|
||||
for index in range(int(sys.argv[4]), int(sys.argv[5])):
|
||||
run_id = '70000000-0000-4000-8000-%012d' % index
|
||||
db.write_execution_start(RunStart(run_id, sys.argv[3], index, sys.argv[2], epoch, 100))
|
||||
finally:
|
||||
db.close_all()
|
||||
`, database, job, session, String(start), String(stop)], { cwd: new URL('../../', import.meta.url) });
|
||||
addRuns(4, 258);
|
||||
assert.equal((await readExecutionRuns(database, job)).rows.length, 256);
|
||||
addRuns(258, 259);
|
||||
await assert.rejects(readExecutionRuns(database, job), /bound|256/);
|
||||
execFileSync(python, ['-c', `import sqlite3,sys
|
||||
with sqlite3.connect(sys.argv[1]) as c:
|
||||
c.execute('DROP TABLE execution_context_capture')
|
||||
`, database]);
|
||||
await assert.rejects(readExecutionRuns(database, job, otherSession), /no such table: execution_context_capture/);
|
||||
await assert.rejects(readExecutionRuns(join(directory, 'missing.db'), job));
|
||||
await writeFile(database, 'corrupt database');
|
||||
await assert.rejects(readExecutionRuns(database, job));
|
||||
|
||||
@@ -0,0 +1,111 @@
|
||||
"""Shared payload reservations with explicit, single-owner transport leases."""
|
||||
|
||||
from dataclasses import dataclass
|
||||
import json
|
||||
import threading
|
||||
|
||||
from .execution_contexts import BUNDLE_STATES, CAPTURE_REASONS, ContextBundle
|
||||
|
||||
|
||||
def _byte_count(value: int) -> None:
|
||||
if type(value) is not int or value < 0:
|
||||
raise ValueError("Byte count must be a nonnegative integer")
|
||||
|
||||
|
||||
class ContextBudget:
|
||||
def __init__(self, limit_bytes: int = 8388608) -> None:
|
||||
_byte_count(limit_bytes)
|
||||
self._limit = limit_bytes
|
||||
self._charged = 0
|
||||
self._lock = threading.Lock()
|
||||
|
||||
@property
|
||||
def charged_bytes(self) -> int:
|
||||
with self._lock:
|
||||
return self._charged
|
||||
|
||||
def reserve(self, byte_count: int) -> "PayloadLease | None":
|
||||
_byte_count(byte_count)
|
||||
with self._lock:
|
||||
if byte_count > self._limit - self._charged:
|
||||
return None
|
||||
self._charged += byte_count
|
||||
return PayloadLease._new(self, byte_count)
|
||||
|
||||
|
||||
class PayloadLease:
|
||||
def __init__(self, pool: ContextBudget, byte_count: int) -> None:
|
||||
raise ValueError("Reserve payload leases through ContextBudget")
|
||||
|
||||
@classmethod
|
||||
def _new(cls, pool: ContextBudget, byte_count: int) -> "PayloadLease":
|
||||
owner = object.__new__(cls)
|
||||
owner._pool = pool
|
||||
owner._bytes = byte_count
|
||||
owner._active = True
|
||||
return owner
|
||||
|
||||
@property
|
||||
def byte_count(self) -> int:
|
||||
with self._pool._lock:
|
||||
self._require_owner()
|
||||
return self._bytes
|
||||
|
||||
def _require_owner(self) -> None:
|
||||
if not self._active:
|
||||
raise ValueError("Inactive payload lease")
|
||||
|
||||
def transfer(self) -> "PayloadLease":
|
||||
with self._pool._lock:
|
||||
self._require_owner()
|
||||
owner = PayloadLease._new(self._pool, self._bytes)
|
||||
self._active = False
|
||||
return owner
|
||||
|
||||
def shrink_to(self, byte_count: int) -> None:
|
||||
_byte_count(byte_count)
|
||||
with self._pool._lock:
|
||||
self._require_owner()
|
||||
if byte_count > self._bytes:
|
||||
raise ValueError("Cannot grow payload lease")
|
||||
self._pool._charged -= self._bytes - byte_count
|
||||
self._bytes = byte_count
|
||||
|
||||
def release(self) -> None:
|
||||
with self._pool._lock:
|
||||
if self._active:
|
||||
self._pool._charged -= self._bytes
|
||||
self._active = False
|
||||
|
||||
|
||||
SHARED_CONTEXT_BUDGET = ContextBudget()
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SealedContext:
|
||||
bundle: ContextBundle | None
|
||||
lease: PayloadLease | None
|
||||
state: str
|
||||
reason: str
|
||||
capture_generation: int | None = None
|
||||
|
||||
def validate_header(self) -> None:
|
||||
if (
|
||||
type(self.state) is not str
|
||||
or self.state not in BUNDLE_STATES
|
||||
or type(self.reason) is not str
|
||||
or self.reason not in CAPTURE_REASONS
|
||||
or (
|
||||
self.capture_generation is not None
|
||||
and (
|
||||
type(self.capture_generation) is not int
|
||||
or not 0 <= self.capture_generation <= 2**63 - 1
|
||||
)
|
||||
)
|
||||
):
|
||||
raise ValueError("Invalid capture status")
|
||||
header = json.dumps(
|
||||
[self.state, self.reason, self.capture_generation], separators=(",", ":")
|
||||
).encode("utf-8")
|
||||
if len(header) > 256:
|
||||
raise ValueError("Oversized capture status")
|
||||
@@ -0,0 +1,451 @@
|
||||
"""Revocable, bounded runtime evidence owned by an actual dequeue attempt."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from contextvars import ContextVar
|
||||
from dataclasses import dataclass, replace
|
||||
import threading
|
||||
from typing import Callable, TYPE_CHECKING
|
||||
from uuid import uuid4
|
||||
import weakref
|
||||
|
||||
from .execution_context_budget import ContextBudget, PayloadLease, SealedContext
|
||||
from .execution_context_graph import (
|
||||
GraphProjection,
|
||||
GraphRole,
|
||||
SourceRef,
|
||||
project_contexts,
|
||||
)
|
||||
from .execution_contexts import (
|
||||
ContextBundle,
|
||||
PromptSnapshot,
|
||||
RecipeContext,
|
||||
TextEvidence,
|
||||
bundle_payload_size,
|
||||
text_size,
|
||||
MAX_ATTEMPT_BYTES,
|
||||
)
|
||||
from .execution_history_events import RunStart, encode_command
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from .execution_history_writer import HistoryWriterStatus
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class AttemptCapability:
|
||||
nonce: str
|
||||
run_id: str
|
||||
session_id: str
|
||||
queue_item_id: int
|
||||
job_id: str
|
||||
storage_epoch: str
|
||||
capture_generation: int
|
||||
sources: tuple[SourceRef, ...]
|
||||
|
||||
|
||||
@dataclass
|
||||
class _Attempt:
|
||||
capability: AttemptCapability
|
||||
projection: GraphProjection
|
||||
lease: PayloadLease | None
|
||||
samples: dict
|
||||
retained_bytes: int
|
||||
reason: str = ""
|
||||
executor_identity: int | None = None
|
||||
executor_rejected: bool = False
|
||||
|
||||
|
||||
_binding: ContextVar[tuple | None] = ContextVar("prompt_manager_attempt", default=None)
|
||||
|
||||
|
||||
def active_capture() -> tuple[ContextCapture, AttemptCapability] | None:
|
||||
binding = _binding.get()
|
||||
if binding is not None:
|
||||
manager = binding[0]()
|
||||
if manager is not None and manager.valid(binding[1]):
|
||||
return manager, binding[1]
|
||||
return None
|
||||
|
||||
|
||||
def unbind_current() -> None:
|
||||
binding = _binding.get()
|
||||
_binding.set(None)
|
||||
if binding is not None:
|
||||
manager = binding[0]()
|
||||
if manager is not None:
|
||||
manager.revoke(binding[1])
|
||||
|
||||
|
||||
def _unknown(reason: str = "missing") -> PromptSnapshot:
|
||||
field = TextEvidence(None, "unknown", "unknown", reason)
|
||||
return PromptSnapshot(field, field, field, field)
|
||||
|
||||
|
||||
def _snapshot_bytes(snapshot: PromptSnapshot) -> int:
|
||||
# Reuse the scalar validator, including provenance and per-field bounds.
|
||||
bundle = ContextBundle(
|
||||
(RecipeContext("x", snapshot, None, "inferred"),), "partial", "partial"
|
||||
)
|
||||
return bundle_payload_size(bundle)
|
||||
|
||||
|
||||
def _bounded_snapshot(snapshot: PromptSnapshot) -> tuple[PromptSnapshot, bool]:
|
||||
if type(snapshot) is not PromptSnapshot:
|
||||
raise ValueError("invalid_snapshot")
|
||||
fields, limited = [], False
|
||||
for field in (snapshot.raw, snapshot.effective, snapshot.prepend, snapshot.append):
|
||||
if type(field) is not TextEvidence:
|
||||
raise ValueError("invalid_field")
|
||||
if field.value is not None:
|
||||
try:
|
||||
text_size(field.value)
|
||||
except ValueError:
|
||||
field = _unknown("limit_reached").raw
|
||||
limited = True
|
||||
fields.append(field)
|
||||
if fields[1].value is not None and fields[1].provenance != "recorded":
|
||||
fields[1] = _unknown("missing").effective
|
||||
bounded = PromptSnapshot(*fields)
|
||||
_snapshot_bytes(bounded)
|
||||
return bounded, limited
|
||||
|
||||
|
||||
class ContextCapture:
|
||||
def __init__(
|
||||
self,
|
||||
queue_identity: int,
|
||||
status_reader: Callable[[], HistoryWriterStatus],
|
||||
budget: ContextBudget,
|
||||
) -> None:
|
||||
self.queue_identity = queue_identity
|
||||
self._status_reader = status_reader
|
||||
self._budget = budget
|
||||
self._lock = threading.RLock()
|
||||
self._attempts: dict[str, _Attempt] = {}
|
||||
|
||||
def _current(self, capability: AttemptCapability) -> _Attempt | None:
|
||||
attempt = self._attempts.get(capability.nonce)
|
||||
if attempt is None or attempt.capability is not capability:
|
||||
return None
|
||||
status = self._status_reader()
|
||||
if (
|
||||
not status.accepting
|
||||
or status.epoch != capability.storage_epoch
|
||||
or status.capture_generation != capability.capture_generation
|
||||
):
|
||||
self._drop(capability)
|
||||
return None
|
||||
return attempt
|
||||
|
||||
def valid(self, capability: AttemptCapability) -> bool:
|
||||
with self._lock:
|
||||
return self._current(capability) is not None
|
||||
|
||||
def begin(
|
||||
self, start: RunStart, generation: int, graph: object, outputs: object
|
||||
) -> AttemptCapability:
|
||||
unbind_current()
|
||||
encode_command(start)
|
||||
with self._lock:
|
||||
cap = AttemptCapability(
|
||||
str(uuid4()),
|
||||
start.run_id,
|
||||
start.session_id,
|
||||
start.queue_item_id,
|
||||
start.job_id,
|
||||
start.storage_epoch,
|
||||
generation,
|
||||
(),
|
||||
)
|
||||
if len(self._attempts) >= 128:
|
||||
return cap
|
||||
lease = self._budget.reserve(MAX_ATTEMPT_BYTES)
|
||||
projection = GraphProjection((), (), "")
|
||||
reason = "budget_exhausted" if lease is None else ""
|
||||
retained = 0
|
||||
if lease is not None:
|
||||
try:
|
||||
projection = project_contexts(graph, outputs)
|
||||
retained = (
|
||||
sum(
|
||||
_snapshot_bytes(role.inferred) if role.inferred else 0
|
||||
for recipe in projection.recipes
|
||||
for role in (recipe.positive, recipe.negative)
|
||||
)
|
||||
+ sum(
|
||||
_snapshot_bytes(snapshot)
|
||||
for _, snapshot in projection.source_snapshots
|
||||
)
|
||||
+ sum(
|
||||
text_size(s.node_id, 512) + text_size(s.node_type, 128) + 32
|
||||
for s in projection.sources
|
||||
)
|
||||
)
|
||||
if retained > MAX_ATTEMPT_BYTES:
|
||||
raise ValueError("projection_limit")
|
||||
except ValueError:
|
||||
projection = GraphProjection((), (), "")
|
||||
reason = "graph_limit"
|
||||
retained = 0
|
||||
cap = replace(cap, sources=projection.sources)
|
||||
self._attempts[cap.nonce] = _Attempt(
|
||||
cap, projection, lease, {}, retained, reason
|
||||
)
|
||||
_binding.set((weakref.ref(self), cap))
|
||||
self._current(cap)
|
||||
return cap
|
||||
|
||||
def bind_executor(self, capability: AttemptCapability, identity: int) -> None:
|
||||
"""Bind one scalar host identity; recursive starts invalidate cache proof."""
|
||||
with self._lock:
|
||||
attempt = self._current(capability)
|
||||
if attempt is not None:
|
||||
if attempt.executor_identity is not None:
|
||||
attempt.executor_rejected = True
|
||||
else:
|
||||
attempt.executor_identity = identity
|
||||
|
||||
def matches_executor(self, capability: AttemptCapability, identity: int) -> bool:
|
||||
with self._lock:
|
||||
attempt = self._current(capability)
|
||||
return bool(
|
||||
attempt is not None
|
||||
and not attempt.executor_rejected
|
||||
and attempt.executor_identity == identity
|
||||
)
|
||||
|
||||
def source_snapshot(
|
||||
self, capability: AttemptCapability, source: SourceRef
|
||||
) -> PromptSnapshot | None:
|
||||
with self._lock:
|
||||
attempt = self._current(capability)
|
||||
if attempt is not None:
|
||||
for ref, snapshot in attempt.projection.source_snapshots:
|
||||
if ref == source:
|
||||
return snapshot
|
||||
return None
|
||||
|
||||
def observe(
|
||||
self,
|
||||
capability: AttemptCapability,
|
||||
source: SourceRef,
|
||||
invocation_index: int,
|
||||
snapshot: PromptSnapshot,
|
||||
) -> None:
|
||||
try:
|
||||
with self._lock:
|
||||
attempt = self._current(capability)
|
||||
if (
|
||||
attempt is None
|
||||
or attempt.lease is None
|
||||
or source not in capability.sources
|
||||
or type(source) is not SourceRef
|
||||
or type(invocation_index) is not int
|
||||
or not 0 <= invocation_index <= 2**63 - 1
|
||||
):
|
||||
return
|
||||
snapshot, limited = _bounded_snapshot(snapshot)
|
||||
size = _snapshot_bytes(snapshot)
|
||||
if (
|
||||
limited
|
||||
or any(
|
||||
field.reason == "limit_reached"
|
||||
for field in (
|
||||
snapshot.raw,
|
||||
snapshot.effective,
|
||||
snapshot.prepend,
|
||||
snapshot.append,
|
||||
)
|
||||
)
|
||||
) and attempt.reason not in ("multiple_samples", "limit_reached"):
|
||||
attempt.reason = "field_limit"
|
||||
key = source, invocation_index
|
||||
if key in attempt.samples:
|
||||
if attempt.samples[key] != snapshot:
|
||||
attempt.reason = "multiple_samples"
|
||||
return
|
||||
if len(attempt.samples) >= 64:
|
||||
attempt.reason = "multiple_samples"
|
||||
return
|
||||
if attempt.retained_bytes + size > MAX_ATTEMPT_BYTES:
|
||||
attempt.reason = "limit_reached"
|
||||
return
|
||||
attempt.samples[key] = snapshot
|
||||
attempt.retained_bytes += size
|
||||
except (ValueError, AttributeError, TypeError):
|
||||
return
|
||||
|
||||
@staticmethod
|
||||
def _discard_payload(attempt: _Attempt) -> None:
|
||||
attempt.projection = GraphProjection((), (), "")
|
||||
attempt.samples.clear()
|
||||
attempt.retained_bytes = 0
|
||||
|
||||
def _drop(self, capability: AttemptCapability) -> _Attempt | None:
|
||||
attempt = self._attempts.get(capability.nonce)
|
||||
if attempt is not None and attempt.capability is capability:
|
||||
self._attempts.pop(capability.nonce)
|
||||
self._discard_payload(attempt)
|
||||
if attempt.lease is not None:
|
||||
attempt.lease.release()
|
||||
return attempt
|
||||
return None
|
||||
|
||||
def revoke(self, capability: AttemptCapability) -> None:
|
||||
with self._lock:
|
||||
self._drop(capability)
|
||||
|
||||
def revoke_all(self) -> None:
|
||||
with self._lock:
|
||||
for attempt in self._attempts.values():
|
||||
self._discard_payload(attempt)
|
||||
if attempt.lease is not None:
|
||||
attempt.lease.release()
|
||||
self._attempts.clear()
|
||||
|
||||
def seal(self, capability: AttemptCapability) -> SealedContext:
|
||||
with self._lock:
|
||||
attempt = self._current(capability)
|
||||
if attempt is None:
|
||||
return SealedContext(None, None, "unavailable", "unavailable")
|
||||
self._attempts.pop(capability.nonce)
|
||||
lease = attempt.lease
|
||||
if lease is None:
|
||||
return SealedContext(None, None, "unavailable", attempt.reason)
|
||||
try:
|
||||
ambiguous = attempt.reason == "multiple_samples"
|
||||
|
||||
def role_snapshot(role: GraphRole) -> PromptSnapshot:
|
||||
nonlocal ambiguous
|
||||
if role.reason == "source-transformed":
|
||||
return _unknown("unsupported")
|
||||
if not role.sources:
|
||||
return role.inferred or _unknown("unsupported")
|
||||
values = [
|
||||
snapshot
|
||||
for (source, _), snapshot in attempt.samples.items()
|
||||
if source in role.sources
|
||||
]
|
||||
if len(values) > 1 or attempt.reason == "multiple_samples":
|
||||
ambiguous = True
|
||||
return _unknown("ambiguous")
|
||||
if len(role.sources) == 1 and len(values) == 1:
|
||||
return values[0]
|
||||
inferred = role.inferred
|
||||
if len(role.sources) == 1:
|
||||
snapshots = attempt.projection.source_snapshots
|
||||
inferred = next(
|
||||
(
|
||||
snapshot
|
||||
for source, snapshot in snapshots
|
||||
if source == role.sources[0]
|
||||
),
|
||||
inferred,
|
||||
)
|
||||
if inferred is not None:
|
||||
s = inferred
|
||||
return PromptSnapshot(
|
||||
s.raw, _unknown().effective, s.prepend, s.append
|
||||
)
|
||||
return _unknown()
|
||||
|
||||
contexts = tuple(
|
||||
RecipeContext(
|
||||
r.anchor_id,
|
||||
role_snapshot(r.positive),
|
||||
role_snapshot(r.negative),
|
||||
"inferred",
|
||||
)
|
||||
for r in attempt.projection.recipes
|
||||
)
|
||||
supported = any(
|
||||
role.sources or role.inferred is not None
|
||||
for r in attempt.projection.recipes
|
||||
for role in (r.positive, r.negative)
|
||||
)
|
||||
complete = bool(contexts) and all(
|
||||
field.value is not None
|
||||
for r in contexts
|
||||
for s in (r.positive, r.negative)
|
||||
for field in (s.raw, s.effective, s.prepend, s.append)
|
||||
)
|
||||
state = (
|
||||
"ambiguous" if ambiguous else "complete" if complete else "partial"
|
||||
)
|
||||
reason = (
|
||||
"multiple_samples"
|
||||
if ambiguous
|
||||
else "captured" if complete else "partial"
|
||||
)
|
||||
if attempt.reason == "field_limit" and not ambiguous:
|
||||
state, reason = "partial", "field_limit"
|
||||
if not supported:
|
||||
contexts, state, reason = (), "unavailable", "no_supported_context"
|
||||
if attempt.reason in ("limit_reached", "graph_limit"):
|
||||
contexts, state, reason = (), "unavailable", attempt.reason
|
||||
bundle = ContextBundle(contexts, state, reason)
|
||||
size = bundle_payload_size(bundle)
|
||||
if size + attempt.retained_bytes > MAX_ATTEMPT_BYTES:
|
||||
raise ValueError("assembly_limit")
|
||||
self._discard_payload(attempt)
|
||||
lease.shrink_to(size)
|
||||
return SealedContext(
|
||||
bundle, lease, state, reason, capability.capture_generation
|
||||
)
|
||||
except ValueError:
|
||||
self._discard_payload(attempt)
|
||||
lease.release()
|
||||
return SealedContext(None, None, "unavailable", "limit_reached")
|
||||
|
||||
|
||||
def record_runtime_context(
|
||||
node_type: str,
|
||||
raw: str,
|
||||
prepend: str,
|
||||
append: str,
|
||||
effective: str,
|
||||
*,
|
||||
role: str | None = None,
|
||||
) -> None:
|
||||
"""Record only a successful node return in a trusted, bound host invocation."""
|
||||
try:
|
||||
binding = active_capture()
|
||||
if binding is None:
|
||||
return
|
||||
# Optional host import is independent of our package namespace.
|
||||
from comfy_execution.utils import get_executing_context
|
||||
|
||||
host = get_executing_context()
|
||||
manager, cap = binding
|
||||
if (
|
||||
host is None
|
||||
or type(host.prompt_id) is not str
|
||||
or host.prompt_id != cap.job_id
|
||||
):
|
||||
return
|
||||
if type(host.node_id) is not str or type(host.list_index) is not int:
|
||||
return
|
||||
if node_type == "PromptManagerPair":
|
||||
slots = (
|
||||
(0, 2) if role == "positive" else (1, 3) if role == "negative" else ()
|
||||
)
|
||||
else:
|
||||
slots = (
|
||||
(0,)
|
||||
if node_type == "PromptManagerText"
|
||||
else (0, 1) if role is None else ()
|
||||
)
|
||||
for slot in slots:
|
||||
source = SourceRef(host.node_id, node_type, slot)
|
||||
if source not in cap.sources:
|
||||
continue
|
||||
snapshot = PromptSnapshot(
|
||||
*(
|
||||
TextEvidence(value, "recorded", "runtime_return", None)
|
||||
for value in (raw, effective, prepend, append)
|
||||
)
|
||||
)
|
||||
manager.observe(cap, source, host.list_index, snapshot)
|
||||
except Exception:
|
||||
# Capture must never change the host result or expose input/exception text.
|
||||
return
|
||||
@@ -0,0 +1,426 @@
|
||||
"""Bounded projection of declared sampler roles, independent of the host."""
|
||||
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
from .execution_contexts import PromptSnapshot, TextEvidence, text_size
|
||||
from .prompt_pair_resolver import _DEFAULT_ADAPTERS
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SourceRef:
|
||||
node_id: str
|
||||
node_type: str
|
||||
output_slot: int
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class GraphRole:
|
||||
sources: tuple[SourceRef, ...]
|
||||
inferred: PromptSnapshot | None
|
||||
reason: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class DeclaredRecipe:
|
||||
anchor_id: str
|
||||
positive: GraphRole
|
||||
negative: GraphRole
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class GraphProjection:
|
||||
recipes: tuple[DeclaredRecipe, ...]
|
||||
sources: tuple[SourceRef, ...]
|
||||
reason: str
|
||||
source_snapshots: tuple[tuple[SourceRef, PromptSnapshot], ...] = ()
|
||||
|
||||
|
||||
_SAMPLERS = {"KSampler", "KSamplerAdvanced", "SamplerCustom"}
|
||||
_PM = {
|
||||
"PromptManager",
|
||||
"PromptManagerFocused",
|
||||
"PromptManagerSplit",
|
||||
"PromptManagerText",
|
||||
"PromptManagerPair",
|
||||
}
|
||||
|
||||
|
||||
class _Unknown(ValueError):
|
||||
pass
|
||||
|
||||
|
||||
class _Limit(_Unknown):
|
||||
pass
|
||||
|
||||
|
||||
def _identifier(value: object) -> str:
|
||||
if type(value) is int:
|
||||
if abs(value) > 10**127:
|
||||
raise _Unknown("invalid_node_identifier")
|
||||
value = str(value)
|
||||
try:
|
||||
text_size(value, 512)
|
||||
if len(value) > 128:
|
||||
raise ValueError("oversized_node_identifier")
|
||||
except ValueError:
|
||||
raise _Unknown("invalid_node_identifier") from None
|
||||
if not value:
|
||||
raise _Unknown("invalid_node_identifier")
|
||||
return value
|
||||
|
||||
|
||||
def _link(value: object) -> tuple[str, int]:
|
||||
if (
|
||||
type(value) not in (tuple, list)
|
||||
or len(value) != 2
|
||||
or type(value[1]) is not int
|
||||
or value[1] < 0
|
||||
):
|
||||
raise _Unknown("expected_typed_graph_link")
|
||||
return _identifier(value[0]), value[1]
|
||||
|
||||
|
||||
class _Projector:
|
||||
def __init__(self, graph: dict) -> None:
|
||||
self.graph = graph
|
||||
self.visits = 0
|
||||
self.retained = 0
|
||||
self.transformed = False
|
||||
self.selected_seen = set()
|
||||
|
||||
def node(self, node_id: str, depth: int) -> tuple[str, dict]:
|
||||
self.visits += 1
|
||||
if self.visits > 500 or depth > 64:
|
||||
raise _Unknown("graph_resolution_limit_reached")
|
||||
node = self.graph.get(node_id)
|
||||
if type(node) is not dict or type(node.get("inputs")) is not dict:
|
||||
raise _Unknown("missing_or_malformed_node")
|
||||
if len(node["inputs"]) > 500:
|
||||
raise _Unknown("node_input_limit_reached")
|
||||
kind = node.get("class_type")
|
||||
try:
|
||||
text_size(kind, 128)
|
||||
except ValueError:
|
||||
raise _Unknown("invalid_node_type") from None
|
||||
return kind, node["inputs"]
|
||||
|
||||
def text(
|
||||
self, value: object, kind: str, depth: int = 1, active: frozenset = frozenset()
|
||||
) -> tuple[PromptSnapshot, tuple]:
|
||||
if type(value) is str and kind == "string":
|
||||
text_size(value)
|
||||
return self.snapshot(value, value, "", ""), ()
|
||||
node_id, slot = _link(value)
|
||||
key = (node_id, slot)
|
||||
if key in active:
|
||||
raise _Unknown("cyclic_prompt_graph")
|
||||
name, inputs = self.node(node_id, depth)
|
||||
active = active | {key}
|
||||
if name == "Reroute" and slot == 0:
|
||||
keys = [key for key in ("value", "input", "in") if key in inputs]
|
||||
if len(keys) != 1:
|
||||
raise _Unknown("ambiguous_reroute")
|
||||
return self.text(inputs[keys[0]], kind, depth + 1, active)
|
||||
if (
|
||||
name in ("StringConcatenate", "Text Concatenate")
|
||||
and slot == 0
|
||||
and kind == "string"
|
||||
):
|
||||
keys = (
|
||||
["string_a", "string_b"]
|
||||
if name == "StringConcatenate"
|
||||
else sorted(
|
||||
key
|
||||
for key in inputs
|
||||
if type(key) is str and re.fullmatch(r"text_[a-z]", key)
|
||||
)
|
||||
)
|
||||
delimiter = inputs.get(
|
||||
"delimiter", "" if name == "StringConcatenate" else " "
|
||||
)
|
||||
text_size(delimiter)
|
||||
clean_value = inputs.get("clean_whitespace", "true")
|
||||
if name == "Text Concatenate" and type(clean_value) not in (str, bool):
|
||||
raise _Unknown("invalid_whitespace_mode")
|
||||
clean = name == "Text Concatenate" and (
|
||||
clean_value is True
|
||||
or type(clean_value) is str
|
||||
and clean_value.lower() == "true"
|
||||
)
|
||||
if not keys:
|
||||
raise _Unknown("missing_concatenation_inputs")
|
||||
parts, sources, size = [], [], 0
|
||||
for input_name in keys:
|
||||
child, refs = self.text(
|
||||
inputs.get(input_name, ""), "string", depth + 1, active
|
||||
)
|
||||
part = child.effective.value.strip() if clean else child.effective.value
|
||||
if part or name == "StringConcatenate":
|
||||
size += text_size(part) + (text_size(delimiter) if parts else 0)
|
||||
if size > 65536:
|
||||
raise _Unknown("text_field_limit_reached")
|
||||
parts.append(part)
|
||||
sources.extend(refs)
|
||||
if len(sources) > 64:
|
||||
raise _Unknown("source_reference_limit_reached")
|
||||
text = delimiter.join(parts)
|
||||
return self.snapshot(
|
||||
text, text, "", "", composed=True, raw_composed=True
|
||||
), tuple(sources)
|
||||
adapter = _DEFAULT_ADAPTERS.get((name, slot))
|
||||
if adapter is None or adapter.kind != kind:
|
||||
raise _Unknown("unsupported_node_semantics_or_output_slot")
|
||||
child, sources = self.text(
|
||||
inputs.get(adapter.input_name), "string", depth + 1, active
|
||||
)
|
||||
if name in _PM:
|
||||
sources = (SourceRef(node_id, name, slot),)
|
||||
if adapter.compose:
|
||||
before, before_sources = self.text(
|
||||
inputs.get("prepend_text", ""), "string", depth + 1, active
|
||||
)
|
||||
after, after_sources = self.text(
|
||||
inputs.get("append_text", ""), "string", depth + 1, active
|
||||
)
|
||||
raw = child.effective.value
|
||||
prepend, append = before.effective.value, after.effective.value
|
||||
parts = [part for part in (prepend.strip(), raw, append.strip()) if part]
|
||||
if sum(text_size(part) for part in parts) + max(0, len(parts) - 1) > 65536:
|
||||
raise _Unknown("text_field_limit_reached")
|
||||
effective = " ".join(parts)
|
||||
text_size(effective)
|
||||
snapshot = self.retain(
|
||||
PromptSnapshot(
|
||||
child.effective,
|
||||
TextEvidence(effective, "inferred", "graph_semantics", None),
|
||||
before.effective,
|
||||
after.effective,
|
||||
)
|
||||
)
|
||||
if name not in _PM:
|
||||
sources += before_sources + after_sources
|
||||
elif name in _PM:
|
||||
snapshot = self.retain(
|
||||
PromptSnapshot(
|
||||
child.effective,
|
||||
child.effective,
|
||||
TextEvidence("", "inferred", "graph_literal", None),
|
||||
TextEvidence("", "inferred", "graph_literal", None),
|
||||
)
|
||||
)
|
||||
else:
|
||||
snapshot = child
|
||||
if len(sources) > 64:
|
||||
raise _Unknown("source_reference_limit_reached")
|
||||
return snapshot, sources
|
||||
|
||||
def source_snapshot(self, source: SourceRef) -> PromptSnapshot:
|
||||
name, inputs = self.node(source.node_id, 1)
|
||||
adapter = _DEFAULT_ADAPTERS[(name, source.output_slot)]
|
||||
unknown = TextEvidence(None, "unknown", "unknown", "missing")
|
||||
|
||||
def literal(value: object) -> TextEvidence:
|
||||
try:
|
||||
snapshot, sources = self.text(value, "string", 2)
|
||||
# Stateful PM outputs cannot prove their injected result from inputs.
|
||||
return unknown if sources else snapshot.effective
|
||||
except _Limit:
|
||||
raise
|
||||
except ValueError:
|
||||
return unknown
|
||||
|
||||
raw = literal(inputs.get(adapter.input_name))
|
||||
before = literal(inputs.get("prepend_text", "") if adapter.compose else "")
|
||||
after = literal(inputs.get("append_text", "") if adapter.compose else "")
|
||||
return self.retain(PromptSnapshot(raw, unknown, before, after))
|
||||
|
||||
def discover_sources(
|
||||
self, value: object, kind: str, depth: int = 1, active: frozenset = frozenset()
|
||||
) -> tuple[SourceRef, ...]:
|
||||
if type(value) is str and kind == "string":
|
||||
return ()
|
||||
node_id, slot = _link(value)
|
||||
key = (node_id, slot)
|
||||
if key in active:
|
||||
raise _Unknown("cyclic_prompt_graph")
|
||||
name, inputs = self.node(node_id, depth)
|
||||
active = active | {key}
|
||||
adapter = _DEFAULT_ADAPTERS.get((name, slot))
|
||||
if adapter is not None and adapter.kind == kind:
|
||||
if name in _PM:
|
||||
return (SourceRef(node_id, name, slot),)
|
||||
return self.discover_sources(
|
||||
inputs.get(adapter.input_name), "string", depth + 1, active
|
||||
)
|
||||
if name == "Reroute" and slot == 0:
|
||||
keys = [key for key in ("value", "input", "in") if key in inputs]
|
||||
if len(keys) != 1:
|
||||
raise _Unknown("ambiguous_reroute")
|
||||
return self.discover_sources(inputs[keys[0]], kind, depth + 1, active)
|
||||
if (
|
||||
name in ("StringConcatenate", "Text Concatenate")
|
||||
and slot == 0
|
||||
and kind == "string"
|
||||
):
|
||||
keys = (
|
||||
["string_a", "string_b"]
|
||||
if name == "StringConcatenate"
|
||||
else sorted(
|
||||
key
|
||||
for key in inputs
|
||||
if type(key) is str and re.fullmatch(r"text_[a-z]", key)
|
||||
)
|
||||
)
|
||||
sources = []
|
||||
for key in keys:
|
||||
sources.extend(
|
||||
self.discover_sources(
|
||||
inputs.get(key, ""), "string", depth + 1, active
|
||||
)
|
||||
)
|
||||
if len(sources) > 64:
|
||||
raise _Unknown("source_limit_reached")
|
||||
self.transformed = self.transformed or bool(sources)
|
||||
return tuple(sources)
|
||||
return ()
|
||||
|
||||
def retain(self, snapshot: PromptSnapshot) -> PromptSnapshot:
|
||||
size = sum(
|
||||
(text_size(field.value) if field.value is not None else 0) + 64
|
||||
for field in (
|
||||
snapshot.raw,
|
||||
snapshot.effective,
|
||||
snapshot.prepend,
|
||||
snapshot.append,
|
||||
)
|
||||
)
|
||||
if self.retained + size > 1048576:
|
||||
raise _Limit("projection_payload_limit_reached")
|
||||
self.retained += size
|
||||
return snapshot
|
||||
|
||||
def snapshot(
|
||||
self,
|
||||
raw: str,
|
||||
effective: str,
|
||||
prepend: str,
|
||||
append: str,
|
||||
composed: bool = False,
|
||||
raw_composed: bool = False,
|
||||
) -> PromptSnapshot:
|
||||
return self.retain(
|
||||
PromptSnapshot(
|
||||
*(
|
||||
TextEvidence(
|
||||
value,
|
||||
"inferred",
|
||||
(
|
||||
"graph_semantics"
|
||||
if (composed and index == 1 or raw_composed and index == 0)
|
||||
else "graph_literal"
|
||||
),
|
||||
None,
|
||||
)
|
||||
for index, value in enumerate((raw, effective, prepend, append))
|
||||
)
|
||||
)
|
||||
)
|
||||
|
||||
def role(self, value: object) -> GraphRole:
|
||||
sources = ()
|
||||
try:
|
||||
self.transformed = False
|
||||
# Only the capture-source-to-consumer path determines transforms.
|
||||
sources = self.discover_sources(value, "conditioning")
|
||||
snapshot, _ = self.text(value, "conditioning")
|
||||
return GraphRole(
|
||||
sources, snapshot, "source-transformed" if self.transformed else ""
|
||||
)
|
||||
except _Limit:
|
||||
raise
|
||||
except ValueError as error:
|
||||
reason = str(error) if isinstance(error, _Unknown) else "invalid_text_value"
|
||||
return GraphRole(
|
||||
sources,
|
||||
None,
|
||||
"source-transformed" if self.transformed and sources else reason,
|
||||
)
|
||||
|
||||
def anchors(self, node_id: str, depth: int = 1) -> tuple[str, ...]:
|
||||
if node_id in self.selected_seen:
|
||||
return ()
|
||||
self.selected_seen.add(node_id)
|
||||
name, inputs = self.node(node_id, depth)
|
||||
anchors = [node_id] if name in _SAMPLERS else []
|
||||
for value in inputs.values():
|
||||
if (
|
||||
type(value) not in (tuple, list)
|
||||
or len(value) != 2
|
||||
or type(value[0]) not in (str, int)
|
||||
or type(value[1]) is not int
|
||||
or value[1] < 0
|
||||
):
|
||||
continue
|
||||
parent, _ = _link(value)
|
||||
anchors.extend(self.anchors(parent, depth + 1))
|
||||
if len(anchors) > 16:
|
||||
raise _Unknown("recipe_context_limit_reached")
|
||||
return tuple(anchors)
|
||||
|
||||
|
||||
def project_contexts(graph: object, outputs: object) -> GraphProjection:
|
||||
"""Keep declared roles and graph inference, never evidence of consumption.
|
||||
|
||||
Sources are PM capture points. An inferred snapshot may compose text; its
|
||||
presence never promotes a source observation to recorded effective text.
|
||||
"""
|
||||
try:
|
||||
if type(graph) is not dict or len(graph) > 500:
|
||||
raise _Unknown("invalid_or_oversized_graph")
|
||||
normalized = {_identifier(key): value for key, value in graph.items()}
|
||||
if len(normalized) != len(graph):
|
||||
raise _Unknown("duplicate_graph_identifiers")
|
||||
resolver = _Projector(normalized)
|
||||
if outputs is None:
|
||||
anchors = [
|
||||
key
|
||||
for key, value in normalized.items()
|
||||
if type(value) is dict
|
||||
and type(value.get("class_type")) is str
|
||||
and value["class_type"] in _SAMPLERS
|
||||
]
|
||||
elif type(outputs) in (list, tuple) and len(outputs) <= 500:
|
||||
anchors = []
|
||||
for output in outputs:
|
||||
anchors.extend(resolver.anchors(_identifier(output)))
|
||||
anchors = list(dict.fromkeys(anchors))
|
||||
else:
|
||||
raise _Unknown("invalid_output_selection")
|
||||
if len(anchors) > 16:
|
||||
raise _Unknown("recipe_context_limit_reached")
|
||||
recipes, sources = [], []
|
||||
for anchor in anchors:
|
||||
_, inputs = resolver.node(anchor, 1)
|
||||
positive = resolver.role(inputs.get("positive"))
|
||||
negative = resolver.role(inputs.get("negative"))
|
||||
recipes.append(DeclaredRecipe(anchor, positive, negative))
|
||||
sources.extend(positive.sources + negative.sources)
|
||||
sources = tuple(dict.fromkeys(sources))
|
||||
if len(sources) > 64:
|
||||
raise _Unknown("source_reference_limit_reached")
|
||||
reason = "" if recipes else "no_selected_sampler_anchors"
|
||||
if any(
|
||||
role.reason
|
||||
for recipe in recipes
|
||||
for role in (recipe.positive, recipe.negative)
|
||||
):
|
||||
reason = "some_roles_require_inferred_or_unresolved_text"
|
||||
return GraphProjection(
|
||||
tuple(recipes),
|
||||
sources,
|
||||
reason,
|
||||
tuple((source, resolver.source_snapshot(source)) for source in sources),
|
||||
)
|
||||
except ValueError as error:
|
||||
reason = str(error) if isinstance(error, _Unknown) else "invalid_text_value"
|
||||
return GraphProjection((), (), reason)
|
||||
@@ -0,0 +1,155 @@
|
||||
"""Pinned-host v1 adapter for local cached STRING evidence only."""
|
||||
|
||||
import functools
|
||||
import threading
|
||||
from typing import Callable
|
||||
|
||||
from .execution_context_capture import active_capture
|
||||
from .execution_contexts import PromptSnapshot, TextEvidence
|
||||
|
||||
_STATE_ATTRIBUTE = "_prompt_manager_executor_messages_v1"
|
||||
_registration_lock = threading.Lock()
|
||||
|
||||
|
||||
class _Messages:
|
||||
def __init__(self, executor_class: type) -> None:
|
||||
self.executor_class = executor_class
|
||||
self.lock = threading.Lock()
|
||||
self.callbacks = {}
|
||||
self.wrapper = None
|
||||
self.original = None
|
||||
self.installed = False
|
||||
|
||||
def subscribe(self, key: str, callback: Callable) -> Callable[[], None]:
|
||||
with self.lock:
|
||||
if not self.installed:
|
||||
original = self.executor_class.add_message
|
||||
if not callable(original):
|
||||
raise ValueError("Executor message capability unavailable")
|
||||
|
||||
@functools.wraps(original)
|
||||
def observe(executor, *args, **kwargs):
|
||||
result = original(executor, *args, **kwargs)
|
||||
with self.lock:
|
||||
callbacks = tuple(entry[1] for entry in self.callbacks.values())
|
||||
for notify in callbacks:
|
||||
try:
|
||||
notify(executor, args, kwargs)
|
||||
except Exception:
|
||||
# Best-effort observers cannot change host results.
|
||||
pass
|
||||
return result
|
||||
|
||||
self.executor_class.add_message = observe
|
||||
self.original, self.wrapper, self.installed = original, observe, True
|
||||
existing = self.callbacks.get(key)
|
||||
token = existing[0] if existing is not None else object()
|
||||
if existing is None:
|
||||
self.callbacks[key] = (token, callback)
|
||||
|
||||
def unsubscribe() -> None:
|
||||
with self.lock:
|
||||
current = self.callbacks.get(key)
|
||||
if current is None or current[0] is not token:
|
||||
return
|
||||
del self.callbacks[key]
|
||||
if (
|
||||
not self.callbacks
|
||||
and self.executor_class.add_message is self.wrapper
|
||||
):
|
||||
self.executor_class.add_message = self.original
|
||||
self.installed = False
|
||||
|
||||
return unsubscribe
|
||||
|
||||
|
||||
def subscribe_executor_messages(key: str, callback: Callable) -> Callable[[], None]:
|
||||
"""Observe successful original calls; preserve earlier and later wrappers."""
|
||||
from execution import PromptExecutor
|
||||
|
||||
with _registration_lock:
|
||||
state = vars(PromptExecutor).get(_STATE_ATTRIBUTE)
|
||||
if state is None:
|
||||
state = _Messages(PromptExecutor)
|
||||
setattr(PromptExecutor, _STATE_ATTRIBUTE, state)
|
||||
return state.subscribe(key, callback)
|
||||
|
||||
|
||||
def observe_cached_context(
|
||||
executor: object, message_type: object, message_data: object
|
||||
) -> None:
|
||||
"""Copy bounded local strings without cache/provider execution or retention."""
|
||||
try:
|
||||
binding = active_capture()
|
||||
if binding is None or type(message_type) is not str:
|
||||
return
|
||||
manager, cap = binding
|
||||
if (
|
||||
type(message_data) is not dict
|
||||
or type(message_data.get("prompt_id")) is not str
|
||||
or message_data["prompt_id"] != cap.job_id
|
||||
or id(executor.server.prompt_queue) != manager.queue_identity
|
||||
):
|
||||
return
|
||||
if message_type == "execution_start":
|
||||
manager.bind_executor(cap, id(executor))
|
||||
return
|
||||
if message_type != "execution_cached" or not manager.matches_executor(
|
||||
cap, id(executor)
|
||||
):
|
||||
return
|
||||
nodes = message_data.get("nodes")
|
||||
if type(nodes) not in (list, tuple) or len(nodes) > 500:
|
||||
return
|
||||
from comfy_execution import caching
|
||||
from execution import CacheEntry
|
||||
|
||||
cache = executor.caches.outputs
|
||||
supported = tuple(
|
||||
cls
|
||||
for name in ("HierarchicalCache", "LRUCache", "RAMPressureCache")
|
||||
if isinstance(cls := getattr(caching, name, None), type)
|
||||
)
|
||||
if type(cache) not in supported:
|
||||
return
|
||||
local = getattr(cache, "get_local", None)
|
||||
if not callable(local):
|
||||
return
|
||||
# Only source IDs retained by the bounded submitted graph are queried.
|
||||
for source in cap.sources:
|
||||
if not any(type(node) is str and node == source.node_id for node in nodes):
|
||||
continue
|
||||
slot = (
|
||||
(3 if source.output_slot in (1, 3) else 2)
|
||||
if source.node_type == "PromptManagerPair"
|
||||
else 0 if source.node_type == "PromptManagerText" else 1
|
||||
)
|
||||
entry = local(source.node_id)
|
||||
if type(entry) is not CacheEntry:
|
||||
continue
|
||||
outputs = entry.outputs
|
||||
if type(outputs) not in (list, tuple) or not slot < len(outputs) <= 4:
|
||||
continue
|
||||
values = outputs[slot]
|
||||
if type(values) not in (list, tuple) or len(values) > 64:
|
||||
continue
|
||||
inferred = manager.source_snapshot(cap, source)
|
||||
if inferred is None:
|
||||
continue
|
||||
for index, value in enumerate(values):
|
||||
if type(value) is not str:
|
||||
continue
|
||||
manager.observe(
|
||||
cap,
|
||||
source,
|
||||
index,
|
||||
PromptSnapshot(
|
||||
inferred.raw,
|
||||
TextEvidence(value, "recorded", "host_cache_output", None),
|
||||
inferred.prepend,
|
||||
inferred.append,
|
||||
),
|
||||
)
|
||||
except Exception:
|
||||
# Unknown host versions and malformed entries leave evidence unavailable.
|
||||
return
|
||||
@@ -0,0 +1,201 @@
|
||||
"""Scalar contracts for bounded execution context capture."""
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class TextEvidence:
|
||||
value: str | None
|
||||
provenance: str
|
||||
source: str
|
||||
reason: str | None
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class PromptSnapshot:
|
||||
raw: TextEvidence
|
||||
effective: TextEvidence
|
||||
prepend: TextEvidence
|
||||
append: TextEvidence
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class RecipeContext:
|
||||
anchor_id: str
|
||||
positive: PromptSnapshot | None
|
||||
negative: PromptSnapshot | None
|
||||
role_provenance: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ContextBundle:
|
||||
contexts: tuple[RecipeContext, ...]
|
||||
state: str
|
||||
reason: str
|
||||
format_version: int = 1
|
||||
|
||||
|
||||
MAX_FIELD_BYTES = 64 * 1024
|
||||
MAX_CONTEXT_BYTES = 256 * 1024
|
||||
MAX_ATTEMPT_BYTES = 1024 * 1024
|
||||
|
||||
|
||||
def text_size(value: str, limit: int = MAX_FIELD_BYTES) -> int:
|
||||
"""Reject oversize strings before allocating encoded copies."""
|
||||
if type(value) is not str or len(value) > limit:
|
||||
raise ValueError("Invalid or oversized text")
|
||||
size = 0
|
||||
for char in value:
|
||||
code = ord(char)
|
||||
if 0xD800 <= code <= 0xDFFF:
|
||||
raise ValueError("Invalid Unicode")
|
||||
size += 1 if code < 128 else 2 if code < 2048 else 3 if code < 65536 else 4
|
||||
if size > limit:
|
||||
raise ValueError("Oversized text")
|
||||
return size
|
||||
|
||||
|
||||
def canonical_content(kind: str, fields: tuple[str | None, ...]) -> bytes:
|
||||
text_size(kind, 128)
|
||||
if type(fields) is not tuple:
|
||||
raise ValueError("Invalid canonical fields")
|
||||
for value in fields:
|
||||
if value is not None:
|
||||
text_size(value)
|
||||
return json.dumps(
|
||||
[1, kind, *fields], ensure_ascii=False, allow_nan=False, separators=(",", ":")
|
||||
).encode("utf-8")
|
||||
|
||||
|
||||
def content_digest(kind: str, fields: tuple[str | None, ...]) -> str:
|
||||
return hashlib.sha256(canonical_content(kind, fields)).hexdigest()
|
||||
|
||||
|
||||
def snapshot_digest(snapshot: PromptSnapshot) -> str:
|
||||
return content_digest(
|
||||
"snapshot",
|
||||
tuple(
|
||||
getattr(snapshot, field).value
|
||||
for field in ("raw", "effective", "prepend", "append")
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def context_digest(context: RecipeContext) -> str:
|
||||
return content_digest(
|
||||
"context",
|
||||
tuple(
|
||||
snapshot_digest(side) if side is not None else None
|
||||
for side in (context.positive, context.negative)
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
BUNDLE_STATES = frozenset(
|
||||
{"complete", "partial", "ambiguous", "unavailable", "conflicting"}
|
||||
)
|
||||
|
||||
CAPTURE_REASONS = frozenset(
|
||||
{
|
||||
"",
|
||||
"captured",
|
||||
"partial",
|
||||
"unavailable",
|
||||
"unsupported",
|
||||
"ambiguous",
|
||||
"limit_reached",
|
||||
"missing",
|
||||
"conflicting",
|
||||
"no_supported_context",
|
||||
"budget_exhausted",
|
||||
"invalid_evidence",
|
||||
"multiple_samples",
|
||||
"source_transformed",
|
||||
"field_limit",
|
||||
"graph_limit",
|
||||
}
|
||||
)
|
||||
UNKNOWN_REASONS = frozenset(
|
||||
{"unavailable", "unsupported", "ambiguous", "limit_reached", "missing"}
|
||||
)
|
||||
EVIDENCE_SOURCES = {
|
||||
"recorded": frozenset({"runtime_return", "host_cache_output"}),
|
||||
"inferred": frozenset({"graph_literal", "graph_semantics"}),
|
||||
"unknown": frozenset({"unknown"}),
|
||||
}
|
||||
|
||||
|
||||
def validate_bundle(bundle: ContextBundle) -> None:
|
||||
bundle_payload_size(bundle)
|
||||
|
||||
|
||||
def bundle_payload_size(bundle: ContextBundle) -> int:
|
||||
"""Validate scalar contracts and measure the charged UTF-8 payload."""
|
||||
if (
|
||||
type(bundle) is not ContextBundle
|
||||
or type(bundle.format_version) is not int
|
||||
or bundle.format_version != 1
|
||||
or type(bundle.contexts) is not tuple
|
||||
or len(bundle.contexts) > 16
|
||||
):
|
||||
raise ValueError("Invalid context bundle")
|
||||
total = text_size(bundle.state, 128) + text_size(bundle.reason, 128)
|
||||
if bundle.state not in BUNDLE_STATES:
|
||||
raise ValueError("Invalid bundle state")
|
||||
if bundle.reason and re.fullmatch(r"[a-z][a-z0-9_-]*", bundle.reason) is None:
|
||||
raise ValueError("Invalid bundle reason code")
|
||||
anchors = set()
|
||||
for context in bundle.contexts:
|
||||
if type(context) is not RecipeContext:
|
||||
raise ValueError("Invalid recipe context")
|
||||
size = text_size(context.anchor_id, 512)
|
||||
if len(context.anchor_id) > 128:
|
||||
raise ValueError("Oversized recipe anchor")
|
||||
if not context.anchor_id or context.anchor_id in anchors:
|
||||
raise ValueError("Duplicate or empty recipe anchor")
|
||||
anchors.add(context.anchor_id)
|
||||
size += text_size(context.role_provenance, 128)
|
||||
if context.role_provenance != "inferred":
|
||||
raise ValueError("Invalid declared role provenance")
|
||||
for snapshot in (context.positive, context.negative):
|
||||
if snapshot is None:
|
||||
continue
|
||||
if type(snapshot) is not PromptSnapshot:
|
||||
raise ValueError("Invalid prompt snapshot")
|
||||
for field in (
|
||||
snapshot.raw,
|
||||
snapshot.effective,
|
||||
snapshot.prepend,
|
||||
snapshot.append,
|
||||
):
|
||||
if type(field) is not TextEvidence:
|
||||
raise ValueError("Invalid text evidence")
|
||||
if (
|
||||
type(field.provenance) is not str
|
||||
or type(field.source) is not str
|
||||
or field.source not in EVIDENCE_SOURCES.get(field.provenance, ())
|
||||
):
|
||||
raise ValueError("Invalid evidence provenance or source")
|
||||
if field.value is None:
|
||||
if (
|
||||
field.provenance != "unknown"
|
||||
or type(field.reason) is not str
|
||||
or field.reason not in UNKNOWN_REASONS
|
||||
):
|
||||
raise ValueError("Unknown evidence requires a reason code")
|
||||
elif field.provenance == "unknown" or field.reason is not None:
|
||||
raise ValueError("Known evidence requires a known provenance")
|
||||
if field.value is not None:
|
||||
size += text_size(field.value)
|
||||
size += text_size(field.provenance, 128) + text_size(field.source, 128)
|
||||
if field.reason is not None:
|
||||
size += text_size(field.reason, 128)
|
||||
if size > MAX_CONTEXT_BYTES:
|
||||
raise ValueError("Context payload limit reached")
|
||||
total += size
|
||||
if total > MAX_ATTEMPT_BYTES:
|
||||
raise ValueError("Attempt payload limit reached")
|
||||
return total
|
||||
+106
-18
@@ -1,5 +1,6 @@
|
||||
"""Own bounded queue-attempt capture without retaining host workflow payloads."""
|
||||
|
||||
from collections import OrderedDict
|
||||
import atexit
|
||||
import threading
|
||||
import time
|
||||
@@ -12,6 +13,8 @@ except ImportError:
|
||||
from database import PromptDatabase
|
||||
|
||||
from .execution_history_events import make_finish, make_start
|
||||
from .execution_context_capture import ContextCapture, active_capture, unbind_current
|
||||
from .execution_context_host import observe_cached_context, subscribe_executor_messages
|
||||
from .execution_history_writer import HistoryWriter
|
||||
from .logging_config import get_logger
|
||||
from .queue_observers import subscribe_dequeue, subscribe_task_done
|
||||
@@ -26,13 +29,27 @@ class HistoryRegistration:
|
||||
"""Own subscriptions, one asynchronous writer and bounded attempt identities."""
|
||||
|
||||
def __init__(
|
||||
self, queue: object, session_id: str, db_factory: Callable[[], PromptDatabase]
|
||||
self,
|
||||
queue: object,
|
||||
session_id: str,
|
||||
db_factory: Callable[[], PromptDatabase],
|
||||
*,
|
||||
high_water: int = -1,
|
||||
) -> None:
|
||||
self.writer = HistoryWriter(db_factory)
|
||||
self.writer = HistoryWriter(
|
||||
db_factory, on_invalidation=self._invalidate_capture
|
||||
)
|
||||
self.capture = ContextCapture(
|
||||
id(queue), self.writer.status, self.writer.context_budget
|
||||
)
|
||||
self._capabilities = {}
|
||||
self._accepted = OrderedDict()
|
||||
self._high_water = high_water
|
||||
self._session_id = session_id
|
||||
self._lock = threading.Lock()
|
||||
self._active: dict[int, str] = {}
|
||||
self._disposed = False
|
||||
self._capture_generation = None
|
||||
self._unsubscribers: list[Callable[[], None]] = []
|
||||
try:
|
||||
self._unsubscribers.append(
|
||||
@@ -41,46 +58,99 @@ class HistoryRegistration:
|
||||
self._unsubscribers.append(
|
||||
subscribe_task_done(queue, "execution_history", self._task_done)
|
||||
)
|
||||
try:
|
||||
self._unsubscribers.append(
|
||||
subscribe_executor_messages(
|
||||
f"execution_history_{id(self)}", self._executor_message
|
||||
)
|
||||
)
|
||||
except (ImportError, AttributeError, TypeError, ValueError):
|
||||
# Optional cache evidence must not prevent lifecycle tracking.
|
||||
pass
|
||||
self.writer.start()
|
||||
atexit.register(self.dispose)
|
||||
except Exception:
|
||||
self.dispose()
|
||||
raise
|
||||
|
||||
def _executor_message(self, executor: object, args: tuple, kwargs: dict) -> None:
|
||||
binding = active_capture()
|
||||
if binding is None or binding[0] is not self.capture:
|
||||
return
|
||||
event = args[0] if args else kwargs.get("event")
|
||||
data = args[1] if len(args) > 1 else kwargs.get("data")
|
||||
observe_cached_context(executor, event, data)
|
||||
|
||||
def _invalidate_capture(self) -> None:
|
||||
self.capture.revoke_all()
|
||||
|
||||
def _ready_epoch(self) -> str | None:
|
||||
status = self.writer.status()
|
||||
epoch = status.epoch if status.accepting else None
|
||||
if any(value != epoch for value in self._active.values()):
|
||||
if status.capture_generation != self._capture_generation or any(
|
||||
value != epoch for value in self._active.values()
|
||||
):
|
||||
self._active.clear()
|
||||
self._capabilities.clear()
|
||||
self.capture.revoke_all()
|
||||
self._capture_generation = status.capture_generation
|
||||
return epoch
|
||||
|
||||
def _dequeue(self, result: object) -> None:
|
||||
try:
|
||||
with self._lock:
|
||||
epoch = self._ready_epoch()
|
||||
command = (
|
||||
make_start(
|
||||
self._session_id, result, epoch, time.time_ns() // 1_000_000
|
||||
)
|
||||
if epoch
|
||||
else None
|
||||
)
|
||||
accepted = (
|
||||
self._accepted.get(command.queue_item_id)
|
||||
if command is not None
|
||||
else None
|
||||
)
|
||||
if command is not None and accepted == (
|
||||
command.job_id,
|
||||
command.storage_epoch,
|
||||
):
|
||||
return
|
||||
unbind_current()
|
||||
if self._disposed:
|
||||
return
|
||||
epoch = self._ready_epoch()
|
||||
if epoch is None:
|
||||
if command is None or command.queue_item_id in self._active:
|
||||
self.writer.record_gap()
|
||||
return
|
||||
command = make_start(
|
||||
self._session_id, result, epoch, time.time_ns() // 1_000_000
|
||||
)
|
||||
if command is None:
|
||||
self.writer.record_gap()
|
||||
return
|
||||
item_id = command.queue_item_id
|
||||
if item_id in self._active:
|
||||
return
|
||||
if len(self._active) >= _MAX_ACTIVE:
|
||||
self.writer.record_gap()
|
||||
return
|
||||
fresh = command.queue_item_id > self._high_water
|
||||
self._high_water = max(self._high_water, command.queue_item_id)
|
||||
if self.writer.submit(command):
|
||||
self._active[item_id] = epoch
|
||||
self._accepted[command.queue_item_id] = (
|
||||
command.job_id,
|
||||
command.storage_epoch,
|
||||
)
|
||||
if len(self._accepted) > 128:
|
||||
self._accepted.popitem(last=False)
|
||||
self._active[command.queue_item_id] = epoch
|
||||
if not fresh:
|
||||
return
|
||||
queue_item = result[0]
|
||||
graph = queue_item[2] if len(queue_item) > 2 else None
|
||||
outputs = queue_item[4] if len(queue_item) > 4 else None
|
||||
cap = self.capture.begin(
|
||||
command, self.writer.status().capture_generation, graph, outputs
|
||||
)
|
||||
self._capabilities[command.queue_item_id] = cap
|
||||
except Exception:
|
||||
unbind_current()
|
||||
self.writer.record_gap()
|
||||
|
||||
def _task_done(self, args: tuple, kwargs: dict) -> None:
|
||||
sealed = None
|
||||
try:
|
||||
with self._lock:
|
||||
if self._disposed:
|
||||
@@ -90,6 +160,11 @@ class HistoryRegistration:
|
||||
self.writer.record_gap()
|
||||
return
|
||||
item_id = args[0] if args else kwargs.get("item_id")
|
||||
if type(item_id) is int:
|
||||
cap = self._capabilities.pop(item_id, None)
|
||||
self._active.pop(item_id, None)
|
||||
if cap is not None:
|
||||
sealed = self.capture.seal(cap)
|
||||
status = args[2] if len(args) > 2 else kwargs.get("status")
|
||||
command = make_finish(
|
||||
self._session_id,
|
||||
@@ -101,10 +176,16 @@ class HistoryRegistration:
|
||||
if command is None:
|
||||
self.writer.record_gap()
|
||||
return
|
||||
self._active.pop(command.queue_item_id, None)
|
||||
self.writer.submit(command)
|
||||
if sealed is not None:
|
||||
self.writer.submit_finalization(command, sealed)
|
||||
sealed = None
|
||||
else:
|
||||
self.writer.submit(command)
|
||||
except Exception:
|
||||
self.writer.record_gap()
|
||||
finally:
|
||||
if sealed is not None and sealed.lease is not None:
|
||||
sealed.lease.release()
|
||||
|
||||
def dispose(self, timeout: float = 2.0) -> bool:
|
||||
"""Stop capture, release subscriptions and drain the owned writer boundedly."""
|
||||
@@ -112,6 +193,8 @@ class HistoryRegistration:
|
||||
if not self._disposed:
|
||||
self._disposed = True
|
||||
self._active.clear()
|
||||
self._capabilities.clear()
|
||||
self.capture.revoke_all()
|
||||
for unsubscribe in self._unsubscribers:
|
||||
unsubscribe()
|
||||
self._unsubscribers.clear()
|
||||
@@ -137,7 +220,12 @@ def register_history_hook(
|
||||
previous = state["registration"]
|
||||
if previous is not None and not previous._disposed:
|
||||
return previous
|
||||
registration = HistoryRegistration(queue, state["session_id"], db_factory)
|
||||
registration = HistoryRegistration(
|
||||
queue,
|
||||
state["session_id"],
|
||||
db_factory,
|
||||
high_water=previous._high_water if previous is not None else -1,
|
||||
)
|
||||
state["registration"] = registration
|
||||
return registration
|
||||
except Exception:
|
||||
|
||||
@@ -1,11 +1,13 @@
|
||||
"""A bounded, single-worker transport for best-effort execution history."""
|
||||
|
||||
from contextlib import contextmanager
|
||||
from dataclasses import dataclass
|
||||
from functools import partial
|
||||
import math
|
||||
import queue
|
||||
import threading
|
||||
import time
|
||||
from typing import Callable
|
||||
from typing import Callable, Iterator
|
||||
|
||||
try:
|
||||
from ..database import PromptDatabase
|
||||
@@ -20,7 +22,14 @@ except ImportError:
|
||||
subscribe_history_restore,
|
||||
)
|
||||
|
||||
from .execution_history_events import RunCommand, RunStart, encode_command
|
||||
from .execution_context_budget import (
|
||||
SHARED_CONTEXT_BUDGET,
|
||||
ContextBudget,
|
||||
PayloadLease,
|
||||
SealedContext,
|
||||
)
|
||||
from .execution_contexts import ContextBundle, bundle_payload_size
|
||||
from .execution_history_events import RunCommand, RunFinish, RunStart, encode_command
|
||||
from .logging_config import get_logger
|
||||
|
||||
_logger = get_logger("prompt_manager.execution_history_writer")
|
||||
@@ -36,6 +45,13 @@ class HistoryWriterStatus:
|
||||
pending: int
|
||||
recording_gaps: int
|
||||
alive: bool
|
||||
capture_generation: int
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class _Finalization:
|
||||
command: RunFinish
|
||||
context: SealedContext
|
||||
|
||||
|
||||
class HistoryWriter:
|
||||
@@ -46,12 +62,21 @@ class HistoryWriter:
|
||||
db_factory: Callable[[], PromptDatabase],
|
||||
*,
|
||||
clock: Callable[[], float] = time.monotonic,
|
||||
context_budget: ContextBudget | None = None,
|
||||
on_invalidation: Callable[[], None] | None = None,
|
||||
) -> None:
|
||||
self.context_budget = (
|
||||
SHARED_CONTEXT_BUDGET if context_budget is None else context_budget
|
||||
)
|
||||
self._on_invalidation = on_invalidation
|
||||
self._db_factory = db_factory
|
||||
self._clock = clock
|
||||
self._context_commit_gate = threading.Lock()
|
||||
self._lock = threading.Lock()
|
||||
self._wake = threading.Event()
|
||||
self._queue: queue.Queue[RunCommand] = queue.Queue(maxsize=_MAX_PENDING)
|
||||
self._queue: queue.Queue[RunCommand | _Finalization] = queue.Queue(
|
||||
maxsize=_MAX_PENDING
|
||||
)
|
||||
self._thread: threading.Thread | None = None
|
||||
self._epoch: str | None = None
|
||||
self._accepting = False
|
||||
@@ -94,6 +119,69 @@ class HistoryWriter:
|
||||
self._wake.set()
|
||||
return accepted
|
||||
|
||||
@staticmethod
|
||||
def _release_context(context: SealedContext) -> None:
|
||||
if isinstance(context, SealedContext) and type(context.lease) is PayloadLease:
|
||||
context.lease.release()
|
||||
|
||||
def _consume_context_locked(self, context: SealedContext) -> SealedContext:
|
||||
try:
|
||||
if type(context) is not SealedContext:
|
||||
raise ValueError("Invalid sealed context")
|
||||
context.validate_header()
|
||||
if context.bundle is None:
|
||||
if context.lease is not None:
|
||||
raise ValueError("Unexpected payload lease")
|
||||
return context
|
||||
size = bundle_payload_size(context.bundle)
|
||||
if (
|
||||
context.capture_generation != self._generation
|
||||
or context.capture_generation is None
|
||||
or type(context.lease) is not PayloadLease
|
||||
or context.lease._pool is not self.context_budget
|
||||
or context.lease.byte_count < size
|
||||
or context.state != context.bundle.state
|
||||
or context.reason != context.bundle.reason
|
||||
):
|
||||
raise ValueError("Invalid context ownership or generation")
|
||||
owner = context.lease.transfer()
|
||||
return SealedContext(
|
||||
context.bundle,
|
||||
owner,
|
||||
context.state,
|
||||
context.reason,
|
||||
context.capture_generation,
|
||||
)
|
||||
except ValueError:
|
||||
self._release_context(context)
|
||||
return SealedContext(None, None, "unavailable", "invalid_evidence")
|
||||
|
||||
def submit_finalization(self, command: RunFinish, context: SealedContext) -> bool:
|
||||
try:
|
||||
if type(command) is not RunFinish:
|
||||
raise ValueError("Finalization requires finish")
|
||||
encode_command(command)
|
||||
except ValueError:
|
||||
self._release_context(context)
|
||||
self.record_gap()
|
||||
return False
|
||||
with self._lock:
|
||||
if (
|
||||
not self._accepting
|
||||
or command.storage_epoch != self._epoch
|
||||
or self._pending >= _MAX_PENDING
|
||||
):
|
||||
self._release_context(context)
|
||||
self._record_gap_locked()
|
||||
accepted = False
|
||||
else:
|
||||
owned = self._consume_context_locked(context)
|
||||
self._queue.put_nowait(_Finalization(command, owned))
|
||||
self._pending += 1
|
||||
accepted = True
|
||||
self._wake.set()
|
||||
return accepted
|
||||
|
||||
def _record_gap_locked(self) -> None:
|
||||
self._gaps = min(_MAX_GAPS, self._gaps + 1)
|
||||
|
||||
@@ -103,13 +191,36 @@ class HistoryWriter:
|
||||
self._record_gap_locked()
|
||||
self._wake.set()
|
||||
|
||||
@contextmanager
|
||||
def _context_generation_guard(
|
||||
self, generation: int, epoch: str, capture_generation: int | None
|
||||
) -> Iterator[bool]:
|
||||
with self._context_commit_gate:
|
||||
with self._lock:
|
||||
current = (
|
||||
generation == self._generation
|
||||
and epoch == self._epoch
|
||||
and (capture_generation is None or capture_generation == generation)
|
||||
)
|
||||
yield current
|
||||
|
||||
def _notify_invalidation(self) -> None:
|
||||
if self._on_invalidation is not None:
|
||||
try:
|
||||
self._on_invalidation()
|
||||
except Exception:
|
||||
# Observer failure must not block revocation of stale captures.
|
||||
pass
|
||||
|
||||
def invalidate(self) -> None:
|
||||
with self._lock:
|
||||
self._accepting = False
|
||||
self._epoch = None
|
||||
self._generation += 1
|
||||
self._next_probe = self._clock() + _PROBE_INTERVAL
|
||||
self._discard_locked()
|
||||
with self._context_commit_gate:
|
||||
with self._lock:
|
||||
self._accepting = False
|
||||
self._epoch = None
|
||||
self._generation += 1
|
||||
self._next_probe = self._clock() + _PROBE_INTERVAL
|
||||
self._discard_locked()
|
||||
self._notify_invalidation()
|
||||
self._wake.set()
|
||||
|
||||
def status(self) -> HistoryWriterStatus:
|
||||
@@ -120,6 +231,7 @@ class HistoryWriter:
|
||||
self._pending,
|
||||
self._gaps,
|
||||
self._thread is not None and self._thread.is_alive(),
|
||||
self._generation,
|
||||
)
|
||||
|
||||
def close(self, timeout: float = 2.0) -> bool:
|
||||
@@ -132,17 +244,31 @@ class HistoryWriter:
|
||||
self._close_deadline = deadline
|
||||
self._accepting = False
|
||||
thread = self._thread
|
||||
self._notify_invalidation()
|
||||
self._wake.set()
|
||||
if thread is not None:
|
||||
thread.join(max(0.0, deadline - time.monotonic()))
|
||||
while thread.is_alive():
|
||||
remaining = deadline - time.monotonic()
|
||||
if remaining <= 0:
|
||||
break
|
||||
thread.join(remaining)
|
||||
if thread.is_alive():
|
||||
remaining = deadline - time.monotonic()
|
||||
if remaining > 0:
|
||||
time.sleep(min(remaining, 0.001))
|
||||
with self._lock:
|
||||
if self._closing and time.monotonic() >= self._close_deadline:
|
||||
self._discard_locked()
|
||||
return thread is None or not thread.is_alive()
|
||||
|
||||
def _discard_locked(self) -> None:
|
||||
while True:
|
||||
try:
|
||||
self._queue.get_nowait()
|
||||
item = self._queue.get_nowait()
|
||||
except queue.Empty:
|
||||
return
|
||||
if isinstance(item, _Finalization):
|
||||
self._release_context(item.context)
|
||||
self._pending -= 1
|
||||
self._record_gap_locked()
|
||||
|
||||
@@ -200,6 +326,10 @@ class HistoryWriter:
|
||||
self._record_gap_locked()
|
||||
if command is not None:
|
||||
failed = False
|
||||
context = None
|
||||
if isinstance(command, _Finalization):
|
||||
context = command.context
|
||||
command = command.command
|
||||
try:
|
||||
# Restore may have reopened this thread's managed connection.
|
||||
db.model.get_connection().execute("PRAGMA busy_timeout=250")
|
||||
@@ -207,12 +337,49 @@ class HistoryWriter:
|
||||
db.write_execution_start(command)
|
||||
else:
|
||||
db.write_execution_finish(command)
|
||||
if context is not None:
|
||||
with self._lock:
|
||||
current = (
|
||||
generation == self._generation
|
||||
and command.storage_epoch == self._epoch
|
||||
and (
|
||||
context.capture_generation is None
|
||||
or context.capture_generation
|
||||
== self._generation
|
||||
)
|
||||
)
|
||||
if current:
|
||||
try:
|
||||
db.write_execution_contexts_for_finish(
|
||||
command,
|
||||
(
|
||||
context.bundle
|
||||
if context.bundle is not None
|
||||
else ContextBundle(
|
||||
(), context.state, context.reason
|
||||
)
|
||||
),
|
||||
generation_guard=partial(
|
||||
self._context_generation_guard,
|
||||
generation,
|
||||
command.storage_epoch,
|
||||
context.capture_generation,
|
||||
),
|
||||
)
|
||||
except ValueError:
|
||||
# Validation/collision cannot revoke the outcome.
|
||||
with self._lock:
|
||||
self._record_gap_locked()
|
||||
except HistoryEpochChanged:
|
||||
# The exception's storage_epoch is only a hint; readiness
|
||||
# comes from a later probe and never replays this command.
|
||||
failed = True
|
||||
except Exception:
|
||||
failed = True
|
||||
finally:
|
||||
if context is not None:
|
||||
self._release_context(context)
|
||||
context = None
|
||||
with self._lock:
|
||||
self._pending -= 1
|
||||
if failed:
|
||||
@@ -222,6 +389,8 @@ class HistoryWriter:
|
||||
self._generation += 1
|
||||
self._next_probe = self._clock() + _PROBE_INTERVAL
|
||||
self._discard_locked()
|
||||
if failed:
|
||||
self._notify_invalidation()
|
||||
with self._lock:
|
||||
gaps = self._gaps
|
||||
has_work = self._epoch is not None and not self._queue.empty()
|
||||
@@ -239,6 +408,7 @@ class HistoryWriter:
|
||||
self._accepting = False
|
||||
self._epoch = None
|
||||
self._discard_locked()
|
||||
self._notify_invalidation()
|
||||
if unsubscribe is not None:
|
||||
unsubscribe()
|
||||
if db is not None:
|
||||
|
||||
Reference in New Issue
Block a user