f356e13bef
The Analyzer collapsed relational phrasing to the bare noun ("a bunch of money
in her hand" -> just "money", losing the held relationship) and could drop the
later half of "X and Y" enumerations, especially on NSFW input. The reference
Knowledge Base already covers the relational tags (holding money, hand on own
hip, in mouth, 400+ holding_* tags), so the loss was entirely at the analysis
step.
- System prompt: add "Relations, positions, and holding", "Conjunctions and
enumerations", and "Completeness (including explicit content)" sections, and
cross-reference them from "Objects". Held/worn/positioned objects now also
yield the relational concept (holding <object>); every conjunct survives;
explicit content gets the same full fidelity.
- Regression tests: relational (holding money, hand on own hip) and conjunction
(holding sword + holding shield) cases prove the reference KB resolves these
concepts end to end, so the improved Analyzer output compiles.
Co-authored-by: kunail0804 <kunail0804@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
97 lines
3.4 KiB
Python
97 lines
3.4 KiB
Python
"""Concept-fidelity regression cases (epic #86, issue #113).
|
|
|
|
Durable, data-driven guards for the two fidelity failure modes so they cannot
|
|
silently return:
|
|
|
|
- **Modifier loss during resolution** (#111): the head-noun reduction used to
|
|
drop leading modifiers ("open white shirt" → "white shirt"), silently losing
|
|
"open" even though it forms a valid compound tag. Each case asserts the
|
|
compiled prompt contains both the head-noun tag and the recovered modifier tag.
|
|
- **List under-transcription** (#112): every item of a tag-list input must survive
|
|
to the compiled prompt. The list case pins that resolution never silently drops
|
|
a known list item.
|
|
|
|
Cases are ``(name, scene, required_tags)`` triples, so a new regression is a
|
|
one-line addition. Assertions run against the shipped reference Knowledge Base
|
|
with a fixed Scene JSON — no live Analyzer backend — so they are deterministic.
|
|
Every case here fails against pre-#86 behaviour and passes once the fidelity
|
|
fixes land.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import pytest
|
|
|
|
from tests.regression.golden_scenes import (
|
|
_character,
|
|
_scene,
|
|
compile_prompt,
|
|
load_reference_knowledge_base,
|
|
)
|
|
|
|
# (name, scene, tags that MUST appear in the compiled prompt)
|
|
FIDELITY_CASES = [
|
|
(
|
|
"modifier_open_white_shirt",
|
|
_scene([_character(0, clothing=["open white shirt"])]),
|
|
["white shirt", "open shirt"],
|
|
),
|
|
(
|
|
"modifier_long_messy_hair",
|
|
_scene([_character(0, appearance=["long messy hair"])]),
|
|
["messy hair", "long hair"],
|
|
),
|
|
(
|
|
"modifier_long_blue_skirt",
|
|
_scene([_character(0, clothing=["long blue skirt"])]),
|
|
["blue skirt", "long skirt"],
|
|
),
|
|
(
|
|
"full_list_transcription",
|
|
_scene(
|
|
[
|
|
_character(
|
|
0, appearance=["long hair", "blue eyes"], clothing=["thighhighs", "jacket"]
|
|
)
|
|
],
|
|
environment=["classroom"],
|
|
),
|
|
["long hair", "blue eyes", "thighhighs", "jacket", "classroom"],
|
|
),
|
|
# Relational fidelity (bug report #3): the Analyzer must keep held/positioned
|
|
# relationships as concepts ("money in her hand" -> holding money), not collapse
|
|
# them to the bare noun. These pin that the reference KB resolves such concepts,
|
|
# so the improved Analyzer output compiles end to end.
|
|
(
|
|
"relational_holding_money",
|
|
_scene([_character(0, actions=["holding money"])]),
|
|
["holding money"],
|
|
),
|
|
(
|
|
"relational_hand_on_hip",
|
|
_scene([_character(0, pose=["hand on own hip"])]),
|
|
["hand on own hip"],
|
|
),
|
|
# Conjunction fidelity: both items of "a sword and a shield" must survive.
|
|
(
|
|
"conjunction_sword_and_shield",
|
|
_scene([_character(0, actions=["holding sword", "holding shield"])]),
|
|
["holding sword", "holding shield"],
|
|
),
|
|
]
|
|
|
|
|
|
@pytest.fixture(scope="module")
|
|
def knowledge_base():
|
|
return load_reference_knowledge_base()
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"name, scene, required_tags", FIDELITY_CASES, ids=[c[0] for c in FIDELITY_CASES]
|
|
)
|
|
def test_fidelity_case(name: str, scene, required_tags: list[str], knowledge_base) -> None:
|
|
prompt = compile_prompt(scene, knowledge_base)
|
|
tags = prompt.split(",")
|
|
missing = [tag for tag in required_tags if tag not in tags]
|
|
assert not missing, f"{name}: prompt {prompt!r} is missing {missing}"
|