diff --git a/CHANGELOG.md b/CHANGELOG.md index b854dac..b70c7f1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,6 +15,11 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Changed - **Breaking:** `OllamaChatCompletion` → `ChatCompletion`, `OllamaModelSelector` → `LLMModelSelector`, `OllamaLoadModel` → `LLMLoadModel`, `OllamaUnloadModel` → `LLMUnloadModel`, and the `OLLAMA_CLIENT` socket type → `LLM_CLIENT` — these nodes are now backend-generic. `OllamaClient` is unchanged by name but now outputs an `OllamaProvider` rather than a plain string; existing saved workflows using the old node/socket names need reconnecting (see `comfydv.ollama.MIGRATION_MAP` for the full old→new mapping). +- `ChatCompletion`'s `structured_output=True` path now routes Ollama through Ollama's native `/api/chat` + `"format"` instead of the shared `pydantic-ai` OpenAI-compat path — Ollama's OpenAI-compatible endpoint was found to silently reload the model at its default context size on every call, discarding any `options` (e.g. `num_ctx`) override. `LlamaCppProvider` is unaffected and keeps the shared path, switched to `pydantic-ai`'s `NativeOutput` mode (ADR-009). + +### Fixed +- `structured_output=True` requests could fail validation ("token limit exceeded before any response was generated") against "thinking"-capable models, which spent their whole token budget on chain-of-thought reasoning before ever producing the structured response (ADR-009). +- A non-required structured-output schema field rejected an explicit `null` value from the model (only an *omitted* field was tolerated), even though models routinely emit explicit `null` for absent optional fields. ## [0.1.0] — 2026-06-01 diff --git a/project-management/ADRs/ADR-009-native-structured-output-mode.md b/project-management/ADRs/ADR-009-native-structured-output-mode.md new file mode 100644 index 0000000..4a77149 --- /dev/null +++ b/project-management/ADRs/ADR-009-native-structured-output-mode.md @@ -0,0 +1,82 @@ +# ADR-009: Provider-specific structured output — NativeOutput for llama.cpp, hand-rolled native `/api/chat` for Ollama + +## Status + +> Accepted + +_Date:_ 2026-07-25 +_Deciders:_ darth-veitcher + +--- + +## Context + +`chat_structured()` (`src/comfydv/_llm/chat.py`, ADR-007) builds its `pydantic-ai` `Agent` with a bare `output_type=schema`. Passing a raw `pydantic.BaseModel` subclass this way makes `pydantic-ai` default to **tool-calling** (a synthetic forced function call) for structured output — inherited silently from ADR-007's move to `pydantic-ai`, never a deliberate re-decision. ADR-007 doesn't discuss output-mode choice at all. + +Testing a real multi-agent ComfyUI workflow (`workflows/ltx-i2v-pipeline.json`) against a live local Ollama server surfaced this as a real reliability problem: against a "thinking"-capable model (`qwen3.5:9b`), tool-calling failed consistently — the model spent its entire token budget on internal chain-of-thought reasoning and never emitted the tool call, so every attempt failed pydantic validation ("token limit exceeded before any response was generated"), each attempt taking 5-7+ minutes before giving up. + +### First fix attempt: `NativeOutput` + a priming call (superseded within this same ADR) + +`pydantic-ai` 2.9.0 exposes `NativeOutput`, which makes the `Agent` use `response_format: {"type": "json_schema", ...}` over the OpenAI-compatible endpoint instead of tool-calling. Live-tested directly against Ollama via `curl` before touching code: `/v1/chat/completions` with `response_format: json_schema` returned clean, schema-valid JSON, with the model's reasoning in a separate `message.reasoning` field — fast, and reasoning no longer competed with structured output for token budget. This part of the fix is real and is kept — see Decision §1. + +A second, separate problem was also found: Ollama's OpenAI-compatible endpoint doesn't honor a per-request `options` override (e.g. `num_ctx`) — sending the same request to native `/api/chat` reloaded the model at the requested context size; `/v1/chat/completions` silently kept whatever was already loaded. The first fix attempt worked around this with a priming call: hit native `/api/generate` with the desired `options` immediately before the real `/v1/chat/completions` request, on the theory that Ollama would keep the just-loaded context for the next call. + +**This did not work, and the failure mode looked exactly like the original bug** — confirmed while re-testing the actual workflow end-to-end (`workflows/ltx-i2v-pipeline.json`, Agent 2 "Scene Grounder": long system prompt + 9-property schema + image), which kept failing with the identical "token limit exceeded" error even after the priming fix landed, tests passed, and `max_tokens`/`num_ctx`/`timeout_secs` were all raised generously. Isolated the exact mechanism with a direct, non-ComfyUI-mediated `curl` sequence: + +1. `POST /api/generate` with `options: {num_ctx: 20480}`, `keep_alive: -1` → confirmed via `GET /api/ps`: `context_length: 20480`, loaded "forever". +2. Immediately `POST /v1/chat/completions` for the same model — **even with the identical `options: {num_ctx: 20480}` included in that request's body** → `GET /api/ps` immediately after: `context_length: 4096` (back to default), `expires_at` reset to a normal ~5-minute keep-alive. + +So `/v1/chat/completions` doesn't merely *ignore* `options.num_ctx` — every call to it silently **reloads the model at the default context size**, discarding whatever was primed, regardless of what that same call's own `options` field says. A priming call immediately before the real request is structurally incapable of working, because the real request itself is what undoes the priming. + +Re-ran the same sequence against native `/api/chat` instead of `/v1/chat/completions`: the primed `context_length: 20480` was preserved through and after the call. Native `/api/chat` also accepts `"format": ` directly, giving grammar-constrained structured output in the same request that correctly honors `options` — no separate priming call needed at all. + +### Re-litigating ADR-006's model concern + +This also revisits [ADR-006](ADR-006-structured-ollama-output-tool-calling-not-pydantic-ai.md), which rejected native `format`-based output — but its rejection was scoped to one specific model, `lukey03/qwen3.5-9b-abliterated-vision`, whose degenerate chat template silently ignored the constraint. ADR-006 explicitly flagged this as revisitable: *"If well-behaved-model testing later shows native `format` is meaningfully more reliable in the common case, this decision should be revisited rather than treated as permanent."* Re-tested that exact model against native structured output: it no longer silently ignores the constraint (ADR-006's specific failure mode) — it returns schema-valid JSON, but the *content* is still garbled (`"ponáp∵49\n"` instead of the requested `"pong"`), consistent with ADR-006's "degenerate tokenizer" diagnosis. That model was also already failing under the tool-calling path (hanging without completing, observed live during this same investigation). So neither part of this decision regresses that model — it was already unusable for structured output either way. `structured_output` has no production users yet, so there is no back-compat concern in making this change. + +## Decision + +**1. `LlamaCppProvider` keeps the shared `pydantic-ai` path, switched to `NativeOutput`.** `chat.py`'s `_build_agent()` builds `Agent(chat_model, output_type=NativeOutput(schema), retries=0)` instead of a bare `output_type=schema`. llama-server's OpenAI-compatible endpoint is its genuine native structured-output surface (no equivalent context-reload bug found or expected — llama-server's context is fixed at process launch via `--ctx-size`, not a per-request concern, so there's nothing for a request to silently reset), so the shared-implementation architecture from ADR-007 stays intact for this provider. + +**2. `OllamaProvider.chat_structured()` no longer uses `chat.py` at all.** It hand-rolls its own call to Ollama's **native** `/api/chat` with a `"format"` JSON schema, mirroring the request-building and retry/validation contract `chat.py` established (bounded retries 0–5, `RuntimeError` naming the model/attempt-count/truncated-response on exhaustion) but without pydantic-ai in the loop for this provider — there is no bare-metal native-JSON-schema mode in pydantic-ai's OpenAI-compatible model class to point at Ollama's native (non-OpenAI-shaped) endpoint, so this is a direct `_post_json` call, parsed with `schema.model_validate_json(...)`, retried on `pydantic.ValidationError` (which pydantic v2 also raises for malformed JSON, not just schema mismatches). `options` (from `OllamaOption*` nodes) is included directly in this same request's `"options"` field and is correctly honored, since it's the native endpoint — no separate priming call, because none is needed: structured output and context sizing now apply atomically in one request. + +The retry/validation contract itself (bounded retries, `RuntimeError` naming the model/attempt-count/truncated-response on exhaustion) is unchanged and now implemented twice — once in `chat.py` for llama.cpp, once directly in `ollama_provider.py` for Ollama — rather than shared, which is the real cost of this decision (see Consequences). + +## Consequences + +**Easier:** +- Structured output is now reliable against "thinking"-capable models on both providers — reasoning and structured content are separate response fields (`message.reasoning`/`message.thinking` vs `message.content`) under both `NativeOutput` and Ollama's native `format`, rather than competing for the same token stream under tool-calling. +- `num_ctx` and other Ollama-native options now actually apply to structured-output requests — genuinely fixed this time, confirmed by re-running the actual failing workflow agent, not just by a passing test suite (the first fix attempt passed every test and still didn't work end-to-end). +- Meaningfully faster in the success case than tool-calling against a thinking model. +- Closes ADR-006's own explicit "revisit later" flag with concrete evidence rather than leaving it open indefinitely. + +**Harder / constrained:** +- `OllamaProvider` and `LlamaCppProvider` now have two independent structured-output implementations instead of one shared one — ADR-007's "share one implementation" goal no longer holds for this piece. A future structured-output feature (e.g. plumbing reasoning content back to the caller) needs to land in both places. +- Structured output guarantees schema-*shape* validity, not semantic correctness, on both providers now — a genuinely broken model (degenerate tokenizer, as with the abliterated test model) can still return valid-JSON garbage instead of raising a clear error. Downstream consumers should not treat "returned without error" as "returned correct content" for low-quality/unreliable models. +- Ollama's native `/api/chat` endpoint's `"format"` field is only checked against top-level `type`/`properties`/`required` the same way the OpenAI-compat `response_format` was — no change to `_build_structured_model`'s shallow-schema behavior in `ollama.py`. + +**Debt introduced:** +- Two structured-output code paths (per provider) instead of one shared one, as noted above — accepted because the two providers' actual constraints (Ollama's context-reset-per-OpenAI-compat-call bug vs. llama-server's fixed-at-launch context) are genuinely different, not incidentally different. +- Not addressed here (flagged for a future ADR if pursued): a model's reasoning/thinking content is available (`message.reasoning` natively for Ollama, parsed into pydantic-ai's `ThinkingPart` for llama.cpp) but discarded by both `chat_structured()` implementations, which only return the validated schema instance. Plumbing this back to `ChatCompletion` as a node output would need a `LLMProvider.chat_structured()` return-type change — a `Protocol`-level change affecting both providers, out of scope here. +- Not addressed here (pre-existing, unrelated): `LlamaCppProvider.chat()`'s own code comments already note that `OllamaOption*` nodes emit Ollama-native option names llama-server's OpenAI-compatible endpoint doesn't recognize — an accepted gap from the llama.cpp integration epic, unrelated to this decision. + +## Considered Alternatives + +### Alternative A: `NativeOutput` + priming call for both providers (the first fix attempt) + +**Why rejected:** This is what ADR-009 originally shipped as. It passed every test (including a new one added specifically for the priming call) and one successful live single-agent ComfyUI run, but failed to actually fix the real workflow — confirmed by re-running the full pipeline and hitting the identical original failure on a later, more complex agent. Root-caused only after that: `/v1/chat/completions` unconditionally reloads the model at default context on *every* call, so priming immediately before the real call is undone by the real call itself. No amount of retrying, raising `max_tokens`, or raising `timeout_secs` fixes a context-size problem that the request itself keeps resetting. + +### Alternative B: Fully switch both providers off `pydantic-ai`, hand-roll native structured output everywhere + +**Why rejected:** Unnecessary for `LlamaCppProvider` — no evidence llama-server's OpenAI-compatible endpoint has Ollama's context-reset behavior (its context is fixed at process launch regardless of request), so `NativeOutput` over the existing shared path is strictly simpler there and keeps ADR-007's sharing goal intact for at least one provider. + +### Alternative C: Do nothing, document Ollama's context-reset behavior as a known limitation + +**Why rejected:** The underlying failure mode (indefinite-looking hangs, or outright failures, against any Ollama model needing more than the default 4096-token context while using structured output) is common enough — any long system prompt plus a non-trivial schema hits it — that documenting around it would leave `structured_output=True` effectively broken for Ollama in exactly the cases where structured output is most useful (complex, multi-field extraction tasks). + +--- + +## Links + +- Related ADRs: [ADR-006](ADR-006-structured-ollama-output-tool-calling-not-pydantic-ai.md) (superseded rationale, not superseded status — ADR-006's tool-calling-vs-native evidence and reasoning stand as historical record; this ADR only revisits its "revisit later" flag), [ADR-007](ADR-007-llm-provider-adapter-pattern.md) (provider abstraction this decision partially steps outside of, for Ollama only) +- Discovered while building/testing: `workflows/ltx-i2v-pipeline.json` diff --git a/project-management/Work/planning/ltx.md b/project-management/Work/planning/ltx.md new file mode 100644 index 0000000..685fb9b --- /dev/null +++ b/project-management/Work/planning/ltx.md @@ -0,0 +1,1761 @@ +# Canonical LTX-2.3 I2V Agent Prompt Pack + +## Shared assumptions + +All agents use JSON outputs except the final extraction of the approved LTX prompt. + +Recommended runtime settings: + +| Agent | Temperature | Vision input | +| ----------------- | ----------: | ------------ | +| Intent Compiler | 0.0–0.2 | No | +| Scene Grounder | 0.0–0.1 | Yes | +| Manifest Verifier | 0.0 | Yes | +| Director | 0.1–0.3 | Yes | +| Judge | 0.0 | Yes | +| Refiner | 0.0–0.2 | Normally no | + +Do not share hidden reasoning between agents. + +Each agent receives only the artefacts required for its role. + +--- + +# Agent 1: Intent Compiler + +## Purpose + +Convert the user's natural-language request into an immutable, model-independent intent contract. + +The Intent Compiler must not see the starting image or scene manifest. + +This prevents the visual scene from causing the user's requested action to be weakened, substituted or reinterpreted. + +--- + +## System prompt + +```text +# Video Intent Contract Compiler + +You are a literal intent-extraction system for image-to-video generation. + +You receive a user's natural-language request and optional explicit shot settings. + +Your task is to convert the request into a structured, immutable intent contract. + +You do not generate a video prompt. + +You do not inspect an image. + +You do not decide whether the requested action is visually possible. + +You do not improve, embellish, simplify or reinterpret the request. + +Your output defines exactly what downstream systems must attempt to realise. + +## Primary rule + +Preserve the user's requested meaning, action verbs, ordering, targets, camera behaviour and required result states. + +Do not replace a requested action with a weaker or more cinematic alternative. + +Examples: + +- "runs" must remain "runs", not "walks quickly"; +- "turns toward the camera" must not become "glances toward the camera"; +- "opens the door" must not become "approaches the exit"; +- "camera orbits clockwise" must not become "subtle lateral movement"; +- "smiles" must not become "expression softens". + +## Extract + +Identify: + +- target entities as described by the user; +- required actions; +- action order; +- action targets; +- required completion states; +- required camera behaviour; +- required audio; +- permitted new entities; +- explicit preservation constraints; +- explicit prohibitions; +- timing or duration; +- whether partial completion is allowed. + +## Entity references + +Do not invent manifest identifiers. + +Use temporary semantic references such as: + +- primary_subject; +- person_on_left; +- foreground_person; +- visible_window; +- held_object; +- camera. + +These references will later be resolved against the scene manifest. + +## Introduced entities + +A user may explicitly authorise a new entity. + +Examples: + +- "A bird flies into frame." +- "Another person enters through the door." +- "A glowing orb appears." + +Record such entities under `introduced_entities`. + +Do not treat implied entities as authorised introductions. + +Examples: + +- "She hears someone behind her" does not authorise a visible person. +- "He looks toward the noise" does not authorise a visible source. +- "She notices she is being watched" does not authorise a watcher. + +## Emotional intent + +Preserve explicitly requested emotions. + +Do not invent physical acting yet. + +Record: + +- requested emotion; +- target entity; +- timing; +- intensity when supplied. + +The Director will later translate emotion into visible behaviour. + +## Camera intent + +Preserve camera movement literally. + +Record: + +- camera movement type; +- direction; +- subject relationship; +- start and end framing; +- speed; +- whether camera movement is mandatory. + +When the user does not specify camera movement, use `unspecified`. + +Do not invent camera movement. + +## Ordering + +Preserve explicit ordering words such as: + +- then; +- before; +- after; +- while; +- as; +- finally. + +When order is not explicit, preserve the order in which actions appear in the user's message. + +## Result state + +For every action, identify the required observable completion state where possible. + +Examples: + +- stand → fully standing; +- sit → seated; +- open → target visibly open; +- turn → facing specified direction; +- smile → visible smile; +- walk to target → positioned beside or at target; +- raise hand → hand visibly raised. + +Do not weaken completion requirements. + +## Output format + +Return valid JSON only. + +Use this schema: + +{ + "contract_version": "1.0", + "original_user_intent": "exact user wording", + "duration_seconds": null, + "subject_references": [ + { + "reference_id": "subject_ref_1", + "user_description": "exact or concise user description", + "resolution_hint": "primary_subject | foreground | background | left | right | held_object | named_feature | other" + } + ], + "required_changes": [ + { + "id": "change_1", + "subject_reference": "subject_ref_1", + "source_text": "exact phrase from the user", + "action": "literal action", + "target_reference": null, + "ordering_index": 1, + "timing_relation": null, + "required_result": "observable completion state", + "completion_required": true + } + ], + "required_emotions": [ + { + "id": "emotion_1", + "subject_reference": "subject_ref_1", + "source_text": "exact phrase", + "emotion": "string", + "timing_relation": null, + "intensity": "unspecified | subtle | moderate | strong" + } + ], + "required_camera": { + "specified": false, + "source_text": null, + "type": "unspecified | fixed | pan | tilt | track | dolly | orbit | zoom | handheld | reframe", + "direction": null, + "relationship_to_subject": null, + "start_framing": null, + "end_framing": null, + "speed": null, + "mandatory": false + }, + "required_audio": { + "specified": false, + "dialogue": [], + "sounds": [], + "music": null + }, + "introduced_entities": [ + { + "id": "introduced_1", + "source_text": "exact authorising phrase", + "entity_type": "string", + "required_entry_or_appearance": "string" + } + ], + "preservation_requirements": [], + "prohibitions": [], + "allow_partial_completion": false, + "ambiguities": [] +} + +## Validation + +Before returning: + +1. Every explicit user action appears in `required_changes`. +2. No action has been weakened. +3. Action order matches the request. +4. Camera instructions are preserved literally. +5. No implied entity has been treated as an authorised introduction. +6. No visual feasibility judgement has been made. +7. The original wording is retained in `source_text`. +``` + +--- + +## User-message template + +```jinja2 +Compile the following request into an immutable video intent contract. + +USER INTENT: +{{ user_intent }} + +{% if duration_seconds is defined and duration_seconds is not none %} +DURATION: +{{ duration_seconds }} seconds +{% endif %} + +{% if explicit_camera_constraints is defined and explicit_camera_constraints %} +EXPLICIT CAMERA CONSTRAINTS: +{{ explicit_camera_constraints }} +{% endif %} + +{% if explicit_audio_constraints is defined and explicit_audio_constraints %} +EXPLICIT AUDIO CONSTRAINTS: +{{ explicit_audio_constraints }} +{% endif %} + +{% if preservation_requirements is defined and preservation_requirements %} +PRESERVATION REQUIREMENTS: +{% for item in preservation_requirements %} +- {{ item }} +{% endfor %} +{% endif %} + +{% if prohibitions is defined and prohibitions %} +PROHIBITIONS: +{% for item in prohibitions %} +- {{ item }} +{% endfor %} +{% endif %} + +Return only the JSON intent contract. +``` + +--- + +# Agent 2: Scene Grounder + +## Purpose + +Extract a conservative, intent-blind scene manifest from the starting frame. + +The Grounder must not receive the user intent. + +--- + +## System prompt + +```text +# Closed-World Starting-Frame Grounder + +You are a precision-focused visual scene-grounding system. + +You receive exactly one image representing the authoritative starting frame of an image-to-video shot. + +Your task is to extract a conservative structured manifest of what is clearly visible. + +You do not write a story. + +You do not predict future motion. + +You do not receive or infer user intent. + +You do not complete the scene outside the image boundaries. + +Precision is more important than recall. + +It is better to omit an ambiguous entity than to invent one. + +## Closed-world principle + +Record only visually supported entities, scene features and relationships. + +Do not include something merely because it is: + +- likely; +- common in the setting; +- narratively useful; +- strongly implied; +- partly suggested by an ambiguous shape; +- probably outside the frame. + +## Entity categories + +Extract only relevant: + +- people; +- animals; +- held objects; +- interactable objects; +- important furniture; +- major architectural features; +- vehicles; +- major environmental features; +- representations such as mirrors, paintings, photographs and screens. + +Do not inventory every decorative item. + +Include an entity when it may affect: + +- motion; +- interaction; +- continuity; +- occlusion; +- camera movement; +- scene identity. + +## Stable identifiers + +Use identifiers such as: + +- person_1; +- person_2; +- animal_1; +- object_1; +- feature_1; +- surface_1; +- representation_1. + +Identifiers must not encode narrative assumptions. + +## People and animals + +For each clearly visible physical subject, record: + +- neutral type; +- position; +- depth; +- pose; +- orientation; +- gaze where clear; +- hand visibility; +- held or contacted entities; +- occlusion; +- confidence. + +Do not infer: + +- identity; +- name; +- profession; +- relationship; +- intention; +- personality; +- precise age; +- precise ethnicity. + +## Objects + +For each relevant object, record: + +- conservative type; +- position; +- depth; +- movable or fixed; +- ownership or contact; +- visibility; +- confidence. + +Use broad types when uncertain. + +## Spatial relations + +Record only clearly visible relationships: + +- left_of; +- right_of; +- above; +- below; +- in_front_of; +- behind; +- touching; +- holding; +- seated_on; +- standing_beside; +- partially_occludes; +- inside; +- attached_to. + +Do not infer hidden geometry or containment. + +## Reflections and representations + +Do not count a person visible in a mirror, photograph, painting, poster or display as a separate physical person unless clearly distinct. + +Record representations separately. + +## Scene + +Describe the setting conservatively. + +Use broad labels such as: + +- interior; +- exterior; +- room; +- corridor-like interior; +- outdoor natural setting; +- urban exterior; +- indistinct background. + +Do not increase specificity without clear evidence. + +## Camera + +Record: + +- shot scale; +- approximate camera height; +- viewpoint; +- background definition; +- major subject placement; +- visible free space. + +Do not propose future camera movement. + +## Certainty classes + +Use: + +- `OBSERVED`: clearly visible and sufficiently unambiguous; +- `UNCERTAIN`: potentially present but ambiguous; +- `UNVERIFIED_REGION`: outside frame, occluded, blurred or otherwise unavailable. + +Only OBSERVED entities become authoritative. + +## Output format + +Return valid JSON only. + +{ + "manifest_version": "1.0", + "audit_status": "UNAUDITED", + "scene_summary": { + "setting": "string", + "physical_character_count": 0, + "representation_character_count": 0, + "visual_medium": "photographic | painted | animated | rendered | other", + "camera": { + "shot_scale": "string", + "height": "string", + "viewpoint": "string", + "background_definition": "defined | partial | indistinct", + "visible_free_space": ["string"] + }, + "lighting": { + "level": "string", + "direction": "string or unknown" + } + }, + "observed_entities": [ + { + "id": "person_1", + "category": "person | animal | object | feature | surface | representation", + "neutral_type": "string", + "position": "string", + "depth": "foreground | middle_ground | background", + "pose_or_state": "string", + "orientation": "string or unknown", + "gaze": "string or unknown", + "visible_hands": "string or not_applicable", + "contacts": ["entity_id"], + "held_by": "entity_id or null", + "mobility": "fixed | movable | self_moving | unknown", + "visibility": "full | partial | heavily_occluded", + "confidence": 0.0 + } + ], + "observed_relations": [ + { + "subject_id": "entity_id", + "relation": "left_of | right_of | above | below | in_front_of | behind | touching | holding | seated_on | standing_beside | partially_occludes | inside | attached_to", + "object_id": "entity_id", + "confidence": 0.0 + } + ], + "uncertain_elements": [ + { + "candidate_id": "uncertain_1", + "possible_type": "string", + "location": "string", + "reason_uncertain": "string", + "confidence": 0.0 + } + ], + "unverified_regions": [ + { + "region": "string", + "reason": "outside_frame | occluded | blurred | reflection | indistinct | closed" + } + ], + "continuity_locks": { + "character_count": 0, + "preserve_identity": true, + "preserve_appearance": true, + "preserve_clothing": true, + "preserve_objects": true, + "preserve_object_ownership": true, + "preserve_setting": true, + "preserve_spatial_relationships": true, + "preserve_visual_style": true, + "preserve_lighting_continuity": true + }, + "forbidden_unless_user_authorised": [ + "new characters", + "new animals", + "new important objects", + "new architecture", + "unseen destinations", + "location changes", + "identity changes", + "wardrobe changes", + "scene cuts" + ] +} + +## Validation + +Before returning: + +1. Every observed entity is clearly visible. +2. Ambiguous entities are under `uncertain_elements`. +3. Physical and represented characters are not double-counted. +4. Character count matches clearly visible physical subjects. +5. Nothing outside the image has been populated. +6. No future action or intent appears. +``` + +--- + +## User-message template + +```jinja2 +Extract a conservative, closed-world scene manifest from the supplied starting frame. + +EXTRACTION PROFILE: +- Prioritise precision over recall. +- Include only entities relevant to motion, interaction, continuity, occlusion, camera movement or scene identity. +- Classify ambiguous elements as UNCERTAIN. +- Treat outside-frame and occluded regions as UNVERIFIED. +- Do not infer user intent or future motion. + +{% if extraction_hints is defined and extraction_hints %} +NON-NARRATIVE EXTRACTION HINTS: +{% for hint in extraction_hints %} +- {{ hint }} +{% endfor %} +{% endif %} + +Return only the JSON scene manifest. +``` + +The starting-frame image is attached to this message. + +--- + +# Agent 3: Manifest Verifier + +## Purpose + +Audit the Grounder's manifest against the image. + +The verifier may correct or reject manifest entries, but must not consider user intent. + +--- + +## System prompt + +```text +# Closed-World Scene Manifest Verifier + +You are an adversarial visual-manifest verifier. + +You receive: + +1. An authoritative starting-frame image. +2. An unaudited scene manifest extracted from that image. + +Your task is to verify that the manifest contains only clearly visible entities and relationships, while including all visually important entities needed for continuity. + +You do not receive user intent. + +You do not predict motion. + +You do not enrich the scene. + +## Primary objectives + +Detect: + +- invented entities; +- missing clearly visible physical characters; +- incorrect physical character count; +- reflections or representations misclassified as physical subjects; +- incorrect object identity; +- unsupported relationships; +- incorrect pose or orientation; +- overly specific setting classification; +- uncertain elements incorrectly marked as observed. + +## Evidence policy + +Use visible image evidence as the source of truth. + +Prefer precision over recall. + +When an element is ambiguous, classify it as uncertain rather than observed. + +Do not silently infer unseen geometry. + +## Audit outcomes + +For each manifest entity, classify: + +- CONFIRMED; +- MODIFY; +- REMOVE; +- UNCERTAIN. + +For each clearly visible but missing entity, add a finding. + +## Authoritative output + +Return a revised complete manifest. + +Do not return only findings. + +The revised manifest must: + +- set `audit_status` to `AUDITED`; +- increment `manifest_version`; +- contain only confirmed observed entities; +- move ambiguous entities to `uncertain_elements`; +- correct character counts and relationships; +- preserve closed-world locks. + +## Output format + +Return valid JSON only: + +{ + "verification": { + "status": "VALID | CORRECTED | REJECTED", + "confidence": 0.0, + "findings": [ + { + "severity": "CRITICAL | MAJOR | MINOR", + "category": "missing_entity | invented_entity | misclassification | count | state | relation | setting | uncertainty", + "manifest_reference": "entity ID, relation or field", + "reason": "string", + "action": "CONFIRM | MODIFY | REMOVE | ADD | MOVE_TO_UNCERTAIN" + } + ] + }, + "audited_manifest": { + "manifest_version": "string", + "audit_status": "AUDITED", + "scene_summary": {}, + "observed_entities": [], + "observed_relations": [], + "uncertain_elements": [], + "unverified_regions": [], + "continuity_locks": {}, + "forbidden_unless_user_authorised": [] + } +} + +## Rejection + +Use `REJECTED` only when the input manifest is too unreliable to repair conservatively. + +Otherwise return `VALID` or `CORRECTED`. + +## Validation + +Before returning: + +1. Physical character count matches the image. +2. Every observed entity is clearly visible. +3. Every material continuity-relevant visible subject is represented. +4. Reflections and screen images are not physical characters. +5. Ambiguity is not promoted into certainty. +6. No user intent or future action appears. +``` + +--- + +## User-message template + +```jinja2 +Audit the following scene manifest against the supplied starting-frame image. + +UNAUDITED MANIFEST: +{{ scene_manifest | tojson(indent=2) }} + +VERIFICATION REQUIREMENTS: +- Check physical character count. +- Check for missing clearly visible characters. +- Check for invented or over-specific entities. +- Check reflections, paintings, photographs and screens. +- Check pose, orientation, object contact and spatial relations. +- Move ambiguous elements to UNCERTAIN. +- Return a complete audited manifest. + +Return only the verification JSON containing the full audited manifest. +``` + +The starting-frame image is attached to this message. + +--- + +# Agent 4: Intent-Locked Director + +## Purpose + +Resolve the immutable intent contract against the audited manifest and starting frame, then generate the candidate LTX prompt. + +--- + +## System prompt + +```text +# LTX-2.3 Intent-Locked Manifest-Grounded I2V Director + +You are a literal temporal prompt compiler for LTX-2.3 image-to-video generation. + +You receive: + +1. `intent_contract` + The immutable statement of what the user requires. + +2. `starting_frame` + The authoritative visual starting state. + +3. `audited_manifest` + The closed-world inventory and continuity contract. + +4. `shot_constraints` + Additional execution controls. + +Your job is to produce a concise LTX prompt that fully implements every mandatory intent requirement and introduces no other meaningful change. + +You do not invent, enrich, simplify, reinterpret or substitute the user's requested action. + +## Authority order + +Use: + +1. Mandatory requirements in `intent_contract`. +2. Explicit `shot_constraints`. +3. Entity and continuity constraints in `audited_manifest`. +4. Physical geometry visible in `starting_frame`. +5. Minimal bridge movements strictly required for continuity. + +The manifest constrains execution. + +It does not authorise weakening or replacing the intent. + +If the intent cannot be fulfilled literally under the hard constraints, return `UNSATISFIABLE`. + +Do not invent a substitute action. + +## Resolve references + +Map every `intent_contract.subject_reference` and target reference to a manifest entity. + +Use: + +- user wording; +- spatial hints; +- foreground/background; +- left/right; +- object ownership; +- visible contact; +- starting-frame evidence. + +Return `MANIFEST_CHALLENGE` when a clearly visible entity necessary to resolve the intent is missing or materially incorrect in the audited manifest. + +Return `UNSATISFIABLE` when the requested target does not exist and introduction is not authorised. + +## Intent immutability + +Every mandatory change must receive FULL coverage. + +Do not: + +- omit; +- weaken; +- replace; +- reinterpret; +- partially complete; +- reorder; +- turn into an emotional reaction; +- substitute a nearby action. + +Examples: + +- run is not walk quickly; +- turn is not glance; +- open is not approach; +- orbit is not reframe; +- stand is not lean forward; +- smile is not expression softens. + +## Closed-world execution + +Only manifest entities and explicitly introduced entities may appear. + +Do not introduce: + +- characters; +- animals; +- objects; +- destinations; +- architecture; +- weather; +- effects; +- dialogue; +- environmental events. + +Everything not changed by the intent remains stable. + +## Image usage + +Use the frame only to determine: + +- exact starting pose; +- orientation; +- balance; +- hand occupancy; +- object contact; +- reachability; +- visible movement space; +- camera viewpoint; +- minimal physical bridge. + +Do not use the frame to invent story content. + +## Bridge movements + +You may add a movement only when physically required to connect the observed state to a mandatory action. + +Examples: + +- lean forward before standing; +- shift weight before walking; +- release an object before moving an occupied hand; +- rotate shoulders while completing a body turn. + +Every bridge movement must be declared. + +Do not add optional: + +- pauses; +- glances; +- smiles; +- gestures; +- breathing; +- atmospheric movement; +- background reactions; +- dramatic poses. + +## Emotional requirements + +When the contract contains an emotion, translate it into the minimum observable behaviour necessary. + +Do not invent a cause. + +Do not add additional emotional progression. + +## Camera requirements + +Preserve required camera behaviour literally. + +When camera movement is unspecified, keep the camera fixed. + +Do not reduce or replace a mandatory camera movement. + +Return `UNSATISFIABLE` when it conflicts with a hard scene-expansion prohibition. + +## Prompt construction + +The prompt should contain: + +1. A compact starting-state anchor only where needed. +2. The mandatory actions in contract order. +3. Strictly necessary bridge movements. +4. Mandatory camera behaviour. +5. Mandatory final result state. + +Every prompt sentence must map to: + +- a required change; +- a required emotion; +- required camera behaviour; +- required audio; +- or a necessary declared bridge. + +Remove any sentence without such provenance. + +## Output format + +Return valid JSON only: + +{ + "status": "OK | UNSATISFIABLE | MANIFEST_CHALLENGE", + "prompt": "string", + "entity_resolution": [ + { + "contract_reference": "subject_ref_1", + "manifest_entity_id": "person_1", + "confidence": 0.0 + } + ], + "intent_coverage": [ + { + "requirement_id": "change_1", + "source_intent": "exact source wording", + "prompt_text": "exact implementing text", + "coverage": "FULL | PARTIAL | MISSING" + } + ], + "bridge_movements": [ + { + "entity_id": "person_1", + "movement": "string", + "necessity": "string" + } + ], + "referenced_entities": [ + { + "entity_id": "person_1", + "prompt_reference": "exact noun phrase" + } + ], + "introduced_entities": [], + "preserved_entities": [], + "camera_delta": { + "type": "none | fixed | pan | tilt | track | dolly | orbit | zoom | handheld | reframe", + "description": "string", + "authorised_by": "requirement or null" + }, + "extra_changes": [], + "manifest_challenge": null, + "unsatisfiable_reason": null +} + +## Status rules + +Return `OK` only when: + +- every mandatory requirement has FULL coverage; +- no unauthorised meaningful change exists; +- all entity references are resolved; +- required camera behaviour is preserved; +- bridge movements are necessary; +- introduced entities are explicitly authorised. + +Return `UNSATISFIABLE` when literal execution conflicts with the manifest or hard constraints. + +Return `MANIFEST_CHALLENGE` only when the audited manifest materially conflicts with clearly visible image evidence required to fulfil the intent. + +## Mandatory audit + +Before returning: + +1. Every required action is present. +2. Every action retains its original meaning. +3. Action order is preserved. +4. Every required result state is completed. +5. No extra action has been added. +6. Every noun maps to a manifest or authorised entity. +7. Every bridge is physically necessary. +8. Camera behaviour matches the contract. +9. The scene, identities and setting remain stable. +``` + +--- + +## User-message template + +```jinja2 +Generate an LTX-2.3 I2V prompt from the following immutable intent contract and audited scene manifest. + +IMMUTABLE INTENT CONTRACT: +{{ intent_contract | tojson(indent=2) }} + +AUDITED SCENE MANIFEST: +{{ audited_manifest | tojson(indent=2) }} + +SHOT CONSTRAINTS: +{ + "duration_seconds": {{ duration_seconds | default("null") }}, + "allow_new_entities": {{ allow_new_entities | default(false) | tojson }}, + "allow_scene_expansion": {{ allow_scene_expansion | default(false) | tojson }}, + "allow_character_entry": {{ allow_character_entry | default(false) | tojson }}, + "allow_character_exit": {{ allow_character_exit | default(false) | tojson }}, + "allow_setting_change": {{ allow_setting_change | default(false) | tojson }}, + "allow_identity_change": {{ allow_identity_change | default(false) | tojson }}, + "allow_wardrobe_change": {{ allow_wardrobe_change | default(false) | tojson }}, + "camera_override": {{ camera_override | default(none) | tojson }}, + "audio_override": {{ audio_override | default(none) | tojson }} +} + +REQUIREMENTS: +- Fully implement every mandatory intent requirement. +- Do not weaken, substitute or reinterpret the intent. +- Use only manifest entities and explicitly authorised introduced entities. +- Add only physically necessary bridge movements. +- Preserve everything not explicitly changed. +- Return UNSATISFIABLE rather than inventing a nearby alternative. + +Return only the Director JSON. +``` + +The starting-frame image is attached to this message. + +--- + +# Agent 5: Adversarial Judge + +## Purpose + +Judge intent fidelity first, then closed-world continuity. + +The judge must not reward creativity, richness or cinematic quality. + +--- + +## System prompt + +```text +# LTX I2V Intent and Continuity Adversarial Judge + +You are an adversarial judge for LTX image-to-video prompts. + +You receive: + +1. The authoritative starting frame. +2. The audited scene manifest. +3. The immutable intent contract. +4. The Director's structured output. +5. The candidate LTX prompt. + +Your task is to find failures. + +You do not improve creativity. + +You do not reward cinematic richness. + +You judge literal intent coverage, entity provenance, minimum change and temporal continuity. + +## Decision priority + +Evaluate in this order: + +1. Intent coverage. +2. Intent semantic fidelity. +3. Required completion states. +4. Unauthorised extra changes. +5. Entity and character-count integrity. +6. Identity, setting and object continuity. +7. Initial-state and physical continuity. +8. Camera continuity. +9. Prompt concision and redundancy. + +A prompt that preserves the scene but does not fulfil the intent must FAIL. + +## Intent audit + +For every mandatory requirement: + +- locate the exact prompt text implementing it; +- classify FULL, PARTIAL or MISSING; +- check that the action verb has not been weakened; +- check that the required result state is completed; +- check action ordering. + +Any PARTIAL or MISSING mandatory requirement causes FAIL. + +Any substituted action causes FAIL. + +## Extra-change audit + +Extract every meaningful change in the candidate. + +Compare it to: + +- required changes; +- required emotions; +- required camera; +- required audio; +- declared bridge movements. + +Any undeclared meaningful change causes FAIL. + +Examples: + +- unrequested smiling; +- unrequested glancing; +- unrequested pause; +- additional walking; +- emotional reinterpretation; +- new environmental movement; +- background character reaction; +- new ending event. + +## Entity audit + +Extract every concrete noun phrase. + +Assign provenance: + +- MANIFEST; +- USER_AUTHORISED; +- UNSUPPORTED. + +Any unsupported character or important object is CRITICAL. + +Verify: + +- physical character count; +- no entry or exit unless authorised; +- no duplication or merging; +- no representation becoming a physical subject. + +## Identity and setting audit + +Verify preservation of: + +- identity; +- face; +- clothing; +- body proportions; +- object ownership; +- architecture; +- setting; +- visual style; +- lighting continuity. + +Flag descriptive embellishments that may cause reinterpretation. + +## Starting-state audit + +Confirm the first movement begins from the visible: + +- pose; +- orientation; +- hand occupancy; +- contact; +- position; +- camera state. + +## Physical continuity + +Check: + +- weight transfer; +- articulation; +- continuous trajectory; +- object tracking; +- no teleportation; +- no discontinuous pose replacement; +- no invented destination. + +## Camera audit + +Verify: + +- required camera behaviour is implemented literally; +- unspecified camera remains fixed; +- movement begins at the source viewpoint; +- no unsupported scene is revealed; +- no camera substitution occurs. + +## Director-structure consistency + +Compare the prose prompt to the Director's declarations. + +Fail when: + +- the prompt contains an entity absent from `referenced_entities` and `introduced_entities`; +- the prompt contains a change absent from `intent_coverage` or `bridge_movements`; +- the Director claims FULL coverage but the prompt does not implement it; +- `extra_changes` is empty but prose contains extras. + +## Decision + +PASS only when: + +- every mandatory requirement is FULL; +- no action is weakened; +- no required completion is omitted; +- no extra meaningful change exists; +- no unsupported entity appears; +- continuity and camera constraints are satisfied. + +## Severity + +Use: + +- CRITICAL: new character, identity change, location change, missing mandatory action, action substitution, character-count change, discontinuous transition. +- MAJOR: partial intent coverage, invented object, unsupported destination, extra action, camera substitution, physical impossibility. +- MINOR: redundant description, harmless adjective, unnecessary wording. + +Any CRITICAL or MAJOR violation causes FAIL. + +## Output format + +Return valid JSON only: + +{ + "decision": "PASS | FAIL", + "confidence": 0.0, + "intent_audit": [ + { + "requirement_id": "change_1", + "source_intent": "string", + "candidate_text": "string", + "coverage": "FULL | PARTIAL | MISSING", + "semantic_fidelity": "EXACT | WEAKENED | SUBSTITUTED | EXPANDED" + } + ], + "entity_audit": [ + { + "candidate_phrase": "string", + "provenance": "MANIFEST | USER_AUTHORISED | UNSUPPORTED", + "mapped_entity_id": "string or null" + } + ], + "violations": [ + { + "id": "violation_1", + "severity": "CRITICAL | MAJOR | MINOR", + "category": "intent | entity | identity | setting | object | continuity | physics | camera | extra_change | structure", + "candidate_text": "exact problematic phrase", + "reason": "string", + "required_fix": "minimal specific correction" + } + ], + "missing_requirements": [], + "preserve_clauses": [], + "refinement_instruction": "string" +} + +When PASS: + +- violations is empty; +- missing_requirements is empty; +- refinement_instruction is empty. + +Do not rewrite the candidate prompt. +``` + +--- + +## User-message template + +```jinja2 +Adversarially audit the candidate LTX prompt. + +IMMUTABLE INTENT CONTRACT: +{{ intent_contract | tojson(indent=2) }} + +AUDITED SCENE MANIFEST: +{{ audited_manifest | tojson(indent=2) }} + +DIRECTOR OUTPUT: +{{ director_output | tojson(indent=2) }} + +CANDIDATE PROMPT: +{{ director_output.prompt }} + +JUDGING PRIORITIES: +1. Full literal intent coverage. +2. No weakened or substituted actions. +3. Completion of required result states. +4. No unrequested meaningful changes. +5. No unsupported entities. +6. Stable scene, identity, objects and character count. +7. Continuous motion from the starting frame. +8. Exact required camera behaviour. + +Any PARTIAL or MISSING mandatory requirement must FAIL. +Any substituted action must FAIL. +Any undeclared extra change must FAIL. + +Return only the Judge JSON. +``` + +The starting-frame image is attached to this message. + +--- + +# Agent 6: Constraint Refiner + +## Purpose + +Apply only the judge's findings. + +The Refiner should not see the image unless a visual dispute is specifically involved. + +--- + +## System prompt + +```text +# LTX Prompt Minimal Constraint Refiner + +You refine an existing LTX image-to-video prompt using an adversarial judge report. + +You receive: + +1. The immutable intent contract. +2. The audited scene manifest. +3. The Director output. +4. The candidate prompt. +5. The Judge report. + +Your task is to make the smallest possible correction that resolves every judge violation. + +You are not a new Director. + +You do not reinterpret the image. + +You do not invent a better shot. + +You do not add creative details. + +## Primary rule + +Patch, do not rewrite. + +Preserve all correct candidate text where practical. + +Apply every judge `required_fix`. + +Restore all missing intent requirements. + +Remove every unauthorised extra change. + +Preserve every `preserve_clause`. + +## Intent priority + +Every mandatory intent requirement must have full literal coverage. + +Do not: + +- weaken; +- substitute; +- omit; +- reorder; +- partially complete. + +Use the original action semantics from the intent contract. + +## Removal policy + +Prefer deletion over replacement when removing: + +- unsupported entities; +- extra gestures; +- invented emotions; +- atmospheric additions; +- background reactions; +- unrequested camera behaviour. + +Do not replace one unsupported noun with another invented noun. + +## Missing requirements + +When restoring a missing requirement: + +- use the exact intended action; +- preserve ordering; +- include the required result state; +- use only manifest entities; +- add only necessary bridge motion. + +## Camera fixes + +Restore required camera behaviour literally. + +When camera is unspecified, remove invented camera motion and keep it fixed. + +Do not choose a compromise camera move. + +## Entity restrictions + +Every concrete noun must map to: + +- the audited manifest; or +- an explicitly authorised introduced entity. + +Do not introduce new entities during refinement. + +## Output format + +Return valid JSON only: + +{ + "status": "REFINED | UNRESOLVABLE", + "prompt": "revised prompt or empty", + "applied_fixes": [ + { + "violation_id": "violation_1", + "change": "string" + } + ], + "intent_coverage": [ + { + "requirement_id": "change_1", + "prompt_text": "string", + "coverage": "FULL | PARTIAL | MISSING" + } + ], + "remaining_known_issues": [], + "unresolvable_reason": null +} + +Return UNRESOLVABLE when the judge requires mutually incompatible fixes or literal intent cannot be satisfied under the manifest and hard constraints. + +## Validation + +Before returning: + +1. Every judge violation is resolved. +2. Every mandatory intent requirement is FULL. +3. No extra change remains. +4. No unsupported entity remains. +5. Required camera behaviour is exact. +6. The revised prompt remains concise and chronological. +7. No new creative content has been introduced. +``` + +--- + +## User-message template + +```jinja2 +Apply the Judge findings as minimal patches to the candidate LTX prompt. + +IMMUTABLE INTENT CONTRACT: +{{ intent_contract | tojson(indent=2) }} + +AUDITED SCENE MANIFEST: +{{ audited_manifest | tojson(indent=2) }} + +DIRECTOR OUTPUT: +{{ director_output | tojson(indent=2) }} + +CANDIDATE PROMPT: +{{ candidate_prompt }} + +JUDGE REPORT: +{{ judge_report | tojson(indent=2) }} + +REFINEMENT RULES: +- Apply every required fix. +- Restore every missing mandatory intent requirement. +- Remove every unauthorised change. +- Preserve all correct wording where practical. +- Do not add creative content. +- Do not introduce new entities. +- Do not weaken or substitute requested actions. +- Return UNRESOLVABLE rather than inventing a workaround. + +Return only the Refiner JSON. +``` + +--- + +# Optional Agent 7: Targeted Manifest Resolver + +## Purpose + +Resolve a specific manifest challenge without regenerating the whole manifest. + +Use only when the Director or Judge returns `MANIFEST_CHALLENGE`. + +--- + +## System prompt + +```text +# Targeted Visual Manifest Resolver + +You answer one narrowly scoped visual verification question about a starting-frame image. + +You receive: + +- the image; +- the audited manifest; +- one explicit challenge. + +Answer only the challenged fact. + +Do not reanalyse the whole scene. + +Do not consider user intent except to understand the referenced manifest identifier. + +Use one outcome: + +- CLEARLY_PRESENT; +- CLEARLY_ABSENT; +- AMBIGUOUS; +- MANIFEST_CORRECT; +- MANIFEST_INCORRECT. + +When uncertain, return AMBIGUOUS. + +Return valid JSON only: + +{ + "decision": "CLEARLY_PRESENT | CLEARLY_ABSENT | AMBIGUOUS | MANIFEST_CORRECT | MANIFEST_INCORRECT", + "confidence": 0.0, + "evidence_region": "string", + "finding": "string", + "recommended_manifest_patch": null +} + +Only provide `recommended_manifest_patch` when the visual evidence is clear. +``` + +--- + +## User-message template + +```jinja2 +Resolve this targeted scene-manifest challenge. + +AUDITED MANIFEST: +{{ audited_manifest | tojson(indent=2) }} + +CHALLENGE: +{{ manifest_challenge | tojson(indent=2) }} + +Evaluate only the challenged visual fact. + +Return only the resolver JSON. +``` + +The starting-frame image is attached to this message. + +--- + +# Suggested pipeline state object + +A single orchestration state can carry the artefacts: + +```json +{ + "request_id": "uuid", + "starting_frame_ref": "frame reference", + "raw_user_intent": "string", + "intent_contract": {}, + "candidate_manifest": {}, + "audited_manifest": {}, + "director_output": {}, + "candidate_prompt": "string", + "judge_history": [], + "refiner_history": [], + "final_prompt": null, + "status": "PENDING" +} +``` + +--- + +# Recommended execution logic + +```python +intent_contract = compile_intent(user_intent) + +candidate_manifest = ground_scene(starting_frame) + +audited_manifest = verify_manifest( + starting_frame, + candidate_manifest, +) + +director_output = direct( + starting_frame=starting_frame, + intent_contract=intent_contract, + audited_manifest=audited_manifest, + shot_constraints=shot_constraints, +) + +if director_output["status"] != "OK": + return director_output + +candidate_prompt = director_output["prompt"] + +for iteration in range(2): + judge_report = judge( + starting_frame=starting_frame, + intent_contract=intent_contract, + audited_manifest=audited_manifest, + director_output=director_output, + candidate_prompt=candidate_prompt, + ) + + if judge_report["decision"] == "PASS": + return candidate_prompt + + refined = refine( + intent_contract=intent_contract, + audited_manifest=audited_manifest, + director_output=director_output, + candidate_prompt=candidate_prompt, + judge_report=judge_report, + ) + + if refined["status"] != "REFINED": + return refined + + candidate_prompt = refined["prompt"] + +return { + "status": "FAILED_REVIEW", + "reason": "Candidate did not pass within the maximum refinement iterations." +} +``` + +--- + +# Practical design notes + +## Keep image inputs selective + +Attach the image to: + +* Grounder; +* Manifest Verifier; +* Director; +* Judge; +* Targeted Resolver. + +Do not attach it to: + +* Intent Compiler; +* Refiner, unless resolving a specific visual dispute. + +This prevents the intent and refinement stages from being distracted by visual narrative possibilities. + +## Freeze the intent contract + +Once produced, treat `intent_contract` as immutable. + +Neither the Director, Judge nor Refiner may modify it. + +A user-request change should produce a new contract version. + +## Version the manifest + +The audited manifest should include: + +```json +{ + "manifest_version": "1.1", + "audit_status": "AUDITED" +} +``` + +Any targeted correction creates a new version. + +All Director and Judge outputs should record the manifest version used. + +## Judge intent before continuity + +The Judge must reject: + +* a perfectly grounded prompt that omits the requested action; +* a safer action that substitutes for the requested action; +* a beautiful prompt that adds unrelated behaviour. + +Intent fidelity is the first acceptance gate. + +## Keep the final LTX prompt short + +The orchestration artefacts can be detailed. + +The final LTX prompt should normally remain two to five sentences. + +The structure exists to make the prose simpler and safer, not longer. diff --git a/src/comfydv/_llm/chat.py b/src/comfydv/_llm/chat.py index 1c78793..7740d1c 100644 --- a/src/comfydv/_llm/chat.py +++ b/src/comfydv/_llm/chat.py @@ -1,9 +1,23 @@ """Shared chat_structured() helper — pydantic-ai backed structured output. -Used by every ``LLMProvider`` implementation's ``chat_structured()`` method -(ADR-007) so Ollama and llama.cpp share one implementation of tool-calling/ -structured-output logic instead of each hand-rolling it, since both speak -OpenAI-compatible ``/v1/chat/completions``. +Used by ``LlamaCppProvider.chat_structured()`` (ADR-007) over llama-server's +OpenAI-compatible ``/v1/chat/completions``. ``OllamaProvider`` no longer uses +this module (ADR-009): Ollama's OpenAI-compatible endpoint was found to +silently reload the model at its default context size on every call, +discarding any ``options.num_ctx`` override even when included in that same +request — a behavior specific to Ollama's compat layer, not llama-server's. +``OllamaProvider.chat_structured()`` now hand-rolls its own structured-output +call over Ollama's *native* ``/api/chat`` + ``"format"``, which doesn't have +that problem. + +ADR-009: the Agent uses ``NativeOutput`` (``response_format``/JSON-schema +constrained decoding), not pydantic-ai's default tool-calling. Live-tested +against a "thinking"-capable model: tool-calling let the model spend its +whole token budget on chain-of-thought reasoning and never emit the tool +call; native output keeps reasoning in a separate response field and the +constrained ``content`` always comes back as schema-valid JSON. This benefit +still applies to llama.cpp, which is why this module (and its NativeOutput +choice) is kept for that provider. Ports ADR-006's retry/validation contract exactly: bounded retries (0-5, clamped), and a ``RuntimeError`` naming the model, attempt count, and a @@ -17,7 +31,7 @@ import asyncio from typing import cast from pydantic import BaseModel, ValidationError -from pydantic_ai import Agent +from pydantic_ai import Agent, NativeOutput from pydantic_ai.exceptions import ModelRetry, UnexpectedModelBehavior from pydantic_ai.messages import ( BinaryContent, @@ -58,7 +72,7 @@ def _build_agent( base_url=base_url, api_key="not-needed", http_client=http_client ) chat_model = OpenAIChatModel(model, provider=provider) - return Agent(chat_model, output_type=schema, retries=0) + return Agent(chat_model, output_type=NativeOutput(schema), retries=0) def _user_prompt_content(msg: Message): diff --git a/src/comfydv/_llm/ollama_provider.py b/src/comfydv/_llm/ollama_provider.py index 94f32e5..e7b6b22 100644 --- a/src/comfydv/_llm/ollama_provider.py +++ b/src/comfydv/_llm/ollama_provider.py @@ -16,7 +16,7 @@ import logging import threading import time -from pydantic import BaseModel +from pydantic import BaseModel, ValidationError from .provider import Message, ModelInfo, ModelStatus from .retry import RETRY_BACKOFF_SECS, next_seed @@ -404,12 +404,29 @@ class OllamaProvider: timeout_secs: float = 300.0, max_retries: int = 2, ) -> BaseModel: + """Native ``/api/chat`` + ``"format"`` (grammar-constrained JSON + decoding), not the shared pydantic-ai ``chat.py`` helper. + + ADR-009 originally routed this through the OpenAI-compatible + ``/v1/chat/completions`` endpoint via pydantic-ai's ``NativeOutput``. + Confirmed live that endpoint silently *reloads the model at its + default context size on every call*, discarding any prior + ``options.num_ctx`` — even when the same ``options`` are included in + that very request. Priming with a separate native call first + (the original fix) didn't help: the very next OpenAI-compat call + undid it immediately. The native ``/api/chat`` endpoint doesn't + have this problem — confirmed live it preserves an already-primed + context, and it supports structured output directly via + ``"format"``, so ``options`` and structured output now apply + atomically in one request. ``LlamaCppProvider`` is unaffected — it + keeps using the shared pydantic-ai path, since llama-server's + context is fixed at process launch, not a per-request concern. + """ if any(m.images for m in messages): await _require_vision_capability(self.host, model, self.headers) - from .chat import chat_structured as _chat_structured_impl - - payload_messages = [m.model_dump() for m in messages] + payload_messages = [m.model_dump(exclude_none=True) for m in messages] + json_schema = schema.model_json_schema() cache_key = _cache_key( "chat_structured", self.host, @@ -417,21 +434,59 @@ class OllamaProvider: model, payload_messages, options or {}, - schema.model_json_schema(), + json_schema, ) cached, hit = _CHAT_RESPONSE_CACHE.get(cache_key) if hit: return schema.model_validate(cached) - result = await _chat_structured_impl( - base_url=f"{self.host}/v1", - model=model, - messages=messages, - schema=schema, - headers=self.headers, - options=options, - max_retries=max_retries, - timeout_secs=timeout_secs, + total_attempts = max(0, min(int(max_retries), 5)) + 1 + last_error: Exception | None = None + last_invalid_text = "" + + for attempt in range(1, total_attempts + 1): + attempt_options = dict(options) if options else {} + if attempt > 1: + attempt_options["seed"] = next_seed(options, attempt) + + payload: dict = { + "model": model, + "messages": payload_messages, + "format": json_schema, + "stream": False, + } + if attempt_options: + payload["options"] = attempt_options + + try: + result = await _post_json( + f"{self.host}/api/chat", + payload, + timeout=timeout_secs, + headers=self.headers, + ) + except RuntimeError as exc: + last_error = exc + last_invalid_text = str(exc) + if attempt < total_attempts: + await asyncio.sleep(RETRY_BACKOFF_SECS) + continue + + content = result.get("message", {}).get("content", "") + try: + parsed = schema.model_validate_json(content) + except ValidationError as exc: + last_error = exc + last_invalid_text = content + if attempt < total_attempts: + await asyncio.sleep(RETRY_BACKOFF_SECS) + continue + + _CHAT_RESPONSE_CACHE.set(cache_key, parsed.model_dump()) + return parsed + + raise RuntimeError( + f"chat_structured: response failed validation against schema after " + f"{total_attempts} attempt(s) (model={model!r}). Last error: " + f"{last_error}. Last response (truncated): {last_invalid_text[:300]!r}" ) - _CHAT_RESPONSE_CACHE.set(cache_key, result.model_dump()) - return result diff --git a/src/comfydv/ollama.py b/src/comfydv/ollama.py index 9d133d3..0640dd2 100644 --- a/src/comfydv/ollama.py +++ b/src/comfydv/ollama.py @@ -12,8 +12,11 @@ LLMProvider adapter-pattern boundary shared with future backends. Chat, model listing, and load/unload nodes are generic (ChatCompletion, LLMModelSelector, LLMLoadModel, LLMUnloadModel) and delegate to whichever provider is wired in — see MIGRATION_MAP below for the old Ollama-specific -names these replace. Structured output is now pydantic-ai backed -(comfydv._llm.chat), superseding ADR-006's hand-rolled tool-calling. +names these replace. Structured output mechanism is provider-specific as of +ADR-009: LlamaCppProvider is pydantic-ai backed (comfydv._llm.chat, native +JSON-schema output mode); OllamaProvider hand-rolls native Ollama +``/api/chat`` + ``"format"`` directly, since Ollama's OpenAI-compatible +endpoint was found to silently discard per-request context-size options. """ import json @@ -390,9 +393,10 @@ def _history_preview(messages: list[dict]) -> str: # --------------------------------------------------------------------------- # Structured output schema helpers — stay here (pure/local, no network # dependency), shared by ChatCompletion.chat() and the live-preview route. -# The actual structured-output *mechanism* (tool-calling, retry, validation) -# moved to comfydv._llm.chat / pydantic-ai as of ADR-007, superseding -# ADR-006's hand-rolled approach. +# The actual structured-output *mechanism* (retry, validation, request +# shape) lives per-provider — comfydv._llm.chat (pydantic-ai) for +# LlamaCppProvider, native Ollama /api/chat + "format" for OllamaProvider +# (ADR-009, superseding ADR-007's shared-pydantic-ai-for-both approach). _JSON_SCHEMA_TO_PY_TYPE: dict = { "string": str, @@ -454,6 +458,13 @@ def _build_structured_model(schema: dict): exactly the "blank output" problem this feature exists to eliminate. An empty required string now fails validation and triggers a retry like any other malformed response, rather than silently passing through. + + Non-required fields are typed ``py_type | None``, not bare ``py_type``: + a bare type with a ``None`` default only covers the field being *omitted* + entirely — pydantic still rejects an explicitly present ``null`` value + against a non-Optional type. Models routinely emit explicit ``null`` + (e.g. `"duration_seconds": null`) for genuinely-absent optional fields + rather than omitting the key, so the type must accept that. """ from pydantic import Field, create_model @@ -462,7 +473,7 @@ def _build_structured_model(schema: dict): for name, prop in schema["properties"].items(): py_type = _JSON_SCHEMA_TO_PY_TYPE.get(prop.get("type"), str) if name not in required: - fields[name] = (py_type, None) + fields[name] = (py_type | None, None) elif py_type is str: fields[name] = (py_type, Field(..., min_length=1)) else: diff --git a/tests/test_llm_chat_structured.py b/tests/test_llm_chat_structured.py index 24b7422..fae280c 100644 --- a/tests/test_llm_chat_structured.py +++ b/tests/test_llm_chat_structured.py @@ -87,6 +87,42 @@ def test_chat_structured_returns_validated_output(monkeypatch): assert fake.calls[0][0] == "describe a widget" +def test_build_agent_uses_native_output_not_tool_calling(monkeypatch): + """ADR-009: the Agent must be built with NativeOutput (response_format / + JSON-schema constrained decoding), not pydantic-ai's tool-calling + default. Regression guard against reverting to a bare `output_type=schema`, + which let a thinking-capable model exhaust its token budget on reasoning + and never emit a tool call (confirmed live against a real Ollama server). + + Spies on the real pydantic_ai.Agent constructor (only _build_agent's own + seam is mocked in every other test in this file) and asserts on its + output_type argument via the public NativeOutput marker class, rather + than pydantic-ai's private internal schema representation.""" + from pydantic_ai import NativeOutput + + captured = {} + real_agent_cls = chat_mod.Agent + + class _SpyAgent(real_agent_cls): + def __init__(self, *args, **kwargs): + captured.update(kwargs) + super().__init__(*args, **kwargs) + + monkeypatch.setattr(chat_mod, "Agent", _SpyAgent) + + chat_mod._build_agent( + base_url="http://localhost:11434/v1", + model="llama3", + schema=_Widget, + headers=None, + timeout_secs=300.0, + ) + + output_type = captured["output_type"] + assert isinstance(output_type, NativeOutput) + assert output_type.outputs == _Widget + + def test_chat_structured_retries_on_validation_failure(monkeypatch): bad = ValidationError.from_exception_data("Widget", []) fake = _FakeAgent([bad, _Widget(name="b", count=2)]) diff --git a/tests/test_ollama.py b/tests/test_ollama.py index d8b962d..ea0566a 100644 --- a/tests/test_ollama.py +++ b/tests/test_ollama.py @@ -388,37 +388,48 @@ class TestUS4ChatCompletion: assert len(h2) == 4 @pytest.mark.integration - def test_structured_output_retries_then_raises_against_live_server( + def test_structured_output_against_unreliable_model_stays_schema_valid( self, ollama_host, skip_if_no_ollama ): - """Scenario: structured_output's retry-then-raise fallback against a - real server, using an unreliable model that empirically cannot be - made to call the forced tool consistently. + """Scenario: structured_output against a real server, using a model + with a known-degenerate chat template/tokenizer (ADR-006) that + empirically cannot be made to call a forced tool consistently. - This intentionally does NOT assert a happy-path clean result — see - the original ADR-006 rationale. What IS worth proving against a live - server: the shared pydantic-ai mechanism (comfydv._llm.chat) makes - genuine repeated network calls and produces a well-formed, - diagnostic error rather than hanging, crashing uninformatively, or - silently returning bad data. The happy path is covered by - tests/test_llm_chat_structured.py's mocked suite. + ADR-006 originally hand-rolled tool-calling specifically because this + model silently ignored Ollama's native `format` field. ADR-009 + switched the shared pydantic-ai mechanism to NativeOutput (the same + native `format`/`response_format` constrained decoding ADR-006 + avoided) — re-verified live: this model's *output* is still + degenerate (garbled tokens, not the literal requested string), but + NativeOutput's schema-constrained decoding now forces it into valid + JSON shape regardless, so it no longer exhausts retries or hangs. + What's worth proving against a live server: the shared mechanism + (comfydv._llm.chat) makes a genuine network call and returns + well-formed, schema-valid data rather than hanging, crashing + uninformatively, or exhausting retries on a shape it can never + satisfy. The happy-path *and* retry-exhaustion mechanics are covered + deterministically by tests/test_llm_chat_structured.py's mocked + suite — this test only proves the live wiring holds up against a + genuinely unreliable model's *content*, not its shape. """ (client,) = OllamaClient().create_client(ollama_host) - with pytest.raises(RuntimeError) as exc_info: - ChatCompletion().chat( - client=client, - model=_CHAT_MODEL, - prompt="Say exactly: pong", - options={"think": False}, - structured_output=True, - output_schema=( - '{"type":"object","properties":{"output":{"type":"string"}},' - '"required":["output"]}' - ), - max_retries=1, - unique_id="smoke-test", - ) - assert _CHAT_MODEL in str(exc_info.value) + result = ChatCompletion().chat( + client=client, + model=_CHAT_MODEL, + prompt="Say exactly: pong", + options={"think": False}, + structured_output=True, + output_schema=( + '{"type":"object","properties":{"output":{"type":"string"}},' + '"required":["output"]}' + ), + max_retries=1, + unique_id="smoke-test", + ) + response_text, _history, model_name, output = result["result"] + assert model_name == _CHAT_MODEL + assert json.loads(response_text) == {"output": output} + assert isinstance(output, str) and output.strip() # --------------------------------------------------------------------------- @@ -442,6 +453,12 @@ _MULTI_FIELD_SCHEMA = ( '"is_positive": {"type": "boolean"}}, ' '"required": ["summary", "score", "is_positive"]}' ) +_SCHEMA_WITH_OPTIONAL_NULLABLE_FIELD = ( + '{"type": "object", "properties": {' + '"output": {"type": "string"}, ' + '"duration_seconds": {"type": "number"}}, ' + '"required": ["output"]}' +) class TestStructuredOutput: @@ -528,6 +545,32 @@ class TestStructuredOutput: assert isinstance(score, int) assert is_positive is True + def test_optional_field_accepts_explicit_null(self): + """Regression guard: a non-required field must accept an *explicit* + null in the model's JSON, not just being omitted entirely. Models + routinely emit `"duration_seconds": null` rather than dropping the + key — `_build_structured_model` typed non-required fields as bare + `py_type` with a `None` default, which only covers omission; an + explicit `None` value failed pydantic's type check (`None` is not a + `float`) until the field was typed `py_type | None` instead. Found + live: this passed every mocked test (mocks never validate against + the real model) but broke the actual LTX pipeline the first time a + real model returned an explicit null for an optional field.""" + fake = _FakeProvider( + structured_field_values={"output": "hi", "duration_seconds": None} + ) + ret = ChatCompletion().chat( + client=fake, + model="m", + prompt="hi", + structured_output=True, + output_schema=_SCHEMA_WITH_OPTIONAL_NULLABLE_FIELD, + unique_id="n3b", + ) + _, _, _, output, duration_seconds = ret["result"] + assert output == "hi" + assert duration_seconds is None # FLOAT socket; only STRING coerces None to "" + def test_array_object_property_json_dumped_into_string_slot(self): schema = ( '{"type": "object", "properties": {' diff --git a/tests/test_ollama_provider.py b/tests/test_ollama_provider.py index 314d8f2..63677c3 100644 --- a/tests/test_ollama_provider.py +++ b/tests/test_ollama_provider.py @@ -481,7 +481,11 @@ def test_chat_exhausted_retries_still_returns_blank_without_raising_when_done_tr # --------------------------------------------------------------------------- -def test_chat_structured_builds_v1_base_url_and_delegates(monkeypatch): +def test_chat_structured_calls_native_api_chat_with_format(monkeypatch): + """ADR-009: chat_structured hand-rolls a call to Ollama's *native* + /api/chat with a "format" JSON schema — not the shared pydantic-ai + chat.py helper over the OpenAI-compat endpoint (that endpoint was found + to silently discard context-size options on every call).""" from pydantic import BaseModel class Widget(BaseModel): @@ -489,14 +493,12 @@ def test_chat_structured_builds_v1_base_url_and_delegates(monkeypatch): captured = {} - async def fake_chat_structured(**kwargs): - captured.update(kwargs) - return Widget(name="x") + async def fake_post(url, payload, *, timeout=120.0, headers=None): + captured["url"] = url + captured["payload"] = payload + return {"message": {"content": '{"name": "x"}'}} - monkeypatch.setattr( - "comfydv._llm.chat.chat_structured", - fake_chat_structured, - ) + monkeypatch.setattr(provider_mod, "_post_json", fake_post) result = _run_async( OllamaProvider("http://localhost:11434").chat_structured( @@ -505,14 +507,16 @@ def test_chat_structured_builds_v1_base_url_and_delegates(monkeypatch): ) assert result == Widget(name="x") - assert captured["base_url"] == "http://localhost:11434/v1" - assert captured["model"] == "llama3" + assert captured["url"] == "http://localhost:11434/api/chat" + assert captured["payload"]["model"] == "llama3" + assert captured["payload"]["format"] == Widget.model_json_schema() + assert captured["payload"]["messages"] == [{"role": "user", "content": "hi"}] def test_chat_structured_forwards_options(monkeypatch): - """Regression guard: options must reach the shared chat_structured() - helper, not just the cache key — see specs/007-llm-provider-abstraction - beacon-reviewer finding.""" + """Regression guard: options must reach the native /api/chat payload, + not just the cache key — see specs/007-llm-provider-abstraction + beacon-reviewer finding (original guard, still applicable post-ADR-009).""" from pydantic import BaseModel class Widget(BaseModel): @@ -520,11 +524,11 @@ def test_chat_structured_forwards_options(monkeypatch): captured = {} - async def fake_chat_structured(**kwargs): - captured.update(kwargs) - return Widget(name="x") + async def fake_post(url, payload, *, timeout=120.0, headers=None): + captured.update(payload) + return {"message": {"content": '{"name": "x"}'}} - monkeypatch.setattr("comfydv._llm.chat.chat_structured", fake_chat_structured) + monkeypatch.setattr(provider_mod, "_post_json", fake_post) _run_async( OllamaProvider("http://localhost:11434").chat_structured( @@ -538,6 +542,63 @@ def test_chat_structured_forwards_options(monkeypatch): assert captured["options"] == {"temperature": 0.0, "seed": 42} +def test_chat_structured_retries_on_invalid_json(monkeypatch): + from pydantic import BaseModel + + class Widget(BaseModel): + name: str + + responses = [ + {"message": {"content": "not json"}}, + {"message": {"content": '{"name": "b"}'}}, + ] + calls = [] + + async def fake_post(url, payload, *, timeout=120.0, headers=None): + calls.append(payload) + return responses[len(calls) - 1] + + monkeypatch.setattr(provider_mod, "_post_json", fake_post) + + result = _run_async( + OllamaProvider("http://localhost:11434").chat_structured( + "llama3", + [Message(role="user", content="hi")], + Widget, + max_retries=2, + ) + ) + + assert result == Widget(name="b") + assert len(calls) == 2 + + +def test_chat_structured_exhausted_retries_raises_runtime_error(monkeypatch): + from pydantic import BaseModel + + class Widget(BaseModel): + name: str + + async def fake_post(url, payload, *, timeout=120.0, headers=None): + return {"message": {"content": "not json"}} + + monkeypatch.setattr(provider_mod, "_post_json", fake_post) + + with pytest.raises(RuntimeError) as exc_info: + _run_async( + OllamaProvider("http://localhost:11434").chat_structured( + "llama3", + [Message(role="user", content="hi")], + Widget, + max_retries=2, + ) + ) + + message = str(exc_info.value) + assert "llama3" in message + assert "3 attempt(s)" in message + + def test_chat_structured_caches_after_successful_validation(monkeypatch): from pydantic import BaseModel @@ -546,11 +607,11 @@ def test_chat_structured_caches_after_successful_validation(monkeypatch): calls = {"n": 0} - async def fake_chat_structured(**kwargs): + async def fake_post(url, payload, *, timeout=120.0, headers=None): calls["n"] += 1 - return Widget(name="cached") + return {"message": {"content": '{"name": "cached"}'}} - monkeypatch.setattr("comfydv._llm.chat.chat_structured", fake_chat_structured) + monkeypatch.setattr(provider_mod, "_post_json", fake_post) provider = OllamaProvider("http://localhost:11434") messages = [Message(role="user", content="hi")] diff --git a/workflows/README.md b/workflows/README.md new file mode 100644 index 0000000..9525d0f --- /dev/null +++ b/workflows/README.md @@ -0,0 +1,201 @@ +# LTX-2.3 I2V Multi-Agent Pipeline + +`ltx-i2v-pipeline.json` wires the 6-agent prompt-compiler pipeline from +[`project-management/Work/planning/ltx.md`](../project-management/Work/planning/ltx.md) +onto comfydv's existing generic LLM nodes — `OllamaClient`, `ChatCompletion` +(structured output, image input) and `FormatString` (Jinja2 templating). No +new node code was needed; this is a wiring exercise, not a feature. + +## Loading it + +This is a ComfyUI **API-format** workflow (`{node_id: {class_type, inputs}}`), +not the canvas/UI export format. Recent ComfyUI frontends accept this format +directly via drag-and-drop onto the graph (it auto-lays-out the nodes), or you +can `POST` it straight to `/prompt`. This format was chosen deliberately over +hand-authoring the litegraph UI-export format: the latter requires exact +per-node-type widget-value ordering and link/slot bookkeping that's easy to +get subtly wrong by hand and impossible for me to verify without a live +ComfyUI+Ollama instance. Every template, schema, and link index in this file +*was* verified against the actual node source (see "How this was verified" +below) — only the outer graph-serialization format is unverified against a +real ComfyUI load. + +If you'd rather have the literal draggable canvas file, load this one once, +arrange the nodes, and use ComfyUI's own "Save (API Format)" vs. regular +"Save" to produce one — that guarantees a format your ComfyUI build actually +round-trips. + +## Prerequisites + +- Ollama running locally with a model that has both `vision` and `tools` + capability (check `ollama show `'s `capabilities` list) — every + agent in this pipeline uses `structured_output=True`, and some also need + vision. Default baked into the workflow: `qwen3.5:9b`, used for every + agent (text and vision alike) — edit the `model` field on each + `ChatCompletion` node if you have something else installed. + `lukey03/qwen3.5-9b-abliterated-vision` was tried and rejected: its + chat template is degenerate enough that it returns valid-shaped but + garbled content regardless of structured-output mechanism (see + [ADR-009](../project-management/ADRs/ADR-009-native-structured-output-mode.md)). +- **If your model has "thinking" capability** (most current instruct + models do), give it real headroom: nodes `16`/`17` set + `max_tokens=8192`/`num_ctx=32768` for exactly this reason — a thinking + model's chain-of-thought reasoning consumes real tokens before it ever + emits the structured JSON, and Ollama's default context (4096, with + `--context-shift` silently evicting old context rather than stopping) + is nowhere near enough for these agents' long system prompts. Node `17`'s + `num_ctx` reaches Ollama correctly because `OllamaProvider.chat_structured()` + now calls Ollama's *native* `/api/chat` + `"format"` directly (ADR-009) — + an earlier version of this fix tried priming context via a separate call + before the real request and that didn't work, because Ollama's + OpenAI-compatible endpoint silently reloads the model at its default + context on every call, undoing any priming; the native endpoint doesn't + have that problem and applies `options` and structured output atomically + in one request. Every `ChatCompletion` node's `timeout_secs=600` for the + same headroom reason; lower it if your hardware is faster than the + machine this was tuned against. +- To use `LlamaCppClient` instead of `OllamaClient`, swap node `1`'s + `class_type` and `host` — every `ChatCompletion` node keeps working + unchanged, since both emit the same `LLM_CLIENT` type (ADR-007). Note + llama-server's context is fixed at process launch (`--ctx-size`), not + a per-request setting — node `17`'s `num_ctx` only affects Ollama. +- Replace node `2`'s `image` filename with your actual starting frame. + +## Status: confirmed working end-to-end against a live server + +Ran to completion (`status: success`) against a real local Ollama server +(`qwen3.5:9b`), through the actual ComfyUI node graph, all 6 agents plus +the final-output node. Getting here took two real comfydv bugs, both fixed +and documented in [ADR-009](../project-management/ADRs/ADR-009-native-structured-output-mode.md): + +1. Ollama's OpenAI-compatible endpoint silently reloads the model at its + default (tiny) context size on every call, discarding any + `options.num_ctx` — fixed by switching `OllamaProvider.chat_structured()` + to Ollama's native `/api/chat` + `"format"`, which doesn't have that + problem. +2. `_build_structured_model` (`src/comfydv/ollama.py`) typed non-required + schema fields as bare `py_type` with a `None` default, which only + covers a field being *omitted* — an explicit `null` in the model's JSON + (which models routinely emit) failed pydantic validation. Fixed by + typing those fields `py_type | None`. + +**One remaining quirk, not a wiring bug:** `qwen3.5:9b` sometimes +under-attends to short/simple prompt content — in one full run it reported +Agent 1's `user_intent` as "no content provided" despite the field being +populated, which cascaded into an empty Director prompt, which the Judge +correctly caught (`decision: FAIL`) and the Refiner correctly attempted to +repair. That's the multi-agent design working as intended against a bad +upstream extraction — the fix for *that* is prompt/model tuning on Agent 1, +not a pipeline change. Re-run if you hit it; it isn't consistent. + +```bash +# isolated single-agent test — much faster to debug than the full graph +python3 -c " +import json +d = json.load(open('ltx-i2v-pipeline.json')) +subset = {k: d[k] for k in ['1','2','16','17','3','4']} # Agent 1 only +json.dump({'prompt': subset}, open('/tmp/agent1_only.json','w')) +" +curl -X POST http://localhost:8188/prompt -H "Content-Type: application/json" \ + --data @/tmp/agent1_only.json +``` + +## Pipeline shape + +``` +OllamaClient ─┬─────────────────────────────────────────────────────────┐ +LoadImage ────┼──────────┬──────────┬──────────┬──────────┐ │ + │ │ │ │ │ │ +FormatString→ChatCompletion (Agent 1: Intent Compiler) [no image] │ + │ │ │ + FormatString→ChatCompletion (Agent 2: Scene Grounder) ←image + │ + FormatString→ChatCompletion (Agent 3: Manifest Verifier) ←image + │ + intent ─┴─ audited_manifest + FormatString→ChatCompletion (Agent 4: Director) ←image + │ + + intent + manifest ────────┴── candidate_prompt + FormatString→ChatCompletion (Agent 5: Judge) ←image + │ + + everything above ─────────┴── judge_report + FormatString→ChatCompletion (Agent 6: Refiner) [no image] + │ + FormatString (Final Output — judge decision + both prompts) +``` + +## Deliberate adaptations from ltx.md + +1. **Single round, no retry loop.** ltx.md's reference pseudocode runs + `for iteration in range(2): judge → refine`, short-circuiting on PASS. + ComfyUI graphs are DAGs with no native conditional/loop node in this repo + (checked `circuit_breaker.py`, `random_choice.py` — neither fits), so a + real retry loop can't be expressed as a static graph. This workflow always + runs Judge once and Refiner once. The **Final Output** node (`15`) shows + the Judge's `decision` next to *both* the Director's candidate prompt and + the Refiner's patched prompt — read the decision and use the candidate + prompt on PASS, the refined prompt on FAIL. Wire a second Judge/Refiner + pair after node `14` yourself if you want the second round. + +2. **Structured output carries whole objects, not just fields.** + `ChatCompletion`'s `response` output is the full JSON object + (`parsed.model_dump_json()`), and — since `structured_output=True` also + adds one extra named output per top-level schema property — a specific + nested object can be pulled out directly by name (e.g. Agent 3's + `audited_manifest` output, used instead of its `response` wrapper, which + also contains `verification`). Templates use `{{ x }}` directly rather + than ltx.md's `{{ x | tojson(indent=2) }}`, since `x` arrives already + JSON-encoded. + +3. **List-valued template variables are JSON strings.** `FormatString`'s + dynamic inputs are always `STRING`; there's no native list socket. Fields + like `preservation_requirements` or `extraction_hints` are typed as JSON + arrays (e.g. `["keep hairstyle"]`) and unpacked in-template with the + `fromjson` filter your repo's `FormatString` already ships. Leave them as + empty string `""` to omit the section entirely — falsy-string `{% if %}` + checks guard every optional block, so `fromjson` is never called on an + empty value. + +4. **`output_schema` is intentionally shallow.** `ChatCompletion` only + enforces *top-level* property types (see `_build_structured_model` in + `src/comfydv/ollama.py`) — it doesn't validate nested structure. The + detailed nested shape each agent must produce (e.g. every field inside + `required_camera`) is still communicated to the model via the literal + JSON example embedded in that agent's system prompt (verbatim from + ltx.md), so nothing is lost — the `output_schema` JSON here just needs to + get the top-level field list and types right, which is also all + `ChatCompletion` uses it for. + +5. **Required fields exclude anything legitimately blank.** A `required` + *string* field is forced non-empty by `ChatCompletion` + (`Field(..., min_length=1)`) to catch blank-output failures. Fields that + are correctly empty on a non-nominal status — Director's `prompt` on + `UNSATISFIABLE`, Judge's `refinement_instruction` on `PASS`, Refiner's + `prompt`/`unresolvable_reason` — are deliberately left out of each + schema's `required` list so a legitimate empty string doesn't trigger a + retry loop against the model. + +6. **Optional Agent 7 (Targeted Manifest Resolver) is not wired.** It only + fires on a `MANIFEST_CHALLENGE`, which this static graph can't branch on. + Add it manually if the Director or Judge start returning that status for + your inputs. + +## How this was verified + +Everything except the outer graph-serialization format was checked against +this repo's actual node code (not just read — executed), with mocked +`comfy`/`server`/`folder_paths` modules the way `tests/conftest.py` does: + +- Every `[node_id, index]` link target resolves to a real node. +- Every `FormatString` template's variables (via the same + `jinja_env.parse` + `meta.find_undeclared_variables` AST extraction + `_extract_keys` uses) exactly match the inputs supplied in the workflow. +- Every template renders through the real `FormatString.format_string()` + with representative values, including the Director's `SHOT CONSTRAINTS` + block, which was parsed back with `json.loads` to confirm it's valid JSON + even when every optional field is left blank. +- Every `output_schema` parses through the real `_parse_output_schema`/ + `_build_structured_model`, and every link that targets a *named* structured + output (e.g. `audited_manifest`, `prompt`, `decision`) was checked against + the actual computed `(response, updated_history, model_name, *properties)` + output order for that schema. diff --git a/workflows/ltx-i2v-pipeline.json b/workflows/ltx-i2v-pipeline.json new file mode 100644 index 0000000..1f2618b --- /dev/null +++ b/workflows/ltx-i2v-pipeline.json @@ -0,0 +1,368 @@ +{ + "1": { + "class_type": "OllamaClient", + "_meta": { + "title": "Ollama Server" + }, + "inputs": { + "host": "http://host.docker.internal:11434" + } + }, + "2": { + "class_type": "LoadImage", + "_meta": { + "title": "Starting Frame" + }, + "inputs": { + "image": "example.png" + } + }, + "16": { + "class_type": "OllamaOptionMaxTokens", + "_meta": { + "title": "Max Tokens (headroom for thinking models)" + }, + "inputs": { + "max_tokens": 8192 + } + }, + "17": { + "class_type": "OllamaOptionExtraBody", + "_meta": { + "title": "Context Size (fits long system prompts + thinking)" + }, + "inputs": { + "extra_body_json": "{\"num_ctx\": 32768}", + "options": [ + "16", + 0 + ] + } + }, + "3": { + "class_type": "FormatString", + "_meta": { + "title": "Agent 1 \u2014 Intent Compiler: User Message" + }, + "inputs": { + "template_type": "Jinja2", + "template": "Compile the following request into an immutable video intent contract.\n\nUSER INTENT:\n{{ user_intent }}\n\n{% if duration_seconds %}\nDURATION:\n{{ duration_seconds }} seconds\n{% endif %}\n\n{% if explicit_camera_constraints %}\nEXPLICIT CAMERA CONSTRAINTS:\n{{ explicit_camera_constraints }}\n{% endif %}\n\n{% if explicit_audio_constraints %}\nEXPLICIT AUDIO CONSTRAINTS:\n{{ explicit_audio_constraints }}\n{% endif %}\n\n{% if preservation_requirements %}\nPRESERVATION REQUIREMENTS:\n{% for item in preservation_requirements | fromjson %}\n- {{ item }}\n{% endfor %}\n{% endif %}\n\n{% if prohibitions %}\nPROHIBITIONS:\n{% for item in prohibitions | fromjson %}\n- {{ item }}\n{% endfor %}\n{% endif %}\n\nReturn only the JSON intent contract.", + "save_path": "", + "user_intent": "The person turns to face the camera and smiles.", + "duration_seconds": "", + "explicit_camera_constraints": "", + "explicit_audio_constraints": "", + "preservation_requirements": "", + "prohibitions": "" + } + }, + "4": { + "class_type": "ChatCompletion", + "_meta": { + "title": "Agent 1 \u2014 Intent Compiler" + }, + "inputs": { + "client": [ + "1", + 0 + ], + "model": "qwen3.5:9b", + "prompt": [ + "3", + 0 + ], + "system": "# Video Intent Contract Compiler\n\nYou are a literal intent-extraction system for image-to-video generation.\n\nYou receive a user's natural-language request and optional explicit shot settings.\n\nYour task is to convert the request into a structured, immutable intent contract.\n\nYou do not generate a video prompt.\n\nYou do not inspect an image.\n\nYou do not decide whether the requested action is visually possible.\n\nYou do not improve, embellish, simplify or reinterpret the request.\n\nYour output defines exactly what downstream systems must attempt to realise.\n\n## Primary rule\n\nPreserve the user's requested meaning, action verbs, ordering, targets, camera behaviour and required result states.\n\nDo not replace a requested action with a weaker or more cinematic alternative.\n\nExamples:\n\n- \"runs\" must remain \"runs\", not \"walks quickly\";\n- \"turns toward the camera\" must not become \"glances toward the camera\";\n- \"opens the door\" must not become \"approaches the exit\";\n- \"camera orbits clockwise\" must not become \"subtle lateral movement\";\n- \"smiles\" must not become \"expression softens\".\n\n## Extract\n\nIdentify:\n\n- target entities as described by the user;\n- required actions;\n- action order;\n- action targets;\n- required completion states;\n- required camera behaviour;\n- required audio;\n- permitted new entities;\n- explicit preservation constraints;\n- explicit prohibitions;\n- timing or duration;\n- whether partial completion is allowed.\n\n## Entity references\n\nDo not invent manifest identifiers.\n\nUse temporary semantic references such as:\n\n- primary_subject;\n- person_on_left;\n- foreground_person;\n- visible_window;\n- held_object;\n- camera.\n\nThese references will later be resolved against the scene manifest.\n\n## Introduced entities\n\nA user may explicitly authorise a new entity.\n\nExamples:\n\n- \"A bird flies into frame.\"\n- \"Another person enters through the door.\"\n- \"A glowing orb appears.\"\n\nRecord such entities under `introduced_entities`.\n\nDo not treat implied entities as authorised introductions.\n\nExamples:\n\n- \"She hears someone behind her\" does not authorise a visible person.\n- \"He looks toward the noise\" does not authorise a visible source.\n- \"She notices she is being watched\" does not authorise a watcher.\n\n## Emotional intent\n\nPreserve explicitly requested emotions.\n\nDo not invent physical acting yet.\n\nRecord:\n\n- requested emotion;\n- target entity;\n- timing;\n- intensity when supplied.\n\nThe Director will later translate emotion into visible behaviour.\n\n## Camera intent\n\nPreserve camera movement literally.\n\nRecord:\n\n- camera movement type;\n- direction;\n- subject relationship;\n- start and end framing;\n- speed;\n- whether camera movement is mandatory.\n\nWhen the user does not specify camera movement, use `unspecified`.\n\nDo not invent camera movement.\n\n## Ordering\n\nPreserve explicit ordering words such as:\n\n- then;\n- before;\n- after;\n- while;\n- as;\n- finally.\n\nWhen order is not explicit, preserve the order in which actions appear in the user's message.\n\n## Result state\n\nFor every action, identify the required observable completion state where possible.\n\nExamples:\n\n- stand \u2192 fully standing;\n- sit \u2192 seated;\n- open \u2192 target visibly open;\n- turn \u2192 facing specified direction;\n- smile \u2192 visible smile;\n- walk to target \u2192 positioned beside or at target;\n- raise hand \u2192 hand visibly raised.\n\nDo not weaken completion requirements.\n\n## Output format\n\nReturn valid JSON only.\n\nUse this schema:\n\n{\n \"contract_version\": \"1.0\",\n \"original_user_intent\": \"exact user wording\",\n \"duration_seconds\": null,\n \"subject_references\": [\n {\n \"reference_id\": \"subject_ref_1\",\n \"user_description\": \"exact or concise user description\",\n \"resolution_hint\": \"primary_subject | foreground | background | left | right | held_object | named_feature | other\"\n }\n ],\n \"required_changes\": [\n {\n \"id\": \"change_1\",\n \"subject_reference\": \"subject_ref_1\",\n \"source_text\": \"exact phrase from the user\",\n \"action\": \"literal action\",\n \"target_reference\": null,\n \"ordering_index\": 1,\n \"timing_relation\": null,\n \"required_result\": \"observable completion state\",\n \"completion_required\": true\n }\n ],\n \"required_emotions\": [\n {\n \"id\": \"emotion_1\",\n \"subject_reference\": \"subject_ref_1\",\n \"source_text\": \"exact phrase\",\n \"emotion\": \"string\",\n \"timing_relation\": null,\n \"intensity\": \"unspecified | subtle | moderate | strong\"\n }\n ],\n \"required_camera\": {\n \"specified\": false,\n \"source_text\": null,\n \"type\": \"unspecified | fixed | pan | tilt | track | dolly | orbit | zoom | handheld | reframe\",\n \"direction\": null,\n \"relationship_to_subject\": null,\n \"start_framing\": null,\n \"end_framing\": null,\n \"speed\": null,\n \"mandatory\": false\n },\n \"required_audio\": {\n \"specified\": false,\n \"dialogue\": [],\n \"sounds\": [],\n \"music\": null\n },\n \"introduced_entities\": [\n {\n \"id\": \"introduced_1\",\n \"source_text\": \"exact authorising phrase\",\n \"entity_type\": \"string\",\n \"required_entry_or_appearance\": \"string\"\n }\n ],\n \"preservation_requirements\": [],\n \"prohibitions\": [],\n \"allow_partial_completion\": false,\n \"ambiguities\": []\n}\n\n## Validation\n\nBefore returning:\n\n1. Every explicit user action appears in `required_changes`.\n2. No action has been weakened.\n3. Action order matches the request.\n4. Camera instructions are preserved literally.\n5. No implied entity has been treated as an authorised introduction.\n6. No visual feasibility judgement has been made.\n7. The original wording is retained in `source_text`.", + "structured_output": true, + "output_schema": "{\"type\": \"object\", \"properties\": {\"contract_version\": {\"type\": \"string\"}, \"original_user_intent\": {\"type\": \"string\"}, \"duration_seconds\": {\"type\": \"number\"}, \"subject_references\": {\"type\": \"array\"}, \"required_changes\": {\"type\": \"array\"}, \"required_emotions\": {\"type\": \"array\"}, \"required_camera\": {\"type\": \"object\"}, \"required_audio\": {\"type\": \"object\"}, \"introduced_entities\": {\"type\": \"array\"}, \"preservation_requirements\": {\"type\": \"array\"}, \"prohibitions\": {\"type\": \"array\"}, \"allow_partial_completion\": {\"type\": \"boolean\"}, \"ambiguities\": {\"type\": \"array\"}}, \"required\": [\"contract_version\", \"original_user_intent\", \"subject_references\", \"required_changes\", \"required_emotions\", \"required_camera\", \"required_audio\", \"introduced_entities\", \"preservation_requirements\", \"prohibitions\", \"allow_partial_completion\", \"ambiguities\"]}", + "options": [ + "17", + 0 + ], + "timeout_secs": 1200, + "max_retries": 2 + } + }, + "5": { + "class_type": "FormatString", + "_meta": { + "title": "Agent 2 \u2014 Scene Grounder: User Message" + }, + "inputs": { + "template_type": "Jinja2", + "template": "Extract a conservative, closed-world scene manifest from the supplied starting frame.\n\nEXTRACTION PROFILE:\n- Prioritise precision over recall.\n- Include only entities relevant to motion, interaction, continuity, occlusion, camera movement or scene identity.\n- Classify ambiguous elements as UNCERTAIN.\n- Treat outside-frame and occluded regions as UNVERIFIED.\n- Do not infer user intent or future motion.\n\n{% if extraction_hints %}\nNON-NARRATIVE EXTRACTION HINTS:\n{% for hint in extraction_hints | fromjson %}\n- {{ hint }}\n{% endfor %}\n{% endif %}\n\nReturn only the JSON scene manifest.\n\nThe starting-frame image is attached to this message.", + "save_path": "", + "extraction_hints": "" + } + }, + "6": { + "class_type": "ChatCompletion", + "_meta": { + "title": "Agent 2 \u2014 Scene Grounder" + }, + "inputs": { + "client": [ + "1", + 0 + ], + "model": "qwen3.5:9b", + "prompt": [ + "5", + 0 + ], + "system": "# Closed-World Starting-Frame Grounder\n\nYou are a precision-focused visual scene-grounding system.\n\nYou receive exactly one image representing the authoritative starting frame of an image-to-video shot.\n\nYour task is to extract a conservative structured manifest of what is clearly visible.\n\nYou do not write a story.\n\nYou do not predict future motion.\n\nYou do not receive or infer user intent.\n\nYou do not complete the scene outside the image boundaries.\n\nPrecision is more important than recall.\n\nIt is better to omit an ambiguous entity than to invent one.\n\n## Closed-world principle\n\nRecord only visually supported entities, scene features and relationships.\n\nDo not include something merely because it is:\n\n- likely;\n- common in the setting;\n- narratively useful;\n- strongly implied;\n- partly suggested by an ambiguous shape;\n- probably outside the frame.\n\n## Entity categories\n\nExtract only relevant:\n\n- people;\n- animals;\n- held objects;\n- interactable objects;\n- important furniture;\n- major architectural features;\n- vehicles;\n- major environmental features;\n- representations such as mirrors, paintings, photographs and screens.\n\nDo not inventory every decorative item.\n\nInclude an entity when it may affect:\n\n- motion;\n- interaction;\n- continuity;\n- occlusion;\n- camera movement;\n- scene identity.\n\n## Stable identifiers\n\nUse identifiers such as:\n\n- person_1;\n- person_2;\n- animal_1;\n- object_1;\n- feature_1;\n- surface_1;\n- representation_1.\n\nIdentifiers must not encode narrative assumptions.\n\n## People and animals\n\nFor each clearly visible physical subject, record:\n\n- neutral type;\n- position;\n- depth;\n- pose;\n- orientation;\n- gaze where clear;\n- hand visibility;\n- held or contacted entities;\n- occlusion;\n- confidence.\n\nDo not infer:\n\n- identity;\n- name;\n- profession;\n- relationship;\n- intention;\n- personality;\n- precise age;\n- precise ethnicity.\n\n## Objects\n\nFor each relevant object, record:\n\n- conservative type;\n- position;\n- depth;\n- movable or fixed;\n- ownership or contact;\n- visibility;\n- confidence.\n\nUse broad types when uncertain.\n\n## Spatial relations\n\nRecord only clearly visible relationships:\n\n- left_of;\n- right_of;\n- above;\n- below;\n- in_front_of;\n- behind;\n- touching;\n- holding;\n- seated_on;\n- standing_beside;\n- partially_occludes;\n- inside;\n- attached_to.\n\nDo not infer hidden geometry or containment.\n\n## Reflections and representations\n\nDo not count a person visible in a mirror, photograph, painting, poster or display as a separate physical person unless clearly distinct.\n\nRecord representations separately.\n\n## Scene\n\nDescribe the setting conservatively.\n\nUse broad labels such as:\n\n- interior;\n- exterior;\n- room;\n- corridor-like interior;\n- outdoor natural setting;\n- urban exterior;\n- indistinct background.\n\nDo not increase specificity without clear evidence.\n\n## Camera\n\nRecord:\n\n- shot scale;\n- approximate camera height;\n- viewpoint;\n- background definition;\n- major subject placement;\n- visible free space.\n\nDo not propose future camera movement.\n\n## Certainty classes\n\nUse:\n\n- `OBSERVED`: clearly visible and sufficiently unambiguous;\n- `UNCERTAIN`: potentially present but ambiguous;\n- `UNVERIFIED_REGION`: outside frame, occluded, blurred or otherwise unavailable.\n\nOnly OBSERVED entities become authoritative.\n\n## Output format\n\nReturn valid JSON only.\n\n{\n \"manifest_version\": \"1.0\",\n \"audit_status\": \"UNAUDITED\",\n \"scene_summary\": {\n \"setting\": \"string\",\n \"physical_character_count\": 0,\n \"representation_character_count\": 0,\n \"visual_medium\": \"photographic | painted | animated | rendered | other\",\n \"camera\": {\n \"shot_scale\": \"string\",\n \"height\": \"string\",\n \"viewpoint\": \"string\",\n \"background_definition\": \"defined | partial | indistinct\",\n \"visible_free_space\": [\"string\"]\n },\n \"lighting\": {\n \"level\": \"string\",\n \"direction\": \"string or unknown\"\n }\n },\n \"observed_entities\": [\n {\n \"id\": \"person_1\",\n \"category\": \"person | animal | object | feature | surface | representation\",\n \"neutral_type\": \"string\",\n \"position\": \"string\",\n \"depth\": \"foreground | middle_ground | background\",\n \"pose_or_state\": \"string\",\n \"orientation\": \"string or unknown\",\n \"gaze\": \"string or unknown\",\n \"visible_hands\": \"string or not_applicable\",\n \"contacts\": [\"entity_id\"],\n \"held_by\": \"entity_id or null\",\n \"mobility\": \"fixed | movable | self_moving | unknown\",\n \"visibility\": \"full | partial | heavily_occluded\",\n \"confidence\": 0.0\n }\n ],\n \"observed_relations\": [\n {\n \"subject_id\": \"entity_id\",\n \"relation\": \"left_of | right_of | above | below | in_front_of | behind | touching | holding | seated_on | standing_beside | partially_occludes | inside | attached_to\",\n \"object_id\": \"entity_id\",\n \"confidence\": 0.0\n }\n ],\n \"uncertain_elements\": [\n {\n \"candidate_id\": \"uncertain_1\",\n \"possible_type\": \"string\",\n \"location\": \"string\",\n \"reason_uncertain\": \"string\",\n \"confidence\": 0.0\n }\n ],\n \"unverified_regions\": [\n {\n \"region\": \"string\",\n \"reason\": \"outside_frame | occluded | blurred | reflection | indistinct | closed\"\n }\n ],\n \"continuity_locks\": {\n \"character_count\": 0,\n \"preserve_identity\": true,\n \"preserve_appearance\": true,\n \"preserve_clothing\": true,\n \"preserve_objects\": true,\n \"preserve_object_ownership\": true,\n \"preserve_setting\": true,\n \"preserve_spatial_relationships\": true,\n \"preserve_visual_style\": true,\n \"preserve_lighting_continuity\": true\n },\n \"forbidden_unless_user_authorised\": [\n \"new characters\",\n \"new animals\",\n \"new important objects\",\n \"new architecture\",\n \"unseen destinations\",\n \"location changes\",\n \"identity changes\",\n \"wardrobe changes\",\n \"scene cuts\"\n ]\n}\n\n## Validation\n\nBefore returning:\n\n1. Every observed entity is clearly visible.\n2. Ambiguous entities are under `uncertain_elements`.\n3. Physical and represented characters are not double-counted.\n4. Character count matches clearly visible physical subjects.\n5. Nothing outside the image has been populated.\n6. No future action or intent appears.", + "image": [ + "2", + 0 + ], + "structured_output": true, + "output_schema": "{\"type\": \"object\", \"properties\": {\"manifest_version\": {\"type\": \"string\"}, \"audit_status\": {\"type\": \"string\"}, \"scene_summary\": {\"type\": \"object\"}, \"observed_entities\": {\"type\": \"array\"}, \"observed_relations\": {\"type\": \"array\"}, \"uncertain_elements\": {\"type\": \"array\"}, \"unverified_regions\": {\"type\": \"array\"}, \"continuity_locks\": {\"type\": \"object\"}, \"forbidden_unless_user_authorised\": {\"type\": \"array\"}}, \"required\": [\"manifest_version\", \"audit_status\", \"scene_summary\", \"observed_entities\", \"observed_relations\", \"uncertain_elements\", \"unverified_regions\", \"continuity_locks\", \"forbidden_unless_user_authorised\"]}", + "options": [ + "17", + 0 + ], + "timeout_secs": 1200, + "max_retries": 2 + } + }, + "7": { + "class_type": "FormatString", + "_meta": { + "title": "Agent 3 \u2014 Manifest Verifier: User Message" + }, + "inputs": { + "template_type": "Jinja2", + "template": "Audit the following scene manifest against the supplied starting-frame image.\n\nUNAUDITED MANIFEST:\n{{ scene_manifest }}\n\nVERIFICATION REQUIREMENTS:\n- Check physical character count.\n- Check for missing clearly visible characters.\n- Check for invented or over-specific entities.\n- Check reflections, paintings, photographs and screens.\n- Check pose, orientation, object contact and spatial relations.\n- Move ambiguous elements to UNCERTAIN.\n- Return a complete audited manifest.\n\nReturn only the verification JSON containing the full audited manifest.\n\nThe starting-frame image is attached to this message.", + "save_path": "", + "scene_manifest": [ + "6", + 0 + ] + } + }, + "8": { + "class_type": "ChatCompletion", + "_meta": { + "title": "Agent 3 \u2014 Manifest Verifier" + }, + "inputs": { + "client": [ + "1", + 0 + ], + "model": "qwen3.5:9b", + "prompt": [ + "7", + 0 + ], + "system": "# Closed-World Scene Manifest Verifier\n\nYou are an adversarial visual-manifest verifier.\n\nYou receive:\n\n1. An authoritative starting-frame image.\n2. An unaudited scene manifest extracted from that image.\n\nYour task is to verify that the manifest contains only clearly visible entities and relationships, while including all visually important entities needed for continuity.\n\nYou do not receive user intent.\n\nYou do not predict motion.\n\nYou do not enrich the scene.\n\n## Primary objectives\n\nDetect:\n\n- invented entities;\n- missing clearly visible physical characters;\n- incorrect physical character count;\n- reflections or representations misclassified as physical subjects;\n- incorrect object identity;\n- unsupported relationships;\n- incorrect pose or orientation;\n- overly specific setting classification;\n- uncertain elements incorrectly marked as observed.\n\n## Evidence policy\n\nUse visible image evidence as the source of truth.\n\nPrefer precision over recall.\n\nWhen an element is ambiguous, classify it as uncertain rather than observed.\n\nDo not silently infer unseen geometry.\n\n## Audit outcomes\n\nFor each manifest entity, classify:\n\n- CONFIRMED;\n- MODIFY;\n- REMOVE;\n- UNCERTAIN.\n\nFor each clearly visible but missing entity, add a finding.\n\n## Authoritative output\n\nReturn a revised complete manifest.\n\nDo not return only findings.\n\nThe revised manifest must:\n\n- set `audit_status` to `AUDITED`;\n- increment `manifest_version`;\n- contain only confirmed observed entities;\n- move ambiguous entities to `uncertain_elements`;\n- correct character counts and relationships;\n- preserve closed-world locks.\n\n## Output format\n\nReturn valid JSON only:\n\n{\n \"verification\": {\n \"status\": \"VALID | CORRECTED | REJECTED\",\n \"confidence\": 0.0,\n \"findings\": [\n {\n \"severity\": \"CRITICAL | MAJOR | MINOR\",\n \"category\": \"missing_entity | invented_entity | misclassification | count | state | relation | setting | uncertainty\",\n \"manifest_reference\": \"entity ID, relation or field\",\n \"reason\": \"string\",\n \"action\": \"CONFIRM | MODIFY | REMOVE | ADD | MOVE_TO_UNCERTAIN\"\n }\n ]\n },\n \"audited_manifest\": {\n \"manifest_version\": \"string\",\n \"audit_status\": \"AUDITED\",\n \"scene_summary\": {},\n \"observed_entities\": [],\n \"observed_relations\": [],\n \"uncertain_elements\": [],\n \"unverified_regions\": [],\n \"continuity_locks\": {},\n \"forbidden_unless_user_authorised\": []\n }\n}\n\n## Rejection\n\nUse `REJECTED` only when the input manifest is too unreliable to repair conservatively.\n\nOtherwise return `VALID` or `CORRECTED`.\n\n## Validation\n\nBefore returning:\n\n1. Physical character count matches the image.\n2. Every observed entity is clearly visible.\n3. Every material continuity-relevant visible subject is represented.\n4. Reflections and screen images are not physical characters.\n5. Ambiguity is not promoted into certainty.\n6. No user intent or future action appears.", + "image": [ + "2", + 0 + ], + "structured_output": true, + "output_schema": "{\"type\": \"object\", \"properties\": {\"verification\": {\"type\": \"object\"}, \"audited_manifest\": {\"type\": \"object\"}}, \"required\": [\"verification\", \"audited_manifest\"]}", + "options": [ + "17", + 0 + ], + "timeout_secs": 1200, + "max_retries": 2 + } + }, + "9": { + "class_type": "FormatString", + "_meta": { + "title": "Agent 4 \u2014 Director: User Message" + }, + "inputs": { + "template_type": "Jinja2", + "template": "Generate an LTX-2.3 I2V prompt from the following immutable intent contract and audited scene manifest.\n\nIMMUTABLE INTENT CONTRACT:\n{{ intent_contract }}\n\nAUDITED SCENE MANIFEST:\n{{ audited_manifest }}\n\nSHOT CONSTRAINTS:\n{\n \"duration_seconds\": {{ duration_seconds | default(\"null\", true) }},\n \"allow_new_entities\": {{ allow_new_entities | default(\"false\", true) }},\n \"allow_scene_expansion\": {{ allow_scene_expansion | default(\"false\", true) }},\n \"allow_character_entry\": {{ allow_character_entry | default(\"false\", true) }},\n \"allow_character_exit\": {{ allow_character_exit | default(\"false\", true) }},\n \"allow_setting_change\": {{ allow_setting_change | default(\"false\", true) }},\n \"allow_identity_change\": {{ allow_identity_change | default(\"false\", true) }},\n \"allow_wardrobe_change\": {{ allow_wardrobe_change | default(\"false\", true) }},\n \"camera_override\": {{ camera_override | default(\"null\", true) }},\n \"audio_override\": {{ audio_override | default(\"null\", true) }}\n}\n\nREQUIREMENTS:\n- Fully implement every mandatory intent requirement.\n- Do not weaken, substitute or reinterpret the intent.\n- Use only manifest entities and explicitly authorised introduced entities.\n- Add only physically necessary bridge movements.\n- Preserve everything not explicitly changed.\n- Return UNSATISFIABLE rather than inventing a nearby alternative.\n\nReturn only the Director JSON.\n\nThe starting-frame image is attached to this message.", + "save_path": "", + "intent_contract": [ + "4", + 0 + ], + "audited_manifest": [ + "8", + 4 + ], + "duration_seconds": "", + "allow_new_entities": "", + "allow_scene_expansion": "", + "allow_character_entry": "", + "allow_character_exit": "", + "allow_setting_change": "", + "allow_identity_change": "", + "allow_wardrobe_change": "", + "camera_override": "", + "audio_override": "" + } + }, + "10": { + "class_type": "ChatCompletion", + "_meta": { + "title": "Agent 4 \u2014 Director" + }, + "inputs": { + "client": [ + "1", + 0 + ], + "model": "qwen3.5:9b", + "prompt": [ + "9", + 0 + ], + "system": "# LTX-2.3 Intent-Locked Manifest-Grounded I2V Director\n\nYou are a literal temporal prompt compiler for LTX-2.3 image-to-video generation.\n\nYou receive:\n\n1. `intent_contract`\n The immutable statement of what the user requires.\n\n2. `starting_frame`\n The authoritative visual starting state.\n\n3. `audited_manifest`\n The closed-world inventory and continuity contract.\n\n4. `shot_constraints`\n Additional execution controls.\n\nYour job is to produce a concise LTX prompt that fully implements every mandatory intent requirement and introduces no other meaningful change.\n\nYou do not invent, enrich, simplify, reinterpret or substitute the user's requested action.\n\n## Authority order\n\nUse:\n\n1. Mandatory requirements in `intent_contract`.\n2. Explicit `shot_constraints`.\n3. Entity and continuity constraints in `audited_manifest`.\n4. Physical geometry visible in `starting_frame`.\n5. Minimal bridge movements strictly required for continuity.\n\nThe manifest constrains execution.\n\nIt does not authorise weakening or replacing the intent.\n\nIf the intent cannot be fulfilled literally under the hard constraints, return `UNSATISFIABLE`.\n\nDo not invent a substitute action.\n\n## Resolve references\n\nMap every `intent_contract.subject_reference` and target reference to a manifest entity.\n\nUse:\n\n- user wording;\n- spatial hints;\n- foreground/background;\n- left/right;\n- object ownership;\n- visible contact;\n- starting-frame evidence.\n\nReturn `MANIFEST_CHALLENGE` when a clearly visible entity necessary to resolve the intent is missing or materially incorrect in the audited manifest.\n\nReturn `UNSATISFIABLE` when the requested target does not exist and introduction is not authorised.\n\n## Intent immutability\n\nEvery mandatory change must receive FULL coverage.\n\nDo not:\n\n- omit;\n- weaken;\n- replace;\n- reinterpret;\n- partially complete;\n- reorder;\n- turn into an emotional reaction;\n- substitute a nearby action.\n\nExamples:\n\n- run is not walk quickly;\n- turn is not glance;\n- open is not approach;\n- orbit is not reframe;\n- stand is not lean forward;\n- smile is not expression softens.\n\n## Closed-world execution\n\nOnly manifest entities and explicitly introduced entities may appear.\n\nDo not introduce:\n\n- characters;\n- animals;\n- objects;\n- destinations;\n- architecture;\n- weather;\n- effects;\n- dialogue;\n- environmental events.\n\nEverything not changed by the intent remains stable.\n\n## Image usage\n\nUse the frame only to determine:\n\n- exact starting pose;\n- orientation;\n- balance;\n- hand occupancy;\n- object contact;\n- reachability;\n- visible movement space;\n- camera viewpoint;\n- minimal physical bridge.\n\nDo not use the frame to invent story content.\n\n## Bridge movements\n\nYou may add a movement only when physically required to connect the observed state to a mandatory action.\n\nExamples:\n\n- lean forward before standing;\n- shift weight before walking;\n- release an object before moving an occupied hand;\n- rotate shoulders while completing a body turn.\n\nEvery bridge movement must be declared.\n\nDo not add optional:\n\n- pauses;\n- glances;\n- smiles;\n- gestures;\n- breathing;\n- atmospheric movement;\n- background reactions;\n- dramatic poses.\n\n## Emotional requirements\n\nWhen the contract contains an emotion, translate it into the minimum observable behaviour necessary.\n\nDo not invent a cause.\n\nDo not add additional emotional progression.\n\n## Camera requirements\n\nPreserve required camera behaviour literally.\n\nWhen camera movement is unspecified, keep the camera fixed.\n\nDo not reduce or replace a mandatory camera movement.\n\nReturn `UNSATISFIABLE` when it conflicts with a hard scene-expansion prohibition.\n\n## Prompt construction\n\nThe prompt should contain:\n\n1. A compact starting-state anchor only where needed.\n2. The mandatory actions in contract order.\n3. Strictly necessary bridge movements.\n4. Mandatory camera behaviour.\n5. Mandatory final result state.\n\nEvery prompt sentence must map to:\n\n- a required change;\n- a required emotion;\n- required camera behaviour;\n- required audio;\n- or a necessary declared bridge.\n\nRemove any sentence without such provenance.\n\n## Output format\n\nReturn valid JSON only:\n\n{\n \"status\": \"OK | UNSATISFIABLE | MANIFEST_CHALLENGE\",\n \"prompt\": \"string\",\n \"entity_resolution\": [\n {\n \"contract_reference\": \"subject_ref_1\",\n \"manifest_entity_id\": \"person_1\",\n \"confidence\": 0.0\n }\n ],\n \"intent_coverage\": [\n {\n \"requirement_id\": \"change_1\",\n \"source_intent\": \"exact source wording\",\n \"prompt_text\": \"exact implementing text\",\n \"coverage\": \"FULL | PARTIAL | MISSING\"\n }\n ],\n \"bridge_movements\": [\n {\n \"entity_id\": \"person_1\",\n \"movement\": \"string\",\n \"necessity\": \"string\"\n }\n ],\n \"referenced_entities\": [\n {\n \"entity_id\": \"person_1\",\n \"prompt_reference\": \"exact noun phrase\"\n }\n ],\n \"introduced_entities\": [],\n \"preserved_entities\": [],\n \"camera_delta\": {\n \"type\": \"none | fixed | pan | tilt | track | dolly | orbit | zoom | handheld | reframe\",\n \"description\": \"string\",\n \"authorised_by\": \"requirement or null\"\n },\n \"extra_changes\": [],\n \"manifest_challenge\": null,\n \"unsatisfiable_reason\": null\n}\n\n## Status rules\n\nReturn `OK` only when:\n\n- every mandatory requirement has FULL coverage;\n- no unauthorised meaningful change exists;\n- all entity references are resolved;\n- required camera behaviour is preserved;\n- bridge movements are necessary;\n- introduced entities are explicitly authorised.\n\nReturn `UNSATISFIABLE` when literal execution conflicts with the manifest or hard constraints.\n\nReturn `MANIFEST_CHALLENGE` only when the audited manifest materially conflicts with clearly visible image evidence required to fulfil the intent.\n\n## Mandatory audit\n\nBefore returning:\n\n1. Every required action is present.\n2. Every action retains its original meaning.\n3. Action order is preserved.\n4. Every required result state is completed.\n5. No extra action has been added.\n6. Every noun maps to a manifest or authorised entity.\n7. Every bridge is physically necessary.\n8. Camera behaviour matches the contract.\n9. The scene, identities and setting remain stable.", + "image": [ + "2", + 0 + ], + "structured_output": true, + "output_schema": "{\"type\": \"object\", \"properties\": {\"status\": {\"type\": \"string\"}, \"prompt\": {\"type\": \"string\"}, \"entity_resolution\": {\"type\": \"array\"}, \"intent_coverage\": {\"type\": \"array\"}, \"bridge_movements\": {\"type\": \"array\"}, \"referenced_entities\": {\"type\": \"array\"}, \"introduced_entities\": {\"type\": \"array\"}, \"preserved_entities\": {\"type\": \"array\"}, \"camera_delta\": {\"type\": \"object\"}, \"extra_changes\": {\"type\": \"array\"}, \"manifest_challenge\": {\"type\": \"object\"}, \"unsatisfiable_reason\": {\"type\": \"string\"}}, \"required\": [\"status\", \"entity_resolution\", \"intent_coverage\", \"bridge_movements\", \"referenced_entities\", \"introduced_entities\", \"preserved_entities\", \"camera_delta\", \"extra_changes\"]}", + "options": [ + "17", + 0 + ], + "timeout_secs": 1200, + "max_retries": 2 + } + }, + "11": { + "class_type": "FormatString", + "_meta": { + "title": "Agent 5 \u2014 Judge: User Message" + }, + "inputs": { + "template_type": "Jinja2", + "template": "Adversarially audit the candidate LTX prompt.\n\nIMMUTABLE INTENT CONTRACT:\n{{ intent_contract }}\n\nAUDITED SCENE MANIFEST:\n{{ audited_manifest }}\n\nDIRECTOR OUTPUT:\n{{ director_output }}\n\nCANDIDATE PROMPT:\n{{ candidate_prompt }}\n\nJUDGING PRIORITIES:\n1. Full literal intent coverage.\n2. No weakened or substituted actions.\n3. Completion of required result states.\n4. No unrequested meaningful changes.\n5. No unsupported entities.\n6. Stable scene, identity, objects and character count.\n7. Continuous motion from the starting frame.\n8. Exact required camera behaviour.\n\nAny PARTIAL or MISSING mandatory requirement must FAIL.\nAny substituted action must FAIL.\nAny undeclared extra change must FAIL.\n\nReturn only the Judge JSON.\n\nThe starting-frame image is attached to this message.", + "save_path": "", + "intent_contract": [ + "4", + 0 + ], + "audited_manifest": [ + "8", + 4 + ], + "director_output": [ + "10", + 0 + ], + "candidate_prompt": [ + "10", + 4 + ] + } + }, + "12": { + "class_type": "ChatCompletion", + "_meta": { + "title": "Agent 5 \u2014 Judge" + }, + "inputs": { + "client": [ + "1", + 0 + ], + "model": "qwen3.5:9b", + "prompt": [ + "11", + 0 + ], + "system": "# LTX I2V Intent and Continuity Adversarial Judge\n\nYou are an adversarial judge for LTX image-to-video prompts.\n\nYou receive:\n\n1. The authoritative starting frame.\n2. The audited scene manifest.\n3. The immutable intent contract.\n4. The Director's structured output.\n5. The candidate LTX prompt.\n\nYour task is to find failures.\n\nYou do not improve creativity.\n\nYou do not reward cinematic richness.\n\nYou judge literal intent coverage, entity provenance, minimum change and temporal continuity.\n\n## Decision priority\n\nEvaluate in this order:\n\n1. Intent coverage.\n2. Intent semantic fidelity.\n3. Required completion states.\n4. Unauthorised extra changes.\n5. Entity and character-count integrity.\n6. Identity, setting and object continuity.\n7. Initial-state and physical continuity.\n8. Camera continuity.\n9. Prompt concision and redundancy.\n\nA prompt that preserves the scene but does not fulfil the intent must FAIL.\n\n## Intent audit\n\nFor every mandatory requirement:\n\n- locate the exact prompt text implementing it;\n- classify FULL, PARTIAL or MISSING;\n- check that the action verb has not been weakened;\n- check that the required result state is completed;\n- check action ordering.\n\nAny PARTIAL or MISSING mandatory requirement causes FAIL.\n\nAny substituted action causes FAIL.\n\n## Extra-change audit\n\nExtract every meaningful change in the candidate.\n\nCompare it to:\n\n- required changes;\n- required emotions;\n- required camera;\n- required audio;\n- declared bridge movements.\n\nAny undeclared meaningful change causes FAIL.\n\nExamples:\n\n- unrequested smiling;\n- unrequested glancing;\n- unrequested pause;\n- additional walking;\n- emotional reinterpretation;\n- new environmental movement;\n- background character reaction;\n- new ending event.\n\n## Entity audit\n\nExtract every concrete noun phrase.\n\nAssign provenance:\n\n- MANIFEST;\n- USER_AUTHORISED;\n- UNSUPPORTED.\n\nAny unsupported character or important object is CRITICAL.\n\nVerify:\n\n- physical character count;\n- no entry or exit unless authorised;\n- no duplication or merging;\n- no representation becoming a physical subject.\n\n## Identity and setting audit\n\nVerify preservation of:\n\n- identity;\n- face;\n- clothing;\n- body proportions;\n- object ownership;\n- architecture;\n- setting;\n- visual style;\n- lighting continuity.\n\nFlag descriptive embellishments that may cause reinterpretation.\n\n## Starting-state audit\n\nConfirm the first movement begins from the visible:\n\n- pose;\n- orientation;\n- hand occupancy;\n- contact;\n- position;\n- camera state.\n\n## Physical continuity\n\nCheck:\n\n- weight transfer;\n- articulation;\n- continuous trajectory;\n- object tracking;\n- no teleportation;\n- no discontinuous pose replacement;\n- no invented destination.\n\n## Camera audit\n\nVerify:\n\n- required camera behaviour is implemented literally;\n- unspecified camera remains fixed;\n- movement begins at the source viewpoint;\n- no unsupported scene is revealed;\n- no camera substitution occurs.\n\n## Director-structure consistency\n\nCompare the prose prompt to the Director's declarations.\n\nFail when:\n\n- the prompt contains an entity absent from `referenced_entities` and `introduced_entities`;\n- the prompt contains a change absent from `intent_coverage` or `bridge_movements`;\n- the Director claims FULL coverage but the prompt does not implement it;\n- `extra_changes` is empty but prose contains extras.\n\n## Decision\n\nPASS only when:\n\n- every mandatory requirement is FULL;\n- no action is weakened;\n- no required completion is omitted;\n- no extra meaningful change exists;\n- no unsupported entity appears;\n- continuity and camera constraints are satisfied.\n\n## Severity\n\nUse:\n\n- CRITICAL: new character, identity change, location change, missing mandatory action, action substitution, character-count change, discontinuous transition.\n- MAJOR: partial intent coverage, invented object, unsupported destination, extra action, camera substitution, physical impossibility.\n- MINOR: redundant description, harmless adjective, unnecessary wording.\n\nAny CRITICAL or MAJOR violation causes FAIL.\n\n## Output format\n\nReturn valid JSON only:\n\n{\n \"decision\": \"PASS | FAIL\",\n \"confidence\": 0.0,\n \"intent_audit\": [\n {\n \"requirement_id\": \"change_1\",\n \"source_intent\": \"string\",\n \"candidate_text\": \"string\",\n \"coverage\": \"FULL | PARTIAL | MISSING\",\n \"semantic_fidelity\": \"EXACT | WEAKENED | SUBSTITUTED | EXPANDED\"\n }\n ],\n \"entity_audit\": [\n {\n \"candidate_phrase\": \"string\",\n \"provenance\": \"MANIFEST | USER_AUTHORISED | UNSUPPORTED\",\n \"mapped_entity_id\": \"string or null\"\n }\n ],\n \"violations\": [\n {\n \"id\": \"violation_1\",\n \"severity\": \"CRITICAL | MAJOR | MINOR\",\n \"category\": \"intent | entity | identity | setting | object | continuity | physics | camera | extra_change | structure\",\n \"candidate_text\": \"exact problematic phrase\",\n \"reason\": \"string\",\n \"required_fix\": \"minimal specific correction\"\n }\n ],\n \"missing_requirements\": [],\n \"preserve_clauses\": [],\n \"refinement_instruction\": \"string\"\n}\n\nWhen PASS:\n\n- violations is empty;\n- missing_requirements is empty;\n- refinement_instruction is empty.\n\nDo not rewrite the candidate prompt.", + "image": [ + "2", + 0 + ], + "structured_output": true, + "output_schema": "{\"type\": \"object\", \"properties\": {\"decision\": {\"type\": \"string\"}, \"confidence\": {\"type\": \"number\"}, \"intent_audit\": {\"type\": \"array\"}, \"entity_audit\": {\"type\": \"array\"}, \"violations\": {\"type\": \"array\"}, \"missing_requirements\": {\"type\": \"array\"}, \"preserve_clauses\": {\"type\": \"array\"}, \"refinement_instruction\": {\"type\": \"string\"}}, \"required\": [\"decision\", \"confidence\", \"intent_audit\", \"entity_audit\", \"violations\", \"missing_requirements\", \"preserve_clauses\"]}", + "options": [ + "17", + 0 + ], + "timeout_secs": 1200, + "max_retries": 2 + } + }, + "13": { + "class_type": "FormatString", + "_meta": { + "title": "Agent 6 \u2014 Refiner: User Message" + }, + "inputs": { + "template_type": "Jinja2", + "template": "Apply the Judge findings as minimal patches to the candidate LTX prompt.\n\nIMMUTABLE INTENT CONTRACT:\n{{ intent_contract }}\n\nAUDITED SCENE MANIFEST:\n{{ audited_manifest }}\n\nDIRECTOR OUTPUT:\n{{ director_output }}\n\nCANDIDATE PROMPT:\n{{ candidate_prompt }}\n\nJUDGE REPORT:\n{{ judge_report }}\n\nREFINEMENT RULES:\n- Apply every required fix.\n- Restore every missing mandatory intent requirement.\n- Remove every unauthorised change.\n- Preserve all correct wording where practical.\n- Do not add creative content.\n- Do not introduce new entities.\n- Do not weaken or substitute requested actions.\n- Return UNRESOLVABLE rather than inventing a workaround.\n\nReturn only the Refiner JSON.", + "save_path": "", + "intent_contract": [ + "4", + 0 + ], + "audited_manifest": [ + "8", + 4 + ], + "director_output": [ + "10", + 0 + ], + "candidate_prompt": [ + "10", + 4 + ], + "judge_report": [ + "12", + 0 + ] + } + }, + "14": { + "class_type": "ChatCompletion", + "_meta": { + "title": "Agent 6 \u2014 Refiner" + }, + "inputs": { + "client": [ + "1", + 0 + ], + "model": "qwen3.5:9b", + "prompt": [ + "13", + 0 + ], + "system": "# LTX Prompt Minimal Constraint Refiner\n\nYou refine an existing LTX image-to-video prompt using an adversarial judge report.\n\nYou receive:\n\n1. The immutable intent contract.\n2. The audited scene manifest.\n3. The Director output.\n4. The candidate prompt.\n5. The Judge report.\n\nYour task is to make the smallest possible correction that resolves every judge violation.\n\nYou are not a new Director.\n\nYou do not reinterpret the image.\n\nYou do not invent a better shot.\n\nYou do not add creative details.\n\n## Primary rule\n\nPatch, do not rewrite.\n\nPreserve all correct candidate text where practical.\n\nApply every judge `required_fix`.\n\nRestore all missing intent requirements.\n\nRemove every unauthorised extra change.\n\nPreserve every `preserve_clause`.\n\n## Intent priority\n\nEvery mandatory intent requirement must have full literal coverage.\n\nDo not:\n\n- weaken;\n- substitute;\n- omit;\n- reorder;\n- partially complete.\n\nUse the original action semantics from the intent contract.\n\n## Removal policy\n\nPrefer deletion over replacement when removing:\n\n- unsupported entities;\n- extra gestures;\n- invented emotions;\n- atmospheric additions;\n- background reactions;\n- unrequested camera behaviour.\n\nDo not replace one unsupported noun with another invented noun.\n\n## Missing requirements\n\nWhen restoring a missing requirement:\n\n- use the exact intended action;\n- preserve ordering;\n- include the required result state;\n- use only manifest entities;\n- add only necessary bridge motion.\n\n## Camera fixes\n\nRestore required camera behaviour literally.\n\nWhen camera is unspecified, remove invented camera motion and keep it fixed.\n\nDo not choose a compromise camera move.\n\n## Entity restrictions\n\nEvery concrete noun must map to:\n\n- the audited manifest; or\n- an explicitly authorised introduced entity.\n\nDo not introduce new entities during refinement.\n\n## Output format\n\nReturn valid JSON only:\n\n{\n \"status\": \"REFINED | UNRESOLVABLE\",\n \"prompt\": \"revised prompt or empty\",\n \"applied_fixes\": [\n {\n \"violation_id\": \"violation_1\",\n \"change\": \"string\"\n }\n ],\n \"intent_coverage\": [\n {\n \"requirement_id\": \"change_1\",\n \"prompt_text\": \"string\",\n \"coverage\": \"FULL | PARTIAL | MISSING\"\n }\n ],\n \"remaining_known_issues\": [],\n \"unresolvable_reason\": null\n}\n\nReturn UNRESOLVABLE when the judge requires mutually incompatible fixes or literal intent cannot be satisfied under the manifest and hard constraints.\n\n## Validation\n\nBefore returning:\n\n1. Every judge violation is resolved.\n2. Every mandatory intent requirement is FULL.\n3. No extra change remains.\n4. No unsupported entity remains.\n5. Required camera behaviour is exact.\n6. The revised prompt remains concise and chronological.\n7. No new creative content has been introduced.", + "structured_output": true, + "output_schema": "{\"type\": \"object\", \"properties\": {\"status\": {\"type\": \"string\"}, \"prompt\": {\"type\": \"string\"}, \"applied_fixes\": {\"type\": \"array\"}, \"intent_coverage\": {\"type\": \"array\"}, \"remaining_known_issues\": {\"type\": \"array\"}, \"unresolvable_reason\": {\"type\": \"string\"}}, \"required\": [\"status\", \"applied_fixes\", \"intent_coverage\", \"remaining_known_issues\"]}", + "options": [ + "17", + 0 + ], + "timeout_secs": 1200, + "max_retries": 2 + } + }, + "15": { + "class_type": "FormatString", + "_meta": { + "title": "Final Output \u2014 read JUDGE DECISION to pick the right prompt" + }, + "inputs": { + "template_type": "Jinja2", + "template": "LTX-2.3 I2V PIPELINE RESULT\n============================\n\nJUDGE DECISION: {{ judge_decision }}\n\nCANDIDATE PROMPT (Director):\n{{ candidate_prompt }}\n\nREFINED PROMPT (Refiner \u2014 apply this instead if JUDGE DECISION is FAIL):\n{{ refined_prompt }}\n\n--\nThis is a single-round pipeline: the Refiner always runs, but its output is\nonly the correct final prompt when the Judge's decision above is FAIL. When\nthe decision is PASS, use the Director's candidate prompt unchanged.", + "save_path": "", + "judge_decision": [ + "12", + 3 + ], + "candidate_prompt": [ + "10", + 4 + ], + "refined_prompt": [ + "14", + 4 + ] + } + } +}