beacon-reviewer caught a real gap: when a caller pins options["seed"],
the retry loop set the new seed on ModelSettings' top-level "seed"
field but left the same request's extra_body.options.seed (the
Ollama-native passthrough) at the original pinned value. A backend
that honors the nested field over the top-level OpenAI one would keep
sending the identical effective seed on every retry, silently
defeating this fix for exactly the pinned-seed case it needs to cover.
Now both fields are kept in sync on every retry attempt, built via
fresh copies rather than mutating the caller's options dict in place
(that dict is shared across every attempt, and possibly across other
calls). Added a regression test asserting the caller's options dict is
untouched after a multi-attempt retry.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0132ojafeazQ3ephcBejEWFj
Reported live on a fresh runpod: structured_output extraction on
ChatCompletion would fail with
RuntimeError: chat_structured: response failed validation against
schema after 3 attempt(s) ... Last error: Exceeded maximum output
retries (0)
...but succeed if a plain (non-structured) chat call with the same
prompt was run first. Root cause: pydantic-ai's Agent is built with
retries=0 (comfydv drives its own outer retry loop, not pydantic-ai's
internal one), so a single bad/empty tool-call response instantly
exhausts one attempt. comfydv's own retry loop then retried with the
*exact same request* every time — if the model is genuinely stuck
(e.g. still warming up on a freshly-started backend) or produces a
degenerate response for a given seed, every attempt fails identically.
Confirmed the seed was unset in the reporting user's workflow (Ollama
picks its own per call) and it still failed 3/3 in a row, which rules
out "stuck on one bad seed" and points at request-level determinism
plus zero backoff between attempts as the real gap.
Plain chat() had zero retry/validation logic at all before this change
— it silently returned whatever content came back, even blank. Given
the underlying failure mode (a fresh model's first response sometimes
being blank) is identical for both chat modes, this fix applies to
both, not just structured output.
Adds src/comfydv/_llm/retry.py: next_seed() (deterministic, starting
from any user-pinned options["seed"], incrementing per retry — attempt
1 is always the caller's original, untouched request) and
RETRY_BACKOFF_SECS (a flat delay between retries, giving a still-
loading model time to finish rather than hammering it with identical
requests back-to-back).
Both OllamaProvider.chat() and LlamaCppProvider.chat() now retry (per
the node's existing max_retries widget) when a response comes back
blank, injecting the incremented seed each retry — Ollama via the
native options.seed field, llama.cpp via the OpenAI spec's top-level
"seed" field (nesting it under "options" like the rest of the
Ollama-native passthrough would silently not work — llama-server's
OpenAI-compatible endpoint doesn't read seed from there, a pre-existing,
documented non-goal for general options passthrough that doesn't apply
to this internal retry mechanism). Neither raises if every retry stays
blank — matches chat()'s existing never-validates contract.
chat_structured() in _llm/chat.py now injects the same incrementing
seed via pydantic-ai's ModelSettings["seed"] (which correctly maps to
the OpenAI API's top-level seed param for both backends) and adds the
same backoff between attempts, on top of its existing validation-retry
loop.
Live-verified against real Ollama: the happy path (model responds
normally first try) still makes exactly one network call for chat()
and completes chat_structured() normally — no regression from the new
retry scaffolding. The specific cold/stuck-model failure itself is
runpod-cold-start-dependent and wasn't independently reproducible in
this session's environment (a second live attempt succeeded on the
very first try) — unit tests directly exercise the retry/seed/backoff
logic instead (test_llm_retry.py, and new cases in
test_ollama_provider.py, test_llamacpp_provider.py, and
test_llm_chat_structured.py), verified against both providers.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0132ojafeazQ3ephcBejEWFj
chat_structured() silently dropped the options dict (temperature, seed,
num_predict, repeat_penalty, etc. — set via the OllamaOption* nodes) in
structured_output mode: OllamaProvider.chat_structured() only used it for
the cache key, and the shared comfydv._llm.chat.chat_structured() helper
had no options parameter at all. On main, the hand-rolled implementation
sent options verbatim in the request body; this migration silently lost it,
contradicting FR-008 (structured-output behavior must be unchanged).
Fixed by forwarding options as pydantic-ai's model_settings.extra_body,
matching the original payload shape exactly rather than lossily remapping
onto ModelSettings' own standardized field names (which don't cover
Ollama-native params like num_predict/repeat_penalty/top_k anyway).
Verified against the live local Ollama server: structured output with
temperature=0.0/seed=42 succeeds on first attempt.
Caught by an independent beacon-reviewer pass on the diff before opening a
PR — exactly what that review step is for.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0132ojafeazQ3ephcBejEWFj
Provider-layer only — src/comfydv/_llm/chat.py doesn't touch ollama.py, so
it's safe to land ahead of the deferred atomic node cutover (issue #16).
Ports ADR-006's exact retry/validation contract: bounded retries (0-5,
clamped), RuntimeError naming model/attempt-count/truncated-last-response
on exhaustion. Agent's own internal retries disabled so the error contract
is comfydv's, not pydantic-ai's.
Required an API-discovery spike (Agent/OpenAIProvider/OpenAIChatModel
constructors and exception surface aren't reliably knowable from training
data for a fast-moving library) before tests could be written meaningfully
— tests and implementation were validated together rather than strictly
red-first; noted in tasks.md rather than presented as pure TDD.
Also fixes a pre-existing gap in test_packaging.py's requirements.txt/
pyproject.toml consistency check: it stripped `[extras]` on one side of the
comparison but not the other, which this change's pydantic-ai-slim[openai]
entry exposed as the first dependency with an extras marker.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0132ojafeazQ3ephcBejEWFj
