James VeitchandClaude Sonnet 5
09a173e552
Reported live on a fresh runpod: structured_output extraction on ChatCompletion would fail with RuntimeError: chat_structured: response failed validation against schema after 3 attempt(s) ... Last error: Exceeded maximum output retries (0) ...but succeed if a plain (non-structured) chat call with the same prompt was run first. Root cause: pydantic-ai's Agent is built with retries=0 (comfydv drives its own outer retry loop, not pydantic-ai's internal one), so a single bad/empty tool-call response instantly exhausts one attempt. comfydv's own retry loop then retried with the *exact same request* every time — if the model is genuinely stuck (e.g. still warming up on a freshly-started backend) or produces a degenerate response for a given seed, every attempt fails identically. Confirmed the seed was unset in the reporting user's workflow (Ollama picks its own per call) and it still failed 3/3 in a row, which rules out "stuck on one bad seed" and points at request-level determinism plus zero backoff between attempts as the real gap. Plain chat() had zero retry/validation logic at all before this change — it silently returned whatever content came back, even blank. Given the underlying failure mode (a fresh model's first response sometimes being blank) is identical for both chat modes, this fix applies to both, not just structured output. Adds src/comfydv/_llm/retry.py: next_seed() (deterministic, starting from any user-pinned options["seed"], incrementing per retry — attempt 1 is always the caller's original, untouched request) and RETRY_BACKOFF_SECS (a flat delay between retries, giving a still- loading model time to finish rather than hammering it with identical requests back-to-back). Both OllamaProvider.chat() and LlamaCppProvider.chat() now retry (per the node's existing max_retries widget) when a response comes back blank, injecting the incremented seed each retry — Ollama via the native options.seed field, llama.cpp via the OpenAI spec's top-level "seed" field (nesting it under "options" like the rest of the Ollama-native passthrough would silently not work — llama-server's OpenAI-compatible endpoint doesn't read seed from there, a pre-existing, documented non-goal for general options passthrough that doesn't apply to this internal retry mechanism). Neither raises if every retry stays blank — matches chat()'s existing never-validates contract. chat_structured() in _llm/chat.py now injects the same incrementing seed via pydantic-ai's ModelSettings["seed"] (which correctly maps to the OpenAI API's top-level seed param for both backends) and adds the same backoff between attempts, on top of its existing validation-retry loop. Live-verified against real Ollama: the happy path (model responds normally first try) still makes exactly one network call for chat() and completes chat_structured() normally — no regression from the new retry scaffolding. The specific cold/stuck-model failure itself is runpod-cold-start-dependent and wasn't independently reproducible in this session's environment (a second live attempt succeeded on the very first try) — unit tests directly exercise the retry/seed/backoff logic instead (test_llm_retry.py, and new cases in test_ollama_provider.py, test_llamacpp_provider.py, and test_llm_chat_structured.py), verified against both providers. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0132ojafeazQ3ephcBejEWFj
25 lines
683 B
Python
25 lines
683 B
Python
"""Tests for comfydv._llm.retry — shared retry-on-blank-output helpers used
|
|
by both providers' chat() and the shared chat_structured() helper.
|
|
"""
|
|
|
|
from comfydv._llm.retry import next_seed
|
|
|
|
|
|
def test_next_seed_attempt_one_is_zero_by_default():
|
|
assert next_seed(None, 1) == 0
|
|
|
|
|
|
def test_next_seed_increments_from_zero_when_unset():
|
|
assert next_seed(None, 2) == 1
|
|
assert next_seed({}, 3) == 2
|
|
|
|
|
|
def test_next_seed_starts_from_pinned_base():
|
|
assert next_seed({"seed": 42}, 1) == 42
|
|
assert next_seed({"seed": 42}, 2) == 43
|
|
assert next_seed({"seed": 42}, 3) == 44
|
|
|
|
|
|
def test_next_seed_ignores_non_int_seed():
|
|
assert next_seed({"seed": "not-an-int"}, 2) == 1
|