- raise a dedicated JobSubmissionError when ComfyUI prompt submission fails
- re-raise submission failures in process_message so consume loop does not delete SQS messages
- emit failed status updates on submit-time errors using status_policy.fail_status fallback
- remove failed-status dedupe guard so each retry attempt remains observable
- align parse-error logging with retry behavior ("Message will be retried")
* feat(config): add Memory Hygiene config scaffolding and defaults for nilor-nodes
- introduce MemoryHygieneConfig and wire into NilorNodesConfig
- parse NILOR_MEMORY_HYGIENE_* from JSON5 and apply env overrides
- add validation for thresholds, policy, cooldowns, retries, and durations
- extend config.json5 with sane Memory Guardian defaults
- update .env.example
* feat(client): add ComfyUI capability detection and supports_hygiene cache
- add one-time probe for /system_stats and /free, cached per session
- expose supports_hygiene() that logs a single warning when unsupported
- use short timeouts and no retries; mark false only on 404/405
- leave transient failures retryable by keeping capability as unknown
* feat(memory): add MemoryHygiene module with typed skeleton and API
- introduce RemediationAction and RemediationResult dataclass
- add MemoryHygiene class with DI for client/config/logger
- implement check_and_remediate skeleton with enablement/capability checks
- add safe stats helper; defer policy/remediation to later commits
- export public symbols via all
* feat(memory): add metrics collector with usage pct and vram_total
- extend SystemStats with vram_total and parse from /system_stats
- add DerivedStats and collect_metrics() computing vram/ram used pct (0–100)
- integrate metrics collection in check_and_remediate skeleton
- safe math with clamping and None handling for incomplete stats
* feat(memory): implement policy engine thresholds, cooldown, and action selection
- add cooldown handling and respect it in check_and_remediate
- detect pressure via percent or absolute MB thresholds for vram/ram
- normalize policy and choose staged initial action (auto => free)
- return actionable RemediationResult with reason; no remediation yet
- helper functions for conversions and comparisons
* feat(memory): add remediation loop with retries, time caps, and cooldown
- implement remediate_cycle with free/unload flags and staged auto escalation
- respect MAX_RETRIES, SLEEP_BETWEEN_ATTEMPTS_SECONDS, MAX_CYCLE_DURATION_SECONDS
- set cooldown after cycle; return after-stats, attempts, action, and outcome reason
- integrate cycle into check_and_remediate; keep helpers in module
* feat(worker): wire MemoryHygiene into idle and post-completion paths
- initialize MemoryHygiene with comfy client and config in consume_jobs
- run hygiene before polling when idle and after prompt finalize with 1s debounce
- guard remediation by setting is_busy to block new intake; reset after
- keep websocket listener unaffected; schedule post-completion hygiene as background task
* feat(worker): throttle hygiene checks and add cadence tracking
- throttle idle hygiene by cfg.hygiene.idle_poll_seconds using monotonic clock
- add last_hygiene_check_ts to avoid overly frequent checks
- keep is_busy gating and post-completion debounce execution
* chore(memory): add structured logs for decisions and remediation
- log disabled/unsupported/cooldown/no-pressure branches
- log start/end of remediation cycles with before/after VRAM/RAM stats
- log each /free invocation flags; guard logging to avoid exceptions
* fix(memory): harden hygiene with session disable and single-warning on unsupported
- add _capability_disabled to short-circuit future runs after unsupported endpoints
- emit a single warning then quietly skip further cycles for the session
- preserve existing retry/backoff/cooldown and safe exception handling
* .env.example update
* improved get_system_stats and added more logging
* improve logging formatting for comfyui_client and memory_hygiene
* feat(nilor-nodes): add startup hygiene summary log; convert hygiene lambda to class method
- add WorkerConsumer._run_memory_hygiene() async method; remove late-bound lambda injection
- delegate to guarded helper to respect busy gate and optional debounce
- emit startup hygiene summary with effective thresholds from _CFG.hygiene
(enabled, idle_poll_s, vram/ram pct caps, min_free_mb, policy, retries, cooldown,
sleep_between, max_cycle)
- keep existing call sites in consume_loop() and _finalize_prompt() using the new method
- no functional changes to remediation logic; new log improves observability at boot
* got rid of redundant .env loading in worker_consumer
* chore(config): simplify global config caching
- keep process-wide _CONFIG singleton for shared configuration instance
- no functional behavior change to config loading paths
* fixed valueerror logging in media_stream
- use client.ws_connect exclusively; if disabled/not constructed, sleep and retry
- keep event parsing semantics and state transitions unchanged
- retain HTTP fallback gating; WS path now simplified for maintainability
- import ComfyUILocalClient and ComfyUIClientError into worker
- construct a shared ComfyUILocalClient with the shared aiohttp session on startup
- replace direct POST to /prompt with client.submit_prompt(payload)
- preserve client_id injection, prompt_id mapping, and busy/current state updates
- catch ComfyUIClientError in submission error handling alongside aiohttp errors
- load Config once with env>JSON5 precedence and pass cfg into WorkerConsumer
- replace direct os.getenv reads with cfg.comfy and cfg.worker values
- compute Comfy HTTP/WS endpoints from cfg and reuse single aiohttp session
- keep behavior identical; no new features introduced
- Accept large preview frames from ComfyUI without dropping the socket.
- Binary preview frames are still ignored by the consumer; this change stops the
connection from closing with code 1009 (message too big).
- Impact: keeps workers from getting stuck "busy" and allows long video job
queues to finish reliably.
Notes:
- To reduce bandwidth instead of disabling client handling, start ComfyUI with
--preview-method none or a smaller --preview-size.
- copy content_id, venue, canvas, scene from job payload into status updates
- use running_status for first progress; fail_status on execution errors (fallbacks preserved)
- manage per-content context lifecycles
- Adds configuration for the new `job_status_updates` queue in the worker consumer.
- After successfully submitting a job to the local ComfyUI instance, the worker publishes a message with `{"status": "running"}`.
- Includes a critical check for `client_id` in the workflow data to ensure a job ID is present before publishing.
- Logs a warning and skips the update if `client_id` is missing, preventing silent failures.
- Errors during the status update publication are logged but do not interrupt the primary job, maintaining system resilience.
- Update .env.example
Sebastian Monroy