Commit Graph
6 Commits
Author SHA1 Message Date
Sebastian Monroy 66c66f045f Fix websocket client crash on websockets 15.x by removing read_limit
Nilor-Nodes worker websocket listener was failing with:
TypeError: BaseEventLoop.create_connection() got an unexpected keyword argument 'read_limit'
On Python 3.12 with websockets>=15, passing read_limit to websockets.connect() ends up forwarding it to asyncio’s loop.create_connection(), which doesn’t accept that kwarg.
This change removes the read_limit argument from ComfyUILocalClient.ws_connect() while keeping max_size=None and existing ping settings, restoring websocket connectivity and preventing reconnect exhaustion.
2026-01-13 20:36:41 +00:00
Sebastian Monroy 527e8dc4ab 663 implement comfyui memory guardian to prevent oom errors (#12)
* feat(config): add Memory Hygiene config scaffolding and defaults for nilor-nodes

- introduce MemoryHygieneConfig and wire into NilorNodesConfig
- parse NILOR_MEMORY_HYGIENE_* from JSON5 and apply env overrides
- add validation for thresholds, policy, cooldowns, retries, and durations
- extend config.json5 with sane Memory Guardian defaults
- update .env.example

* feat(client): add ComfyUI capability detection and supports_hygiene cache

- add one-time probe for /system_stats and /free, cached per session
- expose supports_hygiene() that logs a single warning when unsupported
- use short timeouts and no retries; mark false only on 404/405
- leave transient failures retryable by keeping capability as unknown

* feat(memory): add MemoryHygiene module with typed skeleton and API

- introduce RemediationAction and RemediationResult dataclass
- add MemoryHygiene class with DI for client/config/logger
- implement check_and_remediate skeleton with enablement/capability checks
- add safe stats helper; defer policy/remediation to later commits
- export public symbols via all

* feat(memory): add metrics collector with usage pct and vram_total

- extend SystemStats with vram_total and parse from /system_stats
- add DerivedStats and collect_metrics() computing vram/ram used pct (0–100)
- integrate metrics collection in check_and_remediate skeleton
- safe math with clamping and None handling for incomplete stats

* feat(memory): implement policy engine thresholds, cooldown, and action selection

- add cooldown handling and respect it in check_and_remediate
- detect pressure via percent or absolute MB thresholds for vram/ram
- normalize policy and choose staged initial action (auto => free)
- return actionable RemediationResult with reason; no remediation yet
- helper functions for conversions and comparisons

* feat(memory): add remediation loop with retries, time caps, and cooldown

- implement remediate_cycle with free/unload flags and staged auto escalation
- respect MAX_RETRIES, SLEEP_BETWEEN_ATTEMPTS_SECONDS, MAX_CYCLE_DURATION_SECONDS
- set cooldown after cycle; return after-stats, attempts, action, and outcome reason
- integrate cycle into check_and_remediate; keep helpers in module

* feat(worker): wire MemoryHygiene into idle and post-completion paths

- initialize MemoryHygiene with comfy client and config in consume_jobs
- run hygiene before polling when idle and after prompt finalize with 1s debounce
- guard remediation by setting is_busy to block new intake; reset after
- keep websocket listener unaffected; schedule post-completion hygiene as background task

* feat(worker): throttle hygiene checks and add cadence tracking

- throttle idle hygiene by cfg.hygiene.idle_poll_seconds using monotonic clock
- add last_hygiene_check_ts to avoid overly frequent checks
- keep is_busy gating and post-completion debounce execution

* chore(memory): add structured logs for decisions and remediation

- log disabled/unsupported/cooldown/no-pressure branches
- log start/end of remediation cycles with before/after VRAM/RAM stats
- log each /free invocation flags; guard logging to avoid exceptions

* fix(memory): harden hygiene with session disable and single-warning on unsupported

- add _capability_disabled to short-circuit future runs after unsupported endpoints
- emit a single warning then quietly skip further cycles for the session
- preserve existing retry/backoff/cooldown and safe exception handling

* .env.example update

* improved get_system_stats and added more logging

* improve logging formatting for comfyui_client and memory_hygiene

* feat(nilor-nodes): add startup hygiene summary log; convert hygiene lambda to class method

- add WorkerConsumer._run_memory_hygiene() async method; remove late-bound lambda injection
- delegate to guarded helper to respect busy gate and optional debounce
- emit startup hygiene summary with effective thresholds from _CFG.hygiene
  (enabled, idle_poll_s, vram/ram pct caps, min_free_mb, policy, retries, cooldown,
  sleep_between, max_cycle)
- keep existing call sites in consume_loop() and _finalize_prompt() using the new method
- no functional changes to remediation logic; new log improves observability at boot

* got rid of redundant .env loading in worker_consumer

* chore(config): simplify global config caching

- keep process-wide _CONFIG singleton for shared configuration instance
- no functional behavior change to config loading paths

* fixed valueerror logging in media_stream
2025-10-20 13:56:38 +01:00
Sebastian Monroy 3476fbfbf9 feat(client/lifecycle): add session ownership, context mgmt, and health probe
- implement context-managed internal session (create/close when no external session)
- update HTTP calls to reuse owned session when available
- add async probe() with short timeout and no retries for /system_stats
- extend client protocol to include probe(); keep cooperative cancellation semantics
2025-10-16 19:18:16 +01:00
Sebastian Monroy aafaf87d18 feat(client/ws): add websocket connect with bounded reconnection and events
- implement ws_connect(client_id) using websockets with large-frame defaults
- add bounded reconnection (max 5 attempts, capped 30s total backoff) with jitter
- map websocket errors to ComfyUIClientWsClosed/ComfyUIClientError per plan
- yield parsed JSON events as WsEvent; wrap non-JSON as text/binary events
- add ws policy fields to client; keep cancellation propagation intact
2025-10-16 19:15:28 +01:00
Sebastian Monroy 790a3ce38e feat(client/http): add HTTP methods with idempotent retries and error mapping
- implement submit_prompt, get_system_stats, and free in ComfyUI client
- add exponential backoff + jitter for idempotent routes; no retry for /prompt
- normalize errors to ComfyUIClientError family with safe response snippets
- validate prompt_id contract and tolerant parsing of SystemStats fields
- create ephemeral sessions when no external session is provided (lifecycle later)
2025-10-16 19:12:26 +01:00
Sebastian Monroy 114a3d661f feat(client): scaffold ComfyUI thin client protocol, DTOs, and exceptions
- add `comfyui_client.py` with `ComfyUIClientProtocol` and `ComfyUILocalClient` constructor
- define DTOs: `SystemStats` and `WsEvent` with typed fields and docstrings
- add exception hierarchy: `ComfyUIClientError`, `ComfyUIClientTimeout`, `ComfyUIClientWsClosed`
- leave protocol methods unimplemented (raise NotImplementedError); behavior to follow in next commits
- export public surface via __all__; keep imports lightweight and typing-friendly
2025-10-16 19:07:25 +01:00