develop
1
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Sebastian Monroy
|
527e8dc4ab |
663 implement comfyui memory guardian to prevent oom errors (#12)
* feat(config): add Memory Hygiene config scaffolding and defaults for nilor-nodes - introduce MemoryHygieneConfig and wire into NilorNodesConfig - parse NILOR_MEMORY_HYGIENE_* from JSON5 and apply env overrides - add validation for thresholds, policy, cooldowns, retries, and durations - extend config.json5 with sane Memory Guardian defaults - update .env.example * feat(client): add ComfyUI capability detection and supports_hygiene cache - add one-time probe for /system_stats and /free, cached per session - expose supports_hygiene() that logs a single warning when unsupported - use short timeouts and no retries; mark false only on 404/405 - leave transient failures retryable by keeping capability as unknown * feat(memory): add MemoryHygiene module with typed skeleton and API - introduce RemediationAction and RemediationResult dataclass - add MemoryHygiene class with DI for client/config/logger - implement check_and_remediate skeleton with enablement/capability checks - add safe stats helper; defer policy/remediation to later commits - export public symbols via all * feat(memory): add metrics collector with usage pct and vram_total - extend SystemStats with vram_total and parse from /system_stats - add DerivedStats and collect_metrics() computing vram/ram used pct (0–100) - integrate metrics collection in check_and_remediate skeleton - safe math with clamping and None handling for incomplete stats * feat(memory): implement policy engine thresholds, cooldown, and action selection - add cooldown handling and respect it in check_and_remediate - detect pressure via percent or absolute MB thresholds for vram/ram - normalize policy and choose staged initial action (auto => free) - return actionable RemediationResult with reason; no remediation yet - helper functions for conversions and comparisons * feat(memory): add remediation loop with retries, time caps, and cooldown - implement remediate_cycle with free/unload flags and staged auto escalation - respect MAX_RETRIES, SLEEP_BETWEEN_ATTEMPTS_SECONDS, MAX_CYCLE_DURATION_SECONDS - set cooldown after cycle; return after-stats, attempts, action, and outcome reason - integrate cycle into check_and_remediate; keep helpers in module * feat(worker): wire MemoryHygiene into idle and post-completion paths - initialize MemoryHygiene with comfy client and config in consume_jobs - run hygiene before polling when idle and after prompt finalize with 1s debounce - guard remediation by setting is_busy to block new intake; reset after - keep websocket listener unaffected; schedule post-completion hygiene as background task * feat(worker): throttle hygiene checks and add cadence tracking - throttle idle hygiene by cfg.hygiene.idle_poll_seconds using monotonic clock - add last_hygiene_check_ts to avoid overly frequent checks - keep is_busy gating and post-completion debounce execution * chore(memory): add structured logs for decisions and remediation - log disabled/unsupported/cooldown/no-pressure branches - log start/end of remediation cycles with before/after VRAM/RAM stats - log each /free invocation flags; guard logging to avoid exceptions * fix(memory): harden hygiene with session disable and single-warning on unsupported - add _capability_disabled to short-circuit future runs after unsupported endpoints - emit a single warning then quietly skip further cycles for the session - preserve existing retry/backoff/cooldown and safe exception handling * .env.example update * improved get_system_stats and added more logging * improve logging formatting for comfyui_client and memory_hygiene * feat(nilor-nodes): add startup hygiene summary log; convert hygiene lambda to class method - add WorkerConsumer._run_memory_hygiene() async method; remove late-bound lambda injection - delegate to guarded helper to respect busy gate and optional debounce - emit startup hygiene summary with effective thresholds from _CFG.hygiene (enabled, idle_poll_s, vram/ram pct caps, min_free_mb, policy, retries, cooldown, sleep_between, max_cycle) - keep existing call sites in consume_loop() and _finalize_prompt() using the new method - no functional changes to remediation logic; new log improves observability at boot * got rid of redundant .env loading in worker_consumer * chore(config): simplify global config caching - keep process-wide _CONFIG singleton for shared configuration instance - no functional behavior change to config loading paths * fixed valueerror logging in media_stream |