14 Commits
Author SHA1 Message Date
Sebastian Monroy 62b71b2978 moved NILOR_WORKFLOW_OS_NORMALIZATION_ENABLED param to config 2026-01-15 22:52:37 +00:00
Sebastian Monroy 266fe598c6 tweaks to config.json5 and .env.example 2026-01-07 13:00:46 +00:00
Sebastian Monroy 527e8dc4ab 663 implement comfyui memory guardian to prevent oom errors (#12)
* feat(config): add Memory Hygiene config scaffolding and defaults for nilor-nodes

- introduce MemoryHygieneConfig and wire into NilorNodesConfig
- parse NILOR_MEMORY_HYGIENE_* from JSON5 and apply env overrides
- add validation for thresholds, policy, cooldowns, retries, and durations
- extend config.json5 with sane Memory Guardian defaults
- update .env.example

* feat(client): add ComfyUI capability detection and supports_hygiene cache

- add one-time probe for /system_stats and /free, cached per session
- expose supports_hygiene() that logs a single warning when unsupported
- use short timeouts and no retries; mark false only on 404/405
- leave transient failures retryable by keeping capability as unknown

* feat(memory): add MemoryHygiene module with typed skeleton and API

- introduce RemediationAction and RemediationResult dataclass
- add MemoryHygiene class with DI for client/config/logger
- implement check_and_remediate skeleton with enablement/capability checks
- add safe stats helper; defer policy/remediation to later commits
- export public symbols via all

* feat(memory): add metrics collector with usage pct and vram_total

- extend SystemStats with vram_total and parse from /system_stats
- add DerivedStats and collect_metrics() computing vram/ram used pct (0–100)
- integrate metrics collection in check_and_remediate skeleton
- safe math with clamping and None handling for incomplete stats

* feat(memory): implement policy engine thresholds, cooldown, and action selection

- add cooldown handling and respect it in check_and_remediate
- detect pressure via percent or absolute MB thresholds for vram/ram
- normalize policy and choose staged initial action (auto => free)
- return actionable RemediationResult with reason; no remediation yet
- helper functions for conversions and comparisons

* feat(memory): add remediation loop with retries, time caps, and cooldown

- implement remediate_cycle with free/unload flags and staged auto escalation
- respect MAX_RETRIES, SLEEP_BETWEEN_ATTEMPTS_SECONDS, MAX_CYCLE_DURATION_SECONDS
- set cooldown after cycle; return after-stats, attempts, action, and outcome reason
- integrate cycle into check_and_remediate; keep helpers in module

* feat(worker): wire MemoryHygiene into idle and post-completion paths

- initialize MemoryHygiene with comfy client and config in consume_jobs
- run hygiene before polling when idle and after prompt finalize with 1s debounce
- guard remediation by setting is_busy to block new intake; reset after
- keep websocket listener unaffected; schedule post-completion hygiene as background task

* feat(worker): throttle hygiene checks and add cadence tracking

- throttle idle hygiene by cfg.hygiene.idle_poll_seconds using monotonic clock
- add last_hygiene_check_ts to avoid overly frequent checks
- keep is_busy gating and post-completion debounce execution

* chore(memory): add structured logs for decisions and remediation

- log disabled/unsupported/cooldown/no-pressure branches
- log start/end of remediation cycles with before/after VRAM/RAM stats
- log each /free invocation flags; guard logging to avoid exceptions

* fix(memory): harden hygiene with session disable and single-warning on unsupported

- add _capability_disabled to short-circuit future runs after unsupported endpoints
- emit a single warning then quietly skip further cycles for the session
- preserve existing retry/backoff/cooldown and safe exception handling

* .env.example update

* improved get_system_stats and added more logging

* improve logging formatting for comfyui_client and memory_hygiene

* feat(nilor-nodes): add startup hygiene summary log; convert hygiene lambda to class method

- add WorkerConsumer._run_memory_hygiene() async method; remove late-bound lambda injection
- delegate to guarded helper to respect busy gate and optional debounce
- emit startup hygiene summary with effective thresholds from _CFG.hygiene
  (enabled, idle_poll_s, vram/ram pct caps, min_free_mb, policy, retries, cooldown,
  sleep_between, max_cycle)
- keep existing call sites in consume_loop() and _finalize_prompt() using the new method
- no functional changes to remediation logic; new log improves observability at boot

* got rid of redundant .env loading in worker_consumer

* chore(config): simplify global config caching

- keep process-wide _CONFIG singleton for shared configuration instance
- no functional behavior change to config loading paths

* fixed valueerror logging in media_stream
2025-10-20 13:56:38 +01:00
Sebastian Monroy 5668e09a79 removed legacy code that only existed for controlling rollout, idc bout dat 2025-10-16 20:51:03 +01:00
Sebastian Monroy a94c1a9d60 update .env.example and config.json5 2025-10-16 20:39:33 +01:00
Sebastian Monroy 675fce6a16 update logger to use NILOR_LOG_LEVEL instead of LOG_LEVEL 2025-10-16 20:39:22 +01:00
Sebastian Monroy 0f22deb027 fixed config.json5 2025-10-16 20:00:21 +01:00
Sebastian Monroy 63b44ebff4 chore(config): add NILOR_COMFY_* settings and surface via typed config
- create config.json5 with client enable flag, HTTP retry, and WS reconnect defaults
- extend ComfyApiConfig to include client_enabled, retry, and websocket policy fields
- update loader to parse new keys and honor env overrides (env > file)
- add validations for backoff and reconnect parameters; keep existing fields unchanged
2025-10-16 19:20:39 +01:00
Sebastian Monroy 3c0bf2222c fix crash from mutating fields on frozen dataclasses during env overrides 2025-10-16 18:06:19 +01:00
Sebastian Monroy ff0c7f1209 feat(nilor-nodes-config): add NILOR_SQS_ENABLED and wire into startup gate
- add NILOR_SQS_ENABLED to config.json5 with default false
- extend NilorNodesConfig with sqs_enabled; apply env override when allowed
- update __init__.py to gate worker startup via cfg.sqs_enabled instead of raw env
2025-10-16 17:52:19 +01:00
Sebastian Monroy 14862c1712 chore(nilor-nodes-config): finalize JSON5 defaults and document key consumers
- annotate config.json5 keys with consuming modules and usage patterns
- clarify Comfy HTTP/WS suffixes and SQS usage across worker and media_stream
- state that AWS secret must come from .env; keep allow_env_override true
2025-10-16 17:41:37 +01:00
Sebastian Monroy 4a56d4dff5 feat(nilor-nodes): integrate typed config loader into worker_consumer
- load Config once with env>JSON5 precedence and pass cfg into WorkerConsumer
- replace direct os.getenv reads with cfg.comfy and cfg.worker values
- compute Comfy HTTP/WS endpoints from cfg and reuse single aiohttp session
- keep behavior identical; no new features introduced
2025-10-16 17:36:42 +01:00
Sebastian Monroy 5e579b1aed feat(nilor-nodes-config): implement loader merge, compat mapping, and validation
- implement Config.load with env>JSON5 precedence and JSON5 reading
- add typed parsing into dataclasses
- validate URL schemes, timeouts, poll wait bounds, and positive integers
- auto-generate NILOR_WORKER_CLIENT_ID from hostname with base36 suffix
2025-10-16 17:07:27 +01:00
Sebastian Monroy 9883728f55 feat(nilor-nodes-config): add JSON5 defaults and typed config scaffolding
- add config/config.json5 with non-secret defaults and allow_env_override flag
- introduce config/config.py with ComfyApiConfig, WorkerConfig, NilorNodesConfig
- define Config.load(env, json5_path) API; implementation deferred to next commit
- add types.py with ConfigSource enum for future integration
- add json5 to requirements.txt
2025-10-16 17:00:12 +01:00