Sebastian Monroy 527e8dc4ab 663 implement comfyui memory guardian to prevent oom errors (#12)
* feat(config): add Memory Hygiene config scaffolding and defaults for nilor-nodes

- introduce MemoryHygieneConfig and wire into NilorNodesConfig
- parse NILOR_MEMORY_HYGIENE_* from JSON5 and apply env overrides
- add validation for thresholds, policy, cooldowns, retries, and durations
- extend config.json5 with sane Memory Guardian defaults
- update .env.example

* feat(client): add ComfyUI capability detection and supports_hygiene cache

- add one-time probe for /system_stats and /free, cached per session
- expose supports_hygiene() that logs a single warning when unsupported
- use short timeouts and no retries; mark false only on 404/405
- leave transient failures retryable by keeping capability as unknown

* feat(memory): add MemoryHygiene module with typed skeleton and API

- introduce RemediationAction and RemediationResult dataclass
- add MemoryHygiene class with DI for client/config/logger
- implement check_and_remediate skeleton with enablement/capability checks
- add safe stats helper; defer policy/remediation to later commits
- export public symbols via all

* feat(memory): add metrics collector with usage pct and vram_total

- extend SystemStats with vram_total and parse from /system_stats
- add DerivedStats and collect_metrics() computing vram/ram used pct (0–100)
- integrate metrics collection in check_and_remediate skeleton
- safe math with clamping and None handling for incomplete stats

* feat(memory): implement policy engine thresholds, cooldown, and action selection

- add cooldown handling and respect it in check_and_remediate
- detect pressure via percent or absolute MB thresholds for vram/ram
- normalize policy and choose staged initial action (auto => free)
- return actionable RemediationResult with reason; no remediation yet
- helper functions for conversions and comparisons

* feat(memory): add remediation loop with retries, time caps, and cooldown

- implement remediate_cycle with free/unload flags and staged auto escalation
- respect MAX_RETRIES, SLEEP_BETWEEN_ATTEMPTS_SECONDS, MAX_CYCLE_DURATION_SECONDS
- set cooldown after cycle; return after-stats, attempts, action, and outcome reason
- integrate cycle into check_and_remediate; keep helpers in module

* feat(worker): wire MemoryHygiene into idle and post-completion paths

- initialize MemoryHygiene with comfy client and config in consume_jobs
- run hygiene before polling when idle and after prompt finalize with 1s debounce
- guard remediation by setting is_busy to block new intake; reset after
- keep websocket listener unaffected; schedule post-completion hygiene as background task

* feat(worker): throttle hygiene checks and add cadence tracking

- throttle idle hygiene by cfg.hygiene.idle_poll_seconds using monotonic clock
- add last_hygiene_check_ts to avoid overly frequent checks
- keep is_busy gating and post-completion debounce execution

* chore(memory): add structured logs for decisions and remediation

- log disabled/unsupported/cooldown/no-pressure branches
- log start/end of remediation cycles with before/after VRAM/RAM stats
- log each /free invocation flags; guard logging to avoid exceptions

* fix(memory): harden hygiene with session disable and single-warning on unsupported

- add _capability_disabled to short-circuit future runs after unsupported endpoints
- emit a single warning then quietly skip further cycles for the session
- preserve existing retry/backoff/cooldown and safe exception handling

* .env.example update

* improved get_system_stats and added more logging

* improve logging formatting for comfyui_client and memory_hygiene

* feat(nilor-nodes): add startup hygiene summary log; convert hygiene lambda to class method

- add WorkerConsumer._run_memory_hygiene() async method; remove late-bound lambda injection
- delegate to guarded helper to respect busy gate and optional debounce
- emit startup hygiene summary with effective thresholds from _CFG.hygiene
  (enabled, idle_poll_s, vram/ram pct caps, min_free_mb, policy, retries, cooldown,
  sleep_between, max_cycle)
- keep existing call sites in consume_loop() and _finalize_prompt() using the new method
- no functional changes to remediation logic; new log improves observability at boot

* got rid of redundant .env loading in worker_consumer

* chore(config): simplify global config caching

- keep process-wide _CONFIG singleton for shared configuration instance
- no functional behavior change to config loading paths

* fixed valueerror logging in media_stream
2025-10-20 13:56:38 +01:00
2024-04-19 05:44:31 +02:00
2024-11-15 11:30:03 +08:00

Nilor Nodes Documentation 👺

A collection of utility nodes for ComfyUI focusing on list manipulation, batch operations, and advanced I/O functionality.

Prerequisites

  • comfyui-kjnodes custom_nodes repo

🏭 Generators

Interpolated Float List

Generates a list of interpolated float values based on sections.

Input Type Description
number_of_floats INT Total number of float values to generate
number_of_sections INT Number of sections to divide into
section_number INT Current section being processed
interpolation_type ["slinear", "quadratic", "cubic"] Type of interpolation
Output Type Description
floats FLOAT List of interpolated float values

Notes: Creates smooth transitions between values using scipy's interpolation.

One Minus Float List

Creates an inverted list of float values (1 - x).

Input Type Description
list_of_floats FLOAT Input float list
Output Type Description
floats FLOAT Inverted float values

Notes: Simple inversion operation, useful for creating complementary values.

Remap Float List

Remaps a list of float values from one range to another.

Input Type Description
list_of_floats FLOAT Input float list
min_input FLOAT Minimum input value (default: 0.0)
max_input FLOAT Maximum input value (default: 1.0)
min_output FLOAT Minimum output value (default: 0.0)
max_output FLOAT Maximum output value (default: 1.0)
Output Type Description
remapped_floats FLOAT Remapped float values

Notes: Useful for scaling values between different ranges while preserving relationships.

Inverse Map Float List

Creates a mirror mapping of float values around their midpoint.

Input Type Description
list_of_floats FLOAT Input float list
Output Type Description
floats FLOAT Inverse mapped values

Notes: Automatically determines min/max from input list.

🛠️ Utilities

Int To List Of Bools

Converts an integer into a list of boolean values.

Input Type Description
number_of_images INT Number to convert
Output Type Description
booleans BOOLEAN List of boolean values

Notes: Creates a list where first N values are True, rest are False.

List of Ints

Generates a sequential or shuffled list of integers.

Input Type Description
min INT Starting integer (default: 0)
max INT Ending integer (default: 9)
shuffle BOOLEAN Whether to randomize order
Output Type Description
ints INT List of integers

Notes: Output is always a list, even for single values.

Select Index From List

Extracts a single item from a list at the specified index.

Input Type Description
list_of_any any Input list of any type
index INT Index to select (default: 0)
Output Type Description
any any Selected item

Notes: Uses custom AnyType to accept any input type. Handles tensor unpacking automatically.

Shuffle Image Batch

Randomly reorders images in a batch.

Input Type Description
images IMAGE Batch of images
seed INT Random seed for shuffling
Output Type Description
images IMAGE Shuffled image batch

Notes: Maintains batch dimensions while randomizing order.

💾 I/O Operations

Save Image To HF Dataset

Uploads images to a HuggingFace dataset.

Input Type Description
image IMAGE Image to upload
repository_id STRING HuggingFace dataset repository
hf_auth_token STRING HuggingFace authentication token
filename_prefix STRING Prefix for saved files

Notes: Requires HuggingFace authentication token and repository access.

Save EXR Arbitrary

Saves multi-channel data as an OpenEXR file.

Input Type Description
channels any List of tensor channels
filename_prefix STRING Output filename prefix

Notes: Supports arbitrary number of channels. Each channel must have same dimensions.

Save Video To HF Dataset

Uploads video files to a HuggingFace dataset.

Input Type Description
filenames VHS_FILENAMES List of video files
repository_id STRING HuggingFace dataset repository
hf_auth_token STRING HuggingFace authentication token
filename_prefix STRING Prefix for saved files

Notes: Handles batch upload of multiple video files.

📡 Core Nilor Services

Worker Consumer Service

The worker_consumer.py script is a background service that runs on each ComfyUI worker. It is responsible for pulling jobs from the central ElasticMQ jobs_to_process queue and submitting them to its local ComfyUI instance for processing. This service is essential for the distributed architecture of the system.

Key Responsibilities:

  • Continuously polls the jobs_to_process queue for new jobs using long polling.
  • When a job is received, it extracts the workflow data and submits it to the local ComfyUI server.
  • Deletes the job message from the queue upon successful submission to prevent reprocessing.
  • If submission fails, the message remains on the queue to be picked up by another worker.

⚙️ Configuration and Runtime Model

The sidecar uses a small, typed configuration loader with JSON5 defaults and optional environment overrides.

  • Precedence: environment variables > config/config.json5 (controlled by allow_env_override: true).
  • No hot‑reload: configuration is loaded once at process start and passed to components.
  • Paths/keys: JSON5 at ComfyUI/custom_nodes/nilor-nodes/config/config.json5 with NILOR_* keys (e.g., NILOR_COMFYUI_API_URL, NILOR_SQS_ENDPOINT_URL). Secrets (AWS secret) must be set via .env.
  • Typed object: loader returns a NilorNodesConfig with comfy and worker sections.

Pseudocode usage:

cfg = load_nilor_nodes_config()
# Comfy endpoints
http_url = cfg.comfy.api_url + "/prompt"
ws_url = cfg.comfy.ws_url + "/ws"
# SQS client params
endpoint = cfg.worker.sqs_endpoint_url
region = cfg.worker.aws_region
access_key = cfg.worker.aws_access_key_id
secret_key = cfg.worker.aws_secret_access_key
client_id = cfg.worker.worker_client_id

Current integrations:

  • worker_consumer.py: loads config at startup, reuses a single HTTP session, and uses cfg.comfy/cfg.worker exclusively.
  • media_stream.py: uses cfg.worker for SQS completion notifications.
Environment Variables

The nilor-nodes require a .env file to be present in the ComfyUI directory to configure the connection to the core services (MinIO, ElasticMQ, and the Brain API). To set it up, create a file named .env in the root of your ComfyUI directory by copying the .env.example template.

Instructions:

  1. Create a new file named .env in the ComfyUI directory.
  2. Copy the contents of the .env.example file into your new .env file.
  3. Replace the placeholder values with your actual credentials and endpoint URLs for your local or production environment.
S
Description
No description provided
Readme
347 KiB
Languages
Python 94%
JavaScript 6%