* perf(images): refresh the listing when the monitor sees a new output (#348, PR 2) The image monitor now reports every settled output (linked or not, also when linking raised) with its resolved path through set_output_listener. The API's notify_output_settled hops from the monitor's worker thread onto the server's loop (call_soon_threadsafe) and does nothing before the loop is known or after cleanup. mark_root_dirty matches the path against each walk's resolved_root (normcase(realpath), computed in the executor by walk_root, now its fourth value) and marks the root dirty. A dirty root gets one background walk, no sooner than rewalk_interval = min(15, max(5, 5 x last walk's duration)) after the previous walk started (last_started_at / last_duration are recorded on every walk). _schedule_walk is shared with PR 1's delayed stale refresh: one deferred walk per root, never armed while a walk runs (the walk's end re-checks dirty: the trailing walk), at least 0.5 s away, the earlier deadline wins, and every walk start cancels it. A failed walk clears dirty. Rescan semantics are unchanged. * fix(images): keep removed roots and cancelled walks from walking again (#348, PR 2) Review findings on the monitor-triggered walks: - reset_gallery_roots_cache now retires every listing record: its pending timer is cancelled, dirty is cleared, and no mark, timer or trailing walk starts a walk of it until a reader or the warm-up asks for that root again. Before, a timer or trailing walk pending when the roots changed stored a listing for the removed root, and the monitor (which watches the old roots until restart) kept it walking on every new output. - A walk task cancelled outside cleanup_gallery_listings no longer starts a trailing walk while the loop shuts down. - root_contains ignores a Windows extended-length prefix (\\?\ and \\?\UNC\) on either side, which realpath can keep on a file past MAX_PATH. - The REWALK comment states the ~20% GIL share only for walks up to 3 s. * test(images): prove a cancelled walk starts no trailing walk (#348, PR 2) The previous assertions held even with the trailing walk, because that walk finished before they ran; assert the walk number and the scan count. * perf(images): add a settled output to the listing without a walk (#348, PR 2) Live test: each landing in a 20,000-file folder cost a full walk (~0.4 s of GIL) plus a full workspace rebuild, so Images reads during a run took up to 2.1 s. The monitor's report now patches the stored listing instead. - output_settled (the monitor hop's target) slices the file's part below the root in its own case (relative_part; Windows compares folded), and _run_adds places one root's pending files in one executor batch: _settled_entry applies the walk's filters (suffix, thumbnails, depth, symlink by lstat; macOS on-disk spelling), insert_entry puts the (Path, mtime) entry where the newest-first sort does, add_folder copies the folder sets. Stored only when no walk started and no drop happened meanwhile; walked_at is untouched, so the 30 s refresh still walks. - Anything inexact marks the root dirty and walks as before: a running walk, an unsliceable path, a failed lstat, a tie in mtime, the cap, an unexpected error. Rescan, the refresh, the warm-up and invalidation stay full walks; a walk start forgets the pending adds. - The workspace caches follow a patched listing by entry identity (listing_delta): only added values are built, the newest order places them by bisection, the favorites lookup is copied and extended. Synthetic 20,000 files: per landing 252 ms (walk + full build) -> 18 ms (3 ms add + 15 ms warm build). * fix(images): review fixes for adds without a walk (#348, PR 2) - _newest_step steps the cached newest order only when the per-file cache still holds exactly the values this call built. A stale page read racing a newer warm build (both run in executor threads) stored an order labelled with older values than it held, and the next read inserted the new file a second time. It also sorts in full when a roots reset dropped the per-file values meanwhile. - The stored spelling is checked on Windows too (nt._getfinalpathname: realpath keeps the given case when it cannot open the file), and before the name filters, which compare the walk's stored spelling. macOS opens non-blocking, so a FIFO named like an image never holds the executor. - An output in a folder the walk cannot list (no read permission) is not added, as the walk skips it. - Windows-portable unit tests: POSIX paths in the insert tests, and a str.lower fold for the length guard (Windows' normcase keeps lengths). * test(images): reproduce the newest-order race and fix a Windows clock flake (#348, PR 2) - The race test now has the newer build step from the values the cached order was built from, which is the interleaving that inserted a file twice; it fails without the parts check in _newest_step. - test_a_failed_walk_records_its_timing_and_clears_dirty compared two monotonic readings that Windows' ~15.6 ms clock can make equal; the previous start is moved back instead. - TESTING.md: os.access ignores ACLs on Windows.
247 lines
9.3 KiB
Python
247 lines
9.3 KiB
Python
"""
|
|
ComfyUI_PromptManager: A comprehensive ComfyUI custom node that extends text encoding
|
|
with persistent prompt storage, advanced search capabilities, and an automatic image
|
|
gallery system.
|
|
|
|
This module provides prompt node types:
|
|
- PromptManager: CLIP encoding node that outputs CONDITIONING
|
|
- PromptManagerFocused: Editor-first layout using the same encoding contract
|
|
- PromptManagerSplit: Side-by-side editor and library using the same contract
|
|
- PromptManagerText: Text-only node that outputs STRING with prepend/append
|
|
functionality
|
|
|
|
The nodes share the same database backend for persistent prompt storage and include
|
|
an automatic image gallery system that monitors ComfyUI output directories and links
|
|
generated images to their source prompts.
|
|
|
|
Features:
|
|
- SQLite-based persistent prompt storage with deduplication
|
|
- Advanced search and categorization system
|
|
- Real-time image gallery with metadata extraction
|
|
- Web-based admin dashboard for prompt management
|
|
- Comprehensive logging and diagnostics
|
|
"""
|
|
|
|
import re
|
|
from pathlib import Path
|
|
|
|
|
|
def get_version():
|
|
"""Parse version from pyproject.toml"""
|
|
try:
|
|
pyproject_path = Path(__file__).parent / "pyproject.toml"
|
|
if pyproject_path.exists():
|
|
content = pyproject_path.read_text()
|
|
match = re.search(r'version\s*=\s*["\']([^"\']+)["\']', content)
|
|
if match:
|
|
return match.group(1)
|
|
except Exception:
|
|
pass
|
|
return "unknown"
|
|
|
|
|
|
from .prompt_manager import ( # noqa: E402
|
|
PromptManager,
|
|
PromptManagerFocused,
|
|
PromptManagerSplit,
|
|
)
|
|
from .prompt_manager_pair import PromptManagerPair # noqa: E402
|
|
from .prompt_manager_text import PromptManagerText # noqa: E402
|
|
from .prompt_search_list import PromptSearchList # noqa: E402
|
|
|
|
NODE_CLASS_MAPPINGS = {
|
|
"PromptManager": PromptManager,
|
|
"PromptManagerFocused": PromptManagerFocused,
|
|
"PromptManagerSplit": PromptManagerSplit,
|
|
"PromptManagerPair": PromptManagerPair,
|
|
"PromptManagerText": PromptManagerText,
|
|
"PromptSearchList": PromptSearchList,
|
|
}
|
|
|
|
NODE_DISPLAY_NAME_MAPPINGS = {
|
|
"PromptManager": "Prompt Manager (Classic)",
|
|
"PromptManagerFocused": "Prompt Manager (Focused)",
|
|
"PromptManagerSplit": "Prompt Manager (Split)",
|
|
"PromptManagerPair": "Prompt Manager (+/-)",
|
|
"PromptManagerText": "Prompt Manager Text",
|
|
"PromptSearchList": "Prompt Search List",
|
|
}
|
|
|
|
# Canvas extension only. ComfyUI loads every .js under this directory into the
|
|
# canvas page, so the admin/gallery bundles and vendored libraries under web/
|
|
# are served by our own routes instead (see py/api).
|
|
WEB_DIRECTORY = "web/comfy"
|
|
|
|
# Add API routes (same pattern as ComfyUI_Assets)
|
|
try:
|
|
# Set extension URI
|
|
import os
|
|
|
|
from .py import config
|
|
from .py.api import PromptManagerAPI
|
|
|
|
extension_uri = os.path.dirname(__file__)
|
|
config.extension_uri = extension_uri
|
|
|
|
# Register API routes using the same pattern as ComfyUI_Assets
|
|
routes = config.routes
|
|
api = PromptManagerAPI()
|
|
api.add_routes(routes)
|
|
# Temp files of preview writes a crash interrupted (#317), off the
|
|
# import path: the walk may cover a large thumbnails folder.
|
|
import threading
|
|
|
|
threading.Thread(
|
|
target=api.sweep_thumbnail_temps,
|
|
name="PromptManagerPreviewTempSweep",
|
|
daemon=True,
|
|
).start()
|
|
host_app = getattr(config.server_instance, "app", None)
|
|
if host_app is not None and hasattr(host_app, "on_cleanup"):
|
|
host_app.on_cleanup.append(api.cleanup_history_reads)
|
|
host_app.on_cleanup.append(api.cleanup_annotation_captures)
|
|
host_app.on_cleanup.append(api.cleanup_gallery_listings)
|
|
if host_app is not None and hasattr(host_app, "on_startup"):
|
|
# Build each output root's Images listing once the server's loop runs,
|
|
# so the first open does not wait for the walk (#348).
|
|
host_app.on_startup.append(api.warm_gallery_listings)
|
|
if host_app is not None and hasattr(host_app, "on_shutdown"):
|
|
# Open Needs review change streams end before the server waits on them.
|
|
host_app.on_shutdown.append(api.close_review_events)
|
|
|
|
except Exception as e:
|
|
# Log to internal logging system without console spam
|
|
try:
|
|
from .utils.logging_config import get_logger
|
|
|
|
logger = get_logger("prompt_manager.init")
|
|
logger.error(f"Failed to register API routes: {e}")
|
|
except Exception:
|
|
pass
|
|
|
|
# Shared database, set up with the monitoring system below. Stays None when that
|
|
# fails so the queue hook can report the database as unavailable instead of crashing.
|
|
_global_db = None
|
|
|
|
# Start image monitoring globally at module import time
|
|
# This ensures images are linked regardless of which node is used in the workflow
|
|
try:
|
|
from .database.operations import PromptDatabase
|
|
from .utils.image_monitor import get_image_monitor
|
|
from .utils.prompt_tracker import get_prompt_tracker
|
|
from .utils.logging_config import get_logger
|
|
|
|
_init_logger = get_logger("prompt_manager.init")
|
|
_init_logger.info("Starting global image monitoring system...")
|
|
|
|
# Initialize database and get singleton instances
|
|
_global_db = PromptDatabase()
|
|
_global_prompt_tracker = get_prompt_tracker(_global_db)
|
|
_global_image_monitor = get_image_monitor(_global_db, _global_prompt_tracker)
|
|
|
|
# The Needs review pass for ambiguous new outputs runs through the API's
|
|
# capture store (the one the scan and the Tag dialog use): a linked output
|
|
# is never queued and its pending items leave the queue. It stops before
|
|
# that store is closed at shutdown.
|
|
_review_api = globals().get("api")
|
|
if _review_api is not None:
|
|
_global_image_monitor.set_review_recorder(_review_api.record_review_file)
|
|
# Every settled output's preview is queued for the API's background
|
|
# worker (#249): the workspace never asks to generate thumbnails.
|
|
_global_image_monitor.set_thumbnail_queue(_review_api.queue_capture_thumbnail)
|
|
# Every settled output walks its Images listing again (#348): the
|
|
# API hops from the monitor's thread onto the server's loop.
|
|
_global_image_monitor.set_output_listener(_review_api.notify_output_settled)
|
|
_review_host = getattr(config.server_instance, "app", None)
|
|
if _review_host is not None and hasattr(_review_host, "on_cleanup"):
|
|
|
|
async def _stop_review_recording(
|
|
app, monitor=_global_image_monitor, api=_review_api
|
|
):
|
|
monitor.set_review_recorder(None)
|
|
monitor.set_thumbnail_queue(None)
|
|
monitor.set_output_listener(None)
|
|
api.close_auto_thumbnails(final=True)
|
|
|
|
_review_host.on_cleanup.insert(0, _stop_review_recording)
|
|
|
|
# Start monitoring
|
|
_global_image_monitor.start_monitoring()
|
|
_init_logger.info("Global image monitoring system started successfully")
|
|
|
|
except Exception as e:
|
|
try:
|
|
from .utils.logging_config import get_logger
|
|
|
|
_init_logger = get_logger("prompt_manager.init")
|
|
_init_logger.error(f"Failed to start image monitoring: {e}")
|
|
except Exception:
|
|
print(f"[ComfyUI-PromptManager] Warning: Failed to start image monitoring: {e}")
|
|
|
|
|
|
def _db_factory():
|
|
"""The shared database for the queue hook; called lazily per request."""
|
|
if _global_db is None:
|
|
raise RuntimeError("PromptManager database unavailable")
|
|
return _global_db
|
|
|
|
|
|
# Count prompt runs when ComfyUI dequeues workflows (it skips unchanged nodes, so
|
|
# node execution alone misses re-runs). Separate from monitoring so either can fail.
|
|
try:
|
|
from .py.config import server_instance
|
|
from .utils.usage_tracking import register_queue_hook
|
|
|
|
register_queue_hook(server_instance, _db_factory)
|
|
except Exception as e:
|
|
try:
|
|
from .utils.logging_config import get_logger
|
|
|
|
get_logger("prompt_manager.init").error(f"Failed to register usage hook: {e}")
|
|
except Exception:
|
|
import logging
|
|
|
|
logging.getLogger("prompt_manager.init").error(
|
|
f"Failed to register usage hook: {e}"
|
|
)
|
|
|
|
|
|
def _history_db_factory() -> "PromptDatabase":
|
|
"""Initialize history on its worker even if monitoring initialization failed."""
|
|
if _global_db is not None:
|
|
return _global_db
|
|
from .database.operations import PromptDatabase
|
|
|
|
return PromptDatabase()
|
|
|
|
|
|
_history_registration = None
|
|
try:
|
|
from .py.config import server_instance
|
|
from .utils.execution_history import register_history_hook
|
|
|
|
_history_registration = register_history_hook(server_instance, _history_db_factory)
|
|
except Exception:
|
|
try:
|
|
from .utils.logging_config import get_logger
|
|
|
|
get_logger("prompt_manager.init").error("Failed to register execution history")
|
|
except Exception:
|
|
import logging
|
|
|
|
logging.getLogger("prompt_manager.init").error(
|
|
"Failed to register execution history"
|
|
)
|
|
|
|
__all__ = ["NODE_CLASS_MAPPINGS", "NODE_DISPLAY_NAME_MAPPINGS", "WEB_DIRECTORY"]
|
|
|
|
# Print startup message with loaded tools
|
|
print()
|
|
print(f"\033[94m[ComfyUI-PromptManager] Version:\033[0m {get_version()}")
|
|
for node_key, display_name in NODE_DISPLAY_NAME_MAPPINGS.items():
|
|
# ASCII only: a Windows console using a legacy code page cannot print emoji,
|
|
# and a UnicodeEncodeError here would abort loading the whole node.
|
|
print(f" \033[94mLoaded:\033[0m {display_name}")
|
|
print(f"\033[94mTotal: {len(NODE_CLASS_MAPPINGS)} tools loaded\033[0m")
|
|
print()
|