* perf(images): refresh the listing when the monitor sees a new output (#348, PR 2) The image monitor now reports every settled output (linked or not, also when linking raised) with its resolved path through set_output_listener. The API's notify_output_settled hops from the monitor's worker thread onto the server's loop (call_soon_threadsafe) and does nothing before the loop is known or after cleanup. mark_root_dirty matches the path against each walk's resolved_root (normcase(realpath), computed in the executor by walk_root, now its fourth value) and marks the root dirty. A dirty root gets one background walk, no sooner than rewalk_interval = min(15, max(5, 5 x last walk's duration)) after the previous walk started (last_started_at / last_duration are recorded on every walk). _schedule_walk is shared with PR 1's delayed stale refresh: one deferred walk per root, never armed while a walk runs (the walk's end re-checks dirty: the trailing walk), at least 0.5 s away, the earlier deadline wins, and every walk start cancels it. A failed walk clears dirty. Rescan semantics are unchanged. * fix(images): keep removed roots and cancelled walks from walking again (#348, PR 2) Review findings on the monitor-triggered walks: - reset_gallery_roots_cache now retires every listing record: its pending timer is cancelled, dirty is cleared, and no mark, timer or trailing walk starts a walk of it until a reader or the warm-up asks for that root again. Before, a timer or trailing walk pending when the roots changed stored a listing for the removed root, and the monitor (which watches the old roots until restart) kept it walking on every new output. - A walk task cancelled outside cleanup_gallery_listings no longer starts a trailing walk while the loop shuts down. - root_contains ignores a Windows extended-length prefix (\\?\ and \\?\UNC\) on either side, which realpath can keep on a file past MAX_PATH. - The REWALK comment states the ~20% GIL share only for walks up to 3 s. * test(images): prove a cancelled walk starts no trailing walk (#348, PR 2) The previous assertions held even with the trailing walk, because that walk finished before they ran; assert the walk number and the scan count. * perf(images): add a settled output to the listing without a walk (#348, PR 2) Live test: each landing in a 20,000-file folder cost a full walk (~0.4 s of GIL) plus a full workspace rebuild, so Images reads during a run took up to 2.1 s. The monitor's report now patches the stored listing instead. - output_settled (the monitor hop's target) slices the file's part below the root in its own case (relative_part; Windows compares folded), and _run_adds places one root's pending files in one executor batch: _settled_entry applies the walk's filters (suffix, thumbnails, depth, symlink by lstat; macOS on-disk spelling), insert_entry puts the (Path, mtime) entry where the newest-first sort does, add_folder copies the folder sets. Stored only when no walk started and no drop happened meanwhile; walked_at is untouched, so the 30 s refresh still walks. - Anything inexact marks the root dirty and walks as before: a running walk, an unsliceable path, a failed lstat, a tie in mtime, the cap, an unexpected error. Rescan, the refresh, the warm-up and invalidation stay full walks; a walk start forgets the pending adds. - The workspace caches follow a patched listing by entry identity (listing_delta): only added values are built, the newest order places them by bisection, the favorites lookup is copied and extended. Synthetic 20,000 files: per landing 252 ms (walk + full build) -> 18 ms (3 ms add + 15 ms warm build). * fix(images): review fixes for adds without a walk (#348, PR 2) - _newest_step steps the cached newest order only when the per-file cache still holds exactly the values this call built. A stale page read racing a newer warm build (both run in executor threads) stored an order labelled with older values than it held, and the next read inserted the new file a second time. It also sorts in full when a roots reset dropped the per-file values meanwhile. - The stored spelling is checked on Windows too (nt._getfinalpathname: realpath keeps the given case when it cannot open the file), and before the name filters, which compare the walk's stored spelling. macOS opens non-blocking, so a FIFO named like an image never holds the executor. - An output in a folder the walk cannot list (no read permission) is not added, as the walk skips it. - Windows-portable unit tests: POSIX paths in the insert tests, and a str.lower fold for the length guard (Windows' normcase keeps lengths). * test(images): reproduce the newest-order race and fix a Windows clock flake (#348, PR 2) - The race test now has the newer build step from the values the cached order was built from, which is the interleaving that inserted a file twice; it fails without the parts check in _newest_step. - test_a_failed_walk_records_its_timing_and_clears_dirty compared two monotonic readings that Windows' ~15.6 ms clock can make equal; the previous start is moved back instead. - TESTING.md: os.access ignores ACLs on Windows.
This commit is contained in:
Vito
+41
-6
@@ -362,8 +362,11 @@ class PromptManagerAPI(
|
||||
# background walks refreshing them (py/api/gallery_listing.py, #348).
|
||||
self._root_listings: dict[str, RootListing] = {}
|
||||
self._listing_tasks: set = set()
|
||||
# The loop the walks run on, set by the first walk.
|
||||
# The loop the walks run on, set by the startup warm-up or the first
|
||||
# walk; the monitor's thread hops onto it (notify_output_settled).
|
||||
self._listing_loop = None
|
||||
# Set by cleanup_gallery_listings: no background walk starts after it.
|
||||
self._listings_closed = False
|
||||
# The workspace caches rebuilt behind a stored walk that changed the
|
||||
# listing, once the host's startup warm-up has run (#348).
|
||||
self._workspace_warm_enabled = False
|
||||
@@ -492,7 +495,7 @@ class PromptManagerAPI(
|
||||
"""The shared listing record of one output root (created empty)."""
|
||||
return self._root_listings.setdefault(str(root), RootListing())
|
||||
|
||||
def invalidate_gallery_cache(self):
|
||||
def invalidate_gallery_cache(self, retire=False):
|
||||
"""Drop every root's listing, so the next read waits for a new walk.
|
||||
|
||||
Called after files are deleted through the app (duplicate cleanup and
|
||||
@@ -503,6 +506,9 @@ class PromptManagerAPI(
|
||||
_workspace_listed and _card_facts); its sort orders are dropped, as
|
||||
the walk replaces them anyway, and so is the favorites lookup, which
|
||||
holds the old list.
|
||||
|
||||
``retire`` (the roots changed): no background walk of a root starts
|
||||
again until a reader or the warm-up asks for it.
|
||||
"""
|
||||
loop = self._listing_loop
|
||||
if loop is not None and loop.is_running() and _running_loop() is not loop:
|
||||
@@ -513,7 +519,7 @@ class PromptManagerAPI(
|
||||
|
||||
def drop():
|
||||
try:
|
||||
self._drop_listings()
|
||||
self._drop_listings(retire)
|
||||
finally:
|
||||
dropped.set()
|
||||
|
||||
@@ -521,11 +527,19 @@ class PromptManagerAPI(
|
||||
if not dropped.wait(_OFF_LOOP_DROP_WAIT):
|
||||
self.logger.warning("The Images listing drop is still queued")
|
||||
return
|
||||
self._drop_listings()
|
||||
self._drop_listings(retire)
|
||||
|
||||
def _drop_listings(self):
|
||||
def _drop_listings(self, retire=False):
|
||||
for record in list(self._root_listings.values()):
|
||||
record.drop()
|
||||
if retire:
|
||||
# The roots changed: a removed root must not come back through
|
||||
# a pending timer, a trailing walk or a monitor mark.
|
||||
record.retired = True
|
||||
record.dirty = False
|
||||
handle, record.rewalk_handle = record.rewalk_handle, None
|
||||
if handle is not None:
|
||||
handle.cancel()
|
||||
self._workspace_order_cache = {}
|
||||
self._favorite_listing_cache = {}
|
||||
|
||||
@@ -539,15 +553,36 @@ class PromptManagerAPI(
|
||||
# Each stored walk that changes a listing then rebuilds the workspace's
|
||||
# per-file values, orders and favorites behind it (_warm_workspace).
|
||||
self._workspace_warm_enabled = True
|
||||
self._listing_loop = asyncio.get_running_loop()
|
||||
self._listings_closed = False
|
||||
try:
|
||||
for root in self._get_all_output_dirs():
|
||||
self._root_listing(root).retired = False
|
||||
self._schedule_walk(root)
|
||||
except Exception as e:
|
||||
self.logger.warning(f"Could not warm the Images listing: {e}")
|
||||
|
||||
def notify_output_settled(self, path):
|
||||
"""The image monitor settled an output at *path* (any thread).
|
||||
|
||||
Called on the monitor's worker thread with the resolved path. It only
|
||||
hands the path to the event loop (``output_settled``) and never
|
||||
touches a listing from this thread; before the loop is known, or once
|
||||
the listings are cleaned up, it does nothing.
|
||||
"""
|
||||
loop = self._listing_loop
|
||||
if loop is None or self._listings_closed or not loop.is_running():
|
||||
return
|
||||
try:
|
||||
loop.call_soon_threadsafe(self.output_settled, path)
|
||||
except RuntimeError:
|
||||
pass # the loop closed meanwhile (shutdown)
|
||||
|
||||
async def cleanup_gallery_listings(self, app=None):
|
||||
"""Stop the background listing walks (aiohttp on_cleanup)."""
|
||||
self._workspace_warm_enabled = False
|
||||
# No mark, deferred walk or trailing walk starts a walk from now on.
|
||||
self._listings_closed = True
|
||||
# Before any await, so no delayed refresh can start a walk meanwhile.
|
||||
for record in self._root_listings.values():
|
||||
handle, record.rewalk_handle = record.rewalk_handle, None
|
||||
@@ -571,7 +606,7 @@ class PromptManagerAPI(
|
||||
|
||||
def reset_gallery_roots_cache(self):
|
||||
"""Drop every cached listing and derived workspace value (roots changed)."""
|
||||
self.invalidate_gallery_cache()
|
||||
self.invalidate_gallery_cache(retire=True)
|
||||
self._workspace_listed_cache = {}
|
||||
self._workspace_card_cache = {}
|
||||
|
||||
|
||||
+211
-20
@@ -6,13 +6,20 @@ loop; the walk itself, the comparison with the previous listing and both
|
||||
folder sets run in the executor through ``walk_root``. Routes live in
|
||||
``py/api/images.py`` (``_listing``); this module has no aiohttp import so
|
||||
tests can exercise it without a server.
|
||||
|
||||
A settled output the image monitor reports is added to its root's stored
|
||||
listing without a walk (``output_settled``, ``insert_entry``), in the place
|
||||
a walk would list it. When that cannot be exact, the root is marked dirty
|
||||
(``mark_root_dirty``) instead; a dirty root gets one background walk, no
|
||||
sooner than ``rewalk_interval`` after its previous walk started (PR 2).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
from dataclasses import dataclass
|
||||
from bisect import bisect_left
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import PurePath
|
||||
from typing import Callable, NamedTuple
|
||||
|
||||
@@ -26,6 +33,19 @@ REFRESH_AFTER = 30
|
||||
# (#348 live test: reads during the walk took 2.2 s instead of 0.07 s).
|
||||
REFRESH_DELAY = 1.0
|
||||
|
||||
# No background walk of a root starts sooner than rewalk_interval(last walk's
|
||||
# duration) after its previous walk started: FACTOR times that duration, at
|
||||
# least FLOOR and at most CAP seconds. Walks up to CAP / FACTOR (3 s) then
|
||||
# hold the GIL at most ~20% of the time; the cap keeps a new output listed
|
||||
# within ~20 s, at the cost of a higher share (back to back from 15 s) on
|
||||
# slower walks. Read through this module so tests can patch them.
|
||||
REWALK_FLOOR = 5
|
||||
REWALK_FACTOR = 5
|
||||
REWALK_CAP = 15
|
||||
|
||||
# A deferred walk is never armed sooner than this: no zero-delay loop.
|
||||
MIN_REWALK_DELAY = 0.5
|
||||
|
||||
# A reader with no listing gives up (500) after this many walks in a row were
|
||||
# discarded by invalidations made while they ran.
|
||||
MAX_DISCARDED_WALKS = 3
|
||||
@@ -80,14 +100,35 @@ class RootListing:
|
||||
flight: asyncio.Future | None = None
|
||||
# True while walks keep failing: one warning per streak.
|
||||
failing: bool = False
|
||||
# The delayed refresh not started yet (loop.call_later), or None.
|
||||
# The one deferred background walk not started yet (loop.call_later), or
|
||||
# None: a stale refresh or a dirty root's rewalk, whichever is due first.
|
||||
rewalk_handle: asyncio.TimerHandle | None = None
|
||||
# normcase(realpath(root)), computed by the walk in the executor: monitor
|
||||
# paths are matched against it without file I/O on the loop.
|
||||
resolved_root: str | None = None
|
||||
# Monotonic start and duration of the last walk (rewalk_interval).
|
||||
last_started_at: float | None = None
|
||||
last_duration: float | None = None
|
||||
# A settled output was reported under this root; cleared when a walk starts.
|
||||
dirty: bool = False
|
||||
# Dropped because the output roots changed: no background walk until a
|
||||
# reader (or the warm-up) asks for this root again. The monitor keeps
|
||||
# watching a removed root until ComfyUI restarts.
|
||||
retired: bool = False
|
||||
# Relative paths of settled outputs waiting to be added to ``files``
|
||||
# without a walk, and the one task adding them (``output_settled``).
|
||||
pending_adds: list = field(default_factory=list)
|
||||
add_task: asyncio.Task | None = None
|
||||
# Settled outputs added to the stored listing since the last walk stored.
|
||||
patched: int = 0
|
||||
|
||||
def drop(self) -> None:
|
||||
self.files = None
|
||||
self.folders = None
|
||||
self.folders_with_ancestors = None
|
||||
self.walked_at = None
|
||||
self.pending_adds = []
|
||||
self.patched = 0
|
||||
self.generation += 1
|
||||
|
||||
|
||||
@@ -99,26 +140,174 @@ def folder_sets(root, files) -> tuple[frozenset, frozenset]:
|
||||
files, plus every ancestor joined by ``/``.
|
||||
"""
|
||||
# From the path strings: relative_to per file was half of a large walk.
|
||||
sep, prefix = _root_prefix(root)
|
||||
folders = set()
|
||||
for path, _mtime in files:
|
||||
rel_dir = _folder_of(root, path, sep, prefix)
|
||||
if rel_dir:
|
||||
folders.add(rel_dir)
|
||||
with_ancestors = set(folders)
|
||||
for folder in folders:
|
||||
with_ancestors.update(_ancestors(folder))
|
||||
return frozenset(folders), frozenset(with_ancestors)
|
||||
|
||||
|
||||
def add_folder(root, path, folders, with_ancestors) -> tuple[frozenset, frozenset]:
|
||||
"""Both folder sets once *path* is listed too (the same sets when it adds none)."""
|
||||
sep, prefix = _root_prefix(root)
|
||||
rel_dir = _folder_of(root, path, sep, prefix)
|
||||
if not rel_dir or rel_dir in folders:
|
||||
return folders, with_ancestors
|
||||
return folders | {rel_dir}, with_ancestors | {rel_dir, *_ancestors(rel_dir)}
|
||||
|
||||
|
||||
def _root_prefix(root) -> tuple[str, str]:
|
||||
# The separator is the path flavour's own (tests use PureWindowsPath).
|
||||
sep = str(type(root)("a", "b"))[1]
|
||||
prefix = str(root)
|
||||
if not prefix.endswith(sep):
|
||||
prefix += sep
|
||||
folders = set()
|
||||
for path, _mtime in files:
|
||||
text = str(path)
|
||||
if text.startswith(prefix):
|
||||
rel_dir = text[len(prefix) :].rpartition(sep)[0]
|
||||
else:
|
||||
rel_dir = str(path.relative_to(root).parent)
|
||||
if rel_dir and rel_dir != ".":
|
||||
folders.add(rel_dir)
|
||||
with_ancestors = set(folders)
|
||||
for folder in folders:
|
||||
parts = folder.replace("\\", "/").split("/")
|
||||
for i in range(1, len(parts)):
|
||||
with_ancestors.add("/".join(parts[:i]))
|
||||
return frozenset(folders), frozenset(with_ancestors)
|
||||
return sep, prefix
|
||||
|
||||
|
||||
def _folder_of(root, path, sep, prefix) -> str:
|
||||
"""The folder ``/gallery/subfolders`` names for *path*; "" for the root."""
|
||||
text = str(path)
|
||||
if text.startswith(prefix):
|
||||
rel_dir = text[len(prefix) :].rpartition(sep)[0]
|
||||
else:
|
||||
rel_dir = str(path.relative_to(root).parent)
|
||||
return "" if rel_dir == "." else rel_dir
|
||||
|
||||
|
||||
def _ancestors(folder: str) -> list[str]:
|
||||
parts = folder.replace("\\", "/").split("/")
|
||||
return ["/".join(parts[:i]) for i in range(1, len(parts))]
|
||||
|
||||
|
||||
def relative_part(resolved_root: str, path, pathmod=os.path) -> str | None:
|
||||
"""The part of the resolved *path* below *resolved_root*, as a walk spells it.
|
||||
|
||||
Kept in *path*'s own case (``realpath`` gives a file's on-disk case on
|
||||
Windows) with the native separator. None when *path* is not below the
|
||||
root, or when the case folding changed the length so the slice would
|
||||
not be exact.
|
||||
"""
|
||||
sep = pathmod.sep
|
||||
text = str(path)
|
||||
if sep == "\\":
|
||||
text = _plain(text.replace("/", sep), pathmod)
|
||||
folded = pathmod.normcase(text)
|
||||
prefix = _plain(resolved_root, pathmod)
|
||||
if not prefix.endswith(sep):
|
||||
prefix += sep
|
||||
if len(folded) != len(text) or not folded.startswith(prefix):
|
||||
return None
|
||||
relative = text[len(prefix) :]
|
||||
if any(part in ("", ".", "..") for part in relative.split(sep)):
|
||||
return None
|
||||
return relative
|
||||
|
||||
|
||||
def insert_entry(files: list, entry: tuple, max_files: int) -> list | None:
|
||||
"""A new listing with *entry* ``(path, mtime)`` where a walk would list it.
|
||||
|
||||
*files* is newest first; an entry for the same path is replaced. Returns
|
||||
*files* itself when it already holds *entry*, and None when the place is
|
||||
not exact: another file has the same mtime (a walk orders those by its
|
||||
own visit order), or the listing is at the walk's *max_files* cap.
|
||||
Blocking (one pass over *files*): run it in the executor.
|
||||
"""
|
||||
path, mtime = entry
|
||||
text = str(path)
|
||||
kept = files
|
||||
for index, (listed, listed_mtime) in enumerate(files):
|
||||
if str(listed) == text:
|
||||
if listed_mtime == mtime:
|
||||
return files
|
||||
kept = files[:index] + files[index + 1 :]
|
||||
break
|
||||
else:
|
||||
if len(files) >= max_files:
|
||||
return None
|
||||
index = bisect_left(kept, -mtime, key=lambda item: -item[1])
|
||||
if index < len(kept) and kept[index][1] == mtime:
|
||||
return None
|
||||
return [*kept[:index], entry, *kept[index:]]
|
||||
|
||||
|
||||
# A listing that differs from the cached one by more entries than this is
|
||||
# rebuilt in full by the workspace caches (listing_delta).
|
||||
MAX_DELTA = 256
|
||||
|
||||
|
||||
def listing_delta(old: list, new: list, limit: int = MAX_DELTA):
|
||||
"""``(added, removed)`` entries from *old* to *new*, or None past *limit*.
|
||||
|
||||
A listing patched by ``insert_entry`` shares every unchanged entry object
|
||||
with the one it came from, so entries are told apart by identity: a walk
|
||||
builds new tuples, shares none and is rebuilt in full. *old* must still
|
||||
be alive (the
|
||||
caller's cache holds it), so no identity was reused.
|
||||
"""
|
||||
if abs(len(new) - len(old)) > limit:
|
||||
return None
|
||||
old_ids = set(map(id, old))
|
||||
added = []
|
||||
for entry in new:
|
||||
if id(entry) not in old_ids:
|
||||
added.append(entry)
|
||||
if len(added) > limit:
|
||||
return None
|
||||
if new and len(added) == len(new):
|
||||
return None # nothing shared: a walk
|
||||
if len(old) + len(added) == len(new):
|
||||
return added, []
|
||||
new_ids = set(map(id, new))
|
||||
return added, [entry for entry in old if id(entry) not in new_ids]
|
||||
|
||||
|
||||
def rewalk_interval(last_duration) -> float:
|
||||
"""Seconds from a walk's start to the next background walk of its root."""
|
||||
scaled = REWALK_FACTOR * (last_duration or 0.0)
|
||||
return min(REWALK_CAP, max(REWALK_FLOOR, scaled))
|
||||
|
||||
|
||||
def root_contains(resolved_root: str, path, pathmod=os.path) -> bool:
|
||||
"""True when the resolved *path* lies under *resolved_root* (no file I/O).
|
||||
|
||||
*resolved_root* is already ``normcase``d; *path* is folded the same way,
|
||||
so Windows matches case-insensitively and with either separator. The
|
||||
match ends at a separator: ``/out2/x.png`` is not under ``/out``.
|
||||
"""
|
||||
target = _plain(pathmod.normcase(str(path)), pathmod)
|
||||
prefix = _plain(resolved_root, pathmod)
|
||||
if not prefix.endswith(pathmod.sep):
|
||||
prefix += pathmod.sep
|
||||
return target.startswith(prefix)
|
||||
|
||||
|
||||
def _plain(path: str, pathmod) -> str:
|
||||
"""*path* (normcased or not) without a Windows extended-length prefix.
|
||||
|
||||
``realpath`` may keep ``\\\\?\\`` on a file past MAX_PATH while its shorter
|
||||
root resolves without it.
|
||||
"""
|
||||
if pathmod.sep != "\\":
|
||||
return path
|
||||
if path[:8].lower() == "\\\\?\\unc\\":
|
||||
return "\\\\" + path[8:]
|
||||
if path.startswith("\\\\?\\"):
|
||||
return path[4:]
|
||||
return path
|
||||
|
||||
|
||||
def resolve_root(root) -> str:
|
||||
"""``normcase(realpath(root))``, the spelling monitor paths are matched in.
|
||||
|
||||
File I/O (realpath follows links): call it in the executor.
|
||||
"""
|
||||
return os.path.normcase(os.path.realpath(str(root)))
|
||||
|
||||
|
||||
def same_listing(old, new) -> bool:
|
||||
@@ -130,8 +319,10 @@ def same_listing(old, new) -> bool:
|
||||
|
||||
def walk_root(
|
||||
scan_fn: Callable[[PurePath], list], root, previous_files
|
||||
) -> tuple[list, frozenset, frozenset]:
|
||||
"""The single executor job of a walk: ``(files, folders, with_ancestors)``.
|
||||
) -> tuple[list, frozenset, frozenset, str]:
|
||||
"""The single executor job of a walk.
|
||||
|
||||
Returns ``(files, folders, with_ancestors, resolved_root)``.
|
||||
|
||||
Opens the root first, so a missing or unreadable root raises ``OSError``
|
||||
instead of listing nothing. Returns *previous_files* itself when nothing
|
||||
@@ -143,4 +334,4 @@ def walk_root(
|
||||
if same_listing(previous_files, files):
|
||||
files = previous_files
|
||||
folders, with_ancestors = folder_sets(root, files)
|
||||
return files, folders, with_ancestors
|
||||
return files, folders, with_ancestors, resolve_root(root)
|
||||
|
||||
+392
-33
@@ -10,10 +10,12 @@ import io
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import stat
|
||||
import subprocess # nosec B404 - ffmpeg frames through run_encoder, fixed argv
|
||||
import sys
|
||||
import time as _time
|
||||
import urllib.parse
|
||||
from bisect import bisect_left, bisect_right
|
||||
from bisect import bisect_left, bisect_right, insort
|
||||
from collections.abc import Mapping
|
||||
from pathlib import Path
|
||||
from typing import Callable, NamedTuple
|
||||
@@ -589,6 +591,10 @@ def _workspace_key(entry: tuple, order: str) -> tuple:
|
||||
return (mtime, entry.name, entry.tie, entry.index)
|
||||
|
||||
|
||||
def _newest_key(entry: _Listed) -> tuple:
|
||||
return _workspace_key(entry, "newest")
|
||||
|
||||
|
||||
def _rated_key(entry: _Listed, rank: int) -> tuple:
|
||||
"""Highest rated: rating 5..1, linked unrated (0), unlinked (-1); then newest."""
|
||||
return (-rank, *_workspace_key(entry, "rating"))
|
||||
@@ -764,6 +770,34 @@ def _favorite_relative_key(relative: str) -> str:
|
||||
return _path_flavor.normcase(relative).replace(_path_flavor.sep, "/")
|
||||
|
||||
|
||||
def _listing_posix(root, prefix, path) -> str:
|
||||
"""*path*'s POSIX spelling below *root* (*prefix*: ``join(root, "")``)."""
|
||||
text = str(path)
|
||||
if text.startswith(prefix):
|
||||
relative = text[len(prefix) :]
|
||||
return relative if os.sep == "/" else relative.replace(os.sep, "/")
|
||||
return path.relative_to(root).as_posix()
|
||||
|
||||
|
||||
def _favorite_listing_step(root, listing, added, removed) -> dict | None:
|
||||
"""*listing* after a patched listing's change, or None to rebuild it.
|
||||
|
||||
Only when every removed file is added again under the same spelling (a
|
||||
rewritten output) and no added file's key is already taken by another
|
||||
spelling: the full build keeps the first spelling in listing order.
|
||||
"""
|
||||
prefix = os.path.join(str(root), "")
|
||||
spellings = {_listing_posix(root, prefix, path) for path, _mtime in added}
|
||||
if any(_listing_posix(root, prefix, path) not in spellings for path, _ in removed):
|
||||
return None
|
||||
listing = dict(listing)
|
||||
for posix in spellings:
|
||||
known = listing.setdefault(_favorite_relative_key(posix), posix)
|
||||
if known != posix:
|
||||
return None
|
||||
return listing
|
||||
|
||||
|
||||
def _favorite_root_key(root) -> str:
|
||||
"""Server-side identity of a gallery root: its real, case-normalized path.
|
||||
|
||||
@@ -827,6 +861,91 @@ def _walk_media(output_path, max_files):
|
||||
stack.append((entry.path, rel_parts + (entry.name,)))
|
||||
|
||||
|
||||
def _settled_entry(root, relative):
|
||||
"""``(Path, mtime)`` of a settled output as a walk would list it.
|
||||
|
||||
*relative* is its part below *root* (``gallery_listing.relative_part``).
|
||||
None when a walk would not list it (``_walk_media``'s filters, and every
|
||||
folder on the way must be listable); raises ``OSError`` when that cannot
|
||||
be told (the file is gone or unreadable, or its on-disk spelling differs
|
||||
or cannot be read). Blocking: run it in the executor.
|
||||
"""
|
||||
path = Path(os.path.join(os.fspath(root), relative))
|
||||
info = os.lstat(path)
|
||||
if stat.S_ISLNK(info.st_mode) or stat.S_ISDIR(info.st_mode):
|
||||
return None
|
||||
# Before the name filters: they compare the walk's (stored) spelling.
|
||||
on_disk = _on_disk_path(path)
|
||||
if on_disk is not None and not on_disk.endswith(os.sep + relative):
|
||||
raise OSError(f"{relative} is spelled differently on disk")
|
||||
*folders, name = relative.split(os.sep)
|
||||
if _name_suffix(name).lower() not in MEDIA_EXTENSIONS:
|
||||
return None
|
||||
if "thumbnails" in folders or len(folders) > MAX_SCAN_DEPTH:
|
||||
return None
|
||||
folder = os.fspath(root)
|
||||
for part in ("", *folders):
|
||||
folder = os.path.join(folder, part) if part else folder
|
||||
if not os.access(folder, os.R_OK | os.X_OK):
|
||||
return None # the walk cannot list this folder
|
||||
return path, info.st_mtime
|
||||
|
||||
|
||||
def _on_disk_path(path, platform=None):
|
||||
"""The file's path as its folders store the names, where that can differ.
|
||||
|
||||
Windows and macOS file systems ignore case. Windows' ``realpath`` gives
|
||||
the stored case, but keeps the given one when it cannot open the file;
|
||||
macOS' ``realpath`` always keeps the given one. A walk lists the names as
|
||||
stored, so an add checks them: Windows through ``GetFinalPathNameByHandle``
|
||||
(``nt._getfinalpathname``), macOS through ``F_GETPATH``. Raises OSError
|
||||
when it cannot be read; None elsewhere (case-sensitive file systems).
|
||||
"""
|
||||
platform = platform or sys.platform
|
||||
if platform == "win32":
|
||||
import nt
|
||||
|
||||
return nt._getfinalpathname(os.fspath(path))
|
||||
if platform != "darwin":
|
||||
return None
|
||||
import fcntl
|
||||
|
||||
# Non-blocking: a FIFO named like an image must not hold the executor.
|
||||
descriptor = os.open(path, os.O_RDONLY | getattr(os, "O_NONBLOCK", 0))
|
||||
try:
|
||||
raw = fcntl.fcntl(descriptor, fcntl.F_GETPATH, bytes(1024))
|
||||
finally:
|
||||
os.close(descriptor)
|
||||
return os.fsdecode(raw.split(b"\0", 1)[0])
|
||||
|
||||
|
||||
def _patch_listing(root, files, folders, with_ancestors, relatives):
|
||||
"""The listing with each settled output in *relatives* added (blocking).
|
||||
|
||||
Returns ``(files, folders, with_ancestors, unresolved)``; *files* is the
|
||||
same list when nothing changed. ``unresolved`` is True when an output
|
||||
could not be placed exactly, so the root needs a walk.
|
||||
"""
|
||||
unresolved = False
|
||||
for relative in relatives:
|
||||
try:
|
||||
entry = _settled_entry(root, relative)
|
||||
except OSError:
|
||||
unresolved = True
|
||||
continue
|
||||
if entry is None:
|
||||
continue # a walk would not list it either
|
||||
patched = gallery_listing.insert_entry(files, entry, MAX_GALLERY_FILES)
|
||||
if patched is None:
|
||||
unresolved = True
|
||||
elif patched is not files:
|
||||
files = patched
|
||||
folders, with_ancestors = gallery_listing.add_folder(
|
||||
root, entry[0], folders, with_ancestors
|
||||
)
|
||||
return files, folders, with_ancestors, unresolved
|
||||
|
||||
|
||||
def _iter_media_files(output_path, max_files):
|
||||
"""Yield the ``Path`` of each media file under *output_path* (see _walk_media)."""
|
||||
for entry in _walk_media(output_path, max_files):
|
||||
@@ -1408,6 +1527,7 @@ class ImageRoutesMixin:
|
||||
even if a previous listing exists.
|
||||
"""
|
||||
record = self._root_listing(output_path)
|
||||
record.retired = False # a configured root: read, so listened to
|
||||
needed = record.walk_seq + 1 if fresh else 0
|
||||
discarded = 0
|
||||
while True:
|
||||
@@ -1433,28 +1553,172 @@ class ImageRoutesMixin:
|
||||
)
|
||||
|
||||
def _refresh_if_stale(self, output_path, record):
|
||||
"""Walk a stale listing again, REFRESH_DELAY after this read (on the loop).
|
||||
"""Ask for a background walk when the listing is stale or dirty.
|
||||
|
||||
At most one pending refresh per root, none while a walk runs.
|
||||
A stale listing (older than REFRESH_AFTER) is walked REFRESH_DELAY
|
||||
after this read, so the walk does not share the GIL with the page's
|
||||
own requests; a dirty one as soon as its interval allows.
|
||||
"""
|
||||
age = _time.monotonic() - (record.walked_at or 0.0)
|
||||
if age < gallery_listing.REFRESH_AFTER:
|
||||
if age >= gallery_listing.REFRESH_AFTER:
|
||||
self._schedule_walk(output_path, gallery_listing.REFRESH_DELAY)
|
||||
elif record.dirty:
|
||||
self._schedule_walk(output_path)
|
||||
|
||||
def output_settled(self, path):
|
||||
"""The monitor settled an output at *path*: list it (on the loop).
|
||||
|
||||
*path* is resolved by the monitor's thread (``notify_output_settled``).
|
||||
A root with a stored listing and no walk running gets the file added
|
||||
in the executor (``_run_adds``), without a walk. Every other root
|
||||
holding it is marked dirty (``mark_root_dirty``): one with a walk
|
||||
running gets the trailing walk, as that walk may have passed the
|
||||
file's folder already.
|
||||
"""
|
||||
for key, record in self._settled_records(path):
|
||||
relative = None
|
||||
if (
|
||||
record.files is not None
|
||||
and record.flight is None
|
||||
and record.resolved_root is not None
|
||||
):
|
||||
relative = gallery_listing.relative_part(record.resolved_root, path)
|
||||
if relative is None:
|
||||
self._mark_dirty(key, record)
|
||||
continue
|
||||
record.pending_adds.append(relative)
|
||||
if record.add_task is None:
|
||||
task = asyncio.get_running_loop().create_task(
|
||||
self._run_adds(Path(key), record)
|
||||
)
|
||||
record.add_task = task
|
||||
self._listing_tasks.add(task)
|
||||
task.add_done_callback(self._listing_tasks.discard)
|
||||
|
||||
def mark_root_dirty(self, path):
|
||||
"""A settled output at *path*: walk every root holding it again (on the loop).
|
||||
|
||||
*path* is resolved by the monitor's thread (``notify_output_settled``),
|
||||
so this does no file I/O: it compares ``normcase`` strings with each
|
||||
walk's ``resolved_root``. Only roots with a listing, or a walk running,
|
||||
are marked; a root without either is walked by its next read anyway.
|
||||
"""
|
||||
for key, record in self._settled_records(path):
|
||||
self._mark_dirty(key, record)
|
||||
|
||||
def _settled_records(self, path):
|
||||
"""``(key, record)`` of every listed or walking root holding *path*."""
|
||||
if self._listings_closed:
|
||||
return []
|
||||
found = []
|
||||
for key, record in list(self._root_listings.items()):
|
||||
if record.retired or (record.files is None and record.flight is None):
|
||||
continue
|
||||
resolved = record.resolved_root or os.path.normcase(key)
|
||||
if gallery_listing.root_contains(resolved, path):
|
||||
found.append((key, record))
|
||||
return found
|
||||
|
||||
def _mark_dirty(self, key, record):
|
||||
if self._listings_closed or record.retired:
|
||||
return
|
||||
if record.flight is not None or record.rewalk_handle is not None:
|
||||
if record.files is None and record.flight is None:
|
||||
return # walked by its next read anyway
|
||||
record.dirty = True
|
||||
self._schedule_walk(Path(key))
|
||||
|
||||
async def _run_adds(self, root, record):
|
||||
"""Add the root's pending settled outputs to its listing (on the loop).
|
||||
|
||||
Each batch is placed in the executor against the stored listing, and
|
||||
stored only when no walk started and no drop happened meanwhile: a
|
||||
walk started after the outputs settled lists them itself, and a
|
||||
dropped listing is walked by its next read. An output that cannot be
|
||||
placed exactly marks the root dirty instead.
|
||||
"""
|
||||
key = str(root)
|
||||
try:
|
||||
while record.pending_adds:
|
||||
relatives, record.pending_adds = record.pending_adds, []
|
||||
if record.files is None or record.flight is not None:
|
||||
continue # dropped, or a walk started after they settled
|
||||
base, generation, seq = record.files, record.generation, record.walk_seq
|
||||
files, folders, with_ancestors, unresolved = (
|
||||
await self._run_in_executor(
|
||||
_patch_listing,
|
||||
root,
|
||||
base,
|
||||
record.folders,
|
||||
record.folders_with_ancestors,
|
||||
relatives,
|
||||
)
|
||||
)
|
||||
if self._listings_closed or record.retired:
|
||||
return
|
||||
if record.generation != generation or record.walk_seq != seq:
|
||||
continue
|
||||
if files is not base:
|
||||
record.files = files
|
||||
record.folders = folders
|
||||
record.folders_with_ancestors = with_ancestors
|
||||
record.patched += 1
|
||||
self._schedule_workspace_warm()
|
||||
if unresolved:
|
||||
self._mark_dirty(key, record)
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except Exception as exc:
|
||||
self.logger.warning(f"Could not add a new output to the listing: {exc}")
|
||||
record.pending_adds = []
|
||||
self._mark_dirty(key, record)
|
||||
finally:
|
||||
record.add_task = None
|
||||
|
||||
def _schedule_walk(self, output_path, delay=0.0):
|
||||
"""Start or defer a background walk of *output_path* (on the loop).
|
||||
|
||||
- A walk is running: nothing; its end re-checks ``dirty``.
|
||||
- Otherwise the walk is due *delay* from now, and never sooner than
|
||||
``rewalk_interval`` after the previous walk started. Due now: start
|
||||
it. Later: one deferred walk per root, at least MIN_REWALK_DELAY
|
||||
away; when one is already pending, the earlier deadline wins.
|
||||
Every walk start cancels the pending one (``_start_walk``).
|
||||
"""
|
||||
if self._listings_closed:
|
||||
return
|
||||
record.rewalk_handle = asyncio.get_running_loop().call_later(
|
||||
gallery_listing.REFRESH_DELAY, self._delayed_walk, output_path, record
|
||||
record = self._root_listing(output_path)
|
||||
if record.flight is not None or record.retired:
|
||||
return
|
||||
now = _time.monotonic()
|
||||
due = now + delay
|
||||
if record.last_started_at is not None:
|
||||
interval = gallery_listing.rewalk_interval(record.last_duration)
|
||||
due = max(due, record.last_started_at + interval)
|
||||
if due <= now:
|
||||
self._start_walk(output_path, record)
|
||||
return
|
||||
wait = max(gallery_listing.MIN_REWALK_DELAY, due - now)
|
||||
loop = asyncio.get_running_loop()
|
||||
handle = record.rewalk_handle
|
||||
if handle is not None:
|
||||
if handle.when() - loop.time() <= wait:
|
||||
return # the pending walk comes first
|
||||
handle.cancel()
|
||||
record.rewalk_handle = loop.call_later(
|
||||
wait, self._delayed_walk, output_path, record
|
||||
)
|
||||
|
||||
def _delayed_walk(self, output_path, record):
|
||||
record.rewalk_handle = None
|
||||
self._schedule_walk(output_path)
|
||||
"""The deferred walk's timer: clear the handle first, then walk.
|
||||
|
||||
def _schedule_walk(self, output_path):
|
||||
"""Start a background walk of *output_path* unless one is running."""
|
||||
record = self._root_listing(output_path)
|
||||
if record.flight is None:
|
||||
self._start_walk(output_path, record)
|
||||
It was armed for a stale or dirty listing at the earliest moment the
|
||||
interval allowed. Only a walk start clears either reason, and every
|
||||
start cancels this timer, so when it fires the walk is still due.
|
||||
"""
|
||||
record.rewalk_handle = None
|
||||
if self._listings_closed or record.flight is not None or record.retired:
|
||||
return
|
||||
self._start_walk(output_path, record)
|
||||
|
||||
def _start_walk(self, output_path, record):
|
||||
"""Start the root's one walk; returns its flight future (on the loop)."""
|
||||
@@ -1462,7 +1726,11 @@ class ImageRoutesMixin:
|
||||
self._listing_loop = loop
|
||||
handle, record.rewalk_handle = record.rewalk_handle, None
|
||||
if handle is not None:
|
||||
handle.cancel() # this walk is the refresh it was waiting to start
|
||||
handle.cancel() # this walk is the one it was waiting to start
|
||||
# This walk lists every output reported so far.
|
||||
record.dirty = False
|
||||
record.pending_adds = []
|
||||
record.last_started_at = _time.monotonic()
|
||||
record.walk_seq += 1
|
||||
flight = loop.create_future()
|
||||
record.flight = flight
|
||||
@@ -1482,11 +1750,14 @@ class ImageRoutesMixin:
|
||||
error never reaches a future nobody awaits.
|
||||
"""
|
||||
outcome = None
|
||||
cancelled = False
|
||||
started = record.last_started_at
|
||||
try:
|
||||
files, folders, with_ancestors = await self._run_in_executor(
|
||||
files, folders, with_ancestors, resolved = await self._run_in_executor(
|
||||
walk_root, self._scan_gallery_files_sync, output_path, record.files
|
||||
)
|
||||
except asyncio.CancelledError:
|
||||
cancelled = True
|
||||
outcome = WalkOutcome.failed(RuntimeError(WALK_STOPPED))
|
||||
raise
|
||||
except Exception as exc:
|
||||
@@ -1496,11 +1767,15 @@ class ImageRoutesMixin:
|
||||
f"Could not list output folder {output_path}: {exc}"
|
||||
)
|
||||
record.failing = True
|
||||
# Retried by the 30 s rule, not by a trailing walk.
|
||||
record.dirty = False
|
||||
outcome = WalkOutcome.failed(exc)
|
||||
else:
|
||||
record.failing = False
|
||||
record.resolved_root = resolved
|
||||
if record.generation == generation:
|
||||
changed = files is not record.files
|
||||
record.patched = 0
|
||||
record.files = files
|
||||
record.folders = folders
|
||||
record.folders_with_ancestors = with_ancestors
|
||||
@@ -1512,10 +1787,14 @@ class ImageRoutesMixin:
|
||||
else:
|
||||
outcome = WalkOutcome.discarded()
|
||||
finally:
|
||||
record.last_duration = _time.monotonic() - started
|
||||
if record.flight is flight:
|
||||
record.flight = None
|
||||
if not flight.done():
|
||||
flight.set_result(outcome)
|
||||
if record.dirty and not cancelled:
|
||||
# An output settled during this walk: the trailing walk.
|
||||
self._schedule_walk(output_path)
|
||||
|
||||
async def get_output_images(self, request):
|
||||
"""Get all images from ComfyUI output folder(s)."""
|
||||
@@ -1644,6 +1923,10 @@ class ImageRoutesMixin:
|
||||
cached = self._workspace_listed_cache.get(key)
|
||||
if cached is not None and cached[0] is files:
|
||||
return cached[1]
|
||||
if cached is not None:
|
||||
delta = gallery_listing.listing_delta(cached[0], files)
|
||||
if delta is not None:
|
||||
return self._workspace_listed_step(key, cached, files, delta)
|
||||
previous = cached[2] if cached is not None else {}
|
||||
listed = []
|
||||
for path, mtime in files:
|
||||
@@ -1663,7 +1946,42 @@ class ImageRoutesMixin:
|
||||
self._workspace_card_cache[key] = {
|
||||
file: facts for file, facts in cards.items() if file in lookup
|
||||
}
|
||||
self._workspace_listed_cache[key] = (files, listed, lookup)
|
||||
self._workspace_listed_cache[key] = (files, listed, lookup, None)
|
||||
return listed
|
||||
|
||||
def _workspace_listed_step(self, key, cached, files, delta) -> list:
|
||||
"""The _Listed values after a few files were added or replaced.
|
||||
|
||||
A listing patched by a settled output (``output_settled``) shares
|
||||
every other entry with the cached one, so only the added files are
|
||||
built; the order of the values does not matter (every sort is a total
|
||||
order). The step ``(previous values, added, removed)`` lets the sort
|
||||
orders follow it the same way (``_workspace_orders``).
|
||||
"""
|
||||
index, root = key[0], Path(key[1])
|
||||
_files, old_listed, old_lookup, _step = cached
|
||||
added, removed = delta
|
||||
if not added and not removed:
|
||||
self._workspace_listed_cache[key] = (files, old_listed, old_lookup, None)
|
||||
return old_listed
|
||||
lookup = dict(old_lookup)
|
||||
gone = [lookup.pop(item) for item in removed if item in lookup]
|
||||
fresh = [_listed(path, mtime, root, index) for path, mtime in added]
|
||||
for entry in fresh:
|
||||
lookup[(entry.path, entry.mtime)] = entry
|
||||
if gone:
|
||||
gone_ids = {id(entry) for entry in gone}
|
||||
listed = [entry for entry in old_listed if id(entry) not in gone_ids]
|
||||
cards = self._workspace_card_cache.get(key)
|
||||
if cards:
|
||||
self._workspace_card_cache[key] = {
|
||||
file: facts for file, facts in cards.items() if file in lookup
|
||||
}
|
||||
else:
|
||||
listed = list(old_listed)
|
||||
listed.extend(fresh)
|
||||
step = (old_listed, fresh, gone)
|
||||
self._workspace_listed_cache[key] = (files, listed, lookup, step)
|
||||
return listed
|
||||
|
||||
def _workspace_orders(self, listings: list[tuple]) -> dict:
|
||||
@@ -1678,14 +1996,54 @@ class ImageRoutesMixin:
|
||||
old is new for old, new in zip(cached[0], parts, strict=True)
|
||||
):
|
||||
return cached[1]
|
||||
newest = sorted(
|
||||
(entry for part in parts for entry in part),
|
||||
key=lambda entry: _workspace_key(entry, "newest"),
|
||||
)
|
||||
newest = None
|
||||
if cached is not None:
|
||||
newest = self._newest_step(listings, parts, cached)
|
||||
if newest is None:
|
||||
newest = sorted(
|
||||
(entry for part in parts for entry in part),
|
||||
key=_newest_key,
|
||||
)
|
||||
# The other orders, the wanted paths and the locators follow lazily.
|
||||
orders = {"newest": newest}
|
||||
self._workspace_order_cache[key] = (parts, orders)
|
||||
return orders
|
||||
|
||||
def _newest_step(self, listings, parts, cached) -> list | None:
|
||||
"""The cached newest order with each part's last step applied, or None.
|
||||
|
||||
Only when every changed part is exactly the value list this call
|
||||
built (*parts*) and one step (``_workspace_listed_step``) past the
|
||||
part the cached order was built from: the added values are placed by
|
||||
bisection instead of sorting the whole listing again. Page reads and
|
||||
the warm build run in executor threads at once, so the per-file cache
|
||||
may already hold a newer step than *parts*; then this sorts in full.
|
||||
"""
|
||||
old_parts, orders = cached
|
||||
added, gone = [], []
|
||||
for (index, root, _files), part, old in zip(
|
||||
listings, parts, old_parts, strict=True
|
||||
):
|
||||
if part is old:
|
||||
continue
|
||||
listed = self._workspace_listed_cache.get((index, str(root)))
|
||||
if listed is None or listed[1] is not part:
|
||||
return None
|
||||
step = listed[3]
|
||||
if step is None or step[0] is not old:
|
||||
return None
|
||||
added.extend(step[1])
|
||||
gone.extend(step[2])
|
||||
newest = orders["newest"]
|
||||
if gone:
|
||||
gone_ids = {id(entry) for entry in gone}
|
||||
newest = [entry for entry in newest if id(entry) not in gone_ids]
|
||||
else:
|
||||
newest = list(newest)
|
||||
for entry in added:
|
||||
insort(newest, entry, key=_newest_key)
|
||||
return newest
|
||||
|
||||
def _workspace_wanted(self, listings: list[tuple]) -> frozenset:
|
||||
"""The selected listing's normalized paths (cached with its orders)."""
|
||||
orders = self._workspace_orders(listings)
|
||||
@@ -2967,18 +3325,19 @@ class ImageRoutesMixin:
|
||||
cached = self._favorite_listing_cache.get(cache_key)
|
||||
if cached is not None and cached[0] is files:
|
||||
return cached[1]
|
||||
# From the path strings, as _listed does: relative_to per file was the
|
||||
# largest cost of a cold first open (#348).
|
||||
prefix = os.path.join(str(root), "")
|
||||
listing = {}
|
||||
for path, _mtime in files:
|
||||
text = str(path)
|
||||
if text.startswith(prefix):
|
||||
relative = text[len(prefix) :]
|
||||
posix = relative if os.sep == "/" else relative.replace(os.sep, "/")
|
||||
else:
|
||||
posix = path.relative_to(root).as_posix()
|
||||
listing.setdefault(_favorite_relative_key(posix), posix)
|
||||
listing = None
|
||||
if cached is not None:
|
||||
delta = gallery_listing.listing_delta(cached[0], files)
|
||||
if delta is not None:
|
||||
listing = _favorite_listing_step(root, cached[1], *delta)
|
||||
if listing is None:
|
||||
# From the path strings, as _listed does: relative_to per file was
|
||||
# the largest cost of a cold first open (#348).
|
||||
prefix = os.path.join(str(root), "")
|
||||
listing = {}
|
||||
for path, _mtime in files:
|
||||
posix = _listing_posix(root, prefix, path)
|
||||
listing.setdefault(_favorite_relative_key(posix), posix)
|
||||
self._favorite_listing_cache[cache_key] = (files, listing)
|
||||
return listing
|
||||
|
||||
|
||||
Reference in New Issue
Block a user