Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
34dabe2cd2 |
@@ -1,9 +1,9 @@
|
||||
name: Windows and macOS
|
||||
name: Windows
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
# Non-blocking portability runs (#420): never on pull requests or in the merge queue, and no context
|
||||
# Non-blocking Windows runs (#420; macOS dropped from CI in #571): never on pull requests or in the merge queue, and no context
|
||||
# here is a required check. A red run is fixed at its cause before the workspace cutover (#334).
|
||||
on:
|
||||
push:
|
||||
@@ -25,10 +25,7 @@ env:
|
||||
# run counts (tests/vendor/ci-workflow.test.mjs recomputes it, #493).
|
||||
# Measured 2026-10-29 on main runs 40935337099 (dd247e3), 40938407252 (dd247e3), 40941240624 (dd247e3):
|
||||
# slowest leg: 7m24s (py3.12 windows) / 7m31s (py3.13 windows) / 7m32s (py3.14 windows) -> 15 min
|
||||
# (windows 4m56s-7m32s, macos 4m39s-6m14s)
|
||||
# native-trash (#499) has no green main runs yet: its probe, run 40949370943 (2026-10-29), took
|
||||
# about 1 minute per macOS leg, so ceil(1.5 x 2) + 3 gives it 6 minutes (step 4). Recalibrate
|
||||
# after three green runs.
|
||||
# (windows 4m56s-7m32s)
|
||||
# native-trash-linux (#543) has no green main runs yet: it installs the dev requirements and runs
|
||||
# one test module, about 1 minute by the same estimate, so ceil(1.5 x 2) + 3 gives it 6 minutes.
|
||||
# Recalibrate after three green runs.
|
||||
@@ -40,7 +37,7 @@ jobs:
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
os: [windows-latest, macos-latest]
|
||||
os: [windows-latest]
|
||||
python-version: ["3.12", "3.13", "3.14"]
|
||||
|
||||
steps:
|
||||
@@ -60,7 +57,7 @@ jobs:
|
||||
|
||||
# The runner's TEMP is on C:, the OS disk, where a flush took about 6 ms
|
||||
# and a test database's create-to-delete about 111 ms; on the D: work
|
||||
# disk the same took 0.5 ms and 23 ms (macOS: 19 ms). Tests write their
|
||||
# disk the same took 0.5 ms and 23 ms. Tests write their
|
||||
# databases under TEMP, so it moves to a folder on D: (measured in #487,
|
||||
# run 40915866444). The runner's TEMP was an 8.3 short name
|
||||
# (C:\Users\RUNNER~1\...), so the new one is spelled by its short name too
|
||||
@@ -83,31 +80,6 @@ jobs:
|
||||
PM_TEST_SLOWEST: "40"
|
||||
run: python -m tests.run_suite -v
|
||||
|
||||
# The real Trash on both macOS CPUs (#499 design §10.2). It runs apart from the suite, so a red
|
||||
# suite never skips the native proof.
|
||||
native-trash:
|
||||
name: native trash on ${{ matrix.os }}
|
||||
runs-on: ${{ matrix.os }}
|
||||
timeout-minutes: 6
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
os: [macos-latest, macos-15-intel]
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v7
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v7
|
||||
with:
|
||||
python-version: "3.12"
|
||||
|
||||
- name: Native trash proof
|
||||
timeout-minutes: 4
|
||||
env:
|
||||
PM_NATIVE_TRASH: macos
|
||||
run: python -m unittest tests.test_trash_macos_native tests.test_trash_macos -v
|
||||
|
||||
# The same tests as the fake mount table, on a real ext4 mount used as the output root (#543). A
|
||||
# runner can mount with sudo; the tests run as the normal user, so owner and mode checks are real.
|
||||
native-trash-linux:
|
||||
|
||||
@@ -28,7 +28,7 @@ env:
|
||||
# py matrix, slowest py3.10 on windows-latest: 12m43s / 15m54s / 14m53s -> 27 min
|
||||
# (ubuntu 5m27s-6m27s, macos 6m14s-8m30s, windows 11m42s-15m54s)
|
||||
# 3.10 and 3.11 left the matrix and 3.14 joined it in #356; re-measure after three green main runs.
|
||||
# Windows and macOS moved to portability.yml (#420), which has its own measured limit (#493);
|
||||
# Windows moved to portability.yml (#420; macOS left CI in #571), which has its own measured limit (#493);
|
||||
# this Linux-only matrix keeps 27 minutes until it is re-measured.
|
||||
# real ComfyUI compatibility: 13m32s / 8m55s / 11m46s -> 24 min
|
||||
# coverage gate (85%): 7m26s / 7m22s / 7m54s -> 15 min
|
||||
@@ -39,7 +39,7 @@ env:
|
||||
# Pull requests run a fast subset (#420): Python 3.12 on ubuntu-latest, coverage, lint and frontend
|
||||
# modules. The merge queue (merge_group), pushes to main and manual runs add Python 3.13 and 3.14 on
|
||||
# Linux and the real ComfyUI job. "all tests" is the required gate that holds the queue to that set.
|
||||
# Windows and macOS run in portability.yml (push to main, nightly, on demand) and never gate a merge.
|
||||
# Windows runs in portability.yml (push to main, nightly, on demand) and never gate a merge.
|
||||
jobs:
|
||||
test:
|
||||
name: py${{ matrix.python-version }} on ${{ matrix.os }}
|
||||
|
||||
@@ -236,12 +236,12 @@ The documentation for those pages is in [README-legacy.md](README-legacy.md).
|
||||
|
||||
| Layer | What runs |
|
||||
| --- | --- |
|
||||
| Python | Over 200 test files on Linux, Windows and macOS with Python 3.12 to 3.14, with an 85% coverage gate |
|
||||
| Python | Over 200 test files on Linux and Windows with Python 3.12 to 3.14, with an 85% coverage gate |
|
||||
| Frontend | About 100 unit test files, with coverage floors per file |
|
||||
| Browser | Over 60 Playwright spec files in Chromium, Firefox, WebKit and touch emulation |
|
||||
| Real ComfyUI | A pinned CPU ComfyUI, started in Classic and Nodes 2.0, running real workflows |
|
||||
|
||||
Linux runs on every pull request and in the merge queue. Windows and macOS run on every push to main and nightly. To run the tests yourself, see [TESTING.md](TESTING.md).
|
||||
Linux runs on every pull request and in the merge queue. Windows runs on every push to main and nightly. macOS is not tested; it should work, but support is best-effort. To run the tests yourself, see [TESTING.md](TESTING.md).
|
||||
|
||||
## License
|
||||
|
||||
|
||||
+14
-24
@@ -261,7 +261,7 @@ the process exits non-zero.
|
||||
job fails anyway and the dump is what matters.
|
||||
- `PM_TEST_SLOWEST=N` (off by default) lists the N slowest tests after the
|
||||
summary, each timed from its `start:` line to the next one like the hang
|
||||
limit. The Windows and macOS jobs (`portability.yml`) set it to 40, so every
|
||||
limit. The Windows jobs (`portability.yml`) set it to 40, so every
|
||||
run shows where its time went (#487). The list prints only when the run
|
||||
ends: a job GitHub cancels or a watchdog exit prints none, so read the
|
||||
timestamps between `start:` lines instead.
|
||||
@@ -327,7 +327,7 @@ Python 3.13/3.14, real ComfyUI and Firefox/WebKit failures first show up in the
|
||||
merge queue, on the merge-group commit rather than the
|
||||
pushed one. Find that run with `gh run list --event merge_group --workflow test.yml`
|
||||
(or `viewerjs-check.yml`) and match its head branch
|
||||
(`gh-readonly-queue/main/pr-<N>-...`). Windows and macOS failures show up only
|
||||
(`gh-readonly-queue/main/pr-<N>-...`). Windows failures show up only
|
||||
after the merge, in `gh run list --workflow portability.yml`.
|
||||
|
||||
Start fixing as soon as a failed log shows the cause, but push the fix only after
|
||||
@@ -345,10 +345,8 @@ and the time left. The steps, in order:
|
||||
monitor's callbacks (Needs review recorder, preview queue, Images listing
|
||||
listener).
|
||||
2. `ImageMonitor.stop_monitoring` works against one 5 s deadline. Closing the
|
||||
image handler joins the image worker; watchdog's `observer.stop()` and the
|
||||
observer join get what is left, which can be 0 s. `stop()` joins watchdog's
|
||||
emitters without a limit, so it runs on a daemon thread and is waited for
|
||||
only until the deadline; a later stop waits for that same call (#561).
|
||||
image handler joins the image worker; watchdog's `observer.stop()` (not
|
||||
bounded here) and the observer join get what is left, which can be 0 s.
|
||||
3. `close_auto_thumbnails(final=True)` stops the preview worker (30 s) and
|
||||
kills ffmpeg.
|
||||
4. `PromptTracker.shutdown` stops the prompt cleanup thread (2 s).
|
||||
@@ -363,15 +361,6 @@ stored, with the parts not reached False, and the `atexit` callback is
|
||||
removed: the later steps are skipped, no summary line is logged, and a later
|
||||
call returns that result and runs nothing.
|
||||
|
||||
The observer is started from a short-lived daemon thread, so every thread it
|
||||
creates is a daemon (a thread inherits that flag from the thread that creates
|
||||
it). watchdog 7 on Windows starts a plain `threading.Thread` per watch for its
|
||||
`ReadDirectoryChangesW` reader; created from ComfyUI's main thread it would not
|
||||
be a daemon, and the interpreter waits for every such thread before it runs any
|
||||
`atexit` callback, so the teardown would never start (#561).
|
||||
`tests/test_package_exit.py` runs a child whose observer does both (a plain
|
||||
reader thread, a `stop()` that never returns) and requires its exit.
|
||||
|
||||
ComfyUI never runs aiohttp's `on_cleanup` hooks, because it never calls
|
||||
`runner.cleanup()`. So `atexit` is the live host's path. It runs on Ctrl+C
|
||||
(`main.py` catches `KeyboardInterrupt` and returns) and on a normal exit.
|
||||
@@ -461,7 +450,7 @@ measurement: `ubuntu-latest`, one Python, no coverage, only the wall-clock tests
|
||||
It runs on every push to `main` and on `workflow_dispatch`, never on pull
|
||||
requests, because shared-runner timing must not block a merge. Dispatch it on a
|
||||
branch to measure before merging.
|
||||
Shared Windows and macOS runners do not run them; the structural tests cover
|
||||
Shared Windows runners do not run them; the structural tests cover
|
||||
portability. Run it locally with
|
||||
`PROMPTMANAGER_PERF_BUDGET=1 python -m unittest tests.test_api_workspace_images.TestWorkspaceImages.test_filter_budget`
|
||||
and quote the printed p95 in the PR body.
|
||||
@@ -657,10 +646,10 @@ Before changing paths, SQLite, streams, restore or cleanup:
|
||||
ordinary busy timeout; cross-thread handoffs wait on events with generous
|
||||
ceilings, never a short wall clock (#242).
|
||||
- Run native Windows CI early for these changes. Pull requests and the merge
|
||||
queue run only Linux, so start Windows and macOS on the branch with
|
||||
queue run only Linux, so start Windows on the branch with
|
||||
`gh workflow run portability.yml --ref <branch>`.
|
||||
Include a Windows impact and evidence note in the PR. All six Windows/macOS
|
||||
combinations in that run, every Linux leg and every applicable
|
||||
Include a Windows impact and evidence note in the PR. All three Windows
|
||||
legs in that run, every Linux leg and every applicable
|
||||
exact-head check must pass before merge. Portable `ntpath`/`PureWindowsPath`
|
||||
controls do not establish native filesystem, locking or cleanup success.
|
||||
|
||||
@@ -679,7 +668,7 @@ what runs:
|
||||
| `page workflows (firefox\|webkit, shard N/2)` | skipped | run |
|
||||
| `all tests`, `viewerjs` (gates) | run | run |
|
||||
| `perf budget` | skipped | push and dispatch only (not in the queue) |
|
||||
| `py3.12-3.14 on windows-latest/macos-latest` (`portability.yml`) | not run | push to `main`, nightly, `workflow_dispatch` only; never required |
|
||||
| `py3.12-3.14 on windows-latest` (`portability.yml`) | not run | push to `main`, nightly, `workflow_dispatch` only; never required |
|
||||
|
||||
GitHub has one required set for pull requests and the queue. A job skipped by
|
||||
its `if:` reports Success. A dropped matrix leg never reports, so it would stay
|
||||
@@ -692,8 +681,9 @@ guarantee:
|
||||
So a merge group cannot turn green unless all three Linux Python legs, real
|
||||
ComfyUI and the four browser shards passed.
|
||||
|
||||
Windows and macOS (Python 3.12-3.14) are out of the blocking path for now
|
||||
(maintainer, 2026-10-28). They run in `portability.yml` on every push to `main`,
|
||||
Windows (Python 3.12-3.14) is out of the blocking path for now
|
||||
(maintainer, 2026-10-28). macOS is not tested in CI (maintainer, 2026-10-31, #571);
|
||||
its code stays and support is best-effort. They run in `portability.yml` on every push to `main`,
|
||||
nightly and on demand, and no context there is required. A red portability run
|
||||
is still a failure: fix it at its cause, not by rerunning, and have every one
|
||||
green before the workspace cutover (#334). `tests/vendor/ci-workflow.test.mjs` pins the
|
||||
@@ -2560,8 +2550,8 @@ files to the system trash. **No code path unlinks or overwrites a media file.**
|
||||
("Moving files to the Recycle Bin isn't available on Windows: files larger than the bin's
|
||||
capacity would be deleted permanently."). Library-only removal works everywhere. Tests use `tests/fake_trash.py`; the
|
||||
browser test server installs it, so no test touches a real trash. The native proof,
|
||||
`tests/test_trash_macos_native.py`, runs only with `PM_NATIVE_TRASH=macos` in
|
||||
portability.yml's `native-trash` job (`macos-latest` and `macos-15-intel`): the startup
|
||||
`tests/test_trash_macos_native.py`, runs only with `PM_NATIVE_TRASH=macos` on a Mac, by hand
|
||||
(CI dropped its `native-trash` macOS job in #571; it last passed on `macos-latest` and `macos-15-intel`): the startup
|
||||
volume, a name clash, fresh APFS and FAT32 disk images, an image used as an output root,
|
||||
a read-only image and an unwritable folder (design and probe:
|
||||
`docs/egm/499-macos-trash/`, local).
|
||||
|
||||
+1
-1
@@ -4,7 +4,7 @@ description = "A powerful ComfyUI custom node that extends the standard text enc
|
||||
version = "3.4.1"
|
||||
license = {file = "LICENSE"}
|
||||
requires-python = ">=3.12"
|
||||
dependencies = ["watchdog>=2.1.0", "Pillow>=8.0.0"]
|
||||
dependencies = ["watchdog>=2.1.0,<7", "Pillow>=8.0.0"]
|
||||
|
||||
[build-system]
|
||||
requires = ["setuptools>=84.2.1"]
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
# Runtime dependencies
|
||||
watchdog>=6.0.0
|
||||
watchdog>=6.0.0,<7
|
||||
Pillow>=12.3.0
|
||||
aiohttp>=3.14.5
|
||||
|
||||
|
||||
+1
-1
@@ -1,4 +1,4 @@
|
||||
# Runtime dependencies for PromptManager.
|
||||
# aiohttp, numpy and torch come from ComfyUI itself and are not listed here.
|
||||
watchdog>=2.1.0 # Output directory monitoring
|
||||
watchdog>=2.1.0,<7 # Output directory monitoring
|
||||
Pillow>=8.0.0 # Image metadata extraction and thumbnails
|
||||
|
||||
@@ -392,10 +392,7 @@ class TestStartStop(unittest.TestCase):
|
||||
self.assertLogs("prompt_manager.image_monitor", "WARNING") as logs,
|
||||
):
|
||||
self.assertFalse(self.monitor.stop_monitoring(timeout=5.0))
|
||||
# The stop runs on its own thread and gets 0 s too: it may not have
|
||||
# returned yet, and then nothing is joined at all.
|
||||
joins = [call for call in fake.calls if call[0] == "join"]
|
||||
self.assertTrue(all(call == ("join", 0.0) for call in joins), fake.calls)
|
||||
self.assertEqual(fake.calls[-1], ("join", 0.0))
|
||||
self.assertIn(
|
||||
"watchdog observer still running after 0.0 s", "\n".join(logs.output)
|
||||
)
|
||||
@@ -404,80 +401,6 @@ class TestStartStop(unittest.TestCase):
|
||||
def test_stop_monitoring_without_an_observer_returns_true(self):
|
||||
self.assertTrue(self.monitor.stop_monitoring())
|
||||
|
||||
def test_threads_the_observer_starts_are_daemons(self):
|
||||
# watchdog 7 on Windows starts a plain threading.Thread per watch in
|
||||
# observer.start(); a non-daemon one holds the interpreter's exit
|
||||
# before any atexit callback runs (#561).
|
||||
fake = ReaderObserver()
|
||||
self.addCleanup(fake.release.set)
|
||||
self._start_with(fake)
|
||||
self.assertIsNotNone(fake.reader)
|
||||
self.assertTrue(fake.reader.daemon)
|
||||
self.assertFalse(threading.current_thread().daemon)
|
||||
|
||||
def test_an_observer_that_fails_to_start_on_its_thread_is_reported(self):
|
||||
fake = FakeObserver()
|
||||
fake.start = MagicMock(side_effect=OSError("handle refused"))
|
||||
with self.assertLogs("prompt_manager.image_monitor", "ERROR") as logs:
|
||||
with patch.object(im_mod, "Observer", return_value=fake):
|
||||
self.monitor.start_monitoring([self.tmp.name])
|
||||
self.assertIn("handle refused", "\n".join(logs.output))
|
||||
self.assertFalse(self.monitor.is_monitoring)
|
||||
|
||||
def test_an_observer_stop_that_never_returns_is_bounded(self):
|
||||
# watchdog's observer.stop() joins its emitters (and, in 7.x on
|
||||
# Windows, their reader threads) without a limit (#561).
|
||||
fake = StuckStopObserver()
|
||||
self.addCleanup(fake.release.set)
|
||||
self._start_with(fake)
|
||||
with self.assertLogs("prompt_manager.image_monitor", "WARNING") as logs:
|
||||
self.assertFalse(self.monitor.stop_monitoring(timeout=0.05))
|
||||
self.assertIn("watchdog observer still running", "\n".join(logs.output))
|
||||
self.assertTrue(fake.entered.wait(30))
|
||||
self.assertFalse(self.monitor.is_monitoring)
|
||||
# A second stop waits for the stop already running; it never calls
|
||||
# observer.stop() a second time.
|
||||
fake.release.set()
|
||||
self.assertTrue(self.monitor.stop_monitoring())
|
||||
self.assertEqual(fake.stop_calls, 1)
|
||||
self.assertFalse(self.monitor._unfinished)
|
||||
|
||||
|
||||
class ReaderObserver(FakeObserver):
|
||||
"""A FakeObserver whose start() starts a plain thread, as watchdog 7's
|
||||
Windows emitter does for its ReadDirectoryChangesW reader."""
|
||||
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self.release = threading.Event()
|
||||
self.reader = None
|
||||
|
||||
def start(self):
|
||||
self.reader = threading.Thread(target=self.release.wait)
|
||||
self.reader.start()
|
||||
super().start()
|
||||
|
||||
def stop(self):
|
||||
self.release.set()
|
||||
self.reader.join(30)
|
||||
super().stop()
|
||||
|
||||
|
||||
class StuckStopObserver(FakeObserver):
|
||||
"""A FakeObserver whose stop() does not return until ``release``."""
|
||||
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self.entered = threading.Event()
|
||||
self.release = threading.Event()
|
||||
self.stop_calls = 0
|
||||
|
||||
def stop(self):
|
||||
self.stop_calls += 1
|
||||
self.entered.set()
|
||||
self.release.wait(30)
|
||||
super().stop()
|
||||
|
||||
|
||||
class FakeEvent:
|
||||
def __init__(self, path, is_directory=False):
|
||||
|
||||
@@ -587,16 +587,13 @@ PACKAGE_THREAD_NAMES = ("PromptManagerPromptCleanup", "PromptManagerImageWorker"
|
||||
|
||||
def package_threads():
|
||||
"""Live threads this design owns: the tracker, the image worker, and the
|
||||
watchdog observer and emitters (which have no name of their own).
|
||||
watchdog 7's InotifyFD, one daemon reader per process that no observer
|
||||
stop ends, is not the package's."""
|
||||
watchdog observer and emitters (which have no name of their own)."""
|
||||
from watchdog.utils import BaseThread
|
||||
|
||||
return {
|
||||
thread
|
||||
for thread in threading.enumerate()
|
||||
if thread.name in PACKAGE_THREAD_NAMES
|
||||
or (isinstance(thread, BaseThread) and type(thread).__name__ != "InotifyFD")
|
||||
if thread.name in PACKAGE_THREAD_NAMES or isinstance(thread, BaseThread)
|
||||
}
|
||||
|
||||
|
||||
|
||||
+44
-107
@@ -31,65 +31,22 @@ CHILD = textwrap.dedent("""
|
||||
import threading
|
||||
|
||||
root, tmp = sys.argv[1], sys.argv[2]
|
||||
stuck = sys.argv[3:] == ["stuck-observer"]
|
||||
sys.path.insert(0, root)
|
||||
state = {}
|
||||
|
||||
# watchdog 7's Windows observer at its worst (#561): start() starts a
|
||||
# plain reader thread, and stop() waits for it, which never ends.
|
||||
class StuckObserver:
|
||||
|
||||
def __init__(self, *args, **kwargs):
|
||||
self._never = threading.Event()
|
||||
self._reader = None
|
||||
|
||||
def schedule(self, *args, **kwargs):
|
||||
pass
|
||||
|
||||
def start(self):
|
||||
self._reader = threading.Thread(
|
||||
target=self._never.wait, name="StuckReader"
|
||||
)
|
||||
self._reader.start()
|
||||
|
||||
def stop(self):
|
||||
self._reader.join()
|
||||
|
||||
def join(self, timeout=None):
|
||||
self._reader.join(timeout)
|
||||
|
||||
def is_alive(self):
|
||||
return self._reader.is_alive()
|
||||
|
||||
if stuck:
|
||||
import watchdog.observers
|
||||
|
||||
watchdog.observers.Observer = StuckObserver
|
||||
|
||||
def report():
|
||||
from watchdog.utils import BaseThread
|
||||
|
||||
package = state.get("package")
|
||||
result = getattr(package, "_background_work_result", None)
|
||||
# The image worker, the tracker, and watchdog's observer and emitters.
|
||||
# watchdog 7's InotifyFD is one daemon reader per process that no
|
||||
# observer stop ends, by design; it is not the package's.
|
||||
alive = sorted(
|
||||
t.name
|
||||
for t in threading.enumerate()
|
||||
if t.name in ("PromptManagerImageWorker", "PromptManagerPromptCleanup")
|
||||
or (isinstance(t, BaseThread) and type(t).__name__ != "InotifyFD")
|
||||
)
|
||||
# Threads that would hold the exit before any atexit callback (#561).
|
||||
holding = sorted(
|
||||
t.name
|
||||
for t in threading.enumerate()
|
||||
if not t.daemon and t is not threading.main_thread()
|
||||
)
|
||||
print(
|
||||
"PM_EXIT_REPORT "
|
||||
+ json.dumps({"result": result, "alive": alive, "holding": holding})
|
||||
or isinstance(t, BaseThread)
|
||||
)
|
||||
print("PM_EXIT_REPORT " + json.dumps({"result": result, "alive": alive}))
|
||||
sys.stdout.flush()
|
||||
|
||||
# First registered, so it runs last: after the package's teardown and
|
||||
@@ -107,10 +64,6 @@ CHILD = textwrap.dedent("""
|
||||
fake_server_module(queue_task_done=True),
|
||||
)
|
||||
state["package"] = package
|
||||
if stuck:
|
||||
# The stop gives up after this long instead of the full 5 s.
|
||||
monitor_module = sys.modules[package.__name__ + ".utils.image_monitor"]
|
||||
monitor_module.OBSERVER_JOIN_SECONDS = 0.5
|
||||
monitor = package._global_image_monitor
|
||||
handler = monitor.handler if monitor is not None else None
|
||||
if handler is None:
|
||||
@@ -125,52 +78,47 @@ CHILD = textwrap.dedent("""
|
||||
""")
|
||||
|
||||
|
||||
def run_child(testcase, *args):
|
||||
"""Run CHILD with ``args``; returns (completed, report, detail)."""
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
os.makedirs(os.path.join(tmp, "output"))
|
||||
script = os.path.join(tmp, "child.py")
|
||||
with open(script, "w", encoding="utf-8") as handle:
|
||||
handle.write(CHILD)
|
||||
env = dict(os.environ)
|
||||
env.pop("PROMPT_MANAGER_CONFIG_PATH", None)
|
||||
# The child's library and log file live in its own folder, never
|
||||
# in the checkout's logs/ (which this runner may hold open) or a
|
||||
# data folder the caller set.
|
||||
env["PROMPT_MANAGER_DATA_DIR"] = tmp
|
||||
env["PYTHONIOENCODING"] = "utf-8"
|
||||
try:
|
||||
completed = subprocess.run(
|
||||
[sys.executable, script, ROOT, tmp, *args],
|
||||
cwd=ROOT,
|
||||
env=env,
|
||||
capture_output=True,
|
||||
timeout=CHILD_CEILING_SECONDS,
|
||||
encoding="utf-8",
|
||||
errors="replace",
|
||||
)
|
||||
except subprocess.TimeoutExpired as hung:
|
||||
testcase.fail(
|
||||
f"the child did not exit within {CHILD_CEILING_SECONDS} s\n"
|
||||
f"stdout:\n{hung.stdout}\nstderr:\n{hung.stderr}"
|
||||
)
|
||||
# Leaving the block removes the folder: on Windows that fails
|
||||
# if the child left a handle open, which it cannot once it exited,
|
||||
# so the check here is that the child exited at all.
|
||||
detail = f"stdout:\n{completed.stdout}\nstderr:\n{completed.stderr}"
|
||||
testcase.assertEqual(completed.returncode, 0, detail)
|
||||
reports = [
|
||||
line[len(REPORT_PREFIX) :]
|
||||
for line in completed.stdout.splitlines()
|
||||
if line.startswith(REPORT_PREFIX)
|
||||
]
|
||||
testcase.assertEqual(len(reports), 1, detail)
|
||||
return completed, json.loads(reports[0]), detail
|
||||
|
||||
|
||||
class TestPackageExit(unittest.TestCase):
|
||||
def test_teardown_runs_at_a_normal_exit_with_the_worker_alive(self):
|
||||
completed, report, detail = run_child(self)
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
os.makedirs(os.path.join(tmp, "output"))
|
||||
script = os.path.join(tmp, "child.py")
|
||||
with open(script, "w", encoding="utf-8") as handle:
|
||||
handle.write(CHILD)
|
||||
env = dict(os.environ)
|
||||
env.pop("PROMPT_MANAGER_CONFIG_PATH", None)
|
||||
# The child's library and log file live in its own folder, never
|
||||
# in the checkout's logs/ (which this runner may hold open) or a
|
||||
# data folder the caller set.
|
||||
env["PROMPT_MANAGER_DATA_DIR"] = tmp
|
||||
env["PYTHONIOENCODING"] = "utf-8"
|
||||
try:
|
||||
completed = subprocess.run(
|
||||
[sys.executable, script, ROOT, tmp],
|
||||
cwd=ROOT,
|
||||
env=env,
|
||||
capture_output=True,
|
||||
timeout=CHILD_CEILING_SECONDS,
|
||||
encoding="utf-8",
|
||||
errors="replace",
|
||||
)
|
||||
except subprocess.TimeoutExpired as hung:
|
||||
self.fail(
|
||||
f"the child did not exit within {CHILD_CEILING_SECONDS} s\n"
|
||||
f"stdout:\n{hung.stdout}\nstderr:\n{hung.stderr}"
|
||||
)
|
||||
# Leaving the block removes the folder: on Windows that fails
|
||||
# if the child left a handle open, which it cannot once it exited,
|
||||
# so the check here is that the child exited at all.
|
||||
detail = f"stdout:\n{completed.stdout}\nstderr:\n{completed.stderr}"
|
||||
self.assertEqual(completed.returncode, 0, detail)
|
||||
reports = [
|
||||
line[len(REPORT_PREFIX) :]
|
||||
for line in completed.stdout.splitlines()
|
||||
if line.startswith(REPORT_PREFIX)
|
||||
]
|
||||
self.assertEqual(len(reports), 1, detail)
|
||||
report = json.loads(reports[0])
|
||||
# "thumbnails" is True here because no preview worker started; its
|
||||
# close result is tested in test_thumbnail_auto and test_init_package.
|
||||
self.assertEqual(
|
||||
@@ -179,7 +127,6 @@ class TestPackageExit(unittest.TestCase):
|
||||
detail,
|
||||
)
|
||||
self.assertEqual(report["alive"], [], detail)
|
||||
self.assertEqual(report["holding"], [], detail)
|
||||
# The teardown's own warnings. Import-time warnings (the API's
|
||||
# output-folder search finds no ComfyUI here) are not about exit.
|
||||
for text in (
|
||||
@@ -190,16 +137,6 @@ class TestPackageExit(unittest.TestCase):
|
||||
):
|
||||
self.assertNotIn(text, completed.stderr, detail)
|
||||
|
||||
def test_an_observer_that_never_stops_does_not_hold_the_exit(self):
|
||||
# Before #561 the reader thread kept the interpreter waiting before
|
||||
# any atexit callback ran, so the teardown never started.
|
||||
completed, report, detail = run_child(self, "stuck-observer")
|
||||
self.assertEqual(
|
||||
report["result"],
|
||||
{"monitor": False, "thumbnails": True, "tracker": True},
|
||||
detail,
|
||||
)
|
||||
# The stuck reader is still alive, but as a daemon it holds nothing.
|
||||
self.assertEqual(report["holding"], [], detail)
|
||||
self.assertIn("watchdog observer still running", completed.stderr, detail)
|
||||
self.assertIn("Background work not stopped in time", completed.stderr, detail)
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
Vendored
+5
-18
@@ -331,7 +331,7 @@ const testJobs = () => jobs(readWorkflow('test.yml'));
|
||||
|
||||
test('the Python matrix is Linux 3.12 on a pull request and Linux 3.12-3.14 on every other event', () => {
|
||||
const body = testJobs().get('test');
|
||||
assert.deepEqual(matrixList(body, 'os'), ['ubuntu-latest'], 'Windows and macOS never gate a merge');
|
||||
assert.deepEqual(matrixList(body, 'os'), ['ubuntu-latest'], 'Windows never gates a merge');
|
||||
const match = /^ {8}python-version: \$\{\{ github\.event_name == 'pull_request' && fromJSON\('(\[[^\]]*\])'\) \|\| fromJSON\('(\[[^\]]*\])'\) \}\}$/m.exec(body);
|
||||
assert.ok(match, 'python-version is chosen by event');
|
||||
assert.deepEqual({ pr: JSON.parse(match[1]), other: JSON.parse(match[2]) }, { pr: ['3.12'], other: ['3.12', '3.13', '3.14'] });
|
||||
@@ -339,34 +339,21 @@ test('the Python matrix is Linux 3.12 on a pull request and Linux 3.12-3.14 on e
|
||||
assert.match(body, /^ {4}name: py\$\{\{ matrix\.python-version \}\} on \$\{\{ matrix\.os \}\}$/m);
|
||||
});
|
||||
|
||||
test('Windows and macOS run 3.12-3.14 on main, nightly and on demand, never on a PR or in the queue', () => {
|
||||
test('Windows runs 3.12-3.14 on main, nightly and on demand, never on a PR or in the queue', () => {
|
||||
const text = readWorkflow('portability.yml');
|
||||
const triggers = text.slice(text.indexOf('\non:'), text.indexOf('\nconcurrency:'));
|
||||
for (const event of ['push', 'schedule', 'workflow_dispatch']) assert.match(triggers, new RegExp(`^ {2}${event}:`, 'm'));
|
||||
for (const event of ['pull_request', 'merge_group']) assert.doesNotMatch(triggers, new RegExp(`^ {2}${event}`, 'm'));
|
||||
assert.match(triggers, /^ {4}branches: \[main\]$/m, 'push runs only for main');
|
||||
const body = jobs(text).get('test');
|
||||
assert.deepEqual(matrixList(body, 'os'), ['windows-latest', 'macos-latest']);
|
||||
assert.deepEqual(matrixList(body, 'os'), ['windows-latest'], 'macOS is not tested in CI (#571)');
|
||||
assert.doesNotMatch(text, /macos-/, 'no macOS runner is used');
|
||||
assert.deepEqual(matrixList(body, 'python-version'), ['"3.12"', '"3.13"', '"3.14"']);
|
||||
assert.match(body, /^ {6}fail-fast: false$/m);
|
||||
assert.match(body, /^ {8}run: python -m tests\.run_suite -v$/m);
|
||||
assert.doesNotMatch(text, /cancel-in-progress: true|cancel-in-progress: \$/, 'no portability run is cancelled');
|
||||
});
|
||||
|
||||
// #499: the native trash proof runs on both macOS CPUs, apart from the suite so a red suite never skips it.
|
||||
test('portability runs the native trash proof on arm64 and Intel macOS', () => {
|
||||
const body = jobs(readWorkflow('portability.yml')).get('native-trash');
|
||||
assert.ok(body, 'portability.yml has a native-trash job');
|
||||
assert.deepEqual(matrixList(body, 'os'), ['macos-latest', 'macos-15-intel']);
|
||||
assert.match(body, /^ {6}fail-fast: false$/m);
|
||||
assert.doesNotMatch(body, /^ {4}needs:/m, 'it does not wait for the suite job');
|
||||
assert.match(body, /^ {10}PM_NATIVE_TRASH: macos$/m);
|
||||
assert.match(body, /^ {8}run: python -m unittest tests\.test_trash_macos_native tests\.test_trash_macos -v$/m);
|
||||
const jobMinutes = Number(/^ {4}timeout-minutes: (\d+)$/m.exec(body)[1]);
|
||||
const stepMinutes = Number(/^ {8}timeout-minutes: (\d+)$/m.exec(body)?.[1]);
|
||||
assert.ok(stepMinutes > 0 && jobMinutes - stepMinutes >= 2, 'the proof step stops before the job limit');
|
||||
});
|
||||
|
||||
// #493: the portability limit follows the rule in its comment: ceil(1.5 x the slowest of three
|
||||
// green main runs) + 3 minutes, never under 5, with those runs and times written above jobs:.
|
||||
// The times are whole-job wall times. That the runs were green and on main is the author's word.
|
||||
@@ -427,5 +414,5 @@ test('every required check reports on a pull request and the gates carry the ful
|
||||
].sort());
|
||||
assert.equal(contexts.filter(context => /^py/.test(context)).length, 1,
|
||||
'a dropped matrix leg never reports on a pull request, so it cannot be required');
|
||||
assert.ok(!contexts.some(context => /windows|macos/.test(context)), 'Windows and macOS never gate a merge');
|
||||
assert.ok(!contexts.some(context => /windows|macos/.test(context)), 'Windows never gates a merge');
|
||||
});
|
||||
|
||||
+11
-87
@@ -96,65 +96,6 @@ WORKER_JOIN_SECONDS = 5.0
|
||||
OBSERVER_JOIN_SECONDS = 5.0
|
||||
|
||||
|
||||
def _start_daemon_observer(observer) -> None:
|
||||
"""``observer.start()`` on a short-lived daemon thread (#561).
|
||||
|
||||
A thread is a daemon when the thread that creates it is one. watchdog
|
||||
7 on Windows starts a plain ``threading.Thread`` per watch inside
|
||||
``start()``; created from ComfyUI's main thread it is not a daemon, and
|
||||
the interpreter waits for every such thread before it runs any
|
||||
``atexit`` callback, so the teardown that would stop it never runs.
|
||||
Started from here, every thread the observer creates is a daemon.
|
||||
Waits for the start without a limit (start-up, not exit); re-raises
|
||||
what ``start()`` raised.
|
||||
"""
|
||||
raised: List[BaseException] = []
|
||||
|
||||
def start() -> None:
|
||||
try:
|
||||
observer.start()
|
||||
except BaseException as e: # re-raised on the caller's thread
|
||||
raised.append(e)
|
||||
|
||||
starter = threading.Thread(
|
||||
target=start, name="PromptManagerObserverStart", daemon=True
|
||||
)
|
||||
starter.start()
|
||||
starter.join()
|
||||
if raised:
|
||||
raise raised[0]
|
||||
|
||||
|
||||
class _ObserverStop:
|
||||
"""One ``observer.stop()`` running on a daemon thread (#561).
|
||||
|
||||
watchdog's ``stop()`` joins its emitters (and, in 7.x on Windows, their
|
||||
reader threads) without a limit. Run here, a stop that never returns
|
||||
holds neither the caller past its deadline nor the interpreter's exit.
|
||||
"""
|
||||
|
||||
def __init__(self, observer) -> None:
|
||||
self.error: Optional[Exception] = None
|
||||
self._thread = threading.Thread(
|
||||
target=self._run,
|
||||
args=(observer,),
|
||||
name="PromptManagerObserverStop",
|
||||
daemon=True,
|
||||
)
|
||||
self._thread.start()
|
||||
|
||||
def _run(self, observer) -> None:
|
||||
try:
|
||||
observer.stop()
|
||||
except Exception as e: # reported by the stop that waits for it
|
||||
self.error = e
|
||||
|
||||
def wait(self, timeout: float) -> bool:
|
||||
"""True once ``stop()`` has returned or raised."""
|
||||
self._thread.join(timeout)
|
||||
return not self._thread.is_alive()
|
||||
|
||||
|
||||
class ImageGenerationHandler(FileSystemEventHandler):
|
||||
"""Filesystem event handler for detecting new image generation.
|
||||
|
||||
@@ -1229,9 +1170,6 @@ class ImageMonitor:
|
||||
# (observer, handler) pairs a stop left running: the next stop waits
|
||||
# for them again instead of reporting True.
|
||||
self._unfinished: List[Tuple[Any, Any]] = []
|
||||
# Observer stops still running on their thread, by observer: a later
|
||||
# stop waits for the same call instead of starting another (#561).
|
||||
self._observer_stops: Dict[Any, _ObserverStop] = {}
|
||||
self.logger = get_logger("prompt_manager.image_monitor")
|
||||
self.review_recorder: Optional[Callable[[str], str]] = None
|
||||
self.thumbnail_queue: Optional[Callable[[str], str]] = None
|
||||
@@ -1335,7 +1273,7 @@ class ImageMonitor:
|
||||
self.logger.warning("No directory could be watched")
|
||||
return
|
||||
try:
|
||||
_start_daemon_observer(observer)
|
||||
observer.start()
|
||||
except Exception as e:
|
||||
self.logger.error(f"Image monitoring could not start: {e}")
|
||||
return
|
||||
@@ -1365,12 +1303,12 @@ class ImageMonitor:
|
||||
|
||||
Closes the handler first (its worker is joined and its dispatch
|
||||
returns at once), then stops the watchdog observer and joins it. The
|
||||
worker and the observer share the one deadline: the observer's stop
|
||||
and join get what the worker left, possibly 0 s. watchdog's
|
||||
``observer.stop()`` joins its emitters without a limit, so it runs on
|
||||
a daemon thread and is waited for only until the deadline (#561). A
|
||||
part an earlier call left running is waited for again. Safe to call
|
||||
when monitoring never started and more than once.
|
||||
worker and the observer share the one deadline: the observer's join
|
||||
gets what the worker left, possibly 0 s.
|
||||
watchdog's ``observer.stop()`` joins its emitters without a limit;
|
||||
it counts against the deadline but is not bounded here. A part an
|
||||
earlier call left running is waited for again. Safe to call when
|
||||
monitoring never started and more than once.
|
||||
|
||||
Returns:
|
||||
True when every worker and observer it stopped has ended.
|
||||
@@ -1411,27 +1349,13 @@ class ImageMonitor:
|
||||
return False
|
||||
|
||||
def _stop_observer(self, observer, timeout: float) -> bool:
|
||||
"""``observer.stop()`` on its own daemon thread, then a join, both
|
||||
within ``timeout`` seconds; a stop or join still running then is
|
||||
logged with the time it got. A stop still running from an earlier
|
||||
call is waited for, never started again."""
|
||||
"""``observer.stop()`` (safe to repeat), then a join of ``timeout``
|
||||
seconds; a join that times out is logged with the time it got."""
|
||||
if observer is None:
|
||||
return True
|
||||
deadline = time.monotonic() + timeout
|
||||
stop = self._observer_stops.get(observer)
|
||||
if stop is None:
|
||||
stop = self._observer_stops[observer] = _ObserverStop(observer)
|
||||
if not stop.wait(timeout):
|
||||
self.logger.warning(
|
||||
f"watchdog observer still running after {timeout:.1f} s"
|
||||
)
|
||||
return False
|
||||
del self._observer_stops[observer]
|
||||
if stop.error is not None:
|
||||
self.logger.warning(f"Error while stopping image monitoring: {stop.error}")
|
||||
return False
|
||||
try:
|
||||
observer.join(max(0.0, deadline - time.monotonic()))
|
||||
observer.stop()
|
||||
observer.join(timeout)
|
||||
except Exception as e:
|
||||
self.logger.warning(f"Error while stopping image monitoring: {e}")
|
||||
return False
|
||||
|
||||
Reference in New Issue
Block a user