* perf(images): share and warm one listing per output root (#348) The Images page, its sidebar counts, the subfolder list and the legacy gallery each rebuilt the output listing when the 30 s cache expired, all at once, and the subfolder list computed relative paths on the event loop. Thumbnail writes also threw the listing away. Keep one RootListing per output root (py/api/gallery_listing.py): - one walk per root at a time; concurrent readers wait for it (shielded) - a listing older than 30 s is served at once and refreshed in the background - the walk, the comparison with the previous listing and both folder sets run in the executor; an unchanged walk keeps the same list object - os.scandir probe: a missing root fails the walk and keeps the previous listing (one warning per failure streak) - Rescan waits for a walk started after the click and is a 500 when it fails, even with a previous listing - deletes keep the blocking invalidation; a generation counter that survives drops discards walks that raced it - thumbnail jobs, conversion and the legacy generate/clear routes no longer drop the listing - warm-up on the host's on_startup, cancelled on on_cleanup Adds tools/vendor/perf/images_load_probe.mjs (maintainer tool, not CI) and the TESTING.md section "Images listing cache (#348)". * fix(images): drop listings on the loop when invalidated from a worker thread (#348) A settings save (gallery roots change) and the browser-test seeds call invalidate_gallery_cache() from the executor. Listing records are only touched on the event loop, so an off-loop call now hands the drop to the loop and waits for it, keeping the call synchronous for its callers. The shared-Rescan test waits for the later Rescans to take their walk number instead of sleeping. * test(images): bound the shared-listing test waits (#348) A macOS py3.12 run stopped inside the shared-listing tests and sat until the 6 h job limit with no traceback. It does not reproduce locally (about 300 runs on 3.12 and 3.14, pinned to one core). Requests and background settles in TestSharedListing now give up after 30 s, so a recurrence fails with the stuck await instead of hanging the run. * test(images): let the settle loops yield so a finished walk leaves the set (#348) On CI (ubuntu and macOS py3.12) the runner's watchdog dumped test_a_root_that_recovers_warns_again_on_its_next_failure spinning in _settle: asyncio.gather over tasks that are already done returns without yielding, so the set's discard callback, queued when the walk finished, never ran and the loop never ended (wait_for could not fire either). Delaying discard by one loop turn reproduces it locally; both settle loops now yield one turn after each gather. * test(browser): poll the import review's rows instead of reading them once gridOf reads the dialog's rows once; right after the click the review may not have rendered them yet (firefox CI: received undefined). * perf(images): walk the output with scandir entries and build folders from strings (#348) The live test showed a read during a background refresh taking 2.2 s instead of 0.07 s: the walk held the GIL for about 2 s in the server. py-spy split that walk into two halves: Path.relative_to per file in folder_sets, and Path.stat + Path.is_symlink + Path joins per file in the scan. - _walk_media visits folders in os.walk's order with os.scandir entries: the file type comes from the listing (no lstat per file) and the mtime from entry.stat (one call, none on Windows). _iter_media_files is now a thin wrapper, so its other caller is unchanged. - folder_sets slices the path strings (in the path flavour's separator) instead of calling relative_to, falling back to it for a path that does not start with the root string. On the maintainer's 19,935-file output folder: scan 0.386 s -> 0.112 s, folder sets 0.452 s -> 0.007 s, both results identical to the previous code. A reference copy of the os.walk version checks order, filters, the depth cap, symlinks and the file cap. * perf(images): build the workspace caches behind the warm-up and changed walks (#348) The live test after the faster walk still took 2.9-5.2 s for the first Images open after a restart. py-spy showed the walk was now small; the time went to the workspace's own caches, cold after a restart and built by three parallel first requests at once: the favorites listing (Path.relative_to per file), the per-file values and newest order, and the prompt-link read. - Once the host's startup warm-up has run, a stored walk that changed a listing schedules one executor job that builds what a first page reads, through the request path itself (_select_workspace_page with the default filters, then _workspace_favorite_paths). A walk stored during a build asks for one more build; an unchanged walk (same list kept) builds nothing; a failed build only warns; on_cleanup stops it. Servers that never ran the warm-up (the test servers) start no build. - _favorite_listing slices the path strings as _listed does, falling back to relative_to for a path outside the root string. * perf(images): start a stale listing's refresh walk a second after the read (#348) After the faster walk, a read that found the listing stale still took 0.5-1.0 s against 0.07 s: it was served from the old listing but ran beside the refresh walk it had just started, sharing the GIL. The refresh now starts REFRESH_DELAY (1 s) after that read through a per-root rewalk_handle, so the page's own requests finish first. One pending refresh per root and none while a walk runs; any walk start (a cold read, Rescan, the warm-up) cancels a pending one; on_cleanup cancels them before its first await. * test(browser): accept KB or MB for the picked safety backup's size The restore preview showed '2.5 MB' for the test server's database copy in a CI coverage shard: its size depends on what earlier tests on that server wrote, and the formatter switches to MB at 1 MiB. * fix(images): keep the 3.10-3.13 suffix rule in the scandir walk and harden the warm-up (#348) Review of the walk speedup: - os.path.splitext drops names like ..png that Path.suffix listed on Python 3.10-3.13 (3.14 changed Path.suffix, which hid it from the parity test on a 3.14 venv). _name_suffix spells out the 3.10-3.13 rule; the test now asserts ..png and ...webp are listed and g. is not. - _warm_workspace reads the roots inside its try, so a failing _get_all_output_dirs warns instead of leaving an unretrieved task error. - The folder_sets fallback test now takes the fallback (a Windows case mismatch); the coalescing test pins the walk count.
929 lines
36 KiB
Python
929 lines
36 KiB
Python
"""Tests for the core API module (py/api/__init__.py): helpers and base routes."""
|
|
|
|
import json
|
|
import os
|
|
import shutil
|
|
import sys
|
|
import tempfile
|
|
import types
|
|
import unittest
|
|
from pathlib import Path
|
|
from unittest.mock import MagicMock
|
|
|
|
_mock_server = MagicMock()
|
|
_mock_server.PromptServer.instance.routes = MagicMock()
|
|
sys.modules.setdefault("server", _mock_server)
|
|
|
|
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
|
|
|
try:
|
|
from tests.open_handles import assert_closed # noqa: E402
|
|
except ImportError: # discovered with tests/ as the top-level directory
|
|
from open_handles import assert_closed # noqa: E402
|
|
from database.models import PromptModel # noqa: E402
|
|
|
|
from aiohttp import web # noqa: E402
|
|
from aiohttp.test_utils import AioHTTPTestCase # noqa: E402
|
|
|
|
import py.api as api_module # noqa: E402
|
|
from database.operations import PromptDatabase # noqa: E402
|
|
from py.api import PromptManagerAPI, _public_error, _public_path # noqa: E402
|
|
from py.config import GalleryConfig # noqa: E402
|
|
|
|
EXTRA_ROOTS_ENV = "PROMPT_MANAGER_EXTRA_GALLERY_ROOTS"
|
|
|
|
|
|
class FolderPathsFixture(unittest.TestCase):
|
|
"""Temp ComfyUI tree exposed through a stub folder_paths module."""
|
|
|
|
def setUp(self):
|
|
self.tmpdir = tempfile.mkdtemp()
|
|
self.addCleanup(shutil.rmtree, self.tmpdir, True)
|
|
self.addCleanup(assert_closed, self, self.tmpdir) # just before the rmtree
|
|
self.addCleanup(PromptModel.close_all_instances)
|
|
self.comfy_dir = Path(self.tmpdir) / "ComfyUI"
|
|
self.output_dir = self.comfy_dir / "output"
|
|
self.output_dir.mkdir(parents=True)
|
|
|
|
self._orig_folder_paths = sys.modules.get("folder_paths")
|
|
self.addCleanup(self._restore_folder_paths)
|
|
self._orig_extra = os.environ.pop(EXTRA_ROOTS_ENV, None)
|
|
self.addCleanup(self._restore_extra_env)
|
|
|
|
self.install_folder_paths(base_path=str(self.comfy_dir))
|
|
|
|
def install_folder_paths(self, **attrs):
|
|
attrs.setdefault("get_output_directory", lambda: str(self.output_dir))
|
|
sys.modules["folder_paths"] = types.SimpleNamespace(**attrs)
|
|
|
|
def _restore_folder_paths(self):
|
|
if self._orig_folder_paths is None:
|
|
sys.modules.pop("folder_paths", None)
|
|
else:
|
|
sys.modules["folder_paths"] = self._orig_folder_paths
|
|
|
|
def _restore_extra_env(self):
|
|
if self._orig_extra is None:
|
|
os.environ.pop(EXTRA_ROOTS_ENV, None)
|
|
else:
|
|
os.environ[EXTRA_ROOTS_ENV] = self._orig_extra
|
|
|
|
|
|
class TestPublicPath(FolderPathsFixture):
|
|
|
|
def test_path_under_base_is_relative_posix(self):
|
|
target = self.output_dir / "renders" / "a.png"
|
|
self.assertEqual(_public_path(str(target)), "output/renders/a.png")
|
|
|
|
def test_base_itself_is_dot(self):
|
|
self.assertEqual(_public_path(str(self.comfy_dir)), ".")
|
|
|
|
def test_path_outside_base_is_basename_only(self):
|
|
secret = Path(self.tmpdir) / "secret" / "passwords.txt"
|
|
secret.parent.mkdir()
|
|
secret.write_text("x")
|
|
self.assertEqual(_public_path(str(secret)), "passwords.txt")
|
|
self.assertNotIn(self.tmpdir, _public_path(str(secret)))
|
|
|
|
def test_path_object_accepted(self):
|
|
self.assertEqual(_public_path(self.output_dir), "output")
|
|
|
|
def test_falls_back_to_output_parent_without_base_path(self):
|
|
self.install_folder_paths() # no base_path attribute
|
|
self.assertEqual(_public_path(str(self.output_dir / "x.png")), "output/x.png")
|
|
|
|
def test_env_extra_root_keeps_its_own_name(self):
|
|
extra = Path(self.tmpdir) / "nas" / "gallery"
|
|
(extra / "sub").mkdir(parents=True)
|
|
os.environ[EXTRA_ROOTS_ENV] = str(extra)
|
|
self.assertEqual(_public_path(str(extra / "sub")), "gallery/sub")
|
|
|
|
def test_without_folder_paths_returns_basename(self):
|
|
sys.modules.pop("folder_paths", None)
|
|
self.assertEqual(_public_path(str(self.output_dir / "img.png")), "img.png")
|
|
|
|
def test_empty_or_none_gives_empty_string(self):
|
|
self.assertEqual(_public_path(""), "")
|
|
self.assertEqual(_public_path(None), "")
|
|
|
|
|
|
class TestPublicError(FolderPathsFixture):
|
|
|
|
def test_oserror_uses_strerror_and_basename(self):
|
|
err = FileNotFoundError(
|
|
2, "No such file or directory", str(self.tmpdir) + "/a/b.png"
|
|
)
|
|
text = _public_error(err)
|
|
self.assertIn("No such file or directory", text)
|
|
self.assertIn("b.png", text)
|
|
self.assertNotIn(self.tmpdir, text)
|
|
|
|
def test_oserror_without_filename(self):
|
|
err = PermissionError(13, "Permission denied")
|
|
self.assertEqual(_public_error(err), "Permission denied")
|
|
|
|
def test_oserror_without_strerror_uses_type_name(self):
|
|
err = OSError()
|
|
self.assertEqual(_public_error(err), "OSError")
|
|
|
|
def test_non_oserror_is_str(self):
|
|
self.assertEqual(_public_error(ValueError("bad value")), "bad value")
|
|
|
|
def test_non_oserror_with_empty_message_uses_type_name(self):
|
|
self.assertEqual(_public_error(RuntimeError()), "RuntimeError")
|
|
|
|
|
|
# ── gzip middleware ───────────────────────────────────────────────────
|
|
|
|
|
|
class TestGzipMiddleware(AioHTTPTestCase):
|
|
|
|
async def get_application(self):
|
|
big = json.dumps({"data": "x" * 4000})
|
|
|
|
async def big_json(request):
|
|
return web.json_response(text=big)
|
|
|
|
async def small_json(request):
|
|
return web.json_response({"ok": True})
|
|
|
|
async def already_encoded(request):
|
|
return web.Response(
|
|
text=big, content_type="text/plain", headers={"Content-Encoding": "br"}
|
|
)
|
|
|
|
async def binary(request):
|
|
return web.Response(body=b"\x89PNG" * 1000, content_type="image/png")
|
|
|
|
async def incompressible(request):
|
|
return web.Response(body=os.urandom(4096), content_type="text/plain")
|
|
|
|
async def streamed(request):
|
|
resp = web.StreamResponse()
|
|
await resp.prepare(request)
|
|
await resp.write(b"x" * 4000)
|
|
await resp.write_eof()
|
|
return resp
|
|
|
|
app = web.Application(middlewares=[api_module._gzip_middleware])
|
|
app.router.add_get("/prompt_manager/big", big_json)
|
|
app.router.add_get("/prompt_manager/small", small_json)
|
|
app.router.add_get("/prompt_manager/encoded", already_encoded)
|
|
app.router.add_get("/prompt_manager/binary", binary)
|
|
app.router.add_get("/prompt_manager/random", incompressible)
|
|
app.router.add_get("/prompt_manager/stream", streamed)
|
|
app.router.add_get("/other/big", big_json)
|
|
return app
|
|
|
|
async def _get(self, path, accept="gzip"):
|
|
headers = {"Accept-Encoding": accept} if accept else {}
|
|
return await self.client.request(
|
|
"GET",
|
|
path,
|
|
headers=headers,
|
|
auto_decompress=False,
|
|
skip_auto_headers=["Accept-Encoding"],
|
|
)
|
|
|
|
async def test_large_json_is_gzipped(self):
|
|
resp = await self._get("/prompt_manager/big")
|
|
self.assertEqual(resp.headers.get("Content-Encoding"), "gzip")
|
|
self.assertEqual(resp.headers.get("Vary"), "Accept-Encoding")
|
|
import gzip
|
|
|
|
self.assertIn(b'"data"', gzip.decompress(await resp.read()))
|
|
|
|
async def test_small_body_not_compressed(self):
|
|
resp = await self._get("/prompt_manager/small")
|
|
self.assertIsNone(resp.headers.get("Content-Encoding"))
|
|
|
|
async def test_other_paths_untouched(self):
|
|
resp = await self._get("/other/big")
|
|
self.assertIsNone(resp.headers.get("Content-Encoding"))
|
|
|
|
async def test_without_accept_encoding(self):
|
|
resp = await self._get("/prompt_manager/big", accept=None)
|
|
self.assertIsNone(resp.headers.get("Content-Encoding"))
|
|
|
|
async def test_already_encoded_untouched(self):
|
|
resp = await self._get("/prompt_manager/encoded")
|
|
self.assertEqual(resp.headers.get("Content-Encoding"), "br")
|
|
|
|
async def test_binary_type_untouched(self):
|
|
resp = await self._get("/prompt_manager/binary")
|
|
self.assertIsNone(resp.headers.get("Content-Encoding"))
|
|
|
|
async def test_incompressible_body_untouched(self):
|
|
resp = await self._get("/prompt_manager/random")
|
|
self.assertIsNone(resp.headers.get("Content-Encoding"))
|
|
|
|
async def test_stream_response_untouched(self):
|
|
resp = await self._get("/prompt_manager/stream")
|
|
self.assertIsNone(resp.headers.get("Content-Encoding"))
|
|
self.assertEqual(len(await resp.read()), 4000)
|
|
|
|
|
|
# ── JSON body reader ──────────────────────────────────────────────────
|
|
|
|
|
|
class TestReadJsonBody(AioHTTPTestCase):
|
|
"""_read_json_body: chunked read with a byte cap, object-only payloads."""
|
|
|
|
async def get_application(self):
|
|
async def echo(request):
|
|
data, error = await api_module._read_json_body(request)
|
|
if error is not None:
|
|
return error
|
|
return web.json_response({"success": True, "keys": sorted(data)})
|
|
|
|
async def small(request):
|
|
data, error = await api_module._read_json_body(request, max_bytes=64)
|
|
if error is not None:
|
|
return error
|
|
return web.json_response({"success": True, "data": data})
|
|
|
|
app = web.Application(client_max_size=4 * 1024 * 1024)
|
|
app.router.add_post("/echo", echo)
|
|
app.router.add_post("/small", small)
|
|
return app
|
|
|
|
async def test_valid_object_is_parsed(self):
|
|
resp = await self.client.request("POST", "/echo", json={"b": 1, "a": 2})
|
|
self.assertEqual(resp.status, 200)
|
|
self.assertEqual((await resp.json())["keys"], ["a", "b"])
|
|
|
|
async def test_default_cap_is_one_megabyte(self):
|
|
self.assertEqual(api_module.JSON_BODY_MAX_BYTES, 1_000_000)
|
|
body = json.dumps({"pad": "x" * 1_000_100}).encode()
|
|
resp = await self.client.request(
|
|
"POST", "/echo", data=body, headers={"Content-Type": "application/json"}
|
|
)
|
|
self.assertEqual(resp.status, 413)
|
|
data = await resp.json()
|
|
self.assertFalse(data["success"])
|
|
self.assertIn("too large", data["error"].lower())
|
|
|
|
async def test_body_just_under_cap_is_accepted(self):
|
|
payload = {"pad": "x" * (1_000_000 - 20)}
|
|
body = json.dumps(payload).encode()
|
|
self.assertLessEqual(len(body), 1_000_000)
|
|
resp = await self.client.request(
|
|
"POST", "/echo", data=body, headers={"Content-Type": "application/json"}
|
|
)
|
|
self.assertEqual(resp.status, 200)
|
|
|
|
async def test_chunked_upload_without_content_length_is_capped(self):
|
|
async def gen():
|
|
for _ in range(10):
|
|
yield b'{"pad": "' + b"x" * 20 + b'"}'
|
|
|
|
resp = await self.client.request(
|
|
"POST",
|
|
"/small",
|
|
data=gen(),
|
|
headers={"Content-Type": "application/json"},
|
|
)
|
|
self.assertEqual(resp.status, 413)
|
|
|
|
async def test_declared_oversize_content_length_is_413(self):
|
|
body = b'{"pad": "' + b"x" * 200 + b'"}'
|
|
resp = await self.client.request(
|
|
"POST", "/small", data=body, headers={"Content-Type": "application/json"}
|
|
)
|
|
self.assertEqual(resp.status, 413)
|
|
|
|
async def test_invalid_json_is_400(self):
|
|
for body in (b"{oops", b"\xff\xfe", b""):
|
|
resp = await self.client.request(
|
|
"POST",
|
|
"/echo",
|
|
data=body,
|
|
headers={"Content-Type": "application/json"},
|
|
)
|
|
self.assertEqual(resp.status, 400, body)
|
|
self.assertFalse((await resp.json())["success"])
|
|
|
|
async def test_non_object_json_is_400(self):
|
|
for body in (b"[1, 2]", b'"text"', b"3", b"null", b"true"):
|
|
resp = await self.client.request(
|
|
"POST",
|
|
"/echo",
|
|
data=body,
|
|
headers={"Content-Type": "application/json"},
|
|
)
|
|
self.assertEqual(resp.status, 400, body)
|
|
self.assertIn("object", (await resp.json())["error"])
|
|
|
|
def test_helper_is_exposed_on_the_api_object(self):
|
|
self.assertIs(PromptManagerAPI._read_json_body, api_module._read_json_body)
|
|
|
|
|
|
class TestLongJobLock(unittest.TestCase):
|
|
"""_acquire_long_job: one job per family at a time, released on exit."""
|
|
|
|
def setUp(self):
|
|
database = MagicMock()
|
|
database.cleanup_duplicates.return_value = 0
|
|
self.api = PromptManagerAPI(db=database)
|
|
|
|
def _run(self, coro):
|
|
import asyncio
|
|
|
|
return asyncio.run(coro)
|
|
|
|
def test_second_acquire_is_refused_until_first_releases(self):
|
|
async def scenario():
|
|
first = self.api._acquire_long_job("duplicates")
|
|
self.assertIsNotNone(first)
|
|
async with first:
|
|
self.assertIsNone(self.api._acquire_long_job("duplicates"))
|
|
# Other families are independent
|
|
other = self.api._acquire_long_job("thumbnails")
|
|
self.assertIsNotNone(other)
|
|
async with other:
|
|
pass
|
|
again = self.api._acquire_long_job("duplicates")
|
|
self.assertIsNotNone(again)
|
|
async with again:
|
|
pass
|
|
|
|
self._run(scenario())
|
|
|
|
def test_lock_is_released_when_the_job_raises(self):
|
|
async def scenario():
|
|
with self.assertRaises(RuntimeError):
|
|
async with self.api._acquire_long_job("duplicates"):
|
|
raise RuntimeError("boom")
|
|
self.assertIsNotNone(self.api._acquire_long_job("duplicates"))
|
|
|
|
self._run(scenario())
|
|
|
|
def test_busy_response_is_409_with_envelope(self):
|
|
resp = self.api._long_job_busy_response("Duplicate scan")
|
|
self.assertEqual(resp.status, 409)
|
|
data = json.loads(resp.body)
|
|
self.assertFalse(data["success"])
|
|
self.assertEqual(data["error"], "Duplicate scan already running")
|
|
|
|
|
|
# ── UI and static routes ──────────────────────────────────────────────
|
|
|
|
|
|
class CoreRoutesTestCase(AioHTTPTestCase):
|
|
|
|
async def get_application(self):
|
|
self.tmpdir = tempfile.mkdtemp()
|
|
app = web.Application()
|
|
routes = web.RouteTableDef()
|
|
self.api = PromptManagerAPI(
|
|
db=PromptDatabase(os.path.join(self.tmpdir, "prompts.db"))
|
|
)
|
|
self.api.add_routes(routes)
|
|
app.router.add_routes(routes)
|
|
return app
|
|
|
|
async def tearDownAsync(self):
|
|
await super().tearDownAsync() # closes the aiohttp test client
|
|
# The automatic preview worker writes into the temp tree: stop it first.
|
|
self.api.close_auto_thumbnails()
|
|
self.api.db.close_all()
|
|
assert_closed(self, self.tmpdir)
|
|
shutil.rmtree(self.tmpdir, ignore_errors=True)
|
|
|
|
def _fake_root(self):
|
|
"""Point _get_project_root at the temp dir and return its web/ dir."""
|
|
web_dir = Path(self.tmpdir) / "web"
|
|
(web_dir / "js").mkdir(parents=True)
|
|
(web_dir / "lib").mkdir()
|
|
orig_root = api_module._get_project_root
|
|
api_module._get_project_root = lambda: self.tmpdir
|
|
self.addCleanup(setattr, api_module, "_get_project_root", orig_root)
|
|
return web_dir
|
|
|
|
|
|
UI_PATHS = (
|
|
"/prompt_manager/workspace",
|
|
"/prompt_manager/web",
|
|
"/prompt_manager/gallery.html",
|
|
"/prompt_manager/admin",
|
|
"/prompt_manager/gallery",
|
|
)
|
|
|
|
|
|
class TestUiRoutes(CoreRoutesTestCase):
|
|
|
|
async def test_pages_are_served(self):
|
|
for path in UI_PATHS:
|
|
resp = await self.client.request("GET", path)
|
|
self.assertEqual(resp.status, 200, path)
|
|
self.assertIn("text/html", resp.headers["Content-Type"])
|
|
self.assertIn("<", await resp.text())
|
|
|
|
async def test_pages_are_cached_after_first_read(self):
|
|
await self.client.request("GET", "/prompt_manager/admin")
|
|
self.assertTrue(any(p.endswith("admin.html") for p in self.api._html_cache))
|
|
resp = await self.client.request("GET", "/prompt_manager/admin")
|
|
self.assertEqual(resp.status, 200)
|
|
|
|
async def test_missing_pages_are_404(self):
|
|
self._fake_root()
|
|
for path in UI_PATHS:
|
|
resp = await self.client.request("GET", path)
|
|
self.assertEqual(resp.status, 404, path)
|
|
|
|
async def test_unreadable_pages_are_500(self):
|
|
web_dir = self._fake_root()
|
|
for name in (
|
|
"workspace.html",
|
|
"index.html",
|
|
"metadata.html",
|
|
"admin.html",
|
|
"gallery.html",
|
|
):
|
|
(web_dir / name).mkdir() # a directory cannot be read as a file
|
|
for path in UI_PATHS:
|
|
resp = await self.client.request("GET", path)
|
|
self.assertEqual(resp.status, 500, path)
|
|
self.assertNotIn(self.tmpdir, await resp.text())
|
|
|
|
async def test_test_route(self):
|
|
resp = await self.client.request("GET", "/prompt_manager/test")
|
|
self.assertTrue((await resp.json())["success"])
|
|
|
|
async def test_image_annotation_routes_are_registered(self):
|
|
table = {
|
|
(route.method, route.resource.canonical)
|
|
for route in self.app.router.routes()
|
|
if route.resource is not None
|
|
}
|
|
for entry in (
|
|
("POST", "/prompt_manager/image-annotations/observe"),
|
|
("GET", "/prompt_manager/image-annotations/capture/{token}"),
|
|
("DELETE", "/prompt_manager/image-annotations/capture/{token}"),
|
|
("PUT", "/prompt_manager/image-annotations/tags"),
|
|
):
|
|
self.assertIn(entry, table)
|
|
|
|
async def test_image_tag_job_route_is_registered_beside_the_scan_job(self):
|
|
table = {
|
|
(route.method, route.resource.canonical)
|
|
for route in self.app.router.routes()
|
|
if route.resource is not None
|
|
}
|
|
self.assertIn(("POST", "/prompt_manager/jobs/scan"), table)
|
|
self.assertIn(("POST", "/prompt_manager/jobs/image-tags"), table)
|
|
|
|
async def test_review_routes_are_registered(self):
|
|
table = {
|
|
(route.method, route.resource.canonical)
|
|
for route in self.app.router.routes()
|
|
if route.resource is not None
|
|
}
|
|
for entry in (
|
|
("GET", "/prompt_manager/review/items"),
|
|
("GET", "/prompt_manager/review/items/{item_id}"),
|
|
("POST", "/prompt_manager/review/items/{item_id}/decision"),
|
|
):
|
|
self.assertIn(entry, table)
|
|
|
|
|
|
class TestStaticRoutes(CoreRoutesTestCase):
|
|
|
|
async def test_js_served_with_mime(self):
|
|
resp = await self.client.request("GET", "/prompt_manager/js/admin.js")
|
|
self.assertEqual(resp.status, 200)
|
|
self.assertIn("application/javascript", resp.headers["Content-Type"])
|
|
|
|
async def test_workspace_module_served_with_javascript_mime(self):
|
|
resp = await self.client.request(
|
|
"GET", "/prompt_manager/js/workspace/index.mjs"
|
|
)
|
|
self.assertEqual(resp.status, 200)
|
|
self.assertIn("application/javascript", resp.headers["Content-Type"])
|
|
|
|
async def test_inspection_semantics_table_is_the_one_json_served(self):
|
|
"""Local Inspect reads the node table the server resolves with (#249 PR L3).
|
|
|
|
Only that exact file is served; every other JSON stays forbidden.
|
|
"""
|
|
resp = await self.client.request(
|
|
"GET", "/prompt_manager/js/inspection-semantics.json"
|
|
)
|
|
self.assertEqual(resp.status, 200)
|
|
self.assertIn("application/json", resp.headers["Content-Type"])
|
|
table = Path(__file__).resolve().parents[1] / "web" / "js"
|
|
self.assertEqual(
|
|
await resp.read(), (table / "inspection-semantics.json").read_bytes()
|
|
)
|
|
for path in (
|
|
"js/shared/toolkit/provenance.json",
|
|
"lib/viewerjs/provenance.json",
|
|
"lib/inspection-semantics.json",
|
|
"js/workspace/..%2Finspection-semantics.json",
|
|
"js/INSPECTION-SEMANTICS.json",
|
|
):
|
|
resp = await self.client.request("GET", f"/prompt_manager/{path}")
|
|
self.assertEqual(resp.status, 403, path)
|
|
|
|
async def test_lib_served(self):
|
|
web_dir = self._fake_root()
|
|
(web_dir / "lib" / "x.css").write_text("body{}")
|
|
resp = await self.client.request("GET", "/prompt_manager/lib/x.css")
|
|
self.assertEqual(resp.status, 200)
|
|
self.assertIn("text/css", resp.headers["Content-Type"])
|
|
self.assertEqual(await resp.text(), "body{}")
|
|
|
|
async def test_real_lib_file_served_with_content_type(self):
|
|
resp = await self.client.request(
|
|
"GET", "/prompt_manager/lib/viewerjs/viewer.min.css"
|
|
)
|
|
self.assertEqual(resp.status, 200)
|
|
self.assertIn("text/css", resp.headers["Content-Type"])
|
|
|
|
async def test_allowed_asset_types_get_their_mime(self):
|
|
web_dir = self._fake_root()
|
|
expected = {
|
|
"a.map": "application/json",
|
|
"f.woff": "font/woff",
|
|
"f.woff2": "font/woff2",
|
|
"f.ttf": "font/ttf",
|
|
"i.png": "image/png",
|
|
"i.svg": "image/svg+xml",
|
|
"i.ico": "image/x-icon",
|
|
"p.html": "text/html",
|
|
}
|
|
for name in expected:
|
|
(web_dir / "lib" / name).write_bytes(b"\x00\x01")
|
|
for name, mime in expected.items():
|
|
resp = await self.client.request("GET", f"/prompt_manager/lib/{name}")
|
|
self.assertEqual(resp.status, 200, name)
|
|
self.assertIn(mime, resp.headers["Content-Type"], name)
|
|
|
|
async def test_disallowed_extensions_are_403_even_when_present(self):
|
|
web_dir = self._fake_root()
|
|
for name in ("x.bin", "x.py", "x.json", "noext", "x.JS.bak"):
|
|
(web_dir / "lib" / name).write_bytes(b"\x00")
|
|
(web_dir / "js" / name).write_bytes(b"\x00")
|
|
for prefix in ("lib", "js"):
|
|
for name in ("x.bin", "x.py", "x.json", "noext", "x.JS.bak"):
|
|
resp = await self.client.request(
|
|
"GET", f"/prompt_manager/{prefix}/{name}"
|
|
)
|
|
self.assertEqual(resp.status, 403, f"{prefix}/{name}")
|
|
|
|
async def test_traversal_is_403(self):
|
|
for prefix in ("lib", "js"):
|
|
for encoded in (
|
|
"..%2F..%2Fpyproject.toml",
|
|
"..%5C..%5Cpyproject.toml",
|
|
"sub%2F..%2F..%2F..%2Fpyproject.toml",
|
|
"%2E%2E%2Fpyproject.toml",
|
|
):
|
|
resp = await self.client.request(
|
|
"GET", f"/prompt_manager/{prefix}/{encoded}"
|
|
)
|
|
self.assertEqual(resp.status, 403, f"{prefix}/{encoded}")
|
|
|
|
async def test_absolute_drive_and_unc_paths_are_rejected_on_every_os(self):
|
|
# Built literally so the Windows-only escapes are exercised everywhere:
|
|
# os.path.join(root, "C:/x") drops root on Windows, "\\x" and UNC
|
|
# shares re-anchor the path as well.
|
|
candidates = (
|
|
"C:/Windows/win.ini",
|
|
"C:%5CWindows%5Cwin.ini",
|
|
"c:win.ini",
|
|
"%5C%5Cserver%5Cshare%5Cx.js",
|
|
"%2F%2Fserver%2Fshare%2Fx.js",
|
|
"%5CWindows%5Cwin.ini",
|
|
"%2Fetc%2Fpasswd",
|
|
)
|
|
for prefix in ("lib", "js"):
|
|
for encoded in candidates:
|
|
resp = await self.client.request(
|
|
"GET", f"/prompt_manager/{prefix}/{encoded}"
|
|
)
|
|
self.assertIn(resp.status, (403, 404), f"{prefix}/{encoded}")
|
|
self.assertNotIn("[extensions]", await resp.text())
|
|
|
|
async def test_symlink_escaping_web_dir_is_403(self):
|
|
web_dir = self._fake_root()
|
|
secret = Path(self.tmpdir) / "secret.js"
|
|
secret.write_text("secret")
|
|
link = web_dir / "lib" / "link.js"
|
|
try:
|
|
os.symlink(secret, link)
|
|
except (OSError, NotImplementedError):
|
|
self.skipTest("symlinks not available")
|
|
resp = await self.client.request("GET", "/prompt_manager/lib/link.js")
|
|
self.assertEqual(resp.status, 403)
|
|
|
|
async def test_inspection_semantics_symlink_escaping_web_dir_is_403(self):
|
|
"""The exact-path JSON gets the same realpath containment as any file."""
|
|
web_dir = self._fake_root()
|
|
secret = Path(self.tmpdir) / "secret.json"
|
|
secret.write_text('{"secret": true}')
|
|
try:
|
|
os.symlink(secret, web_dir / "js" / "inspection-semantics.json")
|
|
except (OSError, NotImplementedError):
|
|
self.skipTest("symlinks not available")
|
|
resp = await self.client.request(
|
|
"GET", "/prompt_manager/js/inspection-semantics.json"
|
|
)
|
|
self.assertEqual(resp.status, 403)
|
|
self.assertNotIn("secret", await resp.text())
|
|
|
|
async def test_inspection_semantics_missing_is_404(self):
|
|
self._fake_root()
|
|
resp = await self.client.request(
|
|
"GET", "/prompt_manager/js/inspection-semantics.json"
|
|
)
|
|
self.assertEqual(resp.status, 404)
|
|
|
|
async def test_empty_path_is_not_served(self):
|
|
for prefix in ("lib", "js"):
|
|
resp = await self.client.request("GET", f"/prompt_manager/{prefix}/")
|
|
self.assertIn(resp.status, (403, 404), prefix)
|
|
|
|
async def test_missing_is_404(self):
|
|
for prefix in ("lib", "js"):
|
|
resp = await self.client.request("GET", f"/prompt_manager/{prefix}/nope.js")
|
|
self.assertEqual(resp.status, 404, prefix)
|
|
|
|
async def test_directory_is_404(self):
|
|
web_dir = self._fake_root()
|
|
(web_dir / "lib" / "dir.js").mkdir()
|
|
resp = await self.client.request("GET", "/prompt_manager/lib/dir.js")
|
|
self.assertEqual(resp.status, 404)
|
|
|
|
async def test_root_of_static_dir_is_not_listed(self):
|
|
resp = await self.client.request("GET", "/prompt_manager/lib/tailwind")
|
|
self.assertEqual(resp.status, 403)
|
|
|
|
|
|
# ── instance helpers ──────────────────────────────────────────────────
|
|
|
|
|
|
class TestInstanceHelpers(FolderPathsFixture):
|
|
|
|
def setUp(self):
|
|
super().setUp()
|
|
self.api = PromptManagerAPI(
|
|
db=PromptDatabase(os.path.join(self.tmpdir, "prompts.db"))
|
|
)
|
|
self.addCleanup(self.api.db.close_all) # runs before the rmtree cleanup
|
|
orig_dirs = list(GalleryConfig.MONITORING_DIRECTORIES)
|
|
self.addCleanup(setattr, GalleryConfig, "MONITORING_DIRECTORIES", orig_dirs)
|
|
|
|
def _run(self, coro):
|
|
import asyncio
|
|
|
|
return asyncio.run(coro)
|
|
|
|
def test_run_in_executor_with_kwargs(self):
|
|
def add(a, b=0):
|
|
return a + b
|
|
|
|
self.assertEqual(self._run(self.api._run_in_executor(add, 1, b=2)), 3)
|
|
self.assertEqual(self._run(self.api._run_in_executor(add, 1, 2)), 3)
|
|
|
|
def test_invalidate_gallery_cache(self):
|
|
record = self.api._root_listing(self.output_dir)
|
|
record.files = [(self.output_dir / "a.png", 1.0)]
|
|
record.folders = record.folders_with_ancestors = frozenset()
|
|
record.walked_at = 1.0
|
|
self.api._favorite_listing_cache = {"x": 1}
|
|
self.api._workspace_order_cache = {"x": 1}
|
|
self.api.invalidate_gallery_cache()
|
|
# The record stays: its generation outlives every drop (#348).
|
|
self.assertIs(self.api._root_listings[str(self.output_dir)], record)
|
|
self.assertIsNone(record.files)
|
|
self.assertIsNone(record.folders)
|
|
self.assertIsNone(record.walked_at)
|
|
self.assertEqual(record.generation, 1)
|
|
self.assertEqual(self.api._favorite_listing_cache, {})
|
|
self.assertEqual(self.api._workspace_order_cache, {})
|
|
self.api.reset_gallery_roots_cache()
|
|
self.assertEqual(record.generation, 2)
|
|
|
|
def test_enrich_images_adds_urls_and_thumbnails(self):
|
|
GalleryConfig.MONITORING_DIRECTORIES = [str(self.output_dir)]
|
|
img = self.output_dir / "sub" / "a.png"
|
|
img.parent.mkdir()
|
|
img.write_bytes(b"x")
|
|
thumb = self.output_dir / "thumbnails" / "sub" / "a_thumb.png"
|
|
thumb.parent.mkdir(parents=True)
|
|
thumb.write_bytes(b"t")
|
|
outside = Path(self.tmpdir) / "elsewhere.png"
|
|
outside.write_bytes(b"o")
|
|
|
|
images = [
|
|
{"id": 7, "image_path": str(img)},
|
|
{"id": 8, "image_path": str(outside)},
|
|
{"id": 9, "image_path": ""},
|
|
{"image_path": str(self.output_dir / "missing.png")},
|
|
]
|
|
result = self.api._enrich_images(images)
|
|
|
|
self.assertEqual(result[0]["url"], "/prompt_manager/images/serve/sub/a.png")
|
|
self.assertRegex(
|
|
result[0]["thumbnail_url"],
|
|
r"^/prompt_manager/images/serve/thumbnails/sub/a_thumb\.png\?v=\d+$",
|
|
)
|
|
self.assertEqual(result[0]["relative_path"], os.path.join("sub", "a.png"))
|
|
self.assertEqual(result[1]["url"], "/prompt_manager/images/8/file")
|
|
self.assertNotIn("relative_path", result[1])
|
|
self.assertNotIn("url", result[2])
|
|
self.assertEqual(result[3]["url"], "/prompt_manager/images/serve/missing.png")
|
|
self.assertNotIn("thumbnail_url", result[3])
|
|
|
|
def test_enrich_prompt_images_walks_prompts(self):
|
|
GalleryConfig.MONITORING_DIRECTORIES = [str(self.output_dir)]
|
|
prompts = [
|
|
{"images": [{"id": 1, "image_path": str(self.output_dir / "p.png")}]},
|
|
{},
|
|
]
|
|
result = self.api._enrich_prompt_images(prompts)
|
|
self.assertEqual(
|
|
result[0]["images"][0]["url"], "/prompt_manager/images/serve/p.png"
|
|
)
|
|
|
|
def test_clean_nan_recursive(self):
|
|
nan = float("nan")
|
|
cleaned = self.api._clean_nan_recursive(
|
|
{"a": nan, "b": [nan, 1.5, {"c": nan}], "d": "x"}
|
|
)
|
|
self.assertEqual(cleaned, {"a": None, "b": [None, 1.5, {"c": None}], "d": "x"})
|
|
|
|
def test_find_output_dir_uses_configured_and_caches(self):
|
|
GalleryConfig.MONITORING_DIRECTORIES = [str(self.output_dir)]
|
|
self.api._cached_output_dir = None
|
|
first = self.api._find_comfyui_output_dir()
|
|
self.assertEqual(first, str(self.output_dir.resolve()))
|
|
GalleryConfig.MONITORING_DIRECTORIES = []
|
|
self.assertEqual(self.api._find_comfyui_output_dir(), first)
|
|
|
|
def test_find_output_dir_skips_configured_root_outside_comfyui(self):
|
|
outside = Path(self.tmpdir) / "outside"
|
|
outside.mkdir()
|
|
GalleryConfig.MONITORING_DIRECTORIES = [str(outside)]
|
|
self.api._cached_output_dir = None
|
|
result = self.api._find_comfyui_output_dir()
|
|
self.assertNotEqual(
|
|
os.path.normcase(str(result or "")), os.path.normcase(str(outside))
|
|
)
|
|
|
|
def test_find_output_dir_skips_missing_configured(self):
|
|
GalleryConfig.MONITORING_DIRECTORIES = [os.path.join(self.tmpdir, "nope")]
|
|
self.api._cached_output_dir = None
|
|
result = self.api._find_comfyui_output_dir()
|
|
self.assertNotEqual(result, os.path.join(self.tmpdir, "nope"))
|
|
|
|
|
|
# ── metadata parsing ──────────────────────────────────────────────────
|
|
|
|
|
|
class TestMetadataParsing(unittest.TestCase):
|
|
|
|
def setUp(self):
|
|
database = MagicMock()
|
|
database.cleanup_duplicates.return_value = 0
|
|
self.api = PromptManagerAPI(db=database)
|
|
self.tmpdir = tempfile.mkdtemp()
|
|
self.addCleanup(shutil.rmtree, self.tmpdir, True)
|
|
|
|
def test_extract_metadata_from_png_and_garbage(self):
|
|
from PIL import Image, PngImagePlugin
|
|
|
|
path = os.path.join(self.tmpdir, "m.png")
|
|
info = PngImagePlugin.PngInfo()
|
|
info.add_text("prompt", "{}")
|
|
Image.new("RGB", (1, 1)).save(path, pnginfo=info)
|
|
self.assertEqual(self.api._extract_comfyui_metadata(path), {"prompt": "{}"})
|
|
|
|
garbage = os.path.join(self.tmpdir, "g.png")
|
|
with open(garbage, "wb") as f:
|
|
f.write(b"nope")
|
|
self.assertEqual(self.api._extract_comfyui_metadata(garbage), {})
|
|
|
|
def test_parse_a1111_parameters(self):
|
|
meta = {"parameters": "a cat\nNegative prompt: dog\nSteps: 20"}
|
|
parsed = self.api._parse_comfyui_prompt(meta)
|
|
self.assertEqual(parsed["positive_prompt"], "a cat")
|
|
self.assertEqual(parsed["negative_prompt"], "dog")
|
|
self.assertEqual(parsed["parameters"]["parameters"], meta["parameters"])
|
|
self.assertEqual(self.api._extract_readable_prompt(parsed), "a cat")
|
|
|
|
def test_parse_comfyui_fields(self):
|
|
meta = {
|
|
"prompt": json.dumps({"1": {"class_type": "X", "inputs": {}}}),
|
|
"workflow": "not json",
|
|
"steps": "20",
|
|
"cfg": "not-json-either",
|
|
}
|
|
parsed = self.api._parse_comfyui_prompt(meta)
|
|
self.assertIsInstance(parsed["prompt"], dict)
|
|
self.assertEqual(parsed["workflow"], "not json")
|
|
self.assertEqual(parsed["parameters"]["steps"], 20)
|
|
self.assertEqual(parsed["parameters"]["cfg"], "not-json-either")
|
|
|
|
parsed = self.api._parse_comfyui_prompt(
|
|
{"prompt": "plain text", "workflow": json.dumps({"nodes": []})}
|
|
)
|
|
self.assertEqual(parsed["prompt"], "plain text")
|
|
self.assertEqual(parsed["workflow"], {"nodes": []})
|
|
|
|
def test_extract_readable_prompt_variants(self):
|
|
extract = self.api._extract_readable_prompt
|
|
self.assertEqual(extract({"prompt": "plain"}), "plain")
|
|
self.assertEqual(extract({"prompt": ["a", "", "b"]}), "a b")
|
|
self.assertEqual(extract({"prompt": 42}), "42")
|
|
self.assertEqual(
|
|
extract({"prompt": None, "parameters": {"positive": ["x", "y"]}}), "x y"
|
|
)
|
|
self.assertIsNone(extract({"prompt": {}, "parameters": {}}))
|
|
|
|
graph = {
|
|
"1": {"class_type": "CLIPTextEncode", "inputs": {"text": "pos"}},
|
|
"2": {"class_type": "CLIPTextEncode", "inputs": {"text": "neg"}},
|
|
"3": {
|
|
"class_type": "KSampler",
|
|
"inputs": {"positive": ["1", 0], "negative": ["2", 0]},
|
|
},
|
|
}
|
|
self.assertEqual(extract({"prompt": graph}), "pos")
|
|
self.assertEqual(extract({"prompt": {"x": 1}, "workflow": graph}), "pos")
|
|
|
|
def test_node_inputs_and_text_lookup(self):
|
|
self.assertEqual(self.api._get_node_inputs("nope"), {})
|
|
self.assertEqual(self.api._get_node_inputs({"inputs": 5}), {})
|
|
self.assertEqual(self.api._get_node_inputs({"inputs": {"a": 1}}), {"a": 1})
|
|
listed = self.api._get_node_inputs(
|
|
{"inputs": [{"name": "text", "link": None}, {"nope": 1}, "junk"]}
|
|
)
|
|
self.assertEqual(list(listed), ["text"])
|
|
|
|
self.assertIsNone(self.api._find_text_in_node(None))
|
|
self.assertEqual(self.api._find_text_in_node({"inputs": {"text": "hi"}}), "hi")
|
|
self.assertEqual(
|
|
self.api._find_text_in_node(
|
|
{"type": "CLIPTextEncode", "widgets_values": ["w"]}
|
|
),
|
|
"w",
|
|
)
|
|
self.assertIsNone(
|
|
self.api._find_text_in_node(
|
|
{"type": "CLIPTextEncode", "widgets_values": [" "]}
|
|
)
|
|
)
|
|
self.assertIsNone(self.api._find_text_in_node({"type": "KSampler"}))
|
|
|
|
def test_positive_prompt_from_nodes_array_and_fallbacks(self):
|
|
extract = self.api._extract_positive_prompt_from_comfyui_data
|
|
self.assertIsNone(extract("x"))
|
|
self.assertIsNone(extract({}))
|
|
self.assertIsNone(extract({"nodes": []}))
|
|
|
|
workflow = {
|
|
"nodes": [
|
|
{"id": 1, "type": "CLIPTextEncode", "widgets_values": ["pos"]},
|
|
{
|
|
"id": 2,
|
|
"type": "CLIPTextEncode",
|
|
"widgets_values": ["neg"],
|
|
"title": "Negative",
|
|
},
|
|
{"id": 3, "type": "KSampler", "inputs": [{"name": "seed", "link": 4}]},
|
|
"junk",
|
|
]
|
|
}
|
|
self.assertEqual(extract(workflow), "pos")
|
|
|
|
only_negative = {
|
|
"nodes": [
|
|
{
|
|
"id": 2,
|
|
"type": "CLIPTextEncode",
|
|
"widgets_values": ["neg"],
|
|
"title": "neg",
|
|
},
|
|
]
|
|
}
|
|
self.assertEqual(extract(only_negative), "neg")
|
|
|
|
bad_link = {
|
|
"1": {"class_type": "CLIPTextEncode", "inputs": {"text": "pos"}},
|
|
"3": {
|
|
"class_type": "KSampler",
|
|
"inputs": {"positive": "oops", "negative": []},
|
|
},
|
|
"4": {
|
|
"class_type": "KSampler",
|
|
"inputs": {"positive": ["zz", 0], "negative": []},
|
|
},
|
|
}
|
|
self.assertEqual(extract(bad_link), "pos")
|
|
|
|
|
|
if __name__ == "__main__":
|
|
unittest.main()
|