Vito
3b1d00501f
* fix(config): report settings persistence failures (#220) Settings saves now go through one serialized config store (py/config_store.py). A commit re-reads config.json under a process lock, applies the change to a copy and replaces the file atomically (temp file beside it, fsync, close, os.replace), so a failed write leaves the old file in place. POST /prompt_manager/settings commits first and only then publishes the values at runtime. A write failure is 500 config-write-failed, an unparseable config.json is 409 config-unreadable (the file is kept instead of replaced) and a stale expected_revision is 409 settings-changed. A save reports persisted, revision, runtime_applied and restart_required_fields; GET adds the revision. Only submitted keys are written; other sections and the CivitAI key are kept. * fix(lora): report integration save failures (#220) POST /prompt_manager/lora/enable commits integrations.lora_manager through the config store and publishes the runtime values only after config.json holds them. Before, it set the runtime values, rewrote the whole runtime config (dropping sections it did not know) and reported success even when the write had failed. A write failure is 500 config-write-failed and changes nothing at runtime; an unparseable config.json is 409 config-unreadable. The body goes through the capped JSON reader and the flags must be booleans. A trigger word cache that fails to load is reported as persisted:true, cache_loaded:false. The CivitAI key is never echoed. The LoRA tests now write to a temporary config path. * fix(jobs): refuse legacy record-changing operations while a job runs The job registry gains admission classes. A records job is refused while any records or guarded job runs, a guarded job while a records job runs, and read jobs share only the one-job-per-kind rule. record_changing=True stays as the spelling of records. Job summaries add cancellable and admission, and a non-cancellable job ignores cancel. JobRegistry.hold() registers a running, non-cancellable job for a legacy-page operation and ends it when the request does. The legacy restore, maintenance, prompt cleanup, duplicate image delete, LoRA import and batch auto-tag routes hold as records; thumbnail generation (both routes) and thumbnail clear hold as guarded. A maintenance run of only read checks holds as read. So a restore during a scan, or a scan during a legacy restore, is refused with the running job named instead of interleaving; this closes the gap noted on #269. Refusals keep each route's shape: the JSON routes return the legacy 409 "<title> already running", and the thumbnail and LoRA streams send their existing error event. * test(lora): keep the preserved-section fixture free of secret values CodeQL flagged the fixture that writes the secret constant into a test config file as clear-text storage. The test only needs a marker value to prove the stored key survives a save. * fix(config): read under the commit lock and publish saves in write order - ConfigStore.read takes the commit lock, so no read handle is open while config.json is replaced (Windows refuses that replace). - os.replace is retried briefly on a Windows PermissionError (antivirus or indexer handles) before the save is reported as not written. - The temp file is written beside realpath(config.json), so a symlinked config keeps its link and a bind mount fails truthfully. - commit() takes a publish callback that runs under the lock after the write; settings and LoRA saves publish there, so two concurrent saves reach the runtime in write order and the runtime ends on the last one. - The concurrency tests now wait until the second caller is blocked on the lock (tests/lock_probe.py) before releasing the first. * fix(jobs): require an admission class, admit holds on entry, true Activity states - JobRegistry.start raises TypeError when neither admission nor record_changing is given, so a new job never runs as "read" beside a scan by omission. - hold() admits when its block is entered, so a hold that is never entered leaves no running records job behind; callers that answer a refusal in their own way check blocker() first, with no await before the block. - A held SSE operation that ends in an error event (thumbnail generation, legacy batch auto-tag) is listed as failed, and refused input (duplicate delete without paths, an unreadable thumbnail body) is answered before the hold, so it is not listed as a job. - Activity keeps a job's cancellable flag (true when an older server omits it) and shows Cancel only for a running, cancellable job, in Recent work and in the job view. A held legacy job no longer offers a Cancel that does nothing. * fix(admin): legacy pages show why a request was refused A job running in the workspace refuses the legacy record-changing requests, and a corrupt config.json refuses a settings save. The legacy pages now name the reason instead of hiding it: - LoRA import: the busy and failure frames carry status and progress, and the admin modal handles type "error": it shows the message, notifies and closes, instead of staying on "Initializing..." at 0%. - Legacy batch auto-tag: a refusal by another job is a 200 stream with one error frame ({type, status, message}), which both pages already render; a second batch keeps its exact JSON 409. - Gallery cache clear: a refused server clear shows the reason and stops before the success toast and the "Not generated" reset. - Admin settings save and maintenance show the reply's error ("config.json cannot be read...", "Scan for metadata already running") instead of "Failed to save settings" or "HTTP 409: Conflict". - Tests cover the read-only /maintenance branch: a read-only selection runs during a scan as a read hold; a mixed or default selection is refused. * refactor(api): move the settings routes out of admin.py GET and POST /prompt_manager/settings, their validation, the config commit and runtime publication, and the shared commit-failure reply now live in py/api/settings_routes.py as SettingsRoutesMixin, before S3 adds host fields there. No behaviour change: the same routes, payloads and status codes; admin.py drops about 300 lines. * chore(coverage): raise the floors of the frontend files this round touched admin.js, gallery.js, activity.mjs and activity-job-view.mjs, to their measured coverage. No floor is lowered. * fix(admin): the gallery starts no thumbnail status read while it leaves The gallery's startup thumbnail check runs one second after load. When that second ended while the page was leaving (CI, WebKit: the workflow test navigates from the gallery to the admin page), WebKit failed the new read with "Fetch API cannot load ... due to access control checks." and Playwright reported a page error. Like the admin page (#290), the gallery now has a page lifetime that ends on beforeunload and pagehide (and starts again on a back/forward restore). The check starts no read once it has ended and passes it as the read's signal. A page-clock workflow test that leaves before the second ends failed on both events, in Chromium and WebKit, before this change.
420 lines
16 KiB
Python
420 lines
16 KiB
Python
"""Tests for the serialized config store (py/config_store.py, #220).
|
|
|
|
Every test works on a config.json inside its own temporary directory; the
|
|
repository's config.json (live user data) is never read or written.
|
|
"""
|
|
|
|
import hashlib
|
|
import json
|
|
import os
|
|
import shutil
|
|
import sys
|
|
import tempfile
|
|
import threading
|
|
import unittest
|
|
from pathlib import Path
|
|
from unittest.mock import MagicMock, patch
|
|
|
|
_mock_server = MagicMock()
|
|
_mock_server.PromptServer.instance.routes = MagicMock()
|
|
sys.modules.setdefault("server", _mock_server)
|
|
|
|
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
|
|
|
from py import config_store # noqa: E402
|
|
from py.config_store import ( # noqa: E402
|
|
ABSENT_REVISION,
|
|
ConfigStore,
|
|
ConfigUnreadable,
|
|
RevisionConflict,
|
|
get_config_store,
|
|
revision_of,
|
|
)
|
|
|
|
try:
|
|
from tests.lock_probe import ProbeLock # noqa: E402
|
|
from tests.open_handles import assert_closed # noqa: E402
|
|
except ImportError: # discovered with tests/ as the top-level directory
|
|
from lock_probe import ProbeLock # noqa: E402
|
|
from open_handles import assert_closed # noqa: E402
|
|
|
|
|
|
class ConfigStoreTests(unittest.TestCase):
|
|
def setUp(self):
|
|
self.tmpdir = os.path.realpath(tempfile.mkdtemp())
|
|
self.path = os.path.join(self.tmpdir, "config.json")
|
|
self.store = ConfigStore(lambda: self.path)
|
|
|
|
def tearDown(self):
|
|
try:
|
|
assert_closed(self, self.tmpdir)
|
|
finally:
|
|
shutil.rmtree(self.tmpdir, ignore_errors=True)
|
|
|
|
def _write(self, data):
|
|
Path(self.path).write_text(json.dumps(data), encoding="utf-8")
|
|
|
|
def _bytes(self):
|
|
return Path(self.path).read_bytes()
|
|
|
|
def _leftovers(self):
|
|
return sorted(name for name in os.listdir(self.tmpdir) if name != "config.json")
|
|
|
|
# ── read ──────────────────────────────────────────────────────────
|
|
|
|
def test_read_of_a_missing_file_is_empty_and_absent(self):
|
|
self.assertEqual(self.store.read(), ({}, ABSENT_REVISION))
|
|
|
|
def test_read_reports_the_sha256_of_the_file_bytes(self):
|
|
self._write({"web_ui": {"result_timeout": 3}})
|
|
data, revision = self.store.read()
|
|
self.assertEqual(data, {"web_ui": {"result_timeout": 3}})
|
|
expected = "sha256:" + hashlib.sha256(self._bytes()).hexdigest()
|
|
self.assertEqual(revision, expected)
|
|
self.assertEqual(revision_of(self._bytes()), expected)
|
|
|
|
def test_non_object_config_is_unreadable(self):
|
|
self._write([1, 2])
|
|
with self.assertRaises(ConfigUnreadable):
|
|
self.store.read()
|
|
|
|
def test_unreadable_path_is_unreadable(self):
|
|
os.mkdir(self.path) # a directory where the file should be
|
|
with self.assertRaises(ConfigUnreadable):
|
|
self.store.read()
|
|
|
|
# ── commit ────────────────────────────────────────────────────────
|
|
|
|
def test_commit_preserves_unknown_sections_and_secret(self):
|
|
self._write(
|
|
{
|
|
"custom_section": {"keep": True},
|
|
"integrations": {"lora_manager": {"civitai_api_key": "secret"}},
|
|
"web_ui": {"result_timeout": 1, "show_test_button": True},
|
|
}
|
|
)
|
|
|
|
def mutate(config):
|
|
config.setdefault("web_ui", {})["result_timeout"] = 7
|
|
return config
|
|
|
|
saved, revision = self.store.commit(mutate, expected_revision=None)
|
|
|
|
on_disk = json.loads(self._bytes())
|
|
self.assertEqual(saved, on_disk)
|
|
self.assertEqual(on_disk["custom_section"], {"keep": True})
|
|
self.assertEqual(
|
|
on_disk["integrations"]["lora_manager"]["civitai_api_key"], "secret"
|
|
)
|
|
self.assertEqual(
|
|
on_disk["web_ui"], {"result_timeout": 7, "show_test_button": True}
|
|
)
|
|
self.assertEqual(revision, revision_of(self._bytes()))
|
|
self.assertEqual(self.store.read(), (on_disk, revision))
|
|
|
|
def test_commit_creates_a_missing_file_and_its_parent(self):
|
|
self.path = os.path.join(self.tmpdir, "nested", "config.json")
|
|
saved, revision = self.store.commit(
|
|
lambda config: {**config, "a": 1}, expected_revision=ABSENT_REVISION
|
|
)
|
|
self.assertEqual(saved, {"a": 1})
|
|
self.assertEqual(json.loads(Path(self.path).read_text()), {"a": 1})
|
|
self.assertNotEqual(revision, ABSENT_REVISION)
|
|
|
|
def test_mutate_works_on_a_copy(self):
|
|
self._write({"web_ui": {"result_timeout": 1}})
|
|
seen = {}
|
|
|
|
def mutate(config):
|
|
seen["config"] = config
|
|
config["web_ui"]["result_timeout"] = 2
|
|
raise RuntimeError("validation failed late")
|
|
|
|
before = self._bytes()
|
|
with self.assertRaises(RuntimeError):
|
|
self.store.commit(mutate, expected_revision=None)
|
|
self.assertEqual(self._bytes(), before)
|
|
self.assertEqual(self.store.read()[0], {"web_ui": {"result_timeout": 1}})
|
|
|
|
def test_mutate_must_return_an_object(self):
|
|
self._write({"a": 1})
|
|
before = self._bytes()
|
|
with self.assertRaises(TypeError):
|
|
self.store.commit(lambda config: None, expected_revision=None)
|
|
self.assertEqual(self._bytes(), before)
|
|
|
|
def test_malformed_existing_config_is_refused_and_untouched(self):
|
|
Path(self.path).write_bytes(b"{not json")
|
|
calls = []
|
|
with self.assertRaises(ConfigUnreadable):
|
|
self.store.commit(
|
|
lambda config: calls.append(config) or config, expected_revision=None
|
|
)
|
|
self.assertEqual(calls, [])
|
|
self.assertEqual(self._bytes(), b"{not json")
|
|
self.assertEqual(self._leftovers(), [])
|
|
|
|
def test_write_failure_leaves_bytes_and_raises(self):
|
|
self._write({"a": 1})
|
|
before = self._bytes()
|
|
with patch.object(config_store.os, "fsync", side_effect=OSError("disk full")):
|
|
with self.assertRaises(OSError):
|
|
self.store.commit(lambda c: {**c, "b": 2}, expected_revision=None)
|
|
self.assertEqual(self._bytes(), before)
|
|
self.assertEqual(self._leftovers(), [])
|
|
|
|
def test_unwritable_directory_raises_oserror(self):
|
|
blocker = os.path.join(self.tmpdir, "afile")
|
|
Path(blocker).write_text("x")
|
|
self.path = os.path.join(blocker, "config.json") # parent is a file
|
|
with self.assertRaises(OSError):
|
|
self.store.commit(lambda c: {"a": 1}, expected_revision=None)
|
|
|
|
def test_stale_revision_conflicts_without_write(self):
|
|
self._write({"a": 1})
|
|
_data, revision = self.store.read()
|
|
self._write({"a": 2}) # another writer got there first
|
|
before = self._bytes()
|
|
calls = []
|
|
with self.assertRaises(RevisionConflict) as caught:
|
|
self.store.commit(
|
|
lambda c: calls.append(c) or c, expected_revision=revision
|
|
)
|
|
self.assertEqual(caught.exception.current, revision_of(before))
|
|
self.assertEqual(calls, [])
|
|
self.assertEqual(self._bytes(), before)
|
|
|
|
def test_matching_revision_commits(self):
|
|
self._write({"a": 1})
|
|
_data, revision = self.store.read()
|
|
saved, _ = self.store.commit(
|
|
lambda c: {**c, "b": 2}, expected_revision=revision
|
|
)
|
|
self.assertEqual(saved, {"a": 1, "b": 2})
|
|
|
|
def test_temp_file_removed_after_replace_failure(self):
|
|
self._write({"a": 1})
|
|
before = self._bytes()
|
|
with patch.object(
|
|
config_store.os, "replace", side_effect=PermissionError("locked")
|
|
):
|
|
with self.assertRaises(OSError):
|
|
self.store.commit(lambda c: {**c, "b": 2}, expected_revision=None)
|
|
self.assertEqual(self._bytes(), before)
|
|
self.assertEqual(self._leftovers(), [])
|
|
|
|
@unittest.skipUnless(os.name == "posix", "POSIX file modes")
|
|
def test_committed_file_is_private_to_the_user(self):
|
|
import stat
|
|
|
|
self._write({"a": 1})
|
|
os.chmod(self.path, 0o644)
|
|
self.store.commit(lambda c: c, expected_revision=None)
|
|
self.assertEqual(stat.S_IMODE(os.stat(self.path).st_mode), 0o600)
|
|
|
|
def _probe_lock(self):
|
|
probe = ProbeLock()
|
|
patcher = patch.object(config_store, "_COMMIT_LOCK", probe)
|
|
patcher.start()
|
|
self.addCleanup(patcher.stop)
|
|
return probe
|
|
|
|
def _hold_lock_in_a_commit(self, value="first"):
|
|
"""Start a commit that holds the lock until the returned event is set."""
|
|
inside = threading.Event()
|
|
release = threading.Event()
|
|
errors = []
|
|
|
|
def slow(config):
|
|
inside.set()
|
|
release.wait(10)
|
|
config[value] = 1
|
|
return config
|
|
|
|
def run():
|
|
try:
|
|
self.store.commit(slow, expected_revision=None)
|
|
except Exception as exc: # pragma: no cover - reported by the test
|
|
errors.append(exc)
|
|
|
|
thread = threading.Thread(target=run)
|
|
thread.start()
|
|
self.assertTrue(inside.wait(10))
|
|
return thread, release, errors
|
|
|
|
def test_concurrent_commits_serialize(self):
|
|
# The second commit is observed blocked on the lock before the first
|
|
# one may finish, so this fails whenever commit stops taking the lock.
|
|
self._write({"keep": True})
|
|
probe = self._probe_lock()
|
|
first, release, errors = self._hold_lock_in_a_commit()
|
|
calls = []
|
|
|
|
def fast(config):
|
|
calls.append(dict(config))
|
|
config["second"] = 2
|
|
return config
|
|
|
|
second = threading.Thread(
|
|
target=lambda: self.store.commit(fast, expected_revision=None)
|
|
)
|
|
second.start()
|
|
self.assertTrue(probe.contended.wait(10), "second commit never waited")
|
|
self.assertEqual(calls, []) # not called while the first holds the lock
|
|
release.set()
|
|
first.join(10)
|
|
second.join(10)
|
|
|
|
self.assertEqual(errors, [])
|
|
self.assertEqual(calls, [{"keep": True, "first": 1}]) # re-read inside
|
|
self.assertEqual(
|
|
json.loads(self._bytes()), {"keep": True, "first": 1, "second": 2}
|
|
)
|
|
|
|
def test_read_waits_for_a_running_commit(self):
|
|
# Windows cannot replace a file another handle has open, so a read
|
|
# runs under the commit lock and never overlaps a replace.
|
|
self._write({"keep": True})
|
|
probe = self._probe_lock()
|
|
first, release, errors = self._hold_lock_in_a_commit()
|
|
seen = []
|
|
reader = threading.Thread(target=lambda: seen.append(self.store.read()))
|
|
reader.start()
|
|
self.assertTrue(probe.contended.wait(10), "read did not take the lock")
|
|
self.assertEqual(seen, [])
|
|
release.set()
|
|
first.join(10)
|
|
reader.join(10)
|
|
self.assertEqual(errors, [])
|
|
self.assertEqual(seen[0][0], {"keep": True, "first": 1})
|
|
self.assertEqual(seen[0][1], revision_of(self._bytes()))
|
|
|
|
def test_replace_is_retried_after_a_transient_permission_error(self):
|
|
# Windows: an antivirus or indexer handle makes os.replace fail with
|
|
# PermissionError for a moment; the commit retries before failing.
|
|
self._write({"a": 1})
|
|
real_replace = os.replace
|
|
attempts = []
|
|
|
|
def flaky(src, dst):
|
|
attempts.append(dst)
|
|
if len(attempts) == 1:
|
|
raise PermissionError("in use by another process")
|
|
return real_replace(src, dst)
|
|
|
|
with (
|
|
patch.object(config_store, "_REPLACE_ATTEMPTS", 5),
|
|
patch.object(config_store.os, "replace", side_effect=flaky),
|
|
patch.object(config_store.time, "sleep") as sleep,
|
|
):
|
|
saved, _ = self.store.commit(
|
|
lambda c: {**c, "b": 2}, expected_revision=None
|
|
)
|
|
self.assertEqual(saved, {"a": 1, "b": 2})
|
|
self.assertEqual(json.loads(self._bytes()), {"a": 1, "b": 2})
|
|
self.assertEqual(len(attempts), 2)
|
|
sleep.assert_called_once_with(config_store._REPLACE_BACKOFF_SECONDS)
|
|
self.assertEqual(self._leftovers(), [])
|
|
|
|
def test_replace_gives_up_after_the_last_attempt(self):
|
|
self._write({"a": 1})
|
|
before = self._bytes()
|
|
with (
|
|
patch.object(config_store, "_REPLACE_ATTEMPTS", 3),
|
|
patch.object(
|
|
config_store.os, "replace", side_effect=PermissionError("locked")
|
|
) as replace,
|
|
patch.object(config_store.time, "sleep") as sleep,
|
|
):
|
|
with self.assertRaises(PermissionError):
|
|
self.store.commit(lambda c: {**c, "b": 2}, expected_revision=None)
|
|
self.assertEqual((replace.call_count, sleep.call_count), (3, 2))
|
|
self.assertEqual(self._bytes(), before)
|
|
self.assertEqual(self._leftovers(), [])
|
|
|
|
def test_replace_is_tried_once_outside_windows(self):
|
|
expected = 5 if os.name == "nt" else 1
|
|
self.assertEqual(config_store._REPLACE_ATTEMPTS, expected)
|
|
|
|
@unittest.skipUnless(hasattr(os, "symlink"), "needs symlinks")
|
|
def test_symlinked_config_is_written_through_the_link(self):
|
|
# A config.json that is a symlink (or a single-file bind mount) is
|
|
# written beside its real target, so the link stays a link.
|
|
real_dir = os.path.join(self.tmpdir, "real")
|
|
os.mkdir(real_dir)
|
|
target = os.path.join(real_dir, "settings.json")
|
|
Path(target).write_text(json.dumps({"a": 1}), encoding="utf-8")
|
|
try:
|
|
os.symlink(target, self.path)
|
|
except (OSError, NotImplementedError): # pragma: no cover - Windows
|
|
self.skipTest("symlinks are not permitted here")
|
|
|
|
saved, revision = self.store.commit(
|
|
lambda c: {**c, "b": 2}, expected_revision=None
|
|
)
|
|
|
|
self.assertTrue(os.path.islink(self.path))
|
|
self.assertEqual(json.loads(Path(target).read_text()), {"a": 1, "b": 2})
|
|
self.assertEqual(self.store.read(), (saved, revision))
|
|
self.assertEqual(os.listdir(real_dir), ["settings.json"])
|
|
|
|
def test_publish_runs_under_the_lock_after_the_write(self):
|
|
# Runtimes publish inside the commit, so two saves publish in the
|
|
# order they were written and the runtime ends on the last one.
|
|
self._write({"a": 1})
|
|
seen = []
|
|
|
|
# The lock is re-entrant, so another thread probes whether it is held.
|
|
def probe_publish(config):
|
|
result = []
|
|
thread = threading.Thread(
|
|
target=lambda: result.append(
|
|
config_store._COMMIT_LOCK.acquire(blocking=False)
|
|
)
|
|
)
|
|
thread.start()
|
|
thread.join(10)
|
|
seen.append((config, json.loads(self._bytes()), result[0]))
|
|
|
|
saved, _ = self.store.commit(
|
|
lambda c: {**c, "b": 2}, expected_revision=None, publish=probe_publish
|
|
)
|
|
self.assertEqual(seen, [(saved, {"a": 1, "b": 2}, False)])
|
|
|
|
def test_publish_is_skipped_when_the_write_fails(self):
|
|
self._write({"a": 1})
|
|
published = []
|
|
with patch.object(config_store.os, "fsync", side_effect=OSError("disk full")):
|
|
with self.assertRaises(OSError):
|
|
self.store.commit(
|
|
lambda c: {**c, "b": 2},
|
|
expected_revision=None,
|
|
publish=published.append,
|
|
)
|
|
self.assertEqual(published, [])
|
|
|
|
def test_handles_closed_after_commit(self):
|
|
self._write({"a": 1})
|
|
self.store.read()
|
|
self.store.commit(lambda c: {**c, "b": 2}, expected_revision=None)
|
|
assert_closed(self, self.tmpdir)
|
|
os.unlink(self.path) # Windows refuses this while a handle is open
|
|
self.assertEqual(self._leftovers(), [])
|
|
|
|
|
|
class SingletonTests(unittest.TestCase):
|
|
def test_get_config_store_is_one_process_store_on_the_configured_path(self):
|
|
store = get_config_store()
|
|
self.assertIs(store, get_config_store())
|
|
tmpdir = tempfile.mkdtemp()
|
|
try:
|
|
custom = os.path.join(tmpdir, "c.json")
|
|
with patch.dict(os.environ, {"PROMPT_MANAGER_CONFIG_PATH": custom}):
|
|
self.assertEqual(store.path, custom)
|
|
finally:
|
|
shutil.rmtree(tmpdir, ignore_errors=True)
|
|
|
|
|
|
if __name__ == "__main__":
|
|
unittest.main()
|