2 Commits

Include renames

Author	SHA1	Message	Date
Vito	69de20a427	feat(system): Sources & scans (#309) ... * feat(system): report configured, effective and watched sources GET /prompt_manager/settings adds settings.monitoring_enabled, a sources block (configured roots with valid/code/reason, effective roots with their listing index and watch state, auto_detected) and the monitor state (present, observer_alive, scheduled). Every path is a public path. A source is "watching" only while a live observer scheduled it, "manual" when monitoring is off or the process has no monitor, and otherwise "restart_pending". POST accepts monitoring_enabled as a strict bool, saved under gallery.monitoring.enabled; restart_required_fields names it when the running observer does not already match. A save that changes the sources runs as a "Source change" records hold, so it is refused with 409 busy while a scan runs. Case variants of one folder are saved once. POST /prompt_manager/settings/validate-source {path} checks a path lexically (no "..", no control characters, native path forms only) and then against the gallery-root rules, answering a code and reason, and writes nothing. * fix(system): no existence oracle in source validation A gallery root's containment in the allowed ComfyUI folders is now decided before its existence and type, so a path outside them answers "outside-allowed" whether or not it exists. validate-source also folds the filesystem-root and home-directory codes into "outside-allowed", with one reason, and logs the detail server-side only. Paths inside the allowed folders still say "not-found" or "not-directory". The legacy settings save shares the same check. * feat(scan): scan one selected source POST /prompt_manager/jobs/scan accepts roots: indexes into the effective sources, strict non-negative ints (no bools or floats), deduplicated, at most 64, non-empty when present. An index past the sources is a 400 "unknown-source" and starts nothing. The job walks only the selected sources, resolved when it was requested, and keeps each file's listing root_index. Its scope records files, roots (null for every source) and output_folders, the number of folders walked, which the log's first line counts. The legacy /scan route still scans every source, and attaching to a scoped job keeps its payload keys. * feat(system): manage sources and start scans natively System > Sources & scans renders the concept's section on the server's sources: one row per effective source with its watch state and a scan scoped to that source, the 3.2.0 update notice with the legacy dismissal, Add source checked by validate-source, Remove, monitoring and worker threads as host drafts behind the one Save row, and Refresh files. A save refused by a running scan names the job. Thumbnail generation hands off to the legacy gallery until S5. Deviations D98, D113 and D-NEW-S4-1 are recorded in TESTING.md. * fix(security): decide source containment as text before any filesystem access Every caller (validate-source, POST /settings, config.json at load) now refuses a source through one check, sources.source_problem: the form check first, then GalleryConfig.gallery_root_problem, which compares the normalised spelling with the allowed folders before anything resolves, stats or opens the path. A UNC share, the \\?\ and \\.\ namespaces, a device or any outside folder is refused with no SMB request, no executor stall and no existence or timing oracle; only then is the path resolved and contained again, which still catches a symlink out. The filesystem root and the home directory answer outside-allowed with the same reason as any outside path, so no caller learns the home path. Also: - POST /settings runs the form checks and parses the roots off the loop; an unusable source is a coded 400 invalid-source naming its public path (as text) and its index. - ComfyUI folders outside the base directory are path anchors, so their sources round-trip and never collide by folder name. - A per-source scan sends each index with its public path; a stale or shifted index is 409 sources-changed and starts nothing. - A source still watched after monitoring was switched off is "stopping". - Roots are compared as folders, not spellings, so an unchanged legacy resend takes no source-change hold. - Tests: no realpath/stat/lstat of an outside path (native and Windows forms with ntpath fakes), the alike answers with /, ~ and a link to ~, an event instead of a timer wait, and a teardown that drains jobs before the temporary database goes. * fix(system): say what Sources saved, refused and still watches - A per-source scan sends the source's public path; when the sources changed since they were read the toast says the server's reason and the list and saved values are read again. - "watching until restart" for a source the running monitor still watches after monitoring was saved off, and the status adds "Restart ComfyUI to apply: Watch for new outputs." from restart_required_fields. - A save refused for one unusable source marks that row and the status is the server's message naming it. - A valid configured source the effective list holds under another spelling reads "same folder as another source". - The busy toast names the job the 409 named (ApiError keeps a validated running job and refused source), falling back to the list. - The Add source dialog's listeners live on the page scope and Enter adds; its markup is compared with the concept (D127). - TESTING.md: D-NEW-S4-1 is D127; D-NEW-S4-2 records the new copy; the Sources contract describes the lexical containment, invalid-source and sources-changed. * fix(security): guard the Windows form probes on a token of their own The no-filesystem guard matched "C:", which on a Windows runner is also the temporary ComfyUI tree the save may resolve for its trusted first entry. The probe paths now carry a token no trusted directory holds. * docs(system): number the Sources states deviation D128	2026-10-23 13:15:06 -07:00
Vito	3b1d00501f	fix(config): serialized config store and registry admission (#289) ... * fix(config): report settings persistence failures (#220) Settings saves now go through one serialized config store (py/config_store.py). A commit re-reads config.json under a process lock, applies the change to a copy and replaces the file atomically (temp file beside it, fsync, close, os.replace), so a failed write leaves the old file in place. POST /prompt_manager/settings commits first and only then publishes the values at runtime. A write failure is 500 config-write-failed, an unparseable config.json is 409 config-unreadable (the file is kept instead of replaced) and a stale expected_revision is 409 settings-changed. A save reports persisted, revision, runtime_applied and restart_required_fields; GET adds the revision. Only submitted keys are written; other sections and the CivitAI key are kept. * fix(lora): report integration save failures (#220) POST /prompt_manager/lora/enable commits integrations.lora_manager through the config store and publishes the runtime values only after config.json holds them. Before, it set the runtime values, rewrote the whole runtime config (dropping sections it did not know) and reported success even when the write had failed. A write failure is 500 config-write-failed and changes nothing at runtime; an unparseable config.json is 409 config-unreadable. The body goes through the capped JSON reader and the flags must be booleans. A trigger word cache that fails to load is reported as persisted:true, cache_loaded:false. The CivitAI key is never echoed. The LoRA tests now write to a temporary config path. * fix(jobs): refuse legacy record-changing operations while a job runs The job registry gains admission classes. A records job is refused while any records or guarded job runs, a guarded job while a records job runs, and read jobs share only the one-job-per-kind rule. record_changing=True stays as the spelling of records. Job summaries add cancellable and admission, and a non-cancellable job ignores cancel. JobRegistry.hold() registers a running, non-cancellable job for a legacy-page operation and ends it when the request does. The legacy restore, maintenance, prompt cleanup, duplicate image delete, LoRA import and batch auto-tag routes hold as records; thumbnail generation (both routes) and thumbnail clear hold as guarded. A maintenance run of only read checks holds as read. So a restore during a scan, or a scan during a legacy restore, is refused with the running job named instead of interleaving; this closes the gap noted on #269. Refusals keep each route's shape: the JSON routes return the legacy 409 "<title> already running", and the thumbnail and LoRA streams send their existing error event. * test(lora): keep the preserved-section fixture free of secret values CodeQL flagged the fixture that writes the secret constant into a test config file as clear-text storage. The test only needs a marker value to prove the stored key survives a save. * fix(config): read under the commit lock and publish saves in write order - ConfigStore.read takes the commit lock, so no read handle is open while config.json is replaced (Windows refuses that replace). - os.replace is retried briefly on a Windows PermissionError (antivirus or indexer handles) before the save is reported as not written. - The temp file is written beside realpath(config.json), so a symlinked config keeps its link and a bind mount fails truthfully. - commit() takes a publish callback that runs under the lock after the write; settings and LoRA saves publish there, so two concurrent saves reach the runtime in write order and the runtime ends on the last one. - The concurrency tests now wait until the second caller is blocked on the lock (tests/lock_probe.py) before releasing the first. * fix(jobs): require an admission class, admit holds on entry, true Activity states - JobRegistry.start raises TypeError when neither admission nor record_changing is given, so a new job never runs as "read" beside a scan by omission. - hold() admits when its block is entered, so a hold that is never entered leaves no running records job behind; callers that answer a refusal in their own way check blocker() first, with no await before the block. - A held SSE operation that ends in an error event (thumbnail generation, legacy batch auto-tag) is listed as failed, and refused input (duplicate delete without paths, an unreadable thumbnail body) is answered before the hold, so it is not listed as a job. - Activity keeps a job's cancellable flag (true when an older server omits it) and shows Cancel only for a running, cancellable job, in Recent work and in the job view. A held legacy job no longer offers a Cancel that does nothing. * fix(admin): legacy pages show why a request was refused A job running in the workspace refuses the legacy record-changing requests, and a corrupt config.json refuses a settings save. The legacy pages now name the reason instead of hiding it: - LoRA import: the busy and failure frames carry status and progress, and the admin modal handles type "error": it shows the message, notifies and closes, instead of staying on "Initializing..." at 0%. - Legacy batch auto-tag: a refusal by another job is a 200 stream with one error frame ({type, status, message}), which both pages already render; a second batch keeps its exact JSON 409. - Gallery cache clear: a refused server clear shows the reason and stops before the success toast and the "Not generated" reset. - Admin settings save and maintenance show the reply's error ("config.json cannot be read...", "Scan for metadata already running") instead of "Failed to save settings" or "HTTP 409: Conflict". - Tests cover the read-only /maintenance branch: a read-only selection runs during a scan as a read hold; a mixed or default selection is refused. * refactor(api): move the settings routes out of admin.py GET and POST /prompt_manager/settings, their validation, the config commit and runtime publication, and the shared commit-failure reply now live in py/api/settings_routes.py as SettingsRoutesMixin, before S3 adds host fields there. No behaviour change: the same routes, payloads and status codes; admin.py drops about 300 lines. * chore(coverage): raise the floors of the frontend files this round touched admin.js, gallery.js, activity.mjs and activity-job-view.mjs, to their measured coverage. No floor is lowered. * fix(admin): the gallery starts no thumbnail status read while it leaves The gallery's startup thumbnail check runs one second after load. When that second ended while the page was leaving (CI, WebKit: the workflow test navigates from the gallery to the admin page), WebKit failed the new read with "Fetch API cannot load ... due to access control checks." and Playwright reported a page error. Like the admin page (#290), the gallery now has a page lifetime that ends on beforeunload and pagehide (and starts again on a back/forward restore). The check starts no read once it has ended and passes it as the read's signal. A page-clock workflow test that leaves before the second ends failed on both events, in Chromium and WebKit, before this change.	2026-10-22 21:39:01 -07:00