- Run a bounded model pipeline over the shared media limiter so model
preparation and failure bookkeeping overlap the previous model's tail.
New example_images_model_concurrency defaults to 2 (1 restores serial),
and metadata refreshes are serialized while media continues.
- Let the AIMD limiter probe one slot at a time above its base after a longer
success streak, up to example_images_download_max_concurrency (default 8).
A 429 still halves the limit below the base before recovery.
- Cache Civitai rendition -> blob redirects in Downloader.download_to_memory
for one hour (2048 entries, invalidated on 404), so retries skip the 301.
- Add scripts/benchmark_example_images.py, a reproducible old-vs-new A/B
harness that selects one cache/sidecar subset and runs serial, new-serial,
pipeline and aimd modes against the same temporary examples path.
- Document the measured results (19 models / 185 images, 3 repeats):
pipeline -23.4% vs new-serial, AIMD -22.2% vs pipeline, no 429s.
- Add an atomic .lm-examples-complete marker; only a folder with media and
the marker is skipped, so interrupted or legacy folders are re-checked by
the per-image existence path. Clear the marker before each pass.
- Treat processed/failed folders without a marker as pending so the
auto-download pre-check repairs them, and clear markers when provider
metadata is refreshed or custom images are imported.
- Bound stale-metadata refreshes at 90s so one hung provider cannot park the
serial run; count metadata-missing models as retryable.
- Record failure reasons in the progress file, expose failure_summary, and
render a no-media/unavailable/retry breakdown in the progress panel with a
warning completion toast.
- Add marker, partial-repair, timeout, failure-summary and UI tests.
- Blend a phase baseline with a 120s throughput window; reset the phase
on pause and skip bursts so a resumed run is not estimated from the
folders it skipped.
- Expose pause-aware elapsed_seconds and add a compact status response
for the 2s progress poll.
- Save .download_progress.json from both download loops; the old save
sat behind an early return on the common image path.
- Treat blank aiohttp errors and raised exceptions as transient, and
bound gallery media at 25s (videos 180s) instead of the shared 300s
socket read.
- Add tracker, manager, progress and timeout tests plus the full-run
measurements.
- Add a shared MediaDownloadLimiter (default 6 concurrent) with a per-run
429 cooldown and AIMD recovery; drop the empirical inter-model delay.
- Request width-aware Civitai renditions (800px images, transcoded 450px
videos) and map Genur signed/direct URLs to c.genur.art _small.webp or
_large.webp.
- Return an ImageDownloadOutcome so offline/connect/timeout/5xx failures
stay transient instead of being written as downloadFailed; track
retryable_models and leave those models pending.
- Force retries flagged Genur images and clears downloadFailed/downloadError
only after a successful write, through the normal metadata save/cache path.
- Force bypasses the populated-folder guard; success is false when flagged
images were skipped, so models are no longer falsely marked recovered.
- Add settings, tests, and a design/measurement note.
CivitAI Workflow archives can bundle only a PNG/WebP screenshot whose
metadata carries the graph, with no workflow JSON member. Zip candidate
discovery now falls back to image members (bounded reads, full-size
images first) and parses their metadata in memory, while the archive
keeps its original bytes and SHA256 identity for CivitAI matching.
The LoRAs page search ran difflib.SequenceMatcher over every token of every
cached entry. Matching entries exit at the substring check, so the cost was
paid entirely by the entries that do not match - exactly the state the list is
in while a word is still being typed. At 100k entries one non-matching query
took 1.7-3.2s, and the handler runs that work on the event loop.
Gate the matcher with two exact necessary conditions instead of a heuristic:
2*min(len_a,len_b)/(len_a+len_b) is an upper bound on the ratio, and so is
twice the character-multiset intersection (a matching block can only use
characters present in both strings). Pairs that fail either bound provably
cannot reach the threshold, so every result is unchanged.
A non-matching query now takes 0.28-0.36s at 100k entries (6-9x faster) and
54-72ms at 20k (was 0.32-0.62s). Add a Hypothesis equivalence test against the
previous implementation, a regression test proving obvious misses never reach
the matcher, and a library-scale benchmark.
Fixes#1179
Newly created workflow entries used to be hardcoded to "Unknown". The
base model is now resolved at creation time only (Create LM-managed
workflow and first-time scans); write-back never re-resolves it.
Resolution order:
1. generation-time hint: on create with a prompt context, the primary
checkpoint from the metadata collector (primary-sampler tracing plus
lm_marker_role marks) outranks static analysis when it resolves in
the local checkpoint library
2. static votes: majority base model of the referenced checkpoints in
the local library, falling back to the referenced LoRAs
3. CivitAI URN fallback for one-off user actions (never scans)
Workflow analysis now extracts structured loader model references from
both API-format and UI-format graphs, including loaders whose widget
values are serialized as arrays.
- Replace the multi-root folder_paths.workflows workflow library with a
scalar workflow_path setting (Settings > Library); remove the
extra_folder_paths.workflows path and the standalone Model Paths editor.
- Migrate pre-release settings.json files that still keep
folder_paths.workflows / extra_folder_paths.workflows on startup, so an
upgrade keeps the configured folder instead of silently losing it.
- Add "Create LM-managed workflow" to the ComfyUI Graph menu and the File
menu/command palette: it writes the serialized graph into the configured
folder, stamps extra.lm_source and registers the entry in the scanner
cache without forcing a rescan.
- Attach the decode image of the client's last executed prompt as the
workflow preview. The prompt id is sent with the create request and the
backend only honors it while it is still the registry's current prompt,
so a tab never borrows another tab's output; otherwise the card keeps the
placeholder preview.
- Detect library entries deleted/renamed while a graph still carries the
marker: a link-status endpoint clears the stale extra.lm_source (on graph
load, menu open and before create/update), and the write-back route
returns 404 instead of recreating the document.
- Collapse the workflows.noPaths standalone/ComfyUI split into a single
description/hint; the folder is edited in Settings > Library in both
modes, so the manual settings.json guidance and the standalone
open-settings-folder fallback are gone.
- Add the preview tensor conversion as a shared utility (reused by the
recipe widget save path) and cover create, preview gating, link status,
migration and menu injection with tests.
Anima 2.9B/3.8B insert blocks between the original 28, so old LoRA keys land on the wrong layers and ComfyUI emits no missing-key warning. Add a model-family adapter that rewrites block indices in memory before the generic LoRA loader runs.
- 28->40, 40->52 and composed 28->52 mappings, with anchored kohya/generic key patterns
- Anima detection from image_model/llm_adapter and block count from blocks/num_blocks
- skip text encoder, llm_adapter and semantic connector keys; collision and newer-LoRA errors
- ComfyUI setting (default on) synced to the LM settings API; no base-model-specific node toggle
- docs, unit and loader integration tests
Reuses the manifest data from PR #1104; supersedes its node-level toggle design.
Library entries are the original artifact bytes (SHA256 = CivitAI identity);
container artifacts (.zip, .png, .webp) get their extracted workflow document
materialized as a *.workflow.json sidecar that follows rename/move/delete.
- workflow_extraction: json direct-read, zip candidate pick, png/webp metadata
- scanner whitelist extended; non-workflow files rejected; adjust_metadata
fills analysis fields for downloaded entries; image artifacts self-preview
- download_manager: workflow branch end-to-end, no zip unpacking, file
selection prefers primary then json/zip/image
- lifecycle: delete/rename/pending-delete carry the derived document
- send-to-comfy reads the derived document for container artifacts
- fix: force_refresh with no cache falls back to full initialization
(first scan before deferred background init no longer crashes)
Add a Workflows page that manages ComfyUI workflow JSON files as a
first-class library alongside loras/checkpoints/embeddings:
- opt-in via the new 'Workflow Management' settings toggle (default off)
- dedicated workflow folder roots with sidecar/preview support, separate
from ComfyUI's own workflow storage; 'Send to ComfyUI' loads a workflow
straight into the graph
- workflow scanner parses the graph document (UI/API format sniffing,
node classes, model/civitai refs) and persists the analysis in the
SQLite model cache
- settings UI: workflow folder paths editor in the Library tab and the
standalone Model Paths page
Fixes from sandbox E2E verification:
- include the workflow scanner in startup initialization so the page no
longer hangs on 'initializing' when enabled without folders
- add the missing workflows entry to the frontend page-state map (page
init crashed and the sidebar fell back to the loras tree without it)
- project workflow analysis fields through the scanner cache entry and
persist them in the model cache (new columns with auto-migration)
- keep symlinked paths intact in the directory picker (abspath instead
of resolve), so stored library paths match what the user sees on disk
Also rebuilds the vue-widgets bundle to sync it with web/comfyui/utils.js.
Downloads, imports, reconciles, metadata syncs and bulk updates all add
freshly allocated objects to caches that were frozen at startup, so
ComfyUI's post-prompt gc.collect() grows again within a running session
(issue #1161, follow-up to #1147).
- add_model_to_cache now stores the slim cache representation: license
flags are derived from the full source payload first, then the civitai
payload is slimmed to the scan-path shape. Fresh downloads previously
also ended up with default license flags.
- New schedule_refreeze(): debounced re-freeze with an idle guard that
reschedules while the ComfyUI prompt queue is busy, so the freeze never
lands mid-prompt. Wired into downloads/imports, reconcile, single-model
updates, deferred bulk updates and metadata sync.
- freeze_long_lived_objects() logs collect duration and the tracked-object
count at DEBUG level to tell 'more objects' apart from 'slower objects'.
A file whose directory entry reports zero bytes hashes to the SHA256 of an
empty byte string. That value is shared by every unread or still-being-written
file: the library collapses them into one bogus duplicate group and every
member resolves to the same polluted remote entry (#1162).
- _calculate_sha256_sync returns "" when the read yields no bytes, so no
caller can persist the placeholder as a real hash
- create_default_metadata refuses to create a sidecar for an empty file
- ModelScanner._process_model_file skips zero-byte files before loading or
creating metadata, leaving them unmanaged until they have content
- a file whose metadata cannot be created no longer aborts the scan via assert
Video preview downloads keep streaming to disk, but the saved file now
gets its extension sniffed from the actual bytes and renamed on
mismatch, so a future Civitai transcode format change cannot leave
e.g. WebM bytes in an .mp4 file. (#1155)
Move the Windows-safe chunked FileResponse out of PreviewHandler into a
reusable py/utils/file_responses.py module, with platform_file_response_class()
selecting the transport per platform. Static routes (add_static) can adopt it
later if they also need to bypass native sendfile on Windows.
Add a platform-independent test for ChunkedFileResponse so Linux CI exercises
the chunked transport; the merged Windows-only test never runs there.
CivitAI stores non-ASCII file names percent-encoded; the mini endpoint's
raw fileName (preferred since #1100) returned that encoded string, so it
landed on disk literally as "%E7%8A%AC..." and the path-budget
truncation could cut mid-escape. Decode to valid non-ASCII UTF-8 names
before the on-disk name is fixed; literal % (stored as %25) and invalid
escapes pass through unchanged.
A full rebuild or library switch replaces the frozen cache with freshly
allocated, GC-tracked objects, so ComfyUI's post-prompt gc.collect() would
slow down again until the next restart. Schedule a debounced re-freeze
(10 s settle delay) after ModelScanner full rebuilds and after model/recipe
library-change re-initialization, so repeated rebuilds and multi-scanner
library switches freeze once, after follow-up work settles.
Distinguish same-version quantized models (e.g. GGUF Q4_K_M vs Q8_0) on
cards, which previously showed only the identical CivitAI version name.
Detection is content-based and stored as a three-state sidecar field
(absent = unchecked, "" = checked-unavailable, value = canonical name),
mirroring the autov3 pipeline:
- GGUF: minimal pure-struct header parser reads general.file_type
- safetensors: dominant tensor dtype F8_* -> FP8
- CivitAI files[].metadata.fp == "fp8" fills in as a fallback only
- A one-time backfill service covers existing libraries; each file
costs only a few-hundred-byte header read
The footer renders the badge after the version name (suppressed in
grouped mode), gated by a new show_quantization_on_card setting that
follows the show_version_on_card toggle pattern.
Two platform assumptions broke the Windows informational job: the
alongside-sidecar fixture went through abspath, which prepends the drive
letter on Windows and skewed the asserted lengths by two; and the
download-rename trim test patched a fixed path limit that left no room
for a viable stem under the runner's long temp dir. The fixture now uses
plain dirname, and the limit is patched relative to the actual tmp_path.
ComfyUI runs a full gc.collect() after every prompt, and its cost scales
with the number of GC-tracked objects. With a large library the hydrated
model/recipe caches account for ~1M long-lived objects, so each post-prompt
collection held the GIL for seconds and froze the whole server.
After all scanner init tasks finish, run gc.collect() + gc.freeze() once to
move the long-lived objects into the permanent generation, which the
collector no longer scans. Matches the workaround the reporter verified
manually.
The per-component caps (150/100/50) do not add up to a guarantee: a deep
model root plus template folders plus a max-length stem still overflows
the Windows MAX_PATH limit, and raw download filenames carried no cap at
all. fit_stem_to_path_budget() now measures the actual destination —
the model path and the sidecar location resolved through sidecar_paths
(the centralized mirror tree is typically deeper) — and trims the stem
to fit, reserving room for companion suffixes (.metadata.json.bak,
previews, .aria2).
Applied at all four name/path decision points: bulk filename-template
rename, post-download template rename, the initial download save path
(file_info rewritten to the on-disk name, matching the #1100 raw-name
precedent), and an auto-organize pre-check that skips with an
actionable message instead of failing opaquely on Windows.
Second pass over the informational Windows job (91 failures -> 11 -> 0):
* `_identity`/`_fold` now wrap the assertions my first sweep skipped: the
parenthesised and multi-line ones, and the ones whose token list was
incomplete (`base_models_roots`, `embeddings_roots`).
* The reconciler's `current_name` and `active_roots` carry the drive-prefixed
label on Windows ("C: loras"), so the last direct comparison goes through
`_root_label_names` too.
* `open_settings_location` reveals the settings *file* with
`explorer /select,` on Windows instead of opening the containing folder, so
that test asserts the platform's call instead of the POSIX one.
* Three more expectations are written in the business form the code returns
(metadata payload, extracted download paths, hydrated preview URL).
The new informational Windows job surfaced 91 failures: four real bugs (fixed in
the previous two commits) and a long tail of POSIX assumptions in the tests
themselves. None of them assert behaviour that differs on Windows, so they are
fixed rather than skipped wherever the behaviour is the same:
* Path spelling: expectations built with `str(tmp_path)` are compared through an
identity helper (`normalize_root_identity`) or the platform's separator, since
the code hands out forward-slashed business paths while `str()` is native.
* Root labels: `_root_display_labels` prefixes the drive on Windows ("C: loras"),
so label assertions compare the final path component or derive the expected
label from the same helper.
* `~` expansion reads USERPROFILE (ntpath), not HOME; timer granularity rounds
0.1s down; a case-only rename counts as "already named" via `os.path.normcase`.
* Genuinely POSIX-only strategies are skipped with a reason instead of failing:
the headless clipboard fallback, dbus file-manager selection, xdg-open, and
the case-fold index test that simulates a case-sensitive filesystem.
ComfyUI hands its model folders over in the host's own spelling --
`utils/extra_config.py` runs every configured path through `os.path.normpath()`,
which is all backslashes on Windows, and `main.py` registers `<output>/loras`
unconditionally for its save nodes. `Config` kept that spelling verbatim while
`_prepare_*_paths()` normalized the live root lists to forward slashes, and the
scanner compared the two as raw strings. On Windows every configured root
therefore looked like a root that was not in the live list: the Refresh menu
listed each directory twice (the twin showing "0 models"), the completion toast
claimed reachable roots were "currently inaccessible" and kept their whole cache
count as `kept_unreachable`, and the settings sync persisted host-registered
directories that were never created.
The duplication was not only cosmetic. `_UnreachablePaths.add()` normalizes the
prefix it is given, so the configured twin of a live root registered exactly the
live root's prefix as unreadable: in the prune phase every cached-but-missing
entry under it matched and was "kept", so models deleted from disk were never
removed from the cache again.
* `py/utils/root_paths.py` holds the single `normalize_root_identity()`
(separator style, redundant/trailing separators, and case where the filesystem
folds it: whole paths on Windows, drive letters everywhere). `config.py` and
`settings_manager.py` each carried a private copy and `model_scanner.py`
compared raw strings.
* `ModelScanner` compares root identities in `get_configured_model_roots()`,
`describe_model_roots()` (dedupe + `available`), the reconcile's
`configured_only` list and the `ReconcileScope.roots` membership test, so one
directory is one root whatever the host called it. `_normalized_root_prefix()`
is built on the same identity, and an empty prefix no longer matches every
path.
* A configured root that never made it into the live list is reported only while
it still holds cached entries -- in the menu and in `skipped_roots` /
`unavailable_paths` alike. A directory that neither exists nor was ever scanned
has nothing to keep, so `<output>/loras` stops being an unavailable root.
* The startup sync drops saved roots that neither exist nor hold cached entries
(`PersistentModelCache.roots_with_cached_entries()`, one scan per model type,
separator-agnostic and wildcard-escaped). A root named by the library's
`default_*_root` is kept even when empty: dropping it would let the next boot
"repair" the default onto another root. When the cache cannot be consulted,
every root is kept.
* `GET /scan?roots=` validates requested roots by identity too, so the browser
extension or a script passing the spelling settings.json holds selects the root
instead of answering 400 "Unknown model root(s)".
Verified: 8 of the new tests fail against the pre-fix source (checked with the
fix stashed); 3798 passed, 7 skipped, 11 deselected. The probe costs 71ms for a
100k-model library with seven missing roots (357ms when probing root by root).
- Return the ResourceNotFoundError detail from get_civitai_versions and
get_model_versions_status instead of a bare "Model not found", and let
the download modal display the server-provided message
- Anchor extract_civitai_image_id / extractCivitaiImageId to
^/images/<id> so API paths and partial numeric IDs are rejected, same
as the model-page parsing fix in #1140
CivitAI's public API deliberately omits prices — paidAccess is trimmed to
{permanent, endsAt} because "pricing belongs to the purchase flow" — but the
public model page embeds the site's own model.getById result, including
paidAccess.terms, in its server-rendered payload. That is read anonymously
(no API key, no internal endpoint, no forged Origin), one request per gated
model, so only the ~2% of models that actually carry a gate pay for it.
- optional capture, off by default: price_tracking_enabled,
price_alert_threshold_buzz (0 = alert on "became free" only) and
price_check_ttl_hours; prices refresh on their own TTL and immediately when a
gate changes, and a failed fetch keeps the stored price instead of blanking it
- versions that stop carrying a gate are marked free (persisted gate_lapsed_at)
and gate transitions are reported as events on the refresh response, so a
version already in the library can announce that it became free
- price_alert_state plus a price_drop edge event; new
GET /api/lm/{type}/updates/price-alerts lists what is under the threshold
- versions tab shows the price (effective, with the list price struck through
and a Blue Buzz note) and a Free Now badge; an update check toasts the
transitions in one message
- the parser and the alerts query are unit-tested against a trimmed page
fixture, and every route definition is now asserted to resolve to a handler
Plan, verification notes and the deviations from it are in
docs/plans/paid-model-price-tracking.md.
CivitAI only returns a non-null paidAccess for an *active* gate: a lapsed gate
stays in the database as a tombstone and is filtered out server-side, so
{"permanent": false, "endsAt": null} — a timed gate whose window end has not
been recorded yet — is still enforced. Verified live: on model 1802980 that
version reports canDownload: false while its lapsed siblings report true.
Both the update service and the download gate dropped that shape, so such
versions read as free and "Hide Early Access Updates" missed them — the class
of bug reported in #1060.
The interpretation now lives in py/utils/paid_access.py and is shared, so the
badge, the update filter and the download warning cannot disagree.
Two gaps kept 'import example images' from working on the Other page:
- import_images/delete_custom_image/set_example_image_nsfw_level only
searched the lora/checkpoint/embedding scanners, so Other-category
models were never found ('Model with hash ... not found in cache').
All three now go through a shared scanner list that includes the
Other scanner.
- Other models (and fresh checkpoints) carry hash_status=pending with
an empty sha256, so the frontend sent an empty model_hash and the
import was rejected with 'Missing model_hash parameter'. The modal
now also sends the model's file path, and the import use case
resolves the hash on demand via the scanner's lazy-hash calculation.
The resolved hash is returned to the UI and persisted on the
showcase element so follow-up operations target the same
hash-keyed folder.
CivitAI serves a re-encoded, metadata-free optimized rendition as the recipe
preview, so the ComfyUI workflow embedded in the original image was dropped:
imported recipes reported has_workflow=false and never offered "Send Workflow
to ComfyUI" even when the source image carried one.
Recover the workflow from the original rendition and carry it to the save step
as data, so the stored preview stays the small optimized image:
- ExifUtils: embed a caller-supplied workflow during optimize_image's single
encode pass, and add embed_workflow() to patch WebP EXIF in place (used by
the verbatim skip_optimize branch and as a safety net).
- RecipePersistenceService.save_recipe: embed metadata["workflow"] before
detecting has_workflow.
- analyze_remote_image: return the workflow recovered from the original
rendition it already downloads for EXIF parsing.
- RecipeManagementHandler: add _fetch_original_media() and workflow helpers;
_do_import_from_url reuses them, and _do_import_remote_recipe fetches the
original only when CivitAI reports a ComfyUI payload (meta.comfy) so
workflow-less images pay no extra bandwidth.
- Batch URL imports and the import modal forward the recovered workflow.
Verified against the reported image: has_workflow flips from false to true and
the recovered workflow matches the original (25 nodes, same graph id).
A model root that moved earlier is mirrored under a pinned name derived
from its old path. If anything resolved against the new sidecar path before
the relocation ran, the destination got a map naming the mirror after the
*current* path. migrate_root's keep-newer transfer then dropped the source
map, so the moved metadata stayed orphaned under the pinned component while
reads followed the new name and rebuilt defaults — losing favorites, notes
and tags a second time.
The identity map is now relocated by relocate_root_map() instead of the
generic transfer: entries recorded under the old sidecar root win for the
roots they describe, destination-only entries are preserved, and the cache
is dropped so the next resolution reloads the merged map. A merge that
cannot be written is reported as a migration error rather than silently
stranding the moved metadata.
Reported by the Codex review on #1131.
Centralized sidecars were addressed by a hash of the model root's absolute
path, so moving or renaming a root produced a new mirror directory. The
scanner then found no sidecar there, rebuilt default metadata, and silently
lost favorites, notes, tags and usage tips for every model under that root,
leaving the old metadata orphaned on disk.
Mirrors are now addressed by a root identity pinned in
<sidecar_root>/.lm-sidecar-roots.json. The identity starts as the existing
deterministic <basename>-<path digest> -- so pre-existing mirrors keep
resolving even if the map is lost, and a relocated sidecar root keeps its
names -- and is re-anchored to the root's new path when it moves, matched by
basename and recorded sample directories. Ambiguous matches are never
guessed: the mirror is left untouched and reported.
Also:
- drop the <library> path segment (single-library direction); a legacy
library prefix is only recognised while adopting an existing mirror, which
also keeps the unreleased centralized layout usable
- surface stranded mirrors in Doctor and in the log instead of silently
rebuilding sidecars
- keep the default alongside mode untouched: the identity map is loaded and
reconciled lazily, only while centralized storage is in use
After migrating to centralized sidecar storage users had no indication
where their files went, and portable-mode installs silently placed the
sidecar root inside the plugin folder where a reinstall or git clean
would delete it.
- Migration now also covers models excluded from the library view and
returns the resolved sidecar root in its result payload
- get_settings exposes the resolved sidecar root, whether it is the
default, and whether it lives inside the installation folder
- New POST /api/lm/sidecars/open-location endpoint opens (or copies)
the sidecar storage folder
- Settings UI always shows the effective storage path with an
open-folder button, and warns when the root is inside the
installation folder (portable-mode hazard)
- Migration confirmation shows the destination; on completion a result
dialog summarizes moved/skipped/conflict counts with the storage
location and an open-folder action
- Ignore /sidecars/ at the repository root so portable-mode sidecars
are never committed
Refs #1045
Codex review on #1124:
- P1: mirror layout root component is now <basename>-<roothash>
(sha256 of the normalized root path), so two roots sharing a basename
no longer map to the same mirror directory and overwrite each other's
sidecars
- P1: changing sidecar_storage_path while centralized no longer strands
assets in the old root — new relocate_root migration direction moves
the whole mirror tree, rewrites preview_url prefixes inside sidecars,
reconciles scanner caches, and prunes the emptied old tree; the
settings UI detects the path change and offers the relocation
- P2: migration enumerates the same preview candidates as
find_preview_file — case-insensitive variants (model.WEBP) and the
legacy .example.0.jpeg suffix — instead of exact lowercase
PREVIEW_EXTENSIONS only
- P2: _rollback_model_staging restores staged files with the
EXDEV-tolerant mover, so a failed undoable-delete staging no longer
strands a cross-filesystem centralized sidecar copy
Tests: same-basename root injectivity, mixed-case/example preview
migration, relocate_root happy path + guards + route 400, frontend
relocation prompt flow. Verified end-to-end in a sandboxed standalone
server: uppercase/legacy previews migrate, root relocation moves the
tree and the list API serves the new locations immediately without a
rescan.
Add an opt-in 'centralized' sidecar storage mode alongside the default
'alongside' layout. In centralized mode, .metadata.json sidecars and
preview assets live under a configurable root (sidecar_storage_path,
default <settings_dir>/sidecars), mirroring the library-relative
directory structure: <root>/<library>/<root_basename>/<rel_dir>/.
Backend:
- settings: sidecar_storage_mode / sidecar_storage_path with validation;
changing either refreshes the preview allowlist
- config: centralized root added to preview-serving allowlist
- lifecycle: delete / move / rename / folder-rename / folder-delete and
undoable-delete staging all operate on the mirror tree in centralized
mode (model files themselves never move); EXDEV-tolerant cross-
filesystem moves
- scanners: pending-hash filesystem scan walks the mirror tree in
centralized mode; preview discovery reads from the sidecar dir;
.civitai.info stays co-located in both modes
- migration: SidecarMigrationUseCase moves sidecars+previews between
layouts both directions (keep-newer conflict resolution, preview_url
rewriting, WebSocket progress), exposed as POST+GET
/api/lm/sidecars/migrate with a mode guard (force=true for the
settings-first flow)
Frontend:
- settings modal: sidecar storage section (mode select + path input with
browse/validation), mode-change confirmation offering immediate
migration (force=true), and a 'Migrate Sidecars Now' action
- i18n keys synced to all locales ([TODO: Translate] placeholders)
Docs: metadata-json-schema.md gains a storage-location section;
AGENTS.md records the sidecar_paths helper convention.
Follow-up to the keyword-dump guard. The reported model's tag list is
["lora, character, ... face", "base model"], so skipping the dump left the
"base model" label to be picked as the folder name. That label describes
Civitai's listing rather than the model's content, which makes it as
meaningless as a folder as the blob was.
Add CIVITAI_META_TAGS and is_civitai_meta_tag(), and skip those labels in
the automatic fallback. An explicit priority entry still matches them, so a
user who does want a "base model" folder can configure one.
The reported model now resolves to "Krea 2/no tags" instead of
"Krea 2/base model".
Also correct a comment that listed ".civitai.info" among the files sitting
next to a model. LoRA Manager only reads that sidecar -- other tools write
it -- and writes ".metadata.json" itself.
CivitAI tags are normally short single-concept labels, but some uploaders
pack their entire keyword list into one tag. The model in #1119 carries
"lora, character, rosie, irish, ... face" as a single 181-character tag.
Priority resolution matches aliases by exact equality, so that tag matched
nothing and resolve_priority_tag_for_model fell back to tags[0] -- the blob.
With the default "{base_model}/{first_tag}" template the model was filed
under "Krea 2/<181-character blob>/", and the full path plus the
".civitai.info" sidecar and the preview images next to it ran into the
Windows MAX_PATH limit.
Tags also bypassed sanitization on the way into a path: both
calculate_relative_path_for_model and DownloadManager._calculate_relative_path
sanitized model_name and version_name but interpolated {first_tag} verbatim,
so a tag containing "/" or ":" silently produced nested or illegal folders.
Two changes:
- The fallback skips tags that cannot serve as a folder name.
is_usable_path_tag rejects comma-separated keyword dumps and tags longer
than MAX_PATH_TAG_LENGTH; the resolver returns "" when nothing usable is
left, which callers already render as "no tags". Whole-tag priority
matching is untouched, so existing priority configurations behave the
same.
- sanitize_folder_name gains an optional max_length, and every tag-derived
segment now goes through it. Tags are capped at MAX_PATH_TAG_LENGTH, model
and version names at MAX_FOLDER_NAME_LENGTH, and rendered filename stems at
MAX_FILENAME_STEM_LENGTH.
For the reported model the folder becomes "Krea 2/base model" instead of the
blob, and the full path drops from 235 to 64 characters.
Existing libraries are not migrated up front: a path is only recomputed on
download, on an auto-organize run or when a filename template is applied, and
values already inside the caps are left byte-identical. Models previously
filed under a keyword-dump folder move on the next auto-organize run.
Add Load Image Metadata (LoraManager) to extract reusable prompts,
model references, LoRA stacks, and sampling settings from images.
Prefer saved A1111-style parameters by default, with optional workflow
and subgraph sampler selection. Resolve local model and LoRA names,
report missing resources, and recover extraction failures with explicit
defaults and readable diagnostics.
Include parser, resource-resolution, and node regression tests, plus
usage documentation.
Add per-model-type filename templates ({model_name}, {version_name},
{base_model}, {author}, {first_tag}, {hash_short}, {original_name}) so
downloaded files get informative names instead of e.g. V1.safetensors.
Empty template keeps the current filename (opt-in, off by default).
- apply template automatically after downloads; rename conflicts keep
the original name and never fail the download
- record original_file_name in metadata on rename for traceability
- bulk apply via GET|POST /api/lm/{prefix}/apply-filename-template with
WebSocket progress, sharing the auto-organize lock
- settings UI lives in the new Organization tab with validation, live
preview, and per-type 'apply to library' actions
Standalone users previously had to hand-edit settings.json to configure
primary folder_paths. Add a standalone-only Model Paths section to the
settings modal:
- Backend exposes standalone_mode, folder_paths (with template placeholder
values filtered out) and a data-driven folder_path_schema derived from
OTHER_MODEL_FOLDER_SUBTYPES via GET /api/lm/settings
- The new section renders multi-path editors per model type from the
schema, with inline enable_other_models / sub-type controls so other
model types are configured without leaving the tab
- Persistent restart-required cues after a save: nav dot, inline notice
and a global banner (unique id per change so dismissals don't mute
future reminders)
- The missing-model-paths startup banner and the Other Models no-paths
empty state now deep-link into the new section instead of pointing at
settings.json
Setting LORA_MANAGER_PORTABLE=1 once wrote use_portable_settings: true
into the plugin's own settings.json, and every later run of every
instance sharing that plugin folder then read and wrote the portable
settings directory. There was no way back except editing the file by
hand, which is exactly the trap a user hit while following the FAQ's
instructions for isolating a second instance (#1114).
LORA_MANAGER_PORTABLE=0 is now the explicit exit:
- _should_use_portable_settings honours "0" as a forced off, so the
resolved settings directory no longer depends on the persisted flag.
- SettingsManager clears the persisted flag in that case, so later runs
without the variable stay on the shared settings directory.
Unset or unrecognised values keep the previous behaviour: the persisted
flag decides, so existing portable installs are unaffected.
LORA_MANAGER_SETTINGS_DIR still takes precedence over both.
Installing a second LoRA Manager instance (standalone or a second
ComfyUI install) that shares the settings directory puts two processes
on the same cache databases. Three things made that unsafe.
- The updater preserved cache/ and model_cache/ but not a legacy
recipe_cache/ directory, so a portable install predating the cache/
move lost its recipe database on a git-based update. Add it to
_PRESERVE_DIRS and to .gitignore.
- Cache connections used the sqlite3 default 5s timeout, which a
scanning instance can exceed, turning a concurrent write into
"database is locked". Route every shared cache connection through
connect_cache_db(), which raises the timeout to 30s and sets
busy_timeout + synchronous=NORMAL to match the existing WAL mode.
App-private databases (download queue, update history) are unchanged.
- A full-table cache replace is a read-modify-write that SQLite cannot
make atomic across processes, so two instances could interleave and
one snapshot could overwrite the other. Guard the recipe and model
save_cache paths with a cross-process advisory lock (flock on POSIX,
msvcrt on Windows). Locking is best-effort: if it is unavailable the
call proceeds and the SQLite busy timeout is the fallback.
The lock file is a hidden sibling of the database and is deliberately
never unlinked, so a second process cannot lock a fresh inode.
Example videos added through the "Add examples" flow were stored with a
hardcoded 720x1280 entry. The dimension probe next to it only ran for
images (PIL cannot open .mp4/.webm files), so every video entry stayed
portrait regardless of the source. The showcase viewer then sizes its
container straight from that value (--media-aspect in showcase.css), so
landscape clips were letterboxed inside a 9:16 box. CivitAI-sourced
examples were unaffected because their dimensions come from the API.
PIL cannot read video containers, so add a dependency-free reader that
parses the container headers instead: moov/trak/tkhd for ISO base media
(with the sample description as a fallback), Segment/Tracks/Pixel* for
WebM/Matroska, and RIFF/WebP for animated examples saved with a video
extension. The sniffed signature decides which reader runs, so a .mp4
that is really WebM still reports the right size; the extension is only
a fallback. Both readers seek past mdat rather than reading it, so a
large file costs the same as a small one.
Imported entries now record the file's real size and keep the previous
placeholder only when the file cannot be parsed.
Existing libraries keep their wrong entries, so backfill them once via
the existing naming migration: bump CURRENT_NAMING_VERSION to 3 and
repair each model's empty-url entries from the files on disk, then sync
the scanner cache. Only entries with no remote url are touched -- those
have no other source, which makes the rewrite lossless -- and entries
already carrying the right size are left byte-identical, so the pass is
idempotent and a no-op for libraries that never imported a video.
Will Miao