feat(backup): models' scan results, and Civitai covers on request
A backup now keeps each model's scan file (.info, small; on by default) so names, trigger words and base models survive Civitai removing a model, and on request Civitai's covers (stored once as the civitai_bak copy; they can be hundreds of MB). Putting back writes a scan file only where the model has none (replaced only with "replace") and Civitai's cover only where it has no cover, with its civitai_bak copy so it stays Civitai's and a later user cover still wins. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DemonGatanjieu
co-authored by
Claude Opus 5.5
Claude Opus 5.5
parent
1efa20fd91
commit
d77655fb76
@@ -5,6 +5,7 @@
|
||||
### 💾 Backup and restore (备份与恢复)
|
||||
- **Settings → Backup → Export** packs your Workflow Recipes, combos, materials and saved prompts, parameter sets, the plugin settings and the names, notes and covers you gave your models into one .zip, optionally with ComfyUI's own saved workflows. Keep it on a cloud drive or a USB stick. The Civitai API key is never included.
|
||||
- **Import** first lists what is in the backup next to what this computer has (new, the same, different), and the models it finds again by file hash, even after they were renamed or moved. By default it only adds what is missing; "take the backup's" replaces differing files and puts this computer's in the Recycle Bin. A model's name and notes are filled in where empty, and your own cover takes the place of the Civitai one. Nothing is deleted.
|
||||
- Models' scan results (.info: names, trigger words, base models from Civitai) are in the backup too, so they survive Civitai removing a model; Civitai's covers can be added (they can be hundreds of MB). Both are put back only where this computer has none.
|
||||
- **Download what the new computer lacks**: the backup also keeps a model list (each model's place, size, fingerprint and link). Importing says how many of those models this computer lacks; **Find downloads…** looks them up like Model Check (Civitai by hash, the saved link, Hugging Face), shows sizes and sources, and downloads the ticked ones one by one into the folder and file name they had, each checked against its fingerprint and scanned after.
|
||||
|
||||
### 🧪 Tests in the repository (测试公开)
|
||||
|
||||
+54
-11
@@ -4,7 +4,8 @@ drive, a USB stick), and putting it back on this or another computer.
|
||||
A backup holds the plugin's stores (Workflow Recipes, combos, materials and saved prompts,
|
||||
parameter sets), optionally ComfyUI's own saved workflows, the plugin settings without the
|
||||
Civitai key, what the user set on models: <model>.anomalous.json (name, notes, link) and
|
||||
the user's cover (any cover that is not a copy of Civitai's <model>.civitai_bak.*), and the
|
||||
the user's cover (any cover that is not a copy of Civitai's <model>.civitai_bak.*), what a
|
||||
scan found (<model>.info; Civitai's cover only when asked, it can be hundreds of MB), and the
|
||||
model list (each model's folder, path, size, SHA-256 and link) so the models a new computer
|
||||
lacks can be downloaded again. Models are found again by SHA-256, else by models folder, path
|
||||
and size, so they match after a move.
|
||||
@@ -42,14 +43,15 @@ from .trash import TrashUnavailable, move_to_trash, trash_failure
|
||||
from . import version_manager
|
||||
|
||||
try:
|
||||
from ..model_identity import USER_INFO_SUFFIX, read_json
|
||||
from ..model_identity import USER_INFO_SUFFIX, read_json, scan_info_path
|
||||
except ImportError:
|
||||
from model_identity import USER_INFO_SUFFIX, read_json
|
||||
from model_identity import USER_INFO_SUFFIX, read_json, scan_info_path
|
||||
|
||||
FORMAT = 1
|
||||
APP = "Anomalous Model Browser"
|
||||
ACTIVE_COVER_SUFFIXES = PREVIEW_SUFFIXES + MEDIA_EXTENSIONS # as scraper.py: <model>.preview.png or <model>.png
|
||||
SECRET_KEYS = {"CIVITAI_API_KEY"} # never leaves this computer
|
||||
SCAN_INFO_SUFFIXES = (".info", ".civitai.info")
|
||||
SKIP_FILES = {".legacy_imported.json"} # this computer's own migration marker
|
||||
MAX_BACKUP_BYTES = 4 * 1024 ** 3 # unpacked size of a backup that is still read
|
||||
MAX_MEMBERS = 200_000
|
||||
@@ -170,7 +172,7 @@ def _version():
|
||||
|
||||
# ---------- export ----------
|
||||
|
||||
def build_backup(include_models=True, include_comfy=True, include_library=True):
|
||||
def build_backup(include_models=True, include_comfy=True, include_library=True, include_scan=True, include_civitai_covers=False):
|
||||
"""Writes the backup .zip into the temp folder: (path, manifest)."""
|
||||
stamp = time.strftime("%Y%m%d-%H%M%S")
|
||||
path = os.path.join(work_dir(), f"AMB-backup-{stamp}.zip")
|
||||
@@ -203,16 +205,27 @@ def build_backup(include_models=True, include_comfy=True, include_library=True):
|
||||
base = os.path.splitext(model_path)[0]
|
||||
user_file = base + USER_INFO_SUFFIX
|
||||
user_data = read_json(user_file) if os.path.isfile(user_file) else None
|
||||
covers = cover_owners(base)[1]
|
||||
if not user_data and not covers:
|
||||
civitai, covers = cover_owners(base)
|
||||
scan_info = scan_info_path(base) if include_scan else None
|
||||
backups = [base + suffix for suffix in CIVITAI_BACKUP_SUFFIXES if os.path.isfile(base + suffix)]
|
||||
civitai_cover = civitai[0] if include_civitai_covers and civitai and backups else None
|
||||
if not user_data and not covers and not scan_info and not civitai_cover:
|
||||
continue
|
||||
entry = {"id": len(manifest["models"]), "type": folder_type, "rel": rel,
|
||||
"size": os.path.getsize(model_path), "sha256": _model_hash(model_path), "user": bool(user_data), "cover": ""}
|
||||
folder = f"models/{entry['id']}/"
|
||||
if user_data:
|
||||
archive.writestr(f"models/{entry['id']}/user.json", json.dumps(user_data, ensure_ascii=False))
|
||||
archive.writestr(folder + "user.json", json.dumps(user_data, ensure_ascii=False))
|
||||
if covers:
|
||||
entry["cover"] = covers[0][len(base):]
|
||||
archive.write(covers[0], f"models/{entry['id']}/cover{entry['cover']}")
|
||||
archive.write(covers[0], f"{folder}cover{entry['cover']}")
|
||||
if scan_info:
|
||||
entry["info"] = scan_info[len(base):]
|
||||
archive.write(scan_info, f"{folder}scan{entry['info']}")
|
||||
if civitai_cover:
|
||||
# One copy: it is both the cover and Civitai's saved image (<model>.civitai_bak.*).
|
||||
entry["civitai_cover"], entry["civitai_bak"] = civitai_cover[len(base):], backups[0][len(base):]
|
||||
archive.write(backups[0], f"{folder}civitai{entry['civitai_bak']}")
|
||||
manifest["models"].append(entry)
|
||||
archive.writestr("manifest.json", json.dumps(manifest, ensure_ascii=False, indent=1))
|
||||
os.replace(temporary, path)
|
||||
@@ -393,9 +406,38 @@ def _restore_stores(archive, wanted, replace, result):
|
||||
_write_member(archive, info, target)
|
||||
|
||||
|
||||
def _restore_scan(archive, entry, base, folder, replace, result):
|
||||
"""What a scan found: the scan file where there is none (or replaced on request), and
|
||||
Civitai's cover where the model has no cover at all."""
|
||||
suffix = str(entry.get("info") or "")
|
||||
if suffix:
|
||||
if suffix not in SCAN_INFO_SUFFIXES:
|
||||
raise BackupError(f"unknown scan file name: {suffix}")
|
||||
info = archive.getinfo(folder + "scan" + suffix)
|
||||
local = scan_info_path(base)
|
||||
if not local or (replace and _sha256_file(local) != _member_sha256(archive, info)):
|
||||
if local:
|
||||
move_to_trash(local)
|
||||
_write_member(archive, info, base + suffix)
|
||||
result["model_infos"] += 1
|
||||
cover, saved = str(entry.get("civitai_cover") or ""), str(entry.get("civitai_bak") or "")
|
||||
if cover and saved:
|
||||
if cover not in ACTIVE_COVER_SUFFIXES or saved not in CIVITAI_BACKUP_SUFFIXES:
|
||||
raise BackupError(f"unknown cover name: {cover}")
|
||||
info = archive.getinfo(folder + "civitai" + saved)
|
||||
civitai, mine = cover_owners(base)
|
||||
has_saved = any(os.path.isfile(base + suffix) for suffix in CIVITAI_BACKUP_SUFFIXES)
|
||||
if not has_saved:
|
||||
_write_member(archive, info, base + saved)
|
||||
if not civitai and not mine and not entry.get("cover"):
|
||||
_write_member(archive, info, base + cover)
|
||||
result["model_civitai_covers"] += 1
|
||||
|
||||
|
||||
def _restore_model(archive, entry, model_path, replace, result):
|
||||
base = os.path.splitext(model_path)[0]
|
||||
folder = f"models/{int(entry['id'])}/"
|
||||
_restore_scan(archive, entry, base, folder, replace, result)
|
||||
if entry.get("user"):
|
||||
backup = json.loads(archive.read(folder + "user.json"))
|
||||
user_file = base + USER_INFO_SUFFIX
|
||||
@@ -445,8 +487,8 @@ def _restore_settings(archive, result):
|
||||
def apply_backup(path, parts=PART_ORDER, models=True, settings=False, replace=False):
|
||||
"""Puts the backup back as chosen; returns the counts for the user and the activity log."""
|
||||
archive, manifest = open_backup(path)
|
||||
result = {"added": 0, "replaced": 0, "skipped": 0, "model_notes": 0, "model_covers": 0,
|
||||
"covers_kept": 0, "models_missing": 0, "settings": False, "failed": []}
|
||||
result = {"added": 0, "replaced": 0, "skipped": 0, "model_notes": 0, "model_covers": 0, "model_infos": 0,
|
||||
"model_civitai_covers": 0, "covers_kept": 0, "models_missing": 0, "settings": False, "failed": []}
|
||||
with archive:
|
||||
_restore_stores(archive, set(parts) & set(PART_ORDER), replace, result)
|
||||
if models and manifest.get("models"):
|
||||
@@ -479,7 +521,8 @@ async def api_export(request):
|
||||
except Exception: # noqa: BLE001 - no body: the defaults
|
||||
body = {}
|
||||
path, manifest = await asyncio.to_thread(build_backup, body.get("models", True) is not False,
|
||||
body.get("comfy_workflows", True) is not False, body.get("library", True) is not False)
|
||||
body.get("comfy_workflows", True) is not False, body.get("library", True) is not False,
|
||||
body.get("scan", True) is not False, body.get("civitai_covers") is True)
|
||||
counts = {**manifest["parts"], "models": len(manifest["models"]), "library": len(manifest.get("library") or [])}
|
||||
await asyncio.to_thread(add_entry, "file", "backup_export", os.path.basename(path), {"backup": counts})
|
||||
return web.json_response({"status": "success", "name": os.path.basename(path), "size": os.path.getsize(path), "counts": counts})
|
||||
|
||||
@@ -184,7 +184,9 @@ downloadFrom), so the placement rules and the put-in after a download are the UI
|
||||
under `stores/<part>/` (recipes, combos, materials, parameters, and `comfy_workflows` =
|
||||
`user/default/workflows` when chosen; the notebooks' legacy-migration marker stays out),
|
||||
`settings/config.json` without `CIVITAI_API_KEY` plus `download_settings.json`, and for each
|
||||
model with a user layer or a user cover `models/<id>/user.json` and `cover<suffix>`, listed
|
||||
model with a user layer or a user cover `models/<id>/user.json` and `cover<suffix>` (and,
|
||||
unless left out, its scan file `scan<.info|.civitai.info>`; Civitai's cover only when asked,
|
||||
stored once as `civitai<.civitai_bak suffix>`), listed
|
||||
in `manifest.json` with type, relative path, size and SHA-256 (from its sidecars; none is
|
||||
computed). The manifest's `library` lists every model the same way plus its download page;
|
||||
inspecting a backup returns the ones this computer lacks, which the frontend looks up and
|
||||
@@ -197,7 +199,9 @@ store how many files are new, the same or different, and how many models it find
|
||||
`POST /anomalous/backup/apply` then writes only the chosen parts: new files are added, the
|
||||
same skipped, different ones replaced only with `replace` (the file here goes to the
|
||||
Recycle Bin first). Models are found by SHA-256, else by models folder, path and equal
|
||||
size. Their user layer is merged field by field (empty fields filled; `replace` takes the
|
||||
size. A scan file is written where the model has none (replaced only with `replace`);
|
||||
Civitai's cover where it has no cover at all, with its `civitai_bak` copy so it stays
|
||||
Civitai's. Their user layer is merged field by field (empty fields filled; `replace` takes the
|
||||
backup's, the old file to the Recycle Bin); the backup's cover replaces a Civitai cover
|
||||
(whose `civitai_bak` copy stays) but not another user cover unless `replace`. Settings
|
||||
are written only when chosen, keeping this computer's Civitai key. Export and import are
|
||||
|
||||
@@ -112,6 +112,25 @@ class BackupTests(unittest.TestCase):
|
||||
settings = json.loads(zipfile.ZipFile(path).read("settings/config.json"))
|
||||
self.assertNotIn("CIVITAI_API_KEY", settings) # the key never leaves the computer
|
||||
|
||||
def test_scan_results_and_civitai_covers_come_back_where_missing(self):
|
||||
civitai = os.path.splitext(os.path.join(self.a.models, "plain.safetensors"))[0]
|
||||
self.a.write(civitai + ".info", '{"id": 7, "name": "Plain"}')
|
||||
path, manifest = self.export(include_civitai_covers=True)
|
||||
plain = [m for m in manifest["models"] if m["rel"] == "plain.safetensors"][0]
|
||||
self.assertEqual((plain["info"], plain["civitai_cover"], plain["civitai_bak"]), (".info", ".preview.png", ".civitai_bak.png"))
|
||||
base = self.b.model("plain.safetensors", sha="cd" * 32)
|
||||
_summary, result = self.put_back(path)
|
||||
self.assertEqual((result["model_infos"], result["model_civitai_covers"]), (1, 1))
|
||||
self.assertEqual(Path(base + ".info").read_text(encoding="utf-8"), '{"id": 7, "name": "Plain"}')
|
||||
self.assertEqual(Path(base + ".preview.png").read_bytes(), b"civitai image")
|
||||
self.assertEqual(Path(base + ".civitai_bak.png").read_bytes(), b"civitai image") # still Civitai's, not the user's
|
||||
self.b.write(base + ".info", '{"id": 7, "name": "Newer scan"}')
|
||||
_summary, kept = self.put_back(path)
|
||||
self.assertEqual(kept["model_infos"], 0)
|
||||
self.assertIn("Newer scan", Path(base + ".info").read_text(encoding="utf-8"))
|
||||
_path, small = self.export() # Civitai's covers only when asked
|
||||
self.assertNotIn("civitai_cover", [m for m in small["models"] if m["rel"] == "plain.safetensors"][0])
|
||||
|
||||
def test_the_model_list_names_what_a_new_computer_lacks(self):
|
||||
path, manifest = self.export()
|
||||
self.assertEqual(sorted(m["rel"] for m in manifest["library"]), ["Style/cute.safetensors", "plain.safetensors"])
|
||||
|
||||
@@ -1097,6 +1097,11 @@ export const i18n = {
|
||||
backupExportModels: '模型的名字、备注和自选封面',
|
||||
backupExportModelsHint: '换了电脑也能按文件哈希对回模型。扫描得来的 C 站信息不用带，重新扫描就有。',
|
||||
backupExportComfyHint: 'ComfyUI 工作流菜单里保存的那些（user/default/workflows）。',
|
||||
backupExportScan: '模型的扫描信息',
|
||||
backupExportScanHint: '从 C 站扫到的名字、触发词、底模（.info，很小）。C 站上的模型被删了也不怕。',
|
||||
backupExportCivitaiCovers: 'C 站封面',
|
||||
backupExportCivitaiCoversHint: '扫描下载的封面图和视频，可能有几百 MB。不带的话，导入后重新扫描也能下回来。',
|
||||
backupDoneScan: '扫描信息：写回 {model_infos} 个，C 站封面 {model_civitai_covers} 张（只补这台电脑上没有的）。',
|
||||
backupExportLibrary: '模型清单',
|
||||
backupExportLibraryHint: '每个模型的位置、大小、指纹和下载链接（很小）。换电脑后导入时，可以把缺的模型直接下回来。',
|
||||
backupLibraryMissing: '这台电脑上缺 {count} 个原来有的模型（共 {size}）。',
|
||||
@@ -3213,6 +3218,11 @@ export const i18n = {
|
||||
backupExportModels: "Your models' names, notes and own covers",
|
||||
backupExportModelsHint: 'Found again by file hash on another computer. What a scan fetched from Civitai is not needed: scan again.',
|
||||
backupExportComfyHint: "The workflows saved in ComfyUI's workflow menu (user/default/workflows).",
|
||||
backupExportScan: "Models' scan results",
|
||||
backupExportScanHint: 'Names, trigger words and base models a scan found on Civitai (.info, small). Safe even if Civitai removes the model.',
|
||||
backupExportCivitaiCovers: 'Civitai covers',
|
||||
backupExportCivitaiCoversHint: 'The cover images and videos scans downloaded; can be hundreds of MB. Without them, scanning again after the import downloads them.',
|
||||
backupDoneScan: 'Scan results: {model_infos} put back, {model_civitai_covers} Civitai covers (only where this computer has none).',
|
||||
backupExportLibrary: 'Model list',
|
||||
backupExportLibraryHint: "Each model's place, size, fingerprint and download link (small). On a new computer, the models it lacks can be downloaded again from the import.",
|
||||
backupLibraryMissing: 'This computer lacks {count} of the models the backup knows ({size}).',
|
||||
|
||||
@@ -85,10 +85,12 @@ function download(name) {
|
||||
export function exportBackup() {
|
||||
const { dialog, footer, close } = openDialog(t('backupExportTitle'));
|
||||
const models = checkbox(t('backupExportModels'), true, t('backupExportModelsHint'));
|
||||
const scan = checkbox(t('backupExportScan'), true, t('backupExportScanHint'));
|
||||
const covers = checkbox(t('backupExportCivitaiCovers'), false, t('backupExportCivitaiCoversHint'));
|
||||
const comfy = checkbox(t('backupPart_comfy_workflows'), true, t('backupExportComfyHint'));
|
||||
const library = checkbox(t('backupExportLibrary'), true, t('backupExportLibraryHint'));
|
||||
const status = el('p', 'anomalous-backup-note');
|
||||
dialog.append(el('p', 'anomalous-backup-text', t('backupExportIntro')), models.line, library.line, comfy.line,
|
||||
dialog.append(el('p', 'anomalous-backup-text', t('backupExportIntro')), models.line, scan.line, covers.line, library.line, comfy.line,
|
||||
el('p', 'anomalous-backup-note', t('backupExportKey')), status);
|
||||
const go = button('anomalous-scan-primary', t('backupExportStart'), async () => {
|
||||
go.disabled = true;
|
||||
@@ -96,7 +98,8 @@ export function exportBackup() {
|
||||
status.textContent = t('backupExporting');
|
||||
try {
|
||||
const data = await postJson('/anomalous/backup/export', {
|
||||
models: models.box.checked, library: library.box.checked, comfy_workflows: comfy.box.checked,
|
||||
models: models.box.checked, scan: scan.box.checked, civitai_covers: covers.box.checked,
|
||||
library: library.box.checked, comfy_workflows: comfy.box.checked,
|
||||
});
|
||||
download(data.name);
|
||||
status.textContent = t('backupExported', { name: data.name, size: formatSize(data.size) });
|
||||
@@ -217,7 +220,7 @@ function showChoices(owner, name, summary, { dialog, footer, close }) {
|
||||
if (again && summary.library.missing.length) dialog.append(again);
|
||||
go.remove();
|
||||
cancel.textContent = t('dialogOk');
|
||||
if (result.model_notes || result.model_covers || result.settings) owner?.loadModels?.();
|
||||
if (result.model_notes || result.model_covers || result.model_infos || result.model_civitai_covers || result.settings) owner?.loadModels?.();
|
||||
} catch (error) {
|
||||
status.textContent = t('backupFailed', { error: error.message });
|
||||
status.classList.add('is-bad');
|
||||
@@ -231,6 +234,7 @@ function showChoices(owner, name, summary, { dialog, footer, close }) {
|
||||
function showResult(dialog, result) {
|
||||
const lines = [t('backupDone', result)];
|
||||
if (result.model_notes || result.model_covers || result.covers_kept || result.models_missing) lines.push(t('backupDoneModels', result));
|
||||
if (result.model_infos || result.model_civitai_covers) lines.push(t('backupDoneScan', result));
|
||||
if (result.settings) lines.push(t('backupDoneSettings'));
|
||||
dialog.replaceChildren(...lines.map(line => el('p', 'anomalous-backup-text', line)));
|
||||
if (result.failed?.length) {
|
||||
|
||||
Reference in New Issue
Block a user