Commit Graph
21 Commits
Author SHA1 Message Date
Dariusz L a7f5f70e3c refactor: migrate logging from global functions to per-module loggers
Replace global log_* wrapper functions with create_module_logger(__name__)
instances across the codebase, enabling proper module-level logger
namespacing. Simplifies core/log_system/log_funcs.py by removing legacy
_normalize_module_path and cleaning up _normalize_module_name. Bumps
log_system config to 2.0.0 and introduces PROJECT_ALIASES for cleaner
log module prefix stripping.
2026-06-30 21:24:33 +02:00
Dariusz L 4e576cbd56 feat: add SHA256 hash calculation with cancellation, progress tracking, and remote size probing 2026-06-26 21:51:55 +02:00
Dariusz L a320d4d805 refactor: consolidate duplicated logic into shared utilities
- Add get_comfy_root_path() to core/path_utils.py and use it in __init__.py and core/downloader.py instead of inline logic
- Add select_primary_model_file(), parse_size_to_bytes(), get_version_sort_key(), and check_credential_http() to core/type_utils.py
- Replace duplicated credential-check helpers in civitai.py, huggingface.py with check_credential_http()
- Replace duplicated size parsing and version sorting in civarchive.py, civitai.py with shared utilities
- Replace duplicated primary-file selection in civarchive.py, civitai.py, lora_manager_archive.py with select_primary_model_file()
- Replace inline tempfile-based JSON writes in popular.py with write_json_atomic()
- Remove dead reload_model_list() from model_list.py
- Fix workflow_analyzer.py hint prioritization for diffusion_models vs specific output hints
2026-06-25 18:54:59 +02:00
Dariusz L d979bbcf21 refactor: extract shared utilities and deduplicate metadata path definitions
- Add first_non_empty() and to_int() to core/type_utils.py
- Add get_filename_from_path() and METADATA_DIR to core/path_utils.py
- Replace inline implementations across resolver, downloader, and sources
with calls to the shared utilities
- Move strip_known_model_extension from resolver to matcher
2026-06-25 17:00:27 +02:00
Dariusz L 8655f92515 refactor: centralize category mapping and atomic JSON writing 2026-06-25 16:45:55 +02:00
Dariusz L e4c3b0fb6f refactor: consolidate type and progress utilities
Add core/type_utils.py with as_dict/as_list helpers, and
core/progress.py with a unified report_progress() callback.

Replace inline duplicated implementations of _json_safe_metadata(),
_as_list(), normalize_filename(), and per-source _report_progress()
with calls to the shared modules across downloader, matcher, resolver,
and all source modules (CivitAI, CivArchive, HuggingFace, LoRA Manager).
Extend filename normalization to include dots as separators.
2026-06-25 02:09:47 +02:00
Dariusz L 5445dd4ac0 feat: implement local hash matching via metadata sidecar files
Backend adds SHA256 normalization, metadata sidecar lookup, and hash fields across source modules. Frontend handles local_hash_matches in search responses and missing browser.
2026-06-24 21:59:10 +02:00
Dariusz L 48bc3bf5f1 feat: wire progress callbacks into backend search sources 2026-06-23 03:33:28 +02:00
Dariusz L 2bdc3c2614 refactor: replace ad-hoc log formatting with structured helpers in search flow
Update search logging to use reusable formatters for values, sizes, and field
pairs, and add append-only extra kwargs so downstream logs stay on one line.
Tighten HuggingFace source messages, refresh the author index metadata, and
consolidate LoraManager/CivitAI log text.
2026-06-08 17:21:24 +02:00
Dariusz L 57b8bc7cda feat: add force search (ignore cache) toggle to batch search menu
Adds a checkbox in the resolver footer search menu that bypasses all
search result caches for the duration of a batch search run.

Frontend (queue_methods.js, resolve_download_methods.js,
resolver_dialog.js, search_panel.js, resolver-main.css):
- Introduce checkbox-type footer menu items (type: 'checkbox').
- Add 'Force search (ignore cache)' checkbox to the search menu,
  wired to a new forceBatchSearch flag on ResolverManagerDialog.
- Pass forceSearch through searchMissingBatch and searchOnline to
  the backend API call.
- When the manual search button is clicked and results already exist
  for that model, forceSearch defaults to true.
- mergeSearchResults respects forceRefresh: if a forced search
  returns no results for a source, that source is cleared from the
  cached results.

Backend (__init__.py, core/sources/huggingface.py):
- force_search is read from the incoming API request and, when true,
  clears all active source search caches (local, huggingface,
  civitai, civarchive, lora_manager_archive) before searching.
- force_search is forwarded as force_refresh to
  search_huggingface_for_file, which folds it into the cache key so
  the cache is not hit, and propagates it to _get_author_index.
- Search result timestamps are no longer inherited from prior
  cached signatures when force_search is active; a fresh timestamp
  is written unconditionally.

Updated metadata/huggingface-author-index.json reflects the latest
cached author index after rebuild.
2026-06-06 17:02:25 +02:00
Dariusz L 57018b4e0f feat: add search result timestamps, enhanced cache clearing, and maintenance UI
Add searched_at timestamps to search results so cached entries are accurately tracked and preserved across merges. Expand the clear-search-cache command to also invalidate popular databases, model list, and the new result timestamp store. Frontend gains new cache-clearing methods and a full resolver cache reset for maintenance. The search panel and missing browser now show source result dates and toggle the search button between 'Search' and 'Search Again' when results already exist.
2026-06-06 16:38:30 +02:00
Dariusz L d38e1cbb67 Add inline credential validation endpoints and UI for CivitAI, HuggingFace, and Brave Search 2026-06-05 03:56:15 +02:00
Dariusz L c3fad878af Add persistent HuggingFace author index cache and reorganize options UI 2026-06-05 03:36:58 +02:00
Dariusz L b8515e1441 feat: add configurable HuggingFace search methods with Brave and Comfy-Org fallbacks
- Add new UI toggles for HF search methods: API search, Comfy-Org fallback, Brave fallback
- Integrate Brave Search API as a fallback for finding models on HuggingFace
- Add Comfy-Org author fallback search for authoritative sources
- Refactor HuggingFace search logic into modular helper functions
- Improve filename cleaning to handle fp4 and mixed precision formats
- Add brave_search_api_key, hf_use_api_search, hf_use_comfy_org_fallback, hf_use_brave_fallback settings
2026-05-10 04:27:33 +02:00
Dariusz L 233ed15222 Add CivitAI session token & clear search cache
Add support for an optional CivitAI session token and cache-management for search results. UI: new input for __Secure-civitai-token saved to localStorage (modelLinker.civitaiSessionToken) and cleared via the options dialog; saving/clearing tokens triggers a backend cache clear and updates notifications. Backend: add clear_search_cache() helpers in core.sources.huggingface and core.sources.civitai, register /model_linker/clear-search-cache route to flush both caches, and wire the civitai session token into search flows. CivitAI changes include sending the session token as a Cookie header for tRPC searches, incorporating session and model_type into the search cache key, and passing session_token through search_civitai_for_file and related calls. Purpose: ensure authenticated/session-aware search results (including NSFW/visibility differences) and avoid stale cached search results after token/settings changes.
2026-04-26 02:34:26 +02:00
Dariusz L 1fc404ebfd Add detailed logging to model linker and sources
Add extensive observability/logging across ModelLinker and source search modules. __init__.py: introduce summarize_result and log_search_result helpers, replace ad-hoc logger calls with log_info/log_exception, and add start/result/summary logs for local, popular, model_list, HuggingFace and CivitAI searches (including extra metadata like confidence, files_count, results_count). core/sources/civitai.py: log cache hits, API call start, response counts, model/version/file checks, no-result info and improved exception logging. core/sources/huggingface.py: log cache hits, API call start, response counts, per-repo debug info, repo tree sizes, no-result info and improved exception logging. Changes are focused on diagnostics and do not alter core search logic.
2026-04-25 23:43:17 +02:00
Dariusz L 236ca180e9 Use log_system helpers for module logging
Replace direct uses of the logging module and module-level loggers with the centralized log_system.log_funcs (log_debug/log_info/log_warn/log_error/log_exception) across multiple modules. Updated imports and removed unused logging.getLogger() instances in downloader, scanner, sources (civitai, huggingface, model_list, popular), workflow_analyzer, workflow_updater, and __init__.py. Minor message/text tweaks and import formatting applied; behavior unchanged aside from routing logs through the shared logging helpers.
2026-04-22 00:08:51 +02:00
Dariusz L e8b49dd184 Add AzLogs logging system and integrate
Introduce a new logging subsystem under core/log_system (logger.py, config.py, log_funcs.py) implementing AzLogs (LogLevel enum, colored console output, optional file rotation) and a lightweight wrapper (log_debug/info/warn/error/exception) with safe fallback when AzLogs is unavailable. Integrate these log functions across many core modules (downloader, linker, matcher, scanner, workflow_analyzer, workflow_updater, sources and related files) and export logger and helper log functions from core.__init__. Replace several direct logging calls in linker with the new log_* helpers and update core.__init__ exports. Also include small code style and robustness tweaks (path/filename normalization, JSON/file handling, minor refactors and formatting fixes) to improve matching and model-list/popular lookups.
2026-04-21 23:51:46 +02:00
Dariusz L 0837964874 Add gguf support and nested widget handling
Add support for the gguf model format across the codebase and improve handling of nested/dict-type widget model references. Changes include:

- Add ".gguf" to model extension lists in core/linker.py, core/scanner.py, core/sources/huggingface.py, and core/workflow_analyzer.py so gguf files are detected and suggested.
- Update HuggingFace search/repo listing to consider gguf files and improve some request/response handling and logging.
- Improve scanner implementation: formatting/PEP8 cleanup, robust extension checks, normalized MODEL_EXTENSIONS set, safer path handling, and clearer logging.
- Add NESTED_MODEL_KEYS and logic in workflow_analyzer to detect model filenames stored inside dict-type widget values (e.g. rgthree Power Lora Loader) and record a nested_key for resolution.
- Propagate nested_key through apply_resolution mapping and update update_model_path to write resolved paths back into nested dict keys when present (instead of only replacing the top-level widget value).
- Update web UI (web/linker.js) regexes to strip/display gguf extensions in model name formatting.

These changes allow gguf models to be discovered, resolved and updated in workflows, and enable support for nodes that store model references inside nested dict fields.
2026-04-19 16:43:25 +02:00
unknown 91acff7c78 Enhance download system with speed display, workflow URLs, and UX improvements 2026-01-18 15:59:27 +01:00
unknown 2843373c58 Add download integration with auto-search on startup 2026-01-18 05:44:31 +01:00