Commit Graph
200 Commits
Author SHA1 Message Date
Dariusz L be12494953 refactor: centralize shared utilities across backend and frontend
Backend:
- Move SHA256 normalization logic from path_utils._normalize_header_sha256
  into core/type_utils.normalize_sha256, adding support for 0x-prefixed
  hashes and unifying the normalization path used by
  extract_safetensors_header_sha256.
- Add tests for the 0x prefix normalization cases.

Frontend:
- Introduce copyTextWithFeedback in web/resolver/utils/html_utils.js
  as a shared clipboard helper with configurable feedback text, CSS class,
  and duration, replacing repeated inline clipboard copy logic in
  globals.js, model_info_methods.js, and the notification utility.
- Centralize showNotification on ResolverManagerDialog and remove the
  duplicate implementation from missingBrowserMethods.
- Replace local const localStorage = safeStorage aliases in
  model_info_methods.js, missing_browser_methods.js, and
  options_methods.js with direct safeStorage usage.
2026-07-17 18:06:22 +02:00
Dariusz L 28d24214aa fix: improve resolver error propagation, sanitization, and consistency 2026-07-16 11:11:48 +02:00
Dariusz L 04c3de9533 refactor: reuse shared path and URL normalization utilities
Replace inline regex path/URL normalization in resolver.py and
missing_browser_methods.js with the existing shared helpers:

- core/resolver.py: normalize_workflow_download_url now uses
  prepare_remote_size_probe_url from type_utils instead of a local
  huggingface-specific regex substitution.
- web/resolver/views/missing_browser_methods.js: path identity
  generation now uses normalizePathIdentity from html_utils.js rather
  than ad hoc replace/trim/lowercase logic.

This keeps normalization behavior consistent across the codebase and
removes duplicated regex patterns.
2026-07-16 10:27:06 +02:00
Dariusz L 8823049f9e refactor: centralize filename extraction, remote URL normalization, and storage abstractions 2026-07-16 10:13:38 +02:00
Dariusz L 32394135e9 refactor: centralize catalog management and unify matcher/progress logic 2026-07-16 09:51:47 +02:00
Dariusz L e4485f3bdf Treat SAM and Ultralytics models as base-model-independent in search panel 2026-07-16 09:10:00 +02:00
Dariusz L 54dae10be5 refactor: centralize background task handling and deduplicate title matching
- Add run_in_background_thread() in __init__.py to unify success/cancel/error
  handling for hash calculation and local metadata build progress
- Extract match_model_by_title_generic() in core/matcher.py and use it from
  both CivitAI and CivArchive providers to remove duplicated version/file
  selection logic
- Enable safetensors header SHA256 extraction by default and report hash source
  through progress updates
- Update options_methods.js to match backend progress behavior
- Add tests/test_title_matching_hashing.py covering title matching and hashing
- Minor metadata updates
2026-07-15 13:27:04 +02:00
Dariusz L 8cf688eba6 refactor: centralize source API requests, polling, and path parsing
- Add request_source_response and request_source_json helpers with 429 retry logic
- Introduce shared split_path_segments and parse_provider_model_url utilities
- Migrate CivitAI, CivArchive, HuggingFace, and ModelList sources to centralized helpers
- Add generic pollBackgroundTask utility for frontend progress polling
- Add tests for request_source_json, split_path_segments, parse_provider_model_url
- Add huggingface-hub to requirements
2026-07-14 12:03:35 +02:00
Dariusz L 280ca2e6de Improve HuggingFace Xet downloads with native progress snapshots, transfer metrics, cancellation, and finalizing UI state 2026-07-14 11:02:57 +02:00
Dariusz L 48db08ae1a Harden network requests, downloads, and path handling
- Add core/network_utils.py with URL validation, public-IP enforcement,
  redirect validation, and safe request_public_url/validate_public_http_url helpers
- Replace direct requests calls in downloader/type_utils/sources with
  request_public_url so redirects are validated before follow-up
- Replace netloc string checks with host_matches_domain for CivitAI,
  CivArchive, and HuggingFace host matching
- Add HuggingFace Xet download backend with progress reporting and
  cancellation support via hf_xet/huggingface_hub
- Harden aria2 installer: reject symlinks/special files in archives,
  restrict custom aria2c paths to the managed install, and update UI copy
- Normalize paths with realpath and enforce configured model roots in
  local hash/metadata routes, plus remove caller-selected metadata_path
  injection in sidecar writing
- Sanitize download/network errors to strip signed query strings before
  logging or showing them to the user
2026-07-14 01:33:19 +02:00
Dariusz L 44153d9d7a Make download cancellation workflow-scoped and prevent stale progress from reviving cancelled downloads
- Introduce workflow-scoped download state keys so active downloads and progress are isolated per workflow tab/route.
- Add a transient 'cancelling' status path and a finalizeCancelledDownloadFrontend helper to avoid flicker and prevent stale progress responses from recreating completed/cancelled UI state.
- Guard active-download lookups by current workflow scope to reduce cross-workflow interference.
2026-07-12 23:28:42 +02:00
Dariusz L 6ed34b6a92 Preserve active download local matches and clean up cancelled downloads
Keep path-specific Local Matches for active downloads during refreshes,
URN search, and missing-model display so late hash-only lookups do not
overwrite them.

Remove cancelled download matches from missing-model UI and cached
analysis when a download is cancelled.
2026-07-12 22:04:53 +02:00
Dariusz L e0997ed9fd Refactor workflow dependency marker sizing to use fixed bounds and compact oversized nodes 2026-07-07 02:40:55 +02:00
Dariusz L 6f9aa9cc55 Improve Model Resolver dependency marker node compatibility and rendering
- Adopt ComfyUI's ComfyExtension/comfy_entrypoint API in __init__.py when
  available, falling back to legacy NODE_CLASS_MAPPINGS/DISPLAY_NAME_MAPPINGS
  on older runtimes so the node loads across ComfyUI versions.
- Hook into beforeRegisterNodeDef, loadedGraphNode, and afterConfigureGraph
  in web/resolver.js so marker nodes are patched during registration, graph
  load, and initial render.
- Teach model_resolver.js to recognise the marker node by every known class
  property shape, patch its type on creation, enforce a minimum size with
  setSize() when present, and dirty the canvas so changes are reflected.
2026-07-06 21:59:30 +02:00
Dariusz L c40cbafc00 Add Model Resolver Opener dependency marker node 2026-07-06 21:36:31 +02:00
Dariusz L f4a049eca9 feat(core, web): embed workflow model hash metadata and prefer hash-based matching 2026-07-06 18:09:31 +02:00
Dariusz L 0cecfab2a0 Switch default log level from DEBUG to ERROR across backend, frontend, and options defaults 2026-07-06 16:05:50 +02:00
Dariusz L 23c0f79444 Add paginated show-more behavior to metadata build history 2026-07-06 14:57:44 +02:00
Dariusz L 8f525044e8 feat(options): make metadata panels collapsible and add summary icons 2026-07-06 14:51:15 +02:00
Dariusz L 5a9e81cc70 refactor(options): use visibility-based transitions for metadata build panels and defer history rendering 2026-07-06 13:58:31 +02:00
Dariusz L f3c0024d46 Replace metadata build history select with custom action filter menu 2026-07-05 22:47:07 +02:00
Dariusz L 39943d5e17 Make metadata history filterable and full-length, and stabilize build progress UI 2026-07-05 22:25:01 +02:00
Dariusz L b9f364e3e4 Add Local Metadata Builder UI and backend API
This change introduces a complete Local Metadata Builder feature in the resolver
options panel. It allows users to scan local model files, create or update
.metadata.json sidecars, fill missing SHA256 hashes from safetensors headers or
file hashing, and review run history with cancel/error handling.

Changes include:
- backend API: /model_resolver/metadata-build/{capabilities,start,progress/{id},cancel}
- backend worker: core/metadata_builder.py with parallel metadata generation
- frontend section: Local Metadata Builder tab in Options
- frontend polling UI: progress, worker breakdown, history table, status/notifications
2026-07-05 21:52:57 +02:00
Dariusz L de6b7738e1 Reorganize options sidebar into Tools and System sections
Move Metadata Sizes into a dedicated Tools group and Maintenance into a System group to improve navigation clarity in the resolver options panel.
2026-07-05 20:59:14 +02:00
Dariusz L 85bbbef06d Add metadata size audit for model sidecar metadata 2026-07-05 18:03:29 +02:00
Dariusz L e386930936 Improve base model matching and subfolder suggestion UX 2026-07-05 05:57:54 +02:00
Dariusz L f691af6387 annotate active downloads in local matches 2026-07-05 05:16:20 +02:00
Dariusz L f75f6878a2 Add unet_gguf category support and bad-folder warnings 2026-07-04 18:24:44 +02:00
Dariusz L 44f539ce49 Refactor frontend apply path for auto-resolve and queue selections 2026-07-04 17:45:17 +02:00
Dariusz L 245bf79800 fix: deduplicate missing models by category+filename and expand per-node resolutions
Previously, identify_missing_models grouped missing entries only by filename,
which could merge unrelated models from different categories that happen to
share a filename. This change deduplicates by (category, filename) and tracks
all_node_refs with a reference_count so the frontend can preserve one
resolution per node reference instead of collapsing duplicates.

Frontend changes in resolve_download_methods.js iterate over all_node_refs,
and missing_browser_methods.js now shows '+ N refs' in the missing-model
label and clarifies the locate tooltip when multiple workflow references
share the same missing model.
2026-07-04 17:26:35 +02:00
Dariusz L d9dacc2016 refactor: centralize Model Resolver button activation with fallback click handling 2026-07-04 04:40:00 +02:00
Dariusz L b57412f5cd Switch Model Resolver shortcut to native ComfyUI commands/keybindings with dynamic tooltip and exclude non-model categories 2026-07-04 04:34:34 +02:00
Dariusz L 4c065818e3 feat: add custom URL resolution to missing browser 2026-07-04 00:45:13 +02:00
Dariusz L 4091dbb88d feat(civitai): add public API search fallback and improve search robustness 2026-07-03 22:00:13 +02:00
Dariusz L 42b13bbb60 feat: improve local hash matching and strengthen download selection metadata
- add POST /model_resolver/local-matches-by-hash backend endpoint
- expand local hash lookup sources to lora_manager_archive, popular, and model_list
- deduplicate backend hash matches by SHA256 value instead of source-hash tuple
- add frontend sync for remote hash matches and hash-label disambiguation
- propagate sha256, file_info, selected_file, and selected_version in model selection/download flows
- render local match status with consistent hash labels across search panel and resolvers
2026-07-03 16:37:10 +02:00
Dariusz L 4c777e5c50 feat: add local hash match identity tracking and highlights; simplify CivitAI import and hash lookup 2026-07-03 01:30:06 +02:00
Dariusz L 9c0ec8b26b Add progress reporting for loaded model and workflow analysis
- Add server-side loaded models progress tracking in __init__.py
- Report workflow node analysis progress from workflow_analyzer.py
- Add frontend polling, rendering, and UI flow for loaded models progress
2026-07-03 00:06:04 +02:00
Dariusz L 56ec5c3083 refactor: consolidate shared helpers and deduplicate HTML escaping/URL collection 2026-07-02 23:35:40 +02:00
Dariusz L 28d054b228 refactor: deduplicate progress tracking, file size extraction, and loaded model rendering
- Replace inline analysis/hash/search progress helpers with direct JobProgressTracker calls
- Add shared extract_file_size() utility and remove duplicated extraction in CivitAI/CivArchive/HuggingFace
- Extract getModelNameAndStrength/getModelToken helpers in loaded models view
- Add raw response mode to fetchJson and simplify backend log export handling
2026-07-02 23:13:37 +02:00
Dariusz L ce29f88c6d refactor: centralize repeated filename/search/URL/HTTP helpers
Replace duplicated filename-to-query logic with a single
\uild_filename_search_queries\ helper in matcher/civarchive/huggingface,
standardize Civitai URL generation in the frontend via \getCivitaiModelUrl\,
unify browser user-agent and response file-size extraction in the downloader
path, and derive workflow node-type category hints from widget categories
rather than maintaining a large static mapping.
2026-07-02 22:56:21 +02:00
Dariusz L e0a765af61 Translate url_utils.js comments to English
Translate JSDoc comments for the getModelCardUrl function from Polish to English for better accessibility and consistency across the codebase.
2026-07-02 04:24:11 +02:00
Dariusz L b7e7b1b295 refactor: centralize getModelCardUrl and HTML/URL utilities in resolver/utils 2026-07-02 04:18:04 +02:00
Dariusz L 86bd6c04fc refactor: centralize model-extension handling and search-cache clearing
- Add clear_all_search_caches() in core/sources/__init__.py and replace 4
  individual cache-clearing call sites in __init__.py with the new helper.
- Add shared parse_civitai_model_path() in core/type_utils.py and use it
  from both CivitAI and CivArchive URL parsers to remove duplicated regex logic.
- Rename parse_huggingface_url() to get_huggingface_repo_and_file_from_url()
  in core/resolver.py for clarity.
- Add stripModelExtension() and hasModelExtension() utilities (with a single
  MODEL_EXTENSIONS_REGEX) in web/resolver/utils/render_format_methods.js
  and replace repeated inline regex patterns across search_panel.js,
  model_info_methods.js, tabs_loaded_methods.js, and queue_methods.js.
2026-07-02 03:57:43 +02:00
Dariusz L 2d2e8b4621 refactor: centralize filename and size utilities across sources
- Introduce get_filename_from_path in path_utils and replace repeated
  os.path.basename calls across core/downloader.py, core/resolver.py,
  core/sources/civarchive.py, core/sources/civitai.py, and
  core/sources/huggingface.py
- Move duplicated inline HTTP response size helpers (_response_file_size,
  _parse_size_header, _parse_content_range_size) into core/type_utils.py
  and update civarchive.py and huggingface.py to import them
- Remove inline normalize_sha256_value from __init__.py and reuse the
  existing normalize_sha256 helper
- Refactor options_methods.js localStorage read/write logic into a
  SETTINGS_MAP-driven loop to reduce repetition
2026-07-02 03:37:44 +02:00
Dariusz L 59e61ba077 refactor: centralize progress reporters, default user-agent, improve error propagation and aria2 fetch usage 2026-07-02 02:54:38 +02:00
Dariusz L 11ec45526c refactor: consolidate duplicated path and size utilities
Extract MODEL_EXTENSIONS and format_size_bytes into core/type_utils, removing duplicates from workflow_analyzer and downloader. Add getFilenameFromPath to render_format_methods and replace inline path-splitting logic across JS resolver actions, search, and views.
2026-07-02 02:27:53 +02:00
Dariusz L 01f0ea1194 feat: improve hash comparison UX with source labels and success notifications
- Add getHashCompareSourceLabel to normalize source display names
- Add getHashCompareResultName and getHashCompareResultUrl helpers
- Collect and dedupe hash compare matches instead of returning boolean
- Show success notification listing matched sources when local model hash matches online results
2026-07-02 02:18:21 +02:00
Dariusz L 2078d870b3 refactor: centralize path normalization and extract common options task handler
- Introduce normalizePathToForward/normalizePathToBackward helpers in render_format_methods.js
- Replace inline path normalization regex across 6 files with shared helpers
- Extract executeOptionsTask helper in options_methods.js to deduplicate repeated async loading/error/success handling and add missing error notifications
2026-07-02 02:12:40 +02:00
Dariusz L d97450f119 refactor: extract splitter drag logic into shared utility
Replace duplicated inline drag handlers in queue_methods.js and
missing_browser_methods.js with a new startSplitterDrag() helper in
web/resolver/utils/splitter_drag.js.

The utility centralizes pointer/mouse event binding, RAF throttling,
and touch-cancellation cleanup, cutting net code across the two
callers while preserving existing behaviour (bounds, edge-open
threshold, collapse preview, and settle/cleanup timing).
2026-07-02 01:57:37 +02:00
Dariusz L dc7ed980b9 refactor: deduplicate progress tracking and unify hash polling logic
- Introduce JobProgressTracker in __init__.py to consolidate the duplicated
  thread-safe progress/cancellation state and cleanup logic previously
  implemented separately for search and hash jobs.
- Extract shared pollHashCalculation method in model_info_methods.js to
  centralize hash polling, abort checking, and error handling across all
  call sites.
- Use getMissingModelKey in search_panel.js to eliminate inline key
  construction duplication.
2026-07-02 01:50:19 +02:00