Commit Graph
118 Commits
Author SHA1 Message Date
Dariusz L 6f9aa9cc55 Improve Model Resolver dependency marker node compatibility and rendering
- Adopt ComfyUI's ComfyExtension/comfy_entrypoint API in __init__.py when
  available, falling back to legacy NODE_CLASS_MAPPINGS/DISPLAY_NAME_MAPPINGS
  on older runtimes so the node loads across ComfyUI versions.
- Hook into beforeRegisterNodeDef, loadedGraphNode, and afterConfigureGraph
  in web/resolver.js so marker nodes are patched during registration, graph
  load, and initial render.
- Teach model_resolver.js to recognise the marker node by every known class
  property shape, patch its type on creation, enforce a minimum size with
  setSize() when present, and dirty the canvas so changes are reflected.
2026-07-06 21:59:30 +02:00
Dariusz L c40cbafc00 Add Model Resolver Opener dependency marker node 2026-07-06 21:36:31 +02:00
Dariusz L f4a049eca9 feat(core, web): embed workflow model hash metadata and prefer hash-based matching 2026-07-06 18:09:31 +02:00
Dariusz L 39943d5e17 Make metadata history filterable and full-length, and stabilize build progress UI 2026-07-05 22:25:01 +02:00
Dariusz L b9f364e3e4 Add Local Metadata Builder UI and backend API
This change introduces a complete Local Metadata Builder feature in the resolver
options panel. It allows users to scan local model files, create or update
.metadata.json sidecars, fill missing SHA256 hashes from safetensors headers or
file hashing, and review run history with cancel/error handling.

Changes include:
- backend API: /model_resolver/metadata-build/{capabilities,start,progress/{id},cancel}
- backend worker: core/metadata_builder.py with parallel metadata generation
- frontend section: Local Metadata Builder tab in Options
- frontend polling UI: progress, worker breakdown, history table, status/notifications
2026-07-05 21:52:57 +02:00
Dariusz L 85bbbef06d Add metadata size audit for model sidecar metadata 2026-07-05 18:03:29 +02:00
Dariusz L 629bc791a6 feat: extract richer local metadata from safetensors headers, improve base-model inference, and preserve local data when remote lookups fail 2026-07-05 04:22:25 +02:00
Dariusz L a079733130 Extract SHA256 from safetensors header and track hash source 2026-07-05 03:00:07 +02:00
Dariusz L 3f54bd4979 Restrict file access to configured model roots and sanitize download filenames 2026-07-04 23:56:50 +02:00
Dariusz L cc8fdda32c chore: add pyproject metadata, CI workflows, and reorganize clone badge into badges/ 2026-07-04 19:59:03 +02:00
Dariusz L b57412f5cd Switch Model Resolver shortcut to native ComfyUI commands/keybindings with dynamic tooltip and exclude non-model categories 2026-07-04 04:34:34 +02:00
Dariusz L 1e22b2ba73 refactor: centralize repeated route logic into shared helpers 2026-07-04 01:23:46 +02:00
Dariusz L 546784e68a refactor: simplify aria2 status route and remove wrapper functions
- Merge GET and POST aria2 status handlers into one route
- Remove redundant calculate_file_sha256 wrapper and use path_utils directly
- Remove redundant _normalize_archive_image, _normalize_filename, and _similarity wrappers
- Replace urllib with requests for remote JSON fetching in model_list
- Update metadata timestamps
2026-07-04 01:03:00 +02:00
Dariusz L 4c065818e3 feat: add custom URL resolution to missing browser 2026-07-04 00:45:13 +02:00
Dariusz L cb666902e5 refactor: centralize hash calculation callbacks and catalog backup logic 2026-07-04 00:18:41 +02:00
Dariusz L 5de1500537 refactor: consolidate inline mappings and precondition checks into shared helpers 2026-07-04 00:04:22 +02:00
Dariusz L 4091dbb88d feat(civitai): add public API search fallback and improve search robustness 2026-07-03 22:00:13 +02:00
Dariusz L 65ff2baa30 Add local hash match cache and invalidate it on rescan/downloads 2026-07-03 17:09:35 +02:00
Dariusz L 42b13bbb60 feat: improve local hash matching and strengthen download selection metadata
- add POST /model_resolver/local-matches-by-hash backend endpoint
- expand local hash lookup sources to lora_manager_archive, popular, and model_list
- deduplicate backend hash matches by SHA256 value instead of source-hash tuple
- add frontend sync for remote hash matches and hash-label disambiguation
- propagate sha256, file_info, selected_file, and selected_version in model selection/download flows
- render local match status with consistent hash labels across search panel and resolvers
2026-07-03 16:37:10 +02:00
Dariusz L 4c777e5c50 feat: add local hash match identity tracking and highlights; simplify CivitAI import and hash lookup 2026-07-03 01:30:06 +02:00
Dariusz L 9c0ec8b26b Add progress reporting for loaded model and workflow analysis
- Add server-side loaded models progress tracking in __init__.py
- Report workflow node analysis progress from workflow_analyzer.py
- Add frontend polling, rendering, and UI flow for loaded models progress
2026-07-03 00:06:04 +02:00
Dariusz L 28d054b228 refactor: deduplicate progress tracking, file size extraction, and loaded model rendering
- Replace inline analysis/hash/search progress helpers with direct JobProgressTracker calls
- Add shared extract_file_size() utility and remove duplicated extraction in CivitAI/CivArchive/HuggingFace
- Extract getModelNameAndStrength/getModelToken helpers in loaded models view
- Add raw response mode to fetchJson and simplify backend log export handling
2026-07-02 23:13:37 +02:00
Dariusz L ce29f88c6d refactor: centralize repeated filename/search/URL/HTTP helpers
Replace duplicated filename-to-query logic with a single
\uild_filename_search_queries\ helper in matcher/civarchive/huggingface,
standardize Civitai URL generation in the frontend via \getCivitaiModelUrl\,
unify browser user-agent and response file-size extraction in the downloader
path, and derive workflow node-type category hints from widget categories
rather than maintaining a large static mapping.
2026-07-02 22:56:21 +02:00
Dariusz L 8ce70c30e6 refactor: consolidate shared utilities into type_utils and path_utils
Replace duplicated SHA256 normalization, metadata sidecar discovery, trained-words
extraction, type coercion, and cache helpers into shared modules. Clarify that the
_civitai.py/_civarchive.py/_huggingface.py/_workflow_analyzer.py refactorings are
behavior-preserving so future contributors keep behavior aligned in one place.
2026-07-02 22:41:56 +02:00
Dariusz L 37383ac046 refactor: consolidate logger imports and replace deprecated log.warn usage 2026-07-02 17:27:54 +02:00
Dariusz L 9951e1fbd5 refactor: centralize shared utilities in core/type_utils and remove redundant wrappers
- Move URL model-file validation logic to type_utils.looks_like_model_file and use it from __init__.py, civarchive
- Extract remote file-size probing into type_utils.fetch_remote_file_size and replace duplicated HEAD+Range logic in civarchive and huggingface
- Extract image normalization into type_utils.normalize_model_image, replacing duplicated CivitAI/CivArchive image mapping in civarchive and civitai
- Drop redundant local wrapper functions (_as_dict, _as_list, _version_sort_key, _select_primary_model_file, _extract_primary_file, _read_json_file, _write_json_file_atomic, _get_sha256_hash, _extract_civitai_image_id, _build_civitai_image_url) in favor of existing shared implementations
- Use path_utils.read_json_safe and write_json_atomic directly in model_list
2026-07-02 04:40:23 +02:00
Dariusz L 86bd6c04fc refactor: centralize model-extension handling and search-cache clearing
- Add clear_all_search_caches() in core/sources/__init__.py and replace 4
  individual cache-clearing call sites in __init__.py with the new helper.
- Add shared parse_civitai_model_path() in core/type_utils.py and use it
  from both CivitAI and CivArchive URL parsers to remove duplicated regex logic.
- Rename parse_huggingface_url() to get_huggingface_repo_and_file_from_url()
  in core/resolver.py for clarity.
- Add stripModelExtension() and hasModelExtension() utilities (with a single
  MODEL_EXTENSIONS_REGEX) in web/resolver/utils/render_format_methods.js
  and replace repeated inline regex patterns across search_panel.js,
  model_info_methods.js, tabs_loaded_methods.js, and queue_methods.js.
2026-07-02 03:57:43 +02:00
Dariusz L 2d2e8b4621 refactor: centralize filename and size utilities across sources
- Introduce get_filename_from_path in path_utils and replace repeated
  os.path.basename calls across core/downloader.py, core/resolver.py,
  core/sources/civarchive.py, core/sources/civitai.py, and
  core/sources/huggingface.py
- Move duplicated inline HTTP response size helpers (_response_file_size,
  _parse_size_header, _parse_content_range_size) into core/type_utils.py
  and update civarchive.py and huggingface.py to import them
- Remove inline normalize_sha256_value from __init__.py and reuse the
  existing normalize_sha256 helper
- Refactor options_methods.js localStorage read/write logic into a
  SETTINGS_MAP-driven loop to reduce repetition
2026-07-02 03:37:44 +02:00
Dariusz L 11ec45526c refactor: consolidate duplicated path and size utilities
Extract MODEL_EXTENSIONS and format_size_bytes into core/type_utils, removing duplicates from workflow_analyzer and downloader. Add getFilenameFromPath to render_format_methods and replace inline path-splitting logic across JS resolver actions, search, and views.
2026-07-02 02:27:53 +02:00
Dariusz L dc7ed980b9 refactor: deduplicate progress tracking and unify hash polling logic
- Introduce JobProgressTracker in __init__.py to consolidate the duplicated
  thread-safe progress/cancellation state and cleanup logic previously
  implemented separately for search and hash jobs.
- Extract shared pollHashCalculation method in model_info_methods.js to
  centralize hash polling, abort checking, and error handling across all
  call sites.
- Use getMissingModelKey in search_panel.js to eliminate inline key
  construction duplication.
2026-07-02 01:50:19 +02:00
Dariusz L 1128234bbd feat: add local model hash comparison functionality
Implement end-to-end hash comparison for local models and local matches.

Backend:
- Add /model_resolver/local-model-hashes API endpoint
- Add get_local_model_hash_metadata() to read sidecar SHA256 metadata

Frontend:
- Add Compare Hashes context menu item and CSS
- Pass missing_key/missing_search_key through resolveDownloadMethods
- Enhance local match rendering with hash status and context data
- Add hash normalization, formatting, and candidate collection helpers
- Wire compareLocalModelHashesWithCurrentFinding action

Also update HuggingFace author index metadata.
2026-07-02 00:27:59 +02:00
Dariusz L 25e20eea8f feat: add frontend and backend log export in options 2026-06-30 23:14:37 +02:00
Dariusz L a7f5f70e3c refactor: migrate logging from global functions to per-module loggers
Replace global log_* wrapper functions with create_module_logger(__name__)
instances across the codebase, enabling proper module-level logger
namespacing. Simplifies core/log_system/log_funcs.py by removing legacy
_normalize_module_path and cleaning up _normalize_module_name. Bumps
log_system config to 2.0.0 and introduces PROJECT_ALIASES for cleaner
log module prefix stripping.
2026-06-30 21:24:33 +02:00
Dariusz L 54f861a02e feat: add aria2 daemon start/stop control and install migration
- Add /aria2/start and /aria2/stop API endpoints for explicit daemon lifecycle control
- Track aria2 daemon state (managed/running/version/active_transfers) in get_aria2_status
- Implement idle auto-stop timer (5 minutes) with aria2_auto_stop_daemon setting
- Add start_aria2_daemon() and stop_aria2_daemon() helpers in core/downloader.py
- Migrate legacy release-* install directories to normalized versioned paths
- Normalize aria2 version strings (strip release-/v prefixes) in installer metadata
- Update aria2 archive extraction to extract-to-temp then copy into canonical dir
- Add frontend daemon controls: status display, stop button, auto-stop toggle
2026-06-30 00:03:08 +02:00
Dariusz L 8b15500ce4 Add automatic aria2 binary installation support
- New core/aria2_installer.py module downloads the latest aria2c release
  from GitHub for Windows, macOS, and Linux with safe path extraction.
- New POST /model_resolver/aria2/install endpoint triggers installation
  and saves the resolved aria2c_path plus download_backend=aria2.
- Options UI adds an Install aria2 button, hides aria2-specific fields
  when another backend is selected, and updates status on completion.
- Includes .gitignore entry for tools/aria2/ so installed binaries are
  not committed.
2026-06-29 21:55:27 +02:00
Dariusz L 59e874c17f feat: improve CivitAI download support with session tokens and proper headers
- Add __init__.py: support civitai.red domain and session token cookies
- Add core/downloader.py: build_download_headers for consistent User-Agent,
  Accept, and CivitAI-specific Referer/Origin headers across download backends
- Resolve CivitAI API redirects before handing URLs to aria2
- Strip sensitive URL parameters from download logs
- Forward civitai_session_token from the frontend to the downloader
2026-06-28 03:42:41 +02:00
Dariusz L aa91d1410c feat: add aria2 download backend with pause/resume support
Integrates aria2c as an alternative download backend alongside the built-in
Python downloader. Adds settings for selecting the backend and configuring
the aria2c executable path, plus a status check button in the options panel.

Backend changes:
- core/downloader.py: aria2 daemon lifecycle, JSON-RPC client, pause/resume
  state machine, and transfer tracking
- core/settings.py: download_backend normalization and aria2c_path setting
- __init__.py: new API routes for pause, resume, and aria2 status checks

Frontend changes:
- options_methods.js: backend selector, aria2c path input, check button,
  and local storage persistence
- resolve_download_methods.js: pause/resume handlers, pending status
  transitions, and progress rendering for aria2 downloads
- queue_methods.js: queue panel pause/resume buttons, backend badge, and
  debounced panel updates
- search_panel.js: actionsHtml support for composite action buttons
2026-06-27 23:39:31 +02:00
Dariusz L b73cdb0f40 feat: improve category resolution and missing models browser
- Expand preferred categories to include clip_vision, ipadapter, sams, ultralytics
- Add clip_gguf alias for text_encoders in both backend and frontend
- Add explicit diffusion_models/text_encoders mappings in CATEGORY_MAP
- Refactor get_model_roots to skip custom_nodes/configs, use template key
  aliases, accept new folder keys from known categories, and normalize
  ultralytics paths (yolo/bbox, yolo/segm, yolo -> parent)
- Pass missing object through resolve_download_methods and search_panel
  so category selection can use widget hints
- Refactor frontend download target resolution:
  - getMissingSupportedDownloadCategories respects widget/type hints
  - getNodeTypeDownloadCategory splits exact-node map and adds
    normalizeNodeCategoryToken / getNodeTypeDownloadCategoryFromTokens
    for GGUF and future node families
  - getSourceResultDownloadCategory and _sourceSupportLevel accept
    missing for unified logic
- Prioritize dynamic widget category hints in workflow analyzer and
  include category_hints/folder_key_hints on all model refs
- Add folder count badges and path normalization in missing browser
  (normalizeMissingFolderRootPath handles category suffixes / bbox/segm)
2026-06-27 19:36:47 +02:00
Dariusz L e53d9f2a0b feat: generalize remote metadata lookup across CivitAI, CivArchive, and HuggingFace
Replace CivitAI-only search with a multi-source metadata resolution route using
SHA256 exact-match validation, dead-link filtering, and live download URL
probing. Unify sidecar metadata saving across all remote sources. Add
force-refresh support, local metadata import with preview detection, and
frontend source-aware UI controls.
2026-06-27 03:25:31 +02:00
Dariusz L 3cd302d4ee feat: implement local-only model info fetching and refactor model info dialog
- Add local_only endpoint mode: returns basic file info (filename, size, path,
  location, sha256) without querying CivitAI or computing file hashes
- Validate provided sha256/hash inputs (must be 64-char hex) and fall back to
  empty string when invalid
- Refactor monolithic info response builder into build_info_response() helper
  used across all result paths (local, CivitAI success, CivitAI fallback,
  not-found), reducing duplication
- Add get_model_info_for_file(local_only=True) early return in CivitAI source
  that builds a minimal local-only result without hash lookup
- Split frontend fetchModelInfoForDialog into:
  - fetchLocalModelInfoForDialog: opens dialog with local data immediately,
    marks civitai_checked=false
  - fetchCivitaiModelInfoForDialog: async CivitAI lookup triggered by user
- Add 'Fetch with CivitAI' button in info dialog when local file exists but
  CivitAI has not been checked yet
- Infer model_type from category in both backend and frontend to reduce
  reliance on upstream data shape
- Fix sidecar metadata condition (was skipping save when path already existed)
- Add cursor/disabled styles for info dialog CivitAI link button
2026-06-26 22:16:44 +02:00
Dariusz L 4e576cbd56 feat: add SHA256 hash calculation with cancellation, progress tracking, and remote size probing 2026-06-26 21:51:55 +02:00
Dariusz L c7d6b27fb0 feat: add search cancellation API and harden CivArchive source resolution
Add a /model_resolver/search-cancel/{progress_id} endpoint with a
SearchCancelled exception so in-flight source searches abort cleanly on
batch cancel. Track sourceProgressIds alongside sourceControllers so the
frontend can cancel backend progress polls. Rework CivArchive extraction
with robust __NEXT_DATA__ parsing, hash-page file resolution, model-link
scraping, confidence-based candidate filtering, and expanded type maps.
Add conftest.py for pytest path setup, update .gitignore for scratch/,
and refresh huggingface-author-index.json.
2026-06-26 00:55:38 +02:00
Dariusz L a320d4d805 refactor: consolidate duplicated logic into shared utilities
- Add get_comfy_root_path() to core/path_utils.py and use it in __init__.py and core/downloader.py instead of inline logic
- Add select_primary_model_file(), parse_size_to_bytes(), get_version_sort_key(), and check_credential_http() to core/type_utils.py
- Replace duplicated credential-check helpers in civitai.py, huggingface.py with check_credential_http()
- Replace duplicated size parsing and version sorting in civarchive.py, civitai.py with shared utilities
- Replace duplicated primary-file selection in civarchive.py, civitai.py, lora_manager_archive.py with select_primary_model_file()
- Replace inline tempfile-based JSON writes in popular.py with write_json_atomic()
- Remove dead reload_model_list() from model_list.py
- Fix workflow_analyzer.py hint prioritization for diffusion_models vs specific output hints
2026-06-25 18:54:59 +02:00
Dariusz L 6850ef1267 refactor: centralize type coercion and file I/O helpers to eliminate duplication 2026-06-25 17:33:59 +02:00
Dariusz L d979bbcf21 refactor: extract shared utilities and deduplicate metadata path definitions
- Add first_non_empty() and to_int() to core/type_utils.py
- Add get_filename_from_path() and METADATA_DIR to core/path_utils.py
- Replace inline implementations across resolver, downloader, and sources
with calls to the shared utilities
- Move strip_known_model_extension from resolver to matcher
2026-06-25 17:00:27 +02:00
Dariusz L 8655f92515 refactor: centralize category mapping and atomic JSON writing 2026-06-25 16:45:55 +02:00
Dariusz L 96036f0d6e refactor: DRY up source search functions and update HF author index
- Refactor __init__.py: extract run_source_search and execute_search_with_fallback
  helpers to eliminate duplicated progress-tracking, logging, and error-handling
  patterns across local, HuggingFace, CivitAI, CivArchive, and LoRA archive
  source search tasks; also fix URN variable shadowing in CivitAI/CivArchive

- Update metadata/huggingface-author-index.json with refreshed Comfy-Org
  author index (new repos and model entries)
2026-06-25 16:35:29 +02:00
Dariusz L 91d8e0cf3f Add return_success_on_error to API endpoints and import has_known_model_extension 2026-06-25 02:41:02 +02:00
Dariusz L 92e2601631 refactor: centralize API error handling and confidence calculation logic 2026-06-25 02:24:02 +02:00
Dariusz L 8f890f6992 refactor: centralize path utilities in core/path_utils
Extract repeated path normalization and identity logic from __init__.py,
core/resolver.py, core/downloader.py, and core/scanner.py into a shared
core/path_utils module. Also add a loading state to the local model picker
so it waits for models to load before rendering options.
2026-06-25 01:23:20 +02:00