Commit Graph
109 Commits
Author SHA1 Message Date
Dariusz L cc8fdda32c chore: add pyproject metadata, CI workflows, and reorganize clone badge into badges/ 2026-07-04 19:59:03 +02:00
Dariusz L b57412f5cd Switch Model Resolver shortcut to native ComfyUI commands/keybindings with dynamic tooltip and exclude non-model categories 2026-07-04 04:34:34 +02:00
Dariusz L 1e22b2ba73 refactor: centralize repeated route logic into shared helpers 2026-07-04 01:23:46 +02:00
Dariusz L 546784e68a refactor: simplify aria2 status route and remove wrapper functions
- Merge GET and POST aria2 status handlers into one route
- Remove redundant calculate_file_sha256 wrapper and use path_utils directly
- Remove redundant _normalize_archive_image, _normalize_filename, and _similarity wrappers
- Replace urllib with requests for remote JSON fetching in model_list
- Update metadata timestamps
2026-07-04 01:03:00 +02:00
Dariusz L 4c065818e3 feat: add custom URL resolution to missing browser 2026-07-04 00:45:13 +02:00
Dariusz L cb666902e5 refactor: centralize hash calculation callbacks and catalog backup logic 2026-07-04 00:18:41 +02:00
Dariusz L 5de1500537 refactor: consolidate inline mappings and precondition checks into shared helpers 2026-07-04 00:04:22 +02:00
Dariusz L 4091dbb88d feat(civitai): add public API search fallback and improve search robustness 2026-07-03 22:00:13 +02:00
Dariusz L 65ff2baa30 Add local hash match cache and invalidate it on rescan/downloads 2026-07-03 17:09:35 +02:00
Dariusz L 42b13bbb60 feat: improve local hash matching and strengthen download selection metadata
- add POST /model_resolver/local-matches-by-hash backend endpoint
- expand local hash lookup sources to lora_manager_archive, popular, and model_list
- deduplicate backend hash matches by SHA256 value instead of source-hash tuple
- add frontend sync for remote hash matches and hash-label disambiguation
- propagate sha256, file_info, selected_file, and selected_version in model selection/download flows
- render local match status with consistent hash labels across search panel and resolvers
2026-07-03 16:37:10 +02:00
Dariusz L 4c777e5c50 feat: add local hash match identity tracking and highlights; simplify CivitAI import and hash lookup 2026-07-03 01:30:06 +02:00
Dariusz L 9c0ec8b26b Add progress reporting for loaded model and workflow analysis
- Add server-side loaded models progress tracking in __init__.py
- Report workflow node analysis progress from workflow_analyzer.py
- Add frontend polling, rendering, and UI flow for loaded models progress
2026-07-03 00:06:04 +02:00
Dariusz L 28d054b228 refactor: deduplicate progress tracking, file size extraction, and loaded model rendering
- Replace inline analysis/hash/search progress helpers with direct JobProgressTracker calls
- Add shared extract_file_size() utility and remove duplicated extraction in CivitAI/CivArchive/HuggingFace
- Extract getModelNameAndStrength/getModelToken helpers in loaded models view
- Add raw response mode to fetchJson and simplify backend log export handling
2026-07-02 23:13:37 +02:00
Dariusz L ce29f88c6d refactor: centralize repeated filename/search/URL/HTTP helpers
Replace duplicated filename-to-query logic with a single
\uild_filename_search_queries\ helper in matcher/civarchive/huggingface,
standardize Civitai URL generation in the frontend via \getCivitaiModelUrl\,
unify browser user-agent and response file-size extraction in the downloader
path, and derive workflow node-type category hints from widget categories
rather than maintaining a large static mapping.
2026-07-02 22:56:21 +02:00
Dariusz L 8ce70c30e6 refactor: consolidate shared utilities into type_utils and path_utils
Replace duplicated SHA256 normalization, metadata sidecar discovery, trained-words
extraction, type coercion, and cache helpers into shared modules. Clarify that the
_civitai.py/_civarchive.py/_huggingface.py/_workflow_analyzer.py refactorings are
behavior-preserving so future contributors keep behavior aligned in one place.
2026-07-02 22:41:56 +02:00
Dariusz L 37383ac046 refactor: consolidate logger imports and replace deprecated log.warn usage 2026-07-02 17:27:54 +02:00
Dariusz L 9951e1fbd5 refactor: centralize shared utilities in core/type_utils and remove redundant wrappers
- Move URL model-file validation logic to type_utils.looks_like_model_file and use it from __init__.py, civarchive
- Extract remote file-size probing into type_utils.fetch_remote_file_size and replace duplicated HEAD+Range logic in civarchive and huggingface
- Extract image normalization into type_utils.normalize_model_image, replacing duplicated CivitAI/CivArchive image mapping in civarchive and civitai
- Drop redundant local wrapper functions (_as_dict, _as_list, _version_sort_key, _select_primary_model_file, _extract_primary_file, _read_json_file, _write_json_file_atomic, _get_sha256_hash, _extract_civitai_image_id, _build_civitai_image_url) in favor of existing shared implementations
- Use path_utils.read_json_safe and write_json_atomic directly in model_list
2026-07-02 04:40:23 +02:00
Dariusz L 86bd6c04fc refactor: centralize model-extension handling and search-cache clearing
- Add clear_all_search_caches() in core/sources/__init__.py and replace 4
  individual cache-clearing call sites in __init__.py with the new helper.
- Add shared parse_civitai_model_path() in core/type_utils.py and use it
  from both CivitAI and CivArchive URL parsers to remove duplicated regex logic.
- Rename parse_huggingface_url() to get_huggingface_repo_and_file_from_url()
  in core/resolver.py for clarity.
- Add stripModelExtension() and hasModelExtension() utilities (with a single
  MODEL_EXTENSIONS_REGEX) in web/resolver/utils/render_format_methods.js
  and replace repeated inline regex patterns across search_panel.js,
  model_info_methods.js, tabs_loaded_methods.js, and queue_methods.js.
2026-07-02 03:57:43 +02:00
Dariusz L 2d2e8b4621 refactor: centralize filename and size utilities across sources
- Introduce get_filename_from_path in path_utils and replace repeated
  os.path.basename calls across core/downloader.py, core/resolver.py,
  core/sources/civarchive.py, core/sources/civitai.py, and
  core/sources/huggingface.py
- Move duplicated inline HTTP response size helpers (_response_file_size,
  _parse_size_header, _parse_content_range_size) into core/type_utils.py
  and update civarchive.py and huggingface.py to import them
- Remove inline normalize_sha256_value from __init__.py and reuse the
  existing normalize_sha256 helper
- Refactor options_methods.js localStorage read/write logic into a
  SETTINGS_MAP-driven loop to reduce repetition
2026-07-02 03:37:44 +02:00
Dariusz L 11ec45526c refactor: consolidate duplicated path and size utilities
Extract MODEL_EXTENSIONS and format_size_bytes into core/type_utils, removing duplicates from workflow_analyzer and downloader. Add getFilenameFromPath to render_format_methods and replace inline path-splitting logic across JS resolver actions, search, and views.
2026-07-02 02:27:53 +02:00
Dariusz L dc7ed980b9 refactor: deduplicate progress tracking and unify hash polling logic
- Introduce JobProgressTracker in __init__.py to consolidate the duplicated
  thread-safe progress/cancellation state and cleanup logic previously
  implemented separately for search and hash jobs.
- Extract shared pollHashCalculation method in model_info_methods.js to
  centralize hash polling, abort checking, and error handling across all
  call sites.
- Use getMissingModelKey in search_panel.js to eliminate inline key
  construction duplication.
2026-07-02 01:50:19 +02:00
Dariusz L 1128234bbd feat: add local model hash comparison functionality
Implement end-to-end hash comparison for local models and local matches.

Backend:
- Add /model_resolver/local-model-hashes API endpoint
- Add get_local_model_hash_metadata() to read sidecar SHA256 metadata

Frontend:
- Add Compare Hashes context menu item and CSS
- Pass missing_key/missing_search_key through resolveDownloadMethods
- Enhance local match rendering with hash status and context data
- Add hash normalization, formatting, and candidate collection helpers
- Wire compareLocalModelHashesWithCurrentFinding action

Also update HuggingFace author index metadata.
2026-07-02 00:27:59 +02:00
Dariusz L 25e20eea8f feat: add frontend and backend log export in options 2026-06-30 23:14:37 +02:00
Dariusz L a7f5f70e3c refactor: migrate logging from global functions to per-module loggers
Replace global log_* wrapper functions with create_module_logger(__name__)
instances across the codebase, enabling proper module-level logger
namespacing. Simplifies core/log_system/log_funcs.py by removing legacy
_normalize_module_path and cleaning up _normalize_module_name. Bumps
log_system config to 2.0.0 and introduces PROJECT_ALIASES for cleaner
log module prefix stripping.
2026-06-30 21:24:33 +02:00
Dariusz L 54f861a02e feat: add aria2 daemon start/stop control and install migration
- Add /aria2/start and /aria2/stop API endpoints for explicit daemon lifecycle control
- Track aria2 daemon state (managed/running/version/active_transfers) in get_aria2_status
- Implement idle auto-stop timer (5 minutes) with aria2_auto_stop_daemon setting
- Add start_aria2_daemon() and stop_aria2_daemon() helpers in core/downloader.py
- Migrate legacy release-* install directories to normalized versioned paths
- Normalize aria2 version strings (strip release-/v prefixes) in installer metadata
- Update aria2 archive extraction to extract-to-temp then copy into canonical dir
- Add frontend daemon controls: status display, stop button, auto-stop toggle
2026-06-30 00:03:08 +02:00
Dariusz L 8b15500ce4 Add automatic aria2 binary installation support
- New core/aria2_installer.py module downloads the latest aria2c release
  from GitHub for Windows, macOS, and Linux with safe path extraction.
- New POST /model_resolver/aria2/install endpoint triggers installation
  and saves the resolved aria2c_path plus download_backend=aria2.
- Options UI adds an Install aria2 button, hides aria2-specific fields
  when another backend is selected, and updates status on completion.
- Includes .gitignore entry for tools/aria2/ so installed binaries are
  not committed.
2026-06-29 21:55:27 +02:00
Dariusz L 59e874c17f feat: improve CivitAI download support with session tokens and proper headers
- Add __init__.py: support civitai.red domain and session token cookies
- Add core/downloader.py: build_download_headers for consistent User-Agent,
  Accept, and CivitAI-specific Referer/Origin headers across download backends
- Resolve CivitAI API redirects before handing URLs to aria2
- Strip sensitive URL parameters from download logs
- Forward civitai_session_token from the frontend to the downloader
2026-06-28 03:42:41 +02:00
Dariusz L aa91d1410c feat: add aria2 download backend with pause/resume support
Integrates aria2c as an alternative download backend alongside the built-in
Python downloader. Adds settings for selecting the backend and configuring
the aria2c executable path, plus a status check button in the options panel.

Backend changes:
- core/downloader.py: aria2 daemon lifecycle, JSON-RPC client, pause/resume
  state machine, and transfer tracking
- core/settings.py: download_backend normalization and aria2c_path setting
- __init__.py: new API routes for pause, resume, and aria2 status checks

Frontend changes:
- options_methods.js: backend selector, aria2c path input, check button,
  and local storage persistence
- resolve_download_methods.js: pause/resume handlers, pending status
  transitions, and progress rendering for aria2 downloads
- queue_methods.js: queue panel pause/resume buttons, backend badge, and
  debounced panel updates
- search_panel.js: actionsHtml support for composite action buttons
2026-06-27 23:39:31 +02:00
Dariusz L b73cdb0f40 feat: improve category resolution and missing models browser
- Expand preferred categories to include clip_vision, ipadapter, sams, ultralytics
- Add clip_gguf alias for text_encoders in both backend and frontend
- Add explicit diffusion_models/text_encoders mappings in CATEGORY_MAP
- Refactor get_model_roots to skip custom_nodes/configs, use template key
  aliases, accept new folder keys from known categories, and normalize
  ultralytics paths (yolo/bbox, yolo/segm, yolo -> parent)
- Pass missing object through resolve_download_methods and search_panel
  so category selection can use widget hints
- Refactor frontend download target resolution:
  - getMissingSupportedDownloadCategories respects widget/type hints
  - getNodeTypeDownloadCategory splits exact-node map and adds
    normalizeNodeCategoryToken / getNodeTypeDownloadCategoryFromTokens
    for GGUF and future node families
  - getSourceResultDownloadCategory and _sourceSupportLevel accept
    missing for unified logic
- Prioritize dynamic widget category hints in workflow analyzer and
  include category_hints/folder_key_hints on all model refs
- Add folder count badges and path normalization in missing browser
  (normalizeMissingFolderRootPath handles category suffixes / bbox/segm)
2026-06-27 19:36:47 +02:00
Dariusz L e53d9f2a0b feat: generalize remote metadata lookup across CivitAI, CivArchive, and HuggingFace
Replace CivitAI-only search with a multi-source metadata resolution route using
SHA256 exact-match validation, dead-link filtering, and live download URL
probing. Unify sidecar metadata saving across all remote sources. Add
force-refresh support, local metadata import with preview detection, and
frontend source-aware UI controls.
2026-06-27 03:25:31 +02:00
Dariusz L 3cd302d4ee feat: implement local-only model info fetching and refactor model info dialog
- Add local_only endpoint mode: returns basic file info (filename, size, path,
  location, sha256) without querying CivitAI or computing file hashes
- Validate provided sha256/hash inputs (must be 64-char hex) and fall back to
  empty string when invalid
- Refactor monolithic info response builder into build_info_response() helper
  used across all result paths (local, CivitAI success, CivitAI fallback,
  not-found), reducing duplication
- Add get_model_info_for_file(local_only=True) early return in CivitAI source
  that builds a minimal local-only result without hash lookup
- Split frontend fetchModelInfoForDialog into:
  - fetchLocalModelInfoForDialog: opens dialog with local data immediately,
    marks civitai_checked=false
  - fetchCivitaiModelInfoForDialog: async CivitAI lookup triggered by user
- Add 'Fetch with CivitAI' button in info dialog when local file exists but
  CivitAI has not been checked yet
- Infer model_type from category in both backend and frontend to reduce
  reliance on upstream data shape
- Fix sidecar metadata condition (was skipping save when path already existed)
- Add cursor/disabled styles for info dialog CivitAI link button
2026-06-26 22:16:44 +02:00
Dariusz L 4e576cbd56 feat: add SHA256 hash calculation with cancellation, progress tracking, and remote size probing 2026-06-26 21:51:55 +02:00
Dariusz L c7d6b27fb0 feat: add search cancellation API and harden CivArchive source resolution
Add a /model_resolver/search-cancel/{progress_id} endpoint with a
SearchCancelled exception so in-flight source searches abort cleanly on
batch cancel. Track sourceProgressIds alongside sourceControllers so the
frontend can cancel backend progress polls. Rework CivArchive extraction
with robust __NEXT_DATA__ parsing, hash-page file resolution, model-link
scraping, confidence-based candidate filtering, and expanded type maps.
Add conftest.py for pytest path setup, update .gitignore for scratch/,
and refresh huggingface-author-index.json.
2026-06-26 00:55:38 +02:00
Dariusz L a320d4d805 refactor: consolidate duplicated logic into shared utilities
- Add get_comfy_root_path() to core/path_utils.py and use it in __init__.py and core/downloader.py instead of inline logic
- Add select_primary_model_file(), parse_size_to_bytes(), get_version_sort_key(), and check_credential_http() to core/type_utils.py
- Replace duplicated credential-check helpers in civitai.py, huggingface.py with check_credential_http()
- Replace duplicated size parsing and version sorting in civarchive.py, civitai.py with shared utilities
- Replace duplicated primary-file selection in civarchive.py, civitai.py, lora_manager_archive.py with select_primary_model_file()
- Replace inline tempfile-based JSON writes in popular.py with write_json_atomic()
- Remove dead reload_model_list() from model_list.py
- Fix workflow_analyzer.py hint prioritization for diffusion_models vs specific output hints
2026-06-25 18:54:59 +02:00
Dariusz L 6850ef1267 refactor: centralize type coercion and file I/O helpers to eliminate duplication 2026-06-25 17:33:59 +02:00
Dariusz L d979bbcf21 refactor: extract shared utilities and deduplicate metadata path definitions
- Add first_non_empty() and to_int() to core/type_utils.py
- Add get_filename_from_path() and METADATA_DIR to core/path_utils.py
- Replace inline implementations across resolver, downloader, and sources
with calls to the shared utilities
- Move strip_known_model_extension from resolver to matcher
2026-06-25 17:00:27 +02:00
Dariusz L 8655f92515 refactor: centralize category mapping and atomic JSON writing 2026-06-25 16:45:55 +02:00
Dariusz L 96036f0d6e refactor: DRY up source search functions and update HF author index
- Refactor __init__.py: extract run_source_search and execute_search_with_fallback
  helpers to eliminate duplicated progress-tracking, logging, and error-handling
  patterns across local, HuggingFace, CivitAI, CivArchive, and LoRA archive
  source search tasks; also fix URN variable shadowing in CivitAI/CivArchive

- Update metadata/huggingface-author-index.json with refreshed Comfy-Org
  author index (new repos and model entries)
2026-06-25 16:35:29 +02:00
Dariusz L 91d8e0cf3f Add return_success_on_error to API endpoints and import has_known_model_extension 2026-06-25 02:41:02 +02:00
Dariusz L 92e2601631 refactor: centralize API error handling and confidence calculation logic 2026-06-25 02:24:02 +02:00
Dariusz L 8f890f6992 refactor: centralize path utilities in core/path_utils
Extract repeated path normalization and identity logic from __init__.py,
core/resolver.py, core/downloader.py, and core/scanner.py into a shared
core/path_utils module. Also add a loading state to the local model picker
so it waits for models to load before rendering options.
2026-06-25 01:23:20 +02:00
Dariusz L 0b98054536 refactor: consolidate duplicated credential checks and local match fetch logic 2026-06-24 23:54:18 +02:00
Dariusz L 5445dd4ac0 feat: implement local hash matching via metadata sidecar files
Backend adds SHA256 normalization, metadata sidecar lookup, and hash fields across source modules. Frontend handles local_hash_matches in search responses and missing browser.
2026-06-24 21:59:10 +02:00
Dariusz L 8c0168ab76 feat: skip model analysis for empty or non-model workflows
Add early-return optimizations so both backend and frontend avoid unnecessary
URL extraction and disk scanning when a workflow has no nodes or contains
no potential model references.

Also tighten API input validation by rejecting non-object workflow_json.
2026-06-24 03:46:50 +02:00
Dariusz L 16a32de8de feat: improve model info dialog with clickable location, wider layout, and refined styling 2026-06-23 20:28:48 +02:00
Dariusz L 093b6d33a7 feat: enrich model info dialog with location, size, tags, and trained words from metadata 2026-06-23 19:47:49 +02:00
Dariusz L 7cd57c05cb improve folder path resolution with identity-based deduping and directory preference logic 2026-06-23 03:50:02 +02:00
Dariusz L 48bc3bf5f1 feat: wire progress callbacks into backend search sources 2026-06-23 03:33:28 +02:00
Dariusz L d8124d03da feat: add live backend search progress tracking
Frontend now polls a new backend endpoint (/model_resolver/search-progress/{progress_id})
to display real-time, per-source progress messages and percentages during model searches,
replacing static 'Searching...' indicators with granular status updates.
2026-06-23 02:23:45 +02:00
Dariusz L 1a4156f00d feat: add any-model fallback search for civitai, civarchive, and lora_manager_archive
When a model search returns no results for a specific base_model, retry the
same query with base_model_context=None so the provider can match any model.
Fallback matches are marked with any_model_match and base_model_fallback on
the result payload. The UI reports 'Found (Any Model)' when a result came
from this fallback path, and the cancel/progress button styling is tightened
to 16x16px with border-radius.
2026-06-20 00:53:51 +02:00