22 Commits
Author SHA1 Message Date
Ewald 0db328352a fix: close auth bypass on /api route mirror, revert PrimeVue to MIT 4.5.5
Security
--------
ComfyUI's PromptServer.add_routes() mirrors every registered route under a
second "/api" prefix, so /mjr/am/<mutation> is also served at
/api/mjr/am/<mutation>. The auth middleware matched only the canonical
prefix, so the mirrored path reached the same handler with the
authentication check skipped entirely, and without the hardening headers:

    POST /mjr/am/assets/delete      -> requires_auth = True
    POST /api/mjr/am/assets/delete  -> requires_auth = False

Replaced the prefix test with a shared _is_majoor_path() boundary covering
the canonical prefixes, ComfyUI's /api mirror, and the /api/v2/assets
compat surface. Verified it does not capture ComfyUI's own routes. CSRF,
rate limiting and safe-mode gates are per-handler and were unaffected.

Licensing
---------
PrimeVue 5 is no longer MIT: PrimeTek moved it to the commercial "PrimeUI"
license, which requires a key, shows a license notice without one, and
forbids redistributing it "as a component library or development tool"
without a separate OEM License. Dependabot had auto-merged 4.5.5 -> 5.0.1.
This project is GPL-3.0 and commits dist/ to the repo, so shipping it would
nag every user and conflict with the GPL's redistribution requirement.

Pinned back to 4.5.5 (last MIT release). The 5.x bump had touched only
package.json/package-lock.json, so no source depended on v5 APIs. The
revert drops @primeui/license-manager and its @noble/ed25519 key-signature
stack; the primevue chunk shrinks 555.70 -> 520.61 kB. Dependabot is now
configured to ignore primevue major bumps.

Correctness
-----------
- api_v2_assets: stop returning the DB's absolute filepath on this
  network-reachable surface; subfolder + root_id already convey location
  without disclosing local disk layout.
- watcher: enforce _RECENT_GENERATED_HARD_CAP after inserting a batch;
  pruning only beforehand let a large batch overshoot the cap.
- routes/registry: track registered route groups individually so a group
  failing part-way through does not re-register earlier groups on retry.
- fs_walker/scanner: apply the viewer's root check to symlinked files, so
  the scanner stops indexing symlinks whose target escapes every allowed
  root ("ghost assets" the viewer then refuses to open). Consulted only for
  actual symlinks, so ordinary scans pay no extra I/O.
- routes/registry_app: skip the redundant app.add_routes() on the live
  PromptServer app; ComfyUI adds the shared table itself after node
  loading, so every core and Majoor route was registered twice at startup.
- Result: unwrap()/unwrap_or()/map() gated on `data is not None`, treating
  a legitimate Ok(None) as failure; gate on `ok` instead.
- custom_roots: replace a leftover debug print() that wrote user file paths
  to stderr on every move request with logger.debug.
- schema: drop the redundant UNIQUE(filepath, source, root_id) index
  (migration m022). assets.filepath is already UNIQUE, so it could never
  reject a row the column constraint does not, nor narrow a lookup - it
  only cost a second B-tree write per asset on the scan hot path.

Frontend
--------
- entry.ts: app.registerExtension() throws on a duplicate name and ComfyUI
  exposes no unregisterExtension, so a live re-import threw before setup()
  ran and the reload-safe teardown never executed. Catch it and re-run
  setup() directly.
- Rebuilt dist/ so both the reload fix and the PrimeVue revert ship.

CI / tooling
------------
- frontend-compat-nightly: the types step died with a bare "cp: cannot
  stat" (upstream's build has not emitted dist/index.d.ts since Aug 26).
  Check both candidate paths and dump dist/ with an actionable message.
- package.json: lenient checks that swallowed their exit code are now
  suffixed :advisory, the plain names are strict, and quality:strict runs
  the full set.

Verified: 1841 Python tests pass (5 skipped, symlink privileges), ruff
clean; tsc --noEmit clean, 831 JS tests pass, eslint clean, build succeeds.
2026-09-04 18:41:48 +02:00
Ewald 20a50040a6 Add workflow library scope 2026-06-09 10:28:25 +02:00
Ewald a363a77a6b feat(db): Phase C tag normalization — v18/v19 migrations, normalized FTS, readers v2, stop-write, drop legacy columns
- m018_fts_from_normalized_tags: replace FTS triggers to read from asset_tags+tags
  instead of the legacy JSON asset_metadata.tags column
- m019_drop_legacy_tag_columns: remove asset_metadata.tags and tags_text via
  create-copy-swap; includes invariant check (aborts if legacy tags not backfilled)
- registry.py: MIGRATIONS = [M017, M018, M019]
- schema_fts.py / schema_sql.py: FTS helpers and baseline triggers rewritten to
  source from normalized tables with asset_tags guard
- TagsRepository: normalized read path (get_tags_for_asset, list_assets_with_tag,
  search_assets_by_tags) + sync_from_legacy_row repair helper
- Readers v2: searcher, result_hydrator, metadata_helpers, updater, rating_tags_service,
  rename_service, stacks/service, duplicates/service migrated to asset_tags/tags JOIN
- Stop-Write: writes no longer update asset_metadata.tags / tags_text
- sqlite_facade: ahas_table helper; sqlite_recovery updated
- deps.py: TagsRepository wired into service container
- scripts/tag_drift_doctor.py: detects assets with legacy tags not backfilled
- ADR 0009, ARCHITECTURE_MAP, RATINGS_TAGS_COLLECTIONS, adr/README updated
- Tests: test_drop_legacy_tag_columns (2), test_fts_from_normalized (6),
  test_tags_dual_write, test_tags_normalized_readers + regression fixes across
  test_rating_tags_import, test_index_metadata_helpers, test_index_searcher,
  test_asset_updater, test_deps, test_schema_heal
- Full suite: 1687 passed, 2 skipped, 2 pre-existing failures (test_health_routes)
2026-05-22 15:00:27 +02:00
Ewald 9c19dfb6b5 feat(db): versioned migration runner + Repository pattern + normalized tags (Phase 3.2)
Introduces a forward-only, additive, idempotent migration framework on top of the existing legacy implicit-repair path (which remains the source of truth for schema versions <= 16). Versioned migrations live in mjr_am_backend/adapters/db/migrations/ and are tracked in a new schema_migrations bookkeeping table. The runner is invoked on every startup right after migrate_schema(); pending migrations are applied in version order, already-applied ones short-circuit.

First migration v17 normalize_tags creates a parallel tags + asset_tags relational layer (case-insensitive UNIQUE on tag name, cascade-delete on asset side, covering index on tag_id). Backfill reads asset_metadata.tags JSON safely (malformed rows skipped) and uses INSERT OR IGNORE everywhere for full idempotency. The legacy JSON column is NOT touched; existing v16 readers keep working.

Adds a Repository layer under mjr_am_backend/data/repositories/ to isolate raw SQL from services. First repository TagsRepository exposes ensure / get_by_name / list_all / attach / detach / list_for_asset / list_assets_with_tag / count on the new normalized tables, all returning Result.

Tests: 6 migration-runner tests (version validation, idempotency, bookkeeping, failure propagation, registry application) and 7 TagsRepository tests (CRUD, case-insensitivity, cascade-delete) - 13/13 passing. Existing test suite remains green (regressions limited to 2 pre-existing health_routes failures unrelated to this change).

Phase 3.2 closes the DB hardening item of the v3 improvement plan.
2026-05-21 23:15:55 +02:00
Ewald 80c0aad5e7 audit profond : corrections et améliorations 2026-04-30 08:42:27 +02:00
Ewald eb5684fbdd docs: update README and architecture map
- Update frontend stack description in README
- Add detailed architecture and module responsibilities in ARCHITECTURE_MAP.md
- Update PLAN_REFACTO_COMPLET_MAJOOR_ASSETS_MANAGER.md with latest changes
2026-04-10 09:32:30 +02:00
MajoorWaldi 5c17eb83d4 Refactor hybrid search and vector search handlers for improved query filtering and asset management
- Added `_build_asset_filter_query` and `_query_allowed_asset_ids` functions to streamline asset filtering in hybrid search.
- Enhanced `_filter_semantic_hits` to utilize new asset filtering methods.
- Refactored vector search functions to improve K-means clustering logic and asset filtering.
- Introduced helper functions for parsing and validating query filters in `query_sanitizer.py`.
- Improved viewer context resolution in `viewer.py` with better error handling and path validation.
- Updated schema healing tests to use helper functions for column validation.
- Enhanced test coverage for vector indexing and filesystem entries.
- Added new test images for filesystem entry tests.
2026-03-29 15:33:59 +02:00
MajoorWaldi c1955f26cb feat: Implement live asset management and deduplication logic in InfiniteScroll and asset handling 2026-03-28 21:34:37 +01:00
MajoorWaldi 4411c72e4e Refactor test imports and clean up unused code across multiple test files
- Removed unnecessary imports from various test files to improve readability and maintainability.
- Added new settings for the Generated Feed display in the settingsFeed.js file.
- Introduced a new configuration file for Pyright type checking.
2026-03-28 13:26:28 +01:00
MajoorWaldi fb8a4d39a5 feat: Enhance database handling with new vector DB resolution and nested transaction support 2026-03-26 21:57:26 +01:00
MajoorWaldi 29e5059c4d feat(ai): enhance asset alignment, captioning and tagging
- Add AI alignment score display to asset metadata
- Implement image description fallback for non-workflow image assets
- Improve AI captioning UI and UX
- Optimize AI tagging suggestions with bootstrap generation
- Refactor collections controller to support AI toggling
- Enhance error handling and retry logic for asset indexing
2026-03-05 02:21:13 +01:00
MajoorWaldi 88a89372a2 feat: Enhance vector service with multimodal capabilities and logging settings
- Updated vector_service.py to support SigLIP2, X-CLIP, and Florence-2 models for embedding images, text, and video keyframes.
- Introduced AI verbose logging settings, allowing dynamic configuration of logging verbosity for model loading and inference.
- Modified database maintenance to backfill missing vectors for both image and video assets.
- Added health check routes for managing AI logging settings.
- Implemented enhanced caption generation for images using Florence-2.
- Updated vector search routes to include enhanced prompt generation and improved error handling.
- Added tests for new features, including enhanced prompt generation and dimension fallback mechanisms.
2026-03-03 00:15:58 +01:00
MajoorWaldi e7c2443670 fix(security): improve API token handling and add server-side bootstrapping
- Move token from local to session storage on client side
- Set httpOnly cookie for server-side token on bootstrap/rotate
- Add token validation and auto-sanitization
- Implement retry logic for failed operations
- Add input validation for collection names
- Update toast durations and error handling
2026-02-27 10:09:44 +01:00
MajoorWaldi 495432f19c feat(search): add query sanitization and result filtering helpers
- Introduced `query_sanitizer.py` for inline query, date, and sort sanitization.
- Added `result_filter.py` for runtime filtering helpers, including asset exclusion and enrichment pause functionality.
- Implemented `result_hydrator.py` for database hydration helpers, ensuring assets are properly hydrated with metadata.
- Created unit tests in `test_db_refactor_helpers.py` to validate database lock detection and SQL write query identification.
2026-02-25 13:31:28 +01:00
Your Name 934f10b58a Refactor imports and improve type hints across multiple modules
- Organized and grouped import statements for better readability in health.py, metadata.py, releases.py, scan.py, search.py, version.py, viewer.py, and registry.py.
- Updated type hints to use the new union syntax (e.g., str | None) for better clarity in scan.py, settings.py, tool_detect.py, and result.py.
- Enhanced the handling of optional types in various functions to ensure more robust type checking.
- Removed unnecessary blank lines and improved formatting for consistency.
- Adjusted the logic in several test files to ensure compatibility with the updated import structure and type hints. (See <attachments> above for file contents. You may not need to search or read the file again.)
2026-02-20 19:13:15 +01:00
Your Name f38ab4209e Save changes 2026-02-19 22:44:26 +01:00
Ewald da847f35f9 Add regression tests for route security guards
- Implement tests to ensure file paths are properly allowlisted in custom view and folder info handlers.
- Verify that collections mutations enforce CSRF protection and write access guards.
- Check that settings mutations require write access for specific routes.
2026-02-18 14:37:13 +01:00
Ewald 77fc6ffdcf feat: Enhance asset management with improved rename handling and foreign key updates 2026-02-13 17:39:42 +01:00
Ewald 00ee7806f7 feat: Add tool detection helpers for ExifTool and FFprobe
- Implemented caching for tool availability and versions.
- Added functions to check for ExifTool and FFprobe availability.
- Included version satisfaction checks against minimum required versions.
- Created utility functions for parsing boolean and float values from environment variables.
- Introduced shared utilities for error message sanitization and logging with emoji indicators.
- Established a Result pattern for error handling without exceptions.
- Added scan throttling helpers to manage background scans efficiently.
- Implemented time utilities for timestamps and performance measurement.
- Defined shared types, enums, and constants for consistent error handling and file classification.
- Created version management utilities to retrieve version information from pyproject.toml.
- Developed installation script for runtime dependencies.
- Added tests for ExifTool batch processing and various configuration scenarios.
- Implemented tests for version route aliases and runtime output root configuration.
2026-02-12 17:55:28 +01:00
Your Name 1646e2171b feat: Implement duplicate detection and similarity analysis features
- Added DuplicatesService for background duplicate analysis and similarity detection.
- Introduced endpoints for analyzing duplicates, checking status, and merging tags.
- Created utility functions for file hashing and perceptual hashing.
- Implemented watcher settings for runtime tuning of debounce and deduplication.
- Added database maintenance tools for resetting and deleting the database.
- Enhanced error handling with sanitization for client responses.
- Added tests for duplicates service and error handling.
- Updated ESLint configuration for better code quality checks.
- Introduced filename validation and logging utilities.
2026-02-08 15:11:17 +01:00
Your Name f8898e964c Release 2.2.2: i18n support + viewer/minimap improvements 2026-01-30 03:37:28 +01:00
Your Name 8ed51fbf39 make assets manager docs, license, and code improvements 2026-01-24 19:28:33 +01:00