Commit Graph
148 Commits
Author SHA1 Message Date
Ewald 094cb552c2 fix(security,ci): gate reads behind auth, harden release pipeline, fix cache-provider coupling and delete atomicity
- Require the same Comfy auth on all Majoor GET routes as on writes, closing
  a read-confidentiality gap for remote/multi-user deployments (no-op when
  Comfy auth is disabled, the default single-user setup).
- Gate nightly and stable releases on a quality job (pytest, mypy, frontend
  typecheck/build/tests, dist/ drift check) before publishing, and refresh
  the committed dist/ bundle to match a clean build.
- Pin softprops/action-gh-release and Comfy-Org/publish-node-action to
  commit SHAs.
- Resolve ComfyUI's CacheProvider through the public comfy_api.latest.Caching
  alias instead of the underscore-private _caching module.
- Make asset-delete DB cleanup atomic: a failed cleanup statement now aborts
  the transaction instead of committing a partial delete.
2026-09-04 20:53:57 +02:00
Ewald 0db328352a fix: close auth bypass on /api route mirror, revert PrimeVue to MIT 4.5.5
Security
--------
ComfyUI's PromptServer.add_routes() mirrors every registered route under a
second "/api" prefix, so /mjr/am/<mutation> is also served at
/api/mjr/am/<mutation>. The auth middleware matched only the canonical
prefix, so the mirrored path reached the same handler with the
authentication check skipped entirely, and without the hardening headers:

    POST /mjr/am/assets/delete      -> requires_auth = True
    POST /api/mjr/am/assets/delete  -> requires_auth = False

Replaced the prefix test with a shared _is_majoor_path() boundary covering
the canonical prefixes, ComfyUI's /api mirror, and the /api/v2/assets
compat surface. Verified it does not capture ComfyUI's own routes. CSRF,
rate limiting and safe-mode gates are per-handler and were unaffected.

Licensing
---------
PrimeVue 5 is no longer MIT: PrimeTek moved it to the commercial "PrimeUI"
license, which requires a key, shows a license notice without one, and
forbids redistributing it "as a component library or development tool"
without a separate OEM License. Dependabot had auto-merged 4.5.5 -> 5.0.1.
This project is GPL-3.0 and commits dist/ to the repo, so shipping it would
nag every user and conflict with the GPL's redistribution requirement.

Pinned back to 4.5.5 (last MIT release). The 5.x bump had touched only
package.json/package-lock.json, so no source depended on v5 APIs. The
revert drops @primeui/license-manager and its @noble/ed25519 key-signature
stack; the primevue chunk shrinks 555.70 -> 520.61 kB. Dependabot is now
configured to ignore primevue major bumps.

Correctness
-----------
- api_v2_assets: stop returning the DB's absolute filepath on this
  network-reachable surface; subfolder + root_id already convey location
  without disclosing local disk layout.
- watcher: enforce _RECENT_GENERATED_HARD_CAP after inserting a batch;
  pruning only beforehand let a large batch overshoot the cap.
- routes/registry: track registered route groups individually so a group
  failing part-way through does not re-register earlier groups on retry.
- fs_walker/scanner: apply the viewer's root check to symlinked files, so
  the scanner stops indexing symlinks whose target escapes every allowed
  root ("ghost assets" the viewer then refuses to open). Consulted only for
  actual symlinks, so ordinary scans pay no extra I/O.
- routes/registry_app: skip the redundant app.add_routes() on the live
  PromptServer app; ComfyUI adds the shared table itself after node
  loading, so every core and Majoor route was registered twice at startup.
- Result: unwrap()/unwrap_or()/map() gated on `data is not None`, treating
  a legitimate Ok(None) as failure; gate on `ok` instead.
- custom_roots: replace a leftover debug print() that wrote user file paths
  to stderr on every move request with logger.debug.
- schema: drop the redundant UNIQUE(filepath, source, root_id) index
  (migration m022). assets.filepath is already UNIQUE, so it could never
  reject a row the column constraint does not, nor narrow a lookup - it
  only cost a second B-tree write per asset on the scan hot path.

Frontend
--------
- entry.ts: app.registerExtension() throws on a duplicate name and ComfyUI
  exposes no unregisterExtension, so a live re-import threw before setup()
  ran and the reload-safe teardown never executed. Catch it and re-run
  setup() directly.
- Rebuilt dist/ so both the reload fix and the PrimeVue revert ship.

CI / tooling
------------
- frontend-compat-nightly: the types step died with a bare "cp: cannot
  stat" (upstream's build has not emitted dist/index.d.ts since Aug 26).
  Check both candidate paths and dump dist/ with an actionable message.
- package.json: lenient checks that swallowed their exit code are now
  suffixed :advisory, the plain names are strict, and quality:strict runs
  the full set.

Verified: 1841 Python tests pass (5 skipped, symlink privileges), ruff
clean; tsc --noEmit clean, 831 JS tests pass, eslint clean, build succeeds.
2026-09-04 18:41:48 +02:00
Ewald 59afa1caec Fix viewer video and audio playback 2026-08-13 09:53:09 +02:00
Ewald b700104588 feat: modernize ComfyUI integration and add KJNodes previews 2026-08-12 11:16:06 +02:00
bsawang 1d974209d5 fix: add test adapters for folder-browsing browse mode and missing 'mode' field
- Add _Settings mock returning show_folders=False to 3 DB-index tests
  so they bypass the new browse mode and exercise the index path.
- Add mode='filesystem' to _build_browse_response payload for
  consistency with the filesystem-fallback path.
2026-07-29 14:09:09 +08:00
Ewald 17db2fe9d7 fix(security): harden workflow path containment and bump brace-expansion
- Resolve CodeQL py/path-injection alerts: enforce lexical containment
  (normpath + prefix check) before any filesystem access in
  list_workflow_versions, diff_workflow_versions,
  list_workflow_thumbnail_candidates and _resolve_path_under_roots,
  with post-resolve symlink re-validation. Also removes a file
  existence oracle for paths outside the allowed roots.
- npm audit fix: brace-expansion DoS (GHSA-3jxr-9vmj-r5cp, dev-only).
2026-07-24 16:16:50 +02:00
Ewald e3afb1a0d3 feat: add Collect Files (context menu + sidebar) bundling asset, workflow, traced prompt text and media inputs into a zip
- New POST /mjr/am/collect-files route (CSRF, write token, rate limit, path allowlist)
- collect_service: extracts workflow media/model refs, resolves inputs within allowed roots, builds {stem}_collected.zip next to the asset with writable-dir fallback to output/_mjr_collected
- Zip contains asset, workflow.json, prompt_graph.json, prompt.json (positive/negative traced via geninfo parser), inputs/ and collected_files.txt manifest
- comfy_core adapter: get_model_full_path helper
- Frontend: 'Collect files' grid context menu item + sidebar (D) MButton, collectFiles() API client op
- Tests: tests/features/test_collect_service.py (10 tests)
2026-07-24 15:41:48 +02:00
Ewald 10f851aaae Add metadata FTS backfill, registry logging collision fixes, i18n locales, and doc updates 2026-07-22 16:33:34 +02:00
Ewald b98edbb80f Fix viewer paths and clarify write access 2026-07-10 11:43:00 +02:00
Ewald ed8e33c5e4 Prepare 2.5.0 features and release notes 2026-07-09 11:18:28 +02:00
Ewald cdd0314ad9 Fix search filter parsing and grid query typing 2026-07-01 16:16:26 +02:00
Ewald 2633e67ede Improve workflow metadata and viewer UX 2026-06-30 22:58:41 +02:00
Ewald 5fadbbbeb1 Fix metadata sidebar extraction and display 2026-06-29 18:07:21 +02:00
Ewald 6f116a3eb4 Fix generation prompt tracing 2026-06-29 08:54:20 +02:00
Ewald 30ae52d978 Integrate ComfyUI app mode and workflow UX fixes 2026-06-22 16:03:28 +02:00
Ewald 431faa8557 Fix security findings and test environment 2026-06-22 09:35:10 +02:00
Ewald b97f13d153 Optimize asset grid and viewer performance 2026-06-17 10:09:52 +02:00
Ewald bacc50000e Improve VRAM controls and settings organization 2026-06-16 17:17:19 +02:00
Ewald 5230dd43a2 Fix Python CI quality failures 2026-06-12 14:04:00 +02:00
Ewald 85d6da74fb fix: add type: ignore[index] for Result.data indexing in tests 2026-06-12 10:21:09 +02:00
Ewald 79cece3f01 fix: mypy type-ignore annotations in test_duplicates_service 2026-06-12 10:20:26 +02:00
Ewald 43a1fad285 chore: update dist, tests, and UI components 2026-06-12 10:19:33 +02:00
Ewald 5c31f29ecc Add workflow roots and fix video previews 2026-06-11 20:50:18 +02:00
Ewald 20a50040a6 Add workflow library scope 2026-06-09 10:28:25 +02:00
Ewald 4660c90b28 Fix CI event tests and bundle drift check 2026-06-03 15:05:49 +02:00
Ewald 99f6fe0ec3 test(types): satisfy mypy for StreamReader protocol stub 2026-06-02 17:22:07 +02:00
Ewald 8628122377 fix(ci): resolve ts typecheck and aiohttp test stub incompatibility 2026-06-02 17:21:42 +02:00
Ewald 31e5ae0d73 Fix asset grid refresh and media card handling 2026-06-02 16:25:20 +02:00
MajoorWaldi 99398aad5d Fix mojibake text and update default settings 2026-05-31 20:03:25 +02:00
Ewald a78f4572c8 Align frontend layout with ComfyUI conventions 2026-05-26 10:42:22 +02:00
Ewald 6587940862 refactor(app): enhance ComfyUI integration and settings sync - Implement two-way binding for ComfyUI settings in Majoor - Add new settings for auto-opening viewers and floating sidebar - Improve extension manager compatibility and dialog handling - Extend sidebar tab registration and controller discovery - Introduce graph canvas helpers for centering view and getting viewport bounds - Enhance minimap functionality with new graph canvas methods - Add settings sync with ComfyUI language 2026-05-23 18:35:21 +02:00
Ewald 37c65f984e fix(tests): add mypy None guards in test_fts_from_normalized + type-ignore in test_asset_updater 2026-05-22 15:03:38 +02:00
Ewald a363a77a6b feat(db): Phase C tag normalization — v18/v19 migrations, normalized FTS, readers v2, stop-write, drop legacy columns
- m018_fts_from_normalized_tags: replace FTS triggers to read from asset_tags+tags
  instead of the legacy JSON asset_metadata.tags column
- m019_drop_legacy_tag_columns: remove asset_metadata.tags and tags_text via
  create-copy-swap; includes invariant check (aborts if legacy tags not backfilled)
- registry.py: MIGRATIONS = [M017, M018, M019]
- schema_fts.py / schema_sql.py: FTS helpers and baseline triggers rewritten to
  source from normalized tables with asset_tags guard
- TagsRepository: normalized read path (get_tags_for_asset, list_assets_with_tag,
  search_assets_by_tags) + sync_from_legacy_row repair helper
- Readers v2: searcher, result_hydrator, metadata_helpers, updater, rating_tags_service,
  rename_service, stacks/service, duplicates/service migrated to asset_tags/tags JOIN
- Stop-Write: writes no longer update asset_metadata.tags / tags_text
- sqlite_facade: ahas_table helper; sqlite_recovery updated
- deps.py: TagsRepository wired into service container
- scripts/tag_drift_doctor.py: detects assets with legacy tags not backfilled
- ADR 0009, ARCHITECTURE_MAP, RATINGS_TAGS_COLLECTIONS, adr/README updated
- Tests: test_drop_legacy_tag_columns (2), test_fts_from_normalized (6),
  test_tags_dual_write, test_tags_normalized_readers + regression fixes across
  test_rating_tags_import, test_index_metadata_helpers, test_index_searcher,
  test_asset_updater, test_deps, test_schema_heal
- Full suite: 1687 passed, 2 skipped, 2 pre-existing failures (test_health_routes)
2026-05-22 15:00:27 +02:00
Ewald 9c19dfb6b5 feat(db): versioned migration runner + Repository pattern + normalized tags (Phase 3.2)
Introduces a forward-only, additive, idempotent migration framework on top of the existing legacy implicit-repair path (which remains the source of truth for schema versions <= 16). Versioned migrations live in mjr_am_backend/adapters/db/migrations/ and are tracked in a new schema_migrations bookkeeping table. The runner is invoked on every startup right after migrate_schema(); pending migrations are applied in version order, already-applied ones short-circuit.

First migration v17 normalize_tags creates a parallel tags + asset_tags relational layer (case-insensitive UNIQUE on tag name, cascade-delete on asset side, covering index on tag_id). Backfill reads asset_metadata.tags JSON safely (malformed rows skipped) and uses INSERT OR IGNORE everywhere for full idempotency. The legacy JSON column is NOT touched; existing v16 readers keep working.

Adds a Repository layer under mjr_am_backend/data/repositories/ to isolate raw SQL from services. First repository TagsRepository exposes ensure / get_by_name / list_all / attach / detach / list_for_asset / list_assets_with_tag / count on the new normalized tables, all returning Result.

Tests: 6 migration-runner tests (version validation, idempotency, bookkeeping, failure propagation, registry application) and 7 TagsRepository tests (CRUD, case-insensitivity, cascade-delete) - 13/13 passing. Existing test suite remains green (regressions limited to 2 pre-existing health_routes failures unrelated to this change).

Phase 3.2 closes the DB hardening item of the v3 improvement plan.
2026-05-21 23:15:55 +02:00
Ewald 625920846c feat(plugins): introduce Blueprint manifest format (Phase 3.3)
Adds a declarative blueprint.{yaml,json} manifest layer on top of the existing dynamic-module plugin loader. Each plugin can now ship as a self-contained package with a manifest describing name, version, author, type, entrypoint class, and capabilities. The legacy flat .py plugin format remains fully supported.

Manifest schema v1 with fields: schema_version, name (slug), version (semver), author, type (metadata_extractor), entrypoint.{module,class}, optional capabilities/tags/license/homepage/description/min_majoor_version. Path-traversal and module-name injection are rejected at parse time. YAML manifests require PyYAML (already shipped by ComfyUI core); JSON manifests need no extra deps.

Loader: PluginLoader.discover_plugins now treats subdirectories with a blueprint manifest as blueprint packages (taking precedence over plain Python-package scanning). New PluginLoader.blueprints property and get_stats[blueprints] field expose loaded manifests for listing endpoints.

Migrated bundled examples to blueprint layout: plugins/examples/wanvideo/, plugins/examples/custom_node_template/, plugins/examples/rgthree/ (YAML variant).

Tests: 11 new tests in tests/features/test_blueprint_plugins.py covering manifest parsing edge cases (bad name, bad semver, unknown type, path traversal, schema mismatch), YAML support, discovery skipping invalid manifests, and loader integration including a missing-entrypoint warning path.
2026-05-21 22:31:15 +02:00
Ewald 912ebb975d feat(api): add /api/v2/assets/* OpenAPI compat layer (Phase 3.1)
Read-only surface mirroring ComfyUI core OpenAPI assets spec under a /api/v2/ prefix to avoid collision with the existing /api/assets/* endpoints served by core.

Endpoints:

- GET  /api/v2/assets (list with name_contains, include/exclude_tags, sort, limit/offset)

- GET  /api/v2/assets/{id} (numeric ID, blake3:hex hash, or filename fallback)

- GET  /api/v2/assets/{id}/content (FileResponse, path-allowlist enforced)

- GET  /api/v2/assets/{id}/tags

- HEAD /api/v2/assets/hash/{hash} (existence check)

11 unit tests in tests/features/test_api_v2_assets.py covering payload shape, pagination, hash check (match/miss/malformed), 404 paths, and disallowed content paths.
2026-05-21 22:24:46 +02:00
Ewald 120c822c6f File path refacto 2026-05-21 21:07:16 +02:00
Ewald ddfe817b45 Polish asset manager UI and metadata helpers 2026-05-21 18:59:30 +02:00
Ewald b18cafcb3b Integrate Comfy core asset ingestion 2026-05-21 18:56:04 +02:00
Ewald 9876e36c22 chore: viewer, context menu, stacks, docs updates 2026-05-21 08:09:38 +02:00
Ewald 22648e5961 chore: commit remaining workspace changes 2026-05-17 21:21:49 +02:00
Ewald 4e7207a05d feat: update drag drop, docs, and build outputs 2026-05-15 17:13:04 +02:00
Ewald d174c1c141 test: fix mypy typing in asset operation service tests 2026-05-09 22:35:12 +02:00
Ewald 015c673d6d feat(viewer): add workflow graph map and viewer sync updates 2026-05-09 22:34:37 +02:00
Ewald 0bb9232ca7 Improve geninfo extraction and Happy Horse prompts 2026-05-06 23:14:48 +02:00
Ewald 8a7ab04558 Tighten remaining audit surfaces 2026-05-01 17:47:00 +02:00
Ewald 777d080494 Protect vector mutation routes 2026-05-01 17:36:12 +02:00
Ewald 0b72e50024 Harden mutation security defaults 2026-05-01 17:30:15 +02:00
Ewald 1ffd37f099 Fix mypy gate for search route tests 2026-05-01 04:36:42 +02:00
Ewald dbe876cff2 Fix asset manager pagination and preview stability 2026-05-01 04:35:04 +02:00