diff --git a/.comfyignore b/.comfyignore index 46e2048..e8f24d9 100644 --- a/.comfyignore +++ b/.comfyignore @@ -7,6 +7,8 @@ __pycache__/ .git/ .github/ docs/ +tests/ +GITHUB_RELEASE_CHECKLIST.md temp/ tmp/ diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml new file mode 100644 index 0000000..67ac0a6 --- /dev/null +++ b/.github/workflows/ci.yml @@ -0,0 +1,75 @@ +name: CI + +on: + push: + pull_request: + workflow_dispatch: + +permissions: + contents: read + +jobs: + unit: + name: Python unit tests + runs-on: ubuntu-latest + timeout-minutes: 20 + steps: + - name: Check out code + uses: actions/checkout@v6 + + - name: Set up Python + uses: actions/setup-python@v6 + with: + python-version: "3.11" + + - name: Install unit-test dependencies + run: | + python -m pip install --upgrade pip + python -m pip install --index-url https://download.pytorch.org/whl/cpu torch + python -m pip install "jsonschema>=4.26.0" + + - name: Run unit tests + run: python -m unittest discover -s tests -p "test_*.py" -v + + schema-readiness: + name: Schema readiness + runs-on: ubuntu-latest + timeout-minutes: 10 + env: + DG_PROGRESS: "0" + steps: + - name: Check out code + uses: actions/checkout@v6 + + - name: Set up Python + uses: actions/setup-python@v6 + with: + python-version: "3.11" + + - name: Install schema-test dependencies + run: | + python -m pip install --upgrade pip + python -m pip install "jsonschema>=4.26.0" + + - name: Run readiness and schema smoke + run: python scripts/smoke_prompt_builder.py + + javascript: + name: JavaScript tests + runs-on: ubuntu-latest + timeout-minutes: 10 + steps: + - name: Check out code + uses: actions/checkout@v6 + + - name: Set up Node.js + uses: actions/setup-node@v6 + with: + node-version: "24" + package-manager-cache: false + + - name: Check shipped JavaScript syntax + run: node --check web/js/model_loader.js + + - name: Run JavaScript tests + run: node --test tests/*.mjs diff --git a/.github/workflows/publish_action.yml b/.github/workflows/publish_action.yml index 8d7d111..5f2867a 100644 --- a/.github/workflows/publish_action.yml +++ b/.github/workflows/publish_action.yml @@ -3,13 +3,16 @@ name: Publish to Comfy Registry on: workflow_dispatch: +permissions: + contents: read + jobs: publish-node: name: Publish custom node to registry runs-on: ubuntu-latest steps: - name: Check out code - uses: actions/checkout@v4 + uses: actions/checkout@v6 - name: Publish custom node uses: Comfy-Org/publish-node-action@main diff --git a/GITHUB_RELEASE_CHECKLIST.md b/GITHUB_RELEASE_CHECKLIST.md index 9d52add..e31b04c 100644 --- a/GITHUB_RELEASE_CHECKLIST.md +++ b/GITHUB_RELEASE_CHECKLIST.md @@ -1,38 +1,69 @@ -# GitHub Release Checklist +# GitHub and Comfy Registry Release Checklist -Use this checklist before pushing the node pack to a standalone GitHub repo. +Use this checklist from the standalone `ComfyUI-DiffusionGemmaPromptBuilder` repository before pushing or publishing a release. -## Package Scope +## Product Boundary -- Exported DiffusionGemma nodes are limited to the nodes used by `examples/07_ltx23_character_motion_transfer.json`. -- The example assets under `examples/assets/character_motion_transfer` are intended to ship with the repo. -- The NVFP4 bridge is source code inside `nodes.py`; do not bundle model weights or generated tensor dumps. +- Keep the stable Director Assistant surface at seven exports: + - five-node core: Model Loader, Context Hub, Target Profile, CoT Generator, JSON Splitter + - two companions: H3 Reference Context and Generation Gate +- Keep the two SplatStage planners visibly experimental. Their mapping ids stay compatible, their display names end in `(Experimental)`, and their category is `prompt/diffusiongemma/experimental-motion-planning`. +- Confirm `NODE_CLASS_MAPPINGS` and `NODE_DISPLAY_NAME_MAPPINGS` contain the same nine ids. +- Treat `ltx_reframe.py` and the reframe workflows as unexported prototypes. Do not describe them as installed nodes unless the package entry point and frontend are deliberately restored together. +- Do not bundle model weights or generated tensor dumps. The NVFP4 bridge remains source code in `nodes.py`. -## Before Upload +## Repository Preflight -1. Confirm the GitHub repo URL in `pyproject.toml`. -2. Confirm the Comfy Registry publisher id in `pyproject.toml` if you plan to publish through the Registry. -3. Confirm `LICENSE` and the Python copyright headers still match the intended copyright holder and release policy. -4. Run `C:\ComfyUI\.venv\Scripts\python.exe scripts\smoke_prompt_builder.py`. -5. Run `C:\ComfyUI\.venv\Scripts\python.exe proof_gates.py --model-path C:\ComfyUI\models\LLM\diffusiongemma-26B-A4B-it-NVFP4`. -6. Fresh-install the repo in a clean ComfyUI custom-nodes folder and confirm the five nodes appear. -7. Copy `examples/assets/character_motion_transfer` to `ComfyUI/input/character_motion_transfer` and load the example workflow. -8. Confirm `.comfyignore` excludes `*.mp4` and `*.gif` before publishing to the Comfy Registry. +1. Run `git rev-parse --show-toplevel` and confirm it resolves to this package directory, not the parent ComfyUI checkout. +2. Confirm `origin` is `https://github.com/exportAnything/ComfyUI-DiffusionGemmaPromptBuilder` and that the branch is based on the public repository history. +3. Confirm the worktree is clean and the intended release commit is pushed. +4. Confirm the repository URLs in `pyproject.toml` are current. +5. Keep `[tool.comfy].PublisherId` exactly `exportanything`; this is the immutable Registry id, while `exportAnything` is its display name. +6. Confirm `project.version` is newer than the latest Registry version. Version `0.1.0` is already published, so this release begins at `0.2.0`. +7. Confirm `requires-comfyui = ">=0.30.1"` and record ComfyUI `0.30.1` as the tested baseline. The Registry field is a compatibility range, not a test-report field. +8. Confirm `LICENSE` and the Python copyright headers match the intended release policy. -## Files To Include +## Automated Gates -- `__init__.py` -- `nodes.py` -- `proof_gates.py` -- `README.md` -- `requirements.txt` -- `pyproject.toml` -- `.gitignore` -- `.comfyignore` -- `.github/workflows/publish_action.yml` -- `web/` -- `schemas/` -- `scripts/` -- `examples/07_ltx23_character_motion_transfer.json` -- `examples/README.md` -- `examples/assets/character_motion_transfer/` +Run the same three gates enforced by `.github/workflows/ci.yml`: + +```text +python -m unittest discover -s tests -p "test_*.py" -v +python scripts/smoke_prompt_builder.py +node --check web/js/model_loader.js +node --test tests/*.mjs +``` + +- The unit suite must validate the exact nine-node surface, the Director/experimental category split, both bundled SplatStage schemas, and retained reframe helpers. +- The readiness/schema smoke must finish with `Prompt Builder readiness/schema smoke passed`. +- The JavaScript suite must exercise the shipped `model_loader.js`. No test may import the retired `ltx_reframe_layout.js`. +- Require all three CI jobs to pass on the release commit. + +## Manual Runtime Gates + +1. Run the NVFP4 proof gate on a CUDA workstation with the real model: + + ```text + python proof_gates.py --model-path C:\ComfyUI\models\LLM\diffusiongemma-26B-A4B-it-NVFP4 + ``` + +2. Fresh-install the repository in a clean ComfyUI `0.30.1` custom-nodes folder and confirm all nine nodes appear: seven Director nodes at the stable category and two SplatStage planners only under the experimental motion-planning category. +3. Copy `examples/assets/character_motion_transfer` to `ComfyUI/input/character_motion_transfer` and load `examples/07_ltx23_character_motion_transfer.json`. +4. Load the MiniMax H3 T2VA and Ref2VA examples and confirm the Generation Gate blocks invalid or empty prompt packets. +5. Treat longer experimental SplatStage and reframe workflows as separate opt-in validation, not as evidence for the stable Director compatibility promise. + +## Files Tracked on GitHub + +- Package entry point and source: `__init__.py`, `nodes.py`, `ltx_reframe.py`, `proof_gates.py` +- Metadata and policy: `README.md`, `LICENSE`, `requirements.txt`, `pyproject.toml`, `.gitignore`, `.comfyignore`, this checklist +- Runtime contracts: `schemas/`, `web/`, `scripts/` +- Verification: `tests/`, `.github/workflows/ci.yml` +- Publishing: `.github/workflows/publish_action.yml` +- Documentation, current examples, and GitHub demo assets: `docs/`, `examples/` + +## Registry Archive + +- Keep `.github/`, `docs/`, `tests/`, caches, temporary files, and large demo video/GIF files out of the Registry archive through `.comfyignore`. +- Confirm both SplatStage schemas remain in the Registry archive because the experimental planners load them at runtime. +- Inspect the archive before publishing and confirm it contains no model weights, local paths, logs, caches, or generated outputs. +- Publish only after the GitHub release commit and version metadata agree. diff --git a/README.md b/README.md index 51a0d19..80932cb 100644 --- a/README.md +++ b/README.md @@ -4,21 +4,39 @@ # ComfyUI DiffusionGemma Prompt Builder -Five DiffusionGemma Director Assistant nodes for ComfyUI workflows that use DiffusionGemma as a multimodal prompt writer, not as the image or video sampler. +DiffusionGemma Director provides a stable five-node prompt-authoring core for ComfyUI, with two stable Director companion nodes. The package also exports two experimental SplatStage motion-planning nodes. DiffusionGemma writes and validates prompts or blueprints; it is not the downstream image or video sampler. -This release is packaged around the working LTX 2.3 character motion-transfer workflow in `examples/07_ltx23_character_motion_transfer.json`. +The primary stable workflow is the working LTX 2.3 character motion-transfer graph in `examples/07_ltx23_character_motion_transfer.json`. Additional Director examples cover native MiniMax H3 T2VA and Ref2VA paths. ## Exported Nodes -Only the nodes used by the packaged workflow are registered in ComfyUI: +Nine nodes are registered in ComfyUI. Their stable and experimental surfaces are intentionally separate. + +### Stable Director Assistant + +The five-node core path used by the primary LTX workflow is: - `DiffusionGemmaModelLoader` - loads the local Transformers/NVFP4 DiffusionGemma runtime. - `DiffusionGemmaContextHub` - combines user text, optional image/video inputs, and optional `visual_description` into one `DG_CONTEXT`. -- `DiffusionGemmaTargetProfile` - stores target generation settings such as LTX profile, audio mode, duration, aspect ratio, and negative prompt policy. +- `DiffusionGemmaTargetProfile` - stores the LTX, Ideogram4, or MiniMax H3 target plus audio, duration, aspect ratio, and negative prompt settings. - `DiffusionGemmaCoTGenerator` - asks DiffusionGemma for the final prompt packet and returns final JSON plus debug outputs. -- `DiffusionGemmaJSONSplitter` - turns the final JSON into LTX prompt, negative prompt, metadata, resolution, segment, and readiness outputs. +- `DiffusionGemmaJSONSplitter` - turns the final JSON into LTX, Ideogram4, or MiniMax H3 prompt output plus metadata, resolution, segment, and readiness outputs. -Older compatibility and utility classes remain in source where the final nodes still reuse helper behavior internally, but they are not exported as ComfyUI nodes in this package. +Two stable companions extend that core without changing it: + +- `DiffusionGemmaH3ReferenceContext` - binds an ordered H3 asset-role manifest and optional picture/video evidence into a Ref2VA-aware `DG_CONTEXT`. +- `DiffusionGemmaGenerationGate` - fails closed when validation says a prompt is not ready, preventing an invalid prompt from reaching a downstream generator. + +### Experimental Motion Planning + +These larger planners are exported for opt-in SplatStage work, but they are not part of the stable Director surface or the primary LTX workflow: + +- `DiffusionGemmaSplatStagePlanner` - writes and validates the strict SplatStage show blueprint. +- `DiffusionGemmaSplatStageEfficientPlanner` - writes and validates the higher-variation efficient SplatStage blueprint. + +Both are labeled `(Experimental)` and appear under `prompt/diffusiongemma/experimental-motion-planning` in ComfyUI. Their schemas ship with this repository, while executing a resulting show still requires the appropriate downstream SplatStage workflow and nodes. + +Older compatibility and utility classes remain in source where exported nodes reuse their behavior internally, but they are not registered. The `ltx_reframe.py` module and LTX reframe workflows are retained as unexported prototypes; the retired `ltx_reframe_layout.js` frontend is not part of this release. ## Packaged Workflow @@ -74,6 +92,89 @@ DiffusionGemma JSON Splitter.ready_for_generation -> generation gate or manual c Use `image_identity_video_control` on the Context Hub when the reference image should define the output subject while the video provides motion, pose, camera, depth, Canny/edge layout, blocking, and timing. +For MiniMax H3, select `minimax_h3` on the same Target Profile node and wire that same `target_profile_config` to both the CoT Generator and JSON Splitter: + +```text +DiffusionGemma Target Profile.target_profile_config -> DiffusionGemma CoT Generator.target_profile_config +DiffusionGemma Target Profile.target_profile_config -> DiffusionGemma JSON Splitter.target_profile_config +DiffusionGemma CoT Generator.final_json -> DiffusionGemma JSON Splitter.final_json +DiffusionGemma JSON Splitter.minimax_h3_prompt -> MiniMax H3 text encoder prompt +``` + +The splitter appends `minimax_h3_prompt` after its existing outputs so saved LTX and Ideogram4 workflow output indices remain unchanged. + +## MiniMax H3 T2VA Prompt Profile + +With `minimax_h3_mode=t2va`, the MiniMax H3 profile turns a normal creative brief into H3's canonical text-to-video/audio representation. T2VA does not invent keyframe or full-reference labels when Director has not been given explicit reference roles. Its output preserves section breaks and uses exactly these three fields in order: + +```text +integrated_multimodal_description: [Shot 1] ... + +overall_soundscape: ... + +non_diegetic_music: ... +``` + +The first shot has no timestamp. Later cuts use increasing instants such as `[Shot 2] At 00:03.500, the camera cuts to...`, all inside the selected duration. Hard cuts are the default; cross-dissolves, fades, and wipes are used only when the brief requests them. Synchronized visible action, speech, and scene sounds stay in `integrated_multimodal_description`; ambient sound and Foley belong in `overall_soundscape`; audience-only score belongs in `non_diegetic_music`, where instrumentation, tempo/rhythm, and dynamic development are described concretely instead of as an abstract mood. + +`target_duration_seconds` remains user controlled and has no H3-specific upper clamp. MiniMax documents 4-15 seconds as the standard H3 generation range, so that range is useful quality guidance, but Director does not clamp longer requests. A value such as 20 seconds is retained in the prompt contract; the connected H3 workflow's frame count, sampler settings, VRAM, and generation time determine whether it can be rendered successfully. `0` uses source-media duration when available, then an explicit duration written in the creative brief, and otherwise a 5-second H3 planning default. + +H3 does not expose a separate negative-conditioning prompt through its local text encoder. When MiniMax H3 is selected, Director keeps `negative_prompt` empty and expresses necessary exclusions concisely inside `integrated_multimodal_description` for T2VA or `detailed_description` for Ref2VA instead of inventing an unsupported negative channel. + +See [`examples/09_minimax_h3_t2va_director.json`](examples/09_minimax_h3_t2va_director.json) for a native MiniMax-H3 T2VA workflow. It shares one user-editable duration between Director and H3, routes the dedicated H3 prompt plus computed resolution into the generator, previews both the generated prompt and any validation block reasons, unloads DiffusionGemma before sampling, and is preset to the proven 20-second, 25-step configuration. + +## MiniMax H3 Ref2VA Prompt Profile + +Set `target_profile=minimax_h3`, `minimax_h3_mode=ref2va`, and build the context with `DiffusionGemmaH3ReferenceContext`. Its manifest is the asset-role contract for the native reference sockets, for example: + +```text +: protagonist identity, face, hair, body, wardrobe, and distinguishing traits; preserve fully. +: environment, palette, lighting, and 2D animation style; do not transfer pictured subject identity. +