exportAnything-ComfyUI-DiffusionGemmaPromptBuilder/README.md

<p align="center">
  <img src="examples/assets/character_motion_transfer/diffusiongemma_ltx_character_demo.gif" alt="DiffusionGemma and LTX 2.3 character motion transfer demo" width="720">
</p>

# ComfyUI DiffusionGemma Prompt Builder

DiffusionGemma Director provides a stable five-node prompt-authoring core for ComfyUI, with two stable Director companion nodes. The package also exports two experimental SplatStage motion-planning nodes. DiffusionGemma writes and validates prompts or blueprints; it is not the downstream image or video sampler.

The primary stable workflow is the working LTX 2.3 character motion-transfer graph in `examples/07_ltx23_character_motion_transfer.json`. Additional Director examples cover native MiniMax H3 T2VA and Ref2VA paths.

## Exported Nodes

Nine nodes are registered in ComfyUI. Their stable and experimental surfaces are intentionally separate.

### Stable Director Assistant

The five-node core path used by the primary LTX workflow is:

- `DiffusionGemmaModelLoader` - loads the local Transformers/NVFP4 DiffusionGemma runtime.
- `DiffusionGemmaContextHub` - combines user text, optional image/video inputs, and optional `visual_description` into one `DG_CONTEXT`.
- `DiffusionGemmaTargetProfile` - stores the LTX, Ideogram4, or MiniMax H3 target plus audio, duration, aspect ratio, and negative prompt settings.
- `DiffusionGemmaCoTGenerator` - asks DiffusionGemma for the final prompt packet and returns final JSON plus debug outputs.
- `DiffusionGemmaJSONSplitter` - turns the final JSON into LTX, Ideogram4, or MiniMax H3 prompt output plus metadata, resolution, segment, and readiness outputs.

Two stable companions extend that core without changing it:

- `DiffusionGemmaH3ReferenceContext` - binds an ordered H3 asset-role manifest and optional picture/video evidence into a Ref2VA-aware `DG_CONTEXT`.
- `DiffusionGemmaGenerationGate` - fails closed when validation says a prompt is not ready, preventing an invalid prompt from reaching a downstream generator.

### Experimental Motion Planning

These larger planners are exported for opt-in SplatStage work, but they are not part of the stable Director surface or the primary LTX workflow:

- `DiffusionGemmaSplatStagePlanner` - writes and validates the strict SplatStage show blueprint.
- `DiffusionGemmaSplatStageEfficientPlanner` - writes and validates the higher-variation efficient SplatStage blueprint.

Both are labeled `(Experimental)` and appear under `prompt/diffusiongemma/experimental-motion-planning` in ComfyUI. Their schemas ship with this repository, while executing a resulting show still requires the appropriate downstream SplatStage workflow and nodes.

Older compatibility and utility classes remain in source where exported nodes reuse their behavior internally, but they are not registered. The `ltx_reframe.py` module and LTX reframe workflows are retained as unexported prototypes; the retired `ltx_reframe_layout.js` frontend is not part of this release.

## Packaged Workflow

The release workflow is:

```text
examples/07_ltx23_character_motion_transfer.json
```

It uses a reference image for character identity and a source video for pose, motion, camera choreography, depth, Canny/edge structure, composition, timing, and scene geometry. It includes Canny, Depth, and DWPose control branches for comparison.

Bundled media lives in:

```text
examples/assets/character_motion_transfer
```

Before running the workflow, copy that folder to:

```text
ComfyUI/input/character_motion_transfer
```

The workflow expects these input paths:

```text
character_motion_transfer/character_reference.png
character_motion_transfer/motion_control_video.mp4
```

The demo output is also included:

```text
examples/assets/character_motion_transfer/diffusiongemma_ltx_character_demo.mp4
examples/assets/character_motion_transfer/diffusiongemma_ltx_character_demo.gif
```

## Recommended Wiring

For custom workflows, the core DiffusionGemma path is:

```text
Text String -> DiffusionGemma Context Hub
Image or Video -> DiffusionGemma Context Hub
DiffusionGemma Target Profile -> DiffusionGemma CoT Generator
DiffusionGemma Model Loader (Advanced) -> DiffusionGemma CoT Generator
DiffusionGemma Context Hub -> DiffusionGemma CoT Generator
DiffusionGemma CoT Generator.final_json -> DiffusionGemma JSON Splitter
DiffusionGemma JSON Splitter.ltx_prompt -> LTX positive prompt input
DiffusionGemma JSON Splitter.negative_prompt -> LTX negative prompt input
DiffusionGemma JSON Splitter.ready_for_generation -> generation gate or manual check
```

Use `image_identity_video_control` on the Context Hub when the reference image should define the output subject while the video provides motion, pose, camera, depth, Canny/edge layout, blocking, and timing.

For MiniMax H3, select `minimax_h3` on the same Target Profile node and wire that same `target_profile_config` to both the CoT Generator and JSON Splitter:

```text
DiffusionGemma Target Profile.target_profile_config -> DiffusionGemma CoT Generator.target_profile_config
DiffusionGemma Target Profile.target_profile_config -> DiffusionGemma JSON Splitter.target_profile_config
DiffusionGemma CoT Generator.final_json -> DiffusionGemma JSON Splitter.final_json
DiffusionGemma JSON Splitter.minimax_h3_prompt -> MiniMax H3 text encoder prompt
```

The splitter appends `minimax_h3_prompt` after its existing outputs so saved LTX and Ideogram4 workflow output indices remain unchanged.

## MiniMax H3 T2VA Prompt Profile

With `minimax_h3_mode=t2va`, the MiniMax H3 profile turns a normal creative brief into H3's canonical text-to-video/audio representation. T2VA does not invent keyframe or full-reference labels when Director has not been given explicit reference roles. Its output preserves section breaks and uses exactly these three fields in order:

```text
integrated_multimodal_description: [Shot 1] ...

overall_soundscape: ...

non_diegetic_music: ...
```

The first shot has no timestamp. Later cuts use increasing instants such as `[Shot 2] At 00:03.500, the camera cuts to...`, all inside the selected duration. Hard cuts are the default; cross-dissolves, fades, and wipes are used only when the brief requests them. Synchronized visible action, speech, and scene sounds stay in `integrated_multimodal_description`; ambient sound and Foley belong in `overall_soundscape`; audience-only score belongs in `non_diegetic_music`, where instrumentation, tempo/rhythm, and dynamic development are described concretely instead of as an abstract mood.

`target_duration_seconds` remains user controlled and has no H3-specific upper clamp. MiniMax documents 4-15 seconds as the standard H3 generation range, so that range is useful quality guidance, but Director does not clamp longer requests. A value such as 20 seconds is retained in the prompt contract; the connected H3 workflow's frame count, sampler settings, VRAM, and generation time determine whether it can be rendered successfully. `0` uses source-media duration when available, then an explicit duration written in the creative brief, and otherwise a 5-second H3 planning default.

H3 does not expose a separate negative-conditioning prompt through its local text encoder. When MiniMax H3 is selected, Director keeps `negative_prompt` empty and expresses necessary exclusions concisely inside `integrated_multimodal_description` for T2VA or `detailed_description` for Ref2VA instead of inventing an unsupported negative channel.

See [`examples/09_minimax_h3_t2va_director.json`](examples/09_minimax_h3_t2va_director.json) for a native MiniMax-H3 T2VA workflow. It shares one user-editable duration between Director and H3, routes the dedicated H3 prompt plus computed resolution into the generator, previews both the generated prompt and any validation block reasons, unloads DiffusionGemma before sampling, and is preset to the proven 20-second, 25-step configuration.

## MiniMax H3 Ref2VA Prompt Profile

Set `target_profile=minimax_h3`, `minimax_h3_mode=ref2va`, and build the context with `DiffusionGemmaH3ReferenceContext`. Its manifest is the asset-role contract for the native reference sockets, for example:

```text
<Picture 1>: protagonist identity, face, hair, body, wardrobe, and distinguishing traits; preserve fully.
<Picture 2>: environment, palette, lighting, and 2D animation style; do not transfer pictured subject identity.
<Audio 1>: synchronized soundtrack timing and impact-texture reference for <Video 1>; do not copy speech.
<Video 1>: running motion, camera path, hard-cut rhythm, and landing mechanics; do not transfer performer identity.
```

Tags are one-based and numbered independently by type. Keep the manifest aligned with native H3: all pictures are presented first; a paired video soundtrack receives an Audio label immediately before its Video label; standalone audio follows the videos. Director supports the native local limits of nine pictures, three videos, three paired soundtracks, and three standalone audio inputs. It deliberately blocks an audio-only manifest because its Ref2VA compiler requires at least one visible reference source.

Ref2VA output follows MiniMax's six-section Context-IR contract exactly:

```text
subject_definitions:
...

summary:
...

retention_analysis:
...

detailed_description:
...

overall_soundscape:
...

non_diegetic_music:
...
```

`<Subject N>` means reusable visible content abstracted from one or more assets; it is not an attachment number. Pictures used only for identity, environment, costume, or style are cited inside Subject definitions. A standalone Picture definition is reserved for a concrete composition/storyboard anchor. In the local native Ref2VA path, an identity/style picture is not a guaranteed first- or last-frame pixel lock; use the separate FL2VA path when that is the intended control.

The compiler requires official summary task types and retention markers, exact declared asset tags, consecutive Subject labels, a medium/style opening before `[Shot 1]`, a detailed chronological timeline, camera direction, reference use at the relevant moments, audio-layer fidelity, and a resolved final state. Invalid, missing, duplicate, or undeclared references block the splitter output instead of reaching H3.

See [`examples/10_minimax_h3_ref2va_director.json`](examples/10_minimax_h3_ref2va_director.json) for the native two-picture sample. It wires both original pictures to `MiniMaxH3ReferenceToVideo`, sends an ordered analysis batch to Director, uses a full 2048-token prompt budget, and retains the user-selected 20-second, 481-frame, 25-step configuration without running a render automatically.

Official references:

- [MiniMax H3 repository and model overview](https://github.com/MiniMax-AI/MiniMax-H3)
- [MiniMax H3 prompt-writing skill](https://github.com/MiniMax-AI/MiniMax-H3/blob/main/skills/h3-prompt-writing/SKILL.md)
- [MiniMax H3 Base prompt guide](https://github.com/MiniMax-AI/MiniMax-H3/blob/main/skills/h3-prompt-writing/references/base-en.txt)
- [MiniMax H3 full-reference prompt guide](https://github.com/MiniMax-AI/MiniMax-H3/blob/main/skills/h3-prompt-writing/references/ref-en.txt)

## Compatibility

The verified baseline is ComfyUI `0.30.1` with Python 3.11. Registry metadata declares `requires-comfyui = ">=0.30.1"`: this is the Registry's minimum-compatible-version constraint, while `0.30.1` remains the version exercised by the compatibility and release checks. Raise that floor when an exported node begins to require a newer ComfyUI API.

## Runtime Dependencies

This pack assumes a working ComfyUI install with CUDA-capable PyTorch. The Python dependencies declared in `requirements.txt` are:

- `transformers>=5.12.1`
- `accelerate>=1.14.0`
- `comfy-kitchen>=0.2.10`
- `jsonschema>=4.26.0`
- `safetensors>=0.7.0`
- `tqdm>=4.67.0`

The NVFP4 bridge also depends on ComfyUI exposing `comfy.quant_ops` and `comfy.ops`. On this workstation, `comfy-kitchen==0.2.10` is installed and the proof gate passes.

The local bridge does not require bundling a custom tensor file. It converts the ModelOpt NVFP4 tensors from the Hugging Face checkpoint at runtime: ModelOpt FP4 nibble order and `weight_scale` layout are adapted into Comfy's packed NVFP4 expert matmul layout. Keep the model weights as a normal Hugging Face repo folder and ship only the node source code.

## Models Used By The Example

DiffusionGemma prompt model:

- [nvidia/diffusiongemma-26B-A4B-it-NVFP4](https://huggingface.co/nvidia/diffusiongemma-26B-A4B-it-NVFP4)

Place the whole repo folder at:

```text
ComfyUI/models/LLM/diffusiongemma-26B-A4B-it-NVFP4
```

LTX 2.3 diffusion model:

- [Kijai/LTX2.3_comfy - ltx-2.3-22b-distilled_transformer_only_fp8_input_scaled_v3.safetensors](https://huggingface.co/Kijai/LTX2.3_comfy/blob/main/diffusion_models/ltx-2.3-22b-distilled_transformer_only_fp8_input_scaled_v3.safetensors)

Place in:

```text
ComfyUI/models/unet
```

LTX IC-LoRAs:

- [Lightricks/LTX-2.3-22b-IC-LoRA-Union-Control](https://huggingface.co/Lightricks/LTX-2.3-22b-IC-LoRA-Union-Control)
- [Lightricks/LTX-2.3-22b-IC-LoRA-Decompression](https://huggingface.co/Lightricks/LTX-2.3-22b-IC-LoRA-Decompression)

Place in:

```text
ComfyUI/models/loras
```

VAEs:

- [Kijai/LTX2.3_comfy - LTX23_video_vae_bf16.safetensors](https://huggingface.co/Kijai/LTX2.3_comfy/blob/main/vae/LTX23_video_vae_bf16.safetensors)
- [Kijai/LTX2.3_comfy - LTX23_audio_vae_bf16.safetensors](https://huggingface.co/Kijai/LTX2.3_comfy/tree/main/vae)

Place in:

```text
ComfyUI/models/vae
```

Text encoders:

- [Comfy-Org/ltx-2 - gemma_3_12B_it_fp4_mixed.safetensors](https://huggingface.co/Comfy-Org/ltx-2/blob/main/split_files/text_encoders/gemma_3_12B_it_fp4_mixed.safetensors)
- [Kijai/LTX2.3_comfy - ltx-2.3_text_projection_bf16.safetensors](https://huggingface.co/Kijai/LTX2.3_comfy/blob/main/text_encoders/ltx-2.3_text_projection_bf16.safetensors)

Place in:

```text
ComfyUI/models/text_encoders
```

Latent upscaler:

- [Lightricks/LTX-2.3 - ltx-2.3-spatial-upscaler-x2-1.1.safetensors](https://huggingface.co/Lightricks/LTX-2.3/blob/main/ltx-2.3-spatial-upscaler-x2-1.1.safetensors)

Place in:

```text
ComfyUI/models/latent_upscale_models
```

## Custom Node Dependencies For The Example

Install missing packs through ComfyUI Manager when possible:

- `ComfyUI-LTXVideo`
- `comfyui-videohelpersuite`
- `comfyui-kjnodes`
- `comfyui_controlnet_aux`
- `comfyui_essentials`
- `comfyui-easy-use`
- `RES4LYF`

Core ComfyUI supplies several additional LTX nodes used by the workflow, including video loaders, LTX preprocessing/conditioning, samplers, latent/audio/video split and concat nodes, and latent upscaler loading.

## Proof Gate

The pack imports without loading DiffusionGemma. Before using the real NVFP4 backend, run:

```powershell
C:\ComfyUI\.venv\Scripts\python.exe custom_nodes\ComfyUI-DiffusionGemmaPromptBuilder\proof_gates.py --model-path C:\ComfyUI\models\LLM\diffusiongemma-26B-A4B-it-NVFP4
```

A good NVFP4 result reports:

- `AutoProcessor` resolves.
- `DiffusionGemmaForBlockDiffusion` imports from Transformers.
- CUDA is available.
- `comfy_nvfp4_bridge_supported` is true.
- The NVFP4 bridge smoke test passes.

## Packaging Notes

`pyproject.toml` is included for GitHub and Comfy Registry publishing. The immutable Registry publisher id is lowercase `exportanything` (display name `exportAnything`); keep that id and casing unchanged. Increment `project.version` for every Registry release.

The current GitHub repository URL in metadata is:

```text
https://github.com/exportAnything/ComfyUI-DiffusionGemmaPromptBuilder
```

## License

MIT License. See `LICENSE`.