DiffusionGemma Director provides a stable five-node prompt-authoring core for ComfyUI, with two stable Director companion nodes. The package also exports two experimental SplatStage motion-planning nodes. DiffusionGemma writes and validates prompts or blueprints; it is not the downstream image or video sampler.

The primary stable workflow is the working LTX 2.3 character motion-transfer graph in examples/07_ltx23_character_motion_transfer.json. Additional Director examples cover native MiniMax H3 T2VA and Ref2VA paths.

Exported Nodes

Nine nodes are registered in ComfyUI. Their stable and experimental surfaces are intentionally separate.

Stable Director Assistant

The five-node core path used by the primary LTX workflow is:

DiffusionGemmaModelLoader - loads the local Transformers/NVFP4 DiffusionGemma runtime.
DiffusionGemmaContextHub - combines user text, optional image/video inputs, and optional visual_description into one DG_CONTEXT.
DiffusionGemmaTargetProfile - stores the LTX, Ideogram4, or MiniMax H3 target plus audio, duration, aspect ratio, and negative prompt settings.
DiffusionGemmaCoTGenerator - asks DiffusionGemma for the final prompt packet and returns final JSON plus debug outputs.
DiffusionGemmaJSONSplitter - turns the final JSON into LTX, Ideogram4, or MiniMax H3 prompt output plus metadata, resolution, segment, and readiness outputs.

Two stable companions extend that core without changing it:

DiffusionGemmaH3ReferenceContext - binds an ordered H3 asset-role manifest and optional picture/video evidence into a Ref2VA-aware DG_CONTEXT.
DiffusionGemmaGenerationGate - fails closed when validation says a prompt is not ready, preventing an invalid prompt from reaching a downstream generator.

Experimental Motion Planning

These larger planners are exported for opt-in SplatStage work, but they are not part of the stable Director surface or the primary LTX workflow:

DiffusionGemmaSplatStagePlanner - writes and validates the strict SplatStage show blueprint.
DiffusionGemmaSplatStageEfficientPlanner - writes and validates the higher-variation efficient SplatStage blueprint.

Both are labeled (Experimental) and appear under prompt/diffusiongemma/experimental-motion-planning in ComfyUI. Their schemas ship with this repository, while executing a resulting show still requires the appropriate downstream SplatStage workflow and nodes.

Older compatibility and utility classes remain in source where exported nodes reuse their behavior internally, but they are not registered. The ltx_reframe.py module and LTX reframe workflows are retained as unexported prototypes; the retired ltx_reframe_layout.js frontend is not part of this release.

Packaged Workflow

The release workflow is:

examples/07_ltx23_character_motion_transfer.json

It uses a reference image for character identity and a source video for pose, motion, camera choreography, depth, Canny/edge structure, composition, timing, and scene geometry. It includes Canny, Depth, and DWPose control branches for comparison.

Bundled media lives in:

examples/assets/character_motion_transfer

Before running the workflow, copy that folder to:

ComfyUI/input/character_motion_transfer

The workflow expects these input paths:

character_motion_transfer/character_reference.png
character_motion_transfer/motion_control_video.mp4

The demo output is also included:

examples/assets/character_motion_transfer/diffusiongemma_ltx_character_demo.mp4
examples/assets/character_motion_transfer/diffusiongemma_ltx_character_demo.gif

Recommended Wiring

For custom workflows, the core DiffusionGemma path is:

Text String -> DiffusionGemma Context Hub
Image or Video -> DiffusionGemma Context Hub
DiffusionGemma Target Profile -> DiffusionGemma CoT Generator
DiffusionGemma Model Loader (Advanced) -> DiffusionGemma CoT Generator
DiffusionGemma Context Hub -> DiffusionGemma CoT Generator
DiffusionGemma CoT Generator.final_json -> DiffusionGemma JSON Splitter
DiffusionGemma JSON Splitter.ltx_prompt -> LTX positive prompt input
DiffusionGemma JSON Splitter.negative_prompt -> LTX negative prompt input
DiffusionGemma JSON Splitter.ready_for_generation -> generation gate or manual check

Use image_identity_video_control on the Context Hub when the reference image should define the output subject while the video provides motion, pose, camera, depth, Canny/edge layout, blocking, and timing.

For MiniMax H3, select minimax_h3 on the same Target Profile node and wire that same target_profile_config to both the CoT Generator and JSON Splitter:

DiffusionGemma Target Profile.target_profile_config -> DiffusionGemma CoT Generator.target_profile_config
DiffusionGemma Target Profile.target_profile_config -> DiffusionGemma JSON Splitter.target_profile_config
DiffusionGemma CoT Generator.final_json -> DiffusionGemma JSON Splitter.final_json
DiffusionGemma JSON Splitter.minimax_h3_prompt -> MiniMax H3 text encoder prompt

The splitter appends minimax_h3_prompt after its existing outputs so saved LTX and Ideogram4 workflow output indices remain unchanged.

MiniMax H3 T2VA Prompt Profile

With minimax_h3_mode=t2va, the MiniMax H3 profile turns a normal creative brief into H3's canonical text-to-video/audio representation. T2VA does not invent keyframe or full-reference labels when Director has not been given explicit reference roles. Its output preserves section breaks and uses exactly these three fields in order:

integrated_multimodal_description: [Shot 1] ...

overall_soundscape: ...

non_diegetic_music: ...

The first shot has no timestamp. Later cuts use increasing instants such as [Shot 2] At 00:03.500, the camera cuts to..., all inside the selected duration. Hard cuts are the default; cross-dissolves, fades, and wipes are used only when the brief requests them. Synchronized visible action, speech, and scene sounds stay in integrated_multimodal_description; ambient sound and Foley belong in overall_soundscape; audience-only score belongs in non_diegetic_music, where instrumentation, tempo/rhythm, and dynamic development are described concretely instead of as an abstract mood.

target_duration_seconds remains user controlled and has no H3-specific upper clamp. MiniMax documents 4-15 seconds as the standard H3 generation range, so that range is useful quality guidance, but Director does not clamp longer requests. A value such as 20 seconds is retained in the prompt contract; the connected H3 workflow's frame count, sampler settings, VRAM, and generation time determine whether it can be rendered successfully. 0 uses source-media duration when available, then an explicit duration written in the creative brief, and otherwise a 5-second H3 planning default.

H3 does not expose a separate negative-conditioning prompt through its local text encoder. When MiniMax H3 is selected, Director keeps negative_prompt empty and expresses necessary exclusions concisely inside integrated_multimodal_description for T2VA or detailed_description for Ref2VA instead of inventing an unsupported negative channel.

See examples/09_minimax_h3_t2va_director.json for a native MiniMax-H3 T2VA workflow. It shares one user-editable duration between Director and H3, routes the dedicated H3 prompt plus computed resolution into the generator, previews both the generated prompt and any validation block reasons, unloads DiffusionGemma before sampling, and is preset to the proven 20-second, 25-step configuration.

MiniMax H3 Ref2VA Prompt Profile

Set target_profile=minimax_h3, minimax_h3_mode=ref2va, and build the context with DiffusionGemmaH3ReferenceContext. Its manifest is the asset-role contract for the native reference sockets, for example:

<Picture 1>: protagonist identity, face, hair, body, wardrobe, and distinguishing traits; preserve fully.
<Picture 2>: environment, palette, lighting, and 2D animation style; do not transfer pictured subject identity.
<Audio 1>: synchronized soundtrack timing and impact-texture reference for <Video 1>; do not copy speech.
<Video 1>: running motion, camera path, hard-cut rhythm, and landing mechanics; do not transfer performer identity.

Tags are one-based and numbered independently by type. Keep the manifest aligned with native H3: all pictures are presented first; a paired video soundtrack receives an Audio label immediately before its Video label; standalone audio follows the videos. Director supports the native local limits of nine pictures, three videos, three paired soundtracks, and three standalone audio inputs. It deliberately blocks an audio-only manifest because its Ref2VA compiler requires at least one visible reference source.

Ref2VA output follows MiniMax's six-section Context-IR contract exactly:

subject_definitions:
...

summary:
...

retention_analysis:
...

detailed_description:
...

overall_soundscape:
...

non_diegetic_music:
...

<Subject N> means reusable visible content abstracted from one or more assets; it is not an attachment number. Pictures used only for identity, environment, costume, or style are cited inside Subject definitions. A standalone Picture definition is reserved for a concrete composition/storyboard anchor. In the local native Ref2VA path, an identity/style picture is not a guaranteed first- or last-frame pixel lock; use the separate FL2VA path when that is the intended control.

The compiler requires official summary task types and retention markers, exact declared asset tags, consecutive Subject labels, a medium/style opening before [Shot 1], a detailed chronological timeline, camera direction, reference use at the relevant moments, audio-layer fidelity, and a resolved final state. Invalid, missing, duplicate, or undeclared references block the splitter output instead of reaching H3.

See examples/10_minimax_h3_ref2va_director.json for the native two-picture sample. It wires both original pictures to MiniMaxH3ReferenceToVideo, sends an ordered analysis batch to Director, uses a full 2048-token prompt budget, and retains the user-selected 20-second, 481-frame, 25-step configuration without running a render automatically.

Official references:

Compatibility

The verified baseline is ComfyUI 0.30.1 with Python 3.11. Registry metadata declares requires-comfyui = ">=0.30.1": this is the Registry's minimum-compatible-version constraint, while 0.30.1 remains the version exercised by the compatibility and release checks. Raise that floor when an exported node begins to require a newer ComfyUI API.

Runtime Dependencies

This pack assumes a working ComfyUI install with CUDA-capable PyTorch. The Python dependencies declared in requirements.txt are:

transformers>=5.12.1
accelerate>=1.14.0
comfy-kitchen>=0.2.10
jsonschema>=4.26.0
safetensors>=0.7.0
tqdm>=4.67.0

The NVFP4 bridge also depends on ComfyUI exposing comfy.quant_ops and comfy.ops. On this workstation, comfy-kitchen==0.2.10 is installed and the proof gate passes.

The local bridge does not require bundling a custom tensor file. It converts the ModelOpt NVFP4 tensors from the Hugging Face checkpoint at runtime: ModelOpt FP4 nibble order and weight_scale layout are adapted into Comfy's packed NVFP4 expert matmul layout. Keep the model weights as a normal Hugging Face repo folder and ship only the node source code.

Models Used By The Example

DiffusionGemma prompt model:

nvidia/diffusiongemma-26B-A4B-it-NVFP4

Place the whole repo folder at:

ComfyUI/models/LLM/diffusiongemma-26B-A4B-it-NVFP4

LTX 2.3 diffusion model:

Kijai/LTX2.3_comfy - ltx-2.3-22b-distilled_transformer_only_fp8_input_scaled_v3.safetensors

Place in:

ComfyUI/models/unet

LTX IC-LoRAs:

Place in:

ComfyUI/models/loras

VAEs:

Place in:

ComfyUI/models/vae

Text encoders:

Place in:

ComfyUI/models/text_encoders

Latent upscaler:

Lightricks/LTX-2.3 - ltx-2.3-spatial-upscaler-x2-1.1.safetensors

Place in:

ComfyUI/models/latent_upscale_models

Custom Node Dependencies For The Example

Install missing packs through ComfyUI Manager when possible:

ComfyUI-LTXVideo
comfyui-videohelpersuite
comfyui-kjnodes
comfyui_controlnet_aux
comfyui_essentials
comfyui-easy-use
RES4LYF

Core ComfyUI supplies several additional LTX nodes used by the workflow, including video loaders, LTX preprocessing/conditioning, samplers, latent/audio/video split and concat nodes, and latent upscaler loading.

Proof Gate

The pack imports without loading DiffusionGemma. Before using the real NVFP4 backend, run:

C:\ComfyUI\.venv\Scripts\python.exe custom_nodes\ComfyUI-DiffusionGemmaPromptBuilder\proof_gates.py --model-path C:\ComfyUI\models\LLM\diffusiongemma-26B-A4B-it-NVFP4

A good NVFP4 result reports:

AutoProcessor resolves.
DiffusionGemmaForBlockDiffusion imports from Transformers.
CUDA is available.
comfy_nvfp4_bridge_supported is true.
The NVFP4 bridge smoke test passes.

Packaging Notes

pyproject.toml is included for GitHub and Comfy Registry publishing. The immutable Registry publisher id is lowercase exportanything (display name exportAnything); keep that id and casing unchanged. Increment project.version for every Registry release.

The current GitHub repository URL in metadata is:

https://github.com/exportAnything/ComfyUI-DiffusionGemmaPromptBuilder

License

MIT License. See LICENSE.