DiffusionGemma and LTX 2.3 character motion transfer demo

# ComfyUI DiffusionGemma Prompt Builder DiffusionGemma Director provides a stable five-node prompt-authoring core for ComfyUI, with two stable Director companion nodes. The package also exports two experimental SplatStage motion-planning nodes. DiffusionGemma writes and validates prompts or blueprints; it is not the downstream image or video sampler. The primary stable workflow is the working LTX 2.3 character motion-transfer graph in `examples/07_ltx23_character_motion_transfer.json`. Additional Director examples cover native MiniMax H3 T2VA and Ref2VA paths. ## Exported Nodes Nine nodes are registered in ComfyUI. Their stable and experimental surfaces are intentionally separate. ### Stable Director Assistant The five-node core path used by the primary LTX workflow is: - `DiffusionGemmaModelLoader` - loads the local Transformers/NVFP4 DiffusionGemma runtime. - `DiffusionGemmaContextHub` - combines user text, optional image/video inputs, and optional `visual_description` into one `DG_CONTEXT`. - `DiffusionGemmaTargetProfile` - stores the LTX, Ideogram4, or MiniMax H3 target plus audio, duration, aspect ratio, and negative prompt settings. - `DiffusionGemmaCoTGenerator` - asks DiffusionGemma for the final prompt packet and returns final JSON plus debug outputs. - `DiffusionGemmaJSONSplitter` - turns the final JSON into LTX, Ideogram4, or MiniMax H3 prompt output plus metadata, resolution, segment, and readiness outputs. Two stable companions extend that core without changing it: - `DiffusionGemmaH3ReferenceContext` - binds an ordered H3 asset-role manifest and optional picture/video evidence into a Ref2VA-aware `DG_CONTEXT`. - `DiffusionGemmaGenerationGate` - fails closed when validation says a prompt is not ready, preventing an invalid prompt from reaching a downstream generator. ### Experimental Motion Planning These larger planners are exported for opt-in SplatStage work, but they are not part of the stable Director surface or the primary LTX workflow: - `DiffusionGemmaSplatStagePlanner` - writes and validates the strict SplatStage show blueprint. - `DiffusionGemmaSplatStageEfficientPlanner` - writes and validates the higher-variation efficient SplatStage blueprint. Both are labeled `(Experimental)` and appear under `prompt/diffusiongemma/experimental-motion-planning` in ComfyUI. Their schemas ship with this repository, while executing a resulting show still requires the appropriate downstream SplatStage workflow and nodes. Older compatibility and utility classes remain in source where exported nodes reuse their behavior internally, but they are not registered. The `ltx_reframe.py` module and LTX reframe workflows are retained as unexported prototypes; the retired `ltx_reframe_layout.js` frontend is not part of this release. ## Packaged Workflow The release workflow is: ```text examples/07_ltx23_character_motion_transfer.json ``` It uses a reference image for character identity and a source video for pose, motion, camera choreography, depth, Canny/edge structure, composition, timing, and scene geometry. It includes Canny, Depth, and DWPose control branches for comparison. Bundled media lives in: ```text examples/assets/character_motion_transfer ``` Before running the workflow, copy that folder to: ```text ComfyUI/input/character_motion_transfer ``` The workflow expects these input paths: ```text character_motion_transfer/character_reference.png character_motion_transfer/motion_control_video.mp4 ``` The demo output is also included: ```text examples/assets/character_motion_transfer/diffusiongemma_ltx_character_demo.mp4 examples/assets/character_motion_transfer/diffusiongemma_ltx_character_demo.gif ``` ## Recommended Wiring For custom workflows, the core DiffusionGemma path is: ```text Text String -> DiffusionGemma Context Hub Image or Video -> DiffusionGemma Context Hub DiffusionGemma Target Profile -> DiffusionGemma CoT Generator DiffusionGemma Model Loader (Advanced) -> DiffusionGemma CoT Generator DiffusionGemma Context Hub -> DiffusionGemma CoT Generator DiffusionGemma CoT Generator.final_json -> DiffusionGemma JSON Splitter DiffusionGemma JSON Splitter.ltx_prompt -> LTX positive prompt input DiffusionGemma JSON Splitter.negative_prompt -> LTX negative prompt input DiffusionGemma JSON Splitter.ready_for_generation -> generation gate or manual check ``` Use `image_identity_video_control` on the Context Hub when the reference image should define the output subject while the video provides motion, pose, camera, depth, Canny/edge layout, blocking, and timing. For MiniMax H3, select `minimax_h3` on the same Target Profile node and wire that same `target_profile_config` to both the CoT Generator and JSON Splitter: ```text DiffusionGemma Target Profile.target_profile_config -> DiffusionGemma CoT Generator.target_profile_config DiffusionGemma Target Profile.target_profile_config -> DiffusionGemma JSON Splitter.target_profile_config DiffusionGemma CoT Generator.final_json -> DiffusionGemma JSON Splitter.final_json DiffusionGemma JSON Splitter.minimax_h3_prompt -> MiniMax H3 text encoder prompt ``` The splitter appends `minimax_h3_prompt` after its existing outputs so saved LTX and Ideogram4 workflow output indices remain unchanged. ## MiniMax H3 T2VA Prompt Profile With `minimax_h3_mode=t2va`, the MiniMax H3 profile turns a normal creative brief into H3's canonical text-to-video/audio representation. T2VA does not invent keyframe or full-reference labels when Director has not been given explicit reference roles. Its output preserves section breaks and uses exactly these three fields in order: ```text integrated_multimodal_description: [Shot 1] ... overall_soundscape: ... non_diegetic_music: ... ``` The first shot has no timestamp. Later cuts use increasing instants such as `[Shot 2] At 00:03.500, the camera cuts to...`, all inside the selected duration. Hard cuts are the default; cross-dissolves, fades, and wipes are used only when the brief requests them. Synchronized visible action, speech, and scene sounds stay in `integrated_multimodal_description`; ambient sound and Foley belong in `overall_soundscape`; audience-only score belongs in `non_diegetic_music`, where instrumentation, tempo/rhythm, and dynamic development are described concretely instead of as an abstract mood. `target_duration_seconds` remains user controlled and has no H3-specific upper clamp. MiniMax documents 4-15 seconds as the standard H3 generation range, so that range is useful quality guidance, but Director does not clamp longer requests. A value such as 20 seconds is retained in the prompt contract; the connected H3 workflow's frame count, sampler settings, VRAM, and generation time determine whether it can be rendered successfully. `0` uses source-media duration when available, then an explicit duration written in the creative brief, and otherwise a 5-second H3 planning default. H3 does not expose a separate negative-conditioning prompt through its local text encoder. When MiniMax H3 is selected, Director keeps `negative_prompt` empty and expresses necessary exclusions concisely inside `integrated_multimodal_description` for T2VA or `detailed_description` for Ref2VA instead of inventing an unsupported negative channel. See [`examples/09_minimax_h3_t2va_director.json`](examples/09_minimax_h3_t2va_director.json) for a native MiniMax-H3 T2VA workflow. It shares one user-editable duration between Director and H3, routes the dedicated H3 prompt plus computed resolution into the generator, previews both the generated prompt and any validation block reasons, unloads DiffusionGemma before sampling, and is preset to the proven 20-second, 25-step configuration. ## MiniMax H3 Ref2VA Prompt Profile Set `target_profile=minimax_h3`, `minimax_h3_mode=ref2va`, and build the context with `DiffusionGemmaH3ReferenceContext`. Its manifest is the asset-role contract for the native reference sockets, for example: ```text : protagonist identity, face, hair, body, wardrobe, and distinguishing traits; preserve fully. : environment, palette, lighting, and 2D animation style; do not transfer pictured subject identity.