alisson-anjos-ComfyUI-BFSNodes/MINIMAX_H3_GUIDES.md

MiniMax-H3 downscaled latent guides

BFSNodes 1.46.0 adds two nodes for H3 upscale LoRAs trained in alisson-anjos/ai-toolkit with target_crop_v2 / target_grid_stride_v1 geometry:

• MiniMax-H3 Downscaled Latent Guide (BFS): preprocess and encode an image or video guide at a lower resolution, and return a model with compatible spatial conditioning coordinates.
• MiniMax-H3 Guide Target — Image / Video (BFS): make the joint target latent at the intended output dimensions, with one frame for an image or 17*n+5 frames for video. It does not resize the target canvas implicitly.

The native Add Guide node enlarges guides to the output canvas. This node keeps the smaller encoded guide and selects every factor-th origin from the target patch grid. Guide time stays aligned to the target. It modifies a cloned model through its UNet wrapper; no global PackedLayout replacement is installed, and other models/conditionings continue to use their native paths.

Install / update

cd ComfyUI/custom_nodes/ComfyUI-BFSNodes
git pull --ff-only

Restart ComfyUI. Use a ComfyUI build with native H3 support; the layout tests use the exact implementation in commit 8cfe5e1. Incompatible H3 layout/VAE APIs produce explicit errors.

Image workflow

1. Load the H3 Ref2V model and the trained upscale LoRA with your normal LoRA loader. Load the official complete H3 video VAE with its encoder and the H3 Qwen3-VL text encoder (CLIPLoader type minimax).
2. Encode the caption without passing the guide image into the VLM. Start with the training prompt: h3upscale, increase resolution and restore fine details while preserving the original content and visual style.
3. Create a Guide Target with width 1024, height 768, frames 1.
4. Connect the model after LoRA loading, positive conditioning, target latent, VAE, and source image to Downscaled Latent Guide. Set downscale_factor=4, frame_idx=0. Select the same lora_name to validate training metadata; that selector does not load the LoRA a second time.
5. Connect both returned MODEL and positive CONDITIONING to the guider / sampler. Use the returned model for sigma scheduling too. The sampler's latent_image stays the target latent from step 3, not a guide latent.
6. Decode with the H3 VAE. The third output is the actual reduced pixel guide preview; at this canvas it is 256 x 192.

An API-format image workflow template is included. Replace YOUR_H3_UPSCALE_LORA.safetensors, YOUR_INPUT_IMAGE.png, and model filenames with files installed in your ComfyUI. The template uses seed 42, Euler, 28 steps, full denoise, and BasicGuider. It can be submitted as the prompt field of a ComfyUI /prompt API request. It is an API graph, not a saved frontend canvas workflow.

Video workflow

Create a target with frames=73 (approximately 3 seconds at 24 fps) and supply an IMAGE batch of exactly 73 source video frames resampled to 24 fps, in chronological order. The video VAE produces 22 latent-time slices. Other valid lengths include 5, 22, 39, 56, 90, 107 and 124; a one-frame batch is an image. The node rejects invalid lengths instead of silently truncating or stretching motion. It does not perform video decoding or FPS conversion itself.

Keep the same target canvas / guide factor as training. Increasing target resolution or frame count increases target activation memory even though guide spatial tokens are reduced. Start with a known-fitting canvas on the target GPU. To save video, connect decoded frames to your video encoder/combiner at 24 fps.

An optional audio soundtrack plus audio_vae can add aligned guide audio. Supply audio starting at the guide clip's first frame. It is cropped to the clip and available target duration, and is never sent to the text encoder. Omit it for visual-only inference. Native reference conditioning and native same-resolution guides can be chained alongside this node; their row positions are retained.

Geometry and encoding

• Each target pixel axis must be a multiple of 32 * downscale_factor.
• The source is fitted to the target canvas with the fork's centered bicubic crop, then downscaled with Lanczos before VAE encoding. This also applies when the supplied source is already small: inspect guide_preview.
• H3 VAE spatial compression is 16. At output 1024 x 768, factor 4 yields guide pixels 256 x 192, guide latents 16 x 12, and target latents 64 x 48.
• The posterior is sampled with an isolated CPU generator (seed 42) and rounded through FP16 before latent normalization, matching the fork's image and video sampling paths. The native VAE mean-only encoding is not used.
• An image has one guide latent-time slice; 73 video frames have 22. Video time coordinates use H3's (1,4,4,4,4) temporal pattern, not a uniform stride.
• The corrected packed layout keeps target/reference/audio coordinates and reindexes condition positions, frozen/update masks and segment boundaries. Downscaled guides receive the target spatial grid at the training stride; coordinates from a separately normalized small grid are not multiplied.

Validation and limits

CPU tests cover the audited native layout, factors 1/2/4, exact crop/resize, posterior sampling, mixed/native guides, references, audio row preservation, 73-frame counts, metadata checks, clone isolation and wrapper composition. Golden position fixtures come from the actual AI Toolkit packing implementation. Run python -m unittest discover -s tests -v.

A full pretrained ComfyUI GPU sampling run has not been validated as part of this change. Matching preprocessing and coordinates establishes the conditioning contract; it does not establish the trained LoRA's visual quality. Compare LoRA strength 0 and 1 on held-out inputs before evaluating upscale fidelity.

Experimental reference layouts and source phase (1.47.0)

Optional inputs on Downscaled Guide expose guide_rope_layout (overlap/sidecar), reference_rope_layout (native/overlap/sidecar), reference_source_phase, reference_phase_scale, reference_sidecar_margin, and source_id. Defaults preserve 1.46.0 behavior. H3 sidecar margins use normalized RoPE units, not pixels. Select your LoRA to check these values against minimax_h3_reference_rope metadata. Legacy checkpoints imply overlap/native/phase-off. Changing only inference geometry on a legacy LoRA does not establish that the model can use it.

MiniMax-H3 Identity Reference + RoPE (BFS) adds a separate single-image latent reference without passing it to the VLM. Supply the same target latent and full H3 VAE. Chain both model and conditioning through Guide and Identity Reference, then connect both final outputs to the guider/sampler (and model to the scheduler). Use guide source_id=1, identity source_id=2 for training control channels 1 and 2. The identity reference keeps its aspect and is resized down only to the trainer's /32 reference-image bucket; the guide factor does not resize the portrait. The reference preview shows the actual pixels used.

This node requires a checkpoint trained with a matching identity reference role and control_latent_only: true for caption-only conditioning. The LoRA selector checks that mode too. It does not load LoRA weights. Adding a portrait to a LoRA trained only with a low-resolution guide is not a validated identity-preservation recipe. Keep caption/source conventions consistent during training.

Source phase adds the phase source_id * phase_scale * 10000**(-d/48) to H3's rotary angles. Text and targets remain source zero. The scoped ModelPatcher object patch composes this phase only when this node's marked conditioning is active; unmarked conditioning and other models keep their original rotary path.

For a first experiment use overlap plus phase and compare to the phase-off baseline. Sidecar and phase have CPU geometry/rotary parity tests, including mixed image, video and audio, but no full pretrained H3 quality validation yet. They distinguish sources; they do not guarantee the same face. See the training configuration guide.

Video context in CLIP with aligned guide latents

Select aligned guide + CLIP video context in BFS Shot H3 Conditioning (or its Simple variant). Qwen receives only the CURRENT SHOT as <Video 1> at 2 fps with real timestamps, using the same shot interval passed to Add Guide. It does not receive the full source video on every shot. The original shot frames provide visual context even when source blur modifies the aligned latent guide. Images retain their <Picture 1> through <Picture N> order and native image reference latents. Planner and the new standalone node allow up to 100 pictures (the Comfy autogrow capacity); available model context and memory can limit this. The original H3 node keeps its own unchanged input limits.

The shot itself enters the diffusion model only as aligned minimax_keyframes through Add Guide; the new mode does not create a native minimax_refs video block for it. Its Qwen embeddings still travel in positive. Explicitly enabling the separate native masked-region video reference adds that as <Video 2>.

For a native workflow, use MiniMax H3 Reference to Video — Guide 0 CLIP Context (BFS) in place of MiniMax H3 Reference to Video:

1. Connect your source guide to guide_video_0, and pictures to ref_images. 0 identifies the input's special role; its prompt token is <Video 1> because H3 numbers video tokens from 1. Only this guide is CLIP-only.
2. Optional ref_videos remain ACTUAL native video references, receiving both Qwen encoding and native reference latents. With guide 0 connected they are <Video 2>, <Video 3>, <Video 4>; without it native numbering starts at 1. Up to 3 actual reference videos are available, as in the original node.
3. Connect the returned positive and target latent to MiniMax H3 Add Guide, with the SAME guide frames, VAE and frame_idx = 0.
4. Feed Add Guide's positive and the target latent to your sampler.

Video inputs use the native 24-fps convention, are trimmed to the requested target length, and round down to the native 17*k+5 frame grid. There is no separate 15s hard cap in this helper; practical context/memory and checkpoint training limits still apply. Native reference soundtracks remain paired to their ref_video_N input, not to guide 0. Supply guide audio through Add Guide. The helper resizes only the guide frames sampled for Qwen and never VAE-encodes that guide as a native video reference. Images, actual reference videos and audio retain the native implementation. Target latent starts empty; Add Guide supplies aligned motion. The BFS mode performs the context and aligned-guide steps internally; Shot Join only assembles/composites the generated outputs.

Example: <Subject 1> comes from <Picture 3> and follows the blond performer in the yellow shirt in <Video 1>. <Subject 2> comes from <Picture 1> and <Picture 2> and follows the orange-haired performer in the white shirt in <Video 1>. Keep these assignments through crossings and occlusions. A picture need not mean a separate person; multiple images can define one identity, and there can be N assigned subjects/reference assets within the available context.

Existing modes and defaults are unchanged. Qwen encoding cost and text context length increase. Checkpoints trained with guide_latent_only did not see the source video in Qwen: this is an experimental inference option, not a demonstrated identity-binding improvement. GPU and visual validation remain for the user's local workflow.

Guide sampling rate and console evidence

clip_video_fps is available on both BFS conditioning variants and the Guide 0 node. Default is 2 fps; range is 0.5–24 fps, capped to the source frame rate without duplicating frames. It changes guide sampling in Qwen only, not the aligned guide, output fps, or native reference-video sampling. Try 4 or 8 fps against 2 fps with the same shot, prompt, references, seed and sampler settings. Native H3 timestamps are retained; higher sampling rates are an experiment, not a guarantee of stronger identity binding.

Console messages with [BFS H3 CLIP context] report the input/used frame count, requested/effective fps, sent RGB frames, dimensions and first/last timestamps. A second message confirms that the tokenizer actually emitted H3 video vision blocks; an incompatible CLIP fails explicitly rather than silently ignoring the video. The final message appears only after Qwen encoding returns, with elapsed time and conditioning shapes. Qwen can pad an odd sample count to pairs. These logs confirm the conditioning path ran, not visual quality.

Compare aligned guide (Add Guide) against aligned guide + CLIP video context with all other settings fixed to assess whether video context changes the result. Use a fresh sampler execution (Planner cached renders may otherwise return old results); the CLIP-context logs only appear when conditioning actually executes. Compare 2/4/8 fps separately to isolate the sampling-rate effect.