alisson-anjos-ComfyUI-BFSNodes/MINIMAX_H3_GUIDES.md

# MiniMax-H3 downscaled latent guides

BFSNodes 1.46.0 adds two nodes for H3 upscale LoRAs trained in
[alisson-anjos/ai-toolkit](https://github.com/alisson-anjos/ai-toolkit/tree/minimax-h3-latent-guides)
with `target_crop_v2` / `target_grid_stride_v1` geometry:

- **MiniMax-H3 Downscaled Latent Guide (BFS)**: preprocess and encode an image
  or video guide at a lower resolution, and return a model with compatible
  spatial conditioning coordinates.
- **MiniMax-H3 Guide Target — Image / Video (BFS)**: make the joint target
  latent at the intended output dimensions, with one frame for an image or
  `17*n+5` frames for video. It does not resize the target canvas implicitly.

The native Add Guide node enlarges guides to the output canvas. This node
keeps the smaller encoded guide and selects every factor-th origin from the
**target patch grid**. Guide time stays aligned to the target. It modifies a
cloned model through its UNet wrapper; no global PackedLayout replacement is
installed, and other models/conditionings continue to use their native paths.

## Install / update

```bash
cd ComfyUI/custom_nodes/ComfyUI-BFSNodes
git pull --ff-only
```

Restart ComfyUI. Use a ComfyUI build with native H3 support; the layout tests
use the exact implementation in commit
[`8cfe5e1`](https://github.com/Comfy-Org/ComfyUI/blob/8cfe5e1ecb97512dea8deaac15e1228d7e6feeb1/comfy_extras/nodes_minimax_h3.py#L166).
Incompatible H3 layout/VAE APIs produce explicit errors.

## Image workflow

1. Load the **H3 Ref2V** model and the trained upscale LoRA with your normal
   LoRA loader. Load the official complete H3 video VAE with its encoder and
   the H3 Qwen3-VL text encoder (`CLIPLoader` type `minimax`).
2. Encode the caption without passing the guide image into the VLM. Start
   with the training prompt:
   `h3upscale, increase resolution and restore fine details while preserving the original content and visual style.`
3. Create a **Guide Target** with width `1024`, height `768`, frames `1`.
4. Connect the model after LoRA loading, positive conditioning, target latent,
   VAE, and source image to **Downscaled Latent Guide**. Set
   `downscale_factor=4`, `frame_idx=0`. Select the same `lora_name` to validate
   training metadata; that selector does **not** load the LoRA a second time.
5. Connect **both returned MODEL and positive CONDITIONING** to the guider /
   sampler. Use the returned model for sigma scheduling too. The sampler's
   `latent_image` stays the target latent from step 3, not a guide latent.
6. Decode with the H3 VAE. The third output is the **actual reduced pixel
   guide preview**; at this canvas it is `256 x 192`.

An [API-format image workflow template](examples/minimax_h3_upscale_image_api.json)
is included. Replace `YOUR_H3_UPSCALE_LORA.safetensors`,
`YOUR_INPUT_IMAGE.png`, and model filenames with files installed in your
ComfyUI. The template uses seed 42, Euler, 28 steps, full denoise, and BasicGuider.
It can be submitted as the `prompt` field of a ComfyUI `/prompt` API request.
It is an API graph, not a saved frontend canvas workflow.

## Video workflow

Create a target with `frames=73` (approximately 3 seconds at 24 fps) and supply
an IMAGE batch of **exactly 73 source video frames** resampled to 24 fps, in
chronological order. The video VAE produces 22 latent-time slices. Other valid
lengths include 5, 22, 39, 56, 90, 107 and 124; a one-frame batch is an image.
The node rejects invalid lengths instead of silently truncating or stretching
motion. It does not perform video decoding or FPS conversion itself.

Keep the same target canvas / guide factor as training. Increasing target
resolution or frame count increases target activation memory even though guide
spatial tokens are reduced. Start with a known-fitting canvas on the target GPU.
To save video, connect decoded frames to your video encoder/combiner at 24 fps.

An optional `audio` soundtrack plus `audio_vae` can add aligned guide audio.
Supply audio starting at the guide clip's first frame. It is cropped to the
clip and available target duration, and is never sent to the text encoder.
Omit it for visual-only inference. Native reference conditioning and native
same-resolution guides can be chained alongside this node; their row positions
are retained.

## Geometry and encoding

- Each target pixel axis must be a multiple of `32 * downscale_factor`.
- The source is fitted to the target canvas with the fork's centered bicubic
  crop, then downscaled with Lanczos before VAE encoding. This also applies
  when the supplied source is already small: inspect `guide_preview`.
- H3 VAE spatial compression is 16. At output `1024 x 768`, factor 4 yields
  guide pixels `256 x 192`, guide latents `16 x 12`, and target latents `64 x 48`.
- The posterior is sampled with an isolated CPU generator (seed 42) and
  rounded through FP16 **before** latent normalization, matching the fork's
  image and video sampling paths. The native VAE mean-only encoding is not used.
- An image has one guide latent-time slice; 73 video frames have 22. Video
  time coordinates use H3's `(1,4,4,4,4)` temporal pattern, not a uniform stride.
- The corrected packed layout keeps target/reference/audio coordinates and
  reindexes condition positions, frozen/update masks and segment boundaries.
  Downscaled guides receive the target spatial grid at the training stride;
  coordinates from a separately normalized small grid are not multiplied.

## Validation and limits

CPU tests cover the audited native layout, factors 1/2/4, exact crop/resize,
posterior sampling, mixed/native guides, references, audio row preservation,
73-frame counts, metadata checks, clone isolation and wrapper composition.
Golden position fixtures come from the actual AI Toolkit packing implementation.
Run `python -m unittest discover -s tests -v`.

A full pretrained ComfyUI GPU sampling run has not been validated as part of
this change. Matching preprocessing and coordinates establishes the conditioning
contract; it does not establish the trained LoRA's visual quality. Compare
LoRA strength 0 and 1 on held-out inputs before evaluating upscale fidelity.

## Experimental reference layouts and source phase (1.47.0)

Optional inputs on Downscaled Guide expose `guide_rope_layout` (overlap/sidecar),
`reference_rope_layout` (native/overlap/sidecar), `reference_source_phase`,
`reference_phase_scale`, `reference_sidecar_margin`, and `source_id`. Defaults
preserve 1.46.0 behavior. H3 sidecar margins use normalized RoPE units, not pixels.
Select your LoRA to check these values against `minimax_h3_reference_rope` metadata.
Legacy checkpoints imply overlap/native/phase-off. Changing only inference geometry
on a legacy LoRA does not establish that the model can use it.

**MiniMax-H3 Identity Reference + RoPE (BFS)** adds a separate single-image latent
reference without passing it to the VLM. Supply the same target latent and full H3
VAE. Chain both model and conditioning through Guide and Identity Reference, then
connect both final outputs to the guider/sampler (and model to the scheduler).
Use guide `source_id=1`, identity `source_id=2` for training control channels 1 and 2.
The identity reference keeps its aspect and is resized down only to the trainer's
/32 reference-image bucket; the guide factor does not resize the portrait.
The reference preview shows the actual pixels used.

This node requires a checkpoint trained with a matching identity reference role
and `control_latent_only: true` for caption-only conditioning. The LoRA selector
checks that mode too. It does not load LoRA weights. Adding a portrait to a LoRA
trained only with a low-resolution guide is not a validated identity-preservation
recipe. Keep caption/source conventions consistent during training.

Source phase adds the phase `source_id * phase_scale * 10000**(-d/48)` to H3's
rotary angles. Text and targets remain source zero. The scoped ModelPatcher object
patch composes this phase only when this node's marked conditioning is active;
unmarked conditioning and other models keep their original rotary path.

For a first experiment use overlap plus phase and compare to the phase-off baseline.
Sidecar and phase have CPU geometry/rotary parity tests, including mixed image,
video and audio, but no full pretrained H3 quality validation yet. They distinguish
sources; they do not guarantee the same face. See the
[training configuration guide](https://github.com/alisson-anjos/ai-toolkit/blob/minimax-h3-latent-guides/H3_REFERENCE_ROPE.md).


## Video context in CLIP with aligned guide latents

Select **aligned guide + CLIP video context** in **BFS Shot H3 Conditioning**
(or its Simple variant). Qwen receives only the CURRENT SHOT as `<Video 1>` at
2 fps with real timestamps, using the same shot interval passed to Add Guide.
It does not receive the full source video on every shot. The original shot frames
provide visual context even when source blur modifies the aligned latent guide.
Images retain their `<Picture 1>` through `<Picture N>` order and native image
reference latents. Planner and the new standalone node allow up to 100 pictures
(the Comfy autogrow capacity); available model context and memory can limit this.
The original H3 node keeps its own unchanged input limits.

The shot itself enters the diffusion model only as aligned `minimax_keyframes`
through Add Guide; the new mode does not create a native `minimax_refs` video block
for it. Its Qwen embeddings still travel in `positive`. Explicitly enabling the
separate native masked-region video reference adds that as `<Video 2>`.

For a native workflow, use **MiniMax H3 Reference to Video — Guide 0 CLIP Context
(BFS)** in place of **MiniMax H3 Reference to Video**:

1. Connect your source guide to **guide_video_0**, and pictures to `ref_images`.
   `0` identifies the input's special role; its prompt token is `<Video 1>` because
   H3 numbers video tokens from 1. Only this guide is CLIP-only.
2. Optional `ref_videos` remain ACTUAL native video references, receiving both Qwen
   encoding and native reference latents. With guide 0 connected they are
   `<Video 2>`, `<Video 3>`, `<Video 4>`; without it native numbering starts at 1.
   Up to 3 actual reference videos are available, as in the original node.
3. Connect the returned `positive` and target `latent` to **MiniMax H3 Add Guide**,
   with the SAME guide frames, VAE and `frame_idx = 0`.
4. Feed Add Guide's `positive` and the target `latent` to your sampler.

Video inputs use the native 24-fps convention, are trimmed to the requested target
length, and round down to the native 17*k+5 frame grid. There is no separate 15s
hard cap in this helper; practical context/memory and checkpoint training limits
still apply. Native reference soundtracks remain paired to their `ref_video_N`
input, not to guide 0. Supply guide audio through Add Guide.
The helper resizes only the guide frames sampled for Qwen and never VAE-encodes
that guide as a native video reference. Images, actual reference videos and audio
retain the native implementation. Target latent starts empty; Add Guide supplies
aligned motion. The BFS mode performs the context and aligned-guide steps
internally; Shot Join only assembles/composites the generated outputs.

Example: `<Subject 1> comes from <Picture 3> and follows the blond performer in
the yellow shirt in <Video 1>. <Subject 2> comes from <Picture 1> and <Picture 2>
and follows the orange-haired performer in the white shirt in <Video 1>. Keep
these assignments through crossings and occlusions.` A picture need not mean a
separate person; multiple images can define one identity, and there can be N
assigned subjects/reference assets within the available context.

Existing modes and defaults are unchanged. Qwen encoding cost and text context
length increase. Checkpoints trained with `guide_latent_only` did not see the
source video in Qwen: this is an experimental inference option, not a demonstrated
identity-binding improvement. GPU and visual validation remain for the user's
local workflow.


### Guide sampling rate and console evidence

`clip_video_fps` is available on both BFS conditioning variants and the Guide 0
node. Default is 2 fps; range is 0.5–24 fps, capped to the source frame rate
without duplicating frames. It changes guide sampling in Qwen only, not the
aligned guide, output fps, or native reference-video sampling. Try 4 or 8 fps
against 2 fps with the same shot, prompt, references, seed and sampler settings.
Native H3 timestamps are retained; higher sampling rates are an experiment, not
a guarantee of stronger identity binding.

Console messages with `[BFS H3 CLIP context]` report the input/used frame count,
requested/effective fps, sent RGB frames, dimensions and first/last timestamps.
A second message confirms that the tokenizer actually emitted H3 video vision
blocks; an incompatible CLIP fails explicitly rather than silently ignoring the
video. The final message appears only after Qwen encoding returns, with elapsed
time and conditioning shapes. Qwen can pad an odd sample count to pairs.
These logs confirm the conditioning path ran, not visual quality.

Compare `aligned guide (Add Guide)` against `aligned guide + CLIP video context`
with all other settings fixed to assess whether video context changes the result.
Use a fresh sampler execution (Planner cached renders may otherwise return old
results); the CLIP-context logs only appear when conditioning actually executes.
Compare 2/4/8 fps separately to isolate the sampling-rate effect.