125 lines
5.7 KiB
Markdown
125 lines
5.7 KiB
Markdown
# ComfyUI-Krea2-Reference
|
||
|
||
Reference images for **Krea 2** in ComfyUI — ChatGPT-style: *"the character
|
||
from Picture 1, in the exact art style of Picture 2"*. Character and style
|
||
transfer through native vision tokens, not img2img.
|
||
|
||
Krea 2's text encoder is **Qwen3-VL-4B**, a vision-language model. This node
|
||
interleaves your reference images into the prompt as native vision tokens
|
||
(`Picture N:`), so the DiT reads character/style information from the same
|
||
hidden states it was trained on. Paired with the **reference LoRA** (below),
|
||
the model follows the scene text instead of collaging the reference.
|
||
|
||
## Examples
|
||
|
||
Everything below was generated by Krea 2 itself (Turbo, 8 steps, cfg 1.0).
|
||
The pipeline: one root character and two style images generated text-only,
|
||
then combined through this node + the `krea2_reference_v1` LoRA (layered:
|
||
text 1.0 / dit 0.0).
|
||
|
||
**Step 1 — generate the ingredients (text-only, no references):**
|
||
|
||
| Root character | Style image A (ukiyo-e) | Style image B (rubber-hose) |
|
||
|---|---|---|
|
||
|  |  |  |
|
||
|
||
**Step 2 — combine them:** `"the character from Picture 1, in the exact art
|
||
style of Picture 2, <scene>"`
|
||
|
||
| Girl × ukiyo-e | Girl × rubber-hose |
|
||
|---|---|
|
||
|  |  |
|
||
|
||
**Style by text instead of image** (single reference, style described in the
|
||
prompt) **and multi-panel:**
|
||
|
||
| Watercolor (text style) | Retro 90s cel (text style) | 4-panel comic |
|
||
|---|---|---|
|
||
|  |  |  |
|
||
|
||
Same braid, star hairpin, freckles, bell choker, sunflower dress, cat
|
||
backpack — across styles and panels. With two references (v1), the style
|
||
image is adopted strongly by the scene and only partially by the subject;
|
||
describing the style in text as well pushes it further onto the character.
|
||
|
||
> **Drag & drop:** every PNG in `examples/` embeds its full ComfyUI graph.
|
||
> Drop `examples/girl_in_ukiyoe.png` into ComfyUI to get the two-reference
|
||
> workflow (2× Load Image → Krea 2 Reference Encode → layered LoRA →
|
||
> KSampler), or `examples/style_watercolor.png` for the single-reference one.
|
||
|
||
## Install
|
||
|
||
```bash
|
||
cd ComfyUI/custom_nodes
|
||
git clone https://github.com/KonokoAz/ComfyUI-Krea2-Reference
|
||
```
|
||
|
||
Models (ComfyUI format): Krea 2 checkpoint (`UNETLoader`), `qwen3vl_4b`
|
||
**bf16** text encoder (`CLIPLoader`, type `krea2` — fp8 breaks the vision
|
||
tower), `qwen_image_vae` (`VAELoader`). Download the reference LoRA from
|
||
Civitai / Hugging Face and put it in `models/loras/`.
|
||
|
||
## Nodes
|
||
|
||
### Krea 2 Reference Encode (`TextEncodeKrea2Reference`)
|
||
|
||
Prompt + up to 4 reference images. **Roles live in the prompt**: write
|
||
`character1..4` for identity references and `style1..4` for style-only
|
||
references (their patches are shuffled so the subject can't leak; `characterN`
|
||
wins if both appear). Both expand to the exact clauses the LoRA trains on —
|
||
`"The character1 driving a car in the style2"` is a complete prompt. Plain
|
||
`image1..4` / `Picture 1..4` also work; connected images never mentioned get a
|
||
character clause appended automatically.
|
||
|
||
| Input | Description |
|
||
|---|---|
|
||
| `imageN_detail` | vision tokens per ref: low/normal/high/max = 256/512/1024/1280px. High for characters, low for style/pose hints |
|
||
| `imageN_mask` | crop to mask bbox + white-out background before encoding |
|
||
| `bg_removal` | wire `LoadBackgroundRemovalModel`: auto-cleans content refs |
|
||
| `layout_split` | first fraction of steps conditioned on text only (0.25 kills collage layouts zero-shot; unnecessary with the LoRA) |
|
||
|
||
### Krea 2 Reference LoRA (layered) (`Krea2ReferenceLoraLoader`)
|
||
|
||
Loads `krea2_reference_v1.safetensors` with independent strengths per layer
|
||
group. Reference addressing, anti-bleed and style transfer live in the
|
||
conditioning path (`text_fusion`/`txt_in`); the DiT-block tensors mostly add
|
||
identity drift.
|
||
|
||
- **`strength_text_fusion 1.0` + `strength_dit_body 0.0`** — recommended.
|
||
Base model rendering untouched, references followed. Best identity, best
|
||
for comics.
|
||
- Raise `strength_dit_body` to ~0.3 for extra scene adherence on hard prompts.
|
||
- A regular `LoraLoaderModelOnly` at strength 0.7–1.4 also works (single
|
||
knob, slight identity drift at high strength).
|
||
|
||
|
||
## Reference LoRA v1
|
||
|
||
Trained on ~2.7k reference-following triplets (teacher-generated character
|
||
sets, judged for consistency by Qwen3-VL). What it fixes vs zero-shot:
|
||
|
||
- Scene text followed instead of ignored (no more white-background
|
||
character-sheet collages).
|
||
- `"the character from Picture 1"` / `"in the exact art style of Picture 1"`
|
||
addressing.
|
||
- Style transfer decouples the reference's palette/layout from the scene.
|
||
- Plain text-to-image unaffected (ref-dropout training rows).
|
||
|
||
Known limits (v2 in progress): character-sheet references induce multi-view
|
||
clones (use single-pose refs); style refs strongly tied to a franchise drag
|
||
their cast along; two-reference prompts (character + style) work only
|
||
partially.
|
||
|
||
## Tips
|
||
|
||
- **The reference's colors ARE the character's identity.** To restyle an
|
||
attribute, change it in the reference, not the prompt: a grayscale copy of
|
||
the character ref + "the entire image is black and white" produces a fully
|
||
monochrome scene (`examples/girl_in_rubberhose_bw.png`); with the color
|
||
ref, the character keeps her palette no matter what the prompt says.
|
||
Raising `strength_dit_body` does NOT help here — it pulls toward the LoRA's
|
||
(colorful) training distribution.
|
||
- CFG 1.0 on Turbo skips the negative prompt entirely — it's not a bug.
|
||
- Wide canvases encourage figure duplication; 1024×1024 is the sweet spot.
|
||
- For exact pose: pose image as init latent, denoise 0.70–0.78.
|