EricRollei-Local_LLM_Prompt_Enhancer/REFERENCE_MODE_GUIDE.md

# Reference Mode Complete Guide

## How Reference Modes Work

When you attach a reference image to the **Advanced Video Prompt Expander** node, the system follows this flow:

### The Complete Pipeline

```
1. Reference Image (your uploaded image)
   ↓
2. Qwen3-VL Analysis (mode-specific prompt)
   ↓
3. Mode Instruction (tells LLM how to use analysis)
   ↓
4. Combined with Your Prompt (from GUI)
   ↓
5. LM Studio Expansion (generates final video prompt)
```

---

## All 7 Reference Modes Explained

### 1. recreate_exact (Default)

**When to use:** Animating existing artwork, maintaining exact appearance

**Qwen3-VL analyzes:**
- ✅ Main subject: facial features, expression, hair, body, pose, clothing
- ✅ Exact outfit: style, color, material, accessories
- ✅ Environment: location, background elements, spatial layout
- ✅ Lighting: direction, quality, color temperature, shadows
- ✅ Color palette: dominant colors, saturation
- ✅ Visual style: artistic style, rendering, mood
- ✅ Camera framing and composition
- ✅ Props, objects, environmental details

**LLM Instruction:**
```
[REFERENCE MODE: RECREATE EXACT]
Use the provided image as the exact character and costume reference.
Keep the same face, hair, outfit, lighting, and overall aesthetic.
Animate this character/scene without changing identity or appearance.
Match the visual style, mood, and composition of the reference image.
```

**Example:**
- Image: Character in medieval armor
- Your prompt: "walking through a forest"
- Result: Same character, same armor, walking through forest

---

### 2. subject_only

**When to use:** Place character in completely new environment/lighting

**Qwen3-VL analyzes:**
- ✅ Facial features: face shape, eyes, expression, skin tone
- ✅ Hair: style, length, color, texture
- ✅ Body type: build, proportions, posture
- ✅ Current clothing (for reference)
- ✅ Personality traits in expression
- ✅ Age, gender, distinctive features
- ❌ Background (ignored)
- ❌ Environment (ignored)
- ❌ Lighting (ignored)

**LLM Instruction:**
```
[REFERENCE MODE: SUBJECT ONLY]
Preserve ONLY the subject's face, body identity, and core appearance.
Ignore the original background, lighting, and environment.
Place this character in the new scene with new lighting and atmosphere.
Keep character identity consistent but change everything else.
```

**Example:**
- Image: Woman in studio with soft lighting
- Your prompt: "in a cyberpunk alley at night"
- Result: Same woman, new outfit, cyberpunk environment, neon lighting

---

### 3. style_only

**When to use:** Apply artistic style to new content

**Qwen3-VL analyzes:**
- ✅ Artistic style: photorealism, illustration, anime, painterly
- ✅ Lighting style: quality, mood, direction, color temperature
- ✅ Color grading: palette, saturation, contrast, toning
- ✅ Atmospheric treatment: haze, clarity, depth, mood
- ✅ Cinematographic style: film look, vintage, modern
- ✅ Visual effects: grain, blur, sharpness, filters
- ✅ Overall mood and emotional tone
- ❌ Subject identity (ignored)
- ❌ Character (ignored)
- ❌ Scene content (ignored)

**LLM Instruction:**
```
[REFERENCE MODE: STYLE TRANSFER]
Match the lighting, color palette, visual aesthetic, and cinematic mood.
Create a completely new subject, character, and scene.
Apply the same artistic style, color grading, and atmospheric treatment.
```

**Example:**
- Image: Film noir scene (high contrast, dramatic shadows)
- Your prompt: "a detective investigating a crime scene"
- Result: New detective, new scene, same noir lighting/style

---

### 4. color_palette_only

**When to use:** Match color mood, different subject

**Qwen3-VL analyzes:**
- ✅ Dominant colors (3-5 main colors with percentages)
- ✅ Accent colors
- ✅ Color relationships: complementary, analogous, triadic
- ✅ Saturation level: vivid, muted, desaturated
- ✅ Value range: bright, dark, balanced
- ✅ Color temperature: warm, cool, neutral
- ✅ Color mood: emotional tone
- ❌ Subjects (ignored)
- ❌ Composition (ignored)
- ❌ Lighting style (ignored)

**LLM Instruction:**
```
[REFERENCE MODE: COLOR PALETTE ONLY]
Extract and apply the dominant color scheme from the reference image.
Use the same hues, saturation levels, and color relationships.
Create an entirely new subject and scene.
Maintain color harmony with the reference palette.
```

**Example:**
- Image: Sunset landscape (warm oranges, purples, magentas)
- Your prompt: "a spaceship landing on an alien planet"
- Result: Spaceship scene using sunset color palette

---

### 5. action_only

**When to use:** Recreate pose/motion with different character

**Qwen3-VL analyzes:**
- ✅ Body position: stance, posture, weight distribution
- ✅ Arm positions and hand gestures
- ✅ Leg positions and foot placement
- ✅ Head position and tilt
- ✅ Movement direction
- ✅ Energy level: static, dynamic, tense, relaxed
- ✅ Action being performed
- ✅ Physical dynamics and balance
- ❌ Character identity (ignored)
- ❌ Clothing (ignored)
- ❌ Environment (ignored)

**LLM Instruction:**
```
[REFERENCE MODE: ACTION/POSE ONLY]
Recreate the pose, gesture, body language, and action/movement.
Change the character identity, environment, lighting, costume, and visual style.
Keep only the physical positioning and motion dynamic.
```

**Example:**
- Image: Dancer mid-leap
- Your prompt: "a warrior in battle armor"
- Result: Warrior performing the same leap pose

---

### 6. character_remix

**When to use:** Same character, different story/scenario

**Qwen3-VL analyzes:**
- ✅ Core identity markers: distinctive facial features, expression
- ✅ Character archetype: hero, villain, mystic, warrior
- ✅ Key personality traits in pose/expression
- ✅ Body language and demeanor
- ✅ Age range and build
- ✅ Current context (to change): what they're doing, where they are
- ✅ Outfit and setting (for reference)
- ⚠️ Emphasis on what makes THIS CHARACTER unique

**LLM Instruction:**
```
[REFERENCE MODE: CHARACTER REMIX]
Keep the character's core identity (face, build, personality traits).
Place them in a completely new scenario, environment, time period, or genre.
Change their outfit, lighting, setting, and mood.
Maintain character recognition while adapting to new context.
```

**Example:**
- Image: Samurai warrior in feudal Japan
- Your prompt: "as a detective in 1940s New York"
- Result: Same person's face/identity, detective outfit, noir setting

---

### 7. reimagine (NEW!)

**When to use:** Creative reinterpretation of concept/theme

**Qwen3-VL analyzes:**
- ✅ Core concept or theme
- ✅ Mood and emotional tone
- ✅ Narrative or story suggested
- ✅ Key symbolic elements or motifs
- ✅ Overall compositional approach
- ✅ What makes the image compelling
- ✅ Underlying ideas or themes
- ✅ The "feeling" or atmosphere
- ⚠️ Focus on abstract qualities, not literal details

**LLM Instruction:**
```
[REFERENCE MODE: REIMAGINE]
Use the reference image as loose inspiration for creative reinterpretation.
Take the core concept, mood, or theme and reimagine it in a new way.
Feel free to change subject, style, setting, and execution.
Maintain thematic connection while interpreting the essence freely.
```

**Example:**
- Image: Lonely astronaut on barren planet
- Your prompt: "underwater exploration"
- Result: Lone diver in vast ocean depths (same isolation/exploration theme)

---

## Complete Example Flow

### Using character_remix mode:

**Input:**
- Reference Image: Photo of a woman in Victorian dress
- reference_mode: "character_remix"  
- Your prompt: "in a futuristic cyberpunk city"

**Step 1 - Qwen3-VL Analysis (mode-specific):**
```
Core identity markers: Oval face, high cheekbones, piercing green eyes, 
determined expression, regal bearing. Character archetype: aristocratic 
but with rebellious undertones visible in her expression. Age: mid-20s. 
Build: slender but strong posture. Current context: Standing in Victorian 
parlor, wearing elaborate blue dress with corset. Key personality: 
Confident, intelligent, hint of defiance in her gaze.
```

**Step 2 - Mode Instruction Added:**
```
[REFERENCE MODE: CHARACTER REMIX]
Keep the character's core identity (face, build, personality traits).
Place them in a completely new scenario, environment, time period, or genre.
Change their outfit, lighting, setting, and mood.
Maintain character recognition while adapting to new context.
```

**Step 3 - Combined with Your Prompt:**
```
[Your prompt]: in a futuristic cyberpunk city

[REFERENCE MODE: CHARACTER REMIX]
Keep the character's core identity...

REFERENCE IMAGE ANALYSIS:
Core identity markers: Oval face, high cheekbones, piercing green eyes...
```

**Step 4 - LM Studio Generates:**
```
[GLOBAL SETUP]
The same woman with distinctive green eyes and regal bearing, now wearing 
sleek black tactical gear with neon blue accent lights. Towering holographic 
billboards reflect in rain-soaked streets of a cyberpunk megacity at night.

Shot 1: Close-up on her face as she surveys the neon-lit streets.
Her piercing green eyes catch the glow of holographic advertisements.
Camera slowly dollies in. Rain droplets stream past the lens...
```

---

## Key Improvements in v1.9.2

### Before (v1.9.1):
❌ Same generic Qwen3-VL prompt for all modes  
❌ Analyzed everything regardless of what mode needed  
❌ Wasted tokens on irrelevant details  
❌ LLM had to filter out what to ignore  
❌ Less precise control  
❌ No "reimagine" option

### After (v1.9.2):
✅ Mode-specific Qwen3-VL prompts  
✅ Only analyzes what's relevant for each mode  
✅ Efficient token usage  
✅ LLM gets pre-filtered focused information  
✅ More precise control  
✅ Added "reimagine" mode for creative freedom

---

## Which Mode Should I Use?

| Goal | Best Mode |
|------|-----------|
| Animate existing character art exactly | `recreate_exact` |
| Same character, different environment | `subject_only` |
| Apply film/art style to new scene | `style_only` |
| Match color mood of reference | `color_palette_only` |
| Recreate a specific pose/action | `action_only` |
| Character in different time/genre | `character_remix` |
| Thematic inspiration, creative freedom | `reimagine` |

---

## Technical Details

**Qwen3-VL Model:** Qwen/Qwen2-VL-7B-Instruct  
**Analysis Quality:** High-detail vision-language understanding  
**Token Efficiency:** Mode-specific prompts reduce unnecessary analysis  
**LLM Backend:** LM Studio (OpenAI-compatible API)  
**Integration:** Seamless Qwen3-VL → LM Studio pipeline

---

## Troubleshooting

**"Image not analyzed" error:**
- Check Qwen3-VL model is loaded
- Verify transformers>=4.42.0 installed
- Check image format (should be ComfyUI IMAGE type)

**"Reference mode not working as expected":**
- Clear ComfyUI cache and restart
- Verify you're using v1.9.2+
- Check LM Studio is running and connected
- Review console output for Qwen3-VL analysis

**"Character identity drifts":**
- Use `subject_only` or `character_remix` mode
- Make sure Qwen3-VL analysis includes facial features
- Try more detailed GUI prompt about maintaining identity

---

## Version History

**v1.9.2** (Current)
- Added mode-specific Qwen3-VL prompts
- Added "reimagine" mode
- Improved character_remix focus
- More efficient token usage

**v1.9.1**
- Added 6 reference modes
- Basic mode instructions

**v1.9.0**
- Initial reference image support
- Generic Qwen3-VL analysis