Pis7i-COMFYUI-MODELINSTALLER/CONTROLNET_CHARACTER_SWAP_PLAN.md

ControlNet Character Face Swap Node - Implementation Plan

Concept Overview

Create a ComfyUI node that takes a reference image (for pose, clothing, background, scene) and applies a character's face from your AI model collection. This lets you place your consistent AI characters into any pose/scene while maintaining their identity.

Use Case Example

Input 1: Reference image of a person sitting at a café (pose + scene)
Input 2: Your AI character model/LoRA or reference face image
Output: Your character sitting at the café in the same pose

Technical Approach

1. Multi-ControlNet for Structure Transfer

Extract structure from reference image:

• OpenPose - Body pose and skeleton (PRIMARY - most important)
• Depth - Spatial depth and 3D structure
• Canny Edge - Sharp outlines and composition
• Normal Map - Surface details and lighting direction
• Softedge - Softer structural guidance

2. Character Face Application Methods

Option A: LoRA-Based (Best for consistent characters)

• Use character-specific LoRA trained on your AI character
• ControlNets guide pose/scene
• LoRA ensures consistent face/identity
• Strength: LoRA weight 0.7-1.0

Option B: IP-Adapter FaceID (Best for flexibility)

• Use IP-Adapter with FaceID model
• Input: Reference image structure + Character face reference
• Maintains face identity across different scenes
• Strength: IP-Adapter weight 0.8-1.0

Option C: Reactor/ReActor Face Swap

• Generate image with ControlNets
• Use face swap to replace generated face with character face
• Most accurate face transfer
• Works with any character reference

3. Recommended Hybrid Approach

Combine multiple methods for best results:

Reference Image (pose/scene)
    ↓
┌─────────────────────────────────┐
│  ControlNet Preprocessing       │
│  - OpenPose skeleton            │
│  - Depth map                    │
│  - Canny edges                  │
│  - Normal map                   │
└─────────────────────────────────┘
    ↓
┌─────────────────────────────────┐
│  Initial Generation             │
│  - Base model + Character LoRA  │
│  - Guided by ControlNets        │
│  - Prompt: character description│
└─────────────────────────────────┘
    ↓
┌─────────────────────────────────┐
│  Face Identity Reinforcement    │
│  Option 1: IP-Adapter FaceID    │
│  Option 2: ReActor Face Swap    │
│  - Apply character face         │
│  - Blend seamlessly             │
└─────────────────────────────────┘
    ↓
Output: Character in reference pose/scene

Node Architecture

Input Ports

Required

1. reference_image - Source image for pose/background/scene
2. model - Base Stable Diffusion checkpoint
3. character_face_image - Your character's face reference

Character Method (Choose One)

4. character_lora - LoRA file for your character (optional)
5. lora_strength - Float (0.0-1.5, default 0.8)

ControlNet Settings

6. pose_strength - Float (0.0-1.5, default 1.0) - Most important
7. depth_strength - Float (0.0-1.5, default 0.7)
8. canny_strength - Float (0.0-1.5, default 0.5)

Generation Settings

9. positive_prompt - Character description + scene details
10. negative_prompt - Quality negatives
11. face_method - Choice ["IP-Adapter FaceID", "ReActor", "Both"]
12. face_strength - Float (0.0-1.0, default 0.85)
13. seed - Random seed

Optional

14. preserve_clothes - Boolean (try to keep reference clothing)
15. preserve_background - Boolean (keep original background)
16. style_prompt - Override style (anime, realistic, etc.)

Output Ports

1. image - Final generated image
2. pose_preview - OpenPose skeleton preview
3. depth_preview - Depth map preview
4. face_mask - Face region mask (debugging)

Processing Pipeline

class ControlNetCharacterSwap:
    
    @classmethod
    def INPUT_TYPES(cls):
        return {
            "required": {
                "reference_image": ("IMAGE",),
                "character_face": ("IMAGE",),
                "model": ("MODEL",),
                "positive_prompt": ("STRING", {"multiline": True}),
                "pose_strength": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 2.0}),
                "face_strength": ("FLOAT", {"default": 0.85, "min": 0.0, "max": 1.0}),
            },
            "optional": {
                "character_lora": ("LORA",),
                "lora_strength": ("FLOAT", {"default": 0.8}),
                "depth_strength": ("FLOAT", {"default": 0.7}),
                "canny_strength": ("FLOAT", {"default": 0.5}),
                "face_method": (["IP-Adapter FaceID", "ReActor", "Both"],),
            }
        }
    
    RETURN_TYPES = ("IMAGE", "IMAGE", "IMAGE", "MASK")
    RETURN_NAMES = ("image", "pose_preview", "depth_preview", "face_mask")
    FUNCTION = "generate_character"
    CATEGORY = "PMA Utils/Character"
    
    def generate_character(self, reference_image, character_face, model, 
                          positive_prompt, pose_strength, face_strength,
                          character_lora=None, lora_strength=0.8,
                          depth_strength=0.7, canny_strength=0.5,
                          face_method="IP-Adapter FaceID"):
        
        # STEP 1: Extract ControlNet Maps
        pose_map = openpose_detector(reference_image)
        depth_map = depth_estimator(reference_image)
        canny_map = canny_detector(reference_image)
        
        # STEP 2: Prepare Model
        working_model = model
        if character_lora:
            working_model = apply_lora(model, character_lora, lora_strength)
        
        # STEP 3: Apply ControlNets
        # OpenPose is primary - controls body structure
        latent = apply_controlnet(
            model=working_model,
            controlnet=openpose_controlnet,
            image=pose_map,
            strength=pose_strength
        )
        
        # Depth adds spatial awareness
        latent = apply_controlnet(
            latent=latent,
            controlnet=depth_controlnet,
            image=depth_map,
            strength=depth_strength
        )
        
        # Canny for fine details (lower strength to allow character features)
        latent = apply_controlnet(
            latent=latent,
            controlnet=canny_controlnet,
            image=canny_map,
            strength=canny_strength
        )
        
        # STEP 4: Generate Base Image
        generated_image = generate(
            latent=latent,
            positive=positive_prompt,
            negative=negative_prompt,
            model=working_model
        )
        
        # STEP 5: Apply Character Face
        if face_method == "IP-Adapter FaceID":
            final_image = apply_ipadapter_faceid(
                image=generated_image,
                face_reference=character_face,
                strength=face_strength
            )
        
        elif face_method == "ReActor":
            final_image = reactor_face_swap(
                target_image=generated_image,
                source_face=character_face,
                strength=face_strength
            )
        
        elif face_method == "Both":
            # First IP-Adapter, then ReActor for maximum accuracy
            temp_image = apply_ipadapter_faceid(
                image=generated_image,
                face_reference=character_face,
                strength=face_strength * 0.7
            )
            final_image = reactor_face_swap(
                target_image=temp_image,
                source_face=character_face,
                strength=face_strength
            )
        
        # STEP 6: Generate Previews
        face_mask = detect_face_mask(final_image)
        
        return (final_image, pose_map, depth_map, face_mask)

Workflow Example

Basic Workflow

┌─────────────────┐
│ Reference Image │ (person at café)
└────────┬────────┘
         │
    ┌────▼─────────────────────┐
    │ ControlNet Preprocessors │
    │ - OpenPose               │
    │ - Depth                  │
    │ - Canny                  │
    └────┬─────────────────────┘
         │
    ┌────▼────────────┐      ┌──────────────────┐
    │ SD Generation   │◄─────┤ Character LoRA   │
    │ + ControlNets   │      └──────────────────┘
    └────┬────────────┘
         │
    ┌────▼────────────┐      ┌──────────────────┐
    │ IP-Adapter      │◄─────┤ Character Face   │
    │ FaceID          │      │ Reference        │
    └────┬────────────┘      └──────────────────┘
         │
    ┌────▼────────────┐
    │ Final Output    │ (Your character at café)
    └─────────────────┘

Prompt Strategy

Positive Prompt Template

[CHARACTER NAME/DESCRIPTION], [CHARACTER FEATURES], 
[REFERENCE SCENE DESCRIPTION], [POSE DESCRIPTION],
[STYLE KEYWORDS], high quality, detailed face, 8k

Example:

Emma, young woman with long brown hair, blue eyes, fair skin,
sitting at outdoor café table, holding coffee cup, relaxed pose,
photorealistic, natural lighting, bokeh background, professional photo

Negative Prompt

bad face, deformed, ugly, blurry, low quality, bad anatomy, 
extra limbs, wrong proportions, watermark

Required Models & Dependencies

ControlNet Models (SD 1.5 or SDXL)

• control_sd15_openpose.pth - ESSENTIAL
• control_sd15_depth.pth - ESSENTIAL
• control_sd15_canny.pth - Recommended
• control_sd15_normal.pth - Optional

Face Models

• IP-Adapter FaceID Plus v2 - For face identity transfer

  • Model: ip-adapter-faceid-plusv2_sdxl.bin
  • CLIP: clip_vision_SDXL_vit-h.safetensors

• ReActor - For face swapping

  • InsightFace models (buffalo_l)

ComfyUI Custom Nodes

1. ComfyUI-ControlNet-Aux - Preprocessors
2. ComfyUI-IPAdapter-Plus - IP-Adapter support
3. ReActor Node - Face swapping
4. ComfyUI-Impact-Pack - Face detection utilities

Your Character Assets

• Character LoRAs - Trained on your AI characters
• Character Face References - Clean face shots of each character

Character Library Management

Suggested Structure

/models/loras/characters/
    emma_character.safetensors
    john_character.safetensors
    sarah_character.safetensors

/inputs/character_faces/
    emma_face.png
    john_face.png
    sarah_face.png

Node Enhancement: Character Selector

Add dropdown to select from saved characters:

"character_preset": (self.get_character_list(),)

def get_character_list(self):
    # Scan character directory
    return ["Emma", "John", "Sarah", "Custom..."]

Advanced Features

1. Outfit Preservation Mode

• Extract clothing from reference with segmentation
• Apply clothing ControlNet or inpainting
• Keep character face + reference outfit

2. Background Swapping

• Segment background from reference
• Replace with custom background
• Keep character + pose

3. Multi-Character Scenes

• Detect multiple people in reference
• Apply different character faces to each
• Maintain relative positions

4. Expression Transfer

• Detect facial expression from reference
• Apply expression to character face
• Use expression ControlNet if available

5. Batch Processing

• Process multiple references with same character
• Generate character pose library
• Automatic prompt generation from reference

Testing Scenarios

Scenario	Reference	Character	Expected Result
Portrait	Professional headshot	Emma	Emma in professional setting
Action	Person jumping	John	John jumping, same pose
Group	3 people at party	Sarah (1 person)	Sarah in same scene/pose
Historical	Vintage photo	Emma	Emma in vintage style/setting
Fashion	Model on runway	John	John as fashion model

Optimization Tips

For Best Quality

1. OpenPose Strength: 0.9-1.2 (maintain pose)
2. Character LoRA: 0.7-1.0 (strong character features)
3. Face Method: "Both" (IP-Adapter + ReActor)
4. Resolution: Match reference or use 1024x1024 for SDXL

For Speed

1. Reduce ControlNets: Use only OpenPose + Depth
2. Lower Steps: 20-25 steps sufficient
3. Skip ReActor: Use only IP-Adapter FaceID
4. Batch Process: Queue multiple generations

For Consistency Across Images

1. Use Same LoRA + Weight for all generations
2. Use Same Face Reference image
3. Lock Seed for similar variations
4. Save Workflow as template

Common Issues & Solutions

Problem	Cause	Solution
Wrong face	Weak face strength	Increase face_strength to 0.9-1.0
Wrong pose	Weak pose ControlNet	Increase pose_strength to 1.2
Blurry face	ControlNet conflict	Lower canny/depth, keep pose high
Wrong clothing	Strong clothing from ref	Lower depth_strength, use outfit prompt
Background wrong	ControlNets too strong	Lower all except pose, focus prompt on BG
Face doesn't match character	LoRA not strong enough	Increase lora_strength or use "Both" face method

Implementation Phases

Phase 1: Core Functionality

• Plan architecture
• Implement OpenPose ControlNet integration
• Implement Depth ControlNet integration
• Basic generation pipeline

Phase 2: Face Integration

• IP-Adapter FaceID integration
• ReActor face swap integration
• Face blending quality improvements

Phase 3: LoRA Support

• Character LoRA loading
• LoRA strength controls
• Character library management

Phase 4: UI/UX

• Character preset selector
• Preview outputs for each stage
• Workflow templates

Phase 5: Advanced Features

• Outfit preservation mode
• Expression transfer
• Batch processing

Expected Results

Quality Metrics:

• Face Similarity: 90%+ match to character reference
• Pose Accuracy: 95%+ match to reference pose
• Scene Coherence: Natural integration of character into scene
• Processing Time: 15-30 seconds per image (RTX 3090)

Conclusion

This node solves the problem of placing consistent AI characters into arbitrary poses and scenes. By combining ControlNet for structure transfer with face identity models (IP-Adapter/ReActor), you can maintain character consistency while achieving any pose or setting.

Perfect for:

• Building character portfolio/lookbooks
• Storytelling with consistent characters
• Virtual fashion/product modeling
• Character-based content creation
• AI influencer content

Estimated Development Time: 1-2 weeks
Complexity: Medium-High
Value: Extremely high for character-based workflows