slahiriandClaude Opus 4.5
7f1186d412
New Features: - SID_PromptCompose node with Standard and Enhance with AI modes - SID_PromptSynthesis node for VLM-based prompt generation - SID_ImageAnalysis node for comprehensive image analysis - Text LLM support (Qwen2.5 0.5B-7B) for prompt enhancement - PixAI tagger integration Documentation: - docs/image_tokenization_spec.md - Full technical specification for tokenization → classification → assembly pipeline - docs/implementation_plan.md - 8-phase implementation roadmap - docs/SID_PromptCompose.md - Node usage documentation Infrastructure: - core/compose/ module with base, pipeline, standard, llm generators - core/log.py centralized logging - Config files for Qwen2.5 text models (YAML) - Web assets for ComfyUI integration This commit establishes the foundation for rewriting the compose pipeline based on the tokenization specification. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
109 KiB
109 KiB
Image Tokenization & Classification Specification
Overview
This document defines the approach for extracting, normalizing, classifying, and validating image metadata tokens for prompt composition.
Version: 1.1 Status: Design Specification Date: December 2024 Last Updated: Added Phase 7 (Token Grounding Visualization)
Related Documents
| Document | Description |
|---|---|
| Implementation Plan | Step-by-step implementation roadmap with tasks and priorities |
| CLAUDE.md | Project overview and architecture |
Context Recovery
If resuming after context loss:
- Read this spec for technical details
- Read implementation_plan.md for current progress and next steps
- Check task checkboxes in implementation plan to see what's done
Table of Contents
- Phase 1: Tokenization
- Phase 2: Normalization
- Phase 3: Classification
- Phase 4: Assembly
- Phase 5: Assembly (Rule-Based)
- Phase 6: LLM-Based Assembly
- Phase 7: Token Grounding Visualization
- Validation & Quality Analysis
Phase 1: Tokenization
1.1 Token Structure
ImageToken {
text: string # The actual content
confidence: float # 0.0-1.0 confidence score
source: string # wd14, pixai, joytag, florence_caption, etc.
token_type: enum # TAG | PHRASE | KEY_VALUE | SENTENCE
metadata: object # additional context
}
1.2 Extraction Strategy by Source Type
A. TAGGER OUTPUTS (wd14, pixai, joytag, nudenet)
Input:
"wd14": {
"1girl": 0.999,
"solo": 0.978,
"realistic": 0.975
}
Extraction:
- Direct key-value extraction
- Key → text, Value → confidence
- Token type: TAG
Output:
ImageToken(text="1girl", confidence=0.999, source="wd14", token_type=TAG)
ImageToken(text="solo", confidence=0.978, source="wd14", token_type=TAG)
B. ANALYZER OUTPUTS (photography, iqa, composition, saliency)
Input:
"photography": {
"light gray tones": 0.9,
"high key": 0.8,
"portrait orientation": 0.95
}
Extraction:
- Same as taggers but mark source_type as ANALYZER
- These often have implicit category hints (photography → Lighting/Style)
C. VLM CAPTIONS (florence_caption, florence_description, florence_mixed_caption)
Input:
"Photo of a young woman with long, wavy brown hair, wearing a white Calvin Klein
crop top and matching panties. She has a fair complexion and is standing confidently
with her hands on her hips. The background is plain and light grey, and the lighting
is soft and even."
Extraction Strategy:
Step 1: Sentence Segmentation
- Split by
.,!,? - Handle edge cases (abbreviations, decimals)
Step 2: Clause Extraction
- Split sentences by
,,and,with,while - Preserve meaning units
Step 3: Phrase Pattern Extraction Use regex patterns for common structures:
| Pattern | Example | Extraction |
|---|---|---|
a/an [adj]* [noun] |
"a young woman" | subject phrase |
with [noun phrase] |
"with long, wavy brown hair" | attribute |
wearing [noun phrase] |
"wearing a white crop top" | clothing |
[verb]ing [adverb]* |
"standing confidently" | action |
The [noun] is [adj] |
"The background is plain" | environment |
[noun] is [adj] and [adj] |
"lighting is soft and even" | lighting |
Step 4: Output Multiple Tokens per Caption
ImageToken(text="young woman", token_type=PHRASE, source="florence_caption")
ImageToken(text="long wavy brown hair", token_type=PHRASE, source="florence_caption")
ImageToken(text="wearing white Calvin Klein crop top", token_type=PHRASE, ...)
ImageToken(text="standing confidently", token_type=PHRASE, ...)
ImageToken(text="background plain light grey", token_type=PHRASE, ...)
ImageToken(text="lighting soft and even", token_type=PHRASE, ...)
D. VLM STRUCTURED OUTPUTS (florence_analyze)
Input:
"camera_angle: front view, art_style: photo realistic, location: indoor,
background: plain light grey background, clothing: white crop top,
action: standing, facial_expression: neutral, hair_color: dark brown hair"
Extraction:
- Parse key-value pairs by
:and,/; - Preserve the key as metadata (helps classification)
- Token type: KEY_VALUE
Output:
ImageToken(text="front view", key="camera_angle", token_type=KEY_VALUE, ...)
ImageToken(text="photo realistic", key="art_style", token_type=KEY_VALUE, ...)
ImageToken(text="indoor", key="location", token_type=KEY_VALUE, ...)
E. MIXED TAG SECTIONS (from florence_mixed_caption_plus)
Input:
"1girl, solo, long hair, breasts, looking at viewer, large breasts,
brown hair (long hair), black hair, navel, cleavage, brown eyes, standing"
Extraction:
- Split by
, - Strip whitespace
- Handle parenthetical qualifiers:
brown hair (long hair)→ keep as single token or split - Token type: TAG
- Confidence: 1.0 (VLM generated, no score)
1.3 Phrase Decomposition Algorithm for Natural Text
INPUT: "a woman with a fair complexion, dark brown hair, and a confident expression,
standing with her arms at her sides, looking directly at the camera"
STEP 1: Identify Main Subject
→ "a woman" [SUBJECT]
STEP 2: Extract "with" clauses
→ "fair complexion" [SUBJECT_DETAIL]
→ "dark brown hair" [SUBJECT_DETAIL]
→ "confident expression" [SUBJECT_DETAIL]
→ "arms at her sides" [ACTION_POSE]
STEP 3: Extract "-ing" verb phrases
→ "standing with her arms at her sides" [ACTION_POSE]
→ "looking directly at the camera" [ACTION_POSE]
STEP 4: Chunk remaining phrases
→ Individual attribute phrases
Linguistic Chunking Approach:
Use a lightweight NLP chunker (spaCy, Stanza) to identify:
- NP (Noun Phrases): subjects, objects, attributes
- VP (Verb Phrases): actions
- ADJP (Adjective Phrases): qualities
- PP (Prepositional Phrases): locations, relationships
Phase 2: Normalization
2.1 Text Normalization Pipeline
Raw Token
│
▼
┌─────────────────────────────┐
│ Lowercase │
│ "Brown Hair" → "brown hair"│
└─────────────────────────────┘
│
▼
┌─────────────────────────────┐
│ Whitespace Normalization │
│ "long hair" → "long hair" │
└─────────────────────────────┘
│
▼
┌─────────────────────────────┐
│ Special Char Handling │
│ "1girl" → "1girl" │
│ "calvin klein" → keep as-is│
└─────────────────────────────┘
│
▼
Normalized Token
2.2 Deduplication Strategy
Exact Duplicates:
- Same normalized text → merge, keep highest confidence
Semantic Duplicates:
- "1girl" vs "1 girl" vs "one girl" → merge
- "brown hair" vs "hair brown" → merge
- Use embedding similarity > 0.95 threshold
Hierarchical Duplicates:
- "long wavy brown hair" contains "brown hair" and "long hair"
- Keep the more specific one OR keep both with relationship flag
Confidence Aggregation for Duplicates:
merged_confidence = 1 - ∏(1 - conf_i) # probability union
OR
merged_confidence = max(conf_i) # simpler
OR
merged_confidence = weighted_average(conf_i, source_weight_i)
2.3 Source Reliability Weighting
| Source | Weight | Rationale |
|---|---|---|
| wd14 | 1.0 | Well-calibrated tagger |
| joytag | 0.95 | Good general tagger |
| pixai | 0.9 | Slightly less calibrated |
| florence_caption | 0.85 | Good but verbose |
| florence_analyze | 0.9 | Structured, reliable |
| nudenet | 0.8 | Specific domain |
Phase 3: Classification
3.1 Classification Pipeline
┌─────────────────────────────────────────────────────────────────┐
│ LAYER 1: Deterministic Rules (Structured Keys) │
│ │
│ If token has key from florence_analyze: │
│ camera_angle, distance_to_camera → [Composition] │
│ art_style → [Style/Medium] │
│ location, background → [Environment/Scene] │
│ action, facing_direction → [Action/Pose] │
│ clothing, pants, shoes, accessory → [Subject Details] │
│ hair_*, eye_*, facial_*, body, race → [Subject Details] │
│ gender → [Subject] │
│ text → [Technical Parameters] │
└─────────────────────────────────────────────────────────────────┘
│
Not matched
▼
┌─────────────────────────────────────────────────────────────────┐
│ LAYER 2: Source-Based Routing │
│ │
│ If source == "photography": │
│ terms with "light", "tone", "contrast" → [Lighting] │
│ terms with "sharp", "focus", "quality" → [Quality Boosters] │
│ terms with "orientation", "frame" → [Composition] │
│ │
│ If source == "iqa": │
│ All → [Quality Boosters] │
│ │
│ If source == "composition" or "saliency": │
│ All → [Composition] │
└─────────────────────────────────────────────────────────────────┘
│
Not matched
▼
┌─────────────────────────────────────────────────────────────────┐
│ LAYER 3: Keyword Dictionary Lookup │
│ │
│ For each category, check: │
│ 1. Exact match in category.exact_matches │
│ 2. Prefix/suffix match in category.patterns │
│ 3. Regex match in category.regex_patterns │
│ │
│ If match found → assign to category │
└─────────────────────────────────────────────────────────────────┘
│
Not matched
▼
┌─────────────────────────────────────────────────────────────────┐
│ LAYER 4: Embedding Similarity (Optional) │
│ │
│ Compute embedding of token │
│ Compare to category centroid embeddings │
│ Assign to highest similarity category if > threshold │
└─────────────────────────────────────────────────────────────────┘
│
Not matched
▼
┌─────────────────────────────────────────────────────────────────┐
│ LAYER 5: Uncategorized Bucket │
│ │
│ Preserve for manual review or assembly-phase handling │
└─────────────────────────────────────────────────────────────────┘
3.2 Keyword Dictionary Structure
CategoryDictionary {
category: string
exact_matches: Set[string] # O(1) lookup
prefix_patterns: List[string] # "hair_*", "eye_*"
regex_patterns: List[regex] # complex patterns
semantic_anchors: List[string] # for embedding fallback
negative_keywords: Set[string] # explicit exclusions
}
Category Dictionaries:
[Quality Boosters]
exact_matches: {
"realistic", "photorealistic", "photo realistic", "high quality",
"masterpiece", "best quality", "ultra detailed", "8k", "4k", "hdr",
"sharp", "in focus", "detailed", "professional", "excellent quality",
"good aesthetic quality", "excellent technical quality", "sharp and clean",
"excellent clarity", "photograph", "photography", "photo medium"
}
semantic_anchors: [
"high quality professional image",
"sharp detailed realistic photo",
"masterpiece best quality"
]
[Subject]
exact_matches: {
"1girl", "2girls", "3girls", "1boy", "2boys", "solo", "multiple girls",
"multiple boys", "1woman", "1man", "couple", "group", "crowd",
"female", "male", "person", "people", "model"
}
prefix_patterns: ["*girl*", "*boy*", "*woman*", "*man*"]
semantic_anchors: [
"person woman man subject",
"single solo individual",
"group multiple people"
]
[Subject Details]
subcategories: {
"hair": {
exact: {"long hair", "short hair", "brown hair", "blonde hair", ...}
prefix: ["hair_*", "*_hair"]
},
"eyes": {
exact: {"brown eyes", "blue eyes", "looking at viewer", ...}
prefix: ["eye_*", "*_eyes"]
},
"face": {
exact: {"lips", "nose", "parted lips", "smile", ...}
prefix: ["facial_*"]
},
"body": {
exact: {"breasts", "navel", "slim", "curvy", "toned", ...}
prefix: ["body_*"]
},
"clothing": {
exact: {"underwear", "panties", "bra", "dress", "shirt", ...}
prefix: ["clothing_*", "wearing_*"]
},
"skin": {
exact: {"fair complexion", "tan", "pale skin", "dark skin", ...}
prefix: ["skin_*", "race_*"]
}
}
[Action/Pose]
exact_matches: {
"standing", "sitting", "walking", "running", "lying", "kneeling",
"arms at sides", "hands on hips", "cowboy shot", "looking at viewer",
"looking away", "from behind", "from side", "facing viewer"
}
verb_patterns: ["*ing", "*_shot"] # standing, cowboy_shot
semantic_anchors: [
"person standing sitting posing",
"action movement pose position",
"looking facing direction"
]
[Environment/Scene]
exact_matches: {
"indoor", "outdoor", "background", "simple background", "grey background",
"white background", "gradient background", "plain background", "studio",
"bedroom", "street", "forest", "beach", "city"
}
prefix_patterns: ["background_*", "location_*"]
semantic_anchors: [
"background environment setting location",
"indoor outdoor scene place"
]
[Lighting]
exact_matches: {
"soft light", "hard light", "natural light", "studio light", "rim light",
"backlight", "bright", "dark", "shadows", "high key", "low key",
"soft and even", "dramatic lighting", "neutral tones", "muted colors",
"light gray tones", "desaturated", "low contrast", "high contrast"
}
semantic_anchors: [
"lighting illumination bright dark",
"soft hard natural studio light",
"shadows contrast tones"
]
[Style/Medium]
exact_matches: {
"photo", "photograph", "digital art", "illustration", "3d render",
"anime", "realistic", "oil painting", "watercolor", "sketch",
"minimalist", "modern", "vintage", "retro", "artistic"
}
prefix_patterns: ["art_style_*", "style_*"]
semantic_anchors: [
"art style medium technique",
"photo realistic illustration painting"
]
[Composition]
exact_matches: {
"front view", "side view", "portrait", "landscape", "close up",
"full body", "upper body", "cowboy shot", "dutch angle", "birds eye",
"centered", "rule of thirds", "symmetry", "portrait orientation",
"vertical frame", "horizontal frame", "2:3 aspect ratio"
}
prefix_patterns: ["camera_angle_*", "distance_*", "composition_*"]
semantic_anchors: [
"camera angle view shot framing",
"composition layout arrangement",
"full body close up portrait"
]
[Technical Parameters]
exact_matches: {
"width", "height", "resolution", "aspect ratio", "jpeg artifacts",
"low resolution", "high resolution", "text", "watermark"
}
prefix_patterns: ["image_info_*", "text_*"]
regex_patterns: [r"\d+x\d+", r"\d+:\d+"] # dimensions, ratios
3.3 Embedding-Based Classification
Approach: Category Centroid Matching
Step 1: Build Category Embeddings (One-time, Offline)
For each category, create a centroid embedding:
For category in categories:
anchor_texts = category.semantic_anchors + sample(category.exact_matches, 50)
embeddings = [embed(text) for text in anchor_texts]
category_centroid = mean(embeddings)
category_embeddings[category] = normalize(category_centroid)
Step 2: Runtime Classification
token_embedding = normalize(embed(token.text))
similarities = {}
for category, centroid in category_embeddings:
similarities[category] = cosine_similarity(token_embedding, centroid)
best_category = argmax(similarities)
confidence = similarities[best_category]
if confidence > THRESHOLD: # 0.5 recommended
return (best_category, confidence)
else:
return (UNKNOWN, confidence)
Recommended Embedding Model
| Model | Speed | Quality | Memory |
|---|---|---|---|
all-MiniLM-L6-v2 |
Very Fast | Good | 80MB |
all-mpnet-base-v2 |
Medium | Better | 420MB |
paraphrase-MiniLM-L6-v2 |
Very Fast | Good for short | 80MB |
Recommendation: all-MiniLM-L6-v2 for best speed/quality tradeoff
3.4 Multi-Category Token Handling
Some tokens genuinely belong to multiple categories:
Example: "looking at viewer"
- [Action/Pose] - it's an action
- [Subject Details] → eyes - eye direction
Strategy: Multi-Label with Primary Flag
TokenClassification {
token: ImageToken
primary_category: string
secondary_categories: List[string]
confidence_per_category: Dict[string, float]
}
3.5 Caption → Canonical Category Extraction
The Challenge
Caption text interleaves multiple categories in single sentences:
INPUT:
"she has long, wavy brown hair and is wearing a white crop top with
the Calvin Klein logo, standing confidently against a plain grey background"
CONTAINS:
├── [Subject Details/hair]: "long, wavy brown hair"
├── [Subject Details/clothing]: "white crop top with Calvin Klein logo"
├── [Action/Pose]: "standing confidently"
└── [Environment]: "plain grey background"
Strategy: Hierarchical Clause Extraction
┌─────────────────────────────────────────────────────────────────┐
│ FULL CAPTION │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ LEVEL 1: SENTENCE SEGMENTATION │
│ │
│ Split by: . ! ? │
│ Handle: abbreviations (Dr., Mr.), decimals (0.5) │
│ Result: List of complete sentences │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ LEVEL 2: CLAUSE SEGMENTATION │
│ │
│ Split by: , | and | with | while | as | who | which │
│ Preserve: quoted text, parenthetical content │
│ │
│ "she has long hair, wearing a dress, standing confidently" │
│ ↓ │
│ ["she has long hair", "wearing a dress", "standing confidently"]│
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ LEVEL 3: PHRASE PATTERN MATCHING │
│ │
│ Each clause matched against category-specific patterns │
│ Multiple matches per clause allowed │
│ Unmatched portions preserved for fallback │
└─────────────────────────────────────────────────────────────────┘
Category-Specific Extraction Patterns
[Subject] Patterns
Trigger Structures:
├── "a/an [adjective]* [ethnicity]? [subject_noun]"
│ Examples: "a young woman", "an attractive asian model"
│
├── "the [subject_noun]"
│ Examples: "the woman", "the model"
│
└── Standalone: "woman", "man", "model", "person", "girl", "boy"
Subject Nouns: woman, man, girl, boy, lady, gentleman, person,
model, figure, individual, subject
Ethnicity Terms: asian, indian, european, african, latin,
japanese, korean, chinese, caucasian
[Subject Details] Patterns
Hair:
├── "[length]? [texture]? [color] hair"
│ Examples: "long wavy brown hair", "short blonde hair"
│
├── "hair [is/styled/worn] [description]"
│ Examples: "hair is styled in loose waves"
│
├── "[subject] has [hair description]"
│ Examples: "she has dark curly hair"
│
Length: long, short, medium, shoulder-length
Texture: wavy, curly, straight, frizzy
Color: brown, black, blonde, red, white, gray, dark, light, auburn
Eyes:
├── "[color] eyes"
│ Examples: "brown eyes", "deep blue eyes"
│
├── "eyes are [description]"
│ Examples: "eyes are looking directly at camera"
│
Color: brown, blue, green, hazel, dark, light, amber, gray
Clothing:
├── "wearing [clothing description]"
│ Examples: "wearing a white crop top"
│
├── "dressed in [clothing description]"
│ Examples: "dressed in formal attire"
│
├── "in a [clothing item]"
│ Examples: "in a red dress"
│
├── "[clothing item] with [detail]"
│ Examples: "top with Calvin Klein logo"
│
├── "paired with [clothing]"
│ Examples: "paired with matching briefs"
Clothing Terms: dress, top, shirt, blouse, pants, shorts, skirt,
bra, panties, underwear, bikini, lingerie, suit,
jacket, coat, saree, lehenga, kimono, etc.
Body:
├── "[body descriptor] physique/figure/build"
│ Examples: "slim toned physique", "athletic build"
│
├── "[subject] has a [body description]"
│ Examples: "she has a curvy figure"
Body Terms: slim, curvy, athletic, toned, petite, tall,
muscular, fit, slender
Skin/Complexion:
├── "[skin descriptor] skin/complexion"
│ Examples: "fair complexion", "olive skin"
Skin Terms: fair, pale, tan, olive, dark, brown, warm, smooth,
light, medium, deep
[Action/Pose] Patterns
├── "[verb-ing] [adverb]? [prepositional phrase]?"
│ Examples: "standing confidently", "sitting on a chair"
│
├── "looking [direction/target]"
│ Examples: "looking at the camera", "looking away"
│
├── "[body part] [position]"
│ Examples: "arms at her sides", "hands on hips"
│
├── "[expression type] expression"
│ Examples: "neutral expression", "confident expression"
Action Verbs: standing, sitting, walking, running, lying, kneeling,
posing, leaning, reclining, crouching
Gaze: looking at camera, looking directly, looking away,
looking down, looking up, looking to the side
Expression: neutral, confident, serious, playful, sensual,
relaxed, pensive, happy, smiling
[Environment/Scene] Patterns
├── "against [a/the]? [background description]"
│ Examples: "against a plain grey background"
│
├── "in [a/the]? [location]"
│ Examples: "in a studio", "in an outdoor setting"
│
├── "[location type] background/setting/scene"
│ Examples: "studio background", "outdoor scene"
│
├── "background is [description]"
│ Examples: "background is plain and light"
Location Terms: indoor, outdoor, studio, bedroom, street,
beach, forest, city, garden, office
Background Terms: plain, simple, gradient, white, grey, black,
blurred, bokeh, natural
[Lighting] Patterns
├── "[quality] light/lighting"
│ Examples: "soft lighting", "natural light"
│
├── "lighting is [description]"
│ Examples: "lighting is soft and even"
│
├── "[tone] tones"
│ Examples: "warm tones", "cool tones"
Quality: soft, hard, natural, artificial, dramatic, dim, bright,
diffused, harsh, even, flat
Tone: warm, cool, neutral, muted, vibrant, desaturated
Style: high key, low key, rim light, backlit, side lit,
front lit, Rembrandt
[Style/Medium] Patterns
├── "[style] photograph/photo/shot/shoot"
│ Examples: "photo-realistic shoot", "fashion photograph"
│
├── "[medium] style"
│ Examples: "minimalist style", "editorial style"
Medium: photograph, photo, portrait, editorial, fashion,
artistic, cinematic, fine art, commercial
Style: minimalist, modern, vintage, retro, classic,
professional, high-end, casual
[Composition] Patterns
├── "[angle] camera angle/view/shot"
│ Examples: "front camera angle", "low angle shot"
│
├── "[framing] shot"
│ Examples: "cowboy shot", "full body shot", "close-up"
Angle: front, side, back, low, high, dutch, bird's eye,
worm's eye, three-quarter
Framing: close-up, medium shot, full body, cowboy shot,
portrait, wide shot, headshot, bust shot
Rules: centered, rule of thirds, symmetry, diagonal,
golden ratio
META Patterns (Filter Out)
These patterns indicate meta-commentary that should be removed, not categorized:
FILTER PATTERNS:
├── "the image/photo/picture [shows/captures/depicts/features/has]"
├── "in the [middle/center/foreground/background] of the [image/frame]"
├── "the overall [mood/tone/atmosphere/aesthetic/look]"
├── "with a focus on"
├── "showcasing [her/his/the]"
├── "which [adds/creates/gives/lends]"
├── "conveying a sense of"
├── "creating [a/an] [sense/feeling/atmosphere] of"
├── "the viewer [can/is/may]"
├── "it is worth noting"
Action: Remove entire clause if META pattern matches anywhere in clause.
Multi-Match Handling
A single clause can match multiple categories:
INPUT CLAUSE:
"she has long brown hair and brown eyes"
EXTRACTION:
├── Match 1: "[subject] has [hair description]"
│ → [Subject Details/hair]: "long brown hair"
│
└── Match 2: "[color] eyes"
→ [Subject Details/eyes]: "brown eyes"
RESULT: Both extractions kept, clause fully consumed
Unmatched Content Handling
CLAUSE: "with the Calvin Klein logo on the front"
Pattern Match: None directly
FALLBACK STRATEGY:
1. Check if clause modifies previous extraction
→ Previous: "white crop top" [Subject Details/clothing]
→ This clause is a modifier → append to clothing
→ Result: "white crop top with the Calvin Klein logo"
2. If no modifier relationship, use keyword classification
→ "logo" suggests clothing/branding
→ Assign to [Subject Details/clothing]
3. If still unmatched, preserve in [Uncategorized]
→ Will be processed in Assembly phase
Full Example Walkthrough
INPUT:
"A photo-realistic shoot from a front camera angle about a model in a
white Calvin Klein crop top and matching briefs, photographed against
a plain light grey background. a woman standing in the middle of the
image, with a neutral expression, looking directly at the camera. she
has long, wavy brown hair and is wearing a white crop top with the
Calvin Klein logo on the front, paired with matching white briefs."
SENTENCE 1:
"A photo-realistic shoot from a front camera angle about a model in a
white Calvin Klein crop top and matching briefs, photographed against
a plain light grey background."
Clauses:
├── "A photo-realistic shoot from a front camera angle about a model"
│ → [Style/Medium]: "photo-realistic shoot"
│ → [Composition]: "front camera angle"
│ → [Subject]: "model"
│
├── "in a white Calvin Klein crop top"
│ → [Subject Details/clothing]: "white Calvin Klein crop top"
│
├── "and matching briefs"
│ → [Subject Details/clothing]: "matching briefs"
│
└── "photographed against a plain light grey background"
→ [Environment]: "plain light grey background"
SENTENCE 2:
"a woman standing in the middle of the image, with a neutral expression,
looking directly at the camera."
Clauses:
├── "a woman standing in the middle of the image"
│ → [Subject]: "a woman"
│ → [Action/Pose]: "standing"
│ → [META - FILTER]: "in the middle of the image" ← REMOVED
│
├── "with a neutral expression"
│ → [Action/Pose]: "neutral expression"
│
└── "looking directly at the camera"
→ [Action/Pose]: "looking directly at the camera"
SENTENCE 3:
"she has long, wavy brown hair and is wearing a white crop top with
the Calvin Klein logo on the front, paired with matching white briefs."
Clauses:
├── "she has long, wavy brown hair"
│ → [Subject Details/hair]: "long, wavy brown hair"
│
├── "and is wearing a white crop top with the Calvin Klein logo on the front"
│ → [Subject Details/clothing]: "white crop top with Calvin Klein logo"
│
└── "paired with matching white briefs"
→ [Subject Details/clothing]: "matching white briefs"
AGGREGATED OUTPUT:
{
"Subject": ["model", "a woman"],
"Subject Details": {
"hair": ["long, wavy brown hair"],
"clothing": [
"white Calvin Klein crop top",
"matching briefs",
"white crop top with Calvin Klein logo",
"matching white briefs"
]
},
"Action/Pose": [
"standing",
"neutral expression",
"looking directly at the camera"
],
"Environment": ["plain light grey background"],
"Style/Medium": ["photo-realistic shoot"],
"Composition": ["front camera angle"],
"Filtered (META)": ["in the middle of the image"]
}
Phase 4: Assembly
4.1 Grouping & Ranking
For each category:
tokens = filter(all_tokens, category == this_category)
# Remove duplicates within category
tokens = deduplicate(tokens)
# Rank by confidence
tokens = sort(tokens, key=confidence, descending=True)
result[category] = tokens
4.2 Output Structure
ClassifiedImage {
source_file: string
extraction_timestamp: datetime
total_tokens: int
categories: {
"Quality Boosters": [...],
"Subject": [...],
"Subject Details": {
"hair": [...],
"eyes": [...],
"body": [...],
"clothing": [...],
...
},
"Action/Pose": [...],
"Environment/Scene": [...],
"Lighting": [...],
"Style/Medium": [...],
"Composition": [...],
"Technical Parameters": [...]
},
uncategorized: [...],
filtered_meta: [...]
}
Performance Optimization
Caching Strategy
Token → Classification Cache (LRU, 10k entries)
│
├── Key: normalized_token_text
├── Value: (category, confidence, timestamp)
│
└── Benefits:
- Same tag across images → instant lookup
- Batch processing efficiency
- Reduces embedding computation
Batch Processing
For multiple images:
→ Collect unique tokens across batch
→ Classify unique tokens (batch embedding)
→ Distribute classifications back to images
Estimated Performance
| Component | Time per Token | Batch Size Impact |
|---|---|---|
| Deterministic Rules | <0.01ms | None |
| Dictionary Lookup | <0.1ms | None |
| Fuzzy Match | 1-5ms | Minor |
| Embedding + Similarity | 5-20ms | 10x speedup with batching |
| Full Pipeline (cached) | <1ms | N/A |
| Full Pipeline (cold) | 10-50ms | 5-10x speedup |
Accuracy Improvement Strategies
1. Active Learning Loop
Unclassified/Low-Confidence Tokens
→ Human Review Interface
→ Manual Classification
→ Add to Training Set
→ Retrain Classifier Weekly
→ Update Dictionaries
2. Confidence Calibration
- Track prediction accuracy per confidence bucket
- Adjust thresholds based on real-world performance
- Different thresholds per category (some are easier)
3. Category-Specific Models
For high-importance categories, train specialized classifiers:
- Subject Details → clothing classifier
- Lighting → tone/lighting classifier
- These can be more accurate than generic approach
4. Ensemble Voting
classification_1 = dictionary_lookup(token)
classification_2 = embedding_similarity(token)
classification_3 = trained_classifier(token)
final = weighted_vote([
(classification_1, 0.3),
(classification_2, 0.35),
(classification_3, 0.35)
])
Implementation Priority
| Phase | Priority | Accuracy | Effort |
|---|---|---|---|
| 1. Tokenization Pipeline | High | N/A | Medium |
| 2. Deterministic Rules (Layer 1-2) | High | 95%+ | Low |
| 3. Keyword Dictionaries (Layer 3) | High | 85%+ | Medium |
| 4. Embedding Similarity (Layer 4) | Medium | 75%+ | Medium |
| 5. Trained Classifier (Layer 5) | Low | 80%+ | High |
| 6. Caching & Optimization | Medium | N/A | Medium |
| 7. Active Learning Loop | Low | +5-10% | High |
Validation & Quality Analysis
Overview
This section defines the approach for validating whether the tokenization and classification pipeline is working correctly.
Goals:
- Completeness: Did we lose any content?
- Correctness: Is each phrase in the right category?
- Cleanliness: Did we properly filter meta-commentary?
Validation Strategy 1: Content Preservation Check
Concept
Every meaningful word/phrase from the input caption should appear in exactly one output category (or be explicitly filtered as META).
┌─────────────────────────────────────────────────────────────────┐
│ CONTENT ACCOUNTING │
│ │
│ INPUT CAPTION │
│ ───────────────── │
│ Word count: 47 │
│ Unique content words: 32 (excluding: a, the, is, and, with) │
│ │
│ OUTPUT CATEGORIES │
│ ───────────────────── │
│ Words in [Subject]: 3 │
│ Words in [Subject Details]: 12 │
│ Words in [Action/Pose]: 6 │
│ Words in [Environment]: 4 │
│ Words in [Style]: 2 │
│ Words in [Composition]: 3 │
│ Words in [META/Filtered]: 5 │
│ │
│ TOTAL ACCOUNTED: 35 │
│ UNACCOUNTED: 0 │
│ │
│ COVERAGE: 100% ✓ │
└─────────────────────────────────────────────────────────────────┘
Metrics
| Metric | Formula | Target |
|---|---|---|
| Coverage | (words in categories + filtered) / total content words | > 95% |
| Loss Rate | unaccounted words / total content words | < 5% |
| Filter Rate | filtered words / total content words | 5-15% typical |
Output: Unaccounted Words Report
UNACCOUNTED WORDS:
├── "Calvin" - appeared in input, not in any output
├── "Klein" - appeared in input, not in any output
└── "logo" - appeared in input, not in any output
DIAGNOSIS: Clothing pattern missed brand names
ACTION: Expand clothing pattern to capture brand + logo terms
Validation Strategy 2: Cross-Reference with Tags
Concept
Tags from wd14/joytag/pixai are independently generated labels. If extraction is correct, extracted phrases should align with these tags.
┌─────────────────────────────────────────────────────────────────┐
│ TAG ↔ EXTRACTION ALIGNMENT │
│ │
│ TAG SOURCE: wd14 │
│ ────────────── │
│ "brown hair" (0.915) ←→ [Subject Details/hair]: "brown hair" ✓ │
│ "long hair" (0.9) ←→ [Subject Details/hair]: "long...hair" ✓│
│ "standing" (0.581) ←→ [Action/Pose]: "standing" ✓ │
│ "brown eyes" (0.87) ←→ [Subject Details/eyes]: ??? ✗ │
│ "sports bra" (0.868) ←→ [Subject Details/clothing]: "crop top" ~│
│ │
│ ALIGNMENT SCORE: 4/5 = 80% │
│ SEMANTIC MATCH: 4.5/5 = 90% (sports bra ≈ crop top) │
└─────────────────────────────────────────────────────────────────┘
Alignment Types
| Type | Definition | Example |
|---|---|---|
| Exact Match | Tag text appears in extraction | "brown hair" ↔ "brown hair" |
| Semantic Match | Different words, same meaning | "sports bra" ↔ "crop top" |
| Partial Match | Tag is subset of extraction | "hair" ↔ "long wavy brown hair" |
| Category Match | Tag and extraction in same category | Both in [Subject Details/hair] |
| Missing | Tag has no corresponding extraction | "brown eyes" not extracted |
| Conflict | Tag and extraction in different categories | Tag says pose, extraction says clothing |
Metrics
| Metric | Formula | Target |
|---|---|---|
| Tag Coverage | tags with matching extraction / total tags | > 80% |
| Category Accuracy | correct category matches / total matches | > 90% |
| Conflict Rate | category conflicts / total matches | < 5% |
Validation Strategy 3: Category Coherence Analysis
Concept
Content within a category should be semantically related. Content across categories should be semantically distinct.
┌─────────────────────────────────────────────────────────────────┐
│ INTRA-CATEGORY COHERENCE │
│ │
│ [Subject Details/hair]: │
│ ├── "long wavy brown hair" │
│ ├── "dark hair" │
│ └── "hair styled in loose waves" │
│ │
│ Pairwise Similarity (embedding cosine): │
│ ├── (1,2): 0.82 │
│ ├── (1,3): 0.78 │
│ └── (2,3): 0.71 │
│ │
│ MEAN INTRA-SIMILARITY: 0.77 ✓ (good coherence) │
└─────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────┐
│ INTER-CATEGORY SEPARATION │
│ │
│ [Subject Details/hair] vs [Environment]: │
│ ├── "long wavy brown hair" ↔ "grey background": 0.12 │
│ ├── "dark hair" ↔ "studio setting": 0.08 │
│ │
│ MEAN INTER-SIMILARITY: 0.10 ✓ (good separation) │
└─────────────────────────────────────────────────────────────────┘
Coherence Score
Category Coherence = (Mean Intra-Similarity) / (Mean Inter-Similarity)
Good: > 3.0 (within-category 3x more similar than across)
Acceptable: 2.0 - 3.0
Poor: < 2.0 (categories not well separated)
Contamination Detection
CONTAMINATION CHECK:
─────────────────────
[Action/Pose] contains:
├── "standing confidently" ✓ (belongs here)
├── "neutral expression" ✓ (belongs here)
└── "white crop top" ✗ (should be in clothing!)
CONTAMINATION: 1 item misplaced
CONTAMINATION RATE: 1/3 = 33% ← Problem!
Validation Strategy 4: Round-Trip Reconstruction
Concept
If we extract correctly and reassemble, the meaning should be preserved.
┌─────────────────────────────────────────────────────────────────┐
│ ROUND-TRIP TEST │
│ │
│ ORIGINAL CAPTION: │
│ "A young woman with long brown hair, wearing a white dress, │
│ standing in a garden with soft natural lighting." │
│ │
│ EXTRACTED → REASSEMBLED: │
│ "A young woman. She has long brown hair. She is wearing a │
│ white dress. She is standing. The scene is set in a garden. │
│ The lighting features soft natural lighting." │
│ │
│ SEMANTIC SIMILARITY (embedding): │
│ Original ↔ Reconstructed: 0.89 │
│ │
│ VERDICT: High preservation ✓ │
└─────────────────────────────────────────────────────────────────┘
What Round-Trip Catches
| Issue | Symptom |
|---|---|
| Lost content | Similarity drops significantly |
| Wrong categorization | Reconstructed sentence sounds wrong |
| Over-filtering | Important details missing in reconstruction |
| Duplication | Same phrase appears multiple times |
Validation Strategy 5: META Filter Accuracy
Concept
Verify that filtered content is truly meta-commentary, not descriptive content.
┌─────────────────────────────────────────────────────────────────┐
│ META FILTER VALIDATION │
│ │
│ FILTERED PHRASES: │
│ ├── "in the middle of the image" ← Correct (positional meta) │
│ ├── "the image captures" ← Correct (meta preamble) │
│ ├── "showcasing her curves" ← Correct (meta commentary) │
│ └── "beautiful woman" ← INCORRECT! (descriptive content) │
│ │
│ FALSE POSITIVE: "beautiful woman" was filtered but is content │
│ ACTION: Refine filter to not catch adjective+noun phrases │
└─────────────────────────────────────────────────────────────────┘
Filter Metrics
| Metric | Definition | Target |
|---|---|---|
| True Positive Rate | Correctly filtered meta / total meta | > 95% |
| False Positive Rate | Incorrectly filtered content / total content | < 2% |
| False Negative Rate | Missed meta / total meta | < 10% |
Manual Review Sample
For validation, manually label 50-100 filtered phrases:
- ✓ Correctly filtered (true meta)
- ✗ Incorrectly filtered (was real content)
- ? Ambiguous
Validation Strategy 6: Ground Truth Benchmark
Concept
Create a manually annotated test set of 20-50 captions with correct categorizations.
┌─────────────────────────────────────────────────────────────────┐
│ GROUND TRUTH COMPARISON │
│ │
│ TEST CAPTION #7: │
│ "A woman in a red dress standing by the ocean at sunset" │
│ │
│ GROUND TRUTH (manual): SYSTEM OUTPUT: │
│ ──────────────────── ───────────────── │
│ [Subject]: "woman" [Subject]: "woman" ✓ │
│ [Clothing]: "red dress" [Clothing]: "red dress" ✓│
│ [Action]: "standing" [Action]: "standing" ✓ │
│ [Environment]: "by the ocean" [Environment]: "ocean" ~ │
│ [Lighting]: "at sunset" [Lighting]: "sunset" ✓ │
│ │
│ PRECISION: 5/5 = 100% │
│ RECALL: 4.5/5 = 90% (partial on environment) │
│ F1 SCORE: 0.95 │
└─────────────────────────────────────────────────────────────────┘
Benchmark Metrics
| Metric | Per Category | Overall |
|---|---|---|
| Precision | TP / (TP + FP) | Macro average |
| Recall | TP / (TP + FN) | Macro average |
| F1 Score | 2 * P * R / (P + R) | Macro average |
Combined Validation Report
╔═══════════════════════════════════════════════════════════════════╗
║ EXTRACTION QUALITY REPORT ║
║ Caption: florence_description ║
║ Date: 2024-12-27 ║
╠═══════════════════════════════════════════════════════════════════╣
║ ║
║ 1. CONTENT PRESERVATION ║
║ Coverage: 96.2% ✓ ║
║ Lost words: ["Calvin", "Klein"] (brand names) ║
║ ║
║ 2. TAG ALIGNMENT ║
║ Matched: 18/22 tags (81.8%) ✓ ║
║ Category accuracy: 17/18 (94.4%) ✓ ║
║ Missing: ["brown eyes", "cowboy shot", "3d", "thigh gap"] ║
║ ║
║ 3. CATEGORY COHERENCE ║
║ [Subject Details/hair]: 0.81 intra ✓ ║
║ [Subject Details/clothing]: 0.73 intra ✓ ║
║ [Action/Pose]: 0.68 intra ~ ║
║ [Environment]: 0.85 intra ✓ ║
║ Overall separation ratio: 4.2 ✓ ║
║ ║
║ 4. META FILTERING ║
║ Filtered: 3 phrases ║
║ False positives: 0 ✓ ║
║ Missed meta: 1 ("with a focus on her curves") ║
║ ║
║ 5. ROUND-TRIP SIMILARITY ║
║ Original ↔ Reconstructed: 0.87 ✓ ║
║ ║
╠═══════════════════════════════════════════════════════════════════╣
║ OVERALL SCORE: 91/100 GRADE: A- ║
║ ║
║ ISSUES TO ADDRESS: ║
║ 1. Brand name extraction (Calvin Klein) ║
║ 2. Missing eye color extraction ║
║ 3. One meta phrase not filtered ║
╚═══════════════════════════════════════════════════════════════════╝
Validation Module Structure
ValidationAnalyzer
├── ContentPreservationChecker
│ ├── tokenize_input()
│ ├── collect_output_tokens()
│ ├── compute_coverage()
│ └── report_unaccounted()
│
├── TagAlignmentChecker
│ ├── load_tags_from_metadata()
│ ├── match_tags_to_extractions()
│ ├── classify_match_type()
│ └── compute_alignment_score()
│
├── CoherenceAnalyzer
│ ├── embed_category_contents()
│ ├── compute_intra_similarity()
│ ├── compute_inter_similarity()
│ └── detect_contamination()
│
├── MetaFilterValidator
│ ├── collect_filtered_phrases()
│ ├── check_against_patterns()
│ └── flag_potential_false_positives()
│
├── RoundTripTester
│ ├── reconstruct_from_categories()
│ ├── compute_semantic_similarity()
│ └── identify_lost_content()
│
└── ReportGenerator
├── aggregate_metrics()
├── compute_overall_score()
└── generate_actionable_items()
Running Validation
INPUT:
├── Original metadata (with all captions and tags)
├── Extraction output (categorized tokens)
PROCESS:
├── Run all validation strategies
├── Aggregate results
├── Generate report
OUTPUT:
├── Quality score (0-100)
├── Per-category breakdown
├── Specific issues to fix
├── Unaccounted content list
Appendix: Canonical Categories Reference
| Category | Subcategories | Description |
|---|---|---|
| Quality Boosters | - | Technical quality terms |
| Subject | - | Who/what is the main subject |
| Subject Details | hair, eyes, face, body, clothing, skin | Appearance details |
| Action/Pose | - | What they're doing, expression, gaze |
| Environment/Scene | - | Setting, location, background |
| Lighting | - | Light quality, direction, mood, tones |
| Style/Medium | - | Artistic style, medium |
| Composition | - | Framing, camera angle, shot type |
| Technical Parameters | - | Resolution, aspect ratio, artifacts |
Phase 5: Assembly (Rule-Based Text Generation)
Overview
This phase converts classified tokens into natural, readable text without using LLMs. The approach uses:
- Token → Natural Phrase Mappings
- Category-specific templates
- Grammar rules (adjective ordering, pronoun resolution)
- Sentence fusion techniques
Assembly Pipeline
┌─────────────────────────────────────────────────────────────────┐
│ Classified Tokens (per category) │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ STEP 1: Token → Phrase Mapping │
│ │
│ "1girl" → "a woman" │
│ "looking at viewer" → "looking directly at the camera" │
│ "cowboy shot" → "framed from mid-thigh up" │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ STEP 2: Attribute Extraction & Ordering │
│ │
│ Hair: [long, wavy, brown] → ordered by [length, texture, color]│
│ Result: "long, wavy brown hair" │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ STEP 3: Template Application │
│ │
│ Template: "with {length}, {texture} {color} hair" │
│ Result: "with long, wavy brown hair" │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ STEP 4: Sentence Fusion │
│ │
│ Input: ["She has long hair", "brown hair", "wavy hair"] │
│ Output: "She has long, wavy brown hair" │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ STEP 5: Paragraph Assembly │
│ │
│ Combine sections with appropriate connectors │
│ Apply pronoun resolution │
│ Handle empty categories │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ Final Readable Text │
└─────────────────────────────────────────────────────────────────┘
Category-Specific Generation Strategies
1. [Subject] Generation
Input Tokens:
["1girl", "solo", "young woman", "mid-twenties"]
Semantic Mapping:
| Token Pattern | Role | Example |
|---|---|---|
1girl, 1boy, 1woman |
count + gender | "a woman" |
solo |
composition marker | (implies singular) |
young, mid-twenties |
age modifier | "young", "in her mid-twenties" |
2girls, multiple |
plurality | "two women" |
Token → Natural Phrase Mapping:
{
"1girl": "a woman",
"1boy": "a man",
"2girls": "two women",
"solo": null, # absorbed into singularity
"young woman": "a young woman",
"mid-twenties": "in her mid-twenties",
"model": "a model",
"asian": "Asian"
}
Template:
SUBJECT_TEMPLATE = "{article} {age_modifier} {ethnicity} {gender_noun} {age_phrase}"
Where:
- article = "a" | "an" | "" (for plurals)
- age_modifier = "young" | ""
- ethnicity = "Asian" | "" (optional, only if present)
- gender_noun = "woman" | "man" | "women" | "model"
- age_phrase = "in her mid-twenties" | ""
Generation Logic:
def generate_subject(tokens):
# Priority: most specific description wins
if "young woman" in tokens:
base = "a young woman"
elif "1girl" in tokens or "1woman" in tokens:
base = "a woman"
elif "model" in tokens:
base = "a model"
else:
base = "a person"
# Add age phrase if different from base
if "mid-twenties" in tokens and "young" not in base:
base += " in her mid-twenties"
# Add ethnicity if present
if "asian" in tokens:
base = base.replace("a ", "an Asian ")
return base
Output:
"A young woman in her mid-twenties"
2. [Subject Details] Generation
This is the most complex category with multiple subcategories.
Input Tokens (organized):
{
"hair": ["long hair", "wavy hair", "brown hair"],
"eyes": ["brown eyes", "looking at viewer"],
"face": ["lips", "parted lips", "neutral expression", "fair complexion"],
"body": ["slim", "curvy", "navel", "breasts", "medium breasts"],
"clothing": ["white crop top", "white panties", "Calvin Klein", "underwear"]
}
A. Hair Description
Ordering Rules:
HAIR_ORDER = [length, texture, color, style]
length: short, medium, long
texture: straight, wavy, curly
color: brown, blonde, black, red, etc.
style: ponytail, bun, braids, etc.
Template:
"with {length} {texture} {color} hair"
OR
"with {color} hair styled in {style}"
Token Consolidation:
Input: ["long hair", "wavy hair", "brown hair"]
Step 1: Extract attributes
length = "long"
texture = "wavy"
color = "brown"
Step 2: Apply template
"with long, wavy brown hair"
Grammar Rules:
- If multiple adjectives: use comma between different types
- Color always comes last before noun
- "long, wavy brown hair" ✓
- "brown long wavy hair" ✗
B. Eyes Description
Template:
"{color} eyes {gaze_phrase}"
Gaze Phrase Mapping:
{
"looking at viewer": "looking directly at the camera",
"looking away": "looking away",
"closed eyes": "with eyes closed",
"looking down": "gazing downward"
}
Output:
"brown eyes looking directly at the camera"
C. Face/Expression Description
Template:
"with a {expression} expression and {complexion} complexion"
OR
"with {lip_description}"
Generation:
Input: ["neutral expression", "fair complexion", "parted lips"]
Output: "with a neutral expression, fair complexion, and slightly parted lips"
D. Body Description
Filtering: Some tokens are implicit/redundant for natural description
Include vs Exclude:
Include (natural): slim, curvy, toned, athletic, petite
Exclude (tag-like): navel, breasts, thighs, stomach (unless specifically relevant)
Template:
"with a {body_type} figure"
OR
"with a {body_type} build"
Output:
"with a slim, curvy figure"
E. Clothing Description
Ordering Rules:
CLOTHING_ORDER = [upper_body, lower_body, footwear, accessories]
Grouping Logic:
Input: ["white crop top", "white panties", "Calvin Klein logo"]
Step 1: Identify garments
upper = "white crop top"
lower = "white panties"
brand = "Calvin Klein"
Step 2: Combine logically
If brand present and matches items:
"a white Calvin Klein crop top and matching panties"
Else:
"a white crop top paired with white panties"
Connector Words:
{
"matching_set": "and matching",
"contrast": "paired with",
"layered": "over",
"accessory": "with"
}
Output:
"wearing a white Calvin Klein crop top with matching white briefs"
F. Combined Subject Details Generation
Master Template:
SUBJECT_DETAILS_TEMPLATE = """
{hair_description}, {eye_description}.
She has {face_description} and {body_description}.
She is {clothing_description}.
"""
Full Output:
"with long, wavy brown hair and brown eyes looking directly at the camera.
She has a neutral expression with fair complexion and slightly parted lips,
and a slim, curvy figure. She is wearing a white Calvin Klein crop top
with matching white briefs."
3. [Action/Pose] Generation
Input Tokens:
["standing", "arms at sides", "facing viewer", "cowboy shot"]
Token → Phrase Mapping:
{
"standing": "standing",
"sitting": "sitting",
"arms at sides": "with her arms at her sides",
"hands on hips": "with her hands on her hips",
"facing viewer": "facing the camera",
"cowboy shot": null, # composition, not action
"looking at viewer": "looking at the viewer"
}
Template:
ACTION_TEMPLATE = "{primary_pose} {arm_position}, {facing_direction}"
Ordering:
1. Primary pose (standing, sitting, lying)
2. Body position (arms, legs)
3. Facing direction
4. Gaze direction (if different from facing)
Output:
"standing confidently with her arms at her sides, facing the camera"
4. [Environment/Scene] Generation
Input Tokens:
["indoor", "plain light grey background", "studio", "simple background"]
Deduplication & Priority:
Priority: specific > general
"plain light grey background" > "simple background"
"studio" + "indoor" → "indoor studio setting"
Templates:
BACKGROUND_TEMPLATES = [
"against a {background_description}",
"in {location_type} with a {background_description}",
"set against a {background_description}"
]
Output:
"against a plain, light grey background in an indoor studio setting"
5. [Lighting] Generation
Input Tokens:
["soft light", "even lighting", "high key", "neutral tones", "low contrast"]
Semantic Grouping:
light_quality: soft, hard, natural, artificial
light_style: high key, low key, dramatic
tones: neutral, warm, cool, muted
contrast: low, high
Template:
LIGHTING_TEMPLATE = "The lighting is {quality} and {style}, with {tones} and {contrast} contrast."
OR (simpler)
"lit with {quality} {style} lighting"
Natural Combinations:
{
("soft light", "even lighting"): "soft, even lighting",
("high key", "bright"): "bright, high-key lighting",
("natural light", "warm tones"): "warm natural lighting"
}
Output:
"The image features soft, even lighting with neutral tones and low contrast."
6. [Style/Medium] Generation
Input Tokens:
["photo realistic", "photograph", "minimalist aesthetic", "modern", "professional"]
Template:
STYLE_TEMPLATE = "A {style} {medium} with a {aesthetic} aesthetic."
OR
"This is a {style} {medium}, {aesthetic_description}."
Token Categories:
medium: photograph, digital art, illustration, 3D render
style: realistic, stylized, artistic
aesthetic: minimalist, modern, vintage, professional
Output:
"A photo-realistic photograph with a minimalist, modern aesthetic."
7. [Composition] Generation
Input Tokens:
["front view", "full body", "centered", "cowboy shot", "portrait orientation", "rule of thirds"]
Template:
COMPOSITION_TEMPLATE = "Shot from a {camera_angle}, showing {framing}, {composition_style}."
Mappings:
camera_angle: front view, side view, three-quarter view, from above, from below
framing: full body, upper body, close-up, medium shot, cowboy shot
composition_style: centered, rule of thirds, symmetrical
orientation: portrait, landscape
Natural Phrasing:
{
"cowboy shot": "framed from mid-thigh up",
"full body": "showing the full figure",
"front view": "from a frontal angle",
"centered": "with the subject centered",
"rule of thirds": "following the rule of thirds"
}
Output:
"Captured from a frontal angle, framed from mid-thigh up, with the subject centered in the frame."
8. [Quality Boosters] Generation
Strategy: These are typically not verbalized in natural descriptions, but can be used as prefix qualifiers.
Input Tokens:
["realistic", "sharp", "detailed", "professional quality", "high resolution"]
Template (if needed):
QUALITY_TEMPLATE = "A {quality_list} image..."
Output:
"A sharp, highly detailed, professional-quality image..."
Alternative: Integrate into Style/Medium
"A sharp, realistic photograph with professional quality..."
9. [Technical Parameters] Generation
Strategy: Usually omitted from natural descriptions, but can be appended as metadata.
Input Tokens:
["288x512", "portrait orientation", "2:3 aspect ratio"]
Template (if needed):
TECHNICAL_TEMPLATE = "Image dimensions: {width}x{height} ({orientation}, {aspect_ratio})"
Output (usually separate or omitted):
"(288×512 pixels, portrait orientation, 2:3 aspect ratio)"
Complete Assembly: Full Description Generation
Master Template Structure
FULL_DESCRIPTION_TEMPLATE = """
{quality_prefix}{style_medium}
{subject_intro} {subject_details_hair}, {subject_details_eyes}. {subject_details_face}. {subject_details_body}. {clothing_description}.
{action_pose}.
{environment_scene}. {lighting_description}.
{composition_description}.
"""
Assembly Algorithm
def generate_full_description(classified_tokens):
sections = {}
# Generate each section independently
sections['quality'] = generate_quality(classified_tokens['Quality Boosters'])
sections['style'] = generate_style(classified_tokens['Style/Medium'])
sections['subject'] = generate_subject(classified_tokens['Subject'])
sections['details'] = generate_subject_details(classified_tokens['Subject Details'])
sections['action'] = generate_action(classified_tokens['Action/Pose'])
sections['environment'] = generate_environment(classified_tokens['Environment/Scene'])
sections['lighting'] = generate_lighting(classified_tokens['Lighting'])
sections['composition'] = generate_composition(classified_tokens['Composition'])
# Assemble with appropriate connectors
paragraphs = []
# Paragraph 1: Style intro
p1 = f"{sections['quality']} {sections['style']}"
paragraphs.append(p1.strip())
# Paragraph 2: Subject description
p2 = f"{sections['subject']} {sections['details']}"
paragraphs.append(p2.strip())
# Paragraph 3: Action and environment
p3 = f"{sections['action']}. {sections['environment']}"
paragraphs.append(p3.strip())
# Paragraph 4: Technical (lighting, composition)
p4 = f"{sections['lighting']} {sections['composition']}"
paragraphs.append(p4.strip())
# Join paragraphs
return "\n\n".join(paragraphs)
Example: Full Pipeline Execution
Input (Classified Tokens)
{
"Quality Boosters": ["realistic", "sharp", "professional quality"],
"Subject": ["1girl", "solo", "young woman", "mid-twenties"],
"Subject Details": {
"hair": ["long hair", "wavy hair", "brown hair"],
"eyes": ["brown eyes", "looking at viewer"],
"face": ["neutral expression", "fair complexion", "parted lips"],
"body": ["slim", "curvy"],
"clothing": ["white crop top", "white panties", "Calvin Klein"]
},
"Action/Pose": ["standing", "arms at sides", "facing viewer"],
"Environment/Scene": ["indoor", "plain light grey background", "studio"],
"Lighting": ["soft light", "even lighting", "neutral tones", "low contrast"],
"Style/Medium": ["photograph", "photo realistic", "minimalist", "modern"],
"Composition": ["front view", "cowboy shot", "centered", "portrait orientation"]
}
Output (Generated Description)
A sharp, professional-quality photograph with a minimalist, modern aesthetic.
A young woman in her mid-twenties with long, wavy brown hair and brown eyes
looking directly at the camera. She has a neutral expression with fair complexion
and slightly parted lips, and a slim, curvy figure. She is wearing a white
Calvin Klein crop top with matching white briefs.
She stands confidently with her arms at her sides, facing the camera. The scene
is set against a plain, light grey background in an indoor studio setting.
The image features soft, even lighting with neutral tones and low contrast.
Captured from a frontal angle, framed from mid-thigh up, with the subject
centered in the frame.
Advanced Techniques for Better Output
1. Sentence Variation
Problem: Same template = repetitive output
Solution: Multiple template variants per category
SUBJECT_TEMPLATES = [
"{article} {age} {gender}",
"{article} {gender} {age_phrase}",
"{article} {ethnicity} {gender} who appears to be {age_phrase}",
]
# Random or round-robin selection
template = random.choice(SUBJECT_TEMPLATES)
2. Connector Variation
CONNECTORS = {
"addition": ["and", "with", "featuring", "along with"],
"contrast": ["but", "while", "yet"],
"elaboration": ["specifically", "particularly", "notably"]
}
# Vary connector usage
connector = random.choice(CONNECTORS["addition"])
3. Sentence Fusion
Before:
"She has long hair. Her hair is brown. Her hair is wavy."
After (fused):
"She has long, wavy brown hair."
Fusion Rules:
- Same subject + different attributes → combine attributes
- Use comma for adjective lists
- Apply adjective ordering rules
4. Pronoun Resolution
Track subject for pronoun usage:
First mention: "A young woman..."
Subsequent: "She has..." / "Her hair..." / "She is wearing..."
Gender-neutral option:
First mention: "A person..."
Subsequent: "They have..." / "Their hair..."
5. Confidence-Based Inclusion
def should_include(token, threshold=0.6):
return token.confidence >= threshold
# Only include high-confidence details
details = [t for t in tokens if should_include(t)]
6. Avoiding Redundancy
Rule: Don't repeat information
# If "young woman" is used in Subject, don't repeat "young" in details
if "young" in subject_text:
remove "young" from subject_details modifiers
Implementation Components
Required Libraries (No ML)
| Library | Purpose |
|---|---|
inflect |
Pluralization, articles (a/an) |
pattern |
Grammar, conjugation |
nltk (rules only) |
Tokenization, POS tagging (non-ML) |
Core Data Structures
# Token to natural phrase dictionary
PHRASE_MAPPINGS = {
category: {
token: natural_phrase
}
}
# Grammar rules
ADJECTIVE_ORDER = ["opinion", "size", "age", "shape", "color", "origin", "material"]
# Templates per category
TEMPLATES = {
category: [template1, template2, ...]
}
# Connectors
CONNECTORS = {
type: [variants]
}
Edge Cases & Solutions
1. Empty Categories
if not environment_tokens:
# Option A: Skip entirely
environment_text = ""
# Option B: Generic fallback
environment_text = "against a neutral background"
# Option C: Infer from other categories
if "photograph" in style_tokens:
environment_text = "in a studio setting"
2. Conflicting Information
Input: ["standing", "sitting", "lying down"] # Can't be all three
MUTUALLY_EXCLUSIVE = {
"pose": ["standing", "sitting", "lying", "kneeling"],
"gaze": ["looking at viewer", "looking away", "eyes closed"]
}
def resolve_conflicts(tokens, category):
if category in MUTUALLY_EXCLUSIVE:
# Keep highest confidence
return max(tokens, key=lambda t: t.confidence)
3. Clothing Relationship Detection
def detect_clothing_relationship(items):
colors = extract_colors(items)
if len(set(colors)) == 1:
return "matching"
elif is_formal_set(items):
return "paired_with"
else:
return "combined_with"
4. Context Passing for Pronouns
def generate_action(tokens, context):
pronoun = context.get('subject_pronoun', 'They')
return f"{pronoun} {action_phrase}"
Quality Improvements Without LLM
| Technique | Improvement | Effort |
|---|---|---|
| More templates | +Variety | Low |
| Better phrase mappings | +Naturalness | Medium |
| Adjective ordering rules | +Grammar | Low |
| Sentence fusion | +Fluency | Medium |
| Pronoun tracking | +Coherence | Low |
| Confidence thresholds | +Accuracy | Low |
| Synonym variation | +Variety | Medium |
| Paragraph structure | +Readability | Low |
Limitations vs LLM Approach
| Aspect | Rule-Based | LLM |
|---|---|---|
| Consistency | High (deterministic) | Variable |
| Speed | Very fast | Slower |
| Cost | Zero inference cost | API/GPU cost |
| Creativity | Limited to templates | High |
| Edge cases | Needs manual handling | Handles gracefully |
| Maintenance | Template updates | Prompt tuning |
| Grammar errors | Possible if rules incomplete | Rare |
Verbosity Control
VERBOSITY = {
"minimal": {
"subject_details": ["hair", "clothing"],
"max_adjectives": 2
},
"standard": {
"subject_details": ["hair", "eyes", "clothing", "body"],
"max_adjectives": 3
},
"detailed": {
"subject_details": "all",
"max_adjectives": "unlimited"
}
}
Phase 6: LLM-Based Assembly (Alternative)
Overview
This phase provides an alternative to rule-based assembly using Large Language Models for natural language generation. Use this when higher quality, more natural output is needed and latency/cost are acceptable.
When to Use LLM vs Rule-Based
| Use Case | Rule-Based | LLM |
|---|---|---|
| High volume, low variance | ✓ Best | Overkill |
| Need creative variety | Limited | ✓ Best |
| Complex token relationships | Brittle | ✓ Best |
| Edge cases / unusual combinations | Fails | ✓ Handles gracefully |
| Strict format control | ✓ Best | Needs constraints |
| Cost-sensitive | ✓ Best | Expensive at scale |
| Real-time / low latency | ✓ Best | Slower |
LLM Assembly Approaches
Approach 1: Single Comprehensive Prompt
Input Preparation:
Transform classified tokens into clean structured format:
{
"Quality Boosters": ["realistic", "sharp", "professional quality"],
"Subject": ["1girl", "solo", "young woman", "mid-twenties"],
"Subject Details": {
"hair": ["long hair", "wavy hair", "brown hair"],
"eyes": ["brown eyes", "looking at viewer"],
"face": ["neutral expression", "fair complexion", "parted lips"],
"body": ["slim", "curvy"],
"clothing": ["white crop top", "white panties", "Calvin Klein"]
},
"Action/Pose": ["standing", "arms at sides", "facing viewer"],
"Environment/Scene": ["indoor", "plain light grey background", "studio"],
"Lighting": ["soft light", "even lighting", "neutral tones", "low contrast"],
"Style/Medium": ["photograph", "photo realistic", "minimalist", "modern"],
"Composition": ["front view", "cowboy shot", "centered", "portrait orientation"]
}
System Prompt:
You are an expert image description writer. Your task is to convert structured
image metadata into natural, fluent, professional image descriptions.
RULES:
1. Write in third person, present tense
2. Be concise but comprehensive - aim for 3-4 paragraphs
3. Flow naturally - don't just list attributes
4. Prioritize important visual elements
5. Use appropriate art/photography terminology
6. Don't invent details not present in the input
7. Ignore duplicate or redundant tokens
8. Convert tag-style tokens (like "1girl") into natural language ("a woman")
STRUCTURE:
- Paragraph 1: Overall style, medium, and quality
- Paragraph 2: Subject description (appearance, clothing)
- Paragraph 3: Pose, action, and environment
- Paragraph 4: Lighting and composition (if notable)
OUTPUT FORMAT:
Return only the description text, no headers or labels.
User Prompt:
Convert the following classified image tokens into a natural, readable description:
{json_structured_tokens}
Expected Output:
A sharp, professional-quality photograph with a minimalist, modern aesthetic,
captured in a photo-realistic style.
The image features a young woman, appearing to be in her mid-twenties, with long,
wavy brown hair and brown eyes that gaze directly at the camera. She has a fair
complexion and a neutral expression, her lips slightly parted. Her figure is slim
yet curvy, and she wears a white Calvin Klein crop top paired with matching white briefs.
She stands confidently with her arms relaxed at her sides, facing the viewer
directly. The setting is an indoor studio with a plain, light grey background
that provides clean contrast to her attire.
The lighting is soft and even, creating neutral tones with low contrast that
complement the minimalist aesthetic. The shot is composed from a frontal angle,
framed from mid-thigh up in a classic cowboy shot, with the subject centered
in the portrait-oriented frame.
Approach 2: Section-by-Section Generation
Generate each section independently with focused prompts:
Subject Details Prompt:
Convert these subject detail tokens into 2-3 natural sentences describing
a person's appearance:
Hair: ["long hair", "wavy hair", "brown hair"]
Eyes: ["brown eyes", "looking at viewer"]
Face: ["neutral expression", "fair complexion", "parted lips"]
Body: ["slim", "curvy"]
Clothing: ["white crop top", "white panties", "Calvin Klein"]
Use natural flow, don't list attributes mechanically. Use "she/her" pronouns.
Advantages:
| Benefit | Explanation |
|---|---|
| Parallel execution | 5 calls simultaneously = faster total time |
| Granular control | Different prompts per section |
| Easier debugging | Isolate which section has issues |
| Caching | Cache common section outputs |
| Fallback | If one fails, others still work |
Approach 3: Few-Shot Prompting
Provide examples for consistent output style:
You convert structured image tokens into natural descriptions.
EXAMPLE 1:
Input:
{
"Subject": ["1boy", "solo"],
"Subject Details": {"hair": ["short black hair"], "clothing": ["blue suit", "red tie"]},
"Action/Pose": ["sitting", "arms crossed"],
"Environment/Scene": ["office", "window background"]
}
Output:
A man with short black hair sits with his arms crossed, dressed in a sharp blue
suit with a red tie. The setting is a professional office environment, with
windows visible in the background.
EXAMPLE 2:
Input:
{
"Subject": ["2girls"],
"Subject Details": {"hair": ["blonde hair", "black hair"], "clothing": ["summer dresses"]},
"Action/Pose": ["walking", "holding hands"],
"Environment/Scene": ["beach", "sunset"]
}
Output:
Two women walk hand in hand along the beach - one with flowing blonde hair,
the other with dark black hair. Both wear light summer dresses that catch the
warm glow of the sunset behind them.
NOW CONVERT:
Input:
{your_actual_tokens}
Output:
Approach 4: Structured Output with Schema
Force consistent structure using JSON mode:
Prompt:
Convert the image tokens into a structured description. Return JSON only.
Input tokens:
{json_tokens}
Return this exact JSON structure:
{
"opening_line": "One sentence about style/medium/quality",
"subject_description": "1-2 sentences about the main subject",
"appearance_details": "2-3 sentences about physical appearance and clothing",
"action_and_scene": "1-2 sentences about pose and environment",
"technical_notes": "1 sentence about lighting/composition (or null if unremarkable)",
"full_description": "All above combined into flowing paragraphs"
}
Output:
{
"opening_line": "A sharp, photo-realistic image with a minimalist modern aesthetic.",
"subject_description": "The photograph features a young woman in her mid-twenties.",
"appearance_details": "She has long, wavy brown hair and brown eyes that meet the camera directly. Her expression is neutral with slightly parted lips, and she has a fair complexion. Her slim, curvy figure is dressed in a white Calvin Klein crop top with matching white briefs.",
"action_and_scene": "She stands confidently with arms relaxed at her sides, facing the viewer against a plain light grey studio background.",
"technical_notes": "Soft, even lighting with neutral tones complements the centered, cowboy-shot composition.",
"full_description": "A sharp, photo-realistic image with a minimalist modern aesthetic. The photograph features a young woman in her mid-twenties with long, wavy brown hair and brown eyes that meet the camera directly. Her expression is neutral with slightly parted lips, and she has a fair complexion. Her slim, curvy figure is dressed in a white Calvin Klein crop top with matching white briefs. She stands confidently with arms relaxed at her sides, facing the viewer against a plain light grey studio background. Soft, even lighting with neutral tones complements the centered, cowboy-shot composition."
}
Benefits:
- Guaranteed structure for downstream processing
- Can use individual fields for different purposes
- Easy validation
- Enables A/B testing of different sections
Approach 5: Template + LLM Hybrid (Recommended)
Use templates for structure, LLM for natural phrasing within slots:
Template:
"{style_sentence}
{subject_intro} with {hair_description} and {eye_description}. {face_body_description}. {clothing_description}.
{action_scene_description}.
{lighting_composition_description}."
LLM Calls (Small, Focused):
Call 1: "Convert these hair tokens to a natural phrase: ['long hair', 'wavy hair', 'brown hair']"
→ "long, wavy brown hair"
Call 2: "Convert these clothing tokens to a natural sentence: ['white crop top', 'white panties', 'Calvin Klein']"
→ "She wears a white Calvin Klein crop top with matching white briefs"
... etc
Assembly:
description = TEMPLATE.format(
style_sentence=llm_style_output,
subject_intro=llm_subject_output,
hair_description=llm_hair_output,
eye_description=llm_eye_output,
# ... etc
)
Benefits:
- Controlled overall structure
- Natural language within each slot
- Cheaper (smaller prompts)
- Parallelizable
- Cacheable at slot level
Model Selection
| Model | Speed | Quality | Cost | Best For |
|---|---|---|---|---|
| Claude 3.5 Haiku | Fast | Good | Low | High volume |
| Claude 3.5 Sonnet | Medium | Excellent | Medium | Quality focus |
| GPT-4o-mini | Fast | Good | Low | High volume |
| GPT-4o | Medium | Excellent | High | Quality focus |
| Llama 3 8B (local) | Fast | Decent | Free | Cost-sensitive |
| Mistral 7B (local) | Fast | Decent | Free | Cost-sensitive |
| Qwen 2.5 7B (local) | Fast | Good | Free | Cost-sensitive |
Recommendation by Volume
| Daily Volume | Recommended Approach |
|---|---|
| < 1,000 | Single comprehensive prompt, Sonnet/GPT-4o |
| 1,000 - 10,000 | Section-by-section, Haiku/GPT-4o-mini |
| 10,000 - 100,000 | Hybrid template + LLM slots, Haiku |
| > 100,000 | Rule-based primary, LLM for edge cases only |
Prompt Engineering Best Practices
1. Token Normalization Before LLM
BEFORE sending to LLM:
- Remove duplicate tokens
- Sort by confidence (highest first)
- Limit to top N per category (prevent overload)
- Normalize formats (1girl → "woman", etc.)
2. Negative Instructions
DO NOT:
- Add information not present in the tokens
- Use phrases like "the image shows" repeatedly
- List attributes with bullet points
- Include technical tags verbatim (like "1girl" or "cowboy shot")
- Speculate about context, story, or narrative
- Add emotional interpretations unless expression tokens indicate it
3. Style Calibration
TONE: Professional, neutral, descriptive
VOICE: Third person, present tense
LENGTH: 100-200 words (adjust as needed)
VOCABULARY: Art/photography terminology appropriate
AVOID: Flowery language, excessive adjectives, repetition
4. Temperature Settings
| Use Case | Temperature |
|---|---|
| Consistent output | 0.0 - 0.3 |
| Slight variation | 0.4 - 0.6 |
| Creative variety | 0.7 - 1.0 |
Recommendation: 0.1 - 0.3 for reproducible prompt generation.
5. Confidence Integration
Include confidence in prompt for prioritization:
High confidence (>0.9): ["1girl", "brown hair", "standing"]
Medium confidence (0.7-0.9): ["wavy hair", "slim"]
Lower confidence (0.5-0.7): ["curvy", "parted lips"]
Prioritize high-confidence tokens. Include medium-confidence if relevant.
Mention lower-confidence details only if they add value.
Output Validation
Schema Validation
def validate_output(text):
checks = {
"min_length": len(text) > 50,
"max_length": len(text) < 1000,
"no_raw_tags": not re.search(r'\b1girl\b|\bcowboy shot\b', text),
"has_sentences": text.count('.') >= 2,
"no_bullets": '•' not in text and '- ' not in text,
"no_headers": not re.search(r'^#+\s', text, re.MULTILINE),
}
return all(checks.values()), checks
Semantic Validation
def semantic_check(input_tokens, output_text):
# Check key elements are mentioned
required_elements = []
if "brown hair" in input_tokens["Subject Details"]["hair"]:
required_elements.append(("brown" in output_text and "hair" in output_text))
if "standing" in input_tokens["Action/Pose"]:
required_elements.append("stand" in output_text.lower())
return all(required_elements)
Caching Strategy
┌─────────────────────────────────────────────────────────────────┐
│ Level 1: Full token hash → Complete description │
│ (Exact match cache) │
│ │
│ Level 2: Section token hash → Section text │
│ (Partial reuse) │
│ │
│ Level 3: Common phrase cache │
│ "long wavy brown hair" → "long, wavy brown hair" │
└─────────────────────────────────────────────────────────────────┘
Cache Key Generation
def generate_cache_key(tokens):
# Normalize and sort for consistent hashing
normalized = {
category: sorted(set(token.lower().strip() for token in items))
for category, items in tokens.items()
}
return hashlib.md5(json.dumps(normalized, sort_keys=True).encode()).hexdigest()
Error Handling
Fallback Chain
FALLBACK CHAIN:
1. Retry with backoff (up to 3 attempts)
2. Try alternative model (Haiku → local Llama)
3. Fall back to rule-based assembly
4. Return structured tokens as-is (worst case)
Error Types
ERROR TYPES:
├── Rate Limit
│ → Exponential backoff, queue system
│
├── Timeout (>30s)
│ → Reduce token count, simpler prompt
│
├── Content Filter Triggered
│ → Log, fallback to rule-based
│
├── Invalid JSON (structured output)
│ → Retry with stricter prompt, parse fallback
│
└── Semantic Validation Failed
→ Retry once, then rule-based fallback
Content Safety Handling
MITIGATION:
- Pre-filter tokens before sending to LLM
- Use less restrictive models for certain content
- Have rule-based fallback for filtered requests
Cost Estimation
Per-Image Cost (Approximate)
| Approach | Input Tokens | Output Tokens | Cost (Claude Haiku) |
|---|---|---|---|
| Single prompt | ~500 | ~200 | $0.0002 |
| Section-by-section (5 calls) | ~300 total | ~200 total | $0.0002 |
| Hybrid (10 slot calls) | ~200 total | ~100 total | $0.00012 |
At Scale
| Daily Volume | Monthly Cost (Haiku) | Monthly Cost (Sonnet) |
|---|---|---|
| 1,000 | ~$6 | ~$45 |
| 10,000 | ~$60 | ~$450 |
| 100,000 | ~$600 | ~$4,500 |
| 1,000,000 | ~$6,000 | ~$45,000 |
Typical ComfyUI User (50-200 images/day)
| Approach | Daily Cost | Monthly Cost |
|---|---|---|
| Haiku (single prompt) | $0.01 - $0.04 | $0.30 - $1.20 |
| GPT-4o-mini | $0.01 - $0.05 | $0.30 - $1.50 |
| Local Qwen 7B | $0 (GPU time only) | $0 |
Hybrid Decision: LLM vs Rule-Based
IF all tokens have confidence > 0.8:
Use rule-based (predictable inputs)
ELSE IF any category has unusual/low-confidence tokens:
Use LLM (handles edge cases better)
Mode Selection in Application
MODE = "Standard" → Rule-based assembly (Phase 5)
MODE = "Enhance with AI" → LLM assembly (Phase 6)
Batch Processing for High Volume
For Section-by-Section approach with 5 calls per image:
DON'T: 5 sequential API calls per image
DO: Batch all section calls across N images
- Collect 100 images
- Send all "Subject" sections in one batch
- Send all "Clothing" sections in one batch
- Recombine results
This reduces API overhead significantly.
Prompt Versioning
PROMPT_VERSIONS = {
"v1.0": original_prompt,
"v1.1": improved_prompt,
"v2.0": major_revision
}
# Track which version generated each output
metadata["prompt_version"] = "v1.1"
Essential for debugging and A/B testing.
Comparison Summary: Rule-Based vs LLM Assembly
| Aspect | Rule-Based (Phase 5) | LLM (Phase 6) |
|---|---|---|
| Output Quality | Good, predictable | Excellent, natural |
| Variety | Limited by templates | High variety |
| Edge Cases | May fail gracefully | Handles well |
| Speed | Very fast (<10ms) | Slower (500ms-3s) |
| Cost | Free | API costs |
| Consistency | Deterministic | Variable |
| Maintenance | Template updates | Prompt tuning |
| Dependencies | None (pure Python) | API/GPU required |
Phase 7: Token Grounding Visualization (Optional)
Overview
This phase provides visual debugging and QA capabilities by showing WHERE in the image each token/tag spatially corresponds. This is useful for:
- Validating classification accuracy
- Debugging token extraction
- Understanding model attention
- User-facing visualization features
Priority: LOW (implement after core pipeline is complete)
Grounding Approaches
Approach 1: CLIP-Based Attention Heatmaps
Concept: CLIP encodes images and text into shared embedding space. Extract attention maps to see where the model "looks" for each token.
Image → CLIP Vision Encoder → Attention Maps (per patch)
Token → CLIP Text Encoder → Text Embedding
For each token:
Compute similarity between text embedding and each image patch
→ Heatmap of patch-level similarities
Methods:
- Direct patch similarity
- GradCAM on CLIP similarity score
Tools: CLIP-ViT, pytorch-grad-cam
| Pros | Cons |
|---|---|
| Fast (~20ms GPU) | Low spatial precision |
| Works for abstract concepts | Gradient-based approximation |
| Minimal overhead | Noisy for small regions |
Best For: Abstract tokens (lighting, style, quality) that can't be precisely localized.
Approach 2: Grounding DINO + SAM
Concept: Two-stage approach - Grounding DINO finds bounding boxes for text queries, SAM refines to precise masks.
┌─────────────────────────────────────────────────────────────────┐
│ Token: "white crop top" │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ GROUNDING DINO │ │
│ │ Input: Image + "white crop top" │ │
│ │ Output: Bounding box [(x1,y1), (x2,y2)] │ │
│ └─────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ SAM (Segment Anything) │ │
│ │ Input: Image + Box prompt │ │
│ │ Output: Precise segmentation mask │ │
│ └─────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────┘
Token → Prompt Conversion:
| Category | Token | Grounding Prompt |
|---|---|---|
| Clothing | "white crop top" | "white crop top" |
| Hair | "brown hair" | "hair" |
| Environment | "grey background" | "background" |
| Action/Pose | "arms at sides" | "arms" |
| Pros | Cons |
|---|---|
| Precise segmentation | Slower (two models) |
| State-of-the-art accuracy | ~1.1GB model overhead |
| Handles complex scenes | Abstract concepts can't be grounded |
Best For: Objects, clothing, body parts, props.
Approach 3: Semantic Segmentation + Token Mapping
Concept: Use human parsing model to get pixel-level labels, map tokens to segment classes.
┌─────────────────────────────────────────────────────────────────┐
│ HUMAN PARSING MODEL (SCHP / Graphonomy) │
│ │
│ Input: Image │
│ Output: Per-pixel class labels │
│ │
│ Classes: hair, face, upper_clothes, lower_clothes, arms, │
│ legs, shoes, background, skin, etc. │
└─────────────────────────────────────────────────────────────────┘
Token → Segment Class Mapping:
TOKEN_TO_SEGMENT = {
# Hair tokens
"long hair": "hair",
"brown hair": "hair",
"wavy hair": "hair",
# Face tokens
"brown eyes": "eyes",
"lips": "lips",
"parted lips": "lips",
# Body tokens
"navel": "torso",
"breasts": "torso",
"arms at sides": "arms",
# Clothing tokens
"white crop top": "upper_clothes",
"white panties": "lower_clothes",
"bra": "upper_clothes",
# Background
"grey background": "background",
}
| Pros | Cons |
|---|---|
| Very precise for known classes | Limited to predefined classes |
| Fast inference (~50ms) | Requires class mapping dictionary |
| Consistent results | Can't handle novel concepts |
Best For: Subject Details subcategories (hair, clothing, body, face).
Approach 4: Florence-2 Grounding
Concept: Florence-2 has built-in grounding - can output bounding boxes for concepts it describes.
Florence-2 Tasks:
├── <CAPTION_TO_PHRASE_GROUNDING>
│ Input: "brown hair, white top, standing"
│ Output: Bounding boxes for each phrase
│
├── <REFERRING_EXPRESSION_SEGMENTATION>
│ Input: "the woman's hair"
│ Output: Segmentation mask
│
└── <OPEN_VOCABULARY_DETECTION>
Input: "crop top"
Output: Detection boxes
| Pros | Cons |
|---|---|
| Already loaded for Analysis | Grounding quality varies |
| Multi-task capable | Not all variants support grounding |
| Zero additional models | Box-level (not mask) |
Best For: Leveraging existing Florence-2 model with minimal overhead.
Recommended Strategy: Category-Based Router
def select_grounding_method(token, category):
# Subject Details - use parsing models
if category == "Subject Details":
subcategory = token.metadata.get("subcategory")
if subcategory in ["hair", "face", "body", "clothing"]:
return "human_parsing"
else:
return "grounding_dino"
# Environment - use scene parsing or grounding
elif category == "Environment/Scene":
if "background" in token.text:
return "scene_parsing"
else:
return "grounding_dino"
# Action/Pose - use pose estimation
elif category == "Action/Pose":
return "pose_estimation"
# Abstract concepts (can't ground spatially)
elif category in ["Lighting", "Style/Medium", "Quality Boosters"]:
return "clip_attention" # or skip
# Default fallback
else:
return "grounding_dino"
Visualization Strategies
1. Single Token Heatmap
Overlay mask/heatmap for one token at a time.
┌─────────────────────────────────────────────────────────────────┐
│ Image with "brown hair" highlighted │
│ │
│ ████████████ │
│ ██ HAIR MASK ██ │
│ ████████████ │
│ │
│ [Person] │
│ │
│ Token: brown hair (0.915) │
│ Category: Subject Details > Hair │
│ Source: wd14 │
└─────────────────────────────────────────────────────────────────┘
2. Multi-Token Composite
Color-coded overlays for multiple tokens, grouped by category.
Color Scheme:
├── Hair tokens: Yellow
├── Clothing tokens: Blue
├── Face tokens: Pink
├── Body tokens: Orange
├── Environment: Green
└── Action/Pose: Purple
3. Interactive HTML Viewer
<div id="viewer">
<img src="base_image.png" id="base">
<div class="overlay" data-token="brown_hair">
<img src="mask_hair.png">
</div>
</div>
<div id="token-list">
<label><input type="checkbox" data-token="brown_hair"> brown hair (0.91)</label>
<label><input type="checkbox" data-token="crop_top"> white crop top (0.87)</label>
</div>
<div id="category-filter">
[All] [Subject] [Details] [Environment] [Action]
</div>
Handling Non-Groundable Tokens
Some tokens can't be spatially grounded:
| Category | Examples | Strategy |
|---|---|---|
| Quality Boosters | "realistic", "sharp", "4k" | Global indicator (badge/icon) |
| Style/Medium | "photograph", "minimalist" | Global indicator |
| Lighting | "soft light", "high key" | Whole-image tint or skip |
| Composition | "centered", "rule of thirds" | Grid overlay |
| Technical | "portrait orientation" | Frame indicator |
Output Formats
1. Static Image Export
Output files:
├── image_original.png
├── image_heatmap_hair.png
├── image_heatmap_clothing.png
├── image_heatmap_combined.png
└── image_with_legend.png
2. JSON Annotation Format
{
"image": "image_001.jpg",
"dimensions": [512, 288],
"token_regions": [
{
"token": "brown hair",
"category": "Subject Details > Hair",
"confidence": 0.915,
"source": "wd14",
"region": {
"type": "mask",
"rle": "...",
"bbox": [120, 50, 280, 180],
"area": 12500,
"centroid": [200, 115]
},
"grounding_method": "human_parsing",
"grounding_confidence": 0.92
}
],
"ungrounded_tokens": [
{"token": "realistic", "reason": "abstract_concept"},
{"token": "soft light", "reason": "global_attribute"}
]
}
Recommended Implementation Stack
| Component | Tool | Purpose | Size |
|---|---|---|---|
| Human Parsing | SCHP or Graphonomy |
Body, clothes, hair | ~150MB |
| Face Parsing | face-parsing.PyTorch |
Eyes, lips, nose | ~100MB |
| General Grounding | Grounding DINO |
Objects, props | ~700MB |
| Mask Refinement | SAM (ViT-B) |
Bounding box → mask | ~400MB |
| Fallback | CLIP + GradCAM |
Abstract concepts | ~350MB |
| Visualization | matplotlib + OpenCV |
Overlay composition | - |
Performance Estimates
| Model | GPU | CPU |
|---|---|---|
| SCHP (human parsing) | 50ms | 500ms |
| Face parsing | 30ms | 300ms |
| Grounding DINO | 100ms | 2s |
| SAM (ViT-B) | 50ms | 1s |
| CLIP ViT-B/32 | 20ms | 200ms |
Full pipeline per image: ~300-500ms (GPU), ~3-5s (CPU)
Integration with Validation
Token grounding can enhance validation (Phase 6) by:
- Spatial Coherence Check: Tokens in same category should have overlapping or adjacent regions
- Coverage Verification: Subject Details tokens should cover expected body areas
- Conflict Detection: Mutually exclusive tokens (standing vs sitting) should have non-overlapping regions
- Visual QA Report: Generate annotated images showing classification accuracy
Implementation Priority
This phase is optional and should only be implemented after the core pipeline (Phases 1-6) is complete and validated.
| Task | Priority | Effort |
|---|---|---|
| Human parsing integration | Medium | 4-6 hours |
| Grounding DINO + SAM | Low | 6-8 hours |
| Interactive HTML viewer | Low | 4-6 hours |
| JSON export format | Medium | 2-3 hours |
| Integration with validation | Low | 3-4 hours |