Commit Graph
4 Commits
Author SHA1 Message Date
slahiriandClaude Opus 4.5 ec05fb561f fix: detect and clean garbage LLM output
- Add remove_runaway_repetitions() to clean "dark, dark, dark..." patterns
- Add is_garbage_output() to detect LLM generation failures
- LLM mode now falls back to NLP output when garbage is detected
- Limits any word to max 3 repetitions in output

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-27 20:52:01 +05:30
slahiriandClaude Opus 4.5 d11328ff90 feat: add STRUCTURED prompt style with category-segregated output
- Add STRUCTURED style that outputs each category on separate lines
- Fix '3d' token stripping - only strip booru patterns (1girl, 2boys)
- Increase max_tokens_per_category to 50 (was 5)
- Increase validator limits to preserve more information
- Preserve newlines throughout validation pipeline
- Disable truncation and duplicate removal by default

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-27 19:05:47 +05:30
slahiriandClaude Opus 4.5 700607305b feat(compose): implement Phase 5 - Prompt Validation
Adds the validator module for quality analysis of assembled prompts:
- base.py: ValidatorConfig, ValidationResult, ValidationIssue classes
- rules.py: Conflict detection, duplicate finding, quality checks
- standard.py: StandardValidator with auto-correction

Validates:
- Length constraints (CLIP token limits)
- Duplicate words/phrases detection
- Conflicting terms (e.g., girl vs woman)
- Missing required categories
- Quality issues (punctuation, spacing)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-27 11:41:40 +05:30
slahiriandClaude Opus 4.5 c7859cf9f4 feat(compose): implement Phase 1 - Core Infrastructure
New data structures for tokenization pipeline:

tokenizer/base.py:
- ImageToken: unified token format (text, confidence, source, type)
- TokenType enum: TAG, PHRASE, KEY_VALUE, SENTENCE
- TokenSource enum: WD14, JOYTAG, PIXAI, FLORENCE_*, etc.
- TokenBatch: collection with filtering methods
- SOURCE_WEIGHTS: reliability weighting per source

classifier/categories.py:
- CanonicalCategory enum: 9 categories + UNCATEGORIZED, FILTERED_META
- SubjectDetailType enum: hair, eyes, face, body, clothing, skin, accessories
- CategoryKeywords: exact/prefix/suffix/contains pattern matching
- Keyword dictionaries for all categories
- META_FILTER_PATTERNS: patterns to filter out
- FLORENCE_KEY_TO_CATEGORY: key routing for florence_analyze

classifier/base.py:
- TokenClassification: classification result with primary/secondary categories
- ClassifiedImage: organized tokens by category for assembly

Tests:
- test_phase1_infrastructure.py: 21 unit tests (all passing)
- test_real_image.py: real metadata token extraction demo
- sample_metadata*.json: real image metadata samples

All 21 tests pass. Ready for Phase 2: Tokenization.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-27 09:51:09 +05:30