nodes/chflame163-ComfyUI_LayerStyle_Advance
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Add slow tokenizer fallback for models with incompatible added_tokens

Browse Source 
Some Florence2 variants (e.g. Florence-2-Flux-Large) have added_tokens
in a format that fast tokenizers in transformers 5.x cannot parse.
Fall back to slow tokenizer on TypeError.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Gang Wen
2026-04-07 19:59:26 +02:00
co-authored by Claude Opus 4.6
parent 8d1bfd3874
commit aa016d91de
1 changed files with 5 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
py/florence2_ultra.py
+5 -1
View File
@@ -108,7 +108,11 @@ def _load_model_v5(model_path, attention, dtype):
image_processor.image_seq_length = 577
log(f"[DEBUG] Loading tokenizer from {model_path}")
tokenizer = AutoTokenizer.from_pretrained(model_path)
try:
tokenizer = AutoTokenizer.from_pretrained(model_path)
except TypeError as e:
log(f"[DEBUG] Fast tokenizer failed ({e}), trying slow tokenizer", message_type='warning')
tokenizer = AutoTokenizer.from_pretrained(model_path, use_fast=False)
log(f"[DEBUG] Creating Florence2Processor")
processor = Florence2Processor(image_processor=image_processor, tokenizer=tokenizer)
log(f"[DEBUG] _load_model_v5 completed successfully")