[bugfix]: preserve tokenizer_kwargs customizations across update_model_arch

TextEncoderArchConfig.__post_init__ rebuilt tokenizer_kwargs from scratch, and
TextEncoderLoader.load -> update_model_arch re-runs __post_init__, silently
wiping customizations applied by pipeline configs (kandinsky5/gen3c/longcat
set 'padding'; multi-prompt kandinsky5 runs then crash on ragged sequences).
Merge defaults under existing keys instead of rebuilding.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Will Lin
2026-07-05 14:58:12 -07:00
committed by SolitaryThinker
co-authored by Claude Fable 5
parent 0434e9dcf0
commit 4eb4e252eb
+6 -1
View File
@@ -43,11 +43,16 @@ class TextEncoderArchConfig(EncoderArchConfig):
require_processor: bool = False
def __post_init__(self) -> None:
self.tokenizer_kwargs = {
# update_model_arch re-runs __post_init__ after pipeline configs may
# have customized tokenizer_kwargs (e.g. kandinsky5/gen3c/longcat set
# "padding"); rebuilding the dict here would silently wipe those
# customizations, so only fill in defaults for keys not already set.
defaults = {
"truncation": True,
"max_length": self.text_len,
"return_tensors": "pt",
}
self.tokenizer_kwargs = defaults | self.tokenizer_kwargs
@dataclass