[bugfix]: preserve tokenizer_kwargs customizations across update_model_arch
TextEncoderArchConfig.__post_init__ rebuilt tokenizer_kwargs from scratch, and TextEncoderLoader.load -> update_model_arch re-runs __post_init__, silently wiping customizations applied by pipeline configs (kandinsky5/gen3c/longcat set 'padding'; multi-prompt kandinsky5 runs then crash on ragged sequences). Merge defaults under existing keys instead of rebuilding. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
committed by
SolitaryThinker
co-authored by
Claude Fable 5
parent
0434e9dcf0
commit
4eb4e252eb
@@ -43,11 +43,16 @@ class TextEncoderArchConfig(EncoderArchConfig):
|
||||
require_processor: bool = False
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
self.tokenizer_kwargs = {
|
||||
# update_model_arch re-runs __post_init__ after pipeline configs may
|
||||
# have customized tokenizer_kwargs (e.g. kandinsky5/gen3c/longcat set
|
||||
# "padding"); rebuilding the dict here would silently wipe those
|
||||
# customizations, so only fill in defaults for keys not already set.
|
||||
defaults = {
|
||||
"truncation": True,
|
||||
"max_length": self.text_len,
|
||||
"return_tensors": "pt",
|
||||
}
|
||||
self.tokenizer_kwargs = defaults | self.tokenizer_kwargs
|
||||
|
||||
|
||||
@dataclass
|
||||
|
||||
Reference in New Issue
Block a user