Files
olivv-cs-ComfyUI-FunPack/tests
digital-garbageandClaude Sonnet 5 1e4553e658 fix: H3 attention decoupling was scaling K magnitude, which barely moves softmax
Live 0-1 strength sweep came back flat -- confirmed the mechanism, not just the
value, was wrong. H3's Q/K get RMSNorm'd before attention, so scaling a key
toward zero pushes its logit toward the *mean* of an already near-zero-centered
distribution, not toward excluded. Switched to what actually suppresses a
softmax entry: an additive negative bias on the targeted logits, still passed
through the same per-query-group split (still no S^2 mask, still same total
FLOPs -- the bias is a per-key vector broadcast over the group's queries).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-05 02:23:30 +03:00
..