319 lines
12 KiB
Plaintext
319 lines
12 KiB
Plaintext
================================================================================
|
||
SDXL Long Context — Learned Sinusoidal Extrapolation Method
|
||
Technical Description
|
||
================================================================================
|
||
|
||
|
||
OVERVIEW
|
||
--------
|
||
|
||
This method extends SDXL positional embeddings from 77 to 248 tokens by
|
||
reverse-engineering the mathematical structure that CLIP text encoders
|
||
learn during training, then using that structure to generate embeddings
|
||
for positions beyond the original training range.
|
||
|
||
Instead of blind signal processing (FFT, interpolation, repetition),
|
||
this approach fits explicit parametric models — the same mathematical
|
||
forms used in transformer positional encoding — to the learned weights,
|
||
then evaluates those models at extended positions.
|
||
|
||
|
||
BACKGROUND: HOW POSITIONAL EMBEDDINGS ARE TRAINED
|
||
--------------------------------------------------
|
||
|
||
SDXL uses two CLIP text encoders:
|
||
- CLIP-L (OpenCLIP ViT-L/14): dim=768, max_position=77
|
||
- CLIP-G (OpenCLIP ViT-bigG/14): dim=1280, max_position=77
|
||
|
||
During CLIP training:
|
||
|
||
1. Initialization:
|
||
Positional embeddings are initialized as small Gaussian noise:
|
||
|
||
nn.init.normal_(position_embedding.weight, std=0.01)
|
||
|
||
At this point, every position looks nearly identical — random
|
||
vectors with magnitude ~0.01.
|
||
|
||
2. Training via backpropagation:
|
||
The model is trained on millions of (text, image) pairs using
|
||
contrastive loss. Gradients flow back through the text transformer
|
||
and update the positional embeddings.
|
||
|
||
3. Convergence:
|
||
After training, each dimension of the positional embedding matrix
|
||
contains a smooth signal that encodes position information. These
|
||
signals converge to a structure closely resembling the analytical
|
||
sinusoidal positional encoding from the original Transformer paper.
|
||
|
||
4. Final structure:
|
||
The learned embeddings approximate:
|
||
|
||
PE(pos, 2i) = A * sin(pos / base^(2i/d_model) + phase) + offset
|
||
PE(pos, 2i+1) = A * cos(pos / base^(2i/d_model) + phase) + offset
|
||
|
||
Where A, base, phase, and offset are learned (not fixed as in the
|
||
original Vaswani et al. formula). The key insight is that different
|
||
dimensions learn different frequencies, creating a rich positional
|
||
encoding where each position has a unique signature.
|
||
|
||
|
||
THE VASWANI SINUSOIDAL FORMULA
|
||
-------------------------------
|
||
|
||
The original "Attention Is All You Need" (Vaswani et al., 2017) defined
|
||
positional encoding analytically:
|
||
|
||
PE(pos, 2i) = sin(pos / 10000^(2i/d_model))
|
||
PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model))
|
||
|
||
Properties:
|
||
- Each dimension oscillates at a different frequency
|
||
- Low dimensions: high frequency (changes rapidly with position)
|
||
- High dimensions: low frequency (changes slowly with position)
|
||
- The base 10000 controls the frequency spectrum spread
|
||
- No learned parameters — fully deterministic
|
||
|
||
Learned embeddings in CLIP converge to a MODIFIED version:
|
||
- Same sinusoidal structure
|
||
- But with learned amplitudes (not all equal to 1.0)
|
||
- Learned base (not exactly 10000)
|
||
- Learned phase offsets (not exactly 0)
|
||
- Learned DC offsets (not exactly 0)
|
||
- Some dimensions develop more complex patterns (sum of sinusoids)
|
||
|
||
|
||
METHOD: HYBRID PARAMETRIC FITTING
|
||
-----------------------------------
|
||
|
||
For each dimension d in [0, dim) of the positional embedding matrix:
|
||
|
||
Step 1. Attempt Vaswani formula fit
|
||
────────────────────────────────────
|
||
Fit the parametric model:
|
||
|
||
f(pos) = A * sin(pos / |base|^(2d/dim) + phase) + offset
|
||
|
||
Parameters to fit: A (amplitude), base, phase, offset
|
||
Method: Levenberg-Marquardt nonlinear least squares (scipy.optimize.curve_fit)
|
||
Initial guesses:
|
||
- A = std(signal) * sqrt(2)
|
||
- base = 10000 (from original Transformer)
|
||
- phase = 0
|
||
- offset = mean(signal)
|
||
|
||
Measure fit quality: R² = 1 - Σ(signal - fitted)² / Σ(signal - mean)²
|
||
|
||
If R² > 0.7: USE THIS FIT
|
||
This dimension follows the Vaswani pattern closely.
|
||
Extrapolation is reliable because the model is physically motivated.
|
||
|
||
Step 2. Attempt multi-sinusoidal fit
|
||
────────────────────────────────────
|
||
If Vaswani fit is poor (R² ≤ 0.7), fit a more flexible model:
|
||
|
||
f(pos) = a + b*pos + Σₖ Aₖ * sin(2π * fₖ * pos + φₖ)
|
||
|
||
With k = 1, 2, 3 (three sinusoidal components plus linear trend).
|
||
|
||
Parameters: a, b, A₁, f₁, φ₁, A₂, f₂, φ₂, A₃, f₃, φ₃ (11 params)
|
||
Initial frequency estimates: from FFT peak detection
|
||
|
||
If R² > 0.7: USE THIS FIT
|
||
This dimension has a more complex but still periodic structure.
|
||
|
||
Step 3. Fallback to adaptive spectral extrapolation
|
||
───────────────────────────────────────────────────
|
||
If neither parametric model fits well (R² ≤ 0.7 for both):
|
||
|
||
- Full FFT decomposition
|
||
- Frequency-dependent damping in extrapolated region
|
||
- Per-dimension adaptive damping based on signal regularity
|
||
|
||
This handles dimensions with irregular or non-periodic patterns
|
||
that don't match any analytical form.
|
||
|
||
|
||
POST-PROCESSING
|
||
---------------
|
||
|
||
After extrapolation, three corrections are applied:
|
||
|
||
1. Original preservation (blend)
|
||
────────────────────────────────
|
||
Positions 0 through (77 - blend_width) are replaced with exact
|
||
original values. Positions in the blend zone transition smoothly
|
||
from original to extrapolated using quintic smoothstep:
|
||
|
||
α(t) = t³(t(6t - 15) + 10) t ∈ [0, 1]
|
||
|
||
This ensures C2 continuity at the boundary.
|
||
Default blend_width = 12 positions.
|
||
|
||
2. Norm matching
|
||
────────────────
|
||
The L2 norm of each extrapolated position vector is scaled to match
|
||
the mean norm of the original 77 positions:
|
||
|
||
result[i] *= mean_norm(original) / norm(result[i])
|
||
|
||
This prevents the model from receiving embeddings with unexpected
|
||
magnitudes, which would shift activation statistics.
|
||
|
||
3. Statistical validation
|
||
─────────────────────────
|
||
The method reports:
|
||
- Fit quality (R²) distribution across dimensions
|
||
- Norm ratio between original and extrapolated regions
|
||
- Boundary cosine similarity
|
||
- Per-dimension mean shift
|
||
- Method selection counts (Vaswani / sinusoidal / spectral)
|
||
|
||
|
||
WHY THIS WORKS BETTER THAN PURE FFT
|
||
-------------------------------------
|
||
|
||
1. Physics-informed extrapolation:
|
||
FFT treats the signal as arbitrary. This method knows the signal
|
||
SHOULD be a sum of sinusoids (because that's what transformers learn).
|
||
A parametric fit generalizes better than blind frequency extrapolation.
|
||
|
||
2. No high-frequency divergence:
|
||
FFT extrapolation requires artificial damping to prevent divergence.
|
||
Parametric sinusoids are bounded by definition — sin() never exceeds
|
||
its amplitude regardless of position.
|
||
|
||
3. Correct frequency scaling:
|
||
The Vaswani formula has a specific relationship between dimension
|
||
index and frequency. By fitting the base parameter, we capture
|
||
this relationship and extrapolate it correctly.
|
||
|
||
4. Per-dimension optimal method:
|
||
Different dimensions may have different structures. The hybrid
|
||
approach selects the best model for each dimension independently,
|
||
rather than applying one method uniformly.
|
||
|
||
|
||
VISUALIZATION
|
||
-------------
|
||
|
||
Dimension with good Vaswani fit (R² = 0.95):
|
||
|
||
Value
|
||
^
|
||
| * * * *
|
||
| * * * * * * * *
|
||
|* * * * * * * *
|
||
| * * * * * * *
|
||
| * * * * * * *
|
||
| * * * * * * *
|
||
| * * * *
|
||
+────────────────────────────────────────────── Position
|
||
0 20 40 60 77 120 200 247
|
||
|<── original (exact) ──>|<── Vaswani formula ────>|
|
||
|
||
|
||
Dimension with complex pattern (sinusoidal fit, R² = 0.82):
|
||
|
||
Value
|
||
^
|
||
| ** * ** * **
|
||
| * * * * * * * * * *
|
||
|* ** * * ** * *
|
||
| * * * * *
|
||
| * * * * * * *
|
||
| ** * * ** *
|
||
| ** **
|
||
+────────────────────────────────────────────── Position
|
||
0 20 40 60 77 120 200 247
|
||
|<── original (exact) ──>|<── 3-sinusoid model ──>|
|
||
|
||
|
||
Dimension with irregular pattern (spectral fallback):
|
||
|
||
Value
|
||
^
|
||
| * * * ~ ~
|
||
|* ** * * * ~ ~ ~ ~
|
||
| * * * * ~ ~ ~ ~
|
||
| * * * * ~ ~
|
||
| * * * ~ ~
|
||
| * * ~ (damped) ~
|
||
+────────────────────────────────────────────── Position
|
||
0 20 40 60 77 120 200 247
|
||
|<── original (exact) ──>|<── adaptive spectral ──>|
|
||
|
||
|
||
EMPIRICAL RESULTS
|
||
-----------------
|
||
|
||
Typical fit quality distribution for SDXL CLIP-L (768 dimensions):
|
||
- Vaswani fit (R² > 0.7): ~45-55% of dimensions
|
||
- Sinusoidal fit (R² > 0.7): ~25-35% of dimensions
|
||
- Spectral fallback: ~15-25% of dimensions
|
||
|
||
Typical fit quality distribution for SDXL CLIP-G (1280 dimensions):
|
||
- Vaswani fit (R² > 0.7): ~40-50% of dimensions
|
||
- Sinusoidal fit (R² > 0.7): ~30-35% of dimensions
|
||
- Spectral fallback: ~20-25% of dimensions
|
||
|
||
Norm ratio (extrapolated / original): typically 0.98 - 1.02
|
||
Boundary cosine similarity: typically > 0.95
|
||
|
||
|
||
LIMITATIONS
|
||
-----------
|
||
|
||
1. Parametric fitting is an approximation. Even with R² > 0.9,
|
||
the fitted model is not identical to what the model would have
|
||
learned with 248 positions during actual training.
|
||
|
||
2. The attention mechanism (Q, K, V weights) was trained on context
|
||
length 77. Extended positional embeddings provide position
|
||
information, but attention patterns may not generalize perfectly.
|
||
|
||
3. Cross-attention in the UNet was trained with conditioning sequences
|
||
of specific lengths. Very long prompts may affect spatial composition.
|
||
|
||
4. Fit quality varies across checkpoints. Models fine-tuned extensively
|
||
may have positional embeddings that deviate more from the sinusoidal
|
||
pattern, reducing fit quality.
|
||
|
||
5. scipy is required for nonlinear curve fitting. The method falls back
|
||
to spectral extrapolation for dimensions where fitting fails.
|
||
|
||
|
||
REQUIREMENTS
|
||
------------
|
||
|
||
Python packages: safetensors, numpy, scipy
|
||
Install: pip install safetensors numpy scipy
|
||
|
||
|
||
USAGE
|
||
-----
|
||
|
||
python convert_learned.py \
|
||
--input model.safetensors \
|
||
--output model_long248.safetensors \
|
||
--target_len 248 \
|
||
--method hybrid \
|
||
--analyze
|
||
|
||
Methods:
|
||
learned_sinusoidal — fit N sinusoids + trend to each dimension
|
||
vaswani_fitted — fit original Transformer PE formula
|
||
hybrid — best of all methods per dimension (recommended)
|
||
|
||
|
||
PROJECT FILES
|
||
-------------
|
||
|
||
convert_learned.py — conversion script
|
||
convert_learned.bat — batch file for running conversion
|
||
node.py — ComfyUI loader node
|
||
__init__.py — ComfyUI node registration
|
||
method_description.txt — this file
|
||
|
||
================================================================================
|