Add files via upload
This commit is contained in:
Alexey
@@ -0,0 +1,318 @@
|
||||
================================================================================
|
||||
SDXL Long Context — Learned Sinusoidal Extrapolation Method
|
||||
Technical Description
|
||||
================================================================================
|
||||
|
||||
|
||||
OVERVIEW
|
||||
--------
|
||||
|
||||
This method extends SDXL positional embeddings from 77 to 248 tokens by
|
||||
reverse-engineering the mathematical structure that CLIP text encoders
|
||||
learn during training, then using that structure to generate embeddings
|
||||
for positions beyond the original training range.
|
||||
|
||||
Instead of blind signal processing (FFT, interpolation, repetition),
|
||||
this approach fits explicit parametric models — the same mathematical
|
||||
forms used in transformer positional encoding — to the learned weights,
|
||||
then evaluates those models at extended positions.
|
||||
|
||||
|
||||
BACKGROUND: HOW POSITIONAL EMBEDDINGS ARE TRAINED
|
||||
--------------------------------------------------
|
||||
|
||||
SDXL uses two CLIP text encoders:
|
||||
- CLIP-L (OpenCLIP ViT-L/14): dim=768, max_position=77
|
||||
- CLIP-G (OpenCLIP ViT-bigG/14): dim=1280, max_position=77
|
||||
|
||||
During CLIP training:
|
||||
|
||||
1. Initialization:
|
||||
Positional embeddings are initialized as small Gaussian noise:
|
||||
|
||||
nn.init.normal_(position_embedding.weight, std=0.01)
|
||||
|
||||
At this point, every position looks nearly identical — random
|
||||
vectors with magnitude ~0.01.
|
||||
|
||||
2. Training via backpropagation:
|
||||
The model is trained on millions of (text, image) pairs using
|
||||
contrastive loss. Gradients flow back through the text transformer
|
||||
and update the positional embeddings.
|
||||
|
||||
3. Convergence:
|
||||
After training, each dimension of the positional embedding matrix
|
||||
contains a smooth signal that encodes position information. These
|
||||
signals converge to a structure closely resembling the analytical
|
||||
sinusoidal positional encoding from the original Transformer paper.
|
||||
|
||||
4. Final structure:
|
||||
The learned embeddings approximate:
|
||||
|
||||
PE(pos, 2i) = A * sin(pos / base^(2i/d_model) + phase) + offset
|
||||
PE(pos, 2i+1) = A * cos(pos / base^(2i/d_model) + phase) + offset
|
||||
|
||||
Where A, base, phase, and offset are learned (not fixed as in the
|
||||
original Vaswani et al. formula). The key insight is that different
|
||||
dimensions learn different frequencies, creating a rich positional
|
||||
encoding where each position has a unique signature.
|
||||
|
||||
|
||||
THE VASWANI SINUSOIDAL FORMULA
|
||||
-------------------------------
|
||||
|
||||
The original "Attention Is All You Need" (Vaswani et al., 2017) defined
|
||||
positional encoding analytically:
|
||||
|
||||
PE(pos, 2i) = sin(pos / 10000^(2i/d_model))
|
||||
PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model))
|
||||
|
||||
Properties:
|
||||
- Each dimension oscillates at a different frequency
|
||||
- Low dimensions: high frequency (changes rapidly with position)
|
||||
- High dimensions: low frequency (changes slowly with position)
|
||||
- The base 10000 controls the frequency spectrum spread
|
||||
- No learned parameters — fully deterministic
|
||||
|
||||
Learned embeddings in CLIP converge to a MODIFIED version:
|
||||
- Same sinusoidal structure
|
||||
- But with learned amplitudes (not all equal to 1.0)
|
||||
- Learned base (not exactly 10000)
|
||||
- Learned phase offsets (not exactly 0)
|
||||
- Learned DC offsets (not exactly 0)
|
||||
- Some dimensions develop more complex patterns (sum of sinusoids)
|
||||
|
||||
|
||||
METHOD: HYBRID PARAMETRIC FITTING
|
||||
-----------------------------------
|
||||
|
||||
For each dimension d in [0, dim) of the positional embedding matrix:
|
||||
|
||||
Step 1. Attempt Vaswani formula fit
|
||||
────────────────────────────────────
|
||||
Fit the parametric model:
|
||||
|
||||
f(pos) = A * sin(pos / |base|^(2d/dim) + phase) + offset
|
||||
|
||||
Parameters to fit: A (amplitude), base, phase, offset
|
||||
Method: Levenberg-Marquardt nonlinear least squares (scipy.optimize.curve_fit)
|
||||
Initial guesses:
|
||||
- A = std(signal) * sqrt(2)
|
||||
- base = 10000 (from original Transformer)
|
||||
- phase = 0
|
||||
- offset = mean(signal)
|
||||
|
||||
Measure fit quality: R² = 1 - Σ(signal - fitted)² / Σ(signal - mean)²
|
||||
|
||||
If R² > 0.7: USE THIS FIT
|
||||
This dimension follows the Vaswani pattern closely.
|
||||
Extrapolation is reliable because the model is physically motivated.
|
||||
|
||||
Step 2. Attempt multi-sinusoidal fit
|
||||
────────────────────────────────────
|
||||
If Vaswani fit is poor (R² ≤ 0.7), fit a more flexible model:
|
||||
|
||||
f(pos) = a + b*pos + Σₖ Aₖ * sin(2π * fₖ * pos + φₖ)
|
||||
|
||||
With k = 1, 2, 3 (three sinusoidal components plus linear trend).
|
||||
|
||||
Parameters: a, b, A₁, f₁, φ₁, A₂, f₂, φ₂, A₃, f₃, φ₃ (11 params)
|
||||
Initial frequency estimates: from FFT peak detection
|
||||
|
||||
If R² > 0.7: USE THIS FIT
|
||||
This dimension has a more complex but still periodic structure.
|
||||
|
||||
Step 3. Fallback to adaptive spectral extrapolation
|
||||
───────────────────────────────────────────────────
|
||||
If neither parametric model fits well (R² ≤ 0.7 for both):
|
||||
|
||||
- Full FFT decomposition
|
||||
- Frequency-dependent damping in extrapolated region
|
||||
- Per-dimension adaptive damping based on signal regularity
|
||||
|
||||
This handles dimensions with irregular or non-periodic patterns
|
||||
that don't match any analytical form.
|
||||
|
||||
|
||||
POST-PROCESSING
|
||||
---------------
|
||||
|
||||
After extrapolation, three corrections are applied:
|
||||
|
||||
1. Original preservation (blend)
|
||||
────────────────────────────────
|
||||
Positions 0 through (77 - blend_width) are replaced with exact
|
||||
original values. Positions in the blend zone transition smoothly
|
||||
from original to extrapolated using quintic smoothstep:
|
||||
|
||||
α(t) = t³(t(6t - 15) + 10) t ∈ [0, 1]
|
||||
|
||||
This ensures C2 continuity at the boundary.
|
||||
Default blend_width = 12 positions.
|
||||
|
||||
2. Norm matching
|
||||
────────────────
|
||||
The L2 norm of each extrapolated position vector is scaled to match
|
||||
the mean norm of the original 77 positions:
|
||||
|
||||
result[i] *= mean_norm(original) / norm(result[i])
|
||||
|
||||
This prevents the model from receiving embeddings with unexpected
|
||||
magnitudes, which would shift activation statistics.
|
||||
|
||||
3. Statistical validation
|
||||
─────────────────────────
|
||||
The method reports:
|
||||
- Fit quality (R²) distribution across dimensions
|
||||
- Norm ratio between original and extrapolated regions
|
||||
- Boundary cosine similarity
|
||||
- Per-dimension mean shift
|
||||
- Method selection counts (Vaswani / sinusoidal / spectral)
|
||||
|
||||
|
||||
WHY THIS WORKS BETTER THAN PURE FFT
|
||||
-------------------------------------
|
||||
|
||||
1. Physics-informed extrapolation:
|
||||
FFT treats the signal as arbitrary. This method knows the signal
|
||||
SHOULD be a sum of sinusoids (because that's what transformers learn).
|
||||
A parametric fit generalizes better than blind frequency extrapolation.
|
||||
|
||||
2. No high-frequency divergence:
|
||||
FFT extrapolation requires artificial damping to prevent divergence.
|
||||
Parametric sinusoids are bounded by definition — sin() never exceeds
|
||||
its amplitude regardless of position.
|
||||
|
||||
3. Correct frequency scaling:
|
||||
The Vaswani formula has a specific relationship between dimension
|
||||
index and frequency. By fitting the base parameter, we capture
|
||||
this relationship and extrapolate it correctly.
|
||||
|
||||
4. Per-dimension optimal method:
|
||||
Different dimensions may have different structures. The hybrid
|
||||
approach selects the best model for each dimension independently,
|
||||
rather than applying one method uniformly.
|
||||
|
||||
|
||||
VISUALIZATION
|
||||
-------------
|
||||
|
||||
Dimension with good Vaswani fit (R² = 0.95):
|
||||
|
||||
Value
|
||||
^
|
||||
| * * * *
|
||||
| * * * * * * * *
|
||||
|* * * * * * * *
|
||||
| * * * * * * *
|
||||
| * * * * * * *
|
||||
| * * * * * * *
|
||||
| * * * *
|
||||
+────────────────────────────────────────────── Position
|
||||
0 20 40 60 77 120 200 247
|
||||
|<── original (exact) ──>|<── Vaswani formula ────>|
|
||||
|
||||
|
||||
Dimension with complex pattern (sinusoidal fit, R² = 0.82):
|
||||
|
||||
Value
|
||||
^
|
||||
| ** * ** * **
|
||||
| * * * * * * * * * *
|
||||
|* ** * * ** * *
|
||||
| * * * * *
|
||||
| * * * * * * *
|
||||
| ** * * ** *
|
||||
| ** **
|
||||
+────────────────────────────────────────────── Position
|
||||
0 20 40 60 77 120 200 247
|
||||
|<── original (exact) ──>|<── 3-sinusoid model ──>|
|
||||
|
||||
|
||||
Dimension with irregular pattern (spectral fallback):
|
||||
|
||||
Value
|
||||
^
|
||||
| * * * ~ ~
|
||||
|* ** * * * ~ ~ ~ ~
|
||||
| * * * * ~ ~ ~ ~
|
||||
| * * * * ~ ~
|
||||
| * * * ~ ~
|
||||
| * * ~ (damped) ~
|
||||
+────────────────────────────────────────────── Position
|
||||
0 20 40 60 77 120 200 247
|
||||
|<── original (exact) ──>|<── adaptive spectral ──>|
|
||||
|
||||
|
||||
EMPIRICAL RESULTS
|
||||
-----------------
|
||||
|
||||
Typical fit quality distribution for SDXL CLIP-L (768 dimensions):
|
||||
- Vaswani fit (R² > 0.7): ~45-55% of dimensions
|
||||
- Sinusoidal fit (R² > 0.7): ~25-35% of dimensions
|
||||
- Spectral fallback: ~15-25% of dimensions
|
||||
|
||||
Typical fit quality distribution for SDXL CLIP-G (1280 dimensions):
|
||||
- Vaswani fit (R² > 0.7): ~40-50% of dimensions
|
||||
- Sinusoidal fit (R² > 0.7): ~30-35% of dimensions
|
||||
- Spectral fallback: ~20-25% of dimensions
|
||||
|
||||
Norm ratio (extrapolated / original): typically 0.98 - 1.02
|
||||
Boundary cosine similarity: typically > 0.95
|
||||
|
||||
|
||||
LIMITATIONS
|
||||
-----------
|
||||
|
||||
1. Parametric fitting is an approximation. Even with R² > 0.9,
|
||||
the fitted model is not identical to what the model would have
|
||||
learned with 248 positions during actual training.
|
||||
|
||||
2. The attention mechanism (Q, K, V weights) was trained on context
|
||||
length 77. Extended positional embeddings provide position
|
||||
information, but attention patterns may not generalize perfectly.
|
||||
|
||||
3. Cross-attention in the UNet was trained with conditioning sequences
|
||||
of specific lengths. Very long prompts may affect spatial composition.
|
||||
|
||||
4. Fit quality varies across checkpoints. Models fine-tuned extensively
|
||||
may have positional embeddings that deviate more from the sinusoidal
|
||||
pattern, reducing fit quality.
|
||||
|
||||
5. scipy is required for nonlinear curve fitting. The method falls back
|
||||
to spectral extrapolation for dimensions where fitting fails.
|
||||
|
||||
|
||||
REQUIREMENTS
|
||||
------------
|
||||
|
||||
Python packages: safetensors, numpy, scipy
|
||||
Install: pip install safetensors numpy scipy
|
||||
|
||||
|
||||
USAGE
|
||||
-----
|
||||
|
||||
python convert_learned.py \
|
||||
--input model.safetensors \
|
||||
--output model_long248.safetensors \
|
||||
--target_len 248 \
|
||||
--method hybrid \
|
||||
--analyze
|
||||
|
||||
Methods:
|
||||
learned_sinusoidal — fit N sinusoids + trend to each dimension
|
||||
vaswani_fitted — fit original Transformer PE formula
|
||||
hybrid — best of all methods per dimension (recommended)
|
||||
|
||||
|
||||
PROJECT FILES
|
||||
-------------
|
||||
|
||||
convert_learned.py — conversion script
|
||||
convert_learned.bat — batch file for running conversion
|
||||
node.py — ComfyUI loader node
|
||||
__init__.py — ComfyUI node registration
|
||||
method_description.txt — this file
|
||||
|
||||
================================================================================
|
||||
Reference in New Issue
Block a user