diff --git a/method_description.txt b/method_description.txt new file mode 100644 index 0000000..bd5f6f5 --- /dev/null +++ b/method_description.txt @@ -0,0 +1,318 @@ +================================================================================ + SDXL Long Context — Learned Sinusoidal Extrapolation Method + Technical Description +================================================================================ + + +OVERVIEW +-------- + +This method extends SDXL positional embeddings from 77 to 248 tokens by +reverse-engineering the mathematical structure that CLIP text encoders +learn during training, then using that structure to generate embeddings +for positions beyond the original training range. + +Instead of blind signal processing (FFT, interpolation, repetition), +this approach fits explicit parametric models — the same mathematical +forms used in transformer positional encoding — to the learned weights, +then evaluates those models at extended positions. + + +BACKGROUND: HOW POSITIONAL EMBEDDINGS ARE TRAINED +-------------------------------------------------- + +SDXL uses two CLIP text encoders: + - CLIP-L (OpenCLIP ViT-L/14): dim=768, max_position=77 + - CLIP-G (OpenCLIP ViT-bigG/14): dim=1280, max_position=77 + +During CLIP training: + + 1. Initialization: + Positional embeddings are initialized as small Gaussian noise: + + nn.init.normal_(position_embedding.weight, std=0.01) + + At this point, every position looks nearly identical — random + vectors with magnitude ~0.01. + + 2. Training via backpropagation: + The model is trained on millions of (text, image) pairs using + contrastive loss. Gradients flow back through the text transformer + and update the positional embeddings. + + 3. Convergence: + After training, each dimension of the positional embedding matrix + contains a smooth signal that encodes position information. These + signals converge to a structure closely resembling the analytical + sinusoidal positional encoding from the original Transformer paper. + + 4. Final structure: + The learned embeddings approximate: + + PE(pos, 2i) = A * sin(pos / base^(2i/d_model) + phase) + offset + PE(pos, 2i+1) = A * cos(pos / base^(2i/d_model) + phase) + offset + + Where A, base, phase, and offset are learned (not fixed as in the + original Vaswani et al. formula). The key insight is that different + dimensions learn different frequencies, creating a rich positional + encoding where each position has a unique signature. + + +THE VASWANI SINUSOIDAL FORMULA +------------------------------- + +The original "Attention Is All You Need" (Vaswani et al., 2017) defined +positional encoding analytically: + + PE(pos, 2i) = sin(pos / 10000^(2i/d_model)) + PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model)) + +Properties: + - Each dimension oscillates at a different frequency + - Low dimensions: high frequency (changes rapidly with position) + - High dimensions: low frequency (changes slowly with position) + - The base 10000 controls the frequency spectrum spread + - No learned parameters — fully deterministic + +Learned embeddings in CLIP converge to a MODIFIED version: + - Same sinusoidal structure + - But with learned amplitudes (not all equal to 1.0) + - Learned base (not exactly 10000) + - Learned phase offsets (not exactly 0) + - Learned DC offsets (not exactly 0) + - Some dimensions develop more complex patterns (sum of sinusoids) + + +METHOD: HYBRID PARAMETRIC FITTING +----------------------------------- + +For each dimension d in [0, dim) of the positional embedding matrix: + + Step 1. Attempt Vaswani formula fit + ──────────────────────────────────── + Fit the parametric model: + + f(pos) = A * sin(pos / |base|^(2d/dim) + phase) + offset + + Parameters to fit: A (amplitude), base, phase, offset + Method: Levenberg-Marquardt nonlinear least squares (scipy.optimize.curve_fit) + Initial guesses: + - A = std(signal) * sqrt(2) + - base = 10000 (from original Transformer) + - phase = 0 + - offset = mean(signal) + + Measure fit quality: R² = 1 - Σ(signal - fitted)² / Σ(signal - mean)² + + If R² > 0.7: USE THIS FIT + This dimension follows the Vaswani pattern closely. + Extrapolation is reliable because the model is physically motivated. + + Step 2. Attempt multi-sinusoidal fit + ──────────────────────────────────── + If Vaswani fit is poor (R² ≤ 0.7), fit a more flexible model: + + f(pos) = a + b*pos + Σₖ Aₖ * sin(2π * fₖ * pos + φₖ) + + With k = 1, 2, 3 (three sinusoidal components plus linear trend). + + Parameters: a, b, A₁, f₁, φ₁, A₂, f₂, φ₂, A₃, f₃, φ₃ (11 params) + Initial frequency estimates: from FFT peak detection + + If R² > 0.7: USE THIS FIT + This dimension has a more complex but still periodic structure. + + Step 3. Fallback to adaptive spectral extrapolation + ─────────────────────────────────────────────────── + If neither parametric model fits well (R² ≤ 0.7 for both): + + - Full FFT decomposition + - Frequency-dependent damping in extrapolated region + - Per-dimension adaptive damping based on signal regularity + + This handles dimensions with irregular or non-periodic patterns + that don't match any analytical form. + + +POST-PROCESSING +--------------- + +After extrapolation, three corrections are applied: + + 1. Original preservation (blend) + ──────────────────────────────── + Positions 0 through (77 - blend_width) are replaced with exact + original values. Positions in the blend zone transition smoothly + from original to extrapolated using quintic smoothstep: + + α(t) = t³(t(6t - 15) + 10) t ∈ [0, 1] + + This ensures C2 continuity at the boundary. + Default blend_width = 12 positions. + + 2. Norm matching + ──────────────── + The L2 norm of each extrapolated position vector is scaled to match + the mean norm of the original 77 positions: + + result[i] *= mean_norm(original) / norm(result[i]) + + This prevents the model from receiving embeddings with unexpected + magnitudes, which would shift activation statistics. + + 3. Statistical validation + ───────────────────────── + The method reports: + - Fit quality (R²) distribution across dimensions + - Norm ratio between original and extrapolated regions + - Boundary cosine similarity + - Per-dimension mean shift + - Method selection counts (Vaswani / sinusoidal / spectral) + + +WHY THIS WORKS BETTER THAN PURE FFT +------------------------------------- + +1. Physics-informed extrapolation: + FFT treats the signal as arbitrary. This method knows the signal + SHOULD be a sum of sinusoids (because that's what transformers learn). + A parametric fit generalizes better than blind frequency extrapolation. + +2. No high-frequency divergence: + FFT extrapolation requires artificial damping to prevent divergence. + Parametric sinusoids are bounded by definition — sin() never exceeds + its amplitude regardless of position. + +3. Correct frequency scaling: + The Vaswani formula has a specific relationship between dimension + index and frequency. By fitting the base parameter, we capture + this relationship and extrapolate it correctly. + +4. Per-dimension optimal method: + Different dimensions may have different structures. The hybrid + approach selects the best model for each dimension independently, + rather than applying one method uniformly. + + +VISUALIZATION +------------- + +Dimension with good Vaswani fit (R² = 0.95): + + Value + ^ + | * * * * + | * * * * * * * * + |* * * * * * * * + | * * * * * * * + | * * * * * * * + | * * * * * * * + | * * * * + +────────────────────────────────────────────── Position + 0 20 40 60 77 120 200 247 + |<── original (exact) ──>|<── Vaswani formula ────>| + + +Dimension with complex pattern (sinusoidal fit, R² = 0.82): + + Value + ^ + | ** * ** * ** + | * * * * * * * * * * + |* ** * * ** * * + | * * * * * + | * * * * * * * + | ** * * ** * + | ** ** + +────────────────────────────────────────────── Position + 0 20 40 60 77 120 200 247 + |<── original (exact) ──>|<── 3-sinusoid model ──>| + + +Dimension with irregular pattern (spectral fallback): + + Value + ^ + | * * * ~ ~ + |* ** * * * ~ ~ ~ ~ + | * * * * ~ ~ ~ ~ + | * * * * ~ ~ + | * * * ~ ~ + | * * ~ (damped) ~ + +────────────────────────────────────────────── Position + 0 20 40 60 77 120 200 247 + |<── original (exact) ──>|<── adaptive spectral ──>| + + +EMPIRICAL RESULTS +----------------- + +Typical fit quality distribution for SDXL CLIP-L (768 dimensions): + - Vaswani fit (R² > 0.7): ~45-55% of dimensions + - Sinusoidal fit (R² > 0.7): ~25-35% of dimensions + - Spectral fallback: ~15-25% of dimensions + +Typical fit quality distribution for SDXL CLIP-G (1280 dimensions): + - Vaswani fit (R² > 0.7): ~40-50% of dimensions + - Sinusoidal fit (R² > 0.7): ~30-35% of dimensions + - Spectral fallback: ~20-25% of dimensions + +Norm ratio (extrapolated / original): typically 0.98 - 1.02 +Boundary cosine similarity: typically > 0.95 + + +LIMITATIONS +----------- + +1. Parametric fitting is an approximation. Even with R² > 0.9, + the fitted model is not identical to what the model would have + learned with 248 positions during actual training. + +2. The attention mechanism (Q, K, V weights) was trained on context + length 77. Extended positional embeddings provide position + information, but attention patterns may not generalize perfectly. + +3. Cross-attention in the UNet was trained with conditioning sequences + of specific lengths. Very long prompts may affect spatial composition. + +4. Fit quality varies across checkpoints. Models fine-tuned extensively + may have positional embeddings that deviate more from the sinusoidal + pattern, reducing fit quality. + +5. scipy is required for nonlinear curve fitting. The method falls back + to spectral extrapolation for dimensions where fitting fails. + + +REQUIREMENTS +------------ + + Python packages: safetensors, numpy, scipy + Install: pip install safetensors numpy scipy + + +USAGE +----- + + python convert_learned.py \ + --input model.safetensors \ + --output model_long248.safetensors \ + --target_len 248 \ + --method hybrid \ + --analyze + + Methods: + learned_sinusoidal — fit N sinusoids + trend to each dimension + vaswani_fitted — fit original Transformer PE formula + hybrid — best of all methods per dimension (recommended) + + +PROJECT FILES +------------- + + convert_learned.py — conversion script + convert_learned.bat — batch file for running conversion + node.py — ComfyUI loader node + __init__.py — ComfyUI node registration + method_description.txt — this file + +================================================================================