pixaroma-ComfyUI-Pixaroma/docs/audio-react-math.md

Audio React / AudioReact — Math Reference

Single source of truth for every formula in the Pixaroma audio-reactive nodes (Audio React, AudioReact). Both the Python engine (nodes/_audio_react_engine.py) and the WebGL preview shaders (js/audio_studio/shaders.mjs — landed in Milestone E) implement the formulas defined here.

When you change a formula, change it here FIRST, then update both implementations and re-run the parity check.

1. Audio envelope pipeline

Input: AUDIO dict {waveform: [B, C, S] tensor, sample_rate: int}, fps: int, audio_band: str, smoothing: int.

Output: Float32Array (Python: torch.Tensor) of length total_frames = floor(audio_duration * fps), values in [0, 1].

Pipeline:

1. Mono-mix: if C > 1, average channels.
2. Bandpass FFT — see §3.
3. Resample: split waveform into total_frames non-overlapping chunks of samples_per_frame = max(1, sample_rate // fps) samples each. If the waveform is shorter than total_frames * samples_per_frame, repeat-pad.
4. Per-frame RMS: rms[t] = sqrt(mean(chunk_t ** 2)).
5. Min-max normalize: rms = (rms - rms.min()) / (rms.max() - rms.min()) (returns zeros if rms.max() == rms.min()).
6. Moving-average smoothing: convolve rms with a kernel = ones(sw) / sw where sw is smoothing rounded UP to the nearest odd integer (so the kernel is centered). replicate-pad by sw // 2 on each side first.
7. (Loop-safe ramp — applied separately by the caller, not part of the envelope helper. See §4.)

2. Onset track

Input: envelope: Float32Array[T] in [0, 1]. Output: Float32Array[T] in [0, 1].

Algorithm:

1. diff[t] = max(0, env[t] - env[t-1]) for t >= 1; diff[0] = 0.
2. thresh = max(0.05, quantile(diff, 0.75)).
3. spikes[t] = diff[t] if diff[t] > thresh else 0.
4. Sequential decay: out[0] = spikes[0]; out[t] = max(spikes[t], out[t-1] * 0.85).
5. Peak-normalize: out = out / out.max() (returns zeros if out.max() == 0).

The decay rate 0.85 is hand-tuned. Faster decay (lower) makes onsets spikier; slower (higher) makes them smear into envelopes. Don't change without re-running parity.

3. Bandpass FFT

Input: waveform: Tensor[..., S], sample_rate: int, low_hz: float | None, high_hz: float | None. Output: Tensor[..., S] (real).

spec  = rfft(waveform, dim=-1)
freqs = rfftfreq(S, d=1/sample_rate)
mask  = ones_like(freqs)
if low_hz  is not None: mask *= (freqs >= low_hz)
if high_hz is not None: mask *= (freqs <= high_hz)
spec *= mask
return irfft(spec, n=S, dim=-1)

AUDIO_BANDS_HZ:

• full: (None, None) — no bandpass
• bass: (20, 250)
• mids: (250, 4000)
• treble: (4000, 20000)

The JS implementation must use the same boundaries. The JS FFT is a small inlined real-FFT (radix-2 Cooley-Tukey on real input), zero-padding to the next power-of-2; trim back to S after irfft.

4. Loop-safe ramp

If loop_safe == True AND total_frames >= 4:

1. fade_n = max(2, min(int(fps * 0.5), total_frames // 2)).
2. loop_ramp = linspace(0.0, 1.0, fade_n) — note: starts AT 0.
3. envelope[:fade_n] *= loop_ramp (envelope[0] becomes 0).
4. envelope[-fade_n:] *= flip(loop_ramp) (envelope[-1] becomes 0).

The linspace(0, 1, fade_n) deliberately starts at 0 — that's what makes the loop seamless (motion is fully frozen at boundaries).

5. Base sampling grid

Normalized [-1, 1] grid, [1, H, W, 2] with x-first ordering (ready for F.grid_sample):

y, x = meshgrid(linspace(-1, 1, H), linspace(-1, 1, W), indexing="ij")
base_grid = stack([x, y], dim=-1).unsqueeze(0)  # [1, H, W, 2]

6. Motion modes

All 8 motion functions accept the canonical MotionContext (or its shader-uniform equivalent) and return a transformed sampling grid in the same shape as base_grid. The renderer then samples the source image through the grid (F.grid_sample in Python with mode="bilinear", padding_mode="border", align_corners=False; texture() on a clamp-to-edge texture in GLSL).

6.1 scale_pulse

s = env_t * intensity * 0.15
grid' = grid * (1.0 - s)

Maximum zoom: intensity=2.0, env_t=1.0 → s=0.3 → 30% zoom-in.

6.2 zoom_punch

s = onset_t * intensity * 0.30
grid' = grid * (1.0 - s)

Same shape as scale_pulse but driven by onset_t instead of env_t, and with double the multiplier.

6.3 shake

# Pre-render: build cumulative random walk for the whole clip
gen     = Generator().manual_seed(0)
dx_raw  = randn(total_frames, gen=gen) * onset_array
dy_raw  = randn(total_frames, gen=gen) * onset_array
decay   = 0.7
dx[0]   = dx_raw[0];  dy[0] = dy_raw[0]
dx[k]   = dx[k-1] * decay + dx_raw[k] * (1 - decay)
dy[k]   = dy[k-1] * decay + dy_raw[k] * (1 - decay)

# Per-frame:
amp     = intensity * 0.04
grid'   = grid - (dx[i] * amp, dy[i] * amp)

Approximation note (shader): the WebGL preview cannot bit-match this — torch Generator(seed=0) produces a specific sequence that JS cannot reproduce, and float-precision drift over hundreds of frames diverges trajectories. Shader uses a deterministic same-seeded JS RNG (e.g. mulberry32(0)) approximating the same characteristic motion. Final MP4 output is authoritative; preview is approximate for shake only.

6.4 drift

sway = sin(2π * motion_speed * t)
bob  = cos(2π * motion_speed * t)
amp  = env_t * intensity * 0.04
grid' = grid - (sway * amp, bob * amp)

t in seconds since clip start.

6.5 rotate_pulse

aspect = W / H
sway   = sin(2π * motion_speed * t)
angle  = sway * env_t * intensity * (π / 12)   # max ±15°

xs = grid[..., 0] * aspect
ys = grid[..., 1]
new_x = (xs * cos(angle) - ys * sin(angle)) / aspect
new_y =  xs * sin(angle) + ys * cos(angle)

Aspect correction prevents non-square frames from producing visually elliptical rotation.

6.6 swirl

aspect = W / H
xs     = grid[..., 0] * aspect
ys     = grid[..., 1]
r      = sqrt(xs² + ys²)
θ      = atan2(ys, xs)
twist  = env_t * intensity * (π / 2) * max(0, 1 - r)
θ'     = θ + twist
new_x  = r * cos(θ') / aspect
new_y  = r * sin(θ')

Outside the unit disk (r >= 1), twist = 0 so the corners are stationary.

6.7 ripple

aspect = W / H
ys     = linspace(-1, 1, H) (broadcast)
xs     = linspace(-1, 1, W) (broadcast)
r      = sqrt((xs * aspect)² + ys²)

k      = 6π
omega  = 2π * max(motion_speed * 4.0, 0.5)
A      = env_t * intensity * 0.015 * 2.0   # normalized [-1,1] grid units

dr     = A * sin(k * r - omega * t)
r_safe = max(r, 1e-3)
dx     = dr * (xs * aspect) / r_safe / aspect
dy     = dr * ys / r_safe

grid' = grid + (dx, dy)

6.8 slit_scan

ys     = linspace(-1, 1, H) (broadcast across W)
k      = 4π
omega  = 2π * max(motion_speed * 2.0, 0.4)
A      = env_t * intensity * 0.04

dy     = A * sin(k * ys - omega * t)
dx     = A * 0.5 * cos(k * ys - omega * t)
grid' = grid + (dx, dy)

The horizontal displacement at half-amplitude with cos phase keeps the distortion from looking like a single 1D wave.

7. Overlays

All 4 overlays read frame: [H, W, 3] in [0, 1] and return the modified frame. Each is gated by its strength uniform — if strength <= 0: skip.

7.1 glitch

if onset_t <= 0.001 or strength <= 0: return frame
max_px = max(1, int(onset_t * strength * 0.012 * min(H, W)))

# Per-channel R/G/B horizontal offsets (random sign, ±max_px), seeded
# deterministically by the onset value for reproducibility.
seed   = int(onset_t * 1e6) & 0xFFFF
gen    = Generator().manual_seed(seed)
signs  = randint(0, 2, (3,), gen=gen) * 2 - 1   # ±1 per channel
offsets = signs * max_px

for c in 0..2:
    ox = offsets[c]
    if ox > 0: shift channel c right by ox px (replicate left edge)
    elif ox < 0: shift left by |ox| (replicate right edge)

# Big-spike scanline tear:
if onset_t * strength > 0.7:
    n_swap = max(1, H // 20)
    pick n_swap random row indices, swap row[i] with row[i+1].

GLSL implementation samples each channel from a horizontally-shifted UV coord. Scanline tear can be approximated by per-row pseudo-random amplitude added to the UV.x — a row that's "swapped" effectively reads from the row above or below.

7.2 bloom

if env_t <= 0.001 or strength <= 0: return frame
weight = env_t * strength * 0.6

# Downsample 4x → 9-tap separable Gaussian blur (sigma=2.0) → upsample
small = downsample(frame, 0.25)
small = blur_horizontal(small, kernel=gauss_9, sigma=2.0)
small = blur_vertical(small, kernel=gauss_9, sigma=2.0)
big   = upsample(small, frame.size)
bloom_layer = clip(big * weight, 0, 1)

# Screen blend
out = 1 - (1 - frame) * (1 - bloom_layer)
return clip(out, 0, 1)

GLSL: same multi-pass — downsample to a 1/4-size FBO, two blur passes, upsample, screen-blend. The 9-tap separable Gaussian: weights g[i] = exp(-(i - 4)² / 8), normalized to sum 1.

7.3 vignette

if env_t <= 0.001 or strength <= 0: return frame
ys = linspace(-1, 1, H); xs = linspace(-1, 1, W)
r  = sqrt(xs² + ys²).clip(0, 1.4)
v  = (r / 1.414).clip(0, 1)
mask = 1 - v * env_t * strength * 0.5
return frame * mask

GLSL: compute r from gl_FragCoord normalized to [-1, 1], same formula.

7.4 hue_shift

if env_t <= 0.001 or strength <= 0: return frame
angle = env_t * strength * (30° in radians)
c, s  = cos(angle), sin(angle)
m = [
    [0.299 + 0.701*c + 0.168*s,  0.587 - 0.587*c + 0.330*s,  0.114 - 0.114*c - 0.497*s],
    [0.299 - 0.299*c - 0.328*s,  0.587 + 0.413*c + 0.035*s,  0.114 - 0.114*c + 0.292*s],
    [0.299 - 0.299*c + 1.250*s,  0.587 - 0.587*c - 1.050*s,  0.114 + 0.886*c - 0.203*s],
]
return clip(frame @ m^T, 0, 1)

The 0.299 / 0.587 / 0.114 luma triples MUST be exact in every row — typos drift the gray axis and produce a tint on neutrals at high angles.

8. Constants reference

Constant	Value	Used by	Rationale
scale_pulse multiplier	0.15	scale_pulse	30% max zoom feels punchy without distortion
zoom_punch multiplier	0.30	zoom_punch	2× scale_pulse for spike feel
shake amplitude	0.04	shake	4% half-frame at intensity=1, raw=±1
shake decay	0.7	shake	smooths random walk into believable motion
drift amplitude	0.04	drift	matches shake amplitude
rotate_pulse max angle	π/12 (15°)	rotate_pulse	bigger angles look gimmicky
ripple k	6π	ripple	6 full radial cycles within unit disk
ripple omega coefficient	max(motion_speed*4, 0.5)	ripple	floor prevents stalling at 0 motion_speed
ripple amplitude	0.015 * 2.0	ripple	1.5% of grid range = 1.5% pixel displacement
slit_scan k	4π	slit_scan	2 full vertical cycles
slit_scan omega	max(motion_speed*2, 0.4)	slit_scan	floor matches ripple's rationale
slit_scan amplitude	0.04	slit_scan	matches shake/drift
glitch px coefficient	0.012	glitch	720p → 9px max, 4K → 26px max
glitch tear threshold	0.7	glitch	only big spikes tear
glitch tear row count	H // 20	glitch	5% of rows
bloom weight coefficient	0.6	bloom	screen-blend weight at full env*strength
bloom blur sigma	2.0	bloom	matches 9-tap kernel coverage
bloom downsample	0.25	bloom	1/16 area, 16× compute saving
vignette weight	0.5	vignette	50% darken at corners on full env*strength
hue_shift max angle	30°	hue_shift	bigger looks gimmicky
onset decay	0.85	onset_track	balances spike vs smear
onset thresh floor	0.05	onset_track	minimum sensitivity for quiet music
loop_safe fade	0.5s	loop_safe ramp	invisible on clips ≥ 5s

9. Where preview is approximate

Listed here so users / engineers know what to expect.

• shake — JS RNG cannot reproduce torch's Generator(seed=0). Browser uses mulberry32(0) as a deterministic-but-different stand-in. Final MP4 output (Python) is authoritative.
• bloom — Python uses downsample → 2-pass separable Gaussian blur → upsample → screen blend (3 framebuffer passes). Browser approximates with a single in-shader 9-tap radial blur — visually similar but the falloff shape and strength are not bit-exact. Final MP4 (Python) is authoritative.

All other modes are exact-bit-pattern bound (within shader-precision rounding — highp floats throughout).

10. Parity check

Two layers of test:

1. Python parity (scripts/audio_parity_check.py) — renders 64 reference frames from the engine, diffs against committed PNGs. Tolerance: pixel RMSE ≤ 1.0 (covers PyTorch float-precision drift across releases). Run before each release.
2. Browser parity harness (assets/audio_studio_parity/index.html, landing in Milestone I) — loads the same goldens, runs the WebGL pipeline against the same source, computes per-pair ΔE in JS. Mean ΔE ≤ 5.0 for non-shake / non-bloom tests. Run manually in Chrome / Firefox / Safari before release.