162 lines
4.0 KiB
Markdown
162 lines
4.0 KiB
Markdown
# Estimating Light Direction and Hardness from Normal Passes and Luma Deltas
|
||
|
||
This document describes the **theory** behind using **normal passes** (from OpenGL/DirectX rendering) combined with **luma differences** between two frames (before/after) to estimate:
|
||
|
||
1. Whether the dominant light comes from the **left or right** (and optionally up/down).
|
||
2. Whether the light is **hard or soft**.
|
||
|
||
---
|
||
|
||
## Exposure Analysis Methodology
|
||
|
||
### Hybrid Approach
|
||
- **Display-Referred (IRE) Analysis**
|
||
- Direct sRGB luma values (0-100 IRE scale)
|
||
- Accurate for display-referred comparisons
|
||
- Matches what users see on their monitors
|
||
|
||
- **Scene-Approximated (EV) Analysis**
|
||
- sRGB → Linear conversion (gamma 2.4 expansion)
|
||
- Reference: 18% gray @ 0.18 linear value
|
||
- ΔEV = log₂(linear_after / linear_before)
|
||
- Margin of error: ±0.3 stops due to gamma encoding
|
||
- Valid for relative exposure changes only
|
||
|
||
### Accuracy Considerations
|
||
| Metric | Best For | Limitations |
|
||
|--------|----------|-------------|
|
||
| IRE | Display-referred analysis | Non-linear relationship to scene light |
|
||
| EV | Relative exposure changes | Approximate values from sRGB inputs |
|
||
|
||
## 1. Inputs
|
||
|
||
- **Normal Pass** (`N_rgb`):
|
||
Encoded in RGB channels with X=R, Y=G, Z=B.
|
||
Values in [0, 1] mapped from [-1, 1].
|
||
- OpenGL → Y is **up**
|
||
- DirectX → Y is **down** (requires flipping).
|
||
|
||
- **Luma Before/After** (`Y_before`, `Y_after`):
|
||
Grayscale images representing brightness in two states.
|
||
|
||
---
|
||
|
||
## 2. Decoding Normals
|
||
|
||
To decode:
|
||
|
||
```
|
||
Nx = R * 2 - 1
|
||
Ny = G * 2 - 1
|
||
Nz = B * 2 - 1
|
||
|
||
# Flip Y if using DirectX convention
|
||
Ny = -Ny
|
||
```
|
||
|
||
---
|
||
|
||
## 3. Computing Luma Deltas
|
||
|
||
We compare luma frames to detect lighting changes:
|
||
|
||
```
|
||
ΔY = Y_after - Y_before
|
||
```
|
||
|
||
Optionally normalize by exposure drift:
|
||
|
||
```
|
||
scale = median(|ΔY|) * 6
|
||
ΔYn = clip(ΔY / scale, -3, 3)
|
||
```
|
||
|
||
---
|
||
|
||
## 4. Estimating Light Direction
|
||
|
||
We fit a linear model:
|
||
|
||
```
|
||
ΔY ≈ a * Nx + b * Ny + c
|
||
```
|
||
|
||
- `a` → influence of left/right
|
||
- `b` → influence of up/down
|
||
|
||
The estimated light direction (screen space) is:
|
||
|
||
```
|
||
L = normalize([a, b])
|
||
```
|
||
|
||
- If `L.x > 0` → light from **right**
|
||
- If `L.x < 0` → light from **left**
|
||
|
||
Confidence can be estimated by comparing variance explained by the model vs residuals.
|
||
|
||
---
|
||
|
||
## 5. Estimating Hard vs Soft Light
|
||
|
||
### Approach A – High-Frequency Energy
|
||
Hard light yields sharper shadows → more high-frequency energy in ΔY.
|
||
|
||
- Compute gradient magnitude of ΔY (Sobel filter).
|
||
- Ratio of gradient energy vs signal amplitude gives hardness.
|
||
|
||
### Approach B – Penumbra Width
|
||
Measure transition width of intensity changes along light direction.
|
||
|
||
```
|
||
penumbra ≈ ΔY / gradient_along_light
|
||
```
|
||
|
||
- Larger penumbra width → softer light.
|
||
|
||
### Combined Estimate
|
||
Blend the two measures for robustness:
|
||
|
||
```
|
||
softness = 0.5 * softness_highfreq + 0.5 * softness_penumbra
|
||
hardness = 1 - softness
|
||
```
|
||
|
||
---
|
||
|
||
## 6. Summary of Outputs
|
||
|
||
- **Light Direction (2D vector)** – `[x, y]`
|
||
- **Main Direction** – left/right (from sign of x)
|
||
- **Softness** – [0, 1], where 0 = hard, 1 = soft
|
||
- **Confidence** – ratio of explained variance to residuals
|
||
|
||
---
|
||
|
||
## 7. Recommended Libraries
|
||
|
||
To implement this workflow in Python or node-based systems:
|
||
|
||
- **NumPy** → efficient array math and least-squares fitting
|
||
- **OpenCV (cv2)** → Sobel gradients, Gaussian blur, morphological ops
|
||
- **SciPy** → signal processing, advanced regression tools
|
||
- **PyTorch** or **TensorFlow** (optional) → GPU acceleration for large frames
|
||
- **ComfyUI Custom Nodes** → integrate into node-based AI/graphics workflows
|
||
|
||
---
|
||
|
||
## 8. Notes for Integration
|
||
|
||
- Always check if the normal pass uses **OpenGL (Y up)** or **DirectX (Y down)** before interpreting.
|
||
- Preprocess luma to reduce exposure drift (normalization).
|
||
- Use masks (e.g., Nz > 0, ΔY above threshold) to ignore irrelevant regions.
|
||
- Smooth results over time for video sequences to stabilize estimates.
|
||
|
||
---
|
||
|
||
## 📚 References
|
||
|
||
- OpenGL and DirectX coordinate systems
|
||
- Lambertian reflectance model
|
||
- Image processing techniques for edge detection and frequency analysis
|