CRITICAL FIX: torch.compile fallback was forcing CPU execution instead of GPU

BUG IDENTIFIED:
Line 250: torch._dynamo.config.disable = True
This was COMPLETELY disabling torch.compile, causing SDNQ to fall back to CPU execution
instead of GPU execution, resulting in models using system RAM instead of VRAM.

ROOT CAUSE:
When torch.compile is disabled entirely via disable=True:
- SDNQ's dequantization code has no compiled path
- Fallback implementation runs on CPU instead of GPU
- Model loads into system RAM instead of VRAM
- User experiences slow CPU-only processing

PROPER FIX:
REMOVED: torch._dynamo.config.disable = True
KEPT: torch._dynamo.config.suppress_errors = True

How it works now:
1. torch.compile TRIES to compile (GPU path)
2. If compiler missing, compilation FAILS
3. suppress_errors=True catches the error gracefully
4. torch.compile falls back to EAGER MODE on GPU
5. Model still uses GPU/VRAM (same memory usage)
6. Only difference: No compiled optimizations (slightly slower)

CORRECT BEHAVIOR:
✅ GPU/VRAM used regardless of compiler availability
✅ Quantized weights preserved (same memory savings)
✅ Compilation errors suppressed (no crashes)
✅ Runs on GPU in eager mode (slower than compiled, but still GPU)

INCORRECT BEHAVIOR (before fix):
❌ CPU/System RAM used when compiler missing
❌ Slow processing
❌ No VRAM savings despite using quantized model

Thanks to user for catching this critical bug!
This commit is contained in:
Claude
2025-12-08 00:41:24 +00:00
parent 3ad5f9f46a
commit 0d9bc46f3d
+11 -5
View File
@@ -240,14 +240,20 @@ class SDNQModelLoader:
# Check if C++ compiler is available (needed for torch.compile)
compiler_available = check_cpp_compiler_available()
# Disable torch.compile if compiler not available
# Configure torch.compile error handling
# SDNQ quantized models use torch.compile for dequantization by default
if not compiler_available:
print("⚠ C++ compiler not detected - disabling torch.compile for SDNQ")
print(" (Model will still use quantized weights with same memory savings)")
# Disable torch dynamo compilation
print("⚠ C++ compiler not detected - torch.compile will use fallback mode")
print(" Model will still run on GPU with quantized weights (same VRAM usage)")
print(" Compilation errors will be suppressed and execution will continue")
# Suppress compilation errors - let torch.compile fail gracefully and fall back to eager mode
# CRITICAL: We do NOT disable torch.compile entirely (that would force CPU execution)
# Instead, we let it TRY to compile, catch errors, and fall back to GPU eager mode
torch._dynamo.config.suppress_errors = True
torch._dynamo.config.disable = True
# Optional: Set verbose to False to reduce error spam
torch._dynamo.config.verbose = False
# Pre-load cleanup to clear any leftover state from previous failed loads
# This prevents hanging if a previous load left resources in a bad state