CRITICAL FIX: torch.compile fallback was forcing CPU execution instead of GPU
BUG IDENTIFIED: Line 250: torch._dynamo.config.disable = True This was COMPLETELY disabling torch.compile, causing SDNQ to fall back to CPU execution instead of GPU execution, resulting in models using system RAM instead of VRAM. ROOT CAUSE: When torch.compile is disabled entirely via disable=True: - SDNQ's dequantization code has no compiled path - Fallback implementation runs on CPU instead of GPU - Model loads into system RAM instead of VRAM - User experiences slow CPU-only processing PROPER FIX: REMOVED: torch._dynamo.config.disable = True KEPT: torch._dynamo.config.suppress_errors = True How it works now: 1. torch.compile TRIES to compile (GPU path) 2. If compiler missing, compilation FAILS 3. suppress_errors=True catches the error gracefully 4. torch.compile falls back to EAGER MODE on GPU 5. Model still uses GPU/VRAM (same memory usage) 6. Only difference: No compiled optimizations (slightly slower) CORRECT BEHAVIOR: ✅ GPU/VRAM used regardless of compiler availability ✅ Quantized weights preserved (same memory savings) ✅ Compilation errors suppressed (no crashes) ✅ Runs on GPU in eager mode (slower than compiled, but still GPU) INCORRECT BEHAVIOR (before fix): ❌ CPU/System RAM used when compiler missing ❌ Slow processing ❌ No VRAM savings despite using quantized model Thanks to user for catching this critical bug!
This commit is contained in:
+11
-5
@@ -240,14 +240,20 @@ class SDNQModelLoader:
|
||||
# Check if C++ compiler is available (needed for torch.compile)
|
||||
compiler_available = check_cpp_compiler_available()
|
||||
|
||||
# Disable torch.compile if compiler not available
|
||||
# Configure torch.compile error handling
|
||||
# SDNQ quantized models use torch.compile for dequantization by default
|
||||
if not compiler_available:
|
||||
print("⚠ C++ compiler not detected - disabling torch.compile for SDNQ")
|
||||
print(" (Model will still use quantized weights with same memory savings)")
|
||||
# Disable torch dynamo compilation
|
||||
print("⚠ C++ compiler not detected - torch.compile will use fallback mode")
|
||||
print(" Model will still run on GPU with quantized weights (same VRAM usage)")
|
||||
print(" Compilation errors will be suppressed and execution will continue")
|
||||
|
||||
# Suppress compilation errors - let torch.compile fail gracefully and fall back to eager mode
|
||||
# CRITICAL: We do NOT disable torch.compile entirely (that would force CPU execution)
|
||||
# Instead, we let it TRY to compile, catch errors, and fall back to GPU eager mode
|
||||
torch._dynamo.config.suppress_errors = True
|
||||
torch._dynamo.config.disable = True
|
||||
|
||||
# Optional: Set verbose to False to reduce error spam
|
||||
torch._dynamo.config.verbose = False
|
||||
|
||||
# Pre-load cleanup to clear any leftover state from previous failed loads
|
||||
# This prevents hanging if a previous load left resources in a bad state
|
||||
|
||||
Reference in New Issue
Block a user