Fix value function gradient: move to MLP device before backward pass

MLP lives on CPU, conditioning on GPU — device mismatch silently fell back
to fixed direction. Now moves conditioning to MLP device for the forward/
backward, then returns gradient to original device.

Also surface value function fallback errors instead of swallowing them.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
digital-garbage
2026-05-29 04:35:50 +03:00
co-authored by Claude Sonnet 4.6
parent b0750d5c6d
commit 4dd8fcd674
2 changed files with 5 additions and 3 deletions
+3 -2
View File
@@ -73,11 +73,12 @@ class OnlineValueFunction(nn.Module):
def gradient(self, conditioning):
"""∂reward/∂conditioning — same shape as input."""
mlp_device = next(self.parameters()).device
with torch.inference_mode(False), torch.enable_grad():
c_in = conditioning.detach().clone().float().requires_grad_(True)
c_in = conditioning.detach().clone().float().to(mlp_device).requires_grad_(True)
reward = self.forward(self.compress(c_in).unsqueeze(0))
reward.backward()
return c_in.grad.to(conditioning.dtype)
return c_in.grad.to(conditioning.device, conditioning.dtype)
def is_ready(self):
return len(self.buffer_c) >= self.MIN_SAMPLES