Fix value function gradient: move to MLP device before backward pass
MLP lives on CPU, conditioning on GPU — device mismatch silently fell back to fixed direction. Now moves conditioning to MLP device for the forward/ backward, then returns gradient to original device. Also surface value function fallback errors instead of swallowing them. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Sonnet 4.6
parent
b0750d5c6d
commit
4dd8fcd674
+3
-2
@@ -73,11 +73,12 @@ class OnlineValueFunction(nn.Module):
|
||||
|
||||
def gradient(self, conditioning):
|
||||
"""∂reward/∂conditioning — same shape as input."""
|
||||
mlp_device = next(self.parameters()).device
|
||||
with torch.inference_mode(False), torch.enable_grad():
|
||||
c_in = conditioning.detach().clone().float().requires_grad_(True)
|
||||
c_in = conditioning.detach().clone().float().to(mlp_device).requires_grad_(True)
|
||||
reward = self.forward(self.compress(c_in).unsqueeze(0))
|
||||
reward.backward()
|
||||
return c_in.grad.to(conditioning.dtype)
|
||||
return c_in.grad.to(conditioning.device, conditioning.dtype)
|
||||
|
||||
def is_ready(self):
|
||||
return len(self.buffer_c) >= self.MIN_SAMPLES
|
||||
|
||||
Reference in New Issue
Block a user