Dequantizing produces float32 — four times the width the weight ends up
at. Every tensor was collected before any was cast, so the peak was the
whole checkpoint in float32 plus a copy of each: tens of GB of transient
host allocation on a video model, enough to take the machine down rather
than just the load. ThreadPoolExecutor.map made it worse by submitting
everything at once and buffering results in order.
Now cast-and-release per tensor, with a fixed in-flight window. Peak is
the finished model plus a handful of tensors. Pool capped at 4.
Also prints the path and tensor count before the wait, so a load killed
part way leaves evidence of which backend it chose.
digital-garbage
ae337b5196