As title, there were 2 fundamental sources of graph breaks:
1. using a pre-existing set in a `torch.compile` region, this is a
[known issue](https://github.com/pytorch/pytorch/issues/145761) in
PyTorch and the fix is WIP, but it's a lot easier to change the set
into a tuple here to fix the graph break.
2. using tensor subclass `GGMLTensor`. The full support of tensor
subclass has been gated for `torch.compile`, but with [a recent WIP
stack](https://github.com/pytorch/pytorch/pull/149792),
we'll be able to remove the gating and have `torch.compile` fully
support the `GGMLTensor` in this repo. The only required change here
is avoid changing the `__class__` attribute, and instead using
`torch.Tensor` to convert the subclass object back to Tensor.
Benefit of FP32 for dequantization seems less noticable and feels more like a sidegrade due to differences in numerical precision. Was causing issues with memory not being freed.
#16
Use FP32 to do the calculations while dequantizing to improve existing Q8_0 quants and to avoid double dtype conversion.
Add Q4_1 and Q5_1 dequant kernels
City