Sometimes gguf will return size-one one-dimensional memmaps for scalar values. I'm not exactly sure when or why, but this will accept any dimension as long as it only contains one value
Resolves#384
* Implement GGUF metadata extraction function
Added function to extract metadata from GGUF files.
* Updated the GGUF model loading and patching classes to include metadata handling.
* Clean up return logic for extra metadata
This should be more future proof in case we need to return other attributes in the future. Possible breaking change for anyone using `gguf_sd_loader` directly either way, though.
---------
Co-authored-by: City <125218114+city96@users.noreply.github.com>
According to #361 the thinking/abliterated versions work even though the model wasn't trained with them, maybe the vision one does too.
Does not have mmproj support, since nothing uses that atm.
* add default named_modules_to_munmap variable
Default this to handle to case of cloning a GGUFModelPatcher which then
expects it to exist on the clone.
I could never reproduce this condition, however 4 users have the error.
This error path is very sensitive to VRAM condtions and workflow and I
never got the details so its a bit of a guess.
But from a code point of view, its the right thing anyway
* nodes: carry the mmap_released state through a clone()
There are workflows where the same underlying model will be loaded
twice by two different ModelPatchers.
With the current code as-is, the first patcher will load fine and
intercept the pins while removing already done pins from the free
modules list.
It then will load again with the second MP however, this one will
not get the callbacks for the already pinned stuff and then rip
through and .to().to() them all. This destroys the pinning setup
while comfy core still keeps the modules registered as unpinned.
This causes a crash on einval when core goes to unpin the not pinned
tensors.
Comfy core now has a guard against unpinning the not-pinned, however
what I think is happening, is the RAM of the pin in freed in the
.to().to() and then malloc re-allocates it to new tensors with cuda
keeping its records of previous pinning. einval can happen when
you try and unpin a tensor in the middle of the block which is
definately possible on this use after free pattern.
Comfy core recently introduced a feature where weights may be pinned
when loading, particularly for the case of offloading.
Intercept this, and immediately detached each weight before the pinning.
This avoids a crash that at least some users are experiencing.
Use a little dict on the modules to keep track of whats already done,
and when the catch-all detacher loop comes through, use this dict
(which has already done modules removed) as the iterator basis.
- Apply umt5 hotfix of the token embed runtime cost being too large to llama as well
- Force model size to be re-calculated after the initial patcher is loaded to avoid issue with empty linear layers not being counted
As title, there were 2 fundamental sources of graph breaks:
1. using a pre-existing set in a `torch.compile` region, this is a
[known issue](https://github.com/pytorch/pytorch/issues/145761) in
PyTorch and the fix is WIP, but it's a lot easier to change the set
into a tuple here to fix the graph break.
2. using tensor subclass `GGMLTensor`. The full support of tensor
subclass has been gated for `torch.compile`, but with [a recent WIP
stack](https://github.com/pytorch/pytorch/pull/149792),
we'll be able to remove the gating and have `torch.compile` fully
support the `GGMLTensor` in this repo. The only required change here
is avoid changing the `__class__` attribute, and instead using
`torch.Tensor` to convert the subclass object back to Tensor.
City