Logic has been further cleaned up in preparation for networking
integration. There's not longer unique behaviour for what is being saved
and the onus is placed on the caller to provide data in a safetensor
appropriate format
Supported has been added to cache the output of completed nodes. This
code should be trivially expandable to non-sampler nodes, but will need
to be cautiously enabled on a per-node basis to better regulate dataflow
In preperation for saving checkpoints remotely, all checkpoint
transactions are now wrapped.
Fixed step accumulation on consecutive failures
Support all builtin samplers
When sampling occurs, intermediate steps are cached to disk.
If at any point execution is halted, interrupted, or crashes, these
checkpoints are loaded from to restore execution from the previous step
Currently largely proof of concept. All samplers will cache, but
restoring is only applied to CustomSampler. Boilerplate is largely in
place to allow for caching output of other non-sampling nodes and will
be investigated later.
Austin Mroz