The orchestration code included functionality to track when execution
had completed which is not useful outside of orchestration. As the other
references to completion had been removed, this code was causing
exceptions during execution.
A minimal change set was made to allow the code to function after
changes to the execution model. The changes unrelated to workflow
orchestration have been pulled out so that the code is viable for
general use once again.
The implementation requires modifying frozen parameters, but I still
consider it by far preferable to needing the logic to select the
modified endpoint.
Added LICENSE and brief readme
Updated network checking with more specifics to Salad Simple Storage.
The access token is now fetched, checkpoint size is checked before
uploading and checkpoints are read from the inputs folder to ease file
management on workflow dispatch
Aimed at targeting the Salad Simple Storage,
When net checkpointing is used, there is a system of priority to
requests and requests are uploaded asynchronously.
Logic has been further cleaned up in preparation for networking
integration. There's not longer unique behaviour for what is being saved
and the onus is placed on the caller to provide data in a safetensor
appropriate format
Supported has been added to cache the output of completed nodes. This
code should be trivially expandable to non-sampler nodes, but will need
to be cautiously enabled on a per-node basis to better regulate dataflow
In preperation for saving checkpoints remotely, all checkpoint
transactions are now wrapped.
Fixed step accumulation on consecutive failures
Support all builtin samplers
When sampling occurs, intermediate steps are cached to disk.
If at any point execution is halted, interrupted, or crashes, these
checkpoints are loaded from to restore execution from the previous step
Currently largely proof of concept. All samplers will cache, but
restoring is only applied to CustomSampler. Boilerplate is largely in
place to allow for caching output of other non-sampling nodes and will
be investigated later.
Austin Mroz