164 lines
7.8 KiB
Markdown
164 lines
7.8 KiB
Markdown
# ComfyUI-Keyframed
|
|
|
|
🚧 Work In Progress 🚧 - ComfyUI nodes to facilitate value keyframing by providing an interface for using [keyframed](https://github.com/dmarx/keyframed) in ComfyUI workflows.
|
|
|
|
Similar project you might find more convenient for certain use cases https://github.com/FizzleDorf/ComfyUI_FizzNodes
|
|
|
|
<!--ts-->
|
|
* [ComfyUI-Keyframed](#comfyui-keyframed)
|
|
* [Overview](#overview)
|
|
* [Starter Workflows](#starter-workflows)
|
|
* [Prompt Scheduling](#prompt-scheduling)
|
|
* [Prompt Interleaving (aka Prompt Entanglement, aka Prompt Superposition)](#prompt-interleaving-aka-prompt-entanglement-aka-prompt-superposition)
|
|
* [Simple Curved Parameter](#simple-curved-parameter)
|
|
* [Multi-Prompt Transition With Manually Specified Curves](#multi-prompt-transition-with-manually-specified-curves)
|
|
* [Nodes](#nodes)
|
|
* [Curve Constructors](#curve-constructors)
|
|
* [Curve From String](#curve-from-string)
|
|
* [Curve From YAML](#curve-from-yaml)
|
|
* [Constant-Valued Curve](#constant-valued-curve)
|
|
* [Entangled Curves](#entangled-curves)
|
|
* [Curve Operators](#curve-operators)
|
|
* [Evaluate Curve At T](#evaluate-curve-at-t)
|
|
* [Apply Curve To Conditioning](#apply-curve-to-conditioning)
|
|
* [Add Conditions](#add-conditions)
|
|
* [Curve Arithmetic Operators](#curve-arithmetic-operators)
|
|
* [Scheduling](#scheduling)
|
|
* [Keyframed Condition](#keyframed-condition)
|
|
* [Interpolation Methods](#interpolation-methods)
|
|
* [Set Keyframe](#set-keyframe)
|
|
* [Evaluate Schedule](#evaluate-schedule)
|
|
<!--te-->
|
|
|
|
|
|
# Overview
|
|
|
|
**Philosophy**
|
|
|
|
* Treat curves/schedules and keyframes as objects that can be passed around, plugged and unplugged, interchanged, and manipulated atomically.
|
|
* Leverage nodes to facilitate modularity and flexibility.
|
|
* Facilitate fast iteration
|
|
* Provide convenience functions for most common use cases, and also low-leverl operators capable of reproducing the behavior of those convenience functions to permit user customization in "node space".
|
|
|
|
|
|
# Starter Workflows
|
|
|
|
## Prompt Scheduling
|
|
|
|
|
|
|
|
This one is probably why you are here. This workflow demonstrates how to use the `keyframed/schedule` nodes to achieve similar behavior as [FizzNodes'](https://github.com/FizzleDorf/ComfyUI_FizzNodes) **PromptSchedule** node, but implemented differently.
|
|
|
|
This schedule is essentailly a normal AnimateDiff workflow where several nodes have replaced the normal conditioning setup. Rather than a single `CLIP Text Encode` node, we can have multiple prompts which transition sequentially over time. For documentation detailing how this workflow works, see the [`Nodes > Scheduling`](https://github.com/dmarx/ComfyUI-Keyframed/blob/dev/README.md#scheduling) section below.
|
|
|
|
## Prompt Interleaving (aka Prompt Entanglement, aka Prompt Superposition)
|
|
|
|
|
|
|
|
Which is the node equivalent for achieving this type of thing
|
|
|
|
|
|
|
|
For documentation detailing how this workflow works, see the [`Nodes > Entangled Curves`](https://github.com/dmarx/ComfyUI-Keyframed?tab=readme-ov-file#entangled-curves) section below.
|
|
|
|
## Simple Curved Parameter
|
|
|
|
|
|
|
|
|
|
## Multi-Prompt Transition With Manually Specified Curves
|
|
|
|
|
|
|
|
If you're feeling adventurous, this workflow demonstrates how you would use the curve objects directly to acheive the same thing as a schedule. Each prompt gets its own seaparate curve indicating the weight of the prompt at that time (you probably want the various conditionings weights to sum to 1 when combined. If you need to "fill" missing conditioning weight, try using an empty prompt).
|
|
|
|
# Nodes
|
|
|
|
## Curve Constructors
|
|
|
|
### Curve From String
|
|
|
|
|
|
|
|
|
|
### Curve From YAML
|
|
|
|
|
|
|
|
|
|
### Constant-Valued Curve
|
|
|
|
|
|
|
|
|
|
### Entangled Curves
|
|
|
|
|
|
|
|
Each output curve of the node is a sine wave that oscillates from `0` to `1` at the given frequency or wavelength. The outputs of a given node are phase-offset such that at any given time, the sum of the generated curves is `1`.
|
|
|
|
Reference the [Prompt Interleaving Workflow](https://github.com/dmarx/ComfyUI-Keyframed/blob/dev/README.md#prompt-interleaving-aka-prompt-entanglement-aka-prompt-superposition) for a demonstrative example.
|
|
|
|
## Curve Operators
|
|
|
|
### Evaluate Curve At T
|
|
|
|
|
|
|
|
|
|
### Apply Curve To Conditioning
|
|
|
|
|
|
|
|
Generates a batch of `n` conditionings multiplying each conditioning by th value of the curve at that conditioning's index within the batch. If a batch of latents is provided, `n` is ignored and the size of the latent batch is used. If the provided conditioning object has batch size > 1, the conditioning's batch size takes priority over the latent batch size and `n`.
|
|
|
|
|
|
### Add Conditions
|
|
|
|
|
|
|
|
|
|
### Curve Arithmetic Operators
|
|
|
|
|
|
|
|
NB: the division operator is unreliable at the time of this writing (2023-12-09).
|
|
|
|
|
|
## Scheduling
|
|
|
|
These nodes work together to facilitate transitioning through a sequence of conditionings (i.e. prompts). We'll call this sequence the "schedule" of the conditionings. The primary use case here is for manipulating the positive prompt, i.e. for building a "prompt schedule". Given a particular time (e.g. frame id) in an animation seuquence, we can query the prompt schedule at that time to get the appropriate conditioning to pass to the KSampler.
|
|
|
|
Reference the [Prompt Scheduling Workflow](https://github.com/dmarx/ComfyUI-Keyframed/tree/dev?tab=readme-ov-file#prompt-scheduling) for a demonstrative example
|
|
|
|
### Keyframed Condition
|
|
|
|
|
|
|
|
This node attaches a `conditioning` to a `keyframe`. This let's us assign a time to the conditioning and set what interpolation method to use when we're between keyframes.
|
|
|
|
#### Interpolation Methods
|
|
|
|
Consider three time points `a,b,c` such that `a<b<c`, and two keyframes `X,Y` such that `X.time = a` and `Y.time = c`. To interpolate a value at time `b`, we would use `X.interpolation_method` to "tween" the value between `X.value` and `Y.value`.
|
|
|
|
* **`null`** - If the interpolation value is not set, the default interpolation is "previous".
|
|
* **`previous`** - `X.time`
|
|
* **`next`** - `Y.time`
|
|
* **`linear`** - normal linear lerp. Matches the behavior of Deforum and FizzNodes.
|
|
* **`sine_wave`** - sine function easing. slower close to terminal values, fastest at the middle of the transition.
|
|
* **`eased_lerp** - sin2 easing. similar to sine but starts and ends slower and the fastest part is faster.
|
|
* **`exp_decay`** - starts fast ends slow.
|
|
|
|
### Set Keyframe
|
|
|
|
|
|
|
|
Attaches a keyframe to a schedule. If you haven't created a schedule yet, pass your keyframe into this node to create one, then pass the output schedule to subsequent `Set Keyframe` nodes to attach additional keyframes to the schedule. The first keyframe in the schedule should always be at `time=0`.
|
|
|
|
### Evaluate Schedule
|
|
|
|
|
|
|
|
Evaluate the schedule at a time or time slice to extract a single conditioning or conditioning batch (respectively) to provide to e.g. a KSampler. The non-batch version gives the same kind of conditioning you'd get from a `CLIPTextConditioning` node. The intention of the batch version is to use with AnimateDiff to apply different conditionings per frame.
|
|
|