nodes/DavidPiazza-network_bending
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

WIP: prepare for merge with upstream main, keep VAE/Latent nodes, keep upstream README and audio nodes

Browse Source
This commit is contained in:
David Piazza
2025-07-03 14:58:27 -04:00
parent b791f12225
commit 57a6fa0f76
5 changed files with 1591 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+143
View File
@@ -33,6 +33,54 @@ Mix two models together with various blending modes:
- **Frequency Blend**: Mix models in frequency domain
- **Random Mix**: Randomly select weights from either model
### 🔄 Latent Format Converter Node
Convert between audio and image latent formats to enable cross-modal workflows:
- **Audio to Image**: Convert 3D audio latents [B,C,L] to 4D image latents [B,C,H,W]
- **Image to Audio**: Convert 4D image latents [B,C,H,W] to 3D audio latents [B,C,L]
- **Auto Detect**: Automatically detect format and convert appropriately
- **Channel Conversion**: Convert between different channel counts (e.g., 4→64 for SD→StableAudio)
- **Multiple Methods**: Choose from reshape, interpolate, fold, or tile methods for both spatial and channel dims
### 🌀 VAE Network Bending Node
Apply network bending operations specifically to VAE models:
- **Add Noise**: Inject noise into encoder/decoder weights
- **Scale Weights**: Scale VAE weights for more/less aggressive encoding
- **Asymmetric Noise**: Different noise levels for encoder vs decoder
- **Channel Corruption**: Corrupt specific channels in the VAE
- **Progressive Corruption**: Gradually increase corruption through layers
- **Target Components**: Choose encoder, decoder, or both
### 🎯 VAE Mixer Node
Mix two VAE models with specialized blending strategies:
- **Linear Blend**: Blend both encoder and decoder linearly
- **Encoder Swap**: Use encoder from one VAE with decoder from another
- **Decoder Swap**: Use decoder from one VAE with encoder from another
- **Cross Architecture**: Mix encoder/decoder with different ratios
- **Layer Shuffle**: Randomly select layers from each VAE
### 💫 VAE Latent Bending Node
Perform operations directly on VAE latent streams:
- **Add Noise**: Add noise to latent values for texture variations
- **Channel Swap**: Swap latent channels for color shifts
- **Frequency Filter**: Apply frequency domain filtering
- **Spatial Corruption**: Corrupt specific spatial regions
- **Value Quantization**: Create posterization effects
- **Dimension Warp**: Warp spatial dimensions for distortions
### 🔮 VAE Channel Manipulator Node
Advanced channel-level operations on VAE latents:
- **Channel Attention**: Apply self-attention across channels
- **Cross Channel Mixing**: Mix information between channels
- **Channel Dropout**: Randomly drop channels
- **Channel Amplification**: Amplify important channels
- **Channel Rotation**: Rotate channels in feature space
- **Channel Statistics Swap**: Swap statistics with reference latent
## Installation
1. Clone this repository into your ComfyUI `custom_nodes` directory:
@@ -81,6 +129,33 @@ You can target specific layers using patterns:
4. Set the mix mode and ratio
5. Connect the output to your generation pipeline
### Latent Format Conversion
1. When you encounter dimension mismatch errors between audio and image models
2. Add a "Latent Format Converter" node between the latent source and destination
3. Set conversion mode (or use auto_detect)
4. Choose appropriate reshape method (fold works well for audio→image)
5. Adjust target dimensions if converting to image format
6. Set target_channels to match your model (4 for SD/SDXL, 64 for StableAudio)
7. Use "project" channel mode for best results when converting channels
### VAE Network Bending
1. Load a VAE or get one from a checkpoint loader
2. Add a "VAE Network Bending" node
3. Connect the VAE to the input
4. Choose operation and target component (encoder/decoder/both)
5. Set intensity and preserve_mean option
6. Use the bent VAE for encoding/decoding
### VAE Latent Manipulation
1. Encode an image to latent space using VAE
2. Add "VAE Latent Bending" or "VAE Channel Manipulator" nodes
3. Connect latents through the manipulation nodes
4. Experiment with different operations and intensities
5. Decode back to image to see the effects
## Parameters
### Network Bending Node
@@ -99,6 +174,52 @@ You can target specific layers using patterns:
| mix_mode | Select | Blending mode | See mix modes list |
| mix_ratio | Float | Blend ratio | 0.0 - 1.0 |
### Latent Format Converter Node
| Parameter | Type | Description | Range |
|-----------|------|-------------|-------|
| conversion_mode | Select | Conversion direction | audio_to_image, image_to_audio, auto_detect |
| target_height | Int | Height for image format | 8 - 512 (steps of 8) |
| target_width | Int | Width for image format | 8 - 512 (steps of 8) |
| reshape_method | Select | Method for reshaping | reshape, interpolate, fold, tile |
| target_channels | Int | Target channel count | -1 to 256 (-1 keeps original) |
| channel_mode | Select | Channel conversion method | project, pad, interpolate, tile |
### VAE Network Bending Node
| Parameter | Type | Description | Range |
|-----------|------|-------------|-------|
| operation | Select | VAE bending operation | See VAE operations list |
| intensity | Float | Strength of the operation | 0.0 - 1.0 |
| target_component | Select | Part of VAE to modify | both, encoder, decoder, latent_layers |
| preserve_mean | Boolean | Preserve weight means | True/False |
| seed | Int | Random seed | -1 to 2^63-1 |
### VAE Mixer Node
| Parameter | Type | Description | Range |
|-----------|------|-------------|-------|
| mix_mode | Select | VAE blending mode | See VAE mix modes list |
| mix_ratio | Float | Blend ratio | 0.0 - 1.0 |
### VAE Latent Bending Node
| Parameter | Type | Description | Range |
|-----------|------|-------------|-------|
| operation | Select | Latent operation | See latent operations list |
| intensity | Float | Operation strength | 0.0 - 1.0 |
| channel_specific | Boolean | Target specific channels | True/False |
| target_channels | String | Channel indices | Comma-separated (e.g., "0,1,2,3") |
### VAE Channel Manipulator Node
| Parameter | Type | Description | Range |
|-----------|------|-------------|-------|
| operation | Select | Channel operation | See channel operations list |
| intensity | Float | Operation strength | 0.0 - 1.0 |
| preserve_energy | Boolean | Preserve signal energy | True/False |
| reference_latent | Latent | Reference for some ops | Optional latent input |
## UI Features
- **Visual Feedback**: Operations display information about modified layers
@@ -125,6 +246,28 @@ You can target specific layers using patterns:
- Mix Ratio: `0.5`
- Creates a balanced hybrid of two models
### Cross-Modal Generation
- Use Latent Format Converter with `auto_detect`
- Reshape Method: `fold` (for audio→image)
- Enables using audio latents with image models
### VAE Glitch Effects
- VAE Network Bending: `channel_corruption`
- Target Component: `decoder`
- Intensity: `0.2`
- Creates colorful decoding artifacts
### Dreamy VAE Processing
- VAE Latent Bending: `frequency_filter`
- Intensity: `0.4`
- Creates soft, dream-like images
### Style Transfer via VAE
- VAE Channel Manipulator: `channel_statistics_swap`
- Intensity: `0.7`
- Use reference latent from style image
- Transfers color characteristics between images
## Tips
1. **Start with low intensity values** (0.01-0.1) and increase gradually
examples/vae_bending_example.json
+184
View File
@@ -0,0 +1,184 @@
{
"last_node_id": 15,
"last_link_id": 17,
"nodes": [
{
"id": 1,
"type": "CheckpointLoaderSimple",
"pos": [50, 50],
"size": {"0": 315, "1": 98},
"flags": {},
"order": 0,
"mode": 0,
"outputs": [
{"name": "MODEL", "type": "MODEL", "links": [], "slot_index": 0},
{"name": "CLIP", "type": "CLIP", "links": [1], "slot_index": 1},
{"name": "VAE", "type": "VAE", "links": [2, 3], "slot_index": 2}
],
"properties": {"Node name for S&R": "CheckpointLoaderSimple"},
"widgets_values": ["sd_xl_base_1.0.safetensors"]
},
{
"id": 2,
"type": "VAENetworkBending",
"pos": [400, 200],
"size": {"0": 315, "1": 166},
"flags": {},
"order": 3,
"mode": 0,
"inputs": [
{"name": "vae", "type": "VAE", "link": 2}
],
"outputs": [
{"name": "vae", "type": "VAE", "links": [4], "slot_index": 0}
],
"properties": {"Node name for S&R": "VAENetworkBending"},
"widgets_values": ["add_noise", 0.1, "decoder", true, -1]
},
{
"id": 3,
"type": "CLIPTextEncode",
"pos": [400, 50],
"size": {"0": 400, "1": 140},
"flags": {},
"order": 1,
"mode": 0,
"inputs": [
{"name": "clip", "type": "CLIP", "link": 1}
],
"outputs": [
{"name": "CONDITIONING", "type": "CONDITIONING", "links": [5], "slot_index": 0}
],
"properties": {"Node name for S&R": "CLIPTextEncode"},
"widgets_values": ["a beautiful landscape with mountains and lakes"]
},
{
"id": 4,
"type": "LoadImage",
"pos": [50, 250],
"size": {"0": 315, "1": 314},
"flags": {},
"order": 2,
"mode": 0,
"outputs": [
{"name": "IMAGE", "type": "IMAGE", "links": [6], "slot_index": 0},
{"name": "MASK", "type": "MASK", "links": null}
],
"properties": {"Node name for S&R": "LoadImage"},
"widgets_values": ["example.png", "image"]
},
{
"id": 5,
"type": "VAEEncode",
"pos": [750, 250],
"size": {"0": 210, "1": 46},
"flags": {},
"order": 5,
"mode": 0,
"inputs": [
{"name": "pixels", "type": "IMAGE", "link": 6},
{"name": "vae", "type": "VAE", "link": 4}
],
"outputs": [
{"name": "LATENT", "type": "LATENT", "links": [7, 8], "slot_index": 0}
],
"properties": {"Node name for S&R": "VAEEncode"}
},
{
"id": 6,
"type": "VAELatentBending",
"pos": [1000, 250],
"size": {"0": 315, "1": 142},
"flags": {},
"order": 6,
"mode": 0,
"inputs": [
{"name": "samples", "type": "LATENT", "link": 7}
],
"outputs": [
{"name": "samples", "type": "LATENT", "links": [9], "slot_index": 0}
],
"properties": {"Node name for S&R": "VAELatentBending"},
"widgets_values": ["frequency_filter", 0.3, false, "0,1,2,3"]
},
{
"id": 7,
"type": "VAEChannelManipulator",
"pos": [1350, 250],
"size": {"0": 315, "1": 118},
"flags": {},
"order": 7,
"mode": 0,
"inputs": [
{"name": "samples", "type": "LATENT", "link": 9},
{"name": "reference_latent", "type": "LATENT", "link": 8}
],
"outputs": [
{"name": "samples", "type": "LATENT", "links": [10], "slot_index": 0}
],
"properties": {"Node name for S&R": "VAEChannelManipulator"},
"widgets_values": ["channel_attention", 0.5, true]
},
{
"id": 8,
"type": "VAEDecode",
"pos": [1700, 250],
"size": {"0": 210, "1": 46},
"flags": {},
"order": 8,
"mode": 0,
"inputs": [
{"name": "samples", "type": "LATENT", "link": 10},
{"name": "vae", "type": "VAE", "link": 3}
],
"outputs": [
{"name": "IMAGE", "type": "IMAGE", "links": [11], "slot_index": 0}
],
"properties": {"Node name for S&R": "VAEDecode"}
},
{
"id": 9,
"type": "SaveImage",
"pos": [1950, 250],
"size": {"0": 315, "1": 270},
"flags": {},
"order": 9,
"mode": 0,
"inputs": [
{"name": "images", "type": "IMAGE", "link": 11}
],
"properties": {"Node name for S&R": "SaveImage"},
"widgets_values": ["ComfyUI"]
},
{
"id": 10,
"type": "Note",
"pos": [50, 600],
"size": {"0": 400, "1": 200},
"flags": {},
"order": 4,
"mode": 0,
"properties": {"text": ""},
"widgets_values": ["VAE Network Bending Example\n\nThis workflow demonstrates:\n1. VAENetworkBending - Adds noise to decoder\n2. VAELatentBending - Applies frequency filtering\n3. VAEChannelManipulator - Applies channel attention\n\nTry different operations and intensities to see how they affect the output!"],
"color": "#432",
"bgcolor": "#653"
}
],
"links": [
[1, 1, 1, 3, 0, "CLIP"],
[2, 1, 2, 2, 0, "VAE"],
[3, 1, 2, 8, 1, "VAE"],
[4, 2, 0, 5, 1, "VAE"],
[5, 3, 0, null, null, "CONDITIONING"],
[6, 4, 0, 5, 0, "IMAGE"],
[7, 5, 0, 6, 0, "LATENT"],
[8, 5, 0, 7, 1, "LATENT"],
[9, 6, 0, 7, 0, "LATENT"],
[10, 7, 0, 8, 0, "LATENT"],
[11, 8, 0, 9, 0, "IMAGE"]
],
"groups": [],
"config": {},
"extra": {},
"version": 0.4
}
requirements-audio.txt
+11
View File
@@ -0,0 +1,11 @@
# Audio processing dependencies for Stable Audio latent manipulation nodes
torch>=2.0.0
torchaudio>=2.0.0
librosa>=0.10.0
numpy>=1.21.0
scipy>=1.7.0
# Optional dependencies for advanced features
# soundfile>=0.12.0 # For additional audio format support
# audioread>=3.0.0 # For extended audio format compatibility
# resampy>=0.4.0 # For high-quality resampling
src/network_bending/js/network_bending.js
+204
View File
@@ -117,6 +117,195 @@ You can combine patterns with commas:
return result;
};
}
// Add descriptions for LatentFormatConverter
if (nodeData.name === "LatentFormatConverter") {
const onNodeCreated = nodeType.prototype.onNodeCreated;
nodeType.prototype.onNodeCreated = function() {
const result = onNodeCreated?.apply(this, arguments);
// Add conversion mode help
const conversionModeWidget = this.widgets.find(w => w.name === "conversion_mode");
if (conversionModeWidget) {
const modeHelpWidget = this.addWidget("button", "Conversion Mode Info", null, () => {
const descriptions = {
"audio_to_image": "Converts 3D audio latents [B,C,L] to 4D image latents [B,C,H,W]. Use when feeding audio-generated latents to image models.",
"image_to_audio": "Converts 4D image latents [B,C,H,W] to 3D audio latents [B,C,L]. Use when feeding image-generated latents to audio models.",
"auto_detect": "Automatically detects format based on tensor dimensions and converts accordingly."
};
const currentMode = conversionModeWidget.value;
const desc = descriptions[currentMode] || "No description available.";
app.ui.dialog.show(`${currentMode}:\n\n${desc}`);
});
}
// Add reshape method help
const reshapeMethodWidget = this.widgets.find(w => w.name === "reshape_method");
if (reshapeMethodWidget) {
const methodHelpWidget = this.addWidget("button", "Reshape Method Info", null, () => {
const descriptions = {
"reshape": "Direct reshape with padding/truncation. Fast but may lose spatial relationships.",
"interpolate": "Reshapes to intermediate size then interpolates. Better for preserving features.",
"fold": "Uses fold/unfold operations. Best for preserving local patterns and relationships.",
"tile": "Tiles the input to fill output space. Good for repeating patterns."
};
const currentMethod = reshapeMethodWidget.value;
const desc = descriptions[currentMethod] || "No description available.";
app.ui.dialog.show(`${currentMethod}:\n\n${desc}\n\nFor audio→image: fold often works best\nFor image→audio: reshape is usually sufficient`);
});
}
// Add channel mode help
const channelModeWidget = this.widgets.find(w => w.name === "channel_mode");
if (channelModeWidget) {
const channelHelpWidget = this.addWidget("button", "Channel Mode Info", null, () => {
const descriptions = {
"project": "Uses random projection matrix to convert channels. Best for maintaining information.",
"pad": "Pads with zeros or truncates channels. Simple but may lose information.",
"interpolate": "Interpolates between channels. Smooth transition but may blur features.",
"tile": "Tiles or averages channels. Good for repeating patterns."
};
const currentMode = channelModeWidget.value;
const desc = descriptions[currentMode] || "No description available.";
app.ui.dialog.show(`${currentMode}:\n\n${desc}\n\nCommon channel counts:\n• SD/SDXL: 4 channels\n• StableAudio: 64 channels\n• Set to -1 to keep original`);
});
}
return result;
};
}
// Add descriptions for VAENetworkBending
if (nodeData.name === "VAENetworkBending") {
const onNodeCreated = nodeType.prototype.onNodeCreated;
nodeType.prototype.onNodeCreated = function() {
const result = onNodeCreated?.apply(this, arguments);
const operationWidget = this.widgets.find(w => w.name === "operation");
if (operationWidget) {
const descriptionWidget = this.addWidget("button", "VAE Operation Info", null, () => {
const descriptions = {
"add_noise": "Adds Gaussian noise to VAE encoder/decoder weights. Can create artistic artifacts.",
"scale_weights": "Scales VAE weights. Can make encoding/decoding more or less aggressive.",
"asymmetric_noise": "Adds more noise to decoder than encoder. Creates decoding artifacts while preserving encoding.",
"latent_space_expand": "Modifies latent space dimensions. Can create interesting distortions.",
"channel_corruption": "Corrupts specific channels in the VAE. Creates color/feature artifacts.",
"encoder_decoder_swap": "Experimental: attempts to swap encoder/decoder roles.",
"progressive_corruption": "Gradually increases corruption through VAE layers. Creates gradient effects."
};
const currentOp = operationWidget.value;
const desc = descriptions[currentOp] || "No description available.";
app.ui.dialog.show(`VAE ${currentOp}:\n\n${desc}`);
});
}
// Add target component help
const targetWidget = this.widgets.find(w => w.name === "target_component");
if (targetWidget) {
const targetHelpWidget = this.addWidget("button", "Target Component Info", null, () => {
app.ui.dialog.show(`Target Component:\n\n• both - Modify both encoder and decoder\n• encoder - Only modify the encoder (image→latent)\n• decoder - Only modify the decoder (latent→image)\n• latent_layers - Target layers that process the latent space\n\nEncoder modifications affect how images are compressed.\nDecoder modifications affect how latents are reconstructed.`);
});
}
return result;
};
}
// Add descriptions for VAEMixer
if (nodeData.name === "VAEMixer") {
const onNodeCreated = nodeType.prototype.onNodeCreated;
nodeType.prototype.onNodeCreated = function() {
const result = onNodeCreated?.apply(this, arguments);
const mixModeWidget = this.widgets.find(w => w.name === "mix_mode");
if (mixModeWidget) {
const descriptionWidget = this.addWidget("button", "VAE Mix Mode Info", null, () => {
const descriptions = {
"linear_blend": "Blends both encoder and decoder weights linearly. Most stable option.",
"encoder_swap": "Uses encoder from VAE B with decoder from VAE A.",
"decoder_swap": "Uses decoder from VAE B with encoder from VAE A.",
"cross_architecture": "Mixes encoder more from A, decoder more from B (or vice versa).",
"frequency_mix": "Mixes VAEs in frequency domain for smoother blending.",
"layer_shuffle": "Randomly selects layers from each VAE. Creates hybrid architectures."
};
const currentMode = mixModeWidget.value;
const desc = descriptions[currentMode] || "No description available.";
app.ui.dialog.show(`${currentMode}:\n\n${desc}`);
});
}
return result;
};
}
// Add descriptions for VAELatentBending
if (nodeData.name === "VAELatentBending") {
const onNodeCreated = nodeType.prototype.onNodeCreated;
nodeType.prototype.onNodeCreated = function() {
const result = onNodeCreated?.apply(this, arguments);
const operationWidget = this.widgets.find(w => w.name === "operation");
if (operationWidget) {
const descriptionWidget = this.addWidget("button", "Latent Operation Info", null, () => {
const descriptions = {
"add_noise": "Adds noise directly to latent values. Creates texture variations.",
"channel_swap": "Swaps latent channels. Can create color shifts and inversions.",
"frequency_filter": "Applies frequency filtering to latents. Can smooth or sharpen features.",
"spatial_corruption": "Corrupts spatial regions in latents. Creates localized artifacts.",
"value_quantization": "Quantizes latent values. Creates posterization-like effects.",
"dimension_warp": "Warps spatial dimensions. Creates distortion effects.",
"temporal_shift": "Shifts values temporally. Useful for animation effects."
};
const currentOp = operationWidget.value;
const desc = descriptions[currentOp] || "No description available.";
app.ui.dialog.show(`Latent ${currentOp}:\n\n${desc}`);
});
}
return result;
};
}
// Add descriptions for VAEChannelManipulator
if (nodeData.name === "VAEChannelManipulator") {
const onNodeCreated = nodeType.prototype.onNodeCreated;
nodeType.prototype.onNodeCreated = function() {
const result = onNodeCreated?.apply(this, arguments);
const operationWidget = this.widgets.find(w => w.name === "operation");
if (operationWidget) {
const descriptionWidget = this.addWidget("button", "Channel Operation Info", null, () => {
const descriptions = {
"channel_attention": "Applies self-attention across channels. Enhances channel relationships.",
"cross_channel_mixing": "Mixes information between channels. Creates color blending effects.",
"channel_dropout": "Randomly drops channels. Can create missing color effects.",
"channel_amplification": "Amplifies important channels based on variance. Enhances dominant features.",
"channel_rotation": "Rotates channels in feature space. Creates color shifts.",
"channel_statistics_swap": "Swaps channel statistics with reference. Style transfer effect."
};
const currentOp = operationWidget.value;
const desc = descriptions[currentOp] || "No description available.";
app.ui.dialog.show(`Channel ${currentOp}:\n\n${desc}\n\nTip: Enable 'preserve_energy' to maintain overall signal strength.`);
});
}
return result;
};
}
},
// Add node color coding
@@ -130,6 +319,21 @@ You can combine patterns with commas:
} else if (node.comfyClass === "ModelMixer") {
node.color = "#243"; // Dark teal for mixer
node.bgcolor = "#365";
} else if (node.comfyClass === "LatentFormatConverter") {
node.color = "#324"; // Dark purple for converter
node.bgcolor = "#546";
} else if (node.comfyClass === "VAENetworkBending") {
node.color = "#423"; // Dark orange for VAE bending
node.bgcolor = "#645";
} else if (node.comfyClass === "VAEMixer") {
node.color = "#342"; // Dark green for VAE mixer
node.bgcolor = "#564";
} else if (node.comfyClass === "VAELatentBending") {
node.color = "#234"; // Dark cyan for latent bending
node.bgcolor = "#456";
} else if (node.comfyClass === "VAEChannelManipulator") {
node.color = "#324"; // Dark magenta for channel manipulation
node.bgcolor = "#546";
}
}
});
src/network_bending/nodes.py
+1049 -2
View File
File diff suppressed because it is too large Load Diff