172 lines
4.8 KiB
Markdown
172 lines
4.8 KiB
Markdown
# ComfyUI Coherent Video Sampler Node (V0.3)
|
|
|
|
A custom node for ComfyUI that enables coherent video generation while maintaining efficient memory usage, specifically optimized for heavy models like Flux.
|
|
|
|
|
|
|
|
## Features
|
|
|
|
- 🎥 Frame-by-frame video processing with motion preservation
|
|
- 🧠 Efficient memory management for heavy models
|
|
- 🔄 Progressive denoising with coherence maintenance
|
|
- 💫 Dynamic quality control and motion guidance
|
|
- 🎨 Style preservation across frames
|
|
- 🛠️ Advanced adjustment controls for fine-tuning
|
|
|
|
## Installation
|
|
|
|
Install from ComfyUI manager
|
|
|
|
or
|
|
|
|
Navigate to your ComfyUI custom nodes directory:
|
|
```bash
|
|
cd ComfyUI/custom_nodes
|
|
```
|
|
|
|
Clone this repository:
|
|
```bash
|
|
git clone https://github.com/ShmuelRonen/ComfyUI-CohernetVideoSampler.git
|
|
```
|
|
|
|
Restart ComfyUI
|
|
|
|
## Usage
|
|
|
|
#### For Deforum-like results please use 'shuttle-3-diffusion-fp8.safetensors' 4 steps flux model
|
|
|
|
The node appears in the node menu as "Cohernet Video Sampler".
|
|
|
|
### Core Parameters Guide
|
|
|
|
The sampler now includes four key adjustment parameters that work together to control different aspects of video generation:
|
|
|
|
1. **denoise** (0.0-1.0):
|
|
- Primary denoising control for the sampling process
|
|
- Controls overall deviation from input
|
|
- Lower values (0.3-0.5): Subtle changes, closer to input
|
|
- Higher values (0.7-0.9): More dramatic transformations
|
|
- Recommended: 0.6 for balanced results
|
|
|
|
2. **motion_strength** (0.0-1.0):
|
|
- Controls motion intensity between frames
|
|
- Affects transition smoothness
|
|
- Lower values (0.3-0.4): More static, stable output
|
|
- Higher values (0.7-0.8): Pronounced motion, dynamic transitions
|
|
- Recommended: 0.5 for natural movement
|
|
|
|
3. **consistency_strength** (0.0-1.0):
|
|
- Maintains visual consistency across frames
|
|
- Controls style preservation
|
|
- Lower values (0.7-0.8): More variation allowed
|
|
- Higher values (0.9-1.0): Strict consistency enforcement
|
|
- Recommended: 0.9 for coherent results
|
|
|
|
4. **denoise_strength** (0.0-1.0):
|
|
- Secondary denoising for artifact reduction
|
|
- Fine-tunes final output quality
|
|
- Lower values (0.5-0.7): Preserve more details
|
|
- Higher values (0.8-0.9): Smoother, cleaner output
|
|
- Recommended: 0.8 for balanced detail preservation
|
|
|
|
### Parameter Combinations for Different Effects
|
|
|
|
#### High Quality Stable Video
|
|
```
|
|
denoise: 0.6
|
|
motion_strength: 0.5
|
|
consistency_strength: 0.9
|
|
denoise_strength: 0.8
|
|
```
|
|
|
|
#### Dynamic Movement Priority
|
|
```
|
|
denoise: 0.5
|
|
motion_strength: 0.7
|
|
consistency_strength: 0.8
|
|
denoise_strength: 0.7
|
|
```
|
|
|
|
#### Maximum Detail Preservation
|
|
```
|
|
denoise: 0.4
|
|
motion_strength: 0.4
|
|
consistency_strength: 0.85
|
|
denoise_strength: 0.6
|
|
```
|
|
|
|
### Other Inputs
|
|
- `model`: Your diffusion model (tested extensively with Flux)
|
|
- `positive`: Positive prompt conditioning
|
|
- `negative`: Negative prompt conditioning
|
|
- `video_latents`: Input video in latent space (from VAE Encode)
|
|
- `seed`: Generation seed
|
|
- `steps`: Number of sampling steps
|
|
- `cfg`: Classifier free guidance scale
|
|
- `sampler_name`: Choice of sampler
|
|
- `scheduler`: Choice of scheduler
|
|
|
|
### Memory Management
|
|
The node implements several memory optimization techniques:
|
|
- Progressive batch processing
|
|
- Automatic VRAM cleanup
|
|
- Dynamic batch size adjustment
|
|
- Efficient latent space operations
|
|
|
|
This allows it to work smoothly even with memory-intensive models like Flux without OOM errors.
|
|
|
|
## Memory Usage Examples
|
|
|
|
When using with Flux model:
|
|
- 20 frame video @ 512x512: ~8GB VRAM
|
|
- 40 frame video @ 512x512: ~10GB VRAM
|
|
- Processing happens in windows of frames to maintain stable memory usage
|
|
|
|
## Optimization Tips
|
|
|
|
1. **For Smoother Videos:**
|
|
- Increase consistency_strength
|
|
- Decrease motion_strength slightly
|
|
- Keep denoise moderate
|
|
- Maintain high denoise_strength
|
|
|
|
2. **For More Dynamic Videos:**
|
|
- Increase motion_strength
|
|
- Decrease consistency_strength slightly
|
|
- Lower denoise_strength for detail
|
|
- Adjust denoise based on desired change level
|
|
|
|
3. **For Maximum Quality:**
|
|
- Balance all parameters
|
|
- Use higher consistency_strength
|
|
- Moderate motion_strength
|
|
- Higher denoise_strength
|
|
|
|
## Known Limitations
|
|
|
|
- Very long videos might need to be processed in segments
|
|
- Extreme motion can affect coherence
|
|
- High denoise values might reduce motion preservation
|
|
- Parameter interactions can be complex
|
|
|
|
## Future Plans
|
|
|
|
- Additional motion control parameters
|
|
- Custom denoising patterns
|
|
- Advanced style preservation options
|
|
- Multi-model support optimization
|
|
- Parameter presets for common use cases
|
|
|
|
## Contributing
|
|
|
|
Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.
|
|
|
|
## License
|
|
|
|
[MIT License](LICENSE)
|
|
|
|
## Acknowledgments
|
|
|
|
- ComfyUI team for the amazing framework
|
|
- Flux model team for the inspiration in handling heavy models
|