diff --git a/README.MD b/README.MD index 3243afd..d848b53 100644 --- a/README.MD +++ b/README.MD @@ -14,14 +14,43 @@ Control nodes that generate changing values over separate executions of a workfl - **IntSequence**: Cycles through a comma-separated list of integer values. - **StringSequence**: Cycles through a list of strings (one per line). +### Motion Controls +- **MotionController**: Advanced float-based motion control for smooth animations. +- **IntegerMotionController**: Integer-based motion control for discrete value animations. +- **ROINode**: Region of Interest node for motion tracking and control. + ### Utility Controls - **FPSMonitor**: Generates an FPS overlay as an image and mask, useful for monitoring performance. +- **QuickShapeMask**: Rapidly generate shape masks (circle, square) with customizable dimensions. +- **DTypeConverter**: Convert masks between different data types (float16, uint8, float32, float64). +- **FastWebcamCapture**: High-performance webcam capture node with resizing capabilities. + +### VAE Tools +- **TAESDVaeEncode**: TAESD VAE encoding node. +- **TAESDVaeDecode**: TAESD VAE decoding node. + +## Movement Patterns + +All value and motion controls support various movement patterns: +- **Sine**: Smooth sinusoidal motion +- **Triangle**: Linear interpolation with smooth direction changes +- **Sawtooth**: Linear interpolation with sharp resets +- **Square**: Instant transitions between min/max values +- **Static**: No movement (constant value) +- **and more** ## Usage ### Value Controls Connect any value control node to the input of the node you want to animate. These nodes use movement patterns like sine, bounce, etc. to smoothly transition between values. +### Motion Controls +Use motion controllers for more advanced animation control: +- Set minimum/maximum values +- Control steps per cycle +- Choose movement patterns +- Apply to any numeric parameter + ### Sequence Controls Sequence controls allow you to specify exact values to cycle through. You can control: - Steps per item: How many frames to show each value @@ -30,10 +59,16 @@ Sequence controls allow you to specify exact values to cycle through. You can co ### FPS Monitor Outputs an image and mask showing current and average FPS. Useful for performance monitoring in real-time workflows. -## Demo -In this demo we are controlling the width and height of a shape mask with an Int Control node. Imagine controlling the denoise on a KSampler with a Float Control, though! Or CFG in StreamDiffusion! +## Examples + +### Value Control Demo +In this demo we are controlling the width and height of a shape mask with an Int Control node: ![Value Control Demo](assets/ValueControl.gif) +### Motion Active Blur Demo +Example of motion-based active blur effect: +![Motion Active Blur Demo](https://github.com/user-attachments/assets/e2a7a5e2-651a-4b0d-b3f9-12cb40acce04) + ## Installation ### Option 1: ComfyUI Manager (Coming soon!!) @@ -48,10 +83,58 @@ In this demo we are controlling the width and height of a shape mask with an Int 1. Clone this repository into your ComfyUI custom_nodes folder: ```bash cd ComfyUI/custom_nodes -git clone https://github.com/your-repo/comfyui-control-nodes.git +git clone https://github.com/ryanontheinside/ComfyUI_RealTimeNodes ``` 2. Install the required dependencies: ```bash +cd ComfyUI_RealTimeNodes pip install -r requirements.txt -``` \ No newline at end of file +``` + +## Coming Soon + +### Advanced Detection System +The next major update will introduce a comprehensive detection system with multiple detector types: +- **Motion Detection**: Enhanced motion detection with configurable sensitivity and regions +- **Object Detection**: Real-time object detection and tracking +- **Face Detection**: Face detection and landmark tracking +- **Pose Detection**: Human pose estimation and tracking +- **Depth Detection**: Real-time depth estimation and segmentation + +These detection systems will provide powerful inputs for real-time ComfyUI workflows, enabling dynamic responses to various types of detected changes in the input stream. + +## Project Status + +This is an evolving project that aims to expand the real-time capabilities of ComfyUI. As real-time use cases for ComfyUI continue to emerge and grow, this project will adapt and expand to meet those needs. The goal is to provide a comprehensive suite of tools for real-time workflows, from simple value animations to complex detection and response systems. + +### Contributing + +Your feedback and contributions are more than welcome! This project grows stronger with community input. + +- Have an idea? Open an issue! +- Found a bug? Open an issue! +- Made an improvement? Submit a PR! +- Want to help? Join the discussion! + +Please visit our [GitHub Issues](https://github.com/ryanontheinside/ComfyUI_RealTimeNodes/issues) page to contribute. + +## Related Projects + +### ComfyUI_RyanOnTheInside - Everything Reactivity +Make anything react to anything in your ComfyUI workflows. [ComfyUI_RyanOnTheInside](https://github.com/ryanontheinside/ComfyUI_RyanOnTheInside) - my main custom nodes suite that brings complete reactive control to standard ComfyUI workflows: + +- Dynamic node relationships +- React to audio, MIDI, motion, time, depth, color, Whisper, and more +- Audio source separation and manipulation +- Reactive particle systems +- Reactive text generation +- Reactive image generation +- Reactive video generation +- Optical flow +- Reactive IPAdapters and CogVideo +- Reactive Live Portrait +- Reactive DepthFlow +- Actually more + +Use it alongside these Control Nodes to master both the batch and real-time paradigms in ComfyUI. \ No newline at end of file diff --git a/__init__.py b/__init__.py index be86fd7..c977445 100644 --- a/__init__.py +++ b/__init__.py @@ -4,7 +4,7 @@ from .controls.utility_controls import FPSMonitor from .controls.motion_controls import MotionController, ROINode, IntegerMotionController from .quick_shape_mask import QuickShapeMask from .tiny_vae import TAESDVaeEncode, TAESDVaeDecode -from .misc_nodes import DTypeConverter +from .misc_nodes import DTypeConverter, FastWebcamCapture NODE_CLASS_MAPPINGS = { "FloatControl": FloatControl, @@ -22,7 +22,8 @@ NODE_CLASS_MAPPINGS = { "QuickShapeMask": QuickShapeMask, "TAESDVaeEncode": TAESDVaeEncode, "TAESDVaeDecode": TAESDVaeDecode, - "DTypeConverter": DTypeConverter + "DTypeConverter": DTypeConverter, + "FastWebcamCapture": FastWebcamCapture, } NODE_DISPLAY_NAME_MAPPINGS = { @@ -41,7 +42,8 @@ NODE_DISPLAY_NAME_MAPPINGS = { "QuickShapeMask": "Quick Shape Mask (RyanOnTheInside)", "TAESDVaeEncode": "TAESD VAE Encode (RyanOnTheInside)", "TAESDVaeDecode": "TAESD VAE Decode (RyanOnTheInside)", - "DTypeConverter": "DType Converter (RyanOnTheInside)" + "DTypeConverter": "DType Converter (RyanOnTheInside)", + "FastWebcamCapture": "Fast Webcam Capture (RyanOnTheInside)", } WEB_DIRECTORY = "./web/js" diff --git a/examples/dead_inside_512.png b/examples/dead_inside_512.png new file mode 100644 index 0000000..581b855 Binary files /dev/null and b/examples/dead_inside_512.png differ diff --git a/examples/motioncontrol_example_API.json b/examples/motioncontrol_example_API.json new file mode 100644 index 0000000..93b4097 --- /dev/null +++ b/examples/motioncontrol_example_API.json @@ -0,0 +1,230 @@ +{ + "10": { + "inputs": { + "action": "add", + "value": 2, + "mask": [ + "11", + 0 + ], + "next_roi": [ + "24", + 0 + ] + }, + "class_type": "ROINode", + "_meta": { + "title": "ROI Node (RyanOnTheInside)" + } + }, + "11": { + "inputs": { + "shape": "circle", + "width": 64, + "height": 64, + "x": 85, + "y": 85, + "canvas_width": 512, + "canvas_height": 512, + "batch_size": 1 + }, + "class_type": "QuickShapeMask", + "_meta": { + "title": "Quick Shape Mask (RyanOnTheInside)" + } + }, + "24": { + "inputs": { + "action": "subtract", + "value": 2, + "mask": [ + "25", + 0 + ] + }, + "class_type": "ROINode", + "_meta": { + "title": "ROI Node (RyanOnTheInside)" + } + }, + "25": { + "inputs": { + "shape": "circle", + "width": 64, + "height": 64, + "x": 427, + "y": 85, + "canvas_width": 512, + "canvas_height": 512, + "batch_size": 1 + }, + "class_type": "QuickShapeMask", + "_meta": { + "title": "Quick Shape Mask (RyanOnTheInside)" + } + }, + "28": { + "inputs": { + "value": 0.15, + "width": 512, + "height": 512 + }, + "class_type": "SolidMask", + "_meta": { + "title": "SolidMask" + } + }, + "29": { + "inputs": { + "x": 0, + "y": 0, + "operation": "subtract", + "destination": [ + "11", + 0 + ], + "source": [ + "28", + 0 + ] + }, + "class_type": "MaskComposite", + "_meta": { + "title": "MaskComposite" + } + }, + "30": { + "inputs": { + "x": 0, + "y": 0, + "operation": "subtract", + "destination": [ + "25", + 0 + ], + "source": [ + "28", + 0 + ] + }, + "class_type": "MaskComposite", + "_meta": { + "title": "MaskComposite" + } + }, + "32": { + "inputs": { + "kernel_size": 61, + "sigma": [ + "35", + 0 + ], + "mode": "cuda", + "images": [ + "39", + 0 + ] + }, + "class_type": "GaussianBlur", + "_meta": { + "title": "GaussianBlur" + } + }, + "34": { + "inputs": { + "images": [ + "37", + 0 + ] + }, + "class_type": "PreviewImage", + "_meta": { + "title": "Preview Image" + } + }, + "35": { + "inputs": { + "always_execute": true, + "threshold": 0.1, + "blur_size": 5, + "minimum_value": 1, + "maximum_value": 100, + "starting_value": 10, + "rounding_mode": "round", + "image": [ + "39", + 0 + ], + "roi_chain": [ + "10", + 0 + ] + }, + "class_type": "IntegerMotionController", + "_meta": { + "title": "Integer Motion Controller (RyanOnTheInside)" + } + }, + "36": { + "inputs": { + "x": 0, + "y": 0, + "operation": "add", + "destination": [ + "29", + 0 + ], + "source": [ + "30", + 0 + ] + }, + "class_type": "MaskComposite", + "_meta": { + "title": "MaskComposite" + } + }, + "37": { + "inputs": { + "mode": "cuda", + "foregrounds": [ + "40", + 0 + ], + "backgrounds": [ + "32", + 0 + ], + "foreground_masks": [ + "36", + 0 + ] + }, + "class_type": "Composite", + "_meta": { + "title": "Composite" + } + }, + "39": { + "inputs": { + "image": "dead_inside_512.png", + "upload": "image" + }, + "class_type": "LoadImage", + "_meta": { + "title": "Load Image" + } + }, + "40": { + "inputs": { + "mask": [ + "36", + 0 + ] + }, + "class_type": "MaskToImage", + "_meta": { + "title": "Convert Mask to Image" + } + } +} \ No newline at end of file diff --git a/misc_nodes.py b/misc_nodes.py index 5169809..4f33601 100644 --- a/misc_nodes.py +++ b/misc_nodes.py @@ -1,5 +1,15 @@ import torch import comfy.utils +import numpy as np +import base64 +import re +from io import BytesIO +from PIL import Image +import nodes +import random +from torchvision import transforms + +MAX_RESOLUTION = nodes.MAX_RESOLUTION # Get the same max resolution as core nodes class AnyType(str): def __ne__(self, __value: object) -> bool: @@ -43,3 +53,51 @@ class DTypeConverter: converted = mask.to(target_dtype) return (converted,) + + +class FastWebcamCapture: + @classmethod + def INPUT_TYPES(s): + return { + "required": { + "image": ("WEBCAM", {}), + "width": ("INT", {"default": 640, "min": 0, "max": MAX_RESOLUTION, "step": 1}), + "height": ("INT", {"default": 480, "min": 0, "max": MAX_RESOLUTION, "step": 1}), + "capture_on_queue": ("BOOLEAN", {"default": True}), + } + } + + RETURN_TYPES = ("IMAGE",) + FUNCTION = "process_capture" + + CATEGORY = "image" + + def process_capture(self, image, width, height, capture_on_queue): + # Check if we got a data URL + if isinstance(image, str) and image.startswith('data:image/'): + # Extract the base64 data after the comma + base64_data = re.sub('^data:image/.+;base64,', '', image) + + # Convert base64 to PIL Image + buffer = BytesIO(base64.b64decode(base64_data)) + pil_image = Image.open(buffer).convert("RGB") + + # Convert PIL to numpy array + image = np.array(pil_image) + + # Handle resize if requested + if width > 0 and height > 0: + import cv2 + image = cv2.resize(image, (width, height), interpolation=cv2.INTER_AREA) + + # Convert to float32 and normalize to 0-1 range + image = image.astype(np.float32) / 255.0 + + # Add batch dimension and convert to torch tensor + # ComfyUI expects BHWC format + image = torch.from_numpy(image)[None,...] + + return (image,) + else: + raise ValueError("Invalid image format received from webcam") + diff --git a/web/js/webcam.js b/web/js/webcam.js new file mode 100644 index 0000000..aef382e --- /dev/null +++ b/web/js/webcam.js @@ -0,0 +1,106 @@ +import { app } from "../../../scripts/app.js"; +//TODO: fix occlusion of dimension editing popups +app.registerExtension({ + name: "Core.FastWebcam", + async beforeRegisterNodeDef(nodeType, nodeData, app) { + if (nodeData.name === "FastWebcamCapture") { + console.log("Registering FastWebcamCapture node type"); + + // Store webcam stream at extension level to avoid reinitializing + let webcamStream = null; + let webcamVideo = null; + + // Initialize webcam once for all nodes + const initWebcam = async () => { + // console.log("Initializing global webcam stream..."); + if (!webcamStream) { + try { + webcamStream = await navigator.mediaDevices.getUserMedia({ + video: true, + audio: false + }); + // console.log("Got media stream:", webcamStream); + + webcamVideo = document.createElement("video"); + webcamVideo.srcObject = webcamStream; + webcamVideo.style.display = 'none'; + document.body.appendChild(webcamVideo); + await webcamVideo.play(); + + console.log("Video playing. Video state:", { + paused: webcamVideo.paused, + readyState: webcamVideo.readyState, + videoWidth: webcamVideo.videoWidth, + videoHeight: webcamVideo.videoHeight + }); + } catch (error) { + // console.error("Webcam initialization error:", error); + throw error; + } + } + return { stream: webcamStream, video: webcamVideo }; + }; + + nodeType.prototype.getCustomWidgets = function() { + // console.log("Getting custom widgets"); + return { + WEBCAM: (node, inputName) => { + // console.log("Creating WEBCAM widget"); + const container = document.createElement("div"); + container.style.background = "rgba(0,0,0,0.25)"; + container.style.textAlign = "center"; + + const statusLabel = document.createElement("div"); + statusLabel.textContent = "Webcam Ready"; + container.appendChild(statusLabel); + + const widget = node.addDOMWidget(inputName, "WEBCAM", container); + return { widget }; + } + }; + }; + + const onNodeCreated = nodeType.prototype.onNodeCreated; + nodeType.prototype.onNodeCreated = function() { + // console.log("Node created"); + const r = onNodeCreated ? onNodeCreated.apply(this, arguments) : undefined; + + const camera = this.widgets.find((w) => w.name === "image"); + const w = this.widgets.find((w) => w.name === "width"); + const h = this.widgets.find((w) => w.name === "height"); + + const canvas = document.createElement("canvas"); + + camera.serializeValue = async () => { + // console.log("Serializing camera value"); + + // Only initialize webcam when actually needed + if (!webcamVideo || webcamVideo.paused || webcamVideo.readyState !== 4) { + // console.log("Initializing webcam for first use..."); + const { video } = await initWebcam(); + webcamVideo = video; + } + + console.log("Capturing frame from video:", { + paused: webcamVideo.paused, + readyState: webcamVideo.readyState, + videoWidth: webcamVideo.videoWidth, + videoHeight: webcamVideo.videoHeight + }); + + canvas.width = w.value || webcamVideo.videoWidth || 640; + canvas.height = h.value || webcamVideo.videoHeight || 480; + + const ctx = canvas.getContext("2d"); + ctx.drawImage(webcamVideo, 0, 0, canvas.width, canvas.height); + + const dataUrl = canvas.toDataURL("image/png"); + // console.log("Successfully captured and converted frame to data URL"); + return dataUrl; + }; + + return r; + }; + } + } +}); \ No newline at end of file