Add mask output and options to CameraMotionNode

5 loadpointcloud node errors

README.md

@@ -1,5 +1,5 @@
 # camera-comfyUI
-
+[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/Alexankharin/camera-comfyUI)
 ![ComfyUI Custom Nodes](demo_images/Camera_interpolation_pointcloud.gif)
 
 > Custom ComfyUI nodes for advanced reprojections, point cloud processing, and camera-driven workflows.
@@ -110,7 +110,7 @@ A collection of ComfyUI custom nodes to handle diverse camera projections (pinho
 | `ZDepthToRayDepthNode`    | Converts Z-depth (output of metric-depth-anything) to ray depth to compensate lens curvature.                        |
 | `TransformPointCloud`     | Applies 4×4 rotation matrix to point cloud                                    |
 | `ProjectPointCloud`       | Z-buffer–based projection of point cloud into image + mask.                   |
-| `CameraMotionNode`        | Generates image sequences by moving camera along a trajectory.                |
+| `CameraMotionNode`        | Generates image and mask sequences along a camera trajectory with optional mask dilation/inversion. |
 | `CameraInterpolationNode` | Builds a trajectory tensor from two poses.                                    |
 | `CameraTrajectoryNode`    | Interactive Open3D GUI for recording camera waypoints.                        |
 | `PointCloudCleaner`       | Removes isolated points via voxel filtering.                                  |
@@ -132,6 +132,7 @@ A set of JSON workflows illustrating typical use cases. Each workflow lives in `
 | **Pointcloud.json**                    | Metric‐depth‐anything v2 → point cloud → camera view synthesis |
 | **pointcloud\_inpaint.json**           | Inpaint + backproject to 3D for dynamic camera motion videos   |
 | **Pointcloud\_walker.json**            | GUI‐based camera control via Open3D                            |
+| **sbs180\_workflow.json**              | Generate stereo (side-by-side) wide-angle/fisheye/equirectangular stereo pairs from a high-res input |
 
 ---
 
@@ -190,6 +191,12 @@ Inpaint image with shifted camera and backproject for dynamic camera‐driven vi
 <img src="demo_images/Fisheye_camera_pointcloud_moved_outpainted.png" alt="PointCloud Inpaint" width="40%" />
 <img src="demo_images/Camera_interpolation_pointcloud.gif" alt="PointCloud Inpaint Video" width="40%" />
 
+### 9. `sbs180_workflow.json`
+
+Take a wide-angle (fisheye or equirectangular) high-resolution (e.g., 4096×4096) image and generate a stereo pair by moving the camera horizontally. The output is a wide-angle stereo pair (side-by-side), simulating a fisheye or equirectangular stereo camera.
+
+<img src="demo_images/equirect_stereo.gif" alt="Equirectangular Stereo Demo" width="80%" />
+
 ### 10. `Pointcloud_walker.json`
 
 Interactive Open3D-based GUI for walking and setting camera trajectory inside pointcloud.
@@ -202,7 +209,7 @@ Contributions welcome! Please open issues or PRs to add features, improve docs,
 
 ## TODO List
 
-* [ ] Add processing to pointcloud or depthmap to remove outlier and lonely points at depth borders.
+* [x] Add processing to pointcloud or depthmap to remove outlier and lonely points at depth borders.
 * [x] Use built-in comfyUI mask type an image.
 * [x] Unite nodes into groups to simplify workflows.
 * [ ] Create a single workflow for view synthesis.
@@ -210,3 +217,4 @@ Contributions welcome! Please open issues or PRs to add features, improve docs,
 * [x] Add more examples and documentation for each node.
 * [x] Add pointcloud union
 * [ ] Fix imports for renamed folders (e.g., inpainting_flux)
+* [ ] Integrate camera movement pipeline with video models (e.g., wan2.1) for smooth, high-quality inpainting along camera trajectories.

install.sh

@@ -0,0 +1,49 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# 1. Install PyTorch with CUDA 12.8 wheels
+echo "Installing PyTorch, TorchVision, TorchAudio..."
+pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128
+
+# 2. Update apt repositories and install system dependencies
+echo "Updating apt and installing build-essential, ffmpeg, libsm6, libxext6..."
+sudo apt-get update
+sudo apt-get install -y build-essential ffmpeg libsm6 libxext6
+
+# 3. Clone ComfyUI and install its Python requirements
+echo "Cloning ComfyUI..."
+git clone https://github.com/comfyanonymous/ComfyUI.git
+echo "Installing ComfyUI requirements..."
+pip3 install -r ComfyUI/requirements.txt
+
+# 4. Enter the custom_nodes folder
+cd ComfyUI/custom_nodes
+
+# 5. camera-comfyUI
+echo "Cloning camera-comfyUI..."
+git clone https://github.com/Alexankharin/camera-comfyUI.git
+echo "Installing camera-comfyUI requirements..."
+pip3 install camera-comfyUI/requirements.txt
+
+# 6. ComfyUI-Flux-Inpainting
+echo "Cloning ComfyUI-Flux-Inpainting..."
+git clone https://github.com/rubi-du/ComfyUI-Flux-Inpainting.git
+echo "Installing ComfyUI-Flux-Inpainting requirements..."
+pip3 install ComfyUI-Flux-Inpainting/requirements.txt
+
+# 7. ComfyUI-Image-Filters
+echo "Cloning ComfyUI-Image-Filters..."
+git clone https://github.com/spacepxl/ComfyUI-Image-Filters.git
+echo "Installing ComfyUI-Image-Filters requirements..."
+pip3 install ComfyUI-Image-Filters/requirements.txt
+
+# 8. Tidy up Flux Inpainting folder name
+echo "Renaming Flux Inpainting folder..."
+cd ..
+mv custom_nodes/ComfyUI-Flux-Inpainting-main custom_nodes/inpainting_flux
+
+# 9. Install Hugging Face Hub Python package
+echo "Installing huggingface_hub..."
+pip3 install huggingface_hub
+
+echo "All done! 🎉"

pointcloud_nodes.py

@@ -504,24 +504,45 @@ class LoadPointCloud:
             arr = np.load(file_path)
             tensor_pc = torch.from_numpy(arr)
             return (tensor_pc,)
-        coords = []
-        colors = []
-        with open(file_path, 'r') as f:
-            line = f.readline().strip()
-            while not line.startswith("end_header"):
+
+        if o3d is None:
+            logging.warning("[camera-comfyUI] open3d is not installed. Falling back to manual PLY parser.")
+            coords = []
+            colors = []
+            with open(file_path, 'r') as f:
                 line = f.readline().strip()
-            for line in f:
-                parts = line.strip().split()
-                if len(parts) < 7:
-                    continue
-                x, y, z = map(float, parts[0:3])
-                r, g, b, a = map(int, parts[3:7])
-                coords.append((x, y, z))
-                colors.append((r, g, b, a))
-        np_coords  = np.array(coords,  dtype=np.float32)
-        np_colors  = np.array(colors,  dtype=np.float32)/255.0
-        combined   = np.concatenate([np_coords, np_colors], axis=1)
-        tensor_pc  = torch.from_numpy(combined)
+                while not line.startswith("end_header"):
+                    line = f.readline().strip()
+                for line in f:
+                    parts = line.strip().split()
+                    if len(parts) < 7:
+                        continue
+                    x, y, z = map(float, parts[0:3])
+                    r, g, b, a = map(float, parts[3:7])
+                    coords.append((x, y, z))
+                    colors.append((r, g, b, a))
+            np_coords = np.array(coords, dtype=np.float32)
+            np_colors = np.array(colors, dtype=np.float32)
+            # if colors are > 1, normalize them to [0,1]
+            if np_colors.max() > 1.0:
+                np_colors = np_colors / 255.0
+        else:
+            pc = o3d.t.io.read_point_cloud(file_path)
+            np_coords = pc.point["positions"].numpy().astype(np.float32)
+            if "colors" in pc.point:
+                cols = pc.point["colors"].numpy().astype(np.float32)
+            else:
+                cols = np.ones((np_coords.shape[0], 3), dtype=np.float32)
+            if "alpha" in pc.point:
+                alpha = pc.point["alpha"].numpy().astype(np.float32)
+            else:
+                alpha = np.ones((np_coords.shape[0], 1), dtype=np.float32)
+            np_colors = np.concatenate([cols, alpha], axis=1)
+            if np_colors.max() > 1.0:
+                np_colors = np_colors / 255.0
+        # combine coords and colors into a single tensor
+        combined = np.concatenate([np_coords, np_colors], axis=1)
+        tensor_pc = torch.from_numpy(combined)
         return (tensor_pc,)
 
     @classmethod
@@ -586,24 +607,36 @@ class SavePointCloud:
         os.makedirs(full_output_folder, exist_ok=True)
         base_name = filename.replace("%batch_num%", "0")
         if save_as == "ply":
-            ply_name  = f"{base_name}_{counter:05}.ply"
-            ply_path  = os.path.join(full_output_folder, ply_name)
-            coords = pointcloud[:, :3].cpu().numpy()
-            colors = pointcloud[:, 3:].cpu().numpy().clip(0,1)
-            with open(ply_path, 'w') as f:
-                f.write("ply\n")
-                f.write("format ascii 1.0\n")
-                f.write(f"element vertex {coords.shape[0]}\n")
-                f.write("property float x\n")
-                f.write("property float y\n")
-                f.write("property float z\n")
-                f.write("property uchar red\n")
-                f.write("property uchar green\n")
-                f.write("property uchar blue\n")
-                f.write("property uchar alpha\n")
-                f.write("end_header\n")
-                for (x,y,z), (r,g,b,a) in zip(coords, colors):
-                    f.write(f"{x} {y} {z} {int(r*255)} {int(g*255)} {int(b*255)} {int(a*255)}\n")
+            ply_name = f"{base_name}_{counter:05}.ply"
+            ply_path = os.path.join(full_output_folder, ply_name)
+            coords = pointcloud[:, :3].cpu().numpy().astype(np.float32)
+            colors = pointcloud[:, 3:].cpu().numpy().clip(0, 1).astype(np.float32)
+
+            if o3d is None:
+                logging.warning("[camera-comfyUI] open3d is not installed. Falling back to manual ASCII PLY writer.")
+                with open(ply_path, 'w') as f:
+                    f.write("ply\n")
+                    f.write("format ascii 1.0\n")
+                    f.write(f"element vertex {coords.shape[0]}\n")
+                    f.write("property float x\n")
+                    f.write("property float y\n")
+                    f.write("property float z\n")
+                    f.write("property float red\n")
+                    f.write("property float green\n")
+                    f.write("property float blue\n")
+                    f.write("property float alpha\n")
+                    f.write("end_header\n")
+                    for (x, y, z), (r, g, b, a) in zip(coords, colors):
+                        f.write(f"{x} {y} {z} {r} {g} {b} {a}\n")
+            else:
+                pc = o3d.t.geometry.PointCloud()
+                pc.point["positions"] = o3d.core.Tensor(coords, o3d.core.float32)
+                pc.point["colors"] = o3d.core.Tensor(colors[:, :3], o3d.core.float32)
+                if colors.shape[1] > 3:
+                    pc.point["alpha"] = o3d.core.Tensor(colors[:, 3:], o3d.core.float32)
+                else:
+                    pc.point["alpha"] = o3d.core.Tensor(np.ones((coords.shape[0], 1), dtype=np.float32), o3d.core.float32)
+                o3d.t.io.write_point_cloud(ply_path, pc)
             file_name = ply_name
         else:
             npy_name = f"{base_name}_{counter:05}.npy"
@@ -640,10 +673,13 @@ class CameraMotionNode:
             "output_width":        ("INT",    {"default":512, "min":8, "max":16384}),
             "output_height":       ("INT",    {"default":512, "min":8, "max":16384}),
             "point_size":          ("INT",    {"default":1,   "min":1}),
+            "widen_mask":         ("INT",    {"default":0, "min":0, "max":64}),
+            "invert_mask":        ("BOOLEAN", {"default": False}),
+            "points_to_mask":     ("BOOLEAN", {"default": False, "tooltip": "Output mask frames of projected points"}),
         }}
 
-    RETURN_TYPES = ("IMAGE",)
-    RETURN_NAMES = ("motion_frames",)
+    RETURN_TYPES = ("IMAGE", "MASK")
+    RETURN_NAMES = ("motion_frames", "mask_frames")
     FUNCTION      = "generate_motion_frames"
     CATEGORY      = "Camera/pointcloud"
 
@@ -656,7 +692,10 @@ class CameraMotionNode:
         output_horizontal_fov: float,
         output_width:          int,
         output_height:         int,
-        point_size:            int = 1
+        point_size:            int = 1,
+        widen_mask:           int = 0,
+        invert_mask:          bool = False,
+        points_to_mask:       bool = False
     ) -> Tuple[torch.Tensor]:
         # validate trajectory shape
         if trajectory.dim() != 3 or trajectory.shape[1:] != (4,4):
@@ -683,9 +722,10 @@ class CameraMotionNode:
         proj_node      = ProjectPointCloud()
         transform_node = TransformPointCloud()
         frames = []
+        masks  = []
         for M in tqdm(full_traj):
             pc_t, = transform_node.transform_pointcloud(pointcloud, M)
-            img, _, _ = proj_node.project_pointcloud(
+            img, mask, _ = proj_node.project_pointcloud(
                 pc_t,
                 output_projection,
                 output_horizontal_fov,
@@ -693,10 +733,19 @@ class CameraMotionNode:
                 output_height,
                 point_size
             )
+            if widen_mask > 0:
+                k = 2 * widen_mask + 1
+                pad = widen_mask
+                mask = F.max_pool2d(mask.float().unsqueeze(0).unsqueeze(0), kernel_size=k, stride=1, padding=pad).squeeze(0).squeeze(0)
+            if invert_mask:
+                mask = 1.0 - mask
+            masks.append(mask)
+            if points_to_mask:
+                img = mask.unsqueeze(-1).repeat(1,1,1,3)
             frames.append(img[0])
 
         # output as (T,H,W,3)
-        return (torch.stack(frames, dim=0),)
+        return (torch.stack(frames, dim=0), torch.stack(masks, dim=0))
 
 class CameraInterpolationNode:
     """