update splats stitching

Fix equirectangular projection aspect ratio to 2:1

.gitmodules

@@ -0,0 +1,3 @@
+[submodule "submodules/ml-sharpt"]
+	path = submodules/ml-sharpt
+	url = https://github.com/apple/ml-sharp

README.md

@@ -1,6 +1,7 @@
 # camera-comfyUI
 [![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/Alexankharin/camera-comfyUI)
 ![ComfyUI Custom Nodes](demo_images/Camera_interpolation_pointcloud.gif)
+![Camera Movement Demo](demo_images/camera_movement.gif)
 
 > Custom ComfyUI nodes for advanced reprojections, point cloud processing, and camera-driven workflows.
 
@@ -115,12 +116,20 @@ A collection of ComfyUI custom nodes to handle diverse camera projections (pinho
 | `DepthToPointCloud`       | Converts Depth and image to → 3D point cloud tensor (N×7).                    |
 | `DepthToImageNode`        | Converts depth to image (N×3) using a color map.                              |
 | `ZDepthToRayDepthNode`    | Converts Z-depth (output of metric-depth-anything) to ray depth to compensate lens curvature.                        |
-| `TransformPointCloud`     | Applies 4×4 rotation matrix to point cloud                                    |
+| `TransformPointCloud`     | Applies 4×4 rotation matrix to point cloud.                                   |
 | `ProjectPointCloud`       | Z-buffer–based projection of point cloud into image + mask.                   |
+| `PointCloudCleaner`       | Removes isolated points via voxel filtering.                                  |
+| `PointCloudUnion`         | Combines multiple point clouds into one.                                      |
+| `LoadPointCloud`          | Loads a point cloud from `.npy` or `.ply` format.                             |
+| `SavePointCloud`          | Saves a point cloud to `.npy` or `.ply` format.                               |
 | `CameraMotionNode`        | Generates image and mask sequences along a camera trajectory with optional mask dilation/inversion. |
 | `CameraInterpolationNode` | Builds a trajectory tensor from two poses.                                    |
 | `CameraTrajectoryNode`    | Interactive Open3D GUI for recording camera waypoints.                        |
-| `PointCloudCleaner`       | Removes isolated points via voxel filtering.                                  |
+| `SaveTrajectory`          | Saves a trajectory tensor to a file.                                          |
+| `LoadTrajectory`          | Loads a trajectory tensor from a file.                                        |
+| `VideoCameraMotionSequence` | Processes video frames and depth maps along a camera trajectory, generating reprojected outputs. |
+| `DepthFramesToVideo`      | Converts a sequence of depth maps into video frame tensors for saving.        |
+| `VideoMetricDepthEstimate` | Estimates metric depth for a sequence of frames using VideoDepthAnything.    |
 
 ---
 
@@ -140,6 +149,7 @@ A set of JSON workflows illustrating typical use cases. Each workflow lives in `
 | **pointcloud\_inpaint.json**           | Inpaint + backproject to 3D for dynamic camera motion videos   |
 | **Pointcloud\_walker.json**            | GUI‐based camera control via Open3D                            |
 | **sbs180\_workflow.json**              | Generate stereo (side-by-side) wide-angle/fisheye/equirectangular stereo pairs from a high-res input |
+| **video_camera.json**                  | Camera trajectory movement workflow using `wan-vace` for video inpainting. |
 
 ---
 
@@ -208,11 +218,13 @@ Take a wide-angle (fisheye or equirectangular) high-resolution (e.g., 4096×4096
 
 Interactive Open3D-based GUI for walking and setting camera trajectory inside pointcloud.
 
-### 11. `wan-vace_ref_to_video.json`
+### 11. `video_camera.json`
 
-Integrate the [wan2.1-vace] video generation model to inpaint empty or newly revealed regions during camera movement or view synthesis. This workflow demonstrates how to use the camera-comfyUI nodes to generate camera trajectories and masks, then fill missing areas with the video inpainting model for smooth, high-quality results.
+This workflow demonstrates camera trajectory movement using the `wan-vace` video inpainting model. It generates smooth camera movements along a trajectory while filling missing regions with high-quality inpainting.
 
-<img src="demo_images/wan-vace-camera.gif" alt="wan2.1-vace Camera Inpainting Demo" width="80%" />
+<div style="display:flex; gap:10px;">
+  <img src="demo_images/camera_movement.gif" alt="Camera Movement Demo" width="80%" />
+</div>
 
 ---
 

__init__.py

@@ -4,6 +4,7 @@ from .metric_depth_nodes import NODE_CLASS_MAPPINGS as NCM3
 from .flux_fisheye_filling_nodes import NODE_CLASS_MAPPINGS as NCM4
 from .complex_nodes import NODE_CLASS_MAPPINGS as NCM5
 from .video_nodes import NODE_CLASS_MAPPINGS as NCM6
-NODE_CLASS_MAPPINGS = {**NCM1, **NCM2, **NCM3, **NCM4, **NCM5, **NCM6}
+from .GS_nodes import NODE_CLASS_MAPPINGS as NCM7
+NODE_CLASS_MAPPINGS = {**NCM1, **NCM2, **NCM3, **NCM4, **NCM5, **NCM6, **NCM7}
 
-__all__ = ["NODE_CLASS_MAPPINGS"]
+__all__ = ["NODE_CLASS_MAPPINGS"]

install.sh

@@ -5,76 +5,104 @@ set -euo pipefail
 # Functions
 # ----------------------------------------
 install_pytorch() {
-  echo "Installing PyTorch, TorchVision, TorchAudio..."
+  echo "==> Installing PyTorch, TorchVision, TorchAudio, bitsandbytes, accelerate…"
   pip3 install -U torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128
-  pip3 install bitsandbytes
-  pip3 install accelerate
+  pip3 install -U bitsandbytes accelerate
 }
 
 install_system_deps() {
-  echo "Updating apt and installing system dependencies..."
+  echo "==> Updating apt and installing system packages…"
   sudo apt-get update
-  sudo apt-get install -y build-essential ffmpeg libsm6 libxext6
+  sudo apt-get install -y build-essential ffmpeg libsm6 libxext6 python3.10-dev
 }
 
 clone_and_install_comfyui() {
-  echo "Cloning ComfyUI..."
+  echo "==> Cloning ComfyUI…"
   git clone https://github.com/comfyanonymous/ComfyUI.git
-  echo "Installing ComfyUI requirements..."
+  echo "==> Installing ComfyUI Python requirements…"
   pip3 install -r ComfyUI/requirements.txt
 }
 
 install_camera_node() {
-  echo "Cloning camera‑ComfyUI..."
+  echo "==> Installing camera‑ComfyUI node…"
+  mkdir -p ComfyUI/custom_nodes
   git clone https://github.com/Alexankharin/camera-comfyUI.git \
     ComfyUI/custom_nodes/camera-comfyUI
-  echo "Installing camera‑ComfyUI requirements..."
   pip3 install -r ComfyUI/custom_nodes/camera-comfyUI/requirements.txt
 }
 
 install_image_filters() {
-  echo "Cloning Image‑Filters node..."
+  echo "==> Installing Image‑Filters node…"
+  mkdir -p ComfyUI/custom_nodes
   git clone https://github.com/spacepxl/ComfyUI-Image-Filters.git \
     ComfyUI/custom_nodes/ComfyUI-Image-Filters
-  echo "Installing Image‑Filters requirements..."
   pip3 install -r ComfyUI/custom_nodes/ComfyUI-Image-Filters/requirements.txt
 }
 
 clone_flux_inpainting() {
-  echo "Cloning ComfyUI‑Flux‑Inpainting..."
+  echo "==> Installing Flux‑Inpainting node…"
+  mkdir -p ComfyUI/custom_nodes
   git clone https://github.com/rubi-du/ComfyUI-Flux-Inpainting.git \
-    ComfyUI/custom_nodes/Flux-Inpainting
-  echo "Renaming Flux‑Inpainting folder..."
-  mv ComfyUI/custom_nodes/Flux-Inpainting \
-     ComfyUI/custom_nodes/inpainting_flux
+    ComfyUI/custom_nodes/inpainting_flux
+}
+
+install_metric_video_depth_anything() {
+  echo "==> Installing Metric Video Depth Anything…"
+  # Clone into ComfyUI root, not custom_nodes
+  git clone https://github.com/DepthAnything/Video-Depth-Anything.git \
+    ComfyUI/Video-Depth-Anything
+
+  echo "    • Installing easydict…"
+  pip3 install -U easydict
+
+  echo "    • Copying util.py…"
+  mkdir -p ComfyUI/utils
+  cp ComfyUI/Video-Depth-Anything/metric_depth/utils/util.py \
+     ComfyUI/utils/util.py
+
+  echo "    • Downloading Metric Video Depth checkpoint…"
+  mkdir -p ComfyUI/models/checkpoints
+  wget -q -O ComfyUI/models/checkpoints/metric_video_depth_anything_vitl.pth \
+    "https://huggingface.co/depth-anything/Metric-Video-Depth-Anything-Large/resolve/main/metric_video_depth_anything_vitl.pth"
+}
+
+
+install_comfyui_manager() {
+  echo "==> Installing ComfyUI-Manager extension…"
+  mkdir -p ComfyUI/custom_nodes
+  git clone https://github.com/Comfy-Org/ComfyUI-Manager.git \
+    ComfyUI/custom_nodes/ComfyUI-Manager
+  pip3 install -r ComfyUI/custom_nodes/ComfyUI-Manager/requirements.txt
 }
 
 install_hf_hub() {
-  echo "Installing huggingface_hub..."
-  pip3 install huggingface_hub
+  echo "==> Installing huggingface_hub…"
+  pip3 install -U huggingface_hub
 }
 
 download_vae_models() {
-  echo "Downloading WAN‑VACE models via wget..."
-  wget -O ComfyUI/models/vae/wan_2.1_vae.safetensors \
+  echo "==> Downloading WAN‑VACE models…"
+  mkdir -p ComfyUI/models/vae
+  wget -q -O ComfyUI/models/vae/wan_2.1_vae.safetensors \
     "https://huggingface.co/Comfy-Org/Wan_2.1_ComfyUI_repackaged/resolve/main/split_files/vae/wan_2.1_vae.safetensors?download=true"
 
-  wget -O ComfyUI/models/text_encoders/umt5_xxl_fp16.safetensors \
+  mkdir -p ComfyUI/models/text_encoders
+  wget -q -O ComfyUI/models/text_encoders/umt5_xxl_fp16.safetensors \
     "https://huggingface.co/Comfy-Org/Wan_2.1_ComfyUI_repackaged/resolve/main/split_files/text_encoders/umt5_xxl_fp16.safetensors?download=true"
 
-  wget -O ComfyUI/models/diffusion_models/wan2.1_vace_14B_fp16.safetensors \
+  mkdir -p ComfyUI/models/diffusion_models
+  wget -q -O ComfyUI/models/diffusion_models/wan2.1_vace_14B_fp16.safetensors \
     "https://huggingface.co/Comfy-Org/Wan_2.1_ComfyUI_repackaged/resolve/main/split_files/diffusion_models/wan2.1_vace_14B_fp16.safetensors"
 }
 
 login_hf_hub() {
-  echo "Logging in to Hugging Face Hub..."
+  echo "==> Hugging Face login…"
   huggingface-cli login
 }
 
 # ----------------------------------------
-# Main CLI
+# Main
 # ----------------------------------------
-# default to "install" if no arg given
 MODE="${1:-install}"
 
 case "$MODE" in
@@ -84,6 +112,7 @@ case "$MODE" in
     clone_and_install_comfyui
     install_camera_node
     install_image_filters
+    install_comfyui_manager
     install_hf_hub
     ;;
 
@@ -95,6 +124,10 @@ case "$MODE" in
     download_vae_models
     ;;
 
+  depth)
+    install_metric_video_depth_anything
+    ;;
+
   install)
     install_pytorch
     install_system_deps
@@ -102,7 +135,9 @@ case "$MODE" in
     install_camera_node
     clone_flux_inpainting
     install_image_filters
+    install_comfyui_manager
     install_hf_hub
+    install_metric_video_depth_anything
     ;;
 
   all)
@@ -112,14 +147,16 @@ case "$MODE" in
     install_camera_node
     clone_flux_inpainting
     install_image_filters
+    install_comfyui_manager
     install_hf_hub
     download_vae_models
+    install_metric_video_depth_anything
     login_hf_hub
-    echo "All done! 🎉"
+    echo "✅ All done!"
     ;;
 
   *)
-    echo "Usage: $0 {install|modules|flux|vae|all}"
+    echo "Usage: $0 {install|modules|flux|vae|depth|all}"
     exit 1
     ;;
 esac

pointcloud_nodes.py

@@ -325,141 +325,165 @@ class ProjectPointCloud:
 
     def project_pointcloud(
         self,
-        pointcloud:    torch.Tensor,
+        pointcloud: torch.Tensor,
         output_projection: str,
         output_horizontal_fov: float,
-        output_width:  int,
+        output_width: int,
         output_height: int,
-        point_size:    int = 1,
+        point_size: int = 1,
         return_inverse_depth: bool = False,
     ) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
+        """
+        Projects an (N×6) XYZRGB point cloud into an image,
+        fills occlusion holes robustly, and returns:
+        • img:       [1,H,W,3] RGB image
+        • mask:      [H,W]   foreground mask
+        • depth:     [1,H,W,1] depth (or inverse depth)
+        """
         device = pointcloud.device
-        coords = pointcloud[:, :3]
-        colors = pointcloud[:, 3:].float()
+        xyz, rgb_raw = pointcloud[:, :3], pointcloud[:, 3:6].float()
 
-        # 1) Filter points in front of the camera
-        mask_front = coords[:, 2] > 0
-        coords = coords[mask_front]
-        colors = colors[mask_front]
+        # 1) Keep only points in front of camera
+        in_front = xyz[:, 2] > 0
+        xyz, rgb_raw = xyz[in_front], rgb_raw[in_front]
 
         # 2) Project to normalized UV + depth
-        X, Y, Z = coords.unbind(1)
+        X, Y, Z = xyz.unbind(1)
         if output_projection == "PINHOLE":
-            u, v, depth = XYZ_to_pinhole(X, Y, Z, output_horizontal_fov)
+            u, v, d = XYZ_to_pinhole(X, Y, Z, output_horizontal_fov)
         elif output_projection == "FISHEYE":
-            u, v, depth = XYZ_to_fisheye(X, Y, Z, output_horizontal_fov)
+            u, v, d = XYZ_to_fisheye(X, Y, Z, output_horizontal_fov)
         else:
-            u, v, depth = XYZ_to_equirect(X, Y, Z, output_horizontal_fov)
+            u, v, d = XYZ_to_equirect(X, Y, Z, output_horizontal_fov)
 
         # 3) Rasterize to pixel indices
-        px = (u * (output_width - 1) / 2) + (output_width - 1) / 2
-        py = (v * (output_height - 1) / 2) + (output_height - 1) / 2
-        ix = px.round().clamp(0, output_width - 1).long()
-        iy = py.round().clamp(0, output_height - 1).long()
-        pix = iy * output_width + ix
-        M   = output_width * output_height
+        W, H = output_width, output_height
+        ix = ((u * 0.5 + 0.5) * (W - 1)).round().clamp(0, W - 1).long()
+        iy = ((v * 0.5 + 0.5) * (H - 1)).round().clamp(0, H - 1).long()
+        pix = iy * W + ix
+        valid = (pix >= 0) & (pix < W * H)
+        pix, d, rgb_raw = pix[valid], d[valid], rgb_raw[valid]
 
-        # —— NEW: drop any invalid / NaN→int_min projections ——
-        valid = (pix >= 0) & (pix < M)
-        depth  = depth[valid]
-        colors = colors[valid]
-        pix    = pix[valid]
-        order  = torch.arange(depth.size(0), device=device)
-        # rebuild your "order" to match
+        M = W * H
+        # 3a) Front‐layer (nearest) depth
+        z1 = torch.full((M,), float('inf'), device=device)
+        z1.scatter_reduce_(0, pix, d, reduce='amin', include_self=True)
 
-        # 4) Allocate or reuse buffers
-        if not hasattr(self, '_z_front') or self._z_front.numel() != M:
-            self._z_front = torch.empty((M,), device=device)
-            self._z_back  = torch.empty((M,), device=device)
-            self._idx     = torch.full((M,), -1, dtype=torch.long, device=device)
-            self._flat    = torch.zeros((M, 4), device=device)
-        z_front = self._z_front
-        z_back  = self._z_back
-        idxbuf  = self._idx
-        flat    = self._flat
+        # 3b) Second‐layer (background) depth
+        farther = d > z1[pix]
+        pix2, d2 = pix[farther], d[farther]
+        z2 = torch.full((M,), float('inf'), device=device)
+        z2.scatter_reduce_(0, pix2, d2, reduce='amin', include_self=True)
 
-        # 5) Front z-buffer pass (nearest)
-        z_front.fill_(float('inf'))
-        z_front.scatter_reduce_(0, pix, depth, reduce='amin', include_self=True)
-        sel_front = depth == z_front[pix]
-        order = torch.arange(depth.size(0), device=device)
-        order_m = torch.where(sel_front, order, depth.size(0))
-        idxbuf.fill_(depth.size(0))
-        idxbuf.scatter_reduce_(0, pix, order_m, reduce='amin', include_self=True)
-        win_front = order == idxbuf[pix]
+        # 3c) Foreground colour (from z1)
+        keep = d == z1[pix]
+        rgb = torch.zeros((M, 3), device=device)
+        rgb[pix[keep]] = rgb_raw[keep].clamp(0, 255)
 
-        flat.fill_(0)
-        flat[pix[win_front]] = colors[win_front]
-        img4 = flat.view(output_height, output_width, 4)
-        rgb   = img4[..., :3].clamp(0, 255)
-        alpha = (img4[..., 3] > 0).float()
-        mask_init= (img4[..., 3] > 0)  # initial mask
-        rgb  *= alpha.unsqueeze(-1)
-        depth_img = z_front.view(output_height, output_width)
-        rgb_HR = rgb
+        # reshape to image
+        rgb    = rgb.view(H, W, 3)
+        z1      = z1.view(H, W)
+        z2      = z2.view(H, W)
+        fg_mask = z1 < float('inf')      # has front hit
+        occl    = (z2 < float('inf'))    # has any back hit
+        rear_only = occl & ~fg_mask      # true holes
 
-        # 6) Back z-buffer pass (farthest) for hole-filling
+        # ── Iterative ring‐based in‐painting of rear‐only pixels ─────────────────────
+        ker3 = torch.ones((1,1,3,3), device=device)
+        ker3c = ker3.repeat(3,1,1,1)
+        for _ in range(max(W, H)):
+            # find rear_only pixels adjacent to current FG
+            neigh = (
+                F.max_pool2d(fg_mask.float()[None,None], 3, 1, 1).bool()[0,0]
+                & ~fg_mask
+            )
+            to_fill = rear_only & neigh
+            if not to_fill.any():
+                break
+
+            # average depth + colour from current FG frontier
+            d_t = z1.masked_fill(~fg_mask, 0)[None,None]
+            c_t = rgb.permute(2,0,1)[None]      # [1,3,H,W]
+            m_t = fg_mask.float()[None,None]
+
+            sum_d = F.conv2d(d_t * m_t, ker3, padding=1)
+            cnt_d = F.conv2d(m_t, ker3, padding=1).clamp(min=1)
+            sum_c = F.conv2d(c_t * m_t, ker3c, padding=1, groups=3)
+            cnt_c = cnt_d.repeat(1,3,1,1)
+
+            avg_d = (sum_d / cnt_d).squeeze()
+            avg_c = (sum_c / cnt_c).squeeze().permute(1,2,0)
+
+            z1[to_fill]    = avg_d[to_fill]
+            rgb[to_fill]   = avg_c[to_fill]
+            fg_mask[to_fill] = True
+            rear_only[to_fill] = False
+
+        # ── Depth‐aware generic hole closure ─────────────────────────────────────────
+        # close_rad: radius of hole to close; depth_eps: depth jump tolerance
+        close_rad = max(1, point_size // 2)
+        depth_eps = 0.015
+        pad = close_rad
+        k = 2 * close_rad + 1
+        ker = torch.ones((1,1,k,k), device=device)
+        kerc = ker.repeat(3,1,1,1)
+        front_t = fg_mask.float()[None,None]
+
+        # binary closing: dilate then erode
+        D = F.max_pool2d(front_t, k, 1, pad)
+        E = 1 - F.max_pool2d(1 - D, k, 1, pad)
+        small_hole = E[0,0].bool() & ~fg_mask
+        if small_hole.any():
+            # compute local mean depth of FG
+            z_t = z1.masked_fill(~fg_mask, 0)[None,None]
+            cnt = F.conv2d(front_t, ker, padding=pad).clamp(min=1)
+            z_avg = (F.conv2d(z_t, ker, padding=pad) / cnt)[0,0]
+
+            # depth‐range test
+            z_near = F.max_pool2d(z1[None,None], 3,1,1)[0,0]
+            z_far  = -F.max_pool2d(-z1[None,None],3,1,1)[0,0]
+            flat   = (z_far - z_near) / z_avg.clamp(min=1e-6) < depth_eps
+
+            final = small_hole & flat
+            if final.any():
+                sum_d = F.conv2d(z_t, ker, padding=pad)
+                sum_c = F.conv2d(rgb.permute(2,0,1)[None] * front_t, kerc,
+                                padding=pad, groups=3)
+                avg_d = (sum_d / cnt)[0,0]
+                avg_c = (sum_c / cnt.repeat(1,3,1,1))[0].permute(1,2,0)
+
+                z1[final]      = avg_d[final]
+                rgb[final]     = avg_c[final]
+                fg_mask[final] = True
+
+        # ── Optional morphological blur for larger point_size ───────────────────────
         if point_size > 1:
-            z_back.fill_(-float('inf'))
-            z_back.scatter_reduce_(0, pix, depth, reduce='amax', include_self=True)
-            sel_back = depth == z_back[pix]
-            order_m = torch.where(sel_back, order, -1)
-            idxbuf.fill_(-1)
-            idxbuf.scatter_reduce_(0, pix, order_m, reduce='amax', include_self=True)
-            win_back = idxbuf[pix] >= 0
-            flat.fill_(0)
-            flat[pix[win_back]] = colors[win_back]
-            back4 = flat.view(output_height, output_width, 4)
-            rgb_back   = back4[..., :3].clamp(0,255)
-            alpha_back = (back4[..., 3] > 0).float()
+            r = point_size // 2
+            k = 2 * r + 1
+            pad = r
+            ker = torch.ones((1,1,k,k), device=device)
+            kerc = ker.repeat(3,1,1,1)
+            d_t = z1[None,None]
+            c_t = rgb.permute(2,0,1)[None]
+            m_t = fg_mask.float()[None,None]
 
-            # fill holes where front missed
-            hole = (alpha == 0) & (alpha_back > 0)
-            rgb[hole]       = rgb_back[hole]
-            alpha[hole]     = 1.0
-            depth_img[hole] = z_back.view(output_height, output_width)[hole]
+            z1 = (F.conv2d(d_t * m_t, ker, padding=pad) /
+                F.conv2d(m_t, ker, padding=pad).clamp(min=1)).squeeze()
+            rgb = (F.conv2d(c_t * m_t, kerc, padding=pad, groups=3) /
+                F.conv2d(m_t, ker, padding=pad).repeat(1,3,1,1).clamp(min=1)
+                ).squeeze().permute(1,2,0)
 
-            # 7) Median-filter _only_ in hole regions
-            if hole.any():
-                # prepare for kornia median_blur: [B,C,H,W]
-                rgb_t   = rgb.permute(2,0,1).unsqueeze(0)  # [1,3,H,W]
-                # apply median filter
-                rgb_med = median_blur(rgb_t, (point_size, point_size))
-                # back to HWC
-                rgb_med = rgb_med.squeeze(0).permute(1,2,0)
-                # merge only at hole locations
-                rgb[hole] = rgb_med[hole]
-                # alpha already set to 1.0 for holes
-            # 8 apply median blur to mask if point_size > 1 and to initial image  
-            mask_t = alpha.unsqueeze(0).unsqueeze(0)  # [1,1,H,W]
-            pad    = point_size // 2
-            ksize  = (point_size, point_size)
-
-            # b) grow (dilate) mask by max‑pool
-            mask_grow = F.max_pool2d(mask_t, kernel_size=ksize, stride=1, padding=pad)
-
-            # c) shrink (erode) by inverting, max‑pool, then inverting back
-            mask_shrink = 1.0 - F.max_pool2d(1.0 - mask_grow, kernel_size=ksize, stride=1, padding=pad)
-
-            # d) back to [H,W] and use as our new alpha
-            alpha = mask_shrink.squeeze(0).squeeze(0)
-            print(1)
-            # e) median‑filter the *whole* RGB image
-            #    prep for kornia: [B,C,H,W]
-            rgb_t_full   = rgb.permute(2,0,1).unsqueeze(0)       # [1,3,H,W]
-            rgb_med_full = median_blur(rgb_t_full, ksize)        # [1,3,H,W]
-            rgb_med_full = rgb_med_full.squeeze(0).permute(1,2,0)  # [H,W,3]
-
-            # g) refill *only* the original holes with the median result
-            rgb[~mask_init] = rgb_med_full[~mask_init]
-        # 9) Pack and return with original script shapes
-        img       = rgb.unsqueeze(0)  # [1,H,W,3]
-        mask_out  = alpha             # [H,W]
-        depth4    = depth_img.unsqueeze(0).unsqueeze(-1)  # [1,H,W,1]
+        # 10) Pack outputs
+        img   = rgb.unsqueeze(0)                # [1,H,W,3]
+        mask  = fg_mask.float()                 # [H,W]
+        depth = z1.unsqueeze(0).unsqueeze(-1)   # [1,H,W,1]
         if return_inverse_depth:
-            depth4 = 1.0 / depth4.clamp(min=1e-6)
-        depth4 = depth4 * mask_out.unsqueeze(0).unsqueeze(-1)
-        return img, mask_out, depth4
+            depth = 1.0 / depth.clamp(min=1e-6)
+        depth *= mask.unsqueeze(0).unsqueeze(-1)
+        return img, mask, depth
+
+
+
 
 class PointCloudUnion:
     """
@@ -813,7 +837,7 @@ class CameraTrajectoryNode:
             "pointcloud":    ("TENSOR",),
         },
         "optional": {
-            "initial_matrix": ("MAT_4X4"),
+            "initial_matrix": ("MAT_4X4",),
             }
         }
 

reprojection_nodes.py

@@ -42,7 +42,14 @@ def map_grid(
         output_horizontal_fov = torch.tensor(output_horizontal_fov, device=grid_torch.device).float()
 
         # Calculate vertical field of view for input and output projections
-        output_vertical_fov = output_horizontal_fov  # Assuming square aspect ratio
+        # For equirectangular, use 2:1 aspect ratio (vertical FOV = horizontal FOV / 2)
+        if output_projection == "EQUIRECTANGULAR":
+            output_vertical_fov = output_horizontal_fov / 2.0
+        else:
+            output_vertical_fov = output_horizontal_fov  # Assuming square aspect ratio for other projections
+
+        # Calculate input vertical FOV based on output grid aspect ratio
+        # This allows the input's vertical range to adapt to the output dimensions
         input_vertical_fov = input_horizontal_fov * (grid_torch.shape[0] / grid_torch.shape[1])
 
         # Normalize the grid for vertical FOV adjustment
@@ -222,8 +229,8 @@ class ReprojectImage:
             )
         
         grid_y, grid_x = torch.meshgrid(
-            torch.linspace(-1, 1, output_width, device=image_tensor.device),
             torch.linspace(-1, 1, output_height, device=image_tensor.device),
+            torch.linspace(-1, 1, output_width, device=image_tensor.device),
             indexing="ij"
         )
         grid_init = torch.stack((grid_x, grid_y), dim=-1)

video_nodes.py

@@ -11,17 +11,26 @@ from .pointcloud_nodes import DepthToPointCloud, TransformPointCloud, ProjectPoi
 import folder_paths
 
 # Ensure video_depth_anything is on path
-video_depth_path = os.path.join("/root/Video-Depth-Anything", "metric_depth")
+_here = os.path.dirname(os.path.abspath(__file__))
+
+# climb up 3 levels: camera-comfyUI → custom_nodes → ComfyUI
+COMFYUI_ROOT = os.path.abspath(os.path.join(_here, os.pardir, os.pardir))
+
+# point at metric_depth inside the Video-Depth-Anything clone at the ComfyUI root
+video_depth_path = os.path.join(COMFYUI_ROOT, "Video-Depth-Anything", "metric_depth")
+
+# insert at front so it always wins
 if video_depth_path not in sys.path:
-    sys.path.append(video_depth_path)
+    sys.path.insert(0, video_depth_path)
+NO_VIDEO_DEPTH_ANYTHING= False
 try:
     from video_depth_anything.video_depth import VideoDepthAnything
-    print("video_depth_anything module loaded successfully.")
-except ImportError:
-    VideoDepthAnything = None
-    print("Warning: video_depth_anything module not found. Ensure it is installed correctly.")
-    print("error: ", sys.exc_info()[1])
-
+    print("✅ video_depth_anything module loaded successfully.")
+except ImportError as e:
+    NO_VIDEO_DEPTH_ANYTHING = True
+    print(
+        f"❌ Could not load video_depth_anything from {video_depth_path!r}: {e}"
+    )
 
 class VideoCameraMotionSequence:
     """
@@ -40,6 +49,8 @@ class VideoCameraMotionSequence:
                 "depth_seq": ("TENSOR", {"shape_hint": [None, None, None]}),
                 # Camera trajectory waypoints: Tensor [K, 4, 4]
                 "trajectory": ("TENSOR", {"shape_hint": [None, 4, 4]}),
+                # Optional mask sequence: Tensor [T, H, W] or [T, H, W, 1]
+                "mask_seq": ("MASK", {"shape_hint": [None, None, None], "optional": True}),
                 # Input projection parameters
                 "input_projection": (Projection.PROJECTIONS, {}),
                 "input_horizontal_fov": ("FLOAT", {"default": 90.0}),
@@ -78,6 +89,7 @@ class VideoCameraMotionSequence:
         point_size: int,
         voxel_size: float,
         min_points_per_voxel: int,
+        mask_seq: torch.Tensor = None,
     ) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
         # frames: [T, H, W, 3]
         # depth_seq: [T, H, W] or [T, H, W, 1]
@@ -101,9 +113,21 @@ class VideoCameraMotionSequence:
         out_depths = []
 
         # Add tqdm progress bar for the sequence
-        for frame, depth, pose in tqdm(zip(frames, depth_seq, interp_traj), total=T, desc="Processing video frames"):
+        # If mask_seq is a single mask [H, W] or [H, W, 1], repeat it for all frames
+        if mask_seq is not None:
+            if mask_seq.dim() == 2 or (mask_seq.dim() == 3 and mask_seq.shape[0] == 1):
+                mask_seq = mask_seq.unsqueeze(0) if mask_seq.dim() == 2 else mask_seq
+                mask_seq = mask_seq.repeat(T, 1, 1, 1) if mask_seq.dim() == 4 else mask_seq.repeat(T, 1, 1)
+
+        for i, (frame, depth, pose) in enumerate(tqdm(zip(frames, depth_seq, interp_traj), total=T, desc="Processing video frames")):
             if depth.dim() == 3 and depth.shape[-1] == 1:
                 depth = depth.squeeze(-1)
+                # Use mask if provided
+                mask = None
+            if mask_seq is not None:
+                mask = mask_seq[i]
+            if mask.dim() == 3 and mask.shape[-1] == 1:
+                mask = mask.squeeze(-1)
             # to pointcloud
             pc, = DepthToPointCloud().depth_to_pointcloud(
                 image=frame.permute(2, 0, 1),
@@ -112,7 +136,7 @@ class VideoCameraMotionSequence:
                 depth_scale=depth_scale,
                 invert_depth=invert_depth,
                 depthmap=depth,
-                mask=None,
+                mask=mask,
             )
             # optional cleaning
             if min_points_per_voxel > 1:
@@ -258,8 +282,11 @@ class VideoMetricDepthEstimate:
         return (torch.from_numpy(metric_depths), float(fps))
 
 # Register nodes
-NODE_CLASS_MAPPINGS = {
-    "VideoCameraMotionSequence": VideoCameraMotionSequence,
-    "VideoMetricDepthEstimate": VideoMetricDepthEstimate,
-    "DepthFramesToVideo": DepthFramesToVideo,
-}
+if NO_VIDEO_DEPTH_ANYTHING:
+    NODE_CLASS_MAPPINGS = {}
+else:
+    NODE_CLASS_MAPPINGS = {
+        "VideoCameraMotionSequence": VideoCameraMotionSequence,
+        "VideoMetricDepthEstimate": VideoMetricDepthEstimate,
+        "DepthFramesToVideo": DepthFramesToVideo,
+    }