Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
d49f3f03ce |
@@ -1,19 +0,0 @@
|
||||
.PHONY: install install_all install_modules download_flux download_vae
|
||||
|
||||
# install everything except WAN‑VACE downloads
|
||||
install:
|
||||
./install.sh install
|
||||
|
||||
# install everything + WAN‑VACE + HF login
|
||||
install_all:
|
||||
./install.sh all
|
||||
|
||||
# lower‑level helpers
|
||||
install_modules:
|
||||
./install.sh modules
|
||||
|
||||
download_flux:
|
||||
./install.sh flux
|
||||
|
||||
download_vae:
|
||||
./install.sh vae
|
||||
@@ -1,5 +1,5 @@
|
||||
# camera-comfyUI
|
||||
[](https://deepwiki.com/Alexankharin/camera-comfyUI)
|
||||
|
||||

|
||||
|
||||
> Custom ComfyUI nodes for advanced reprojections, point cloud processing, and camera-driven workflows.
|
||||
@@ -80,25 +80,18 @@ A collection of ComfyUI custom nodes to handle diverse camera projections (pinho
|
||||
* ### Reprojection Nodes
|
||||
|
||||
* `ReprojectImage`, `ReprojectDepth`, `OutpaintAnyProjection`
|
||||
|
||||
* ### Matrix Nodes
|
||||
|
||||
* `TransformToMatrix`, `TransformToMatrixManual`
|
||||
|
||||
* ### Depth Nodes
|
||||
|
||||
* `DepthEstimatorNode`, `DepthToImageNode`, `ZDepthToRayDepthNode`
|
||||
* `CombineDepthsNode`, `DepthRenormalizer`, `FisheyeDepthEstimator`
|
||||
* `CombineDepthsNode`, `DepthRenormalizer`
|
||||
|
||||
* ### Point Cloud Nodes
|
||||
|
||||
* `DepthToPointCloud`, `TransformPointCloud`, `ProjectPointCloud`, `PointCloudUnion`
|
||||
* `PointCloudCleaner`, `LoadPointCloud`, `SavePointCloud`, `ProjectAndClean`
|
||||
|
||||
* ### Trajectory Nodes
|
||||
|
||||
* `DepthToPointCloud`, `TransformPointCloud`, `ProjectPointCloud`
|
||||
* `PointCloudUnion`, `PointCloudCleaner`, `LoadPointCloud`, `SavePointCloud`
|
||||
* `CameraMotionNode`, `CameraInterpolationNode`, `CameraTrajectoryNode`
|
||||
* `SaveTrajectory`, `LoadTrajectory`, `PointcloudTrajectoryEnricher`
|
||||
|
||||
---
|
||||
|
||||
@@ -117,7 +110,7 @@ A collection of ComfyUI custom nodes to handle diverse camera projections (pinho
|
||||
| `ZDepthToRayDepthNode` | Converts Z-depth (output of metric-depth-anything) to ray depth to compensate lens curvature. |
|
||||
| `TransformPointCloud` | Applies 4×4 rotation matrix to point cloud |
|
||||
| `ProjectPointCloud` | Z-buffer–based projection of point cloud into image + mask. |
|
||||
| `CameraMotionNode` | Generates image and mask sequences along a camera trajectory with optional mask dilation/inversion. |
|
||||
| `CameraMotionNode` | Generates image sequences by moving camera along a trajectory. |
|
||||
| `CameraInterpolationNode` | Builds a trajectory tensor from two poses. |
|
||||
| `CameraTrajectoryNode` | Interactive Open3D GUI for recording camera waypoints. |
|
||||
| `PointCloudCleaner` | Removes isolated points via voxel filtering. |
|
||||
@@ -139,7 +132,6 @@ A set of JSON workflows illustrating typical use cases. Each workflow lives in `
|
||||
| **Pointcloud.json** | Metric‐depth‐anything v2 → point cloud → camera view synthesis |
|
||||
| **pointcloud\_inpaint.json** | Inpaint + backproject to 3D for dynamic camera motion videos |
|
||||
| **Pointcloud\_walker.json** | GUI‐based camera control via Open3D |
|
||||
| **sbs180\_workflow.json** | Generate stereo (side-by-side) wide-angle/fisheye/equirectangular stereo pairs from a high-res input |
|
||||
|
||||
---
|
||||
|
||||
@@ -198,22 +190,10 @@ Inpaint image with shifted camera and backproject for dynamic camera‐driven vi
|
||||
<img src="demo_images/Fisheye_camera_pointcloud_moved_outpainted.png" alt="PointCloud Inpaint" width="40%" />
|
||||
<img src="demo_images/Camera_interpolation_pointcloud.gif" alt="PointCloud Inpaint Video" width="40%" />
|
||||
|
||||
### 9. `sbs180_workflow.json`
|
||||
|
||||
Take a wide-angle (fisheye or equirectangular) high-resolution (e.g., 4096×4096) image and generate a stereo pair by moving the camera horizontally. The output is a wide-angle stereo pair (side-by-side), simulating a fisheye or equirectangular stereo camera.
|
||||
|
||||
<img src="demo_images/equirect_stereo.gif" alt="Equirectangular Stereo Demo" width="80%" />
|
||||
|
||||
### 10. `Pointcloud_walker.json`
|
||||
|
||||
Interactive Open3D-based GUI for walking and setting camera trajectory inside pointcloud.
|
||||
|
||||
### 11. `wan-vace_ref_to_video.json`
|
||||
|
||||
Integrate the [wan2.1-vace] video generation model to inpaint empty or newly revealed regions during camera movement or view synthesis. This workflow demonstrates how to use the camera-comfyUI nodes to generate camera trajectories and masks, then fill missing areas with the video inpainting model for smooth, high-quality results.
|
||||
|
||||
<img src="demo_images/wan-vace-camera.gif" alt="wan2.1-vace Camera Inpainting Demo" width="80%" />
|
||||
|
||||
---
|
||||
|
||||
## Contributing
|
||||
@@ -222,7 +202,7 @@ Contributions welcome! Please open issues or PRs to add features, improve docs,
|
||||
|
||||
## TODO List
|
||||
|
||||
* [x] Add processing to pointcloud or depthmap to remove outlier and lonely points at depth borders.
|
||||
* [ ] Add processing to pointcloud or depthmap to remove outlier and lonely points at depth borders.
|
||||
* [x] Use built-in comfyUI mask type an image.
|
||||
* [x] Unite nodes into groups to simplify workflows.
|
||||
* [ ] Create a single workflow for view synthesis.
|
||||
@@ -230,4 +210,3 @@ Contributions welcome! Please open issues or PRs to add features, improve docs,
|
||||
* [x] Add more examples and documentation for each node.
|
||||
* [x] Add pointcloud union
|
||||
* [ ] Fix imports for renamed folders (e.g., inpainting_flux)
|
||||
* [x] Integrate camera movement pipeline with video models (e.g., wan2.1) for smooth, high-quality inpainting along camera trajectories.
|
||||
|
||||
+376
-168
@@ -64,7 +64,7 @@ class FisheyeDepthEstimator:
|
||||
RETURN_TYPES = ("TENSOR","MASK")
|
||||
RETURN_NAMES = ("depthmap","mask")
|
||||
FUNCTION = "estimate_fisheye_depth"
|
||||
CATEGORY = "Camera/Depth"
|
||||
CATEGORY = "Camera/depth"
|
||||
|
||||
def estimate_fisheye_depth(
|
||||
self,
|
||||
@@ -95,76 +95,46 @@ class FisheyeDepthEstimator:
|
||||
depth_full, = de_node.estimate_depth(image, model_name, depth_scale)
|
||||
mask_full = (depth_full > 0).float()
|
||||
|
||||
# 2) Pinhole orientations (5 views)
|
||||
rotations = [
|
||||
(0, 0, 0), # front
|
||||
(0, 45, 0), # right
|
||||
(0, -45, 0), # left
|
||||
(45, 0, 0), # up
|
||||
(-45, 0, 0), # down
|
||||
]
|
||||
# 2) Generate Pinhole Views
|
||||
fisheye_depths, fisheye_masks = self._generate_pinhole_views(
|
||||
image,
|
||||
de_node, z2r_node, ri_node, rd_node,
|
||||
fisheye_fov, pinhole_fov,
|
||||
pin_w, pin_h, fish_w, fish_h,
|
||||
model_name, depth_scale, median_blur_kernel
|
||||
)
|
||||
|
||||
fisheye_depths = []
|
||||
fisheye_masks = []
|
||||
|
||||
# euler → matrix
|
||||
def euler_to_matrix(pitch, yaw, roll):
|
||||
p, y, r = map(math.radians, (pitch, yaw, roll))
|
||||
Rx = torch.tensor([[1,0,0],[0,math.cos(p),-math.sin(p)],[0,math.sin(p),math.cos(p)]], dtype=torch.float32)
|
||||
Ry = torch.tensor([[math.cos(y),0,math.sin(y)],[0,1,0],[-math.sin(y),0,math.cos(y)]], dtype=torch.float32)
|
||||
Rz = torch.tensor([[math.cos(r),-math.sin(r),0],[math.sin(r),math.cos(r),0],[0,0,1]], dtype=torch.float32)
|
||||
R = Rz @ Ry @ Rx
|
||||
M = torch.eye(4, dtype=torch.float32)
|
||||
M[:3, :3] = R
|
||||
return M
|
||||
|
||||
# 3) Process each orientation
|
||||
for pitch, yaw, roll in rotations:
|
||||
M = euler_to_matrix(pitch, yaw, roll)
|
||||
M_np = M.numpy()
|
||||
M_inv = torch.inverse(M).numpy()
|
||||
|
||||
# fisheye → pinhole
|
||||
img_pin, mask_pin = ri_node.reproject_image(
|
||||
image,
|
||||
input_horiszontal_fov = fisheye_fov,
|
||||
output_horiszontal_fov= pinhole_fov,
|
||||
input_projection = "FISHEYE",
|
||||
output_projection = "PINHOLE",
|
||||
output_width = pin_w,
|
||||
output_height = pin_h,
|
||||
transform_matrix = M_np,
|
||||
feathering = 0,
|
||||
)
|
||||
|
||||
# estimate pinhole depth
|
||||
depth_pin, = de_node.estimate_depth(img_pin, model_name, depth_scale, median_blur_kernel=median_blur_kernel)
|
||||
depth_pin, = z2r_node.depth_to_ray_depth(
|
||||
depth_pin,
|
||||
pinhole_fov,
|
||||
)
|
||||
# pinhole → fisheye
|
||||
fish_depth, fish_mask = rd_node.reproject_depth(
|
||||
depth_pin,
|
||||
input_horizontal_fov = pinhole_fov,
|
||||
output_horizontal_fov= fisheye_fov,
|
||||
input_projection = "PINHOLE",
|
||||
output_projection = "FISHEYE",
|
||||
output_width = fish_w,
|
||||
output_height = fish_h,
|
||||
transform_matrix = M_inv,
|
||||
)
|
||||
# squeeze mask to [B,H,W]
|
||||
fish_mask = fish_mask.squeeze(1)
|
||||
|
||||
fisheye_depths.append(fish_depth) # [B,H,W]
|
||||
fisheye_masks.append(fish_mask)
|
||||
fisheye_depths.append(depth_full) # [B,H,W]
|
||||
fisheye_masks.append(mask_full.squeeze(-1)) # [B,H,W 1]
|
||||
# merged mask
|
||||
merged_mask = torch.sum(torch.stack(fisheye_masks), dim=0) > 0.5
|
||||
# print(fisheye_depths[0].shape, fisheye_depths[-1].shape, merged_mask.shape)
|
||||
# 4) Merge in sequence
|
||||
d_acc, m_acc = self._merge_depths(
|
||||
fisheye_depths, fisheye_masks,
|
||||
ren_node, comb_node,
|
||||
mode, softmerge_radius
|
||||
)
|
||||
|
||||
# 5) Circular mask
|
||||
ys = torch.arange(fish_h, device=d_acc.device).view(1, fish_h, 1)
|
||||
xs = torch.arange(fish_w, device=d_acc.device).view(1, 1, fish_w)
|
||||
cy = (fish_h - 1) / 2.0
|
||||
cx = (fish_w - 1) / 2.0
|
||||
dist2 = (ys - cy)**2 + (xs - cx)**2
|
||||
radius2 = (min(fish_w, fish_h) / 2.0)**2
|
||||
circ_mask = (dist2 <= radius2).float()
|
||||
return d_acc, circ_mask
|
||||
|
||||
def _merge_depths(
|
||||
self,
|
||||
fisheye_depths: list,
|
||||
fisheye_masks: list,
|
||||
ren_node: DepthRenormalizer,
|
||||
comb_node: CombineDepthsNode,
|
||||
mode: str,
|
||||
softmerge_radius: int,
|
||||
) -> Tuple[torch.Tensor, torch.Tensor]:
|
||||
d_acc = fisheye_depths[0]
|
||||
m_acc = fisheye_masks[0]
|
||||
for d_new, m_new in zip(fisheye_depths[1:-1], fisheye_masks[1:-1]):
|
||||
@@ -187,20 +157,120 @@ class FisheyeDepthEstimator:
|
||||
m_acc,
|
||||
d_norm,
|
||||
m_new_last,
|
||||
mode = "SRC",
|
||||
mode = "SRC", # Use SRC for the full fisheye to preserve its details
|
||||
invert_mask = False,
|
||||
softmerge_radius = softmerge_radius
|
||||
)
|
||||
return d_acc, m_acc
|
||||
|
||||
# 5) Circular mask
|
||||
ys = torch.arange(fish_h, device=d_acc.device).view(1, fish_h, 1)
|
||||
xs = torch.arange(fish_w, device=d_acc.device).view(1, 1, fish_w)
|
||||
cy = (fish_h - 1) / 2.0
|
||||
cx = (fish_w - 1) / 2.0
|
||||
dist2 = (ys - cy)**2 + (xs - cx)**2
|
||||
radius2 = (min(fish_w, fish_h) / 2.0)**2
|
||||
circ_mask = (dist2 <= radius2).float()
|
||||
return d_acc, circ_mask
|
||||
def _generate_pinhole_views(
|
||||
self,
|
||||
image: torch.Tensor,
|
||||
de_node: DepthEstimatorNode,
|
||||
z2r_node: ZDepthToRayDepthNode,
|
||||
ri_node: ReprojectImage,
|
||||
rd_node: ReprojectDepth,
|
||||
fisheye_fov: float,
|
||||
pinhole_fov: float,
|
||||
pin_w: int,
|
||||
pin_h: int,
|
||||
fish_w: int,
|
||||
fish_h: int,
|
||||
model_name: str,
|
||||
depth_scale: float,
|
||||
median_blur_kernel: int,
|
||||
) -> Tuple[list, list]:
|
||||
rotations = [
|
||||
(0, 0, 0), # front
|
||||
(0, 45, 0), # right
|
||||
(0, -45, 0), # left
|
||||
(45, 0, 0), # up
|
||||
(-45, 0, 0), # down
|
||||
]
|
||||
|
||||
fisheye_depths = []
|
||||
fisheye_masks = []
|
||||
|
||||
for pitch, yaw, roll in rotations:
|
||||
M = self._euler_to_matrix(pitch, yaw, roll)
|
||||
M_np = M.numpy()
|
||||
M_inv = torch.inverse(M).numpy()
|
||||
|
||||
fish_depth, fish_mask = self._process_view(
|
||||
image, M_np, M_inv,
|
||||
de_node, z2r_node, ri_node, rd_node,
|
||||
fisheye_fov, pinhole_fov,
|
||||
pin_w, pin_h, fish_w, fish_h,
|
||||
model_name, depth_scale, median_blur_kernel
|
||||
)
|
||||
|
||||
fisheye_depths.append(fish_depth)
|
||||
fisheye_masks.append(fish_mask)
|
||||
|
||||
return fisheye_depths, fisheye_masks
|
||||
|
||||
def _process_view(
|
||||
self,
|
||||
image: torch.Tensor,
|
||||
M_np: np.ndarray,
|
||||
M_inv: np.ndarray,
|
||||
de_node: DepthEstimatorNode,
|
||||
z2r_node: ZDepthToRayDepthNode,
|
||||
ri_node: ReprojectImage,
|
||||
rd_node: ReprojectDepth,
|
||||
fisheye_fov: float,
|
||||
pinhole_fov: float,
|
||||
pin_w: int,
|
||||
pin_h: int,
|
||||
fish_w: int,
|
||||
fish_h: int,
|
||||
model_name: str,
|
||||
depth_scale: float,
|
||||
median_blur_kernel: int,
|
||||
) -> Tuple[torch.Tensor, torch.Tensor]:
|
||||
# fisheye → pinhole
|
||||
img_pin, mask_pin = ri_node.reproject_image(
|
||||
image,
|
||||
input_horiszontal_fov = fisheye_fov,
|
||||
output_horiszontal_fov= pinhole_fov,
|
||||
input_projection = "FISHEYE",
|
||||
output_projection = "PINHOLE",
|
||||
output_width = pin_w,
|
||||
output_height = pin_h,
|
||||
transform_matrix = M_np,
|
||||
feathering = 0,
|
||||
)
|
||||
|
||||
# estimate pinhole depth
|
||||
depth_pin, = de_node.estimate_depth(img_pin, model_name, depth_scale, median_blur_kernel=median_blur_kernel)
|
||||
depth_pin, = z2r_node.depth_to_ray_depth(
|
||||
depth_pin,
|
||||
pinhole_fov,
|
||||
)
|
||||
# pinhole → fisheye
|
||||
fish_depth, fish_mask = rd_node.reproject_depth(
|
||||
depth_pin,
|
||||
input_horizontal_fov = pinhole_fov,
|
||||
output_horizontal_fov= fisheye_fov,
|
||||
input_projection = "PINHOLE",
|
||||
output_projection = "FISHEYE",
|
||||
output_width = fish_w,
|
||||
output_height = fish_h,
|
||||
transform_matrix = M_inv,
|
||||
)
|
||||
# squeeze mask to [B,H,W]
|
||||
fish_mask = fish_mask.squeeze(1)
|
||||
return fish_depth, fish_mask
|
||||
|
||||
def _euler_to_matrix(self, pitch, yaw, roll):
|
||||
p, y, r = map(math.radians, (pitch, yaw, roll))
|
||||
Rx = torch.tensor([[1,0,0],[0,math.cos(p),-math.sin(p)],[0,math.sin(p),math.cos(p)]], dtype=torch.float32)
|
||||
Ry = torch.tensor([[math.cos(y),0,math.sin(y)],[0,1,0],[-math.sin(y),0,math.cos(y)]], dtype=torch.float32)
|
||||
Rz = torch.tensor([[math.cos(r),-math.sin(r),0],[math.sin(r),math.cos(r),0],[0,0,1]], dtype=torch.float32)
|
||||
R = Rz @ Ry @ Rx
|
||||
M = torch.eye(4, dtype=torch.float32)
|
||||
M[:3, :3] = R
|
||||
return M
|
||||
|
||||
class PointcloudTrajectoryEnricher:
|
||||
"""
|
||||
@@ -241,7 +311,7 @@ class PointcloudTrajectoryEnricher:
|
||||
RETURN_TYPES = ("TENSOR","IMAGE","TENSOR")
|
||||
RETURN_NAMES = ("enriched_pointcloud","debug_image","debug_depth")
|
||||
FUNCTION = "enrich_trajectory"
|
||||
CATEGORY = "Camera/Trajectory"
|
||||
CATEGORY = "Camera/pointcloud"
|
||||
|
||||
def enrich_trajectory(
|
||||
self,
|
||||
@@ -288,106 +358,244 @@ class PointcloudTrajectoryEnricher:
|
||||
debug_img = torch.zeros((1, height, width, 3), device=device)
|
||||
debug_depth = torch.zeros((1, height, width, 1), device=device)
|
||||
enriched_pc = pointcloud
|
||||
# Initialize debug_img and debug_depth which will be updated in the loop
|
||||
# and will hold the values from the last processed view.
|
||||
debug_img = torch.zeros((1, height, width, 3), device=device)
|
||||
debug_depth = torch.zeros((1, height, width, 1), device=device)
|
||||
|
||||
# loop over trajectory (limit or full)
|
||||
for M in tqdm(trajectory[:15], desc="Enriching trajectory"):
|
||||
M_np = M.cpu().numpy()
|
||||
M_inv = np.linalg.inv(M_np)
|
||||
enriched_pc, view_debug_img, view_debug_depth = self._process_single_view(
|
||||
M, enriched_pc, device,
|
||||
proj_node, outpaint_node, depth_node, renorm_node,
|
||||
depth2pc_node, transform_node, clean_node, zdepth_node,
|
||||
camera_type, horizontal_fov, width, height,
|
||||
patch_projection, patch_horiz_fov, patch_res,
|
||||
patch_phi, patch_theta, prompt,
|
||||
num_inference_steps, guidance_scale, mask_blur,
|
||||
voxel_size, min_points_per_voxel, model_name
|
||||
)
|
||||
debug_img = view_debug_img
|
||||
debug_depth = view_debug_depth
|
||||
return enriched_pc, debug_img, debug_depth
|
||||
|
||||
# transform and select front points
|
||||
rotated, = transform_node.transform_pointcloud(enriched_pc, M_np)
|
||||
pc_front = rotated[rotated[:, 2] > 0]
|
||||
def _process_single_view(
|
||||
self,
|
||||
M_matrix: torch.Tensor,
|
||||
current_enriched_pc: torch.Tensor,
|
||||
device: torch.device,
|
||||
proj_node: ProjectPointCloud,
|
||||
outpaint_node: OutpaintAnyProjection,
|
||||
depth_node: DepthEstimatorNode,
|
||||
renorm_node: DepthRenormalizer,
|
||||
depth2pc_node: DepthToPointCloud,
|
||||
transform_node: TransformPointCloud,
|
||||
clean_node: PointCloudCleaner,
|
||||
zdepth_node: ZDepthToRayDepthNode,
|
||||
camera_type: str,
|
||||
horizontal_fov: float,
|
||||
width: int,
|
||||
height: int,
|
||||
patch_projection: str,
|
||||
patch_horiz_fov: float,
|
||||
patch_res: int,
|
||||
patch_phi: float,
|
||||
patch_theta: float,
|
||||
prompt: str,
|
||||
num_inference_steps: int,
|
||||
guidance_scale: float,
|
||||
mask_blur: int,
|
||||
voxel_size: float,
|
||||
min_points_per_voxel: int,
|
||||
model_name: str,
|
||||
) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
|
||||
M_np = M_matrix.cpu().numpy()
|
||||
M_inv = np.linalg.inv(M_np)
|
||||
|
||||
# clean front points
|
||||
pc_front, = clean_node.clean_pointcloud(
|
||||
pc_front,
|
||||
voxel_size=voxel_size,
|
||||
min_points_per_voxel=min_points_per_voxel,
|
||||
width=4096,
|
||||
height=4096,
|
||||
)
|
||||
img, mask, depth_map, pc_front = self._prepare_view_data(
|
||||
current_enriched_pc, M_np, transform_node, clean_node, proj_node,
|
||||
voxel_size, min_points_per_voxel, camera_type, horizontal_fov,
|
||||
width, height
|
||||
)
|
||||
|
||||
# project to image + depth
|
||||
img, mask, depth_map = proj_node.project_pointcloud(
|
||||
pc_front,
|
||||
camera_type,
|
||||
horizontal_fov,
|
||||
width,
|
||||
height,
|
||||
point_size=3,
|
||||
return_inverse_depth=False,
|
||||
)
|
||||
debug_img = img
|
||||
# fill nan in depthmap with (-1)
|
||||
# outpaint missing regions
|
||||
hole_mask = (mask < 0.5).float()
|
||||
out_img, out_mask = outpaint_node.outpaint_any(
|
||||
img,
|
||||
input_projection = camera_type,
|
||||
input_horiz_fov = horizontal_fov,
|
||||
output_projection = camera_type,
|
||||
output_horiz_fov = horizontal_fov,
|
||||
output_width = width,
|
||||
output_height = height,
|
||||
patch_projection = patch_projection,
|
||||
patch_horiz_fov = patch_horiz_fov,
|
||||
patch_res = patch_res,
|
||||
patch_phi = patch_phi,
|
||||
patch_theta = patch_theta,
|
||||
prompt = prompt,
|
||||
num_inference_steps = num_inference_steps,
|
||||
cached = False,
|
||||
guidance_scale = guidance_scale,
|
||||
mask_blur = mask_blur,
|
||||
mask = hole_mask,
|
||||
debug = False,
|
||||
)
|
||||
debug_img = out_img
|
||||
# estimate and renormalize depth
|
||||
nan_mask = torch.isnan(depth_map)
|
||||
# …and replace them with –1.0 (in-place)
|
||||
depth_map[nan_mask] = 0
|
||||
# clip from -1 to 1000
|
||||
depth_map = torch.clamp(depth_map, 0, 1000.0)
|
||||
new_depth, = depth_node.estimate_depth(out_img, model_name, depth_scale=1.0)
|
||||
new_depth, = zdepth_node.depth_to_ray_depth(
|
||||
new_depth,
|
||||
horizontal_fov,
|
||||
)
|
||||
# renormalize depth
|
||||
norm_depth, = renorm_node.renormalize_depth(
|
||||
new_depth,
|
||||
depth_map,
|
||||
depth_mask=(mask>=0.5)*1,
|
||||
guidance_mask=(mask<0.5)*1,
|
||||
use_inverse=False,
|
||||
)
|
||||
# median blur on depth
|
||||
k = 5
|
||||
d = norm_depth.permute(0,3,1,2) # [B,1,H,W]
|
||||
pad = k//2
|
||||
pd = F.pad(d, (pad, pad, pad, pad), mode='reflect')
|
||||
patches = pd.unfold(2, k, 1).unfold(3, k, 1)
|
||||
patches = patches.contiguous().view(d.shape[0], d.shape[1], d.shape[2], d.shape[3], k*k)
|
||||
d, _ = patches.median(dim=-1)
|
||||
norm_depth = d.permute(0,2,3,1) # [B,H,W,1]
|
||||
debug_depth = norm_depth*hole_mask.unsqueeze(0).unsqueeze(-1)+depth_map*(1-hole_mask.unsqueeze(0).unsqueeze(-1))
|
||||
# fill nan in depthmap with (-1)
|
||||
# outpaint missing regions
|
||||
hole_mask = (mask < 0.5).float()
|
||||
out_img = self._outpaint_missing_regions(
|
||||
img, hole_mask, # Pass hole_mask instead of the full mask
|
||||
outpaint_node, camera_type, horizontal_fov, width, height,
|
||||
patch_projection, patch_horiz_fov, patch_res,
|
||||
patch_phi, patch_theta, prompt,
|
||||
num_inference_steps, guidance_scale, mask_blur
|
||||
)
|
||||
# estimate and renormalize depth
|
||||
norm_depth, debug_depth_view = self._estimate_and_refine_depth(
|
||||
out_img, depth_map, mask, hole_mask,
|
||||
depth_node, zdepth_node, renorm_node,
|
||||
model_name, horizontal_fov
|
||||
)
|
||||
|
||||
# back to pointcloud
|
||||
pc_new, = depth2pc_node.depth_to_pointcloud(
|
||||
out_img,
|
||||
camera_type,
|
||||
horizontal_fov,
|
||||
depth_scale=1.0,
|
||||
invert_depth=False,
|
||||
depthmap=norm_depth,
|
||||
mask=hole_mask,
|
||||
)
|
||||
# back to pointcloud
|
||||
pc_world = self._convert_depth_to_world_pointcloud(
|
||||
out_img, norm_depth, hole_mask, M_inv,
|
||||
depth2pc_node, transform_node,
|
||||
camera_type, horizontal_fov
|
||||
)
|
||||
# enriched_pc is not rotated
|
||||
current_enriched_pc = torch.cat([current_enriched_pc, pc_world.to(device)], dim=0)
|
||||
return current_enriched_pc, out_img, debug_depth_view # Return out_img and the depth for this view
|
||||
|
||||
pc_world, = transform_node.transform_pointcloud(pc_new, M_inv)
|
||||
# enriched_pc is not rotated
|
||||
enriched_pc = torch.cat([enriched_pc, pc_world.to(device)], dim=0)
|
||||
return enriched_pc, debug_img, norm_depth
|
||||
|
||||
def _convert_depth_to_world_pointcloud(
|
||||
self,
|
||||
out_img: torch.Tensor,
|
||||
norm_depth: torch.Tensor,
|
||||
hole_mask: torch.Tensor,
|
||||
M_inv: np.ndarray,
|
||||
depth2pc_node: DepthToPointCloud,
|
||||
transform_node: TransformPointCloud,
|
||||
camera_type: str,
|
||||
horizontal_fov: float,
|
||||
) -> torch.Tensor:
|
||||
pc_new, = depth2pc_node.depth_to_pointcloud(
|
||||
out_img,
|
||||
camera_type,
|
||||
horizontal_fov,
|
||||
depth_scale=1.0,
|
||||
invert_depth=False,
|
||||
depthmap=norm_depth,
|
||||
mask=hole_mask,
|
||||
)
|
||||
pc_world, = transform_node.transform_pointcloud(pc_new, M_inv)
|
||||
return pc_world
|
||||
|
||||
def _estimate_and_refine_depth(
|
||||
self,
|
||||
out_img: torch.Tensor,
|
||||
depth_map: torch.Tensor,
|
||||
original_mask: torch.Tensor, # Mask from projection
|
||||
hole_mask: torch.Tensor,
|
||||
depth_node: DepthEstimatorNode,
|
||||
zdepth_node: ZDepthToRayDepthNode,
|
||||
renorm_node: DepthRenormalizer,
|
||||
model_name: str,
|
||||
horizontal_fov: float,
|
||||
) -> Tuple[torch.Tensor, torch.Tensor]:
|
||||
nan_mask = torch.isnan(depth_map)
|
||||
depth_map[nan_mask] = 0 # In-place modification
|
||||
depth_map = torch.clamp(depth_map, 0, 1000.0)
|
||||
|
||||
new_depth, = depth_node.estimate_depth(out_img, model_name, depth_scale=1.0)
|
||||
new_depth, = zdepth_node.depth_to_ray_depth(
|
||||
new_depth,
|
||||
horizontal_fov,
|
||||
)
|
||||
# renormalize depth
|
||||
# Use original_mask for depth_mask as it represents valid projected areas
|
||||
norm_depth, = renorm_node.renormalize_depth(
|
||||
new_depth,
|
||||
depth_map,
|
||||
depth_mask=(original_mask >= 0.5) * 1,
|
||||
guidance_mask=(hole_mask >= 0.5) * 1, # hole_mask is appropriate here
|
||||
use_inverse=False,
|
||||
)
|
||||
# median blur on depth
|
||||
k = 5
|
||||
d = norm_depth.permute(0,3,1,2) # [B,1,H,W]
|
||||
pad = k//2
|
||||
pd = F.pad(d, (pad, pad, pad, pad), mode='reflect')
|
||||
patches = pd.unfold(2, k, 1).unfold(3, k, 1)
|
||||
patches = patches.contiguous().view(d.shape[0], d.shape[1], d.shape[2], d.shape[3], k*k)
|
||||
d_median, _ = patches.median(dim=-1) # Renamed to avoid conflict
|
||||
norm_depth_blurred = d_median.permute(0,2,3,1) # [B,H,W,1]
|
||||
|
||||
# Create debug_depth_view using the blurred normalized depth for holes
|
||||
# and the original depth_map for non-holes.
|
||||
debug_depth_view = norm_depth_blurred * hole_mask.unsqueeze(0).unsqueeze(-1) + \
|
||||
depth_map * (1 - hole_mask.unsqueeze(0).unsqueeze(-1))
|
||||
|
||||
return norm_depth_blurred, debug_depth_view
|
||||
|
||||
|
||||
def _outpaint_missing_regions(
|
||||
self,
|
||||
img: torch.Tensor,
|
||||
hole_mask: torch.Tensor, # Expects the specific hole_mask
|
||||
outpaint_node: OutpaintAnyProjection,
|
||||
camera_type: str,
|
||||
horizontal_fov: float,
|
||||
width: int,
|
||||
height: int,
|
||||
patch_projection: str,
|
||||
patch_horiz_fov: float,
|
||||
patch_res: int,
|
||||
patch_phi: float,
|
||||
patch_theta: float,
|
||||
prompt: str,
|
||||
num_inference_steps: int,
|
||||
guidance_scale: float,
|
||||
mask_blur: int,
|
||||
) -> torch.Tensor: # Returns only out_img, out_mask is not used later
|
||||
out_img, _ = outpaint_node.outpaint_any( # Assign out_mask to _
|
||||
img,
|
||||
input_projection = camera_type,
|
||||
input_horiz_fov = horizontal_fov,
|
||||
output_projection = camera_type,
|
||||
output_horiz_fov = horizontal_fov,
|
||||
output_width = width,
|
||||
output_height = height,
|
||||
patch_projection = patch_projection,
|
||||
patch_horiz_fov = patch_horiz_fov,
|
||||
patch_res = patch_res,
|
||||
patch_phi = patch_phi,
|
||||
patch_theta = patch_theta,
|
||||
prompt = prompt,
|
||||
num_inference_steps = num_inference_steps,
|
||||
cached = False,
|
||||
guidance_scale = guidance_scale,
|
||||
mask_blur = mask_blur,
|
||||
mask = hole_mask, # Use the passed hole_mask
|
||||
debug = False,
|
||||
)
|
||||
return out_img
|
||||
|
||||
def _prepare_view_data(
|
||||
self,
|
||||
current_enriched_pc: torch.Tensor,
|
||||
M_np: np.ndarray,
|
||||
transform_node: TransformPointCloud,
|
||||
clean_node: PointCloudCleaner,
|
||||
proj_node: ProjectPointCloud,
|
||||
voxel_size: float,
|
||||
min_points_per_voxel: int,
|
||||
camera_type: str,
|
||||
horizontal_fov: float,
|
||||
width: int,
|
||||
height: int,
|
||||
) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]:
|
||||
# transform and select front points
|
||||
rotated, = transform_node.transform_pointcloud(current_enriched_pc, M_np)
|
||||
pc_front = rotated[rotated[:, 2] > 0]
|
||||
|
||||
# clean front points
|
||||
pc_front, = clean_node.clean_pointcloud(
|
||||
pc_front,
|
||||
voxel_size=voxel_size,
|
||||
min_points_per_voxel=min_points_per_voxel,
|
||||
width=4096, # Consider passing these as params if they vary
|
||||
height=4096, # Consider passing these as params if they vary
|
||||
)
|
||||
|
||||
# project to image + depth
|
||||
img, mask, depth_map = proj_node.project_pointcloud(
|
||||
pc_front,
|
||||
camera_type,
|
||||
horizontal_fov,
|
||||
width,
|
||||
height,
|
||||
point_size=3,
|
||||
return_inverse_depth=False,
|
||||
)
|
||||
return img, mask, depth_map, pc_front
|
||||
|
||||
NODE_CLASS_MAPPINGS = {"FisheyeDepthEstimator": FisheyeDepthEstimator,
|
||||
"PointcloudTrajectoryEnricher": PointcloudTrajectoryEnricher}
|
||||
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 224 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 13 MiB |
@@ -129,7 +129,7 @@ class OutpaintAnyProjection:
|
||||
patch_mask = torch.ones_like(patch_mask) * 1 if debug else patch_mask
|
||||
# 5) Reproject inpainted patch back
|
||||
|
||||
back_img, back_mask = reproj.reproject_image(
|
||||
back_img, inpainted_patch_reproj_mask_raw = reproj.reproject_image( # Store raw mask output
|
||||
inpainted_patch,
|
||||
patch_horiz_fov, output_horiz_fov,
|
||||
patch_projection, output_projection,
|
||||
@@ -138,19 +138,37 @@ class OutpaintAnyProjection:
|
||||
transform_matrix=rot_m,
|
||||
feathering=0,
|
||||
)
|
||||
back_mask = normalize_mask(back_mask).bool() # True where patch contributes
|
||||
# inpainted_patch_coverage_mask: 1.0 where the reprojected inpainted patch has content, 0.0 otherwise.
|
||||
inpainted_patch_coverage_mask = normalize_mask(back_mask) # Ensure it's float [0,1]
|
||||
|
||||
# original coverage: True = had data, False = hole
|
||||
orig_covered = ~base_mask.bool()
|
||||
base_img=base_img * orig_covered.unsqueeze(-1)
|
||||
# fill only holes where back_mask is False
|
||||
filled = back_img * (~back_mask.unsqueeze(-1))*base_mask.unsqueeze(-1)
|
||||
final_img = base_img+filled
|
||||
# --- Define Masks based on Conventions ---
|
||||
# base_mask: 1.0 where original reprojected image has content, 0.0 for holes.
|
||||
# initial_hole_mask: 1.0 where original reprojected image has holes (inverse of base_mask).
|
||||
initial_hole_mask = 1.0 - base_mask
|
||||
# inpainted_patch_coverage_mask: 1.0 where reprojected inpainted patch has content.
|
||||
|
||||
# anything that’s still a hole after back‐projection needs inpaint
|
||||
needs_inpaint = (~((orig_covered) | (~back_mask))).to(torch.float32)
|
||||
# --- Compositing Logic ---
|
||||
# Goal: Inpainted patch takes precedence in overlapping areas. Original content is used elsewhere.
|
||||
|
||||
return final_img, needs_inpaint
|
||||
# Contribution from the original image:
|
||||
# Valid original pixels, excluding areas covered by the inpainted patch.
|
||||
original_content_contribution = base_img * base_mask.unsqueeze(-1) * \
|
||||
(1.0 - inpainted_patch_coverage_mask.unsqueeze(-1))
|
||||
|
||||
# Contribution from the inpainted patch (reprojected as back_img):
|
||||
# Valid inpainted pixels, where the patch provides coverage.
|
||||
inpainted_patch_contribution = back_img * inpainted_patch_coverage_mask.unsqueeze(-1)
|
||||
|
||||
# Combine:
|
||||
final_img = original_content_contribution + inpainted_patch_contribution
|
||||
|
||||
# --- needs_inpaint_mask Derivation ---
|
||||
# Identifies areas that were initially holes AND remain un-filled by the reprojected inpainted patch.
|
||||
# These are areas that still require inpainting if a further pass was to be made.
|
||||
not_covered_by_inpainted_patch = 1.0 - inpainted_patch_coverage_mask
|
||||
needs_inpaint_mask = initial_hole_mask * not_covered_by_inpainted_patch # Element-wise multiplication (AND logic)
|
||||
|
||||
return final_img, needs_inpaint_mask
|
||||
|
||||
# register
|
||||
NODE_CLASS_MAPPINGS = {
|
||||
|
||||
-125
@@ -1,125 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# ----------------------------------------
|
||||
# Functions
|
||||
# ----------------------------------------
|
||||
install_pytorch() {
|
||||
echo "Installing PyTorch, TorchVision, TorchAudio..."
|
||||
pip3 install -U torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128
|
||||
pip3 install bitsandbytes
|
||||
pip3 install accelerate
|
||||
}
|
||||
|
||||
install_system_deps() {
|
||||
echo "Updating apt and installing system dependencies..."
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y build-essential ffmpeg libsm6 libxext6
|
||||
}
|
||||
|
||||
clone_and_install_comfyui() {
|
||||
echo "Cloning ComfyUI..."
|
||||
git clone https://github.com/comfyanonymous/ComfyUI.git
|
||||
echo "Installing ComfyUI requirements..."
|
||||
pip3 install -r ComfyUI/requirements.txt
|
||||
}
|
||||
|
||||
install_camera_node() {
|
||||
echo "Cloning camera‑ComfyUI..."
|
||||
git clone https://github.com/Alexankharin/camera-comfyUI.git \
|
||||
ComfyUI/custom_nodes/camera-comfyUI
|
||||
echo "Installing camera‑ComfyUI requirements..."
|
||||
pip3 install -r ComfyUI/custom_nodes/camera-comfyUI/requirements.txt
|
||||
}
|
||||
|
||||
install_image_filters() {
|
||||
echo "Cloning Image‑Filters node..."
|
||||
git clone https://github.com/spacepxl/ComfyUI-Image-Filters.git \
|
||||
ComfyUI/custom_nodes/ComfyUI-Image-Filters
|
||||
echo "Installing Image‑Filters requirements..."
|
||||
pip3 install -r ComfyUI/custom_nodes/ComfyUI-Image-Filters/requirements.txt
|
||||
}
|
||||
|
||||
clone_flux_inpainting() {
|
||||
echo "Cloning ComfyUI‑Flux‑Inpainting..."
|
||||
git clone https://github.com/rubi-du/ComfyUI-Flux-Inpainting.git \
|
||||
ComfyUI/custom_nodes/Flux-Inpainting
|
||||
echo "Renaming Flux‑Inpainting folder..."
|
||||
mv ComfyUI/custom_nodes/Flux-Inpainting \
|
||||
ComfyUI/custom_nodes/inpainting_flux
|
||||
}
|
||||
|
||||
install_hf_hub() {
|
||||
echo "Installing huggingface_hub..."
|
||||
pip3 install huggingface_hub
|
||||
}
|
||||
|
||||
download_vae_models() {
|
||||
echo "Downloading WAN‑VACE models via wget..."
|
||||
wget -O ComfyUI/models/vae/wan_2.1_vae.safetensors \
|
||||
"https://huggingface.co/Comfy-Org/Wan_2.1_ComfyUI_repackaged/resolve/main/split_files/vae/wan_2.1_vae.safetensors?download=true"
|
||||
|
||||
wget -O ComfyUI/models/text_encoders/umt5_xxl_fp16.safetensors \
|
||||
"https://huggingface.co/Comfy-Org/Wan_2.1_ComfyUI_repackaged/resolve/main/split_files/text_encoders/umt5_xxl_fp16.safetensors?download=true"
|
||||
|
||||
wget -O ComfyUI/models/diffusion_models/wan2.1_vace_14B_fp16.safetensors \
|
||||
"https://huggingface.co/Comfy-Org/Wan_2.1_ComfyUI_repackaged/resolve/main/split_files/diffusion_models/wan2.1_vace_14B_fp16.safetensors"
|
||||
}
|
||||
|
||||
login_hf_hub() {
|
||||
echo "Logging in to Hugging Face Hub..."
|
||||
huggingface-cli login
|
||||
}
|
||||
|
||||
# ----------------------------------------
|
||||
# Main CLI
|
||||
# ----------------------------------------
|
||||
# default to "install" if no arg given
|
||||
MODE="${1:-install}"
|
||||
|
||||
case "$MODE" in
|
||||
modules)
|
||||
install_pytorch
|
||||
install_system_deps
|
||||
clone_and_install_comfyui
|
||||
install_camera_node
|
||||
install_image_filters
|
||||
install_hf_hub
|
||||
;;
|
||||
|
||||
flux)
|
||||
clone_flux_inpainting
|
||||
;;
|
||||
|
||||
vae)
|
||||
download_vae_models
|
||||
;;
|
||||
|
||||
install)
|
||||
install_pytorch
|
||||
install_system_deps
|
||||
clone_and_install_comfyui
|
||||
install_camera_node
|
||||
clone_flux_inpainting
|
||||
install_image_filters
|
||||
install_hf_hub
|
||||
;;
|
||||
|
||||
all)
|
||||
install_pytorch
|
||||
install_system_deps
|
||||
clone_and_install_comfyui
|
||||
install_camera_node
|
||||
clone_flux_inpainting
|
||||
install_image_filters
|
||||
install_hf_hub
|
||||
download_vae_models
|
||||
login_hf_hub
|
||||
echo "All done! 🎉"
|
||||
;;
|
||||
|
||||
*)
|
||||
echo "Usage: $0 {install|modules|flux|vae|all}"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
@@ -41,7 +41,7 @@ class DepthEstimatorNode:
|
||||
RETURN_TYPES = ("TENSOR",)
|
||||
RETURN_NAMES = ("depth tensor",)
|
||||
FUNCTION = "estimate_depth"
|
||||
CATEGORY = "Camera/Depth"
|
||||
CATEGORY = "Camera/depth"
|
||||
|
||||
def estimate_depth(
|
||||
self,
|
||||
@@ -110,7 +110,7 @@ class DepthToImageNode:
|
||||
RETURN_TYPES = ("IMAGE",)
|
||||
RETURN_NAMES = ("depth image",)
|
||||
FUNCTION = "depth_to_image"
|
||||
CATEGORY = "Camera/Depth"
|
||||
CATEGORY = "Camera/depth"
|
||||
|
||||
def depth_to_image(
|
||||
self,
|
||||
@@ -158,7 +158,7 @@ class ZDepthToRayDepthNode:
|
||||
RETURN_TYPES = ("TENSOR",)
|
||||
RETURN_NAMES = ("ray depth",)
|
||||
FUNCTION = "depth_to_ray_depth"
|
||||
CATEGORY = "Camera/Depth"
|
||||
CATEGORY = "Camera/depth"
|
||||
|
||||
def depth_to_ray_depth(
|
||||
self,
|
||||
@@ -229,7 +229,7 @@ class CombineDepthsNode:
|
||||
RETURN_TYPES = ("TENSOR","MASK")
|
||||
RETURN_NAMES = ("combined_depth","combined_mask")
|
||||
FUNCTION = "combine_depths"
|
||||
CATEGORY = "Camera/Depth"
|
||||
CATEGORY = "Camera/depth"
|
||||
|
||||
def combine_depths(
|
||||
self,
|
||||
@@ -358,7 +358,7 @@ class DepthRenormalizer:
|
||||
RETURN_TYPES = ("TENSOR",)
|
||||
RETURN_NAMES = ("depth tensor",)
|
||||
FUNCTION = "renormalize_depth"
|
||||
CATEGORY = "Camera/Depth"
|
||||
CATEGORY = "Camera/depth"
|
||||
|
||||
def renormalize_depth(
|
||||
self,
|
||||
|
||||
+54
-103
@@ -113,7 +113,7 @@ def XYZ_to_equirect(X: torch.Tensor, Y: torch.Tensor, Z: torch.Tensor, fov: floa
|
||||
Convert XYZ coordinates to normalized UV and depth using equirectangular projection.
|
||||
"""
|
||||
# full 360°×180°
|
||||
fov_rad = math.radians(fov) / 2
|
||||
fov_rad = math.radians(fov)
|
||||
depth = torch.sqrt(X**2 + Y**2 + Z**2)
|
||||
lon = torch.atan2(X, Z) # –π → +π
|
||||
lat = torch.asin(Y / depth) # –π/2 → +π/2
|
||||
@@ -167,7 +167,7 @@ class DepthToPointCloud:
|
||||
RETURN_TYPES = ("TENSOR",)
|
||||
RETURN_NAMES = ("pointcloud",)
|
||||
FUNCTION = "depth_to_pointcloud"
|
||||
CATEGORY = "Camera/PointCloud"
|
||||
CATEGORY = "Camera/pointcloud"
|
||||
|
||||
def depth_to_pointcloud(
|
||||
self,
|
||||
@@ -273,7 +273,7 @@ class TransformPointCloud:
|
||||
RETURN_TYPES = ("TENSOR",)
|
||||
RETURN_NAMES = ("transformed pointcloud",)
|
||||
FUNCTION = "transform_pointcloud"
|
||||
CATEGORY = "Camera/PointCloud"
|
||||
CATEGORY = "Camera/pointcloud"
|
||||
|
||||
def transform_pointcloud(
|
||||
self,
|
||||
@@ -321,7 +321,7 @@ class ProjectPointCloud:
|
||||
RETURN_TYPES = ("IMAGE", "MASK", "TENSOR")
|
||||
RETURN_NAMES = ("image", "mask", "depth")
|
||||
FUNCTION = "project_pointcloud"
|
||||
CATEGORY = "Camera/PointCloud"
|
||||
CATEGORY = "Camera/pointcloud"
|
||||
|
||||
def project_pointcloud(
|
||||
self,
|
||||
@@ -456,7 +456,7 @@ class PointCloudUnion:
|
||||
RETURN_TYPES = ("TENSOR",)
|
||||
RETURN_NAMES =("merged pointcloud",)
|
||||
FUNCTION = "union_pointclouds"
|
||||
CATEGORY = "Camera/PointCloud"
|
||||
CATEGORY = "Camera/pointcloud"
|
||||
|
||||
def union_pointclouds(
|
||||
self,
|
||||
@@ -492,7 +492,7 @@ class LoadPointCloud:
|
||||
}
|
||||
}
|
||||
|
||||
CATEGORY = "Camera/PointCloud"
|
||||
CATEGORY = "Camera/pointcloud"
|
||||
RETURN_TYPES = ("TENSOR",)
|
||||
RETURN_NAMES = ("loaded pointcloud",)
|
||||
FUNCTION = "load_pointcloud"
|
||||
@@ -504,45 +504,24 @@ class LoadPointCloud:
|
||||
arr = np.load(file_path)
|
||||
tensor_pc = torch.from_numpy(arr)
|
||||
return (tensor_pc,)
|
||||
|
||||
if o3d is None:
|
||||
logging.warning("[camera-comfyUI] open3d is not installed. Falling back to manual PLY parser.")
|
||||
coords = []
|
||||
colors = []
|
||||
with open(file_path, 'r') as f:
|
||||
coords = []
|
||||
colors = []
|
||||
with open(file_path, 'r') as f:
|
||||
line = f.readline().strip()
|
||||
while not line.startswith("end_header"):
|
||||
line = f.readline().strip()
|
||||
while not line.startswith("end_header"):
|
||||
line = f.readline().strip()
|
||||
for line in f:
|
||||
parts = line.strip().split()
|
||||
if len(parts) < 7:
|
||||
continue
|
||||
x, y, z = map(float, parts[0:3])
|
||||
r, g, b, a = map(float, parts[3:7])
|
||||
coords.append((x, y, z))
|
||||
colors.append((r, g, b, a))
|
||||
np_coords = np.array(coords, dtype=np.float32)
|
||||
np_colors = np.array(colors, dtype=np.float32)
|
||||
# if colors are > 1, normalize them to [0,1]
|
||||
if np_colors.max() > 1.0:
|
||||
np_colors = np_colors / 255.0
|
||||
else:
|
||||
pc = o3d.t.io.read_point_cloud(file_path)
|
||||
np_coords = pc.point["positions"].numpy().astype(np.float32)
|
||||
if "colors" in pc.point:
|
||||
cols = pc.point["colors"].numpy().astype(np.float32)
|
||||
else:
|
||||
cols = np.ones((np_coords.shape[0], 3), dtype=np.float32)
|
||||
if "alpha" in pc.point:
|
||||
alpha = pc.point["alpha"].numpy().astype(np.float32)
|
||||
else:
|
||||
alpha = np.ones((np_coords.shape[0], 1), dtype=np.float32)
|
||||
np_colors = np.concatenate([cols, alpha], axis=1)
|
||||
if np_colors.max() > 1.0:
|
||||
np_colors = np_colors / 255.0
|
||||
# combine coords and colors into a single tensor
|
||||
combined = np.concatenate([np_coords, np_colors], axis=1)
|
||||
tensor_pc = torch.from_numpy(combined)
|
||||
for line in f:
|
||||
parts = line.strip().split()
|
||||
if len(parts) < 7:
|
||||
continue
|
||||
x, y, z = map(float, parts[0:3])
|
||||
r, g, b, a = map(int, parts[3:7])
|
||||
coords.append((x, y, z))
|
||||
colors.append((r, g, b, a))
|
||||
np_coords = np.array(coords, dtype=np.float32)
|
||||
np_colors = np.array(colors, dtype=np.float32)/255.0
|
||||
combined = np.concatenate([np_coords, np_colors], axis=1)
|
||||
tensor_pc = torch.from_numpy(combined)
|
||||
return (tensor_pc,)
|
||||
|
||||
@classmethod
|
||||
@@ -590,7 +569,7 @@ class SavePointCloud:
|
||||
RETURN_TYPES = ()
|
||||
FUNCTION = "save_pointcloud"
|
||||
OUTPUT_NODE = True
|
||||
CATEGORY = "Camera/PointCloud"
|
||||
CATEGORY = "Camera/pointcloud"
|
||||
DESCRIPTION = "Saves the input point cloud to your ComfyUI output directory as .ply or .npy."
|
||||
|
||||
def save_pointcloud(self, pointcloud: torch.Tensor, filename_prefix: str, save_as: str = "ply"):
|
||||
@@ -607,36 +586,24 @@ class SavePointCloud:
|
||||
os.makedirs(full_output_folder, exist_ok=True)
|
||||
base_name = filename.replace("%batch_num%", "0")
|
||||
if save_as == "ply":
|
||||
ply_name = f"{base_name}_{counter:05}.ply"
|
||||
ply_path = os.path.join(full_output_folder, ply_name)
|
||||
coords = pointcloud[:, :3].cpu().numpy().astype(np.float32)
|
||||
colors = pointcloud[:, 3:].cpu().numpy().clip(0, 1).astype(np.float32)
|
||||
|
||||
if o3d is None:
|
||||
logging.warning("[camera-comfyUI] open3d is not installed. Falling back to manual ASCII PLY writer.")
|
||||
with open(ply_path, 'w') as f:
|
||||
f.write("ply\n")
|
||||
f.write("format ascii 1.0\n")
|
||||
f.write(f"element vertex {coords.shape[0]}\n")
|
||||
f.write("property float x\n")
|
||||
f.write("property float y\n")
|
||||
f.write("property float z\n")
|
||||
f.write("property float red\n")
|
||||
f.write("property float green\n")
|
||||
f.write("property float blue\n")
|
||||
f.write("property float alpha\n")
|
||||
f.write("end_header\n")
|
||||
for (x, y, z), (r, g, b, a) in zip(coords, colors):
|
||||
f.write(f"{x} {y} {z} {r} {g} {b} {a}\n")
|
||||
else:
|
||||
pc = o3d.t.geometry.PointCloud()
|
||||
pc.point["positions"] = o3d.core.Tensor(coords, o3d.core.float32)
|
||||
pc.point["colors"] = o3d.core.Tensor(colors[:, :3], o3d.core.float32)
|
||||
if colors.shape[1] > 3:
|
||||
pc.point["alpha"] = o3d.core.Tensor(colors[:, 3:], o3d.core.float32)
|
||||
else:
|
||||
pc.point["alpha"] = o3d.core.Tensor(np.ones((coords.shape[0], 1), dtype=np.float32), o3d.core.float32)
|
||||
o3d.t.io.write_point_cloud(ply_path, pc)
|
||||
ply_name = f"{base_name}_{counter:05}.ply"
|
||||
ply_path = os.path.join(full_output_folder, ply_name)
|
||||
coords = pointcloud[:, :3].cpu().numpy()
|
||||
colors = pointcloud[:, 3:].cpu().numpy().clip(0,1)
|
||||
with open(ply_path, 'w') as f:
|
||||
f.write("ply\n")
|
||||
f.write("format ascii 1.0\n")
|
||||
f.write(f"element vertex {coords.shape[0]}\n")
|
||||
f.write("property float x\n")
|
||||
f.write("property float y\n")
|
||||
f.write("property float z\n")
|
||||
f.write("property uchar red\n")
|
||||
f.write("property uchar green\n")
|
||||
f.write("property uchar blue\n")
|
||||
f.write("property uchar alpha\n")
|
||||
f.write("end_header\n")
|
||||
for (x,y,z), (r,g,b,a) in zip(coords, colors):
|
||||
f.write(f"{x} {y} {z} {int(r*255)} {int(g*255)} {int(b*255)} {int(a*255)}\n")
|
||||
file_name = ply_name
|
||||
else:
|
||||
npy_name = f"{base_name}_{counter:05}.npy"
|
||||
@@ -673,15 +640,12 @@ class CameraMotionNode:
|
||||
"output_width": ("INT", {"default":512, "min":8, "max":16384}),
|
||||
"output_height": ("INT", {"default":512, "min":8, "max":16384}),
|
||||
"point_size": ("INT", {"default":1, "min":1}),
|
||||
"widen_mask": ("INT", {"default":0, "min":0, "max":64}),
|
||||
"invert_mask": ("BOOLEAN", {"default": False}),
|
||||
"points_to_mask": ("BOOLEAN", {"default": False, "tooltip": "Output mask frames of projected points"}),
|
||||
}}
|
||||
|
||||
RETURN_TYPES = ("IMAGE", "MASK")
|
||||
RETURN_NAMES = ("motion_frames", "mask_frames")
|
||||
RETURN_TYPES = ("IMAGE",)
|
||||
RETURN_NAMES = ("motion_frames",)
|
||||
FUNCTION = "generate_motion_frames"
|
||||
CATEGORY = "Camera/Trajectory"
|
||||
CATEGORY = "Camera/pointcloud"
|
||||
|
||||
def generate_motion_frames(
|
||||
self,
|
||||
@@ -692,10 +656,7 @@ class CameraMotionNode:
|
||||
output_horizontal_fov: float,
|
||||
output_width: int,
|
||||
output_height: int,
|
||||
point_size: int = 1,
|
||||
widen_mask: int = 0,
|
||||
invert_mask: bool = False,
|
||||
points_to_mask: bool = False
|
||||
point_size: int = 1
|
||||
) -> Tuple[torch.Tensor]:
|
||||
# validate trajectory shape
|
||||
if trajectory.dim() != 3 or trajectory.shape[1:] != (4,4):
|
||||
@@ -722,10 +683,9 @@ class CameraMotionNode:
|
||||
proj_node = ProjectPointCloud()
|
||||
transform_node = TransformPointCloud()
|
||||
frames = []
|
||||
masks = []
|
||||
for M in tqdm(full_traj):
|
||||
pc_t, = transform_node.transform_pointcloud(pointcloud, M)
|
||||
img, mask, _ = proj_node.project_pointcloud(
|
||||
img, _, _ = proj_node.project_pointcloud(
|
||||
pc_t,
|
||||
output_projection,
|
||||
output_horizontal_fov,
|
||||
@@ -733,19 +693,10 @@ class CameraMotionNode:
|
||||
output_height,
|
||||
point_size
|
||||
)
|
||||
if widen_mask > 0:
|
||||
k = 2 * widen_mask + 1
|
||||
pad = widen_mask
|
||||
mask = F.max_pool2d(mask.float().unsqueeze(0).unsqueeze(0), kernel_size=k, stride=1, padding=pad).squeeze(0).squeeze(0)
|
||||
if invert_mask:
|
||||
mask = 1.0 - mask
|
||||
masks.append(mask)
|
||||
if points_to_mask:
|
||||
img = mask.unsqueeze(-1).repeat(1,1,1,3)
|
||||
frames.append(img[0])
|
||||
|
||||
# output as (T,H,W,3)
|
||||
return (torch.stack(frames, dim=0), torch.stack(masks, dim=0))
|
||||
return (torch.stack(frames, dim=0),)
|
||||
|
||||
class CameraInterpolationNode:
|
||||
"""
|
||||
@@ -764,7 +715,7 @@ class CameraInterpolationNode:
|
||||
RETURN_TYPES = ("TENSOR",)
|
||||
RETURN_NAMES = ("trajectory",)
|
||||
FUNCTION = "interpolate"
|
||||
CATEGORY = "Camera/Trajectory"
|
||||
CATEGORY = "Camera/pointcloud"
|
||||
|
||||
def interpolate(
|
||||
self,
|
||||
@@ -799,7 +750,7 @@ class CameraTrajectoryNode:
|
||||
RETURN_TYPES = ("TENSOR",)
|
||||
RETURN_NAMES = ("trajectory",)
|
||||
FUNCTION = "build_trajectory"
|
||||
CATEGORY = "Camera/Trajectory"
|
||||
CATEGORY = "Camera/pointcloud"
|
||||
|
||||
def build_trajectory(
|
||||
self,
|
||||
@@ -939,7 +890,7 @@ class PointCloudCleaner:
|
||||
RETURN_TYPES = ("TENSOR",)
|
||||
RETURN_NAMES = ("cleaned_pointcloud",)
|
||||
FUNCTION = "clean_pointcloud"
|
||||
CATEGORY = "Camera/PointCloud"
|
||||
CATEGORY = "Camera/pointcloud"
|
||||
|
||||
def clean_pointcloud(
|
||||
self,
|
||||
@@ -1015,7 +966,7 @@ class ProjectAndClean:
|
||||
RETURN_TYPES = ("TENSOR",)
|
||||
RETURN_NAMES = ("cleaned_pointcloud",)
|
||||
FUNCTION = "project_and_clean"
|
||||
CATEGORY = "Camera/PointCloud"
|
||||
CATEGORY = "Camera/pointcloud"
|
||||
|
||||
def project_and_clean(
|
||||
self,
|
||||
@@ -1129,7 +1080,7 @@ class SaveTrajectory:
|
||||
RETURN_TYPES = ()
|
||||
FUNCTION = "save_trajectory"
|
||||
OUTPUT_NODE = True
|
||||
CATEGORY = "Camera/Trajectory"
|
||||
CATEGORY = "Camera/pointcloud"
|
||||
DESCRIPTION = "Saves the input trajectory tensor (N,4,4) to your ComfyUI output directory as .npy."
|
||||
|
||||
def save_trajectory(self, trajectory: torch.Tensor, filename_prefix: str):
|
||||
@@ -1179,7 +1130,7 @@ class LoadTrajectory:
|
||||
}
|
||||
}
|
||||
|
||||
CATEGORY = "Camera/Trajectory"
|
||||
CATEGORY = "Camera/pointcloud"
|
||||
RETURN_TYPES = ("TENSOR",)
|
||||
RETURN_NAMES = ("loaded_trajectory",)
|
||||
FUNCTION = "load_trajectory"
|
||||
|
||||
@@ -148,7 +148,7 @@ class ReprojectImage:
|
||||
RETURN_TYPES: Tuple[str, str] = ("IMAGE", "MASK")
|
||||
RETURN_NAMES = ("reprojected image", "reprojected mask")
|
||||
FUNCTION: str = "reproject_image"
|
||||
CATEGORY: str = "Camera/Reprojection"
|
||||
CATEGORY: str = "Camera/reproject"
|
||||
|
||||
def reproject_image(
|
||||
self,
|
||||
@@ -303,7 +303,7 @@ class TransformToMatrix:
|
||||
RETURN_TYPES: Tuple[str] = ("MAT_4X4",)
|
||||
RETURN_NAMES = ("transformation matrix",)
|
||||
FUNCTION: str = "generate_matrix"
|
||||
CATEGORY: str = "Camera/Matrix"
|
||||
CATEGORY: str = "Camera/reproject"
|
||||
|
||||
def generate_matrix(
|
||||
self,
|
||||
@@ -392,7 +392,7 @@ class TransformToMatrixManual:
|
||||
RETURN_TYPES: Tuple[str] = ("MAT_4X4",)
|
||||
RETURN_NAMES = ("transformation matrix",)
|
||||
FUNCTION: str = "generate_matrix"
|
||||
CATEGORY: str = "Camera/Matrix"
|
||||
CATEGORY: str = "Camera/reproject"
|
||||
|
||||
def generate_matrix(
|
||||
self,
|
||||
@@ -444,7 +444,7 @@ class ReprojectDepth:
|
||||
RETURN_TYPES: Tuple[str, str] = ("TENSOR", "MASK")
|
||||
RETURN_NAMES = ("reprojected_depth", "reprojected_mask")
|
||||
FUNCTION: str = "reproject_depth"
|
||||
CATEGORY: str = "Camera/Reprojection"
|
||||
CATEGORY: str = "Camera/reproject"
|
||||
|
||||
def reproject_depth(
|
||||
self,
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user