snomiaoandClaude Opus 4.6
dc2c298ce7
Two nodes: InfiniDepthModelLoader and InfiniDepthEstimate. Supports CUDA, MPS, and CPU via ComfyUI device management. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
111 lines
4.8 KiB
Markdown
111 lines
4.8 KiB
Markdown
# ComfyUI-InfiniDepth Development Notes
|
|
|
|
Notes from the initial development and testing of this custom node.
|
|
|
|
## Setup Experience
|
|
|
|
### ComfyUI Version
|
|
- Tested with ComfyUI v0.18.1 (latest as of 2026-03)
|
|
- Both V1 (NODE_CLASS_MAPPINGS) and V3 (comfy_entrypoint) registration styles are supported
|
|
- We use V1 style for broader compatibility
|
|
|
|
### Node Registration
|
|
- ComfyUI discovers custom nodes by importing `__init__.py` from directories under `custom_nodes/`
|
|
- The module must export `NODE_CLASS_MAPPINGS` dict (node_id -> class) and optionally `NODE_DISPLAY_NAME_MAPPINGS`
|
|
- Import errors during node loading are silently swallowed — check "Import times for custom nodes" in startup logs to verify the node loaded
|
|
|
|
### sys.path for InfiniDepth
|
|
- The `InfiniDepth` Python package lives at the repo root, not on pip
|
|
- `__init__.py` auto-detects the repo root via `os.path.realpath(__file__)` — this resolves symlinks correctly
|
|
- Three install methods supported:
|
|
1. Symlink: `ln -s <repo>/ComfyUI-InfiniDepth <ComfyUI>/custom_nodes/ComfyUI-InfiniDepth` (recommended)
|
|
2. Environment variable: `export INFINIDEPTH_ROOT=/path/to/repo`
|
|
3. Direct copy into `custom_nodes/` (less ideal, requires INFINIDEPTH_ROOT env var)
|
|
|
|
## Issues Encountered & Solutions
|
|
|
|
### 1. CUDA requirement in model init
|
|
**Problem:** `InfiniDepth/model/model.py` line ~99 does:
|
|
```python
|
|
if not torch.cuda.is_available():
|
|
raise RuntimeError("CUDA is required to initialize InfiniDepth models.")
|
|
self.cuda()
|
|
self.eval()
|
|
```
|
|
This hard-blocks loading on macOS (MPS) or CPU-only systems.
|
|
|
|
**Solution:** Monkey-patch `_BaseInfiniDepthModel.__init__` in the loader node to:
|
|
- Temporarily mock `torch.cuda.is_available()` to return `True`
|
|
- Replace `self.cuda()` with a no-op
|
|
- Let ComfyUI handle device placement via `comfy.model_management.get_torch_device()`
|
|
|
|
### 2. Missing grpc module
|
|
**Problem:** `InfiniDepth/model/block/implicit_decoder.py` imports `from grpc import insecure_channel`. This is not in `requirements.txt`.
|
|
|
|
**Solution:** Add `grpcio` to the custom node's `requirements.txt`.
|
|
|
|
### 3. Missing MoGe dependency
|
|
**Problem:** MoGe (Microsoft Monocular Geometry Estimation) is a git-only dependency needed for metric scale recovery when no depth sensor input is provided.
|
|
|
|
**Solution:** `install.py` handles this via `pip install git+https://github.com/microsoft/MoGe.git`. Users without MoGe will see a clear error at execution time.
|
|
|
|
### 4. HuggingFace checkpoint download
|
|
**Problem:** Checkpoints are hosted at `ritianyu/InfiniDepth` (not `YvanYin/InfiniDepth`). The repo downloads files flat, but InfiniDepth expects them in subdirectories.
|
|
|
|
**Expected structure:**
|
|
```
|
|
checkpoints/
|
|
depth/infinidepth.ckpt
|
|
depth/infinidepth_depthsensor.ckpt
|
|
gs/infinidepth_gs.ckpt
|
|
gs/infinidepth_depthsensor_gs.ckpt
|
|
moge-2-vitl-normal/model.pt
|
|
sky/skyseg.onnx
|
|
```
|
|
|
|
### 5. RGBA images (4 channels instead of 3)
|
|
**Problem:** Some ComfyUI image sources (e.g. LoadImage with PNG alpha, or nodes that output RGBA) produce `[B, H, W, 4]` tensors. InfiniDepth and MoGe-2 expect exactly 3 channels `[1, 3, H, W]`, causing:
|
|
```
|
|
ValueError: Expected image shape [1,3,H,W], got (1, 4, 768, 1024)
|
|
```
|
|
|
|
**Solution:** Slice to RGB before processing: `image[i][:, :, :3]`. Alpha channel is irrelevant for depth estimation.
|
|
|
|
### 6. Port conflict during testing
|
|
**Problem:** Default ComfyUI port 8188 was already in use by another instance.
|
|
|
|
**Solution:** Use `--port 8199` for testing. Always check before assuming the API returned data from your instance.
|
|
|
|
## IMAGE Format
|
|
- ComfyUI: `[B, H, W, C]` float32, channels-last, 0-1 range
|
|
- InfiniDepth: `[B, C, H, W]` float32, channels-first, 0-1 range
|
|
- Conversion: `.permute(0, 3, 1, 2)` before model, `.permute(0, 2, 3, 1)` after
|
|
|
|
## Performance
|
|
- Full inference on CPU (Apple M4 Max, 36GB): ~340 seconds for a single 1200x900 image at original resolution
|
|
- Model loading takes ~10 seconds (1.1GB checkpoint + DINOv3 backbone)
|
|
- MoGe-2 metric depth estimation adds ~60 seconds on CPU
|
|
- On CUDA GPU this should be dramatically faster (seconds, not minutes)
|
|
|
|
## Testing Workflow (API)
|
|
Minimal workflow JSON for testing via `POST /prompt`:
|
|
```json
|
|
{
|
|
"prompt": {
|
|
"1": {"class_type": "LoadImage", "inputs": {"image": "courtyard.jpg"}},
|
|
"2": {"class_type": "InfiniDepthModelLoader", "inputs": {
|
|
"model_type": "InfiniDepth",
|
|
"depth_ckpt": "checkpoints/depth/infinidepth.ckpt",
|
|
"moge2_ckpt": "checkpoints/moge-2-vitl-normal/model.pt"
|
|
}},
|
|
"3": {"class_type": "InfiniDepthEstimate", "inputs": {
|
|
"model": ["2", 0], "image": ["1", 0],
|
|
"output_resolution_mode": "original",
|
|
"upsample_ratio": 1, "output_height": 768, "output_width": 1024
|
|
}},
|
|
"4": {"class_type": "PreviewImage", "inputs": {"images": ["3", 0]}},
|
|
"5": {"class_type": "PreviewImage", "inputs": {"images": ["3", 1]}}
|
|
}
|
|
}
|
|
```
|