Capturing execution frames: da3_streaming.json Frame 1 saved (frame_000.jpg, t=0.0s) Validating workflow... [ComfyUI] === /validate endpoint called === [ComfyUI] Received JSON with keys: ['prompt'] [ComfyUI] Prompt has 5 nodes [ComfyUI] Node 2: DownloadAndLoadDepthAnythingV3Model [ComfyUI] Inputs: {"model": "da3_large.safetensors", "precision": "auto", "attention": "auto"} Queuing workflow for execution... [ComfyUI] Node 3: LoadVideo [ComfyUI] Inputs: {"file": "robot_unitree.mp4", "video-preview": ""} [ComfyUI] Node 4: SaveVideo [ComfyUI] Inputs: {"filename_prefix": "video/preview_da3", "format": "auto", "codec": "auto", "video": ["6", 0]} [ComfyUI] Node 5: LoadSALADModel [ComfyUI] Inputs: {} [ComfyUI] Node 6: DepthAnythingV3_Streaming [ComfyUI] Inputs: {"normalization_mode": "Standard", "chunk_size": 30, "overlap": 8, "align_lib": "auto", "align_method": "sim3", "resize_method": "resize", "invert_depth": true, "save_pointcloud": false, "sample_ratio [ComfyUI] Generated prompt_id: 966cf7aa-3736-47b1-b074-32ae7fdd616a [ComfyUI] Calling execution.validate_prompt()... [ComfyUI] Validation result: valid=True [ComfyUI] === Validation PASSED === [ComfyUI] got prompt [ComfyUI] [comfy-env] Running LoadSALADModel: [ComfyUI] [VRAM] Before LoadSALADModel: 7555 / 7672 MB [ComfyUI] [SubprocessWorker] call_method: nodes.load_model.LoadSALADModel.EXECUTE_NORMALIZED [ComfyUI] [SubprocessWorker] acquired lock, ensuring started... [ComfyUI] [ComfyUI-DepthAnythingV3] Health check: ping (timeout=5.0s)... [ComfyUI] [ComfyUI-DepthAnythingV3] Health check: failed [ComfyUI] [ComfyUI-DepthAnythingV3] Socket unhealthy, restarting worker... [ComfyUI] [ComfyUI-DepthAnythingV3] python: /root/.ce/.pixi/envs/depthanythingv3-nodes/bin/python [ComfyUI] [ComfyUI-DepthAnythingV3] sys_path sent to worker: ['/workspaces/DepthAnythingV3-1149/ComfyUI', '/workspaces/DepthAnythingV3-1149/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3', '/root/.ce/.pixi/envs/depthanythingv3-nodes/lib/python3.13/site-packages', '/workspaces/DepthAnythingV3-1149/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3'] [ComfyUI] [SubprocessWorker] is_pixi=True, python=/root/.ce/.pixi/envs/depthanythingv3-nodes/bin/python [ComfyUI] [SubprocessWorker] launching cmd=['/root/.pixi/bin/pixi', 'run', '--as-is', '--manifest-path', '/root/.ce/pixi.toml', '-e', 'depthanythingv3-nodes', 'python', '/tmp/comfyui_pvenv_c7dez6_l/persistent_worker.py', 'abstract://comfy_worker_47471f1f2f9d']... [ComfyUI] [SubprocessWorker] socket_addr=abstract://comfy_worker_47471f1f2f9d (abstract, no filesystem path) [ComfyUI] [worker:ComfyUI-DepthAnythingV3] ERROR: Failed to import comfy_kitchen, Error: No module named 'comfy_kitchen', fp8 and fp4 support will not be available. [ComfyUI] [SubprocessWorker] worker started/confirmed [ComfyUI] [SubprocessWorker] sending request via socket... [ComfyUI] [SubprocessWorker] call_id=25 sending call_method [ComfyUI] Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. [ComfyUI] [worker:ComfyUI-DepthAnythingV3] WARNING: Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. [ComfyUI] Frame 2 saved (frame_001.jpg, t=2.4s) [capture-periodic] freeze=2ms shot=142ms unfreeze=7ms saved=True Frame 3 saved (frame_002.jpg, t=4.6s) [capture-periodic] freeze=2ms shot=143ms unfreeze=6ms saved=True [ComfyUI] dino_salad.safetensors: 0%| | 0.00/352M [00:00...\n timeout=600.0,\n )\n', ' File "/workspaces/DepthAnythingV3-1149/.venv/lib/python3.13/site-packages/comfy_env/isolation/workers/subprocess.py", line 674, in call_method\n raise WorkerError(\n ...<2 lines>...\n )\n'], 'current_inputs': {'normalization_mode': ['Standard'], 'chunk_size': [30], 'overlap': [8], 'align_lib': ['auto'], 'align_method': ['sim3'], 'resize_method': ['resize'], 'invert_depth': [True], 'save_pointcloud': [False], 'sample_ratio': [0.015], 'conf_threshold_coef': [0.75], 'da3_model': ["{'model_path': '/workspaces/DepthAnythingV3-1149/ComfyUI/models/depthanything3/da3_large.safetensors', 'model_key': 'da3-large', 'dtype': 'bf16', 'attention': 'auto'}"], 'video': [''], 'salad_model': ["{'checkpoint_path': '/workspaces/DepthAnythingV3-1149/ComfyUI/models/salad/dino_salad.safetensors'}"]}, 'current_outputs': ['5', '3', '6', '2', '4'], 'timestamp': 1778846106492} [ComfyUI] File "/workspaces/DepthAnythingV3-1149/ComfyUI/execution.py", line 297, in process_inputs Status: FAILED Error: Workflow execution failed: {'prompt_id': 'd3813888-bfe2-473d-836b-4c5aa030194d', 'node_id': '6', 'node_type': 'DepthAnythingV3_Streaming', 'executed': ['5', '2', '3'], 'exception_message': 'Allocation on device 0 would exceed allowed memory. (out of memory)\nCurrently allocated : 4.84 GiB\nRequested : 286.44 MiB\nDevice limit : 7.49 GiB\nFree (according to CUDA): 117.44 MiB\nPyTorch limit (set by user-supplied memory fraction)\n : 17179869184.00 GiB\n\nWorker traceback:\nTraceback (most recent call last):\n File "/tmp/comfyui_pvenv_c7dez6_l/persistent_worker.py", line 1762, in main\n result = method(**inputs)\n File "/workspaces/DepthAnythingV3-1149/ComfyUI/comfy_api/latest/_io.py", line 1833, in EXECUTE_NORMALIZED\n to_return = cls.execute(*args, **kwargs)\n File "/workspaces/DepthAnythingV3-1149/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/streaming/node.py", line 249, in execute\n result = pipeline.run(normalized_images, pbar=pbar,\n salad_model=salad_nn_model, video_frames=images)\n File "/workspaces/DepthAnythingV3-1149/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/streaming/pipeline.py", line 277, in run\n result = self._process_chunk(normalized_images, start, end)\n File "/workspaces/DepthAnythingV3-1149/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/streaming/pipeline.py", line 330, in _process_chunk\n output = self.model(chunk_input)\n File "/root/.ce/.pixi/envs/depthanythingv3-nodes/lib/python3.13/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl\n return self._call_impl(*args, **kwargs)\n ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^\n File "/root/.ce/.pixi/envs/depthanythingv3-nodes/lib/python3.13/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl\n return forward_call(*args, **kwargs)\n File "/workspaces/DepthAnythingV3-1149/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/depth_anything_v3/model.py", line 1577, in forward\n output = self._process_depth_head(feats, H, W)\n File "/workspaces/DepthAnythingV3-1149/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/depth_anything_v3/model.py", line 1591, in _process_depth_head\n return self.head(feats, H, W, patch_start_idx=0)\n ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/root/.ce/.pixi/envs/depthanythingv3-nodes/lib/python3.13/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl\n return self._call_impl(*args, **kwargs)\n ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^\n File "/root/.ce/.pixi/envs/depthanythingv3-nodes/lib/python3.13/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl\n return forward_call(*args, **kwargs)\n File "/workspaces/DepthAnythingV3-1149/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/depth_anything_v3/model.py", line 1379, in forward\n self._forward_impl([feat[s0:s1] for feat in feats], H, W, patch_start_idx)\n ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/workspaces/DepthAnythingV3-1149/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/depth_anything_v3/model.py", line 1419, in _forward_impl\n fused_main = self._add_pos_embed(fused_main, W, H)\n File "/workspaces/DepthAnythingV3-1149/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/depth_anything_v3/model.py", line 1477, in _add_pos_embed\n pe = position_grid_to_embed(pe, x.shape[1]) * ratio\n ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~\ntorch.OutOfMemoryError: Allocation on device 0 would exceed allowed memory. (out of memory)\nCurrently allocated : 4.84 GiB\nRequested : 286.44 MiB\nDevice limit : 7.49 GiB\nFree (according to CUDA): 117.44 MiB\nPyTorch limit (set by user-supplied memory fraction)\n : 17179869184.00 GiB\n\n', 'exception_type': 'comfy_env.isolation.workers.base.WorkerError', 'traceback': [' File "/workspaces/DepthAnythingV3-1149/ComfyUI/execution.py", line 535, in execute\n output_data, output_ui, has_subgraph, has_pending_tasks = await get_output_data(prompt_id, unique_id, obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', ' File "/workspaces/DepthAnythingV3-1149/ComfyUI/execution.py", line 335, in get_output_data\n return_values = await _async_map_node_over_list(prompt_id, unique_id, obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', ' File "/workspaces/DepthAnythingV3-1149/ComfyUI/execution.py", line 309, in _async_map_node_over_list\n await process_inputs(input_dict, i)\n', ' File "/workspaces/DepthAnythingV3-1149/ComfyUI/execution.py", line 297, in process_inputs\n result = f(**inputs)\n', ' File "/workspaces/DepthAnythingV3-1149/.venv/lib/python3.13/site-packages/comfy_env/isolation/metadata.py", line 556, in proxy\n result = worker.call_method(\n module_name=mod,\n ...<4 lines>...\n timeout=600.0,\n )\n', ' File "/workspaces/DepthAnythingV3-1149/.venv/lib/python3.13/site-packages/comfy_env/isolation/workers/subprocess.py", line 674, in call_method\n raise WorkerError(\n ...<2 lines>...\n )\n'], 'current_inputs': {'normalization_mode': ['Standard'], 'chunk_size': [30], 'overlap': [8], 'align_lib': ['auto'], 'align_method': ['sim3'], 'resize_method': ['resize'], 'invert_depth': [True], 'save_pointcloud': [False], 'sample_ratio': [0.015], 'conf_threshold_coef': [0.75], 'da3_model': ["{'model_path': '/workspaces/DepthAnythingV3-1149/ComfyUI/models/depthanything3/da3_large.safetensors', 'model_key': 'da3-large', 'dtype': 'bf16', 'attention': 'auto'}"], 'video': [''], 'salad_model': ["{'checkpoint_path': '/workspaces/DepthAnythingV3-1149/ComfyUI/models/salad/dino_salad.safetensors'}"]}, 'current_outputs': ['5', '3', '6', '2', '4'], 'timestamp': 1778846106492} Details: Workflow: /node/ComfyUI-DepthAnythingV3/workflows/da3_streaming.json Node error: DepthAnythingV3_Streaming [ComfyUI] result = f(**inputs) [ComfyUI] File "/workspaces/DepthAnythingV3-1149/.venv/lib/python3.13/site-packages/comfy_env/isolation/metadata.py", line 556, in proxy [ComfyUI] result = worker.call_method( [ComfyUI] module_name=mod, [ComfyUI] ...<4 lines>... [ComfyUI] timeout=600.0, [ComfyUI] ) [ComfyUI] File "/workspaces/DepthAnythingV3-1149/.venv/lib/python3.13/site-packages/comfy_env/isolation/workers/subprocess.py", line 674, in call_method [ComfyUI] raise WorkerError( [ComfyUI] ...<2 lines>... [ComfyUI] ) [ComfyUI] comfy_env.isolation.workers.base.WorkerError: Allocation on device 0 would exceed allowed memory. (out of memory) [ComfyUI] Currently allocated : 4.84 GiB [ComfyUI] Requested : 286.44 MiB [ComfyUI] Device limit : 7.49 GiB [ComfyUI] Free (according to CUDA): 117.44 MiB [ComfyUI] PyTorch limit (set by user-supplied memory fraction) [ComfyUI] : 17179869184.00 GiB [ComfyUI] [ComfyUI] Worker traceback: [ComfyUI] Traceback (most recent call last): [ComfyUI] File "/tmp/comfyui_pvenv_c7dez6_l/persistent_worker.py", line 1762, in main [ComfyUI] result = method(**inputs) [ComfyUI] File "/workspaces/DepthAnythingV3-1149/ComfyUI/comfy_api/latest/_io.py", line 1833, in EXECUTE_NORMALIZED [ComfyUI] to_return = cls.execute(*args, **kwargs) [ComfyUI] File "/workspaces/DepthAnythingV3-1149/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/streaming/node.py", line 249, in execute [ComfyUI] result = pipeline.run(normalized_images, pbar=pbar, [ComfyUI] salad_model=salad_nn_model, video_frames=images) [ComfyUI] File "/workspaces/DepthAnythingV3-1149/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/streaming/pipeline.py", line 277, in run [ComfyUI] result = self._process_chunk(normalized_images, start, end) [ComfyUI] File "/workspaces/DepthAnythingV3-1149/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/streaming/pipeline.py", line 330, in _process_chunk [ComfyUI] output = self.model(chunk_input) [ComfyUI] File "/root/.ce/.pixi/envs/depthanythingv3-nodes/lib/python3.13/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl [ComfyUI] return self._call_impl(*args, **kwargs) [ComfyUI] ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^ [ComfyUI] File "/root/.ce/.pixi/envs/depthanythingv3-nodes/lib/python3.13/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl [ComfyUI] return forward_call(*args, **kwargs) [ComfyUI] File "/workspaces/DepthAnythingV3-1149/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/depth_anything_v3/model.py", line 1577, in forward [ComfyUI] output = self._process_depth_head(feats, H, W) [ComfyUI] File "/workspaces/DepthAnythingV3-1149/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/depth_anything_v3/model.py", line 1591, in _process_depth_head [ComfyUI] return self.head(feats, H, W, patch_start_idx=0) [ComfyUI] ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [ComfyUI] File "/root/.ce/.pixi/envs/depthanythingv3-nodes/lib/python3.13/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl [ComfyUI] return self._call_impl(*args, **kwargs) [ComfyUI] ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^ [ComfyUI] File "/root/.ce/.pixi/envs/depthanythingv3-nodes/lib/python3.13/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl [ComfyUI] return forward_call(*args, **kwargs) [ComfyUI] File "/workspaces/DepthAnythingV3-1149/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/depth_anything_v3/model.py", line 1379, in forward [ComfyUI] self._forward_impl([feat[s0:s1] for feat in feats], H, W, patch_start_idx) [ComfyUI] ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [ComfyUI] File "/workspaces/DepthAnythingV3-1149/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/depth_anything_v3/model.py", line 1419, in _forward_impl [ComfyUI] fused_main = self._add_pos_embed(fused_main, W, H) [ComfyUI] File "/workspaces/DepthAnythingV3-1149/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/depth_anything_v3/model.py", line 1477, in _add_pos_embed [ComfyUI] pe = position_grid_to_embed(pe, x.shape[1]) * ratio [ComfyUI] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~ [ComfyUI] torch.OutOfMemoryError: Allocation on device 0 would exceed allowed memory. (out of memory) [ComfyUI] Currently allocated : 4.84 GiB [ComfyUI] Requested : 286.44 MiB [ComfyUI] Device limit : 7.49 GiB [ComfyUI] Free (according to CUDA): 117.44 MiB [ComfyUI] PyTorch limit (set by user-supplied memory fraction) [ComfyUI] : 17179869184.00 GiB [ComfyUI] [ComfyUI] [ComfyUI] Prompt executed in 22.73 seconds