306 lines
46 KiB
Plaintext
306 lines
46 KiB
Plaintext
Capturing execution frames: da3_streaming.json
|
|
Frame 1 saved (frame_000.jpg, t=0.0s)
|
|
Validating workflow...
|
|
[ComfyUI] === /validate endpoint called ===
|
|
Queuing workflow for execution...
|
|
[ComfyUI] Received JSON with keys: ['prompt']
|
|
[ComfyUI] Prompt has 5 nodes
|
|
[ComfyUI] Node 2: DownloadAndLoadDepthAnythingV3Model
|
|
[ComfyUI] Inputs: {"model": "da3_large.safetensors", "precision": "auto", "attention": "auto"}
|
|
[ComfyUI] Node 3: LoadVideo
|
|
[ComfyUI] Inputs: {"file": "robot_unitree.mp4", "video-preview": ""}
|
|
[ComfyUI] Node 4: SaveVideo
|
|
[ComfyUI] Inputs: {"filename_prefix": "video/preview_da3", "format": "auto", "codec": "auto", "video": ["6", 0]}
|
|
[ComfyUI] Node 5: LoadSALADModel
|
|
[ComfyUI] Inputs: {}
|
|
[ComfyUI] [ComfyUI-DepthAnythingV3] python: /root/.ce/.pixi/envs/depthanythingv3-nodes/bin/python
|
|
[ComfyUI] Node 6: DepthAnythingV3_Streaming
|
|
[ComfyUI] [ComfyUI-DepthAnythingV3] sys_path sent to worker: ['/workspaces/DepthAnythingV3-1231/ComfyUI', '/workspaces/DepthAnythingV3-1231/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3', '/root/.ce/.pixi/envs/depthanythingv3-nodes/lib/python3.1/site-packages', '/workspaces/DepthAnythingV3-1231/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3']
|
|
[ComfyUI] Inputs: {"normalization_mode": "Standard", "chunk_size": 30, "overlap": 8, "align_lib": "auto", "align_method": "sim3", "resize_method": "resize", "invert_depth": true, "save_pointcloud": false, "sample_ratio
|
|
[ComfyUI] [SubprocessWorker] is_pixi=True, python=/root/.ce/.pixi/envs/depthanythingv3-nodes/bin/python
|
|
[ComfyUI] Generated prompt_id: ffe6fd91-1e21-46b9-864a-4cde5948a389
|
|
[ComfyUI] [SubprocessWorker] launching cmd=['/root/.pixi/bin/pixi', 'run', '--as-is', '--manifest-path', '/root/.ce/pixi.toml', '-e', 'depthanythingv3-nodes', 'python', '/tmp/comfyui_pvenv_c1iflygq/persistent_worker.py', 'abstract://comfy_worker_8bf0e56c5d2a']...
|
|
[ComfyUI] Calling execution.validate_prompt()...
|
|
[ComfyUI] [SubprocessWorker] socket_addr=abstract://comfy_worker_8bf0e56c5d2a (abstract, no filesystem path)
|
|
[ComfyUI] Validation result: valid=True
|
|
[ComfyUI] === Validation PASSED ===
|
|
[ComfyUI] got prompt
|
|
[ComfyUI] [comfy-env] Running LoadSALADModel:
|
|
[ComfyUI] [VRAM] Before LoadSALADModel: 24106 / 24124 MB
|
|
[ComfyUI] [SubprocessWorker] call_method: nodes.load_model.LoadSALADModel.EXECUTE_NORMALIZED
|
|
[ComfyUI] [SubprocessWorker] acquired lock, ensuring started...
|
|
[ComfyUI] [ComfyUI-DepthAnythingV3] Health check: ping (timeout=5.0s)...
|
|
[ComfyUI] [ComfyUI-DepthAnythingV3] Health check: failed
|
|
[ComfyUI] [ComfyUI-DepthAnythingV3] Socket unhealthy, restarting worker...
|
|
[ComfyUI] [worker:ComfyUI-DepthAnythingV3] ERROR: Failed to import comfy_kitchen, Error: No module named 'comfy_kitchen', fp8 and fp4 support will not be available.
|
|
[ComfyUI] [SubprocessWorker] worker started/confirmed
|
|
[ComfyUI] [SubprocessWorker] sending request via socket...
|
|
[ComfyUI] [SubprocessWorker] call_id=12 sending call_method
|
|
Frame 2 saved (frame_001.jpg, t=2.5s)
|
|
[capture-periodic] freeze=3ms shot=195ms unfreeze=7ms saved=True
|
|
[ComfyUI]
|
|
Frame 3 saved (frame_002.jpg, t=4.8s)
|
|
[capture-periodic] freeze=4ms shot=222ms unfreeze=7ms saved=True
|
|
[capture-loop] iter=50 t=6.3s state={'complete': False, 'error': None, 'executedCount': 0, 'wsState': 1} eval_ms=4 ram=22.7/31.2GB (73%) swap=22.1/32.0GB (69%) cpu=7% browser_rss=0.94GB
|
|
[capture-loop] iter=100 t=12.2s state={'complete': False, 'error': None, 'executedCount': 0, 'wsState': 1} eval_ms=4 ram=22.9/31.2GB (73%) swap=22.1/32.0GB (69%) cpu=6% browser_rss=0.93GB
|
|
[capture-loop] iter=150 t=18.0s state={'complete': False, 'error': None, 'executedCount': 0, 'wsState': 1} eval_ms=4 ram=22.9/31.2GB (74%) swap=22.1/32.0GB (69%) cpu=4% browser_rss=0.90GB
|
|
[capture-loop] iter=200 t=23.9s state={'complete': False, 'error': None, 'executedCount': 0, 'wsState': 1} eval_ms=4 ram=22.9/31.2GB (74%) swap=22.1/32.0GB (69%) cpu=6% browser_rss=0.90GB
|
|
[capture-loop] iter=250 t=29.9s state={'complete': False, 'error': None, 'executedCount': 0, 'wsState': 1} eval_ms=3 ram=23.0/31.2GB (74%) swap=22.1/32.0GB (69%) cpu=4% browser_rss=0.90GB
|
|
[capture-loop] iter=300 t=35.6s state={'complete': False, 'error': None, 'executedCount': 0, 'wsState': 1} eval_ms=4 ram=22.9/31.2GB (74%) swap=22.1/32.0GB (69%) cpu=4% browser_rss=0.90GB
|
|
[capture-loop] iter=350 t=41.5s state={'complete': False, 'error': None, 'executedCount': 0, 'wsState': 1} eval_ms=4 ram=22.9/31.2GB (73%) swap=22.1/32.0GB (69%) cpu=4% browser_rss=0.90GB
|
|
[capture-loop] iter=400 t=47.5s state={'complete': False, 'error': None, 'executedCount': 0, 'wsState': 1} eval_ms=4 ram=22.8/31.2GB (73%) swap=22.1/32.0GB (69%) cpu=5% browser_rss=0.90GB
|
|
[capture-loop] iter=450 t=53.2s state={'complete': False, 'error': None, 'executedCount': 0, 'wsState': 1} eval_ms=4 ram=23.0/31.2GB (74%) swap=22.1/32.0GB (69%) cpu=7% browser_rss=0.90GB
|
|
[capture-loop] iter=500 t=59.2s state={'complete': False, 'error': None, 'executedCount': 0, 'wsState': 1} eval_ms=3 ram=23.0/31.2GB (74%) swap=22.1/32.0GB (69%) cpu=5% browser_rss=0.90GB
|
|
[ComfyUI] dino_salad.safetensors: 0%| | 0.00/352M [00:00<?, ?B/s]
|
|
[ComfyUI] dino_salad.safetensors: 63%|██████▎ | 223M/352M [00:56<00:32, 3.96MB/s][SubprocessWorker] callback 'report_progress' from call_id=12
|
|
[ComfyUI]
|
|
[ComfyUI] dino_salad.safetensors: 100%|██████████| 352M/352M [00:56<00:00, 7.20MB/s][SubprocessWorker] callback 'report_progress' from call_id=12
|
|
[ComfyUI]
|
|
[ComfyUI] dino_salad.safetensors: 100%|██████████| 352M/352M [00:56<00:00, 6.23MB/s]
|
|
[ComfyUI] [SubprocessWorker] got response: ok
|
|
[ComfyUI] [comfy-env] LoadSALADModel done (59.41s): result: NodeOutput
|
|
[ComfyUI] [VRAM] After LoadSALADModel: 22614 / 24124 MB
|
|
[ComfyUI] [comfy-env] Running DownloadAndLoadDepthAnythingV3Model: model: 'da3_large.safetensors', precision: 'auto', attention: 'auto'
|
|
[ComfyUI] [VRAM] Before DownloadAndLoadDepthAnythingV3Model: 22614 / 24124 MB
|
|
[ComfyUI] [SubprocessWorker] call_method: nodes.load_model.DownloadAndLoadDepthAnythingV3Model.EXECUTE_NORMALIZED
|
|
[ComfyUI] [SubprocessWorker] acquired lock, ensuring started...
|
|
[ComfyUI] [ComfyUI-DepthAnythingV3] Health check: ping (timeout=5.0s)...
|
|
[ComfyUI] [ComfyUI-DepthAnythingV3] Health check: ok
|
|
[ComfyUI] [SubprocessWorker] worker started/confirmed
|
|
[ComfyUI] [SubprocessWorker] serializing kwargs to shm...
|
|
[ComfyUI] [SubprocessWorker] created 0 shm blocks
|
|
[ComfyUI] [SubprocessWorker] sending request via socket...
|
|
[ComfyUI] [SubprocessWorker] call_id=13 sending call_method
|
|
[ComfyUI] [SubprocessWorker] got response: ok
|
|
[ComfyUI] [comfy-env] DownloadAndLoadDepthAnythingV3Model done (0.10s): result: NodeOutput
|
|
[ComfyUI] [VRAM] After DownloadAndLoadDepthAnythingV3Model: 22614 / 24124 MB
|
|
[ComfyUI] [comfy-env] Running DepthAnythingV3_Streaming: normalization_mode: 'Standard', chunk_size: 30, overlap: 8, align_lib: 'auto', align_method: 'sim3', resize_method: 'resize', invert_depth: True, save_pointcloud: False, sample_ratio: 0.015, conf_threshold_coef: 0.75, da3_model: dict, video: VideoFromFile, salad_model: dict
|
|
[ComfyUI] [VRAM] Before DepthAnythingV3_Streaming: 22614 / 24124 MB
|
|
[ComfyUI] [SubprocessWorker] call_method: nodes.streaming.node.DepthAnythingV3_Streaming.EXECUTE_NORMALIZED
|
|
[ComfyUI] [SubprocessWorker] acquired lock, ensuring started...
|
|
[ComfyUI] [ComfyUI-DepthAnythingV3] Health check: ping (timeout=5.0s)...
|
|
[ComfyUI] [ComfyUI-DepthAnythingV3] Health check: ok
|
|
[ComfyUI] [SubprocessWorker] worker started/confirmed
|
|
[ComfyUI] [SubprocessWorker] serializing kwargs to shm...
|
|
[ComfyUI] [SubprocessWorker] created 1 shm blocks
|
|
[ComfyUI] [SubprocessWorker] sending request via socket...
|
|
[ComfyUI] [SubprocessWorker] call_id=14 sending call_method
|
|
[ComfyUI] [worker:ComfyUI-DepthAnythingV3]
|
|
[ComfyUI] === WATCHDOG TICK 1 ===
|
|
[ComfyUI] [worker:ComfyUI-DepthAnythingV3] Thread 0x0000782d43844640 (most recent call first):
|
|
[ComfyUI] File "/root/.ce/.pixi/envs/depthanythingv3-nodes/lib/python3.11/threading.py", line 331 in wait
|
|
[ComfyUI] File "/root/.ce/.pixi/envs/depthanythingv3-nodes/lib/python3.11/threading.py", line 629 in wait
|
|
[ComfyUI] File "/root/.ce/.pixi/envs/depthanythingv3-nodes/lib/python3.11/site-packages/tqdm/_monitor.py", line 60 in run
|
|
[ComfyUI] File "/root/.ce/.pixi/envs/depthanythingv3-nodes/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
|
|
[ComfyUI] File "/root/.ce/.pixi/envs/depthanythingv3-nodes/lib/python3.11/threading.py", line 1002 in _bootstrap
|
|
[ComfyUI]
|
|
[ComfyUI] Thread 0x0000782ef9e76640 (most recent call first):
|
|
[ComfyUI] File "/tmp/comfyui_pvenv_c1iflygq/persistent_worker.py", line 106 in _vram_poller
|
|
[ComfyUI] File "/root/.ce/.pixi/envs/depthanythingv3-nodes/lib/python3.11/threading.py", line 982 in run
|
|
[ComfyUI] File "/root/.ce/.pixi/envs/depthanythingv3-nodes/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
|
|
[ComfyUI] File "/root/.ce/.pixi/envs/depthanythingv3-nodes/lib/python3.11/threading.py", line 1002 in _bootstrap
|
|
[ComfyUI]
|
|
[ComfyUI] Current thread 0x0000782efa67b640 (most recent call first):
|
|
[ComfyUI] File "/tmp/comfyui_pvenv_c1iflygq/persistent_worker.py", line 52 in _watchdog
|
|
[ComfyUI] File "/root/.ce/.pixi/envs/depthanythingv3-nodes/lib/python3.11/threading.py", line 982 in run
|
|
[ComfyUI] File "/root/.ce/.pixi/envs/depthanythingv3-nodes/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
|
|
[ComfyUI] File "/root/.ce/.pixi/envs/depthanythingv3-nodes/lib/python3.11/threading.py", line 1002 in _bootstrap
|
|
[ComfyUI]
|
|
[ComfyUI] Thread 0x0000782efac9b740 (most recent call first):
|
|
[ComfyUI] File "/workspaces/DepthAnythingV3-1231/ComfyUI/comfy_api/latest/_input_impl/video_types.py", line 308 in get_components_internal
|
|
[ComfyUI] File "/workspaces/DepthAnythingV3-1231/ComfyUI/comfy_api/latest/_input_impl/video_types.py", line 366 in get_components
|
|
[ComfyUI] File "/workspaces/DepthAnythingV3-1231/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/streaming/node.py", line 170 in execute
|
|
[ComfyUI] File "/workspaces/DepthAnythingV3-1231/ComfyUI/comfy_api/latest/_io.py", line 1833 in EXECUTE_NORMALIZED
|
|
[ComfyUI] File "/tmp/comfyui_pvenv_c1iflygq/persistent_worker.py", line 1762 in main
|
|
[ComfyUI] File "/tmp/comfyui_pvenv_c1iflygq/persistent_worker.py", line 1807 in <module>
|
|
[ComfyUI]
|
|
[ComfyUI] [worker:ComfyUI-DepthAnythingV3] === END ===
|
|
[ComfyUI]
|
|
Frame 4 saved (frame_003.jpg, t=60.6s)
|
|
[capture-periodic] freeze=2ms shot=144ms unfreeze=6ms saved=True
|
|
[ComfyUI] [worker:ComfyUI-DepthAnythingV3] WARNING: No comfy kitchen, using old apply_rope functions.
|
|
[capture-loop] iter=550 t=64.9s state={'complete': False, 'error': None, 'executedCount': 0, 'wsState': 1} eval_ms=4 ram=24.4/31.2GB (78%) swap=22.9/32.0GB (72%) cpu=10% browser_rss=0.91GB
|
|
[ComfyUI] [SubprocessWorker] callback 'request_vram_budget' from call_id=14
|
|
[ComfyUI] [comfy-env] VRAM request: 1.17GB, free before eviction: 1.58GB
|
|
[ComfyUI] [comfy-env] VRAM after eviction: 1.58GB
|
|
[ComfyUI] [worker:ComfyUI-DepthAnythingV3] [VRAM] +270MB (now 270MB) reserved=288MB peak=270MB
|
|
[capture-loop] iter=600 t=70.9s state={'complete': False, 'error': None, 'executedCount': 0, 'wsState': 1} eval_ms=4 ram=24.3/31.2GB (78%) swap=25.7/32.0GB (80%) cpu=11% browser_rss=0.85GB
|
|
[ComfyUI] [SubprocessWorker] callback 'report_progress' from call_id=14
|
|
[ComfyUI] [worker:ComfyUI-DepthAnythingV3] [VRAM] +278MB (now 549MB) reserved=576MB peak=549MB
|
|
[ComfyUI] [worker:ComfyUI-DepthAnythingV3] ERROR: Error running sage attention: Allocation on device 0 would exceed allowed memory. (out of memory)
|
|
[ComfyUI] Currently allocated : 1.24 GiB
|
|
[ComfyUI] Requested : 175.43 MiB
|
|
[ComfyUI] Device limit : 23.56 GiB
|
|
[ComfyUI] Free (according to CUDA): 22.25 MiB
|
|
[ComfyUI] PyTorch limit (set by user-supplied memory fraction)
|
|
[ComfyUI] : 17179869184.00 GiB, using pytorch attention instead.
|
|
[ComfyUI] [SubprocessWorker] got response: error
|
|
[ComfyUI] !!! Exception during processing !!! Allocation on device 0 would exceed allowed memory. (out of memory)
|
|
[ComfyUI] Currently allocated : 1.24 GiB
|
|
[ComfyUI] Requested : 175.43 MiB
|
|
[ComfyUI] Device limit : 23.56 GiB
|
|
[ComfyUI] Free (according to CUDA): 22.25 MiB
|
|
[ComfyUI] PyTorch limit (set by user-supplied memory fraction)
|
|
[ComfyUI] : 17179869184.00 GiB
|
|
[ComfyUI]
|
|
[ComfyUI] Worker traceback:
|
|
[ComfyUI] Traceback (most recent call last):
|
|
[ComfyUI] File "/tmp/comfyui_pvenv_c1iflygq/persistent_worker.py", line 1762, in main
|
|
[ComfyUI] result = method(**inputs)
|
|
[ComfyUI] ^^^^^^^^^^^^^^^^
|
|
[ComfyUI] File "/workspaces/DepthAnythingV3-1231/ComfyUI/comfy_api/latest/_io.py", line 1833, in EXECUTE_NORMALIZED
|
|
[ComfyUI] to_return = cls.execute(*args, **kwargs)
|
|
[ComfyUI] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
[ComfyUI] File "/workspaces/DepthAnythingV3-1231/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/streaming/node.py", line 249, in execute
|
|
[ComfyUI] result = pipeline.run(normalized_images, pbar=pbar,
|
|
[ComfyUI] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
[ComfyUI] File "/workspaces/DepthAnythingV3-1231/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/streaming/pipeline.py", line 277, in run
|
|
[ComfyUI] result = self._process_chunk(normalized_images, start, end)
|
|
[ComfyUI] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
[ComfyUI] File "/workspaces/DepthAnythingV3-1231/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/streaming/pipeline.py", line 330, in _process_chunk
|
|
[ComfyUI] output = self.model(chunk_input)
|
|
[ComfyUI] ^^^^^^^^^^^^^^^^^^^^^^^
|
|
[ComfyUI] File "/root/.ce/.pixi/envs/depthanythingv3-nodes/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
|
|
[ComfyUI] return self._call_impl(*args, **kwargs)
|
|
[ComfyUI] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
[ComfyUI] File "/root/.ce/.pixi/envs/depthanythingv3-nodes/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl
|
|
[ComfyUI] return forward_call(*args, **kwargs)
|
|
[ComfyUI] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
[ComfyUI] File "/workspaces/DepthAnythingV3-1231/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/depth_anything_v3/model.py", line 1560, in forward
|
|
[ComfyUI] feats, aux_feats = self.backbone(
|
|
[ComfyUI] ^^^^^^^^^^^^^^
|
|
[ComfyUI] File "/root/.ce/.pixi/envs/depthanythingv3-nodes/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
|
|
[ComfyUI] return self._call_impl(*args, **kwargs)
|
|
[ComfyUI] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
[ComfyUI] File "/root/.ce/.pixi/envs/depthanythingv3-nodes/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl
|
|
[ComfyUI] return forward_call(*args, **kwargs)
|
|
[ComfyUI] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
[ComfyUI] File "/workspaces/DepthAnythingV3-1231/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/depth_anything_v3/model.py", line 891, in forward
|
|
[ComfyUI] return self.pretrained.get_intermediate_layers(
|
|
[ComfyUI] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
[ComfyUI] File "/workspaces/DepthAnythingV3-1231/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/depth_anything_v3/model.py", line 800, in get_intermediate_layers
|
|
[ComfyUI] outputs, aux_outputs = self._get_intermediate_layers_not_chunked(
|
|
[ComfyUI] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
[ComfyUI] File "/workspaces/DepthAnythingV3-1231/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/depth_anything_v3/model.py", line 750, in _get_intermediate_layers_not_chunked
|
|
[ComfyUI] x = self.process_attention(x, blk, "local", pos=l_pos)
|
|
[ComfyUI] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
[ComfyUI] File "/workspaces/DepthAnythingV3-1231/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/depth_anything_v3/model.py", line 791, in process_attention
|
|
[ComfyUI] x = block(x, pos=pos, attn_mask=attn_mask)
|
|
[ComfyUI] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
[ComfyUI] File "/root/.ce/.pixi/envs/depthanythingv3-nodes/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
|
|
[ComfyUI] return self._call_impl(*args, **kwargs)
|
|
[ComfyUI] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
[ComfyUI] File "/root/.ce/.pixi/envs/depthanythingv3-nodes/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl
|
|
[ComfyUI] return forward_call(*args, **kwargs)
|
|
[ComfyUI] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
[ComfyUI] File "/workspaces/DepthAnythingV3-1231/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/depth_anything_v3/model.py", line 521, in forward
|
|
[ComfyUI] x = x + self.drop_path1(attn_residual_func(x, pos=pos, attn_mask=attn_mask))
|
|
[ComfyUI] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
[ComfyUI] File "/workspaces/DepthAnythingV3-1231/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/depth_anything_v3/model.py", line 516, in attn_residual_func
|
|
[ComfyUI] return self.ls1(self.attn(self.norm1(x), pos=pos, attn_mask=attn_mask))
|
|
[ComfyUI] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
[ComfyUI] File "/root/.ce/.pixi/envs/depthanythingv3-nodes/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
|
|
[ComfyUI] return self._call_impl(*args, **kwargs)
|
|
[ComfyUI] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
[ComfyUI] File "/root/.ce/.pixi/envs/depthanythingv3-nodes/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl
|
|
[ComfyUI] return forward_call(*args, **kwargs)
|
|
[ComfyUI] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
[ComfyUI] File "/workspaces/DepthAnythingV3-1231/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/depth_anything_v3/model.py", line 453, in forward
|
|
[ComfyUI] x = optimized_attention_for_device(q.device)(q, k, v, heads=self.num_heads, skip_reshape=True)
|
|
[ComfyUI] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
[ComfyUI] File "/workspaces/DepthAnythingV3-1231/ComfyUI/comfy/ldm/modules/attention.py", line 139, in wrapper
|
|
[ComfyUI] return func(*args, **kwargs)
|
|
[ComfyUI] ^^^^^^^^^^^^^^^^^^^^^
|
|
[ComfyUI] File "/workspaces/DepthAnythingV3-1231/ComfyUI/comfy/ldm/modules/attention.py", line 579, in attention_sage
|
|
[ComfyUI] return attention_pytorch(q, k, v, heads, mask=mask, skip_reshape=True, skip_output_reshape=skip_output_reshape, **kwargs)
|
|
[ComfyUI] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
[ComfyUI] File "/workspaces/DepthAnythingV3-1231/ComfyUI/comfy/ldm/modules/attention.py", line 139, in wrapper
|
|
[ComfyUI] return func(*args, **kwargs)
|
|
[ComfyUI] ^^^^^^^^^^^^^^^^^^^^^
|
|
[ComfyUI] File "/workspaces/DepthAnythingV3-1231/ComfyUI/comfy/ldm/modules/attention.py", line 520, in attention_pytorch
|
|
[ComfyUI] out = comfy.ops.scaled_dot_product_attention(q, k, v, attn_mask=mask, dropout_p=0.0, is_causal=False, **sdpa_extra)
|
|
[ComfyUI] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
[ComfyUI] File "/workspaces/DepthAnythingV3-1231/ComfyUI/comfy/ops.py", line 39, in scaled_dot_product_attention
|
|
[ComfyUI] return torch.nn.functional.scaled_dot_product_attention(q, k, v, *args, **kwargs)
|
|
[ComfyUI] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
[ComfyUI] torch.OutOfMemoryError: Allocation on device 0 would exceed allowed memory. (out of memory)
|
|
[ComfyUI] Currently allocated : 1.24 GiB
|
|
[ComfyUI] Requested : 175.43 MiB
|
|
[ComfyUI] Device limit : 23.56 GiB
|
|
[ComfyUI] Free (according to CUDA): 22.25 MiB
|
|
[ComfyUI] PyTorch limit (set by user-supplied memory fraction)
|
|
[ComfyUI] : 17179869184.00 GiB
|
|
[ComfyUI]
|
|
[ComfyUI] Traceback (most recent call last):
|
|
[ComfyUI] File "/workspaces/DepthAnythingV3-1231/ComfyUI/execution.py", line 535, in execute
|
|
[ComfyUI] output_data, output_ui, has_subgraph, has_pending_tasks = await get_output_data(prompt_id, unique_id, obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)
|
|
[ComfyUI] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
[ComfyUI] File "/workspaces/DepthAnythingV3-1231/ComfyUI/execution.py", line 335, in get_output_data
|
|
[ComfyUI] return_values = await _async_map_node_over_list(prompt_id, unique_id, obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)
|
|
[ComfyUI] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
[ComfyUI] File "/workspaces/DepthAnythingV3-1231/ComfyUI/execution.py", line 309, in _async_map_node_over_list
|
|
[ComfyUI] await process_inputs(input_dict, i)
|
|
[ComfyUI] File "/workspaces/DepthAnythingV3-1231/ComfyUI/execution.py", line 297, in process_inputs
|
|
[ComfyUI] result = f(**inputs)
|
|
[ComfyUI] ^^^^^^^^^^^
|
|
[ComfyUI] File "/workspaces/DepthAnythingV3-1231/.venv/lib/python3.11/site-packages/comfy_env/isolation/metadata.py", line 556, in proxy
|
|
[ComfyUI] result = worker.call_method(
|
|
[ComfyUI] ^^^^^^^^^^^^^^^^^^^
|
|
[ComfyUI] File "/workspaces/DepthAnythingV3-1231/.venv/lib/python3.11/site-packages/comfy_env/isolation/workers/subprocess.py", line 674, in call_method
|
|
[ComfyUI] raise WorkerError(
|
|
[ComfyUI] comfy_env.isolation.workers.base.WorkerError: Allocation on device 0 would exceed allowed memory. (out of memory)
|
|
[ComfyUI] Currently allocated : 1.24 GiB
|
|
[ComfyUI] Requested : 175.43 MiB
|
|
[ComfyUI] Device limit : 23.56 GiB
|
|
[ComfyUI] Free (according to CUDA): 22.25 MiB
|
|
[ComfyUI] PyTorch limit (set by user-supplied memory fraction)
|
|
[ComfyUI] : 17179869184.00 GiB
|
|
[ComfyUI]
|
|
[ComfyUI] Worker traceback:
|
|
[ComfyUI] Traceback (most recent call last):
|
|
[ComfyUI] File "/tmp/comfyui_pvenv_c1iflygq/persistent_worker.py", line 1762, in main
|
|
[ComfyUI] result = method(**inputs)
|
|
[ComfyUI] ^^^^^^^^^^^^^^^^
|
|
[ComfyUI] File "/workspaces/DepthAnythingV3-1231/ComfyUI/comfy_api/latest/_io.py", line 1833, in EXECUTE_NORMALIZED
|
|
[ComfyUI] to_return = cls.execute(*args, **kwargs)
|
|
[ComfyUI] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
[ComfyUI] File "/workspaces/DepthAnythingV3-1231/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/streaming/node.py", line 249, in execute
|
|
[ComfyUI] result = pipeline.run(normalized_images, pbar=pbar,
|
|
[ComfyUI] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
[ComfyUI] File "/workspaces/DepthAnythingV3-1231/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/streaming/pipeline.py", line 277, in run
|
|
[ComfyUI] result = self._process_chunk(normalized_images, start, end)
|
|
[ComfyUI] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
[ComfyUI] File "/workspaces/DepthAnythingV3-1231/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/streaming/pipeline.py", line 330, in _process_chunk
|
|
[ComfyUI] output = self.model(chunk_input)
|
|
[ComfyUI] ^^^^^^^^^^^^^^^^^^^^^^^
|
|
[ComfyUI] File "/root/.ce/.pixi/envs/depthanythingv3-nodes/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
|
|
[ComfyUI] return self._call_impl(*args, **kwargs)
|
|
[ComfyUI] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
[ComfyUI] File "/root/.ce/.pixi/envs/depthanythingv3-nodes/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl
|
|
[ComfyUI] return forward_call(*args, **kwargs)
|
|
[ComfyUI] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
[ComfyUI] File "/workspaces/DepthAnythingV3-1231/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/depth_anything_v3/model.py", line 1560, in forward
|
|
[ComfyUI] feats, aux_feats = self.backbone(
|
|
[ComfyUI] ^^^^^^^^^^^^^^
|
|
[ComfyUI] File "/root/.ce/.pixi/envs/depthanythingv3-nodes/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
|
|
[ComfyUI] return self._call_impl(*args, **kwargs)
|
|
[ComfyUI] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
[ComfyUI] File "/root/.ce/.pixi/envs/depthanythingv3-nodes/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl
|
|
Execution complete (t=73.0s)
|
|
[ComfyUI] return forward_call(*args, **kwargs)
|
|
Execution error: {'prompt_id': 'd221db26-2dc6-46c6-bcaf-0363bb5f4cb0', 'node_id': '6', 'node_type': 'DepthAnythingV3_Streaming', 'executed': ['3', '5', '2'], 'exception_message': 'Allocation on device 0 would exceed allowed memory. (out of memory)\nCurrently allocated : 1.24 GiB\nRequested : 175.43 MiB\nDevice limit : 23.56 GiB\nFree (according to CUDA): 22.25 MiB\nPyTorch limit (set by user-supplied memory fraction)\n : 17179869184.00 GiB\n\nWorker traceback:\nTraceback (most recent call last):\n File "/tmp/comfyui_pvenv_c1iflygq/persistent_worker.py", line 1762, in main\n result = method(**inputs)\n ^^^^^^^^^^^^^^^^\n File "/workspaces/DepthAnythingV3-1231/ComfyUI/comfy_api/latest/_io.py", line 1833, in EXECUTE_NORMALIZED\n to_return = cls.execute(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/workspaces/DepthAnythingV3-1231/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/streaming/node.py", line 249, in execute\n result = pipeline.run(normalized_images, pbar=pbar,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/workspaces/DepthAnythingV3-1231/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/streaming/pipeline.py", line 277, in run\n result = self._process_chunk(normalized_images, start, end)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/workspaces/DepthAnythingV3-1231/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/streaming/pipeline.py", line 330, in _process_chunk\n output = self.model(chunk_input)\n ^^^^^^^^^^^^^^^^^^^^^^^\n File "/root/.ce/.pixi/envs/depthanythingv3-nodes/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl\n return self._call_impl(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/root/.ce/.pixi/envs/depthanythingv3-nodes/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl\n return forward_call(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/workspaces/DepthAnythingV3-1231/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/depth_anything_v3/model.py", line 1560, in forward\n feats, aux_feats = self.backbone(\n ^^^^^^^^^^^^^^\n File "/root/.ce/.pixi/envs/depthanythingv3-nodes/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl\n return self._call_impl(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/root/.ce/.pixi/envs/depthanythingv3-nodes/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl\n return forward_call(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/workspaces/DepthAnythingV3-1231/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/depth_anything_v3/model.py", line 891, in forward\n return self.pretrained.get_intermediate_layers(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/workspaces/DepthAnythingV3-1231/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/depth_anything_v3/model.py", line 800, in get_intermediate_layers\n outputs, aux_outputs = self._get_intermediate_layers_not_chunked(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/workspaces/DepthAnythingV3-1231/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/depth_anything_v3/model.py", line 750, in _get_intermediate_layers_not_chunked\n x = self.process_attention(x, blk, "local", pos=l_pos)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/workspaces/DepthAnythingV3-1231/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/depth_anything_v3/model.py", line 791, in process_attention\n x = block(x, pos=pos, attn_mask=attn_mask)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/root/.ce/.pixi/envs/depthanythingv3-nodes/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl\n return self._call_impl(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/root/.ce/.pixi/envs/depthanythingv3-nodes/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl\n return forward_call(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/workspaces/DepthAnythingV3-1231/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/depth_anything_v3/model.py", line 521, in forward\n x = x + self.drop_path1(attn_residual_func(x, pos=pos, attn_mask=attn_mask))\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/workspaces/DepthAnythingV3-1231/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/depth_anything_v3/model.py", line 516, in attn_residual_func\n return self.ls1(self.attn(self.norm1(x), pos=pos, attn_mask=attn_mask))\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/root/.ce/.pixi/envs/depthanythingv3-nodes/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl\n return self._call_impl(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/root/.ce/.pixi/envs/depthanythingv3-nodes/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl\n return forward_call(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/workspaces/DepthAnythingV3-1231/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/depth_anything_v3/model.py", line 453, in forward\n x = optimized_attention_for_device(q.device)(q, k, v, heads=self.num_heads, skip_reshape=True)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/workspaces/DepthAnythingV3-1231/ComfyUI/comfy/ldm/modules/attention.py", line 139, in wrapper\n return func(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^\n File "/workspaces/DepthAnythingV3-1231/ComfyUI/comfy/ldm/modules/attention.py", line 579, in attention_sage\n return attention_pytorch(q, k, v, heads, mask=mask, skip_reshape=True, skip_output_reshape=skip_output_reshape, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/workspaces/DepthAnythingV3-1231/ComfyUI/comfy/ldm/modules/attention.py", line 139, in wrapper\n return func(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^\n File "/workspaces/DepthAnythingV3-1231/ComfyUI/comfy/ldm/modules/attention.py", line 520, in attention_pytorch\n out = comfy.ops.scaled_dot_product_attention(q, k, v, attn_mask=mask, dropout_p=0.0, is_causal=False, **sdpa_extra)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/workspaces/DepthAnythingV3-1231/ComfyUI/comfy/ops.py", line 39, in scaled_dot_product_attention\n return torch.nn.functional.scaled_dot_product_attention(q, k, v, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\ntorch.OutOfMemoryError: Allocation on device 0 would exceed allowed memory. (out of memory)\nCurrently allocated : 1.24 GiB\nRequested : 175.43 MiB\nDevice limit : 23.56 GiB\nFree (according to CUDA): 22.25 MiB\nPyTorch limit (set by user-supplied memory fraction)\n : 17179869184.00 GiB\n\n', 'exception_type': 'comfy_env.isolation.workers.base.WorkerError', 'traceback': [' File "/workspaces/DepthAnythingV3-1231/ComfyUI/execution.py", line 535, in execute\n output_data, output_ui, has_subgraph, has_pending_tasks = await get_output_data(prompt_id, unique_id, obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', ' File "/workspaces/DepthAnythingV3-1231/ComfyUI/execution.py", line 335, in get_output_data\n return_values = await _async_map_node_over_list(prompt_id, unique_id, obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', ' File "/workspaces/DepthAnythingV3-1231/ComfyUI/execution.py", line 309, in _async_map_node_over_list\n await process_inputs(input_dict, i)\n', ' File "/workspaces/DepthAnythingV3-1231/ComfyUI/execution.py", line 297, in process_inputs\n result = f(**inputs)\n ^^^^^^^^^^^\n', ' File "/workspaces/DepthAnythingV3-1231/.venv/lib/python3.11/site-packages/comfy_env/isolation/metadata.py", line 556, in proxy\n result = worker.call_method(\n ^^^^^^^^^^^^^^^^^^^\n', ' File "/workspaces/DepthAnythingV3-1231/.venv/lib/python3.11/site-packages/comfy_env/isolation/workers/subprocess.py", line 674, in call_method\n raise WorkerError(\n'], 'current_inputs': {'normalization_mode': ['Standard'], 'chunk_size': [30], 'overlap': [8], 'align_lib': ['auto'], 'align_method': ['sim3'], 'resize_method': ['resize'], 'invert_depth': [True], 'save_pointcloud': [False], 'sample_ratio': [0.015], 'conf_threshold_coef': [0.75], 'da3_model': ["{'model_path': '/workspaces/DepthAnythingV3-1231/ComfyUI/models/depthanything3/da3_large.safetensors', 'model_key': 'da3-large', 'dtype': 'bf16', 'attention': 'auto'}"], 'video': ['<comfy_api.latest._input_impl.video_types.VideoFromFile object at 0x7b7a6df75510>'], 'salad_model': ["{'checkpoint_path': '/workspaces/DepthAnythingV3-1231/ComfyUI/models/salad/dino_salad.safetensors'}"]}, 'current_outputs': ['6', '3', '2', '5', '4'], 'timestamp': 1778676069429}
|
|
[ComfyUI] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
Status: FAILED
|
|
Error: Workflow execution failed: {'prompt_id': 'd221db26-2dc6-46c6-bcaf-0363bb5f4cb0', 'node_id': '6', 'node_type': 'DepthAnythingV3_Streaming', 'executed': ['3', '5', '2'], 'exception_message': 'Allocation on device 0 would exceed allowed memory. (out of memory)\nCurrently allocated : 1.24 GiB\nRequested : 175.43 MiB\nDevice limit : 23.56 GiB\nFree (according to CUDA): 22.25 MiB\nPyTorch limit (set by user-supplied memory fraction)\n : 17179869184.00 GiB\n\nWorker traceback:\nTraceback (most recent call last):\n File "/tmp/comfyui_pvenv_c1iflygq/persistent_worker.py", line 1762, in main\n result = method(**inputs)\n ^^^^^^^^^^^^^^^^\n File "/workspaces/DepthAnythingV3-1231/ComfyUI/comfy_api/latest/_io.py", line 1833, in EXECUTE_NORMALIZED\n to_return = cls.execute(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/workspaces/DepthAnythingV3-1231/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/streaming/node.py", line 249, in execute\n result = pipeline.run(normalized_images, pbar=pbar,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/workspaces/DepthAnythingV3-1231/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/streaming/pipeline.py", line 277, in run\n result = self._process_chunk(normalized_images, start, end)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/workspaces/DepthAnythingV3-1231/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/streaming/pipeline.py", line 330, in _process_chunk\n output = self.model(chunk_input)\n ^^^^^^^^^^^^^^^^^^^^^^^\n File "/root/.ce/.pixi/envs/depthanythingv3-nodes/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl\n return self._call_impl(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/root/.ce/.pixi/envs/depthanythingv3-nodes/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl\n return forward_call(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/workspaces/DepthAnythingV3-1231/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/depth_anything_v3/model.py", line 1560, in forward\n feats, aux_feats = self.backbone(\n ^^^^^^^^^^^^^^\n File "/root/.ce/.pixi/envs/depthanythingv3-nodes/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl\n return self._call_impl(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/root/.ce/.pixi/envs/depthanythingv3-nodes/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl\n return forward_call(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/workspaces/DepthAnythingV3-1231/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/depth_anything_v3/model.py", line 891, in forward\n return self.pretrained.get_intermediate_layers(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/workspaces/DepthAnythingV3-1231/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/depth_anything_v3/model.py", line 800, in get_intermediate_layers\n outputs, aux_outputs = self._get_intermediate_layers_not_chunked(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/workspaces/DepthAnythingV3-1231/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/depth_anything_v3/model.py", line 750, in _get_intermediate_layers_not_chunked\n x = self.process_attention(x, blk, "local", pos=l_pos)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/workspaces/DepthAnythingV3-1231/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/depth_anything_v3/model.py", line 791, in process_attention\n x = block(x, pos=pos, attn_mask=attn_mask)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/root/.ce/.pixi/envs/depthanythingv3-nodes/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl\n return self._call_impl(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/root/.ce/.pixi/envs/depthanythingv3-nodes/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl\n return forward_call(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/workspaces/DepthAnythingV3-1231/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/depth_anything_v3/model.py", line 521, in forward\n x = x + self.drop_path1(attn_residual_func(x, pos=pos, attn_mask=attn_mask))\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/workspaces/DepthAnythingV3-1231/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/depth_anything_v3/model.py", line 516, in attn_residual_func\n return self.ls1(self.attn(self.norm1(x), pos=pos, attn_mask=attn_mask))\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/root/.ce/.pixi/envs/depthanythingv3-nodes/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl\n return self._call_impl(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/root/.ce/.pixi/envs/depthanythingv3-nodes/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl\n return forward_call(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/workspaces/DepthAnythingV3-1231/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/depth_anything_v3/model.py", line 453, in forward\n x = optimized_attention_for_device(q.device)(q, k, v, heads=self.num_heads, skip_reshape=True)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/workspaces/DepthAnythingV3-1231/ComfyUI/comfy/ldm/modules/attention.py", line 139, in wrapper\n return func(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^\n File "/workspaces/DepthAnythingV3-1231/ComfyUI/comfy/ldm/modules/attention.py", line 579, in attention_sage\n return attention_pytorch(q, k, v, heads, mask=mask, skip_reshape=True, skip_output_reshape=skip_output_reshape, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/workspaces/DepthAnythingV3-1231/ComfyUI/comfy/ldm/modules/attention.py", line 139, in wrapper\n return func(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^\n File "/workspaces/DepthAnythingV3-1231/ComfyUI/comfy/ldm/modules/attention.py", line 520, in attention_pytorch\n out = comfy.ops.scaled_dot_product_attention(q, k, v, attn_mask=mask, dropout_p=0.0, is_causal=False, **sdpa_extra)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/workspaces/DepthAnythingV3-1231/ComfyUI/comfy/ops.py", line 39, in scaled_dot_product_attention\n return torch.nn.functional.scaled_dot_product_attention(q, k, v, *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\ntorch.OutOfMemoryError: Allocation on device 0 would exceed allowed memory. (out of memory)\nCurrently allocated : 1.24 GiB\nRequested : 175.43 MiB\nDevice limit : 23.56 GiB\nFree (according to CUDA): 22.25 MiB\nPyTorch limit (set by user-supplied memory fraction)\n : 17179869184.00 GiB\n\n', 'exception_type': 'comfy_env.isolation.workers.base.WorkerError', 'traceback': [' File "/workspaces/DepthAnythingV3-1231/ComfyUI/execution.py", line 535, in execute\n output_data, output_ui, has_subgraph, has_pending_tasks = await get_output_data(prompt_id, unique_id, obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', ' File "/workspaces/DepthAnythingV3-1231/ComfyUI/execution.py", line 335, in get_output_data\n return_values = await _async_map_node_over_list(prompt_id, unique_id, obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', ' File "/workspaces/DepthAnythingV3-1231/ComfyUI/execution.py", line 309, in _async_map_node_over_list\n await process_inputs(input_dict, i)\n', ' File "/workspaces/DepthAnythingV3-1231/ComfyUI/execution.py", line 297, in process_inputs\n result = f(**inputs)\n ^^^^^^^^^^^\n', ' File "/workspaces/DepthAnythingV3-1231/.venv/lib/python3.11/site-packages/comfy_env/isolation/metadata.py", line 556, in proxy\n result = worker.call_method(\n ^^^^^^^^^^^^^^^^^^^\n', ' File "/workspaces/DepthAnythingV3-1231/.venv/lib/python3.11/site-packages/comfy_env/isolation/workers/subprocess.py", line 674, in call_method\n raise WorkerError(\n'], 'current_inputs': {'normalization_mode': ['Standard'], 'chunk_size': [30], 'overlap': [8], 'align_lib': ['auto'], 'align_method': ['sim3'], 'resize_method': ['resize'], 'invert_depth': [True], 'save_pointcloud': [False], 'sample_ratio': [0.015], 'conf_threshold_coef': [0.75], 'da3_model': ["{'model_path': '/workspaces/DepthAnythingV3-1231/ComfyUI/models/depthanything3/da3_large.safetensors', 'model_key': 'da3-large', 'dtype': 'bf16', 'attention': 'auto'}"], 'video': ['<comfy_api.latest._input_impl.video_types.VideoFromFile object at 0x7b7a6df75510>'], 'salad_model': ["{'checkpoint_path': '/workspaces/DepthAnythingV3-1231/ComfyUI/models/salad/dino_salad.safetensors'}"]}, 'current_outputs': ['6', '3', '2', '5', '4'], 'timestamp': 1778676069429}
|
|
Details: Workflow: /node/ComfyUI-DepthAnythingV3/workflows/da3_streaming.json
|
|
Node error: DepthAnythingV3_Streaming
|
|
[ComfyUI] File "/workspaces/DepthAnythingV3-1231/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/depth_anything_v3/model.py", line 891, in forward
|
|
[ComfyUI] return self.pretrained.get_intermediate_layers(
|
|
[ComfyUI] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
[ComfyUI] File "/workspaces/DepthAnythingV3-1231/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/depth_anything_v3/model.py", line 800, in get_intermediate_layers
|
|
[ComfyUI] outputs, aux_outputs = self._get_intermediate_layers_not_chunked(
|
|
[ComfyUI] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
[ComfyUI] File "/workspaces/DepthAnythingV3-1231/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/depth_anything_v3/model.py", line 750, in _get_intermediate_layers_not_chunked
|
|
[ComfyUI] x = self.process_attention(x, blk, "local", pos=l_pos)
|
|
[ComfyUI] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
[ComfyUI] File "/workspaces/DepthAnythingV3-1231/ComfyUI/custom_nodes/ComfyUI-DepthAnythingV3/nodes/depth_anything_v3/model.py", line 791, in process_attention |